2024-04-24T23:56:48.426 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-24T23:56:48.426 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} --name teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi --verbose --archive /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672232 --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672232/orig.config.yaml 2024-04-24T23:56:48.483 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T23:56:48.567 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672232 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7672232' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8528 sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 sleep_before_teardown: 0 subset: 122/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLSDZVNSqEv8+KGIdLUVBmLv5RtL2H9zle6dcE32r3uhkriKsN3xilDdFvBDK3PUO7IuIiLIlaChNBVaZRjPM+g= smithi139.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNYXtU6cRjuTd95twTZGUcwPNkHbecDw4IH2eNlRZM88Dv0zbzmiNwRSB6+HOW53iGWlIDxG/MGWBl3EvxfXMck= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIxjdHPjlcuLPLrWBzrP1/PbZQ+d7K44mxqLxO43O/0gGcqjDyA/FrtoTHuYpH+o3uyfWBGGLD3EHOB0TVvNOUA= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-24T23:56:48.567 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa; will attempt to use it 2024-04-24T23:56:48.568 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks 2024-04-24T23:56:48.568 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-24T23:56:48.570 INFO:teuthology.task.internal:Checking packages... 2024-04-24T23:56:48.593 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd2c45d977142e024a59b68a8f1dce8f0b9840551' 2024-04-24T23:56:48.593 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-24T23:56:48.593 INFO:teuthology.packaging:ref: None 2024-04-24T23:56:48.593 INFO:teuthology.packaging:tag: None 2024-04-24T23:56:48.593 INFO:teuthology.packaging:branch: squid 2024-04-24T23:56:48.593 INFO:teuthology.packaging:sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:56:48.593 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-24T23:56:48.806 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2445.gd2c45d97 2024-04-24T23:56:48.808 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-24T23:56:48.815 INFO:teuthology.task.internal:no buildpackages task found 2024-04-24T23:56:48.815 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-24T23:56:48.822 INFO:teuthology.task.internal:Saving configuration 2024-04-24T23:56:48.842 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-24T23:56:48.934 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T23:56:48.956 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672232', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 23:50:26.487192', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLSDZVNSqEv8+KGIdLUVBmLv5RtL2H9zle6dcE32r3uhkriKsN3xilDdFvBDK3PUO7IuIiLIlaChNBVaZRjPM+g='} 2024-04-24T23:56:48.977 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi139.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672232', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 23:50:26.490986', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNYXtU6cRjuTd95twTZGUcwPNkHbecDw4IH2eNlRZM88Dv0zbzmiNwRSB6+HOW53iGWlIDxG/MGWBl3EvxfXMck='} 2024-04-24T23:56:48.997 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672232', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 23:50:26.489216', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIxjdHPjlcuLPLrWBzrP1/PbZQ+d7K44mxqLxO43O/0gGcqjDyA/FrtoTHuYpH+o3uyfWBGGLD3EHOB0TVvNOUA='} 2024-04-24T23:56:48.997 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-24T23:56:49.003 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-24T23:56:49.003 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-24T23:56:49.003 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-24T23:56:49.003 INFO:teuthology.run_tasks:Running task console_log... 2024-04-24T23:56:49.113 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fafc06fb160>, signals=[15]) 2024-04-24T23:56:49.114 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-24T23:56:49.120 INFO:teuthology.task.internal:Opening connections... 2024-04-24T23:56:49.120 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-04-24T23:56:49.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:56:49.201 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi139.front.sepia.ceph.com 2024-04-24T23:56:49.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:56:49.274 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-04-24T23:56:49.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:56:49.348 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-24T23:56:49.355 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-04-24T23:56:49.374 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-04-24T23:56:49.374 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-04-24T23:56:49.433 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2024-04-24T23:56:49.433 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="9" 2024-04-24T23:56:49.433 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2024-04-24T23:56:49.433 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2024-04-24T23:56:49.433 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="9" 2024-04-24T23:56:49.433 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el9" 2024-04-24T23:56:49.433 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T23:56:49.433 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2024-04-24T23:56:49.433 INFO:teuthology.orchestra.run.smithi027.stdout:LOGO="fedora-logo-icon" 2024-04-24T23:56:49.433 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T23:56:49.433 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2024-04-24T23:56:49.434 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T23:56:49.434 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T23:56:49.434 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T23:56:49.434 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-04-24T23:56:49.460 DEBUG:teuthology.orchestra.run.smithi139:> uname -m 2024-04-24T23:56:49.478 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64 2024-04-24T23:56:49.478 DEBUG:teuthology.orchestra.run.smithi139:> cat /etc/os-release 2024-04-24T23:56:49.535 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="CentOS Stream" 2024-04-24T23:56:49.535 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="9" 2024-04-24T23:56:49.536 INFO:teuthology.orchestra.run.smithi139.stdout:ID="centos" 2024-04-24T23:56:49.536 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE="rhel fedora" 2024-04-24T23:56:49.536 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="9" 2024-04-24T23:56:49.536 INFO:teuthology.orchestra.run.smithi139.stdout:PLATFORM_ID="platform:el9" 2024-04-24T23:56:49.536 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T23:56:49.536 INFO:teuthology.orchestra.run.smithi139.stdout:ANSI_COLOR="0;31" 2024-04-24T23:56:49.536 INFO:teuthology.orchestra.run.smithi139.stdout:LOGO="fedora-logo-icon" 2024-04-24T23:56:49.536 INFO:teuthology.orchestra.run.smithi139.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T23:56:49.536 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://centos.org/" 2024-04-24T23:56:49.536 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T23:56:49.536 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T23:56:49.536 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T23:56:49.537 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2024-04-24T23:56:49.560 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-04-24T23:56:49.577 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-04-24T23:56:49.578 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-04-24T23:56:49.634 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-04-24T23:56:49.634 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-04-24T23:56:49.635 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-04-24T23:56:49.635 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-04-24T23:56:49.635 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-04-24T23:56:49.635 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-04-24T23:56:49.635 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T23:56:49.635 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-04-24T23:56:49.635 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-04-24T23:56:49.635 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T23:56:49.635 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-04-24T23:56:49.635 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T23:56:49.635 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T23:56:49.635 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T23:56:49.636 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-04-24T23:56:49.660 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-24T23:56:49.671 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-24T23:56:49.749 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-24T23:56:49.749 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-04-24T23:56:49.751 DEBUG:teuthology.orchestra.run.smithi139:> test '!' -e /home/ubuntu/cephtest 2024-04-24T23:56:49.754 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-04-24T23:56:49.771 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-24T23:56:49.829 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-24T23:56:49.830 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-04-24T23:56:49.832 DEBUG:teuthology.orchestra.run.smithi139:> test -z $(ls -A /var/lib/ceph) 2024-04-24T23:56:49.834 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-04-24T23:56:50.031 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-24T23:56:50.110 INFO:teuthology.run_tasks:Running task kernel... 2024-04-24T23:56:50.124 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-04-24T23:56:50.125 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-24T23:56:50.125 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-24T23:56:50.125 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T23:56:50.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:56:50.143 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-04-24T23:56:50.201 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-437.el9.x86_64 2024-04-24T23:56:50.201 INFO:teuthology.task.kernel:Running kernel on smithi027: 5.14.0-437.el9.x86_64 2024-04-24T23:56:50.201 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum install -y kernel 2024-04-24T23:56:52.542 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-24T23:56:55.460 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-04-24T23:56:59.844 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.2 MB 00:00 2024-04-24T23:57:01.286 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 16 kB 00:00 2024-04-24T23:57:03.556 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 9.7 MB/s | 21 MB 00:02 2024-04-24T23:57:08.688 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-24T23:57:10.628 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-24T23:57:10.628 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-24T23:57:10.721 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-24T23:57:10.722 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-24T23:57:10.722 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-24T23:57:10.825 DEBUG:teuthology.orchestra.run.smithi027:> echo no | sudo yum reinstall kernel || true 2024-04-24T23:57:11.386 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Apr 2024 11:57:08 PM UTC. 2024-04-24T23:57:11.516 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-24T23:57:11.522 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-24T23:57:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-24T23:57:11.523 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-24T23:57:11.523 INFO:teuthology.orchestra.run.smithi027.stdout:Reinstalling: 2024-04-24T23:57:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-24T23:57:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-24T23:57:11.523 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-24T23:57:11.523 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-24T23:57:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-24T23:57:11.523 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 255 k 2024-04-24T23:57:11.523 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 0 2024-04-24T23:57:11.523 INFO:teuthology.orchestra.run.smithi027.stderr:Operation aborted. 2024-04-24T23:57:11.590 INFO:teuthology.orchestra.run.smithi027.stdout:Is this ok [y/N]: 2024-04-24T23:57:11.590 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum reinstall -y kernel || true 2024-04-24T23:57:12.105 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:04 ago on Wed 24 Apr 2024 11:57:08 PM UTC. 2024-04-24T23:57:12.225 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-24T23:57:12.227 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-24T23:57:12.227 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-24T23:57:12.227 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-24T23:57:12.227 INFO:teuthology.orchestra.run.smithi027.stdout:Reinstalling: 2024-04-24T23:57:12.227 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-24T23:57:12.227 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-24T23:57:12.228 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-24T23:57:12.228 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-24T23:57:12.228 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-24T23:57:12.228 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 255 k 2024-04-24T23:57:12.228 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 0 2024-04-24T23:57:12.228 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-24T23:57:12.776 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-437.el9.x86_64.rpm 921 kB/s | 255 kB 00:00 2024-04-24T23:57:12.776 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-24T23:57:12.776 INFO:teuthology.orchestra.run.smithi027.stdout:Total 464 kB/s | 255 kB 00:00 2024-04-24T23:57:12.794 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-24T23:57:12.986 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-24T23:57:12.986 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-24T23:57:12.991 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-24T23:57:12.992 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-24T23:57:13.131 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-24T23:57:13.224 INFO:teuthology.orchestra.run.smithi027.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-24T23:57:13.225 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T23:57:13.902 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T23:57:13.902 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-24T23:57:14.639 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T23:57:14.639 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-24T23:57:14.639 INFO:teuthology.orchestra.run.smithi027.stdout:Reinstalled: 2024-04-24T23:57:14.639 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-24T23:57:14.639 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-24T23:57:14.639 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-24T23:57:14.803 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-04-24T23:57:14.851 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-24T23:57:14.851 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-24T23:57:14.851 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-24T23:57:14.851 DEBUG:teuthology.parallel:result is None 2024-04-24T23:57:14.852 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-24T23:57:14.863 INFO:teuthology.task.internal:Creating test directory... 2024-04-24T23:57:14.863 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T23:57:14.865 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T23:57:14.868 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T23:57:14.886 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-24T23:57:14.892 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-24T23:57:14.903 INFO:teuthology.task.internal:Creating archive directory... 2024-04-24T23:57:14.903 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T23:57:14.921 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T23:57:14.926 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T23:57:15.024 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-24T23:57:15.031 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-24T23:57:15.031 DEBUG:teuthology.orchestra.run.smithi139:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T23:57:15.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:57:15.047 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T23:57:15.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:57:15.063 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T23:57:15.065 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T23:57:15.090 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T23:57:15.106 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:57:15.125 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:57:15.139 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:57:15.141 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:57:15.157 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:57:15.158 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:57:15.160 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-24T23:57:15.167 INFO:teuthology.task.internal:Configuring sudo... 2024-04-24T23:57:15.167 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T23:57:15.169 DEBUG:teuthology.orchestra.run.smithi139:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T23:57:15.172 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T23:57:15.227 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-24T23:57:15.236 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-24T23:57:15.236 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T23:57:15.238 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T23:57:15.240 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T23:57:15.283 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:57:15.333 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:57:15.393 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:57:15.429 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:57:15.474 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-24T23:57:15.474 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T23:57:15.538 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:57:15.580 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:57:15.633 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:57:15.716 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:57:15.759 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-24T23:57:15.759 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T23:57:15.825 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:57:15.871 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:57:15.918 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:57:16.001 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:57:16.042 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-24T23:57:16.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T23:57:16.107 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-04-24T23:57:16.109 DEBUG:teuthology.orchestra.run.smithi139:> sudo service rsyslog restart 2024-04-24T23:57:16.111 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-04-24T23:57:16.140 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:57:16.143 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:57:16.182 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:57:16.560 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-24T23:57:16.568 INFO:teuthology.task.internal:Starting timer... 2024-04-24T23:57:16.568 INFO:teuthology.run_tasks:Running task pcp... 2024-04-24T23:57:16.578 INFO:teuthology.run_tasks:Running task selinux... 2024-04-24T23:57:16.586 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-24T23:57:16.586 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2024-04-24T23:57:16.633 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2024-04-24T23:57:16.635 DEBUG:teuthology.orchestra.run.smithi139:> sudo service auditd rotate 2024-04-24T23:57:16.698 INFO:teuthology.orchestra.run.smithi139.stdout:Rotating logs: 2024-04-24T23:57:16.700 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-04-24T23:57:16.759 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-04-24T23:57:16.761 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-24T23:57:16.761 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2024-04-24T23:57:16.794 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2024-04-24T23:57:16.795 DEBUG:teuthology.orchestra.run.smithi139:> /usr/sbin/getenforce 2024-04-24T23:57:16.835 INFO:teuthology.orchestra.run.smithi139.stdout:Permissive 2024-04-24T23:57:16.836 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-04-24T23:57:16.890 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-04-24T23:57:16.891 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi139.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive'} 2024-04-24T23:57:16.891 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T23:57:16.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:57:16.917 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T23:57:16.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:57:16.944 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T23:57:16.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:57:16.969 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-24T23:57:16.969 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2024-04-24T23:57:16.994 DEBUG:teuthology.orchestra.run.smithi139:> sudo /usr/sbin/setenforce permissive 2024-04-24T23:57:17.022 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-04-24T23:57:17.047 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-24T23:57:17.057 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-24T23:57:17.073 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-24T23:57:17.075 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi139.front.sepia.ceph.com,smithi153.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T00:01:16.725 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi139.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com')] 2024-04-25T00:01:16.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-25T00:01:16.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T00:01:16.809 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-04-25T00:01:16.890 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-25T00:01:16.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-04-25T00:01:16.891 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T00:01:16.967 DEBUG:teuthology.orchestra.run.smithi139:> true 2024-04-25T00:01:17.055 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-04-25T00:01:17.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-25T00:01:17.056 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T00:01:17.130 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-04-25T00:01:17.221 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-25T00:01:17.222 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T00:01:17.243 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T00:01:17.243 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T00:01:17.244 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T00:01:17.246 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T00:01:17.246 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T00:01:17.248 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T00:01:17.249 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T00:01:17.275 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T00:01:17.278 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T00:01:17.290 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T00:01:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T00:01:17.293 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T00:01:17.306 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T00:01:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2024-04-25T00:01:17.317 INFO:teuthology.orchestra.run.smithi139.stderr:sudo: ntpd: command not found 2024-04-25T00:01:17.325 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2024-04-25T00:01:17.328 INFO:teuthology.orchestra.run.smithi139.stdout:506 Cannot talk to daemon 2024-04-25T00:01:17.329 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-04-25T00:01:17.339 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-04-25T00:01:17.341 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T00:01:17.342 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T00:01:17.352 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T00:01:17.353 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T00:01:17.355 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T00:01:17.365 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T00:01:17.413 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-04-25T00:01:17.417 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T00:01:17.417 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-04-25T00:01:17.417 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T00:01:17.417 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T00:01:17.417 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T00:01:17.417 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T00:01:17.422 INFO:teuthology.orchestra.run.smithi139.stderr:bash: line 1: ntpq: command not found 2024-04-25T00:01:17.426 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-04-25T00:01:17.427 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T00:01:17.427 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2024-04-25T00:01:17.428 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T00:01:17.428 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T00:01:17.428 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T00:01:17.428 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T00:01:17.429 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T00:01:17.429 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-04-25T00:01:17.429 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T00:01:17.429 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T00:01:17.429 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T00:01:17.429 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T00:01:17.430 INFO:teuthology.run_tasks:Running task install... 2024-04-25T00:01:17.479 DEBUG:teuthology.task.install:project ceph 2024-04-25T00:01:17.479 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}} 2024-04-25T00:01:17.479 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'} 2024-04-25T00:01:17.479 INFO:teuthology.task.install:Using flavor: default 2024-04-25T00:01:17.485 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T00:01:17.486 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T00:01:17.486 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T00:01:17.486 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:01:17.489 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T00:01:17.489 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:01:17.490 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T00:01:17.490 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:01:17.711 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T00:01:17.711 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T00:01:17.732 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T00:01:17.733 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T00:01:17.743 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T00:01:17.743 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T00:01:17.828 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T00:01:17.828 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:01:17.828 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T00:01:17.855 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T00:01:17.855 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T00:01:17.855 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T00:01:17.856 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T00:01:17.856 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T00:01:17.881 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T00:01:17.881 DEBUG:teuthology.orchestra.run.smithi139:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T00:01:17.903 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T00:01:17.903 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T00:01:17.903 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T00:01:17.922 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T00:01:17.927 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T00:01:17.927 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T00:01:17.947 DEBUG:teuthology.orchestra.run.smithi139:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T00:01:17.991 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T00:01:17.996 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T00:01:18.020 DEBUG:teuthology.orchestra.run.smithi139:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T00:01:18.060 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2024-04-25T00:01:18.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2024-04-25T00:01:18.063 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T00:01:18.083 INFO:teuthology.orchestra.run.smithi139.stdout:check_obsoletes = 1 2024-04-25T00:01:18.084 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum clean all 2024-04-25T00:01:18.127 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-04-25T00:01:18.129 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-04-25T00:01:18.296 INFO:teuthology.orchestra.run.smithi027.stdout:48 files removed 2024-04-25T00:01:18.305 INFO:teuthology.orchestra.run.smithi139.stdout:48 files removed 2024-04-25T00:01:18.324 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T00:01:18.332 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T00:01:18.353 INFO:teuthology.orchestra.run.smithi153.stdout:48 files removed 2024-04-25T00:01:18.380 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T00:01:18.981 INFO:teuthology.orchestra.run.smithi139.stdout:ceph packages for x86_64 215 kB/s | 83 kB 00:00 2024-04-25T00:01:19.057 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 201 kB/s | 83 kB 00:00 2024-04-25T00:01:19.075 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 174 kB/s | 83 kB 00:00 2024-04-25T00:01:19.406 INFO:teuthology.orchestra.run.smithi139.stdout:ceph noarch packages 30 kB/s | 11 kB 00:00 2024-04-25T00:01:19.415 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 37 kB/s | 11 kB 00:00 2024-04-25T00:01:19.459 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 34 kB/s | 11 kB 00:00 2024-04-25T00:01:19.724 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 6.9 kB/s | 1.9 kB 00:00 2024-04-25T00:01:19.740 INFO:teuthology.orchestra.run.smithi139.stdout:ceph source packages 6.3 kB/s | 1.9 kB 00:00 2024-04-25T00:01:19.835 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-04-25T00:01:20.376 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-04-25T00:01:23.798 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:04 2024-04-25T00:01:26.691 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-04-25T00:01:28.457 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 947 kB/s | 8.0 MB 00:08 2024-04-25T00:01:31.866 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.2 MB 00:01 2024-04-25T00:01:32.091 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 7.4 MB/s | 19 MB 00:02 2024-04-25T00:01:37.743 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:16 2024-04-25T00:01:39.658 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.2 MB 00:03 2024-04-25T00:01:43.719 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.2 MB 00:02 2024-04-25T00:01:44.877 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 4.0 kB/s | 16 kB 00:04 2024-04-25T00:01:46.929 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 8.2 kB/s | 16 kB 00:01 2024-04-25T00:01:47.281 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - Extras packages 1.1 kB/s | 16 kB 00:14 2024-04-25T00:01:51.003 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 3.5 MB/s | 21 MB 00:06 2024-04-25T00:01:53.040 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 3.5 MB/s | 21 MB 00:05 2024-04-25T00:01:53.891 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 3.2 MB/s | 21 MB 00:06 2024-04-25T00:01:55.961 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-04-25T00:01:58.071 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T00:01:58.071 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T00:01:58.081 INFO:teuthology.orchestra.run.smithi153.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T00:01:58.082 INFO:teuthology.orchestra.run.smithi153.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T00:01:58.083 INFO:teuthology.orchestra.run.smithi153.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T00:01:58.084 INFO:teuthology.orchestra.run.smithi153.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T00:01:58.085 INFO:teuthology.orchestra.run.smithi153.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T00:01:58.088 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T00:01:58.089 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T00:01:58.148 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-25T00:01:58.159 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-04-25T00:01:58.159 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-25T00:01:58.159 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-04-25T00:01:58.159 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-25T00:01:58.159 INFO:teuthology.orchestra.run.smithi153.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T00:01:58.159 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T00:01:58.159 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T00:01:58.159 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T00:01:58.159 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T00:01:58.159 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T00:01:58.159 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T00:01:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T00:01:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T00:01:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T00:01:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T00:01:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T00:01:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T00:01:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T00:01:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T00:01:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T00:01:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T00:01:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T00:01:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T00:01:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T00:01:58.160 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T00:01:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T00:01:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T00:01:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T00:01:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T00:01:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T00:01:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T00:01:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T00:01:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T00:01:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T00:01:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T00:01:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T00:01:58.161 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T00:01:58.162 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T00:01:58.163 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T00:01:58.163 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T00:01:58.163 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T00:01:58.164 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-04-25T00:01:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T00:01:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T00:01:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T00:01:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T00:01:58.164 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-25T00:01:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T00:01:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T00:01:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T00:01:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T00:01:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T00:01:58.164 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T00:01:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T00:01:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T00:01:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T00:01:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T00:01:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T00:01:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T00:01:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T00:01:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T00:01:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T00:01:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T00:01:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T00:01:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T00:01:58.165 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T00:01:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T00:01:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T00:01:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T00:01:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T00:01:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T00:01:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T00:01:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T00:01:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T00:01:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T00:01:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T00:01:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T00:01:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T00:01:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T00:01:58.166 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T00:01:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T00:01:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T00:01:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T00:01:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T00:01:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T00:01:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T00:01:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T00:01:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T00:01:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T00:01:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T00:01:58.169 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T00:01:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T00:01:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T00:01:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T00:01:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T00:01:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T00:01:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T00:01:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T00:01:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T00:01:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T00:01:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T00:01:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T00:01:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T00:01:58.170 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T00:01:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T00:01:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T00:01:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T00:01:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T00:01:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T00:01:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T00:01:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T00:01:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T00:01:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T00:01:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T00:01:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T00:01:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T00:01:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T00:01:58.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T00:01:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T00:01:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T00:01:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T00:01:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T00:01:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T00:01:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T00:01:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T00:01:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T00:01:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T00:01:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T00:01:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T00:01:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T00:01:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T00:01:58.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T00:01:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T00:01:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T00:01:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T00:01:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T00:01:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T00:01:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T00:01:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T00:01:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T00:01:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T00:01:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T00:01:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T00:01:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T00:01:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T00:01:58.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T00:01:58.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T00:01:58.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T00:01:58.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T00:01:58.174 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T00:01:58.174 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T00:01:58.174 INFO:teuthology.orchestra.run.smithi153.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T00:01:58.174 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T00:01:58.174 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T00:01:58.174 INFO:teuthology.orchestra.run.smithi153.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T00:01:58.174 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T00:01:58.174 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T00:01:58.174 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-04-25T00:01:58.174 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T00:01:58.174 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:01:58.175 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-25T00:01:58.175 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-04-25T00:01:58.175 INFO:teuthology.orchestra.run.smithi153.stdout:Install 148 Packages 2024-04-25T00:01:58.175 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 4 Packages 2024-04-25T00:01:58.175 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:01:58.175 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 220 M 2024-04-25T00:01:58.176 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-25T00:01:58.246 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-04-25T00:01:58.882 INFO:teuthology.orchestra.run.smithi139.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-25T00:02:00.472 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T00:02:00.473 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T00:02:00.484 INFO:teuthology.orchestra.run.smithi027.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T00:02:00.485 INFO:teuthology.orchestra.run.smithi027.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T00:02:00.486 INFO:teuthology.orchestra.run.smithi027.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T00:02:00.487 INFO:teuthology.orchestra.run.smithi027.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T00:02:00.489 INFO:teuthology.orchestra.run.smithi027.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T00:02:00.491 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T00:02:00.492 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T00:02:00.558 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-25T00:02:00.567 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-04-25T00:02:00.567 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-25T00:02:00.567 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-04-25T00:02:00.567 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-25T00:02:00.567 INFO:teuthology.orchestra.run.smithi027.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T00:02:00.567 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T00:02:00.568 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T00:02:00.568 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T00:02:00.568 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T00:02:00.568 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T00:02:00.568 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T00:02:00.568 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T00:02:00.568 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T00:02:00.568 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T00:02:00.568 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T00:02:00.568 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T00:02:00.568 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T00:02:00.568 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T00:02:00.568 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T00:02:00.569 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T00:02:00.569 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T00:02:00.569 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T00:02:00.569 INFO:teuthology.orchestra.run.smithi027.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T00:02:00.569 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T00:02:00.569 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T00:02:00.569 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T00:02:00.569 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T00:02:00.569 INFO:teuthology.orchestra.run.smithi027.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T00:02:00.569 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T00:02:00.569 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T00:02:00.569 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T00:02:00.569 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T00:02:00.569 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T00:02:00.570 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T00:02:00.570 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T00:02:00.570 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T00:02:00.570 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T00:02:00.570 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T00:02:00.570 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T00:02:00.570 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T00:02:00.570 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T00:02:00.570 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-04-25T00:02:00.570 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T00:02:00.570 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T00:02:00.570 INFO:teuthology.orchestra.run.smithi027.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T00:02:00.571 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T00:02:00.571 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-25T00:02:00.571 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T00:02:00.571 INFO:teuthology.orchestra.run.smithi027.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T00:02:00.571 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T00:02:00.571 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T00:02:00.571 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T00:02:00.571 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T00:02:00.571 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T00:02:00.571 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T00:02:00.571 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T00:02:00.571 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T00:02:00.571 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T00:02:00.571 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T00:02:00.571 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T00:02:00.572 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T00:02:00.572 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T00:02:00.572 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T00:02:00.572 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T00:02:00.572 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T00:02:00.572 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T00:02:00.572 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T00:02:00.572 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T00:02:00.572 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T00:02:00.572 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T00:02:00.572 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T00:02:00.572 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T00:02:00.572 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T00:02:00.572 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T00:02:00.573 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T00:02:00.573 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T00:02:00.573 INFO:teuthology.orchestra.run.smithi027.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T00:02:00.573 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T00:02:00.573 INFO:teuthology.orchestra.run.smithi027.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T00:02:00.573 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T00:02:00.573 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T00:02:00.573 INFO:teuthology.orchestra.run.smithi027.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T00:02:00.573 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T00:02:00.573 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T00:02:00.573 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T00:02:00.573 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T00:02:00.573 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T00:02:00.573 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T00:02:00.574 INFO:teuthology.orchestra.run.smithi027.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T00:02:00.574 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T00:02:00.574 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T00:02:00.574 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T00:02:00.574 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T00:02:00.574 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T00:02:00.574 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T00:02:00.574 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T00:02:00.574 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T00:02:00.574 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T00:02:00.574 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T00:02:00.574 INFO:teuthology.orchestra.run.smithi027.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T00:02:00.574 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T00:02:00.574 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T00:02:00.575 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T00:02:00.575 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T00:02:00.575 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T00:02:00.575 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T00:02:00.575 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T00:02:00.575 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T00:02:00.575 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T00:02:00.575 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T00:02:00.575 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T00:02:00.575 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T00:02:00.575 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T00:02:00.575 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T00:02:00.575 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T00:02:00.575 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T00:02:00.576 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T00:02:00.576 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T00:02:00.576 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T00:02:00.576 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T00:02:00.576 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T00:02:00.576 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T00:02:00.576 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T00:02:00.576 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T00:02:00.576 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T00:02:00.576 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T00:02:00.576 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T00:02:00.576 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T00:02:00.576 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T00:02:00.577 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T00:02:00.577 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T00:02:00.577 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T00:02:00.577 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T00:02:00.577 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T00:02:00.577 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T00:02:00.577 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T00:02:00.577 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T00:02:00.577 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T00:02:00.577 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T00:02:00.577 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T00:02:00.577 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T00:02:00.577 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T00:02:00.577 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T00:02:00.577 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T00:02:00.578 INFO:teuthology.orchestra.run.smithi027.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T00:02:00.578 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T00:02:00.578 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T00:02:00.578 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T00:02:00.578 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T00:02:00.578 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T00:02:00.578 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T00:02:00.578 INFO:teuthology.orchestra.run.smithi027.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T00:02:00.578 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T00:02:00.578 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T00:02:00.578 INFO:teuthology.orchestra.run.smithi027.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T00:02:00.578 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T00:02:00.578 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T00:02:00.579 INFO:teuthology.orchestra.run.smithi027.stdout:Installing weak dependencies: 2024-04-25T00:02:00.579 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T00:02:00.579 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:02:00.579 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-25T00:02:00.579 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-04-25T00:02:00.579 INFO:teuthology.orchestra.run.smithi027.stdout:Install 148 Packages 2024-04-25T00:02:00.579 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 4 Packages 2024-04-25T00:02:00.579 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:02:00.580 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 220 M 2024-04-25T00:02:00.580 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-25T00:02:00.991 INFO:teuthology.orchestra.run.smithi139.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T00:02:00.991 INFO:teuthology.orchestra.run.smithi139.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T00:02:01.002 INFO:teuthology.orchestra.run.smithi139.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T00:02:01.002 INFO:teuthology.orchestra.run.smithi139.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T00:02:01.003 INFO:teuthology.orchestra.run.smithi139.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T00:02:01.004 INFO:teuthology.orchestra.run.smithi139.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T00:02:01.006 INFO:teuthology.orchestra.run.smithi139.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T00:02:01.008 INFO:teuthology.orchestra.run.smithi139.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T00:02:01.009 INFO:teuthology.orchestra.run.smithi139.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T00:02:01.069 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-04-25T00:02:01.082 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2024-04-25T00:02:01.082 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-04-25T00:02:01.083 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2024-04-25T00:02:01.083 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-04-25T00:02:01.083 INFO:teuthology.orchestra.run.smithi139.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T00:02:01.083 INFO:teuthology.orchestra.run.smithi139.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T00:02:01.083 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T00:02:01.083 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T00:02:01.083 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T00:02:01.083 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T00:02:01.083 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T00:02:01.083 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T00:02:01.083 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T00:02:01.083 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T00:02:01.083 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T00:02:01.083 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T00:02:01.083 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T00:02:01.084 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T00:02:01.084 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T00:02:01.084 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T00:02:01.084 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T00:02:01.084 INFO:teuthology.orchestra.run.smithi139.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T00:02:01.084 INFO:teuthology.orchestra.run.smithi139.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T00:02:01.084 INFO:teuthology.orchestra.run.smithi139.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T00:02:01.084 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T00:02:01.084 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T00:02:01.084 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T00:02:01.084 INFO:teuthology.orchestra.run.smithi139.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T00:02:01.084 INFO:teuthology.orchestra.run.smithi139.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T00:02:01.084 INFO:teuthology.orchestra.run.smithi139.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T00:02:01.084 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T00:02:01.084 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T00:02:01.085 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T00:02:01.085 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T00:02:01.085 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T00:02:01.085 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T00:02:01.085 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T00:02:01.085 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T00:02:01.085 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T00:02:01.085 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T00:02:01.085 INFO:teuthology.orchestra.run.smithi139.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T00:02:01.085 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrading: 2024-04-25T00:02:01.085 INFO:teuthology.orchestra.run.smithi139.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T00:02:01.085 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T00:02:01.085 INFO:teuthology.orchestra.run.smithi139.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T00:02:01.085 INFO:teuthology.orchestra.run.smithi139.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T00:02:01.086 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-04-25T00:02:01.086 INFO:teuthology.orchestra.run.smithi139.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T00:02:01.086 INFO:teuthology.orchestra.run.smithi139.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T00:02:01.086 INFO:teuthology.orchestra.run.smithi139.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T00:02:01.086 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T00:02:01.086 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T00:02:01.086 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T00:02:01.086 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T00:02:01.086 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T00:02:01.086 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T00:02:01.086 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T00:02:01.086 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T00:02:01.086 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T00:02:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T00:02:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T00:02:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T00:02:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T00:02:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T00:02:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T00:02:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T00:02:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T00:02:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T00:02:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T00:02:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T00:02:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T00:02:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T00:02:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T00:02:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T00:02:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T00:02:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T00:02:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T00:02:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T00:02:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T00:02:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T00:02:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T00:02:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T00:02:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T00:02:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T00:02:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T00:02:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T00:02:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T00:02:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T00:02:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T00:02:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T00:02:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T00:02:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T00:02:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T00:02:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T00:02:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T00:02:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T00:02:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T00:02:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T00:02:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T00:02:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T00:02:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T00:02:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T00:02:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T00:02:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T00:02:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T00:02:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T00:02:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T00:02:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T00:02:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T00:02:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T00:02:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T00:02:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T00:02:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T00:02:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T00:02:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T00:02:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T00:02:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T00:02:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T00:02:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T00:02:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T00:02:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T00:02:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T00:02:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T00:02:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T00:02:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T00:02:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T00:02:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T00:02:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T00:02:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T00:02:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T00:02:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T00:02:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T00:02:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T00:02:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T00:02:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T00:02:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T00:02:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T00:02:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T00:02:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T00:02:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T00:02:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T00:02:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T00:02:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T00:02:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T00:02:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T00:02:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T00:02:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T00:02:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T00:02:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T00:02:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T00:02:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T00:02:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T00:02:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T00:02:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T00:02:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T00:02:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T00:02:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T00:02:01.094 INFO:teuthology.orchestra.run.smithi139.stdout:Installing weak dependencies: 2024-04-25T00:02:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T00:02:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T00:02:01.094 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-04-25T00:02:01.094 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2024-04-25T00:02:01.094 INFO:teuthology.orchestra.run.smithi139.stdout:Install 148 Packages 2024-04-25T00:02:01.094 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrade 4 Packages 2024-04-25T00:02:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T00:02:01.095 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 220 M 2024-04-25T00:02:01.095 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-04-25T00:02:10.645 INFO:teuthology.orchestra.run.smithi153.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-04-25T00:02:10.861 INFO:teuthology.orchestra.run.smithi153.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 4.0 MB/s | 890 kB 00:00 2024-04-25T00:02:10.920 INFO:teuthology.orchestra.run.smithi153.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 2.4 MB/s | 145 kB 00:00 2024-04-25T00:02:11.187 INFO:teuthology.orchestra.run.smithi153.stdout:(4/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 8.9 MB/s | 2.4 MB 00:00 2024-04-25T00:02:11.387 INFO:teuthology.orchestra.run.smithi153.stdout:(5/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 7.5 MB/s | 1.5 MB 00:00 2024-04-25T00:02:11.929 INFO:teuthology.orchestra.run.smithi153.stdout:(6/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 8.7 MB/s | 4.7 MB 00:00 2024-04-25T00:02:12.163 INFO:teuthology.orchestra.run.smithi153.stdout:(7/152): ceph-common-19.0.0-2445.gd2c45d97.el9. 12 MB/s | 21 MB 00:01 2024-04-25T00:02:12.396 INFO:teuthology.orchestra.run.smithi153.stdout:(8/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 2.8 MB/s | 5.4 MB 00:01 2024-04-25T00:02:12.488 INFO:teuthology.orchestra.run.smithi153.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 275 kB/s | 25 kB 00:00 2024-04-25T00:02:12.680 INFO:teuthology.orchestra.run.smithi027.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-04-25T00:02:12.863 INFO:teuthology.orchestra.run.smithi153.stdout:(10/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 15 MB/s | 10 MB 00:00 2024-04-25T00:02:12.956 INFO:teuthology.orchestra.run.smithi153.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 2.4 MB/s | 222 kB 00:00 2024-04-25T00:02:13.022 INFO:teuthology.orchestra.run.smithi153.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 406 kB/s | 27 kB 00:00 2024-04-25T00:02:13.114 INFO:teuthology.orchestra.run.smithi153.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 345 kB/s | 32 kB 00:00 2024-04-25T00:02:13.130 INFO:teuthology.orchestra.run.smithi027.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.9 MB/s | 890 kB 00:00 2024-04-25T00:02:13.198 INFO:teuthology.orchestra.run.smithi153.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 8.7 MB/s | 745 kB 00:00 2024-04-25T00:02:13.273 INFO:teuthology.orchestra.run.smithi153.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 2.1 MB/s | 163 kB 00:00 2024-04-25T00:02:13.323 INFO:teuthology.orchestra.run.smithi153.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 2.5 MB/s | 127 kB 00:00 2024-04-25T00:02:13.346 INFO:teuthology.orchestra.run.smithi027.stdout:(3/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 5.6 MB/s | 5.4 MB 00:00 2024-04-25T00:02:13.390 INFO:teuthology.orchestra.run.smithi153.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 7.3 MB/s | 501 kB 00:00 2024-04-25T00:02:13.822 INFO:teuthology.orchestra.run.smithi027.stdout:(4/152): ceph-immutable-object-cache-19.0.0-244 209 kB/s | 145 kB 00:00 2024-04-25T00:02:13.824 INFO:teuthology.orchestra.run.smithi153.stdout:(18/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 12 MB/s | 5.3 MB 00:00 2024-04-25T00:02:13.883 INFO:teuthology.orchestra.run.smithi153.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 776 kB/s | 45 kB 00:00 2024-04-25T00:02:13.949 INFO:teuthology.orchestra.run.smithi153.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 2.0 MB/s | 133 kB 00:00 2024-04-25T00:02:14.016 INFO:teuthology.orchestra.run.smithi153.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 2.4 MB/s | 162 kB 00:00 2024-04-25T00:02:14.092 INFO:teuthology.orchestra.run.smithi153.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 4.2 MB/s | 321 kB 00:00 2024-04-25T00:02:14.167 INFO:teuthology.orchestra.run.smithi153.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 3.9 MB/s | 299 kB 00:00 2024-04-25T00:02:14.242 INFO:teuthology.orchestra.run.smithi153.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.3 MB/s | 99 kB 00:00 2024-04-25T00:02:14.272 INFO:teuthology.orchestra.run.smithi027.stdout:(5/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 2.6 MB/s | 2.4 MB 00:00 2024-04-25T00:02:14.301 INFO:teuthology.orchestra.run.smithi153.stdout:(25/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.4 MB/s | 85 kB 00:00 2024-04-25T00:02:14.468 INFO:teuthology.orchestra.run.smithi153.stdout:(26/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x8 6.6 MB/s | 17 MB 00:02 2024-04-25T00:02:14.489 INFO:teuthology.orchestra.run.smithi027.stdout:(6/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 2.2 MB/s | 1.5 MB 00:00 2024-04-25T00:02:14.560 INFO:teuthology.orchestra.run.smithi153.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 1.8 MB/s | 171 kB 00:00 2024-04-25T00:02:14.635 INFO:teuthology.orchestra.run.smithi153.stdout:(28/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 9.3 MB/s | 3.1 MB 00:00 2024-04-25T00:02:14.661 INFO:teuthology.orchestra.run.smithi153.stdout:(29/152): ceph-grafana-dashboards-19.0.0-2445.g 317 kB/s | 32 kB 00:00 2024-04-25T00:02:14.711 INFO:teuthology.orchestra.run.smithi153.stdout:(30/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.8 MB/s | 141 kB 00:00 2024-04-25T00:02:15.537 INFO:teuthology.orchestra.run.smithi153.stdout:(31/152): ceph-mgr-diskprediction-local-19.0.0- 9.0 MB/s | 7.4 MB 00:00 2024-04-25T00:02:15.573 INFO:teuthology.orchestra.run.smithi027.stdout:(7/152): ceph-common-19.0.0-2445.gd2c45d97.el9. 6.6 MB/s | 21 MB 00:03 2024-04-25T00:02:15.619 INFO:teuthology.orchestra.run.smithi153.stdout:(32/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 3.7 MB/s | 3.5 MB 00:00 2024-04-25T00:02:15.645 INFO:teuthology.orchestra.run.smithi153.stdout:(33/152): ceph-mgr-modules-core-19.0.0-2445.gd2 2.2 MB/s | 249 kB 00:00 2024-04-25T00:02:15.670 INFO:teuthology.orchestra.run.smithi153.stdout:(34/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 981 kB/s | 49 kB 00:00 2024-04-25T00:02:15.681 INFO:teuthology.orchestra.run.smithi027.stdout:(8/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 3.3 MB/s | 4.7 MB 00:01 2024-04-25T00:02:15.695 INFO:teuthology.orchestra.run.smithi153.stdout:(35/152): ceph-prometheus-alerts-19.0.0-2445.gd 307 kB/s | 15 kB 00:00 2024-04-25T00:02:15.748 INFO:teuthology.orchestra.run.smithi027.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 376 kB/s | 25 kB 00:00 2024-04-25T00:02:15.762 INFO:teuthology.orchestra.run.smithi153.stdout:(36/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 2.8 MB/s | 264 kB 00:00 2024-04-25T00:02:15.837 INFO:teuthology.orchestra.run.smithi153.stdout:(37/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 5.2 MB/s | 762 kB 00:00 2024-04-25T00:02:15.863 INFO:teuthology.orchestra.run.smithi153.stdout:(38/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 182 kB/s | 18 kB 00:00 2024-04-25T00:02:16.055 INFO:teuthology.orchestra.run.smithi153.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 370 kB/s | 80 kB 00:00 2024-04-25T00:02:16.172 INFO:teuthology.orchestra.run.smithi153.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-04-25T00:02:16.205 INFO:teuthology.orchestra.run.smithi153.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-04-25T00:02:16.239 INFO:teuthology.orchestra.run.smithi153.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-04-25T00:02:16.298 INFO:teuthology.orchestra.run.smithi153.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-25T00:02:16.356 INFO:teuthology.orchestra.run.smithi153.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-04-25T00:02:16.390 INFO:teuthology.orchestra.run.smithi027.stdout:(10/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 13 MB/s | 10 MB 00:00 2024-04-25T00:02:16.415 INFO:teuthology.orchestra.run.smithi153.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-25T00:02:16.457 INFO:teuthology.orchestra.run.smithi027.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 3.3 MB/s | 222 kB 00:00 2024-04-25T00:02:16.474 INFO:teuthology.orchestra.run.smithi153.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-25T00:02:16.516 INFO:teuthology.orchestra.run.smithi027.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 464 kB/s | 27 kB 00:00 2024-04-25T00:02:16.582 INFO:teuthology.orchestra.run.smithi027.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 476 kB/s | 32 kB 00:00 2024-04-25T00:02:16.674 INFO:teuthology.orchestra.run.smithi027.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 8.0 MB/s | 745 kB 00:00 2024-04-25T00:02:16.741 INFO:teuthology.orchestra.run.smithi153.stdout:(47/152): python3-cryptography-36.0.1-4.el9.x86 2.3 MB/s | 1.2 MB 00:00 2024-04-25T00:02:16.741 INFO:teuthology.orchestra.run.smithi027.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 2.4 MB/s | 163 kB 00:00 2024-04-25T00:02:16.808 INFO:teuthology.orchestra.run.smithi027.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.9 MB/s | 127 kB 00:00 2024-04-25T00:02:16.892 INFO:teuthology.orchestra.run.smithi027.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 5.9 MB/s | 501 kB 00:00 2024-04-25T00:02:17.158 INFO:teuthology.orchestra.run.smithi153.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 1.0 MB/s | 689 kB 00:00 2024-04-25T00:02:17.335 INFO:teuthology.orchestra.run.smithi027.stdout:(18/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 12 MB/s | 5.3 MB 00:00 2024-04-25T00:02:17.401 INFO:teuthology.orchestra.run.smithi027.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 684 kB/s | 45 kB 00:00 2024-04-25T00:02:17.469 INFO:teuthology.orchestra.run.smithi027.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.9 MB/s | 133 kB 00:00 2024-04-25T00:02:17.535 INFO:teuthology.orchestra.run.smithi027.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 2.4 MB/s | 162 kB 00:00 2024-04-25T00:02:17.609 INFO:teuthology.orchestra.run.smithi153.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 238 kB/s | 107 kB 00:00 2024-04-25T00:02:17.610 INFO:teuthology.orchestra.run.smithi027.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 4.2 MB/s | 321 kB 00:00 2024-04-25T00:02:17.736 INFO:teuthology.orchestra.run.smithi027.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 2.3 MB/s | 299 kB 00:00 2024-04-25T00:02:17.811 INFO:teuthology.orchestra.run.smithi027.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.3 MB/s | 99 kB 00:00 2024-04-25T00:02:17.878 INFO:teuthology.orchestra.run.smithi027.stdout:(25/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.3 MB/s | 85 kB 00:00 2024-04-25T00:02:17.883 INFO:teuthology.orchestra.run.smithi153.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 78 kB/s | 21 kB 00:00 2024-04-25T00:02:18.279 INFO:teuthology.orchestra.run.smithi027.stdout:(26/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 7.7 MB/s | 3.1 MB 00:00 2024-04-25T00:02:18.337 INFO:teuthology.orchestra.run.smithi027.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 2.9 MB/s | 171 kB 00:00 2024-04-25T00:02:18.404 INFO:teuthology.orchestra.run.smithi027.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 476 kB/s | 32 kB 00:00 2024-04-25T00:02:18.475 INFO:teuthology.orchestra.run.smithi153.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 530 kB/s | 313 kB 00:00 2024-04-25T00:02:18.534 INFO:teuthology.orchestra.run.smithi153.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 526 kB/s | 943 kB 00:01 2024-04-25T00:02:18.596 INFO:teuthology.orchestra.run.smithi027.stdout:(29/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x8 4.1 MB/s | 17 MB 00:04 2024-04-25T00:02:18.622 INFO:teuthology.orchestra.run.smithi027.stdout:(30/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 651 kB/s | 141 kB 00:00 2024-04-25T00:02:18.751 INFO:teuthology.orchestra.run.smithi153.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 119 kB/s | 33 kB 00:00 2024-04-25T00:02:19.010 INFO:teuthology.orchestra.run.smithi153.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 70 kB/s | 18 kB 00:00 2024-04-25T00:02:19.193 INFO:teuthology.orchestra.run.smithi153.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 61 kB/s | 11 kB 00:00 2024-04-25T00:02:19.213 INFO:teuthology.orchestra.run.smithi027.stdout:(31/152): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-04-25T00:02:19.305 INFO:teuthology.orchestra.run.smithi027.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 2.7 MB/s | 249 kB 00:00 2024-04-25T00:02:19.389 INFO:teuthology.orchestra.run.smithi027.stdout:(33/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 4.4 MB/s | 3.5 MB 00:00 2024-04-25T00:02:19.415 INFO:teuthology.orchestra.run.smithi027.stdout:(34/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 452 kB/s | 49 kB 00:00 2024-04-25T00:02:19.456 INFO:teuthology.orchestra.run.smithi027.stdout:(35/152): ceph-prometheus-alerts-19.0.0-2445.gd 228 kB/s | 15 kB 00:00 2024-04-25T00:02:19.499 INFO:teuthology.orchestra.run.smithi027.stdout:(36/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 3.1 MB/s | 264 kB 00:00 2024-04-25T00:02:19.590 INFO:teuthology.orchestra.run.smithi027.stdout:(37/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 199 kB/s | 18 kB 00:00 2024-04-25T00:02:19.682 INFO:teuthology.orchestra.run.smithi027.stdout:(38/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 3.3 MB/s | 762 kB 00:00 2024-04-25T00:02:19.758 INFO:teuthology.orchestra.run.smithi027.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 480 kB/s | 80 kB 00:00 2024-04-25T00:02:19.768 INFO:teuthology.orchestra.run.smithi153.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 279 kB/s | 160 kB 00:00 2024-04-25T00:02:19.841 INFO:teuthology.orchestra.run.smithi027.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-04-25T00:02:20.008 INFO:teuthology.orchestra.run.smithi027.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-04-25T00:02:20.042 INFO:teuthology.orchestra.run.smithi027.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-04-25T00:02:20.110 INFO:teuthology.orchestra.run.smithi153.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 132 kB/s | 45 kB 00:00 2024-04-25T00:02:20.133 INFO:teuthology.orchestra.run.smithi027.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-04-25T00:02:20.201 INFO:teuthology.orchestra.run.smithi027.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-25T00:02:20.284 INFO:teuthology.orchestra.run.smithi027.stdout:(45/152): python3-cryptography-36.0.1-4.el9.x86 4.4 MB/s | 1.2 MB 00:00 2024-04-25T00:02:20.310 INFO:teuthology.orchestra.run.smithi027.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-04-25T00:02:20.410 INFO:teuthology.orchestra.run.smithi027.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.7 MB/s | 215 kB 00:00 2024-04-25T00:02:20.469 INFO:teuthology.orchestra.run.smithi027.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 4.2 MB/s | 689 kB 00:00 2024-04-25T00:02:20.510 INFO:teuthology.orchestra.run.smithi027.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 107 kB 00:00 2024-04-25T00:02:20.552 INFO:teuthology.orchestra.run.smithi027.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 515 kB/s | 21 kB 00:00 2024-04-25T00:02:20.586 INFO:teuthology.orchestra.run.smithi027.stdout:(51/152): bison-3.7.4-5.el9.x86_64.rpm 5.2 MB/s | 943 kB 00:00 2024-04-25T00:02:20.611 INFO:teuthology.orchestra.run.smithi027.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-04-25T00:02:20.636 INFO:teuthology.orchestra.run.smithi027.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 653 kB/s | 33 kB 00:00 2024-04-25T00:02:20.678 INFO:teuthology.orchestra.run.smithi027.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 438 kB/s | 18 kB 00:00 2024-04-25T00:02:20.729 INFO:teuthology.orchestra.run.smithi027.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-04-25T00:02:20.754 INFO:teuthology.orchestra.run.smithi027.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 148 kB/s | 11 kB 00:00 2024-04-25T00:02:20.779 INFO:teuthology.orchestra.run.smithi027.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-25T00:02:20.804 INFO:teuthology.orchestra.run.smithi027.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 902 kB/s | 45 kB 00:00 2024-04-25T00:02:20.838 INFO:teuthology.orchestra.run.smithi027.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-25T00:02:20.888 INFO:teuthology.orchestra.run.smithi027.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 6.8 MB/s | 582 kB 00:00 2024-04-25T00:02:20.914 INFO:teuthology.orchestra.run.smithi027.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 338 kB/s | 25 kB 00:00 2024-04-25T00:02:20.939 INFO:teuthology.orchestra.run.smithi027.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-04-25T00:02:20.967 INFO:teuthology.orchestra.run.smithi027.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 292 kB 00:00 2024-04-25T00:02:20.992 INFO:teuthology.orchestra.run.smithi027.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.5 MB/s | 188 kB 00:00 2024-04-25T00:02:21.010 INFO:teuthology.orchestra.run.smithi153.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 736 kB/s | 662 kB 00:00 2024-04-25T00:02:21.017 INFO:teuthology.orchestra.run.smithi027.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 704 kB/s | 35 kB 00:00 2024-04-25T00:02:21.126 INFO:teuthology.orchestra.run.smithi027.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 36 MB/s | 4.7 MB 00:00 2024-04-25T00:02:21.193 INFO:teuthology.orchestra.run.smithi027.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 25 MB/s | 4.4 MB 00:00 2024-04-25T00:02:21.243 INFO:teuthology.orchestra.run.smithi027.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-04-25T00:02:21.285 INFO:teuthology.orchestra.run.smithi027.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 9.7 MB/s | 911 kB 00:00 2024-04-25T00:02:21.310 INFO:teuthology.orchestra.run.smithi027.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.9 MB/s | 133 kB 00:00 2024-04-25T00:02:21.361 INFO:teuthology.orchestra.run.smithi027.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 20 MB/s | 1.0 MB 00:00 2024-04-25T00:02:21.436 INFO:teuthology.orchestra.run.smithi027.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 38 MB/s | 5.8 MB 00:00 2024-04-25T00:02:21.461 INFO:teuthology.orchestra.run.smithi027.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 875 kB/s | 87 kB 00:00 2024-04-25T00:02:21.503 INFO:teuthology.orchestra.run.smithi027.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 5.8 MB/s | 244 kB 00:00 2024-04-25T00:02:21.586 INFO:teuthology.orchestra.run.smithi027.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 40 MB/s | 6.0 MB 00:00 2024-04-25T00:02:21.612 INFO:teuthology.orchestra.run.smithi027.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-04-25T00:02:21.637 INFO:teuthology.orchestra.run.smithi027.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 510 kB/s | 26 kB 00:00 2024-04-25T00:02:21.662 INFO:teuthology.orchestra.run.smithi027.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-04-25T00:02:21.696 INFO:teuthology.orchestra.run.smithi027.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-25T00:02:21.721 INFO:teuthology.orchestra.run.smithi027.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 592 kB/s | 35 kB 00:00 2024-04-25T00:02:21.789 INFO:teuthology.orchestra.run.smithi027.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-04-25T00:02:21.810 INFO:teuthology.orchestra.run.smithi153.stdout:(59/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 934 kB/s | 3.0 MB 00:03 2024-04-25T00:02:21.855 INFO:teuthology.orchestra.run.smithi027.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 5.1 MB 00:00 2024-04-25T00:02:21.881 INFO:teuthology.orchestra.run.smithi027.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-04-25T00:02:21.906 INFO:teuthology.orchestra.run.smithi027.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-25T00:02:21.923 INFO:teuthology.orchestra.run.smithi027.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-04-25T00:02:21.956 INFO:teuthology.orchestra.run.smithi027.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-25T00:02:21.982 INFO:teuthology.orchestra.run.smithi027.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 159 kB 00:00 2024-04-25T00:02:22.007 INFO:teuthology.orchestra.run.smithi027.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-25T00:02:22.032 INFO:teuthology.orchestra.run.smithi027.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-25T00:02:22.058 INFO:teuthology.orchestra.run.smithi027.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-25T00:02:22.100 INFO:teuthology.orchestra.run.smithi027.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-04-25T00:02:22.102 INFO:teuthology.orchestra.run.smithi153.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 87 kB/s | 25 kB 00:00 2024-04-25T00:02:22.142 INFO:teuthology.orchestra.run.smithi027.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-04-25T00:02:22.184 INFO:teuthology.orchestra.run.smithi027.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-04-25T00:02:22.226 INFO:teuthology.orchestra.run.smithi027.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 866 kB/s | 36 kB 00:00 2024-04-25T00:02:22.384 INFO:teuthology.orchestra.run.smithi027.stdout:(95/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-04-25T00:02:22.386 INFO:teuthology.orchestra.run.smithi153.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 423 kB/s | 582 kB 00:01 2024-04-25T00:02:22.410 INFO:teuthology.orchestra.run.smithi027.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 258 kB/s | 47 kB 00:00 2024-04-25T00:02:22.435 INFO:teuthology.orchestra.run.smithi027.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 305 kB 00:00 2024-04-25T00:02:22.453 INFO:teuthology.orchestra.run.smithi153.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 135 kB/s | 47 kB 00:00 2024-04-25T00:02:22.460 INFO:teuthology.orchestra.run.smithi027.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-25T00:02:22.486 INFO:teuthology.orchestra.run.smithi027.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-25T00:02:22.511 INFO:teuthology.orchestra.run.smithi027.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-25T00:02:22.536 INFO:teuthology.orchestra.run.smithi027.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 893 kB/s | 45 kB 00:00 2024-04-25T00:02:22.644 INFO:teuthology.orchestra.run.smithi027.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-04-25T00:02:22.770 INFO:teuthology.orchestra.run.smithi027.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 3.6 MB/s | 863 kB 00:00 2024-04-25T00:02:22.987 INFO:teuthology.orchestra.run.smithi027.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.4 MB/s | 309 kB 00:00 2024-04-25T00:02:23.012 INFO:teuthology.orchestra.run.smithi153.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 468 kB/s | 292 kB 00:00 2024-04-25T00:02:23.037 INFO:teuthology.orchestra.run.smithi153.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 323 kB/s | 188 kB 00:00 2024-04-25T00:02:23.079 INFO:teuthology.orchestra.run.smithi027.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.2 MB/s | 548 kB 00:00 2024-04-25T00:02:23.112 INFO:teuthology.orchestra.run.smithi027.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-04-25T00:02:23.114 INFO:teuthology.orchestra.run.smithi139.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 13 kB/s | 6.6 kB 00:00 2024-04-25T00:02:23.287 INFO:teuthology.orchestra.run.smithi153.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 128 kB/s | 35 kB 00:00 2024-04-25T00:02:23.296 INFO:teuthology.orchestra.run.smithi027.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 142 kB/s | 26 kB 00:00 2024-04-25T00:02:23.405 INFO:teuthology.orchestra.run.smithi027.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-04-25T00:02:23.563 INFO:teuthology.orchestra.run.smithi027.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 426 kB/s | 67 kB 00:00 2024-04-25T00:02:23.630 INFO:teuthology.orchestra.run.smithi027.stdout:(110/152): libarrow-9.0.0-10.el9.x86_64.rpm 8.0 MB/s | 4.4 MB 00:00 2024-04-25T00:02:23.631 INFO:teuthology.orchestra.run.smithi139.stdout:(2/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 5.2 MB/s | 5.4 MB 00:01 2024-04-25T00:02:23.681 INFO:teuthology.orchestra.run.smithi139.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 2.9 MB/s | 145 kB 00:00 2024-04-25T00:02:23.814 INFO:teuthology.orchestra.run.smithi027.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 603 kB/s | 151 kB 00:00 2024-04-25T00:02:23.865 INFO:teuthology.orchestra.run.smithi139.stdout:(4/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.2 MB/s | 890 kB 00:00 2024-04-25T00:02:23.915 INFO:teuthology.orchestra.run.smithi139.stdout:(5/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 10 MB/s | 2.4 MB 00:00 2024-04-25T00:02:24.039 INFO:teuthology.orchestra.run.smithi027.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.0 MB/s | 838 kB 00:00 2024-04-25T00:02:24.139 INFO:teuthology.orchestra.run.smithi027.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 1.6 MB/s | 548 kB 00:00 2024-04-25T00:02:24.198 INFO:teuthology.orchestra.run.smithi027.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 274 kB/s | 43 kB 00:00 2024-04-25T00:02:24.256 INFO:teuthology.orchestra.run.smithi027.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 275 kB/s | 32 kB 00:00 2024-04-25T00:02:24.323 INFO:teuthology.orchestra.run.smithi139.stdout:(6/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 11 MB/s | 4.7 MB 00:00 2024-04-25T00:02:24.340 INFO:teuthology.orchestra.run.smithi027.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 105 kB/s | 15 kB 00:00 2024-04-25T00:02:24.374 INFO:teuthology.orchestra.run.smithi027.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-04-25T00:02:24.524 INFO:teuthology.orchestra.run.smithi027.stdout:(118/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.0 MB/s | 303 kB 00:00 2024-04-25T00:02:24.549 INFO:teuthology.orchestra.run.smithi027.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 1.7 MB/s | 358 kB 00:00 2024-04-25T00:02:24.616 INFO:teuthology.orchestra.run.smithi027.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 384 kB/s | 35 kB 00:00 2024-04-25T00:02:24.674 INFO:teuthology.orchestra.run.smithi139.stdout:(7/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 1.8 MB/s | 1.5 MB 00:00 2024-04-25T00:02:24.963 INFO:teuthology.orchestra.run.smithi153.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 2.5 MB/s | 4.7 MB 00:01 2024-04-25T00:02:25.167 INFO:teuthology.orchestra.run.smithi027.stdout:(121/152): ceph-test-19.0.0-2445.gd2c45d97.el9. 5.1 MB/s | 48 MB 00:09 2024-04-25T00:02:25.192 INFO:teuthology.orchestra.run.smithi027.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 320 kB/s | 206 kB 00:00 2024-04-25T00:02:25.309 INFO:teuthology.orchestra.run.smithi027.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 2.9 MB/s | 2.0 MB 00:00 2024-04-25T00:02:25.417 INFO:teuthology.orchestra.run.smithi027.stdout:(124/152): python3-grpcio-tools-1.46.7-10.el9.x 578 kB/s | 144 kB 00:00 2024-04-25T00:02:25.443 INFO:teuthology.orchestra.run.smithi027.stdout:(125/152): python3-jaraco-8.2.1-3.el9.noarch.rp 43 kB/s | 11 kB 00:00 2024-04-25T00:02:25.468 INFO:teuthology.orchestra.run.smithi027.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 111 kB/s | 18 kB 00:00 2024-04-25T00:02:25.568 INFO:teuthology.orchestra.run.smithi027.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 154 kB/s | 23 kB 00:00 2024-04-25T00:02:25.594 INFO:teuthology.orchestra.run.smithi027.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 129 kB/s | 19 kB 00:00 2024-04-25T00:02:25.618 INFO:teuthology.orchestra.run.smithi027.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 131 kB/s | 20 kB 00:00 2024-04-25T00:02:25.624 INFO:teuthology.orchestra.run.smithi139.stdout:(8/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x86 13 MB/s | 17 MB 00:01 2024-04-25T00:02:25.699 INFO:teuthology.orchestra.run.smithi139.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 336 kB/s | 25 kB 00:00 2024-04-25T00:02:25.794 INFO:teuthology.orchestra.run.smithi027.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 265 kB/s | 46 kB 00:00 2024-04-25T00:02:25.819 INFO:teuthology.orchestra.run.smithi027.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 394 kB/s | 79 kB 00:00 2024-04-25T00:02:25.853 INFO:teuthology.orchestra.run.smithi027.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 3.6 MB/s | 1.0 MB 00:00 2024-04-25T00:02:25.920 INFO:teuthology.orchestra.run.smithi027.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 460 kB/s | 58 kB 00:00 2024-04-25T00:02:25.962 INFO:teuthology.orchestra.run.smithi027.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-04-25T00:02:25.987 INFO:teuthology.orchestra.run.smithi027.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-04-25T00:02:26.037 INFO:teuthology.orchestra.run.smithi027.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 768 kB/s | 90 kB 00:00 2024-04-25T00:02:26.087 INFO:teuthology.orchestra.run.smithi027.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 307 kB/s | 31 kB 00:00 2024-04-25T00:02:26.179 INFO:teuthology.orchestra.run.smithi027.stdout:(138/152): python3-pyperclip-1.8.2-2.el9.noarch 130 kB/s | 28 kB 00:00 2024-04-25T00:02:26.204 INFO:teuthology.orchestra.run.smithi027.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.1 MB/s | 188 kB 00:00 2024-04-25T00:02:26.305 INFO:teuthology.orchestra.run.smithi027.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 272 kB/s | 59 kB 00:00 2024-04-25T00:02:26.330 INFO:teuthology.orchestra.run.smithi027.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 237 kB/s | 36 kB 00:00 2024-04-25T00:02:26.349 INFO:teuthology.orchestra.run.smithi139.stdout:(10/152): ceph-common-19.0.0-2445.gd2c45d97.el9 5.6 MB/s | 21 MB 00:03 2024-04-25T00:02:26.355 INFO:teuthology.orchestra.run.smithi027.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 338 kB/s | 51 kB 00:00 2024-04-25T00:02:26.425 INFO:teuthology.orchestra.run.smithi139.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 2.9 MB/s | 222 kB 00:00 2024-04-25T00:02:26.438 INFO:teuthology.orchestra.run.smithi153.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 1.1 MB/s | 1.6 MB 00:01 2024-04-25T00:02:26.464 INFO:teuthology.orchestra.run.smithi027.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 672 kB/s | 90 kB 00:00 2024-04-25T00:02:26.489 INFO:teuthology.orchestra.run.smithi027.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-04-25T00:02:26.508 INFO:teuthology.orchestra.run.smithi139.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 325 kB/s | 27 kB 00:00 2024-04-25T00:02:26.556 INFO:teuthology.orchestra.run.smithi027.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.1 MB/s | 427 kB 00:00 2024-04-25T00:02:26.592 INFO:teuthology.orchestra.run.smithi139.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 378 kB/s | 32 kB 00:00 2024-04-25T00:02:26.598 INFO:teuthology.orchestra.run.smithi027.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 149 kB/s | 20 kB 00:00 2024-04-25T00:02:26.734 INFO:teuthology.orchestra.run.smithi139.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 5.1 MB/s | 745 kB 00:00 2024-04-25T00:02:26.773 INFO:teuthology.orchestra.run.smithi027.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-04-25T00:02:26.826 INFO:teuthology.orchestra.run.smithi139.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 1.7 MB/s | 163 kB 00:00 2024-04-25T00:02:26.855 INFO:teuthology.orchestra.run.smithi153.stdout:(68/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 3.4 MB/s | 48 MB 00:14 2024-04-25T00:02:26.865 INFO:teuthology.orchestra.run.smithi027.stdout:(148/152): re2-20211101-3.el9.x86_64.rpm 510 kB/s | 192 kB 00:00 2024-04-25T00:02:26.910 INFO:teuthology.orchestra.run.smithi139.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.5 MB/s | 127 kB 00:00 2024-04-25T00:02:26.921 INFO:teuthology.orchestra.run.smithi153.stdout:(69/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 1.2 MB/s | 4.4 MB 00:03 2024-04-25T00:02:27.010 INFO:teuthology.orchestra.run.smithi139.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 4.9 MB/s | 501 kB 00:00 2024-04-25T00:02:27.160 INFO:teuthology.orchestra.run.smithi139.stdout:(18/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 4.1 MB/s | 10 MB 00:02 2024-04-25T00:02:27.215 INFO:teuthology.orchestra.run.smithi027.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 3.9 MB/s | 1.3 MB 00:00 2024-04-25T00:02:27.260 INFO:teuthology.orchestra.run.smithi139.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 452 kB/s | 45 kB 00:00 2024-04-25T00:02:27.272 INFO:teuthology.orchestra.run.smithi153.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 320 kB/s | 133 kB 00:00 2024-04-25T00:02:27.377 INFO:teuthology.orchestra.run.smithi139.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.1 MB/s | 133 kB 00:00 2024-04-25T00:02:27.494 INFO:teuthology.orchestra.run.smithi139.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 1.4 MB/s | 162 kB 00:00 2024-04-25T00:02:27.499 INFO:teuthology.orchestra.run.smithi027.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 8.5 MB/s | 2.4 MB 00:00 2024-04-25T00:02:27.603 INFO:teuthology.orchestra.run.smithi139.stdout:(22/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 9.0 MB/s | 5.3 MB 00:00 2024-04-25T00:02:27.608 INFO:teuthology.orchestra.run.smithi027.stdout:(151/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 3.8 MB/s | 3.1 MB 00:00 2024-04-25T00:02:27.628 INFO:teuthology.orchestra.run.smithi139.stdout:(23/152): python3-rados-19.0.0-2445.gd2c45d97.e 2.3 MB/s | 321 kB 00:00 2024-04-25T00:02:27.678 INFO:teuthology.orchestra.run.smithi139.stdout:(24/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 3.9 MB/s | 299 kB 00:00 2024-04-25T00:02:27.704 INFO:teuthology.orchestra.run.smithi139.stdout:(25/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.3 MB/s | 99 kB 00:00 2024-04-25T00:02:27.716 INFO:teuthology.orchestra.run.smithi027.stdout:(152/152): librados2-19.0.0-2445.gd2c45d97.el9. 3.0 MB/s | 3.4 MB 00:01 2024-04-25T00:02:27.719 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-25T00:02:27.720 INFO:teuthology.orchestra.run.smithi027.stdout:Total 8.1 MB/s | 220 MB 00:27 2024-04-25T00:02:27.746 INFO:teuthology.orchestra.run.smithi139.stdout:(26/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.2 MB/s | 85 kB 00:00 2024-04-25T00:02:27.821 INFO:teuthology.orchestra.run.smithi139.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 2.2 MB/s | 171 kB 00:00 2024-04-25T00:02:27.855 INFO:teuthology.orchestra.run.smithi153.stdout:(71/152): postgresql-contrib-13.14-1.el9.x86_64 643 kB/s | 911 kB 00:01 2024-04-25T00:02:27.896 INFO:teuthology.orchestra.run.smithi139.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 423 kB/s | 32 kB 00:00 2024-04-25T00:02:27.972 INFO:teuthology.orchestra.run.smithi139.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.8 MB/s | 141 kB 00:00 2024-04-25T00:02:28.239 INFO:teuthology.orchestra.run.smithi139.stdout:(30/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 5.8 MB/s | 3.1 MB 00:00 2024-04-25T00:02:28.280 INFO:teuthology.orchestra.run.smithi153.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:01 2024-04-25T00:02:28.306 INFO:teuthology.orchestra.run.smithi153.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 194 kB/s | 87 kB 00:00 2024-04-25T00:02:28.338 INFO:teuthology.orchestra.run.smithi139.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 9.6 MB/s | 3.5 MB 00:00 2024-04-25T00:02:28.406 INFO:teuthology.orchestra.run.smithi139.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 3.7 MB/s | 249 kB 00:00 2024-04-25T00:02:28.481 INFO:teuthology.orchestra.run.smithi139.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 658 kB/s | 49 kB 00:00 2024-04-25T00:02:28.539 INFO:teuthology.orchestra.run.smithi139.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 263 kB/s | 15 kB 00:00 2024-04-25T00:02:28.631 INFO:teuthology.orchestra.run.smithi139.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 2.8 MB/s | 264 kB 00:00 2024-04-25T00:02:28.740 INFO:teuthology.orchestra.run.smithi139.stdout:(36/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 6.9 MB/s | 762 kB 00:00 2024-04-25T00:02:28.815 INFO:teuthology.orchestra.run.smithi139.stdout:(37/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 244 kB/s | 18 kB 00:00 2024-04-25T00:02:28.858 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-25T00:02:28.999 INFO:teuthology.orchestra.run.smithi139.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 438 kB/s | 80 kB 00:00 2024-04-25T00:02:29.016 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-25T00:02:29.016 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-25T00:02:29.148 INFO:teuthology.orchestra.run.smithi153.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 290 kB/s | 244 kB 00:00 2024-04-25T00:02:29.166 INFO:teuthology.orchestra.run.smithi139.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 4.7 MB/s | 803 kB 00:00 2024-04-25T00:02:29.283 INFO:teuthology.orchestra.run.smithi139.stdout:(40/152): ceph-mgr-diskprediction-local-19.0.0- 7.1 MB/s | 7.4 MB 00:01 2024-04-25T00:02:29.308 INFO:teuthology.orchestra.run.smithi139.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-04-25T00:02:29.408 INFO:teuthology.orchestra.run.smithi139.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.2 MB 00:00 2024-04-25T00:02:29.466 INFO:teuthology.orchestra.run.smithi153.stdout:(75/152): postgresql-server-13.14-1.el9.x86_64. 2.3 MB/s | 5.8 MB 00:02 2024-04-25T00:02:29.467 INFO:teuthology.orchestra.run.smithi139.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-25T00:02:29.740 INFO:teuthology.orchestra.run.smithi153.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 93 kB/s | 26 kB 00:00 2024-04-25T00:02:29.765 INFO:teuthology.orchestra.run.smithi153.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 402 kB/s | 248 kB 00:00 2024-04-25T00:02:30.015 INFO:teuthology.orchestra.run.smithi153.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 70 kB/s | 19 kB 00:00 2024-04-25T00:02:30.125 INFO:teuthology.orchestra.run.smithi139.stdout:(44/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 11 MB/s | 48 MB 00:04 2024-04-25T00:02:30.151 INFO:teuthology.orchestra.run.smithi139.stdout:(45/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 292 kB/s | 253 kB 00:00 2024-04-25T00:02:30.176 INFO:teuthology.orchestra.run.smithi139.stdout:(46/152): python3-pycparser-2.20-6.el9.noarch.r 191 kB/s | 135 kB 00:00 2024-04-25T00:02:30.285 INFO:teuthology.orchestra.run.smithi139.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.6 MB/s | 215 kB 00:00 2024-04-25T00:02:30.290 INFO:teuthology.orchestra.run.smithi153.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 126 kB/s | 35 kB 00:00 2024-04-25T00:02:30.335 INFO:teuthology.orchestra.run.smithi139.stdout:(48/152): python3-requests-2.25.1-8.el9.noarch. 598 kB/s | 125 kB 00:00 2024-04-25T00:02:30.399 INFO:teuthology.orchestra.run.smithi153.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 272 kB/s | 172 kB 00:00 2024-04-25T00:02:30.724 INFO:teuthology.orchestra.run.smithi153.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 2.4 MB/s | 6.0 MB 00:02 2024-04-25T00:02:30.777 INFO:teuthology.orchestra.run.smithi139.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 242 kB/s | 107 kB 00:00 2024-04-25T00:02:30.876 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-25T00:02:30.877 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-25T00:02:30.877 INFO:teuthology.orchestra.run.smithi139.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 215 kB/s | 21 kB 00:00 2024-04-25T00:02:30.911 INFO:teuthology.orchestra.run.smithi139.stdout:(51/152): automake-1.16.2-8.el9.noarch.rpm 938 kB/s | 689 kB 00:00 2024-04-25T00:02:31.000 INFO:teuthology.orchestra.run.smithi153.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 417 kB/s | 250 kB 00:00 2024-04-25T00:02:31.019 INFO:teuthology.orchestra.run.smithi139.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 302 kB/s | 33 kB 00:00 2024-04-25T00:02:31.161 INFO:teuthology.orchestra.run.smithi139.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 1.1 MB/s | 313 kB 00:00 2024-04-25T00:02:31.194 INFO:teuthology.orchestra.run.smithi139.stdout:(54/152): bison-3.7.4-5.el9.x86_64.rpm 1.0 MB/s | 943 kB 00:00 2024-04-25T00:02:31.262 INFO:teuthology.orchestra.run.smithi139.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 181 kB/s | 18 kB 00:00 2024-04-25T00:02:31.295 INFO:teuthology.orchestra.run.smithi139.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 11 kB 00:00 2024-04-25T00:02:31.350 INFO:teuthology.orchestra.run.smithi153.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 355 kB/s | 222 kB 00:00 2024-04-25T00:02:31.370 INFO:teuthology.orchestra.run.smithi139.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-04-25T00:02:31.404 INFO:teuthology.orchestra.run.smithi139.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 418 kB/s | 45 kB 00:00 2024-04-25T00:02:31.417 INFO:teuthology.orchestra.run.smithi153.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 185 kB/s | 77 kB 00:00 2024-04-25T00:02:31.513 INFO:teuthology.orchestra.run.smithi139.stdout:(59/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.1 MB/s | 3.0 MB 00:00 2024-04-25T00:02:31.588 INFO:teuthology.orchestra.run.smithi139.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 3.1 MB/s | 582 kB 00:00 2024-04-25T00:02:31.613 INFO:teuthology.orchestra.run.smithi139.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 254 kB/s | 25 kB 00:00 2024-04-25T00:02:31.638 INFO:teuthology.orchestra.run.smithi139.stdout:(62/152): librdkafka-1.6.1-102.el9.x86_64.rpm 2.4 MB/s | 662 kB 00:00 2024-04-25T00:02:31.688 INFO:teuthology.orchestra.run.smithi139.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 472 kB/s | 47 kB 00:00 2024-04-25T00:02:31.709 INFO:teuthology.orchestra.run.smithi153.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 118 kB/s | 42 kB 00:00 2024-04-25T00:02:31.722 INFO:teuthology.orchestra.run.smithi139.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-04-25T00:02:31.747 INFO:teuthology.orchestra.run.smithi139.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-25T00:02:31.789 INFO:teuthology.orchestra.run.smithi139.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 351 kB/s | 35 kB 00:00 2024-04-25T00:02:31.917 INFO:teuthology.orchestra.run.smithi153.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 538 kB/s | 269 kB 00:00 2024-04-25T00:02:32.065 INFO:teuthology.orchestra.run.smithi139.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 5.8 MB/s | 1.6 MB 00:00 2024-04-25T00:02:32.134 INFO:teuthology.orchestra.run.smithi153.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 373 kB/s | 159 kB 00:00 2024-04-25T00:02:32.183 INFO:teuthology.orchestra.run.smithi139.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 7.5 MB/s | 911 kB 00:00 2024-04-25T00:02:32.292 INFO:teuthology.orchestra.run.smithi139.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-04-25T00:02:32.493 INFO:teuthology.orchestra.run.smithi153.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 142 kB/s | 51 kB 00:00 2024-04-25T00:02:32.518 INFO:teuthology.orchestra.run.smithi153.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 464 kB/s | 279 kB 00:00 2024-04-25T00:02:32.851 INFO:teuthology.orchestra.run.smithi153.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 150 kB/s | 54 kB 00:00 2024-04-25T00:02:33.127 INFO:teuthology.orchestra.run.smithi153.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 151 kB/s | 42 kB 00:00 2024-04-25T00:02:33.477 INFO:teuthology.orchestra.run.smithi153.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 126 kB/s | 44 kB 00:00 2024-04-25T00:02:33.544 INFO:teuthology.orchestra.run.smithi153.stdout:(93/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.6 MB/s | 5.1 MB 00:03 2024-04-25T00:02:33.811 INFO:teuthology.orchestra.run.smithi153.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 134 kB/s | 36 kB 00:00 2024-04-25T00:02:34.061 INFO:teuthology.orchestra.run.smithi153.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 378 kB/s | 221 kB 00:00 2024-04-25T00:02:34.117 INFO:teuthology.orchestra.run.smithi139.stdout:(70/152): openblas-openmp-0.3.21-2.el9.x86_64.r 2.0 MB/s | 4.7 MB 00:02 2024-04-25T00:02:34.162 INFO:teuthology.orchestra.run.smithi153.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 135 kB/s | 47 kB 00:00 2024-04-25T00:02:34.520 INFO:teuthology.orchestra.run.smithi153.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 161 kB/s | 58 kB 00:00 2024-04-25T00:02:34.650 INFO:teuthology.orchestra.run.smithi139.stdout:(71/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 1.5 MB/s | 4.4 MB 00:02 2024-04-25T00:02:34.679 INFO:teuthology.orchestra.run.smithi153.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 494 kB/s | 305 kB 00:00 2024-04-25T00:02:34.826 INFO:teuthology.orchestra.run.smithi139.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 500 kB/s | 87 kB 00:00 2024-04-25T00:02:34.954 INFO:teuthology.orchestra.run.smithi153.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 216 kB/s | 94 kB 00:00 2024-04-25T00:02:35.029 INFO:teuthology.orchestra.run.smithi153.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 182 kB/s | 64 kB 00:00 2024-04-25T00:02:35.113 INFO:teuthology.orchestra.run.smithi153.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-04-25T00:02:35.313 INFO:teuthology.orchestra.run.smithi153.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 125 kB/s | 45 kB 00:00 2024-04-25T00:02:35.480 INFO:teuthology.orchestra.run.smithi153.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.3 MB/s | 863 kB 00:00 2024-04-25T00:02:35.542 INFO:teuthology.orchestra.run.smithi139.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 725 kB/s | 1.0 MB 00:01 2024-04-25T00:02:35.747 INFO:teuthology.orchestra.run.smithi153.stdout:(104/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.0 MB/s | 16 MB 00:03 2024-04-25T00:02:35.872 INFO:teuthology.orchestra.run.smithi153.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 981 kB/s | 548 kB 00:00 2024-04-25T00:02:35.884 INFO:teuthology.orchestra.run.smithi139.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 714 kB/s | 244 kB 00:00 2024-04-25T00:02:35.964 INFO:teuthology.orchestra.run.smithi153.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 90 kB/s | 19 kB 00:00 2024-04-25T00:02:36.056 INFO:teuthology.orchestra.run.smithi153.stdout:(107/152): gperftools-libs-2.9.1-2.el9.x86_64.r 536 kB/s | 309 kB 00:00 2024-04-25T00:02:36.073 INFO:teuthology.orchestra.run.smithi153.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 239 kB/s | 26 kB 00:00 2024-04-25T00:02:36.140 INFO:teuthology.orchestra.run.smithi153.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 582 kB/s | 49 kB 00:00 2024-04-25T00:02:36.215 INFO:teuthology.orchestra.run.smithi153.stdout:(110/152): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-25T00:02:36.240 INFO:teuthology.orchestra.run.smithi153.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 403 kB/s | 67 kB 00:00 2024-04-25T00:02:36.243 INFO:teuthology.orchestra.run.smithi139.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 692 kB/s | 248 kB 00:00 2024-04-25T00:02:36.266 INFO:teuthology.orchestra.run.smithi153.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-25T00:02:36.333 INFO:teuthology.orchestra.run.smithi153.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-04-25T00:02:36.343 INFO:teuthology.orchestra.run.smithi139.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-04-25T00:02:36.349 INFO:teuthology.orchestra.run.smithi153.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 521 kB/s | 43 kB 00:00 2024-04-25T00:02:36.416 INFO:teuthology.orchestra.run.smithi153.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 385 kB/s | 32 kB 00:00 2024-04-25T00:02:36.433 INFO:teuthology.orchestra.run.smithi153.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 179 kB/s | 15 kB 00:00 2024-04-25T00:02:36.443 INFO:teuthology.orchestra.run.smithi139.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 192 kB/s | 19 kB 00:00 2024-04-25T00:02:36.475 INFO:teuthology.orchestra.run.smithi153.stdout:(117/152): python3-asyncssh-2.13.2-5.el9.noarch 2.3 MB/s | 548 kB 00:00 2024-04-25T00:02:36.517 INFO:teuthology.orchestra.run.smithi153.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-04-25T00:02:36.568 INFO:teuthology.orchestra.run.smithi153.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-04-25T00:02:36.593 INFO:teuthology.orchestra.run.smithi153.stdout:(120/152): python3-cherrypy-18.6.1-2.el9.noarch 2.2 MB/s | 358 kB 00:00 2024-04-25T00:02:36.618 INFO:teuthology.orchestra.run.smithi153.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 351 kB/s | 35 kB 00:00 2024-04-25T00:02:36.660 INFO:teuthology.orchestra.run.smithi153.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.2 MB/s | 206 kB 00:00 2024-04-25T00:02:36.710 INFO:teuthology.orchestra.run.smithi153.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-04-25T00:02:36.718 INFO:teuthology.orchestra.run.smithi139.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 626 kB/s | 172 kB 00:00 2024-04-25T00:02:36.753 INFO:teuthology.orchestra.run.smithi153.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-04-25T00:02:36.803 INFO:teuthology.orchestra.run.smithi153.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 191 kB/s | 18 kB 00:00 2024-04-25T00:02:36.827 INFO:teuthology.orchestra.run.smithi139.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 321 kB/s | 35 kB 00:00 2024-04-25T00:02:36.870 INFO:teuthology.orchestra.run.smithi153.stdout:(126/152): python3-grpcio-1.46.7-10.el9.x86_64. 7.4 MB/s | 2.0 MB 00:00 2024-04-25T00:02:36.895 INFO:teuthology.orchestra.run.smithi153.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 163 kB/s | 23 kB 00:00 2024-04-25T00:02:36.900 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-25T00:02:36.920 INFO:teuthology.orchestra.run.smithi153.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 166 kB/s | 19 kB 00:00 2024-04-25T00:02:36.962 INFO:teuthology.orchestra.run.smithi153.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-04-25T00:02:36.987 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T00:02:37.012 INFO:teuthology.orchestra.run.smithi153.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 504 kB/s | 46 kB 00:00 2024-04-25T00:02:37.026 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T00:02:37.054 INFO:teuthology.orchestra.run.smithi153.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 857 kB/s | 79 kB 00:00 2024-04-25T00:02:37.088 INFO:teuthology.orchestra.run.smithi153.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-04-25T00:02:37.088 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T00:02:37.113 INFO:teuthology.orchestra.run.smithi153.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 574 kB/s | 58 kB 00:00 2024-04-25T00:02:37.122 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T00:02:37.147 INFO:teuthology.orchestra.run.smithi153.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-04-25T00:02:37.147 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T00:02:37.180 INFO:teuthology.orchestra.run.smithi153.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 178 kB/s | 16 kB 00:00 2024-04-25T00:02:37.190 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T00:02:37.206 INFO:teuthology.orchestra.run.smithi153.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 975 kB/s | 90 kB 00:00 2024-04-25T00:02:37.239 INFO:teuthology.orchestra.run.smithi153.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 306 kB/s | 28 kB 00:00 2024-04-25T00:02:37.273 INFO:teuthology.orchestra.run.smithi153.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 333 kB/s | 31 kB 00:00 2024-04-25T00:02:37.306 INFO:teuthology.orchestra.run.smithi153.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-25T00:02:37.332 INFO:teuthology.orchestra.run.smithi153.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 641 kB/s | 59 kB 00:00 2024-04-25T00:02:37.365 INFO:teuthology.orchestra.run.smithi153.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 389 kB/s | 36 kB 00:00 2024-04-25T00:02:37.399 INFO:teuthology.orchestra.run.smithi153.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 553 kB/s | 51 kB 00:00 2024-04-25T00:02:37.425 INFO:teuthology.orchestra.run.smithi153.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-25T00:02:37.458 INFO:teuthology.orchestra.run.smithi153.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 971 kB/s | 90 kB 00:00 2024-04-25T00:02:37.500 INFO:teuthology.orchestra.run.smithi153.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-04-25T00:02:37.517 INFO:teuthology.orchestra.run.smithi153.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 216 kB/s | 20 kB 00:00 2024-04-25T00:02:37.538 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T00:02:37.550 INFO:teuthology.orchestra.run.smithi153.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-04-25T00:02:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T00:02:37.764 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T00:02:37.777 INFO:teuthology.orchestra.run.smithi139.stdout:(80/152): postgresql-server-13.14-1.el9.x86_64. 1.1 MB/s | 5.8 MB 00:05 2024-04-25T00:02:37.791 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T00:02:37.875 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T00:02:37.903 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T00:02:38.061 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T00:02:38.109 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T00:02:38.119 INFO:teuthology.orchestra.run.smithi139.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 734 kB/s | 250 kB 00:00 2024-04-25T00:02:38.136 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T00:02:38.242 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T00:02:38.271 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T00:02:38.334 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T00:02:38.355 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T00:02:38.376 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T00:02:38.461 INFO:teuthology.orchestra.run.smithi139.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 650 kB/s | 222 kB 00:00 2024-04-25T00:02:38.475 INFO:teuthology.orchestra.run.smithi153.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 1.6 MB/s | 1.6 MB 00:00 2024-04-25T00:02:38.589 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T00:02:38.622 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T00:02:38.645 INFO:teuthology.orchestra.run.smithi139.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 421 kB/s | 77 kB 00:00 2024-04-25T00:02:38.732 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T00:02:38.753 INFO:teuthology.orchestra.run.smithi139.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 391 kB/s | 42 kB 00:00 2024-04-25T00:02:38.829 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T00:02:38.834 INFO:teuthology.orchestra.run.smithi153.stdout:(149/152): librados2-19.0.0-2445.gd2c45d97.el9. 2.6 MB/s | 3.4 MB 00:01 2024-04-25T00:02:38.862 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T00:02:38.898 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T00:02:38.972 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T00:02:38.976 INFO:teuthology.orchestra.run.smithi153.stdout:(150/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 2.2 MB/s | 3.1 MB 00:01 2024-04-25T00:02:39.009 INFO:teuthology.orchestra.run.smithi153.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 2.5 MB/s | 1.3 MB 00:00 2024-04-25T00:02:39.014 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T00:02:39.057 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T00:02:39.090 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T00:02:39.117 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T00:02:39.187 INFO:teuthology.orchestra.run.smithi139.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 621 kB/s | 269 kB 00:00 2024-04-25T00:02:39.190 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T00:02:39.201 INFO:teuthology.orchestra.run.smithi153.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 6.5 MB/s | 2.4 MB 00:00 2024-04-25T00:02:39.204 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-25T00:02:39.204 INFO:teuthology.orchestra.run.smithi153.stdout:Total 5.4 MB/s | 220 MB 00:41 2024-04-25T00:02:39.241 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T00:02:39.270 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T00:02:39.305 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T00:02:39.384 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T00:02:39.416 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T00:02:39.445 INFO:teuthology.orchestra.run.smithi139.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 615 kB/s | 159 kB 00:00 2024-04-25T00:02:39.477 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T00:02:39.505 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T00:02:39.579 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T00:02:39.628 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T00:02:39.707 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T00:02:39.856 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T00:02:39.870 INFO:teuthology.orchestra.run.smithi139.stdout:(87/152): python3-pyasn1-modules-0.4.8-6.el9.no 656 kB/s | 279 kB 00:00 2024-04-25T00:02:39.910 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T00:02:39.971 INFO:teuthology.orchestra.run.smithi139.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 510 kB/s | 51 kB 00:00 2024-04-25T00:02:40.080 INFO:teuthology.orchestra.run.smithi139.stdout:(89/152): python3-requests-oauthlib-1.3.0-12.el 494 kB/s | 54 kB 00:00 2024-04-25T00:02:40.262 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-25T00:02:40.403 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-25T00:02:40.403 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-25T00:02:40.458 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T00:02:40.537 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T00:02:40.572 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T00:02:40.615 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T00:02:40.653 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T00:02:40.695 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T00:02:40.877 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T00:02:40.947 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T00:02:41.003 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T00:02:41.078 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T00:02:41.114 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T00:02:41.151 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T00:02:41.199 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T00:02:41.242 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T00:02:41.286 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T00:02:41.328 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T00:02:41.474 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T00:02:41.516 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T00:02:41.547 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T00:02:41.588 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T00:02:41.686 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T00:02:41.721 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T00:02:41.754 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T00:02:41.778 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T00:02:41.803 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T00:02:41.881 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T00:02:42.019 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T00:02:42.053 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T00:02:42.080 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T00:02:42.114 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T00:02:42.154 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T00:02:42.171 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-25T00:02:42.172 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-25T00:02:42.194 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T00:02:42.224 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T00:02:42.250 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T00:02:42.283 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T00:02:42.325 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T00:02:42.359 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T00:02:42.450 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T00:02:42.498 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T00:02:42.529 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T00:02:42.576 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T00:02:42.687 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T00:02:42.746 INFO:teuthology.orchestra.run.smithi139.stdout:(90/152): python3-babel-2.9.1-2.el9.noarch.rpm 771 kB/s | 6.0 MB 00:07 2024-04-25T00:02:42.748 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T00:02:42.854 INFO:teuthology.orchestra.run.smithi139.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 385 kB/s | 42 kB 00:00 2024-04-25T00:02:42.954 INFO:teuthology.orchestra.run.smithi139.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 442 kB/s | 44 kB 00:00 2024-04-25T00:02:43.297 INFO:teuthology.orchestra.run.smithi139.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 646 kB/s | 221 kB 00:00 2024-04-25T00:02:43.447 INFO:teuthology.orchestra.run.smithi139.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 240 kB/s | 36 kB 00:00 2024-04-25T00:02:43.520 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T00:02:43.555 INFO:teuthology.orchestra.run.smithi139.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 438 kB/s | 47 kB 00:00 2024-04-25T00:02:43.561 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T00:02:43.597 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T00:02:43.697 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T00:02:43.731 INFO:teuthology.orchestra.run.smithi139.stdout:(96/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 756 kB/s | 5.1 MB 00:06 2024-04-25T00:02:43.753 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T00:02:43.839 INFO:teuthology.orchestra.run.smithi139.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 533 kB/s | 58 kB 00:00 2024-04-25T00:02:43.854 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T00:02:43.888 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T00:02:43.998 INFO:teuthology.orchestra.run.smithi139.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 689 kB/s | 305 kB 00:00 2024-04-25T00:02:44.032 INFO:teuthology.orchestra.run.smithi139.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 488 kB/s | 94 kB 00:00 2024-04-25T00:02:44.072 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T00:02:44.103 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T00:02:44.132 INFO:teuthology.orchestra.run.smithi139.stdout:(100/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 448 kB/s | 45 kB 00:00 2024-04-25T00:02:44.174 INFO:teuthology.orchestra.run.smithi139.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 362 kB/s | 64 kB 00:00 2024-04-25T00:02:44.183 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T00:02:44.233 INFO:teuthology.orchestra.run.smithi139.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-04-25T00:02:44.499 INFO:teuthology.orchestra.run.smithi139.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 2.0 MB/s | 548 kB 00:00 2024-04-25T00:02:44.599 INFO:teuthology.orchestra.run.smithi139.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 3.0 MB/s | 309 kB 00:00 2024-04-25T00:02:44.750 INFO:teuthology.orchestra.run.smithi139.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 130 kB/s | 19 kB 00:00 2024-04-25T00:02:44.866 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T00:02:44.990 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T00:02:45.325 INFO:teuthology.orchestra.run.smithi139.stdout:(106/152): libarrow-9.0.0-10.el9.x86_64.rpm 7.7 MB/s | 4.4 MB 00:00 2024-04-25T00:02:45.442 INFO:teuthology.orchestra.run.smithi139.stdout:(107/152): protobuf-compiler-3.14.0-13.el9.x86_ 681 kB/s | 863 kB 00:01 2024-04-25T00:02:45.467 INFO:teuthology.orchestra.run.smithi139.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 183 kB/s | 26 kB 00:00 2024-04-25T00:02:45.526 INFO:teuthology.orchestra.run.smithi139.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-25T00:02:45.593 INFO:teuthology.orchestra.run.smithi139.stdout:(110/152): liboath-2.6.7-2.el9.x86_64.rpm 323 kB/s | 49 kB 00:00 2024-04-25T00:02:45.751 INFO:teuthology.orchestra.run.smithi139.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 671 kB/s | 151 kB 00:00 2024-04-25T00:02:45.885 INFO:teuthology.orchestra.run.smithi139.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.8 MB/s | 838 kB 00:00 2024-04-25T00:02:45.910 INFO:teuthology.orchestra.run.smithi139.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-04-25T00:02:45.969 INFO:teuthology.orchestra.run.smithi139.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 551 kB/s | 32 kB 00:00 2024-04-25T00:02:46.019 INFO:teuthology.orchestra.run.smithi139.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 325 kB/s | 43 kB 00:00 2024-04-25T00:02:46.044 INFO:teuthology.orchestra.run.smithi139.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-04-25T00:02:46.086 INFO:teuthology.orchestra.run.smithi139.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-04-25T00:02:46.154 INFO:teuthology.orchestra.run.smithi139.stdout:(118/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-04-25T00:02:46.195 INFO:teuthology.orchestra.run.smithi139.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 2.3 MB/s | 358 kB 00:00 2024-04-25T00:02:46.237 INFO:teuthology.orchestra.run.smithi139.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 422 kB/s | 35 kB 00:00 2024-04-25T00:02:46.262 INFO:teuthology.orchestra.run.smithi139.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 3.0 MB/s | 206 kB 00:00 2024-04-25T00:02:46.296 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T00:02:46.337 INFO:teuthology.orchestra.run.smithi139.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-04-25T00:02:46.374 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T00:02:46.402 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T00:02:46.413 INFO:teuthology.orchestra.run.smithi139.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-04-25T00:02:46.435 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T00:02:46.472 INFO:teuthology.orchestra.run.smithi139.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 303 kB/s | 18 kB 00:00 2024-04-25T00:02:46.555 INFO:teuthology.orchestra.run.smithi139.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 278 kB/s | 23 kB 00:00 2024-04-25T00:02:46.613 INFO:teuthology.orchestra.run.smithi139.stdout:(126/152): python3-jaraco-functools-3.5.0-2.el9 334 kB/s | 19 kB 00:00 2024-04-25T00:02:46.655 INFO:teuthology.orchestra.run.smithi139.stdout:(127/152): python3-grpcio-1.46.7-10.el9.x86_64. 4.9 MB/s | 2.0 MB 00:00 2024-04-25T00:02:46.723 INFO:teuthology.orchestra.run.smithi139.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-04-25T00:02:46.770 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T00:02:46.781 INFO:teuthology.orchestra.run.smithi139.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 793 kB/s | 46 kB 00:00 2024-04-25T00:02:46.814 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T00:02:46.815 INFO:teuthology.orchestra.run.smithi139.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 6.4 MB/s | 1.0 MB 00:00 2024-04-25T00:02:46.857 INFO:teuthology.orchestra.run.smithi139.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-04-25T00:02:46.930 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T00:02:46.940 INFO:teuthology.orchestra.run.smithi139.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 461 kB/s | 58 kB 00:00 2024-04-25T00:02:46.961 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T00:02:46.965 INFO:teuthology.orchestra.run.smithi139.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-04-25T00:02:46.997 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T00:02:46.999 INFO:teuthology.orchestra.run.smithi139.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 280 kB/s | 16 kB 00:00 2024-04-25T00:02:47.024 INFO:teuthology.orchestra.run.smithi139.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-04-25T00:02:47.058 INFO:teuthology.orchestra.run.smithi139.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 484 kB/s | 28 kB 00:00 2024-04-25T00:02:47.108 INFO:teuthology.orchestra.run.smithi139.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 368 kB/s | 31 kB 00:00 2024-04-25T00:02:47.142 INFO:teuthology.orchestra.run.smithi139.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-04-25T00:02:47.167 INFO:teuthology.orchestra.run.smithi139.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-04-25T00:02:47.201 INFO:teuthology.orchestra.run.smithi139.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 611 kB/s | 36 kB 00:00 2024-04-25T00:02:47.226 INFO:teuthology.orchestra.run.smithi139.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 871 kB/s | 51 kB 00:00 2024-04-25T00:02:47.286 INFO:teuthology.orchestra.run.smithi139.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-25T00:02:47.310 INFO:teuthology.orchestra.run.smithi139.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-04-25T00:02:47.393 INFO:teuthology.orchestra.run.smithi139.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-04-25T00:02:47.419 INFO:teuthology.orchestra.run.smithi139.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-04-25T00:02:47.469 INFO:teuthology.orchestra.run.smithi139.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-25T00:02:47.493 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T00:02:47.576 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T00:02:47.611 INFO:teuthology.orchestra.run.smithi139.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2024-04-25T00:02:47.650 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T00:02:47.677 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T00:02:48.111 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-25T00:02:48.205 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T00:02:48.243 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T00:02:48.307 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T00:02:48.347 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T00:02:48.373 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T00:02:48.415 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T00:02:48.453 INFO:teuthology.orchestra.run.smithi139.stdout:(148/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 3.7 MB/s | 3.1 MB 00:00 2024-04-25T00:02:48.603 INFO:teuthology.orchestra.run.smithi139.stdout:(149/152): librados2-19.0.0-2445.gd2c45d97.el9. 3.0 MB/s | 3.4 MB 00:01 2024-04-25T00:02:48.728 INFO:teuthology.orchestra.run.smithi139.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 4.9 MB/s | 1.3 MB 00:00 2024-04-25T00:02:48.817 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T00:02:48.874 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T00:02:48.995 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T00:02:49.016 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T00:02:49.090 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T00:02:49.129 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T00:02:49.136 INFO:teuthology.orchestra.run.smithi139.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 4.5 MB/s | 2.4 MB 00:00 2024-04-25T00:02:49.279 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T00:02:49.334 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T00:02:49.362 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T00:02:49.465 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T00:02:49.488 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T00:02:49.550 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T00:02:49.572 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T00:02:49.595 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T00:02:49.756 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T00:02:49.797 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T00:02:49.905 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T00:02:50.000 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T00:02:50.024 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T00:02:50.029 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T00:02:50.066 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T00:02:50.073 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T00:02:50.138 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T00:02:50.172 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T00:02:50.216 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T00:02:50.257 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T00:02:50.284 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T00:02:50.352 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T00:02:50.406 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T00:02:50.437 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T00:02:50.481 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T00:02:50.558 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T00:02:50.591 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T00:02:50.635 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T00:02:50.636 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T00:02:50.663 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T00:02:50.678 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T00:02:50.741 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T00:02:50.774 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T00:02:50.774 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T00:02:50.774 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T00:02:50.774 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:02:50.794 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T00:02:50.804 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T00:02:50.869 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T00:02:51.017 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T00:02:51.072 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T00:02:51.588 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T00:02:51.675 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T00:02:51.704 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T00:02:51.747 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T00:02:51.785 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T00:02:51.827 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T00:02:51.980 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T00:02:52.063 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T00:02:52.117 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T00:02:52.189 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T00:02:52.222 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T00:02:52.258 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T00:02:52.305 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T00:02:52.341 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T00:02:52.384 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T00:02:52.425 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T00:02:52.562 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T00:02:52.606 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T00:02:52.636 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T00:02:52.678 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T00:02:52.773 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T00:02:52.812 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T00:02:52.851 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T00:02:52.878 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T00:02:52.902 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T00:02:52.976 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T00:02:53.104 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T00:02:53.142 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T00:02:53.170 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T00:02:53.204 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T00:02:53.243 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T00:02:53.275 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T00:02:53.305 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T00:02:53.339 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T00:02:53.372 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T00:02:53.414 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T00:02:53.449 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T00:02:53.536 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T00:02:53.588 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T00:02:53.618 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T00:02:53.665 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T00:02:53.763 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T00:02:53.811 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T00:02:54.530 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T00:02:54.575 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T00:02:54.609 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T00:02:54.710 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T00:02:54.767 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T00:02:54.854 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T00:02:54.886 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T00:02:55.067 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T00:02:55.100 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T00:02:55.187 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T00:02:55.832 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T00:02:55.948 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T00:02:57.190 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T00:02:57.270 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T00:02:57.298 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T00:02:57.332 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T00:02:57.645 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T00:02:57.694 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T00:02:57.785 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T00:02:57.816 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T00:02:57.851 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T00:02:58.326 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T00:02:58.372 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T00:02:58.426 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T00:02:58.448 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T00:03:00.704 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T00:03:00.736 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T00:03:01.286 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T00:03:01.331 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T00:03:01.421 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T00:03:01.421 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T00:03:01.421 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T00:03:01.421 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:03:01.457 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T00:03:01.918 INFO:teuthology.orchestra.run.smithi139.stdout:(152/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 762 kB/s | 16 MB 00:21 2024-04-25T00:03:01.921 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-04-25T00:03:01.921 INFO:teuthology.orchestra.run.smithi139.stdout:Total 3.6 MB/s | 220 MB 01:00 2024-04-25T00:03:02.789 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T00:03:02.789 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2024-04-25T00:03:02.789 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2024-04-25T00:03:02.789 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2024-04-25T00:03:02.789 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2024-04-25T00:03:02.789 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2024-04-25T00:03:02.789 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /root 2024-04-25T00:03:02.789 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2024-04-25T00:03:02.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:03:02.967 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T00:03:02.986 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T00:03:02.986 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:02.986 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T00:03:02.986 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:03:02.990 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-04-25T00:03:03.132 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-04-25T00:03:03.132 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-04-25T00:03:04.504 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T00:03:04.531 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T00:03:04.531 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:04.531 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T00:03:04.531 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T00:03:04.531 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T00:03:04.532 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:03:04.762 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T00:03:04.786 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T00:03:04.786 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:04.786 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T00:03:04.787 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T00:03:04.787 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T00:03:04.787 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:03:04.963 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-04-25T00:03:04.963 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-04-25T00:03:05.246 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T00:03:05.270 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T00:03:05.270 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T00:03:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T00:03:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T00:03:05.271 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:03:05.386 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T00:03:05.444 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T00:03:05.507 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T00:03:05.702 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T00:03:05.748 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T00:03:05.828 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T00:03:05.927 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T00:03:06.068 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T00:03:06.106 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T00:03:06.138 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T00:03:06.165 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T00:03:06.584 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T00:03:06.614 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T00:03:06.646 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T00:03:06.710 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T00:03:06.745 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T00:03:07.836 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T00:03:07.864 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T00:03:08.882 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T00:03:08.906 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T00:03:09.015 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T00:03:09.185 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T00:03:09.216 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T00:03:09.241 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T00:03:09.241 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:09.241 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T00:03:09.241 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T00:03:09.241 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T00:03:09.241 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:03:09.292 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T00:03:09.303 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T00:03:09.405 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T00:03:09.458 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T00:03:10.402 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T00:03:10.428 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T00:03:10.452 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T00:03:10.452 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:10.452 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T00:03:10.452 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T00:03:10.452 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T00:03:10.452 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:03:10.495 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T00:03:10.518 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T00:03:10.518 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:10.519 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T00:03:10.519 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:03:10.563 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T00:03:10.586 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T00:03:10.586 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:10.587 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T00:03:10.587 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T00:03:10.587 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T00:03:10.587 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:03:10.913 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T00:03:10.926 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-04-25T00:03:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T00:03:10.937 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:10.937 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T00:03:10.937 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T00:03:10.937 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T00:03:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:03:11.013 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T00:03:11.060 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T00:03:11.122 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T00:03:11.156 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T00:03:11.181 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T00:03:11.224 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T00:03:11.563 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T00:03:11.674 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T00:03:11.795 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T00:03:11.825 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T00:03:11.916 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T00:03:11.946 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T00:03:12.098 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T00:03:12.143 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T00:03:12.170 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T00:03:12.273 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T00:03:12.297 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T00:03:12.342 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T00:03:12.364 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T00:03:12.386 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T00:03:12.530 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T00:03:12.563 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T00:03:12.676 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T00:03:12.769 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T00:03:12.804 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T00:03:12.840 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T00:03:12.914 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T00:03:12.918 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T00:03:12.918 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-04-25T00:03:12.918 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-04-25T00:03:12.919 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-04-25T00:03:12.919 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-04-25T00:03:12.919 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-04-25T00:03:12.919 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-04-25T00:03:12.919 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-04-25T00:03:12.919 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:03:12.956 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T00:03:13.000 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T00:03:13.031 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T00:03:13.059 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T00:03:13.085 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T00:03:13.100 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T00:03:13.100 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:13.100 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T00:03:13.100 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:03:13.129 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T00:03:13.182 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T00:03:13.212 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T00:03:13.256 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T00:03:13.333 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T00:03:13.366 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T00:03:13.410 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T00:03:13.438 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T00:03:13.508 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T00:03:13.562 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T00:03:13.636 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T00:03:13.772 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T00:03:13.822 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T00:03:14.355 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T00:03:14.434 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T00:03:14.463 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T00:03:14.505 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T00:03:14.543 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T00:03:14.571 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T00:03:14.585 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T00:03:14.597 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T00:03:14.597 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:14.597 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T00:03:14.597 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T00:03:14.597 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T00:03:14.597 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:03:14.753 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T00:03:14.822 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T00:03:14.822 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T00:03:14.844 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T00:03:14.844 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:14.844 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T00:03:14.844 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T00:03:14.845 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T00:03:14.845 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:03:14.877 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T00:03:14.950 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T00:03:14.980 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T00:03:15.017 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T00:03:15.064 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T00:03:15.108 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T00:03:15.150 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T00:03:15.193 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T00:03:15.298 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T00:03:15.320 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T00:03:15.320 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:15.320 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T00:03:15.320 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T00:03:15.320 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T00:03:15.320 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:03:15.330 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T00:03:15.373 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T00:03:15.403 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T00:03:15.428 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T00:03:15.445 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T00:03:15.484 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T00:03:15.539 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T00:03:15.540 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T00:03:15.570 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T00:03:15.618 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T00:03:15.643 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T00:03:15.668 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T00:03:15.692 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T00:03:15.725 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T00:03:15.734 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T00:03:15.743 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T00:03:15.773 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T00:03:15.776 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T00:03:15.807 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T00:03:15.834 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T00:03:15.851 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T00:03:15.873 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T00:03:15.909 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T00:03:15.936 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T00:03:15.939 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T00:03:15.971 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T00:03:16.010 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T00:03:16.032 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T00:03:16.041 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T00:03:16.072 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T00:03:16.072 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T00:03:16.079 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T00:03:16.093 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T00:03:16.105 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T00:03:16.131 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T00:03:16.139 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T00:03:16.163 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T00:03:16.180 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T00:03:16.199 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T00:03:16.206 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T00:03:16.294 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T00:03:16.345 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T00:03:16.377 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T00:03:16.424 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T00:03:16.524 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T00:03:16.579 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T00:03:16.590 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T00:03:16.631 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T00:03:16.662 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T00:03:16.719 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T00:03:16.724 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T00:03:16.754 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T00:03:16.757 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T00:03:16.778 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T00:03:17.323 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T00:03:17.366 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T00:03:17.400 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T00:03:17.499 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T00:03:17.558 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T00:03:17.646 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T00:03:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T00:03:17.693 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T00:03:17.771 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T00:03:17.797 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T00:03:17.813 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T00:03:17.830 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T00:03:17.872 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T00:03:17.898 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T00:03:17.908 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T00:03:17.933 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T00:03:17.934 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T00:03:17.986 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T00:03:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T00:03:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T00:03:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T00:03:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T00:03:18.210 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T00:03:18.642 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T00:03:18.748 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T00:03:18.797 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T00:03:18.839 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T00:03:18.940 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T00:03:19.102 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T00:03:19.133 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T00:03:19.155 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T00:03:19.155 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:19.156 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T00:03:19.156 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T00:03:19.156 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T00:03:19.156 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:03:19.200 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T00:03:19.211 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T00:03:19.310 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T00:03:19.357 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T00:03:20.005 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T00:03:20.086 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T00:03:20.114 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T00:03:20.147 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T00:03:20.254 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T00:03:20.286 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T00:03:20.308 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T00:03:20.308 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:20.309 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T00:03:20.309 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T00:03:20.309 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T00:03:20.309 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:03:20.362 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T00:03:20.383 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T00:03:20.384 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:20.384 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T00:03:20.384 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:03:20.437 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T00:03:20.459 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T00:03:20.459 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T00:03:20.460 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:20.460 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T00:03:20.460 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T00:03:20.460 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T00:03:20.460 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:03:20.501 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T00:03:20.591 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T00:03:20.614 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T00:03:20.650 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T00:03:20.764 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T00:03:20.786 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T00:03:20.786 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:20.786 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T00:03:20.786 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T00:03:20.786 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T00:03:20.786 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:03:21.121 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T00:03:21.154 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T00:03:21.217 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T00:03:21.247 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T00:03:23.578 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T00:03:23.618 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T00:03:23.854 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T00:03:23.855 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T00:03:23.855 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T00:03:23.855 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T00:03:23.855 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T00:03:23.855 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T00:03:23.855 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T00:03:23.855 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T00:03:23.855 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T00:03:23.855 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T00:03:23.855 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T00:03:23.855 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T00:03:23.856 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T00:03:23.856 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T00:03:23.856 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T00:03:23.856 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T00:03:23.856 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T00:03:23.856 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T00:03:23.856 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T00:03:23.856 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T00:03:23.856 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T00:03:23.856 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T00:03:23.856 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T00:03:23.856 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T00:03:23.857 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T00:03:23.857 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T00:03:23.857 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T00:03:23.857 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T00:03:23.857 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T00:03:23.857 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T00:03:23.857 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T00:03:23.857 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T00:03:23.857 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T00:03:23.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T00:03:23.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T00:03:23.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T00:03:23.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T00:03:23.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T00:03:23.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T00:03:23.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T00:03:23.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T00:03:23.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T00:03:23.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T00:03:23.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T00:03:23.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T00:03:23.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T00:03:23.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T00:03:23.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T00:03:23.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T00:03:23.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T00:03:23.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T00:03:23.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T00:03:23.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T00:03:23.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T00:03:23.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T00:03:23.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T00:03:23.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T00:03:23.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T00:03:23.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T00:03:23.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T00:03:23.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T00:03:23.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T00:03:23.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T00:03:23.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T00:03:23.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T00:03:23.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T00:03:23.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T00:03:23.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T00:03:23.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T00:03:23.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T00:03:23.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T00:03:23.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T00:03:23.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T00:03:23.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T00:03:23.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T00:03:23.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T00:03:23.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T00:03:23.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T00:03:23.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T00:03:23.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T00:03:23.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T00:03:23.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T00:03:23.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T00:03:23.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T00:03:23.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T00:03:23.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T00:03:23.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T00:03:23.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T00:03:23.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T00:03:23.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T00:03:23.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T00:03:23.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T00:03:23.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T00:03:23.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T00:03:23.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T00:03:23.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T00:03:23.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T00:03:23.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T00:03:23.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T00:03:23.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T00:03:23.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T00:03:23.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T00:03:23.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T00:03:23.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T00:03:23.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T00:03:23.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T00:03:23.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T00:03:23.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T00:03:23.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T00:03:23.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T00:03:23.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T00:03:23.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T00:03:23.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T00:03:23.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T00:03:23.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T00:03:23.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T00:03:23.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T00:03:23.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T00:03:23.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T00:03:23.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T00:03:23.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T00:03:23.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T00:03:23.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T00:03:23.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T00:03:23.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T00:03:23.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T00:03:23.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T00:03:23.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T00:03:23.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T00:03:23.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T00:03:23.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T00:03:23.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T00:03:23.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T00:03:23.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T00:03:23.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T00:03:23.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T00:03:23.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T00:03:23.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T00:03:23.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T00:03:23.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T00:03:23.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T00:03:23.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T00:03:23.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T00:03:23.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T00:03:23.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T00:03:23.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T00:03:23.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T00:03:23.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T00:03:23.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T00:03:23.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T00:03:23.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T00:03:23.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T00:03:23.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T00:03:23.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T00:03:23.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T00:03:23.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T00:03:24.164 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T00:03:24.206 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T00:03:24.292 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T00:03:24.292 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T00:03:24.292 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T00:03:24.292 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T00:03:24.323 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T00:03:25.300 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T00:03:25.342 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T00:03:25.383 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T00:03:25.415 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T00:03:25.442 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T00:03:25.625 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T00:03:25.679 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T00:03:25.717 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T00:03:26.084 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T00:03:26.084 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:03:26.085 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-04-25T00:03:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T00:03:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T00:03:26.085 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-25T00:03:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T00:03:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T00:03:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T00:03:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T00:03:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T00:03:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T00:03:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T00:03:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T00:03:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T00:03:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T00:03:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T00:03:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T00:03:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T00:03:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T00:03:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T00:03:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T00:03:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T00:03:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T00:03:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T00:03:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T00:03:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T00:03:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T00:03:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T00:03:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T00:03:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T00:03:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T00:03:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T00:03:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T00:03:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T00:03:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T00:03:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T00:03:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T00:03:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T00:03:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T00:03:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T00:03:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T00:03:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T00:03:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T00:03:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T00:03:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T00:03:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T00:03:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T00:03:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T00:03:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T00:03:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T00:03:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T00:03:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T00:03:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T00:03:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T00:03:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T00:03:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T00:03:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T00:03:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T00:03:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T00:03:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T00:03:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T00:03:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T00:03:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T00:03:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T00:03:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T00:03:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T00:03:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T00:03:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T00:03:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T00:03:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T00:03:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T00:03:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T00:03:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T00:03:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T00:03:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T00:03:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T00:03:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T00:03:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T00:03:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T00:03:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T00:03:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T00:03:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T00:03:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T00:03:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T00:03:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T00:03:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T00:03:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T00:03:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T00:03:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T00:03:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T00:03:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T00:03:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T00:03:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T00:03:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T00:03:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T00:03:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T00:03:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T00:03:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T00:03:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T00:03:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T00:03:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T00:03:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T00:03:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T00:03:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T00:03:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T00:03:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T00:03:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T00:03:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T00:03:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T00:03:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T00:03:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T00:03:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T00:03:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T00:03:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T00:03:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:03:26.096 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-25T00:03:26.313 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T00:03:26.357 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T00:03:26.376 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T00:03:26.488 DEBUG:teuthology.parallel:result is None 2024-04-25T00:03:27.201 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T00:03:27.280 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T00:03:27.321 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T00:03:27.403 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T00:03:27.434 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T00:03:27.434 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T00:03:27.518 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T00:03:27.543 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T00:03:27.543 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T00:03:27.654 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T00:03:27.724 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T00:03:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T00:03:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T00:03:33.449 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T00:03:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T00:03:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T00:03:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T00:03:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T00:03:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T00:03:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T00:03:33.450 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T00:03:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T00:03:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T00:03:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T00:03:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T00:03:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T00:03:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T00:03:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T00:03:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T00:03:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T00:03:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T00:03:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T00:03:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T00:03:33.451 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T00:03:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T00:03:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T00:03:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T00:03:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T00:03:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T00:03:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T00:03:33.452 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T00:03:33.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T00:03:33.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T00:03:33.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T00:03:33.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T00:03:33.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T00:03:33.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T00:03:33.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T00:03:33.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T00:03:33.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T00:03:33.464 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T00:03:33.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T00:03:33.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T00:03:33.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T00:03:33.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T00:03:33.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T00:03:33.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T00:03:33.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T00:03:33.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T00:03:33.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T00:03:33.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T00:03:33.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T00:03:33.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T00:03:33.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T00:03:33.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T00:03:33.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T00:03:33.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T00:03:33.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T00:03:33.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T00:03:33.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T00:03:33.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T00:03:33.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T00:03:33.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T00:03:33.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T00:03:33.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T00:03:33.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T00:03:33.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T00:03:33.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T00:03:33.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T00:03:33.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T00:03:33.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T00:03:33.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T00:03:33.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T00:03:33.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T00:03:33.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T00:03:33.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T00:03:33.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T00:03:33.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T00:03:33.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T00:03:33.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T00:03:33.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T00:03:33.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T00:03:33.468 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T00:03:33.468 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T00:03:33.468 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T00:03:33.468 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T00:03:33.468 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T00:03:33.468 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T00:03:33.468 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T00:03:33.468 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T00:03:33.468 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T00:03:33.468 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T00:03:33.468 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T00:03:33.468 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T00:03:33.468 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T00:03:33.469 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T00:03:33.469 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T00:03:33.469 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T00:03:33.469 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T00:03:33.469 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T00:03:33.469 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T00:03:33.469 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T00:03:33.469 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T00:03:33.469 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T00:03:33.469 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T00:03:33.469 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T00:03:33.469 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T00:03:33.469 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T00:03:33.469 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T00:03:33.470 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T00:03:33.470 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T00:03:33.470 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T00:03:33.470 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T00:03:33.470 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T00:03:33.470 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T00:03:33.470 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T00:03:33.470 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T00:03:33.470 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T00:03:33.470 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T00:03:33.470 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T00:03:33.470 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T00:03:33.470 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T00:03:33.470 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T00:03:33.471 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T00:03:33.471 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T00:03:33.471 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T00:03:33.471 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T00:03:33.471 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T00:03:33.471 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T00:03:33.471 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T00:03:33.471 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T00:03:33.471 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T00:03:33.471 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T00:03:33.471 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T00:03:33.471 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T00:03:33.471 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T00:03:33.471 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T00:03:33.472 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T00:03:33.472 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T00:03:33.472 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T00:03:33.472 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T00:03:33.472 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T00:03:33.472 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T00:03:33.472 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T00:03:33.472 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T00:03:33.472 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T00:03:33.472 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T00:03:33.472 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T00:03:33.472 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T00:03:33.472 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T00:03:33.472 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T00:03:33.473 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T00:03:33.473 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T00:03:33.473 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T00:03:33.473 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T00:03:33.473 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T00:03:33.473 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T00:03:35.456 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T00:03:35.457 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /sys 2024-04-25T00:03:35.457 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /proc 2024-04-25T00:03:35.457 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /mnt 2024-04-25T00:03:35.457 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /var/tmp 2024-04-25T00:03:35.457 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /home 2024-04-25T00:03:35.457 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /root 2024-04-25T00:03:35.457 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /tmp 2024-04-25T00:03:35.457 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T00:03:35.531 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T00:03:35.531 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:03:35.531 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-04-25T00:03:35.531 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.531 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.531 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T00:03:35.531 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T00:03:35.532 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-25T00:03:35.532 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T00:03:35.532 INFO:teuthology.orchestra.run.smithi153.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T00:03:35.532 INFO:teuthology.orchestra.run.smithi153.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T00:03:35.532 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T00:03:35.532 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.532 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.532 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.532 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.532 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:35.532 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.533 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.533 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.533 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:35.533 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:35.533 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:35.533 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:35.533 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:35.533 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.533 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.533 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:35.533 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.534 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.534 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.534 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:35.534 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:35.534 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.534 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.534 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:35.534 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T00:03:35.534 INFO:teuthology.orchestra.run.smithi153.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T00:03:35.534 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T00:03:35.535 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T00:03:35.535 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T00:03:35.535 INFO:teuthology.orchestra.run.smithi153.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T00:03:35.535 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T00:03:35.535 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T00:03:35.535 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T00:03:35.535 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T00:03:35.535 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T00:03:35.535 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.535 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.536 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.536 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T00:03:35.536 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T00:03:35.536 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T00:03:35.536 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T00:03:35.536 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T00:03:35.536 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.536 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.536 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T00:03:35.537 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.537 INFO:teuthology.orchestra.run.smithi153.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T00:03:35.537 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T00:03:35.537 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T00:03:35.537 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T00:03:35.537 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T00:03:35.537 INFO:teuthology.orchestra.run.smithi153.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T00:03:35.537 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T00:03:35.537 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T00:03:35.537 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T00:03:35.538 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T00:03:35.538 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T00:03:35.538 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T00:03:35.538 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T00:03:35.538 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T00:03:35.538 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T00:03:35.538 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T00:03:35.538 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T00:03:35.538 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T00:03:35.538 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T00:03:35.538 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T00:03:35.539 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T00:03:35.539 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T00:03:35.539 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T00:03:35.539 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.539 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.539 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.539 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T00:03:35.539 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T00:03:35.539 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T00:03:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T00:03:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T00:03:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T00:03:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T00:03:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T00:03:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T00:03:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T00:03:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T00:03:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T00:03:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T00:03:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T00:03:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T00:03:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T00:03:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T00:03:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T00:03:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T00:03:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T00:03:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T00:03:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T00:03:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T00:03:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T00:03:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T00:03:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T00:03:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T00:03:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T00:03:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T00:03:35.542 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T00:03:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T00:03:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T00:03:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T00:03:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T00:03:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T00:03:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T00:03:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T00:03:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T00:03:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T00:03:35.545 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T00:03:35.546 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T00:03:35.546 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:03:35.546 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-25T00:03:35.623 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T00:03:35.638 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T00:03:35.639 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:35.639 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T00:03:35.639 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T00:03:35.960 DEBUG:teuthology.parallel:result is None 2024-04-25T00:03:37.144 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T00:03:37.170 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T00:03:37.170 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:37.170 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T00:03:37.170 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T00:03:37.170 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T00:03:37.170 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T00:03:37.395 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T00:03:37.417 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T00:03:37.417 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:37.418 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T00:03:37.418 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T00:03:37.418 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T00:03:37.418 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T00:03:37.862 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T00:03:37.884 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T00:03:37.885 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:37.885 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T00:03:37.885 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T00:03:37.885 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T00:03:37.885 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T00:03:37.992 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T00:03:38.049 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T00:03:38.105 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T00:03:38.293 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T00:03:38.338 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T00:03:38.416 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T00:03:38.505 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T00:03:38.644 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T00:03:38.687 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T00:03:38.719 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T00:03:38.747 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T00:03:39.187 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T00:03:39.220 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T00:03:39.251 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T00:03:39.314 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T00:03:39.351 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T00:03:40.390 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T00:03:40.436 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T00:03:41.402 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T00:03:41.428 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T00:03:41.532 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T00:03:41.692 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T00:03:41.722 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T00:03:41.745 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T00:03:41.745 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:41.745 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T00:03:41.745 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T00:03:41.745 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T00:03:41.745 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T00:03:41.798 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T00:03:41.808 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T00:03:41.908 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T00:03:41.963 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T00:03:42.861 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T00:03:42.892 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T00:03:42.914 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T00:03:42.914 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:42.914 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T00:03:42.914 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T00:03:42.914 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T00:03:42.914 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T00:03:42.967 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T00:03:42.989 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T00:03:42.989 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:42.989 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T00:03:42.989 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T00:03:43.035 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T00:03:43.057 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T00:03:43.057 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:43.057 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T00:03:43.057 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T00:03:43.057 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T00:03:43.057 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T00:03:43.361 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T00:03:43.383 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T00:03:43.383 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T00:03:43.383 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T00:03:43.383 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T00:03:43.383 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T00:03:43.383 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T00:03:47.889 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T00:03:47.930 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T00:03:47.989 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T00:03:48.020 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T00:03:48.047 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T00:03:48.237 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T00:03:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T00:03:48.314 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T00:03:48.942 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T00:03:48.978 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T00:03:48.998 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T00:03:49.840 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T00:03:49.916 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T00:03:49.957 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T00:03:50.041 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T00:03:50.087 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T00:03:50.087 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T00:03:50.170 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T00:03:50.195 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T00:03:50.195 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T00:03:50.289 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T00:03:50.360 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T00:03:56.081 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T00:03:56.081 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T00:03:56.081 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T00:03:56.081 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T00:03:56.081 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T00:03:56.081 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T00:03:56.081 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T00:03:56.081 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T00:03:56.081 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T00:03:56.081 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T00:03:56.082 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T00:03:56.082 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T00:03:56.082 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T00:03:56.082 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T00:03:56.082 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T00:03:56.082 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T00:03:56.082 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T00:03:56.082 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T00:03:56.082 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T00:03:56.082 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T00:03:56.082 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T00:03:56.082 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T00:03:56.082 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T00:03:56.083 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T00:03:56.083 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T00:03:56.083 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T00:03:56.097 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T00:03:56.097 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T00:03:56.097 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T00:03:56.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T00:03:56.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T00:03:56.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T00:03:56.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T00:03:56.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T00:03:56.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T00:03:56.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T00:03:56.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T00:03:56.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T00:03:56.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T00:03:56.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T00:03:56.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T00:03:56.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T00:03:56.099 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T00:03:56.099 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T00:03:56.099 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T00:03:56.099 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T00:03:56.099 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T00:03:56.099 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T00:03:56.099 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T00:03:56.099 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T00:03:56.099 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T00:03:56.099 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T00:03:56.099 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T00:03:56.099 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T00:03:56.099 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T00:03:56.100 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T00:03:56.100 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T00:03:56.100 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T00:03:56.100 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T00:03:56.100 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T00:03:56.100 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T00:03:56.100 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T00:03:56.101 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T00:03:56.101 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T00:03:56.101 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T00:03:56.101 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T00:03:56.101 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T00:03:56.101 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T00:03:56.101 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T00:03:56.101 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T00:03:56.101 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T00:03:56.101 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T00:03:56.101 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T00:03:56.102 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T00:03:56.102 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T00:03:56.102 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T00:03:56.102 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T00:03:56.102 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T00:03:56.102 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T00:03:56.102 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T00:03:56.102 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T00:03:56.102 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T00:03:56.102 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T00:03:56.102 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T00:03:56.102 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T00:03:56.102 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T00:03:56.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T00:03:56.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T00:03:56.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T00:03:56.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T00:03:56.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T00:03:56.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T00:03:56.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T00:03:56.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T00:03:56.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T00:03:56.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T00:03:56.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T00:03:56.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T00:03:56.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T00:03:56.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T00:03:56.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T00:03:56.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T00:03:56.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T00:03:56.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T00:03:56.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T00:03:56.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T00:03:56.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T00:03:56.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T00:03:56.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T00:03:56.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T00:03:56.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T00:03:56.105 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T00:03:56.105 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T00:03:56.105 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T00:03:56.105 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T00:03:56.105 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T00:03:56.105 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T00:03:56.105 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T00:03:56.105 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T00:03:56.106 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T00:03:56.106 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T00:03:56.106 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T00:03:56.106 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T00:03:56.106 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T00:03:56.106 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T00:03:56.106 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T00:03:56.106 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T00:03:56.106 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T00:03:56.106 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T00:03:56.106 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T00:03:56.106 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T00:03:56.107 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T00:03:56.107 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T00:03:56.107 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T00:03:56.107 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T00:03:56.107 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T00:03:56.107 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T00:03:56.107 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T00:03:56.107 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T00:03:56.107 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T00:03:56.107 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T00:03:56.107 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T00:03:56.107 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T00:03:56.107 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T00:03:56.107 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T00:03:56.108 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T00:03:56.108 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T00:03:56.108 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T00:03:56.108 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T00:03:56.108 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T00:03:56.108 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T00:03:56.108 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T00:03:56.108 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T00:03:56.108 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T00:03:56.108 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T00:03:56.108 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T00:03:58.187 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T00:03:58.187 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T00:03:58.187 INFO:teuthology.orchestra.run.smithi139.stdout:Upgraded: 2024-04-25T00:03:58.187 INFO:teuthology.orchestra.run.smithi139.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.187 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.187 INFO:teuthology.orchestra.run.smithi139.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T00:03:58.187 INFO:teuthology.orchestra.run.smithi139.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T00:03:58.187 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-04-25T00:03:58.187 INFO:teuthology.orchestra.run.smithi139.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T00:03:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T00:03:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T00:03:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T00:03:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T00:03:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T00:03:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T00:03:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T00:03:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T00:03:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T00:03:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T00:03:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T00:03:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T00:03:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T00:03:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T00:03:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T00:03:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T00:03:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T00:03:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T00:03:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T00:03:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T00:03:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T00:03:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T00:03:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T00:03:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T00:03:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T00:03:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T00:03:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T00:03:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T00:03:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T00:03:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T00:03:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T00:03:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T00:03:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T00:03:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T00:03:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T00:03:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T00:03:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T00:03:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T00:03:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T00:03:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T00:03:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T00:03:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T00:03:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T00:03:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T00:03:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T00:03:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T00:03:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T00:03:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T00:03:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T00:03:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T00:03:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T00:03:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T00:03:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T00:03:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T00:03:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T00:03:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T00:03:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T00:03:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T00:03:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T00:03:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T00:03:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T00:03:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T00:03:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T00:03:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T00:03:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T00:03:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T00:03:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T00:03:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T00:03:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T00:03:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T00:03:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T00:03:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T00:03:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T00:03:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T00:03:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T00:03:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T00:03:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T00:03:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T00:03:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T00:03:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T00:03:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T00:03:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T00:03:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T00:03:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T00:03:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T00:03:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T00:03:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T00:03:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T00:03:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T00:03:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T00:03:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T00:03:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T00:03:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T00:03:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T00:03:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T00:03:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T00:03:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T00:03:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T00:03:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T00:03:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T00:03:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T00:03:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T00:03:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T00:03:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T00:03:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T00:03:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T00:03:58.198 INFO:teuthology.orchestra.run.smithi139.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T00:03:58.198 INFO:teuthology.orchestra.run.smithi139.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T00:03:58.198 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T00:03:58.198 INFO:teuthology.orchestra.run.smithi139.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T00:03:58.198 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T00:03:58.198 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-04-25T00:03:58.579 DEBUG:teuthology.parallel:result is None 2024-04-25T00:03:58.579 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:03:58.804 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T00:03:58.831 INFO:teuthology.orchestra.run.smithi027.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T00:03:58.832 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T00:03:58.832 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T00:03:58.833 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:03:58.984 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T00:03:59.005 INFO:teuthology.orchestra.run.smithi139.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T00:03:59.005 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T00:03:59.006 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T00:03:59.007 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:03:59.232 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T00:03:59.258 INFO:teuthology.orchestra.run.smithi153.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T00:03:59.258 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T00:03:59.258 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T00:03:59.260 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T00:03:59.260 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:03:59.260 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T00:03:59.292 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T00:03:59.292 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T00:03:59.318 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T00:03:59.319 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T00:03:59.349 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T00:03:59.350 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:03:59.350 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T00:03:59.378 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T00:03:59.447 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T00:03:59.447 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T00:03:59.471 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T00:03:59.538 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T00:03:59.538 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T00:03:59.563 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T00:03:59.627 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T00:03:59.627 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:03:59.627 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T00:03:59.652 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T00:03:59.716 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T00:03:59.717 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T00:03:59.742 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T00:03:59.809 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T00:03:59.809 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T00:03:59.834 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T00:03:59.897 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T00:03:59.897 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:03:59.898 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T00:03:59.934 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T00:03:59.999 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T00:03:59.999 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T00:04:00.023 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T00:04:00.087 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T00:04:00.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T00:04:00.111 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T00:04:00.175 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-25T00:04:00.297 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-25T00:04:00.297 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:04:00.298 INFO:tasks.cephadm:Cluster fsid is 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:04:00.298 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-25T00:04:00.298 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.27', 'mon.b': '172.21.15.139', 'mon.c': '172.21.15.153'} 2024-04-25T00:04:00.298 INFO:tasks.cephadm:First mon is mon.a on smithi027 2024-04-25T00:04:00.298 INFO:tasks.cephadm:First mgr is x 2024-04-25T00:04:00.298 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-25T00:04:00.299 DEBUG:teuthology.orchestra.run.smithi027:> sudo hostname $(hostname -s) 2024-04-25T00:04:00.323 DEBUG:teuthology.orchestra.run.smithi139:> sudo hostname $(hostname -s) 2024-04-25T00:04:00.347 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-04-25T00:04:00.371 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-25T00:04:00.371 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:04:00.627 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54987/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2445-gd2c45d97', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2445.gd2c45d97'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 22:37:46.422067', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54981/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2445-gd2c45d97', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2445.gd2c45d97'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 17:00:00.872117', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-25T00:04:00.847 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 d2c45d977142e024a59b68a8f1dce8f0b9840551 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:04:00.849 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm 2024-04-25T00:04:00.849 DEBUG:teuthology.orchestra.run.smithi027:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T00:04:01.408 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 00:04 /home/ubuntu/cephtest/cephadm 2024-04-25T00:04:01.408 DEBUG:teuthology.orchestra.run.smithi139:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T00:04:01.851 INFO:teuthology.orchestra.run.smithi139.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 00:04 /home/ubuntu/cephtest/cephadm 2024-04-25T00:04:01.852 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T00:04:02.421 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 00:04 /home/ubuntu/cephtest/cephadm 2024-04-25T00:04:02.421 DEBUG:teuthology.orchestra.run.smithi027:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T00:04:02.441 DEBUG:teuthology.orchestra.run.smithi139:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T00:04:02.460 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T00:04:02.505 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 on all hosts... 2024-04-25T00:04:02.505 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T00:04:02.508 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T00:04:02.511 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T00:04:03.985 INFO:teuthology.orchestra.run.smithi027.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T00:04:04.026 INFO:teuthology.orchestra.run.smithi153.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T00:04:04.165 INFO:teuthology.orchestra.run.smithi139.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T00:04:36.787 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-04-25T00:04:36.788 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T00:04:36.788 INFO:teuthology.orchestra.run.smithi139.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T00:04:36.788 INFO:teuthology.orchestra.run.smithi139.stdout: "repo_digests": [ 2024-04-25T00:04:36.788 INFO:teuthology.orchestra.run.smithi139.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T00:04:36.788 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-04-25T00:04:36.788 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-04-25T00:05:01.449 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-25T00:05:01.449 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T00:05:01.449 INFO:teuthology.orchestra.run.smithi153.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T00:05:01.449 INFO:teuthology.orchestra.run.smithi153.stdout: "repo_digests": [ 2024-04-25T00:05:01.449 INFO:teuthology.orchestra.run.smithi153.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T00:05:01.450 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-25T00:05:01.450 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-25T00:05:01.569 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:05:01.569 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T00:05:01.569 INFO:teuthology.orchestra.run.smithi027.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T00:05:01.570 INFO:teuthology.orchestra.run.smithi027.stdout: "repo_digests": [ 2024-04-25T00:05:01.570 INFO:teuthology.orchestra.run.smithi027.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T00:05:01.570 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T00:05:01.570 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:05:01.593 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph 2024-04-25T00:05:01.652 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /etc/ceph 2024-04-25T00:05:01.684 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-04-25T00:05:01.713 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /etc/ceph 2024-04-25T00:05:01.737 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 777 /etc/ceph 2024-04-25T00:05:01.764 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-04-25T00:05:01.790 INFO:tasks.cephadm:Writing seed config... 2024-04-25T00:05:01.792 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-25T00:05:01.792 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-25T00:05:01.792 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-25T00:05:01.792 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-25T00:05:01.792 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-25T00:05:01.792 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-25T00:05:01.792 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-25T00:05:01.792 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-25T00:05:01.792 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-25T00:05:01.793 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-25T00:05:01.793 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-25T00:05:01.793 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-25T00:05:01.793 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-25T00:05:01.793 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-25T00:05:01.793 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-25T00:05:01.793 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-25T00:05:01.793 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-25T00:05:01.793 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-25T00:05:01.793 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-25T00:05:01.793 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-25T00:05:01.793 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-25T00:05:01.793 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-25T00:05:01.793 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-25T00:05:01.794 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-25T00:05:01.794 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-25T00:05:01.794 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-25T00:05:01.794 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-25T00:05:01.794 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-25T00:05:01.794 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-25T00:05:01.794 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-25T00:05:01.794 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-25T00:05:01.794 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-25T00:05:01.794 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-25T00:05:01.794 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-25T00:05:01.794 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-25T00:05:01.794 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-25T00:05:01.794 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-25T00:05:01.795 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-25T00:05:01.795 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-25T00:05:01.795 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2024-04-25T00:05:01.795 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-25T00:05:01.795 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-25T00:05:01.796 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-25T00:05:01.796 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-25T00:05:01.796 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2024-04-25T00:05:01.796 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-25T00:05:01.796 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-25T00:05:01.796 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-25T00:05:01.796 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-04-25T00:05:01.797 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:05:01.797 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-25T00:05:01.814 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 518857f8-0297-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-25T00:05:01.815 DEBUG:teuthology.orchestra.run.smithi027:mon.a> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.a.service 2024-04-25T00:05:01.857 DEBUG:teuthology.orchestra.run.smithi027:mgr.x> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mgr.x.service 2024-04-25T00:05:01.899 INFO:tasks.cephadm:Bootstrapping... 2024-04-25T00:05:01.899 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 -v bootstrap --fsid 518857f8-0297-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.27 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-25T00:05:02.095 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-25T00:05:02.095 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551', '-v', 'bootstrap', '--fsid', '518857f8-0297-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.27', '--skip-admin-label'] 2024-04-25T00:05:02.096 INFO:teuthology.orchestra.run.smithi027.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-25T00:05:02.096 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying podman|docker is present... 2024-04-25T00:05:02.122 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T00:05:02.122 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying lvm2 is present... 2024-04-25T00:05:02.122 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying time synchronization is in place... 2024-04-25T00:05:02.130 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T00:05:02.130 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T00:05:02.136 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T00:05:02.136 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T00:05:02.143 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout enabled 2024-04-25T00:05:02.149 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout active 2024-04-25T00:05:02.149 INFO:teuthology.orchestra.run.smithi027.stdout:Unit chronyd.service is enabled and running 2024-04-25T00:05:02.150 INFO:teuthology.orchestra.run.smithi027.stdout:Repeating the final host check... 2024-04-25T00:05:02.172 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T00:05:02.173 INFO:teuthology.orchestra.run.smithi027.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-25T00:05:02.173 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl is present 2024-04-25T00:05:02.173 INFO:teuthology.orchestra.run.smithi027.stdout:lvcreate is present 2024-04-25T00:05:02.179 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T00:05:02.179 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T00:05:02.185 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T00:05:02.185 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T00:05:02.192 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout enabled 2024-04-25T00:05:02.198 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout active 2024-04-25T00:05:02.198 INFO:teuthology.orchestra.run.smithi027.stdout:Unit chronyd.service is enabled and running 2024-04-25T00:05:02.198 INFO:teuthology.orchestra.run.smithi027.stdout:Host looks OK 2024-04-25T00:05:02.198 INFO:teuthology.orchestra.run.smithi027.stdout:Cluster fsid: 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:05:02.198 INFO:teuthology.orchestra.run.smithi027.stdout:Acquiring lock 139710674047904 on /run/cephadm/518857f8-0297-11ef-bc93-c7b262605968.lock 2024-04-25T00:05:02.198 INFO:teuthology.orchestra.run.smithi027.stdout:Lock 139710674047904 acquired on /run/cephadm/518857f8-0297-11ef-bc93-c7b262605968.lock 2024-04-25T00:05:02.199 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying IP 172.21.15.27 port 3300 ... 2024-04-25T00:05:02.199 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying IP 172.21.15.27 port 6789 ... 2024-04-25T00:05:02.199 INFO:teuthology.orchestra.run.smithi027.stdout:Base mon IP(s) is [172.21.15.27:3300, 172.21.15.27:6789], mon addrv is [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2024-04-25T00:05:02.202 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.27 metric 100 2024-04-25T00:05:02.202 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.27 metric 100 2024-04-25T00:05:02.205 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-25T00:05:02.205 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-25T00:05:02.205 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-25T00:05:02.207 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-25T00:05:02.207 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-25T00:05:02.207 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T00:05:02.207 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-25T00:05:02.207 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-04-25T00:05:02.207 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T00:05:02.208 INFO:teuthology.orchestra.run.smithi027.stdout:Mon IP `172.21.15.27` is in CIDR network `172.21.0.0/20` 2024-04-25T00:05:02.208 INFO:teuthology.orchestra.run.smithi027.stdout:Mon IP `172.21.15.27` is in CIDR network `172.21.0.0/20` 2024-04-25T00:05:02.208 INFO:teuthology.orchestra.run.smithi027.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-25T00:05:02.208 INFO:teuthology.orchestra.run.smithi027.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-25T00:05:02.209 INFO:teuthology.orchestra.run.smithi027.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T00:05:02.697 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e 2024-04-25T00:05:02.697 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T00:05:02.697 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Getting image source signatures 2024-04-25T00:05:02.697 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying blob sha256:58a99251f41e8a467bafa0ec849428709568906ca732f10972304f37056be6c5 2024-04-25T00:05:02.697 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-25T00:05:02.697 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying config sha256:f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e 2024-04-25T00:05:02.698 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-25T00:05:03.575 INFO:teuthology.orchestra.run.smithi027.stdout:ceph: stdout ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev) 2024-04-25T00:05:03.576 INFO:teuthology.orchestra.run.smithi027.stdout:Ceph version: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev) 2024-04-25T00:05:03.576 INFO:teuthology.orchestra.run.smithi027.stdout:Extracting ceph user uid/gid from container image... 2024-04-25T00:05:04.460 INFO:teuthology.orchestra.run.smithi027.stdout:stat: stdout 167 167 2024-04-25T00:05:04.460 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial keys... 2024-04-25T00:05:05.358 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQAwnilmbGoCMRAA58MDmva5LLA3HnuBrH/Mgg== 2024-04-25T00:05:06.240 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQAxnilmFCMAKhAAV6+qMT75sCBejgcN2gs+Kw== 2024-04-25T00:05:07.090 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQAynilmjDLTIhAANoQ2vm3hq0iWNPaKe/W/KQ== 2024-04-25T00:05:07.091 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial monmap... 2024-04-25T00:05:07.996 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T00:05:07.996 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-25T00:05:07.996 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:05:07.996 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T00:05:07.997 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool for a [v2:172.21.15.27:3300,v1:172.21.15.27:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T00:05:07.997 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = quincy 2024-04-25T00:05:07.997 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: set fsid to 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:05:07.997 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T00:05:07.997 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:05:07.997 INFO:teuthology.orchestra.run.smithi027.stdout:Creating mon... 2024-04-25T00:05:08.987 INFO:teuthology.orchestra.run.smithi027.stdout:create mon.a on 2024-04-25T00:05:09.203 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-25T00:05:09.354 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-25T00:05:09.526 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-518857f8-0297-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-518857f8-0297-11ef-bc93-c7b262605968.target. 2024-04-25T00:05:09.527 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-518857f8-0297-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-518857f8-0297-11ef-bc93-c7b262605968.target. 2024-04-25T00:05:09.778 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.a 2024-04-25T00:05:09.779 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to reset failed state of unit ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-25T00:05:09.953 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-518857f8-0297-11ef-bc93-c7b262605968.target.wants/ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-518857f8-0297-11ef-bc93-c7b262605968@.service. 2024-04-25T00:05:10.420 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T00:05:10.420 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-04-25T00:05:10.427 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T00:05:10.427 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T00:05:10.427 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-04-25T00:05:10.427 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T00:05:10.428 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mon to start... 2024-04-25T00:05:10.428 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mon... 2024-04-25T00:05:10.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:10 smithi027 systemd[1]: Started Ceph mon.a for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:05:11.476 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:11 smithi027 ceph-mon[28838]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T00:05:11.476 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:11 smithi027 ceph-mon[28838]: monmap epoch 1 2024-04-25T00:05:11.476 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:11 smithi027 ceph-mon[28838]: fsid 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:05:11.476 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:11 smithi027 ceph-mon[28838]: last_changed 2024-04-25T00:05:07.441245+0000 2024-04-25T00:05:11.476 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:11 smithi027 ceph-mon[28838]: created 2024-04-25T00:05:07.441245+0000 2024-04-25T00:05:11.476 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:11 smithi027 ceph-mon[28838]: min_mon_release 19 (squid) 2024-04-25T00:05:11.476 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:11 smithi027 ceph-mon[28838]: election_strategy: 1 2024-04-25T00:05:11.477 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:11 smithi027 ceph-mon[28838]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T00:05:11.477 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:11 smithi027 ceph-mon[28838]: fsmap 2024-04-25T00:05:11.477 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:11 smithi027 ceph-mon[28838]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T00:05:11.477 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:11 smithi027 ceph-mon[28838]: mgrmap e1: no daemons active 2024-04-25T00:05:11.477 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:11 smithi027 ceph-mon[28838]: from='client.? 172.21.15.27:0/1101777955' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-25T00:05:11.484 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout cluster: 2024-04-25T00:05:11.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout id: 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:05:11.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-25T00:05:11.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T00:05:11.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout services: 2024-04-25T00:05:11.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.57552s) 2024-04-25T00:05:11.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-25T00:05:11.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-25T00:05:11.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T00:05:11.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout data: 2024-04-25T00:05:11.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-25T00:05:11.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-25T00:05:11.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-25T00:05:11.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout pgs: 2024-04-25T00:05:11.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T00:05:11.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.869+0000 7f8f69a46700 1 Processor -- start 2024-04-25T00:05:11.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.869+0000 7f8f69a46700 1 -- start start 2024-04-25T00:05:11.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.870+0000 7f8f69a46700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f641082f0 0x7f8f64108710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:11.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.870+0000 7f8f69a46700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8f64108ce0 con 0x7f8f641082f0 2024-04-25T00:05:11.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.870+0000 7f8f62ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f641082f0 0x7f8f64108710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:11.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.870+0000 7f8f62ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f641082f0 0x7f8f64108710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52478/0 (socket says 172.21.15.27:52478) 2024-04-25T00:05:11.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.870+0000 7f8f62ffd700 1 -- 172.21.15.27:0/4070305917 learned_addr learned my addr 172.21.15.27:0/4070305917 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:11.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.870+0000 7f8f62ffd700 1 -- 172.21.15.27:0/4070305917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f64108e60 con 0x7f8f641082f0 2024-04-25T00:05:11.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.871+0000 7f8f62ffd700 1 --2- 172.21.15.27:0/4070305917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f641082f0 0x7f8f64108710 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009e80 tx=0x7f8f5400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb348f95e2af06c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:11.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.871+0000 7f8f61ffb700 1 -- 172.21.15.27:0/4070305917 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f54017070 con 0x7f8f641082f0 2024-04-25T00:05:11.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.871+0000 7f8f61ffb700 1 -- 172.21.15.27:0/4070305917 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f54003a20 con 0x7f8f641082f0 2024-04-25T00:05:11.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.871+0000 7f8f61ffb700 1 -- 172.21.15.27:0/4070305917 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f54003d20 con 0x7f8f641082f0 2024-04-25T00:05:11.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.871+0000 7f8f69a46700 1 -- 172.21.15.27:0/4070305917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f641082f0 msgr2=0x7f8f64108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:11.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.871+0000 7f8f69a46700 1 --2- 172.21.15.27:0/4070305917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f641082f0 0x7f8f64108710 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009e80 tx=0x7f8f5400b330 comp rx=0 tx=0).stop 2024-04-25T00:05:11.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.871+0000 7f8f69a46700 1 -- 172.21.15.27:0/4070305917 shutdown_connections 2024-04-25T00:05:11.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.871+0000 7f8f69a46700 1 --2- 172.21.15.27:0/4070305917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f641082f0 0x7f8f64108710 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:11.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.871+0000 7f8f69a46700 1 -- 172.21.15.27:0/4070305917 >> 172.21.15.27:0/4070305917 conn(0x7f8f64101a20 msgr2=0x7f8f64103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:11.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.872+0000 7f8f69a46700 1 -- 172.21.15.27:0/4070305917 shutdown_connections 2024-04-25T00:05:11.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.872+0000 7f8f69a46700 1 -- 172.21.15.27:0/4070305917 wait complete. 2024-04-25T00:05:11.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.872+0000 7f8f69a46700 1 Processor -- start 2024-04-25T00:05:11.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.872+0000 7f8f69a46700 1 -- start start 2024-04-25T00:05:11.487 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.873+0000 7f8f69a46700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f641082f0 0x7f8f6419d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:11.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.873+0000 7f8f69a46700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8f64106120 con 0x7f8f641082f0 2024-04-25T00:05:11.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.873+0000 7f8f62ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f641082f0 0x7f8f6419d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:11.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.873+0000 7f8f62ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f641082f0 0x7f8f6419d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52480/0 (socket says 172.21.15.27:52480) 2024-04-25T00:05:11.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.873+0000 7f8f62ffd700 1 -- 172.21.15.27:0/1101777955 learned_addr learned my addr 172.21.15.27:0/1101777955 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:11.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.873+0000 7f8f62ffd700 1 -- 172.21.15.27:0/1101777955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f6419dbe0 con 0x7f8f641082f0 2024-04-25T00:05:11.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.873+0000 7f8f62ffd700 1 --2- 172.21.15.27:0/1101777955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f641082f0 0x7f8f6419d6a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8f54000f80 tx=0x7f8f54004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:11.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.873+0000 7f8f68a44700 1 -- 172.21.15.27:0/1101777955 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f54022020 con 0x7f8f641082f0 2024-04-25T00:05:11.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.873+0000 7f8f68a44700 1 -- 172.21.15.27:0/1101777955 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f54018040 con 0x7f8f641082f0 2024-04-25T00:05:11.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.873+0000 7f8f69a46700 1 -- 172.21.15.27:0/1101777955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f6419ded0 con 0x7f8f641082f0 2024-04-25T00:05:11.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.873+0000 7f8f68a44700 1 -- 172.21.15.27:0/1101777955 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f54017070 con 0x7f8f641082f0 2024-04-25T00:05:11.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.873+0000 7f8f69a46700 1 -- 172.21.15.27:0/1101777955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f641a0b60 con 0x7f8f641082f0 2024-04-25T00:05:11.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.874+0000 7f8f68a44700 1 -- 172.21.15.27:0/1101777955 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8f54025020 con 0x7f8f641082f0 2024-04-25T00:05:11.488 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.874+0000 7f8f68a44700 1 -- 172.21.15.27:0/1101777955 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8f5400f050 con 0x7f8f641082f0 2024-04-25T00:05:11.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.874+0000 7f8f69a46700 1 -- 172.21.15.27:0/1101777955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f64063890 con 0x7f8f641082f0 2024-04-25T00:05:11.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.877+0000 7f8f68a44700 1 -- 172.21.15.27:0/1101777955 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8f5402c030 con 0x7f8f641082f0 2024-04-25T00:05:11.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.950+0000 7f8f69a46700 1 -- 172.21.15.27:0/1101777955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f8f6402de30 con 0x7f8f641082f0 2024-04-25T00:05:11.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.950+0000 7f8f68a44700 1 -- 172.21.15.27:0/1101777955 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7f8f54012610 con 0x7f8f641082f0 2024-04-25T00:05:11.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.951+0000 7f8f69a46700 1 -- 172.21.15.27:0/1101777955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f641082f0 msgr2=0x7f8f6419d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:11.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.951+0000 7f8f69a46700 1 --2- 172.21.15.27:0/1101777955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f641082f0 0x7f8f6419d6a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8f54000f80 tx=0x7f8f54004ad0 comp rx=0 tx=0).stop 2024-04-25T00:05:11.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.951+0000 7f8f69a46700 1 -- 172.21.15.27:0/1101777955 shutdown_connections 2024-04-25T00:05:11.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.951+0000 7f8f69a46700 1 --2- 172.21.15.27:0/1101777955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f641082f0 0x7f8f6419d6a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:11.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.951+0000 7f8f69a46700 1 -- 172.21.15.27:0/1101777955 >> 172.21.15.27:0/1101777955 conn(0x7f8f64101a20 msgr2=0x7f8f64194360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:11.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.951+0000 7f8f69a46700 1 -- 172.21.15.27:0/1101777955 shutdown_connections 2024-04-25T00:05:11.489 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:10.951+0000 7f8f69a46700 1 -- 172.21.15.27:0/1101777955 wait complete. 2024-04-25T00:05:11.489 INFO:teuthology.orchestra.run.smithi027.stdout:mon is available 2024-04-25T00:05:11.490 INFO:teuthology.orchestra.run.smithi027.stdout:Assimilating anything we can from ceph.conf... 2024-04-25T00:05:12.568 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:12 smithi027 ceph-mon[28838]: from='client.? 172.21.15.27:0/4113687024' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T00:05:12.568 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:12 smithi027 ceph-mon[28838]: from='client.? 172.21.15.27:0/4113687024' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-25T00:05:12.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T00:05:12.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T00:05:12.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout fsid = 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:05:12.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2024-04-25T00:05:12.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T00:05:12.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T00:05:12.578 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T00:05:12.578 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T00:05:12.578 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T00:05:12.578 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T00:05:12.578 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T00:05:12.578 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T00:05:12.578 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T00:05:12.578 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-04-25T00:05:12.578 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T00:05:12.578 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T00:05:12.578 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T00:05:12.578 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T00:05:12.579 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T00:05:12.579 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.882+0000 7f5fa6be7700 1 Processor -- start 2024-04-25T00:05:12.579 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.882+0000 7f5fa6be7700 1 -- start start 2024-04-25T00:05:12.579 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.883+0000 7f5fa6be7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa0107ed0 0x7f5fa01082f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:12.579 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.883+0000 7f5fa6be7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa0108830 con 0x7f5fa0107ed0 2024-04-25T00:05:12.579 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.883+0000 7f5fa4983700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa0107ed0 0x7f5fa01082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:12.579 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.883+0000 7f5fa4983700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa0107ed0 0x7f5fa01082f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52496/0 (socket says 172.21.15.27:52496) 2024-04-25T00:05:12.579 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.883+0000 7f5fa4983700 1 -- 172.21.15.27:0/2840143198 learned_addr learned my addr 172.21.15.27:0/2840143198 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:12.579 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.883+0000 7f5fa4983700 1 -- 172.21.15.27:0/2840143198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa0108fe0 con 0x7f5fa0107ed0 2024-04-25T00:05:12.579 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.883+0000 7f5fa4983700 1 --2- 172.21.15.27:0/2840143198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa0107ed0 0x7f5fa01082f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5f940098a0 tx=0x7f5f94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ae5e055285c119 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:12.579 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.883+0000 7f5f9f7fe700 1 -- 172.21.15.27:0/2840143198 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5f94004160 con 0x7f5fa0107ed0 2024-04-25T00:05:12.579 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.883+0000 7f5f9f7fe700 1 -- 172.21.15.27:0/2840143198 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f94004300 con 0x7f5fa0107ed0 2024-04-25T00:05:12.579 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.883+0000 7f5f9f7fe700 1 -- 172.21.15.27:0/2840143198 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5f940036a0 con 0x7f5fa0107ed0 2024-04-25T00:05:12.579 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.884+0000 7f5fa6be7700 1 -- 172.21.15.27:0/2840143198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa0107ed0 msgr2=0x7f5fa01082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:12.580 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.884+0000 7f5fa6be7700 1 --2- 172.21.15.27:0/2840143198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa0107ed0 0x7f5fa01082f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5f940098a0 tx=0x7f5f94009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:12.580 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.884+0000 7f5fa6be7700 1 -- 172.21.15.27:0/2840143198 shutdown_connections 2024-04-25T00:05:12.580 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.884+0000 7f5fa6be7700 1 --2- 172.21.15.27:0/2840143198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa0107ed0 0x7f5fa01082f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:12.580 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.884+0000 7f5fa6be7700 1 -- 172.21.15.27:0/2840143198 >> 172.21.15.27:0/2840143198 conn(0x7f5fa0103490 msgr2=0x7f5fa01058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:12.580 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.884+0000 7f5fa6be7700 1 -- 172.21.15.27:0/2840143198 shutdown_connections 2024-04-25T00:05:12.580 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.884+0000 7f5fa6be7700 1 -- 172.21.15.27:0/2840143198 wait complete. 2024-04-25T00:05:12.580 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.885+0000 7f5fa6be7700 1 Processor -- start 2024-04-25T00:05:12.580 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.885+0000 7f5fa6be7700 1 -- start start 2024-04-25T00:05:12.580 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.885+0000 7f5fa6be7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa0107ed0 0x7f5fa007db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:12.580 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.885+0000 7f5fa6be7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa01094b0 con 0x7f5fa0107ed0 2024-04-25T00:05:12.580 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.885+0000 7f5fa4983700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa0107ed0 0x7f5fa007db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:12.580 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.885+0000 7f5fa4983700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa0107ed0 0x7f5fa007db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52500/0 (socket says 172.21.15.27:52500) 2024-04-25T00:05:12.580 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.885+0000 7f5fa4983700 1 -- 172.21.15.27:0/4113687024 learned_addr learned my addr 172.21.15.27:0/4113687024 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:12.580 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.885+0000 7f5fa4983700 1 -- 172.21.15.27:0/4113687024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa007e070 con 0x7f5fa0107ed0 2024-04-25T00:05:12.580 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.885+0000 7f5fa4983700 1 --2- 172.21.15.27:0/4113687024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa0107ed0 0x7f5fa007db30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5f94000c00 tx=0x7f5f94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:12.581 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.885+0000 7f5f9dffb700 1 -- 172.21.15.27:0/4113687024 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5f94003db0 con 0x7f5fa0107ed0 2024-04-25T00:05:12.581 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.886+0000 7f5f9dffb700 1 -- 172.21.15.27:0/4113687024 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f94011630 con 0x7f5fa0107ed0 2024-04-25T00:05:12.581 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.886+0000 7f5f9dffb700 1 -- 172.21.15.27:0/4113687024 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5f94011950 con 0x7f5fa0107ed0 2024-04-25T00:05:12.581 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.886+0000 7f5fa6be7700 1 -- 172.21.15.27:0/4113687024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fa007c260 con 0x7f5fa0107ed0 2024-04-25T00:05:12.581 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.886+0000 7f5fa6be7700 1 -- 172.21.15.27:0/4113687024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fa007c7a0 con 0x7f5fa0107ed0 2024-04-25T00:05:12.581 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.886+0000 7f5f9dffb700 1 -- 172.21.15.27:0/4113687024 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5f94011af0 con 0x7f5fa0107ed0 2024-04-25T00:05:12.581 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.886+0000 7f5f9dffb700 1 -- 172.21.15.27:0/4113687024 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5f9401c450 con 0x7f5fa0107ed0 2024-04-25T00:05:12.581 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.887+0000 7f5fa6be7700 1 -- 172.21.15.27:0/4113687024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fa0063890 con 0x7f5fa0107ed0 2024-04-25T00:05:12.581 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.889+0000 7f5f9dffb700 1 -- 172.21.15.27:0/4113687024 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f5f94004260 con 0x7f5fa0107ed0 2024-04-25T00:05:12.581 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.958+0000 7f5fa6be7700 1 -- 172.21.15.27:0/4113687024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f5fa0194980 con 0x7f5fa0107ed0 2024-04-25T00:05:12.581 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.962+0000 7f5f9dffb700 1 -- 172.21.15.27:0/4113687024 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+454 (secure 0 0 0) 0x7f5fa0194980 con 0x7f5fa0107ed0 2024-04-25T00:05:12.581 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.962+0000 7f5f9dffb700 1 -- 172.21.15.27:0/4113687024 <== mon.0 v2:172.21.15.27:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5f9401cbb0 con 0x7f5fa0107ed0 2024-04-25T00:05:12.581 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.963+0000 7f5fa6be7700 1 -- 172.21.15.27:0/4113687024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa0107ed0 msgr2=0x7f5fa007db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:12.581 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.963+0000 7f5fa6be7700 1 --2- 172.21.15.27:0/4113687024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa0107ed0 0x7f5fa007db30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5f94000c00 tx=0x7f5f94004750 comp rx=0 tx=0).stop 2024-04-25T00:05:12.582 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.963+0000 7f5fa6be7700 1 -- 172.21.15.27:0/4113687024 shutdown_connections 2024-04-25T00:05:12.582 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.963+0000 7f5fa6be7700 1 --2- 172.21.15.27:0/4113687024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa0107ed0 0x7f5fa007db30 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:12.582 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.963+0000 7f5fa6be7700 1 -- 172.21.15.27:0/4113687024 >> 172.21.15.27:0/4113687024 conn(0x7f5fa0103490 msgr2=0x7f5fa0192030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:12.582 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.963+0000 7f5fa6be7700 1 -- 172.21.15.27:0/4113687024 shutdown_connections 2024-04-25T00:05:12.582 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:11.963+0000 7f5fa6be7700 1 -- 172.21.15.27:0/4113687024 wait complete. 2024-04-25T00:05:12.582 INFO:teuthology.orchestra.run.smithi027.stdout:Generating new minimal ceph.conf... 2024-04-25T00:05:13.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:13 smithi027 ceph-mon[28838]: from='client.? 172.21.15.27:0/715895970' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:05:13.723 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.002+0000 7f9fff80a700 1 Processor -- start 2024-04-25T00:05:13.723 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.003+0000 7f9fff80a700 1 -- start start 2024-04-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.003+0000 7f9fff80a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ff8107ed0 0x7f9ff81082f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.003+0000 7f9fff80a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff8108830 con 0x7f9ff8107ed0 2024-04-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.003+0000 7f9ffd5a6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ff8107ed0 0x7f9ff81082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.003+0000 7f9ffd5a6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ff8107ed0 0x7f9ff81082f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52512/0 (socket says 172.21.15.27:52512) 2024-04-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.003+0000 7f9ffd5a6700 1 -- 172.21.15.27:0/2480486734 learned_addr learned my addr 172.21.15.27:0/2480486734 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.003+0000 7f9ffd5a6700 1 -- 172.21.15.27:0/2480486734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff8108fe0 con 0x7f9ff8107ed0 2024-04-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.003+0000 7f9ffd5a6700 1 --2- 172.21.15.27:0/2480486734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ff8107ed0 0x7f9ff81082f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9fe80098a0 tx=0x7f9fe8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1daa7aacd00e5d17 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.004+0000 7f9feffff700 1 -- 172.21.15.27:0/2480486734 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fe8004160 con 0x7f9ff8107ed0 2024-04-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.004+0000 7f9feffff700 1 -- 172.21.15.27:0/2480486734 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9fe8004300 con 0x7f9ff8107ed0 2024-04-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.004+0000 7f9feffff700 1 -- 172.21.15.27:0/2480486734 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fe8011b50 con 0x7f9ff8107ed0 2024-04-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.004+0000 7f9fff80a700 1 -- 172.21.15.27:0/2480486734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ff8107ed0 msgr2=0x7f9ff81082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.004+0000 7f9fff80a700 1 --2- 172.21.15.27:0/2480486734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ff8107ed0 0x7f9ff81082f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9fe80098a0 tx=0x7f9fe8009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.004+0000 7f9fff80a700 1 -- 172.21.15.27:0/2480486734 shutdown_connections 2024-04-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.004+0000 7f9fff80a700 1 --2- 172.21.15.27:0/2480486734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ff8107ed0 0x7f9ff81082f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.004+0000 7f9fff80a700 1 -- 172.21.15.27:0/2480486734 >> 172.21.15.27:0/2480486734 conn(0x7f9ff8103490 msgr2=0x7f9ff81058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.004+0000 7f9fff80a700 1 -- 172.21.15.27:0/2480486734 shutdown_connections 2024-04-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.004+0000 7f9fff80a700 1 -- 172.21.15.27:0/2480486734 wait complete. 2024-04-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.005+0000 7f9fff80a700 1 Processor -- start 2024-04-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.005+0000 7f9fff80a700 1 -- start start 2024-04-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.005+0000 7f9fff80a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ff8107ed0 0x7f9ff807db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.005+0000 7f9fff80a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff810c2a0 con 0x7f9ff8107ed0 2024-04-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.005+0000 7f9ffd5a6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ff8107ed0 0x7f9ff807db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.005+0000 7f9ffd5a6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ff8107ed0 0x7f9ff807db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52514/0 (socket says 172.21.15.27:52514) 2024-04-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.005+0000 7f9ffd5a6700 1 -- 172.21.15.27:0/715895970 learned_addr learned my addr 172.21.15.27:0/715895970 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.005+0000 7f9ffd5a6700 1 -- 172.21.15.27:0/715895970 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff807e070 con 0x7f9ff8107ed0 2024-04-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.006+0000 7f9ffd5a6700 1 --2- 172.21.15.27:0/715895970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ff8107ed0 0x7f9ff807db30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8011b60 tx=0x7f9fe8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.006+0000 7f9fee7fc700 1 -- 172.21.15.27:0/715895970 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fe8003b20 con 0x7f9ff8107ed0 2024-04-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.006+0000 7f9fee7fc700 1 -- 172.21.15.27:0/715895970 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9fe8003cc0 con 0x7f9ff8107ed0 2024-04-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.006+0000 7f9fff80a700 1 -- 172.21.15.27:0/715895970 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ff807c260 con 0x7f9ff8107ed0 2024-04-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.006+0000 7f9fee7fc700 1 -- 172.21.15.27:0/715895970 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fe8024840 con 0x7f9ff8107ed0 2024-04-25T00:05:13.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.006+0000 7f9fff80a700 1 -- 172.21.15.27:0/715895970 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ff807c7a0 con 0x7f9ff8107ed0 2024-04-25T00:05:13.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.006+0000 7f9fee7fc700 1 -- 172.21.15.27:0/715895970 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9fe8022020 con 0x7f9ff8107ed0 2024-04-25T00:05:13.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.006+0000 7f9fee7fc700 1 -- 172.21.15.27:0/715895970 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9fe8011e00 con 0x7f9ff8107ed0 2024-04-25T00:05:13.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.007+0000 7f9fff80a700 1 -- 172.21.15.27:0/715895970 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ff8063890 con 0x7f9ff8107ed0 2024-04-25T00:05:13.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.009+0000 7f9fee7fc700 1 -- 172.21.15.27:0/715895970 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9fe8043dd0 con 0x7f9ff8107ed0 2024-04-25T00:05:13.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.079+0000 7f9fff80a700 1 -- 172.21.15.27:0/715895970 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f9ff807ce40 con 0x7f9ff8107ed0 2024-04-25T00:05:13.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.080+0000 7f9fee7fc700 1 -- 172.21.15.27:0/715895970 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f9fe8032030 con 0x7f9ff8107ed0 2024-04-25T00:05:13.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.080+0000 7f9fff80a700 1 -- 172.21.15.27:0/715895970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ff8107ed0 msgr2=0x7f9ff807db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:13.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.080+0000 7f9fff80a700 1 --2- 172.21.15.27:0/715895970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ff8107ed0 0x7f9ff807db30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8011b60 tx=0x7f9fe8011c40 comp rx=0 tx=0).stop 2024-04-25T00:05:13.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.080+0000 7f9fff80a700 1 -- 172.21.15.27:0/715895970 shutdown_connections 2024-04-25T00:05:13.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.080+0000 7f9fff80a700 1 --2- 172.21.15.27:0/715895970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ff8107ed0 0x7f9ff807db30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:13.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.080+0000 7f9fff80a700 1 -- 172.21.15.27:0/715895970 >> 172.21.15.27:0/715895970 conn(0x7f9ff8103490 msgr2=0x7f9ff8103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:13.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.080+0000 7f9fff80a700 1 -- 172.21.15.27:0/715895970 shutdown_connections 2024-04-25T00:05:13.727 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:13.080+0000 7f9fff80a700 1 -- 172.21.15.27:0/715895970 wait complete. 2024-04-25T00:05:13.727 INFO:teuthology.orchestra.run.smithi027.stdout:Restarting the monitor... 2024-04-25T00:05:13.978 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:13 smithi027 systemd[1]: Stopping Ceph mon.a for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:05:14.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:13 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-a[28834]: 2024-04-25T00:05:13.977+0000 7eff8d4d5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:05:14.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:13 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-a[28834]: 2024-04-25T00:05:13.977+0000 7eff8d4d5700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-25T00:05:14.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:14 smithi027 podman[29081]: 2024-04-25 00:05:14.151964889 +0000 UTC m=+0.260086396 container died 6682fbe3488a930f6b3a81945108bcaea24768dc556416703aa88f4ecae33240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-a, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:05:14.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:14 smithi027 podman[29081]: 2024-04-25 00:05:14.527849232 +0000 UTC m=+0.635970745 container cleanup 6682fbe3488a930f6b3a81945108bcaea24768dc556416703aa88f4ecae33240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T00:05:14.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:14 smithi027 bash[29081]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-a 2024-04-25T00:05:15.126 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:14 smithi027 podman[29095]: 2024-04-25 00:05:14.770511878 +0000 UTC m=+0.617447983 container remove 6682fbe3488a930f6b3a81945108bcaea24768dc556416703aa88f4ecae33240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-25T00:05:15.126 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:14 smithi027 systemd[1]: ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-25T00:05:15.126 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:14 smithi027 systemd[1]: Stopped Ceph mon.a for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:05:15.126 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:14 smithi027 systemd[1]: Starting Ceph mon.a for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:05:15.423 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 podman[29190]: 2024-04-25 00:05:15.029998957 +0000 UTC m=+0.023285845 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:05:15.423 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 podman[29190]: 2024-04-25 00:05:15.221288005 +0000 UTC m=+0.214574908 container create f663a30b3ae1a8b72665aae6baae930fb2ff028b8c889d424da825be51b63cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-a, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T00:05:15.423 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 podman[29190]: 2024-04-25 00:05:15.42353298 +0000 UTC m=+0.416819852 container init f663a30b3ae1a8b72665aae6baae930fb2ff028b8c889d424da825be51b63cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-25T00:05:15.503 INFO:teuthology.orchestra.run.smithi027.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-25T00:05:15.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 podman[29190]: 2024-04-25 00:05:15.426757777 +0000 UTC m=+0.420044639 container start f663a30b3ae1a8b72665aae6baae930fb2ff028b8c889d424da825be51b63cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-25T00:05:15.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T00:05:15.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev), process ceph-mon, pid 2 2024-04-25T00:05:15.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: pidfile_write: ignore empty --pid-file 2024-04-25T00:05:15.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: load: jerasure load: lrc 2024-04-25T00:05:15.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: RocksDB version: 7.9.2 2024-04-25T00:05:15.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Git sha 0 2024-04-25T00:05:15.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Compile date 2024-04-24 16:41:23 2024-04-25T00:05:15.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: DB SUMMARY 2024-04-25T00:05:15.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: DB Session ID: 64CAV5WVOQ80ZY4RH8IH 2024-04-25T00:05:15.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: CURRENT file: CURRENT 2024-04-25T00:05:15.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T00:05:15.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-25T00:05:15.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-25T00:05:15.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105758 ; 2024-04-25T00:05:15.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.error_if_exists: 0 2024-04-25T00:05:15.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.create_if_missing: 0 2024-04-25T00:05:15.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.paranoid_checks: 1 2024-04-25T00:05:15.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T00:05:15.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T00:05:15.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T00:05:15.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.env: 0x55f7fb38cc60 2024-04-25T00:05:15.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T00:05:15.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.info_log: 0x55f7fc325aa0 2024-04-25T00:05:15.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T00:05:15.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.statistics: (nil) 2024-04-25T00:05:15.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.use_fsync: 0 2024-04-25T00:05:15.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_log_file_size: 0 2024-04-25T00:05:15.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T00:05:15.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T00:05:15.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T00:05:15.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T00:05:15.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.allow_fallocate: 1 2024-04-25T00:05:15.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T00:05:15.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T00:05:15.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.use_direct_reads: 0 2024-04-25T00:05:15.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T00:05:15.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T00:05:15.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.db_log_dir: 2024-04-25T00:05:15.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.wal_dir: 2024-04-25T00:05:15.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T00:05:15.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T00:05:15.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T00:05:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T00:05:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T00:05:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T00:05:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T00:05:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T00:05:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.write_buffer_manager: 0x55f7fc2f81e0 2024-04-25T00:05:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T00:05:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T00:05:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T00:05:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T00:05:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T00:05:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T00:05:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T00:05:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T00:05:15.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.unordered_write: 0 2024-04-25T00:05:15.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T00:05:15.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T00:05:15.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T00:05:15.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T00:05:15.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.row_cache: None 2024-04-25T00:05:15.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.wal_filter: None 2024-04-25T00:05:15.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T00:05:15.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T00:05:15.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.two_write_queues: 0 2024-04-25T00:05:15.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T00:05:15.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.wal_compression: 0 2024-04-25T00:05:15.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.atomic_flush: 0 2024-04-25T00:05:15.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T00:05:15.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T00:05:15.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T00:05:15.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.log_readahead_size: 0 2024-04-25T00:05:15.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T00:05:15.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T00:05:15.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T00:05:15.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T00:05:15.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T00:05:15.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T00:05:15.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T00:05:15.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_background_jobs: 2 2024-04-25T00:05:15.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_background_compactions: -1 2024-04-25T00:05:15.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_subcompactions: 1 2024-04-25T00:05:15.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T00:05:15.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T00:05:15.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T00:05:15.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T00:05:15.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T00:05:15.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T00:05:15.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T00:05:15.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T00:05:15.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_open_files: -1 2024-04-25T00:05:15.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T00:05:15.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T00:05:15.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T00:05:15.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T00:05:15.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_background_flushes: -1 2024-04-25T00:05:15.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Compression algorithms supported: 2024-04-25T00:05:15.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T00:05:15.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: kZSTD supported: 0 2024-04-25T00:05:15.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: kXpressCompression supported: 0 2024-04-25T00:05:15.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T00:05:15.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: kZlibCompression supported: 1 2024-04-25T00:05:15.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: kSnappyCompression supported: 1 2024-04-25T00:05:15.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: kLZ4Compression supported: 1 2024-04-25T00:05:15.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: kBZip2Compression supported: 0 2024-04-25T00:05:15.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T00:05:15.703 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T00:05:15.703 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-25T00:05:15.703 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T00:05:15.703 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T00:05:15.703 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.merge_operator: 2024-04-25T00:05:15.703 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compaction_filter: None 2024-04-25T00:05:15.703 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T00:05:15.703 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T00:05:15.703 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T00:05:15.703 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T00:05:15.703 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f7fc325ba0) 2024-04-25T00:05:15.703 INFO:journalctl@ceph.mon.a.smithi027.stdout: cache_index_and_filter_blocks: 1 2024-04-25T00:05:15.703 INFO:journalctl@ceph.mon.a.smithi027.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T00:05:15.704 INFO:journalctl@ceph.mon.a.smithi027.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T00:05:15.704 INFO:journalctl@ceph.mon.a.smithi027.stdout: pin_top_level_index_and_filter: 1 2024-04-25T00:05:15.704 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_type: 0 2024-04-25T00:05:15.704 INFO:journalctl@ceph.mon.a.smithi027.stdout: data_block_index_type: 0 2024-04-25T00:05:15.704 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_shortening: 1 2024-04-25T00:05:15.704 INFO:journalctl@ceph.mon.a.smithi027.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T00:05:15.704 INFO:journalctl@ceph.mon.a.smithi027.stdout: checksum: 4 2024-04-25T00:05:15.704 INFO:journalctl@ceph.mon.a.smithi027.stdout: no_block_cache: 0 2024-04-25T00:05:15.705 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache: 0x55f7fc329090 2024-04-25T00:05:15.705 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_name: BinnedLRUCache 2024-04-25T00:05:15.705 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_options: 2024-04-25T00:05:15.705 INFO:journalctl@ceph.mon.a.smithi027.stdout: capacity : 536870912 2024-04-25T00:05:15.705 INFO:journalctl@ceph.mon.a.smithi027.stdout: num_shard_bits : 4 2024-04-25T00:05:15.705 INFO:journalctl@ceph.mon.a.smithi027.stdout: strict_capacity_limit : 0 2024-04-25T00:05:15.706 INFO:journalctl@ceph.mon.a.smithi027.stdout: high_pri_pool_ratio: 0.000 2024-04-25T00:05:15.706 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_compressed: (nil) 2024-04-25T00:05:15.706 INFO:journalctl@ceph.mon.a.smithi027.stdout: persistent_cache: (nil) 2024-04-25T00:05:15.706 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_size: 4096 2024-04-25T00:05:15.706 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_size_deviation: 10 2024-04-25T00:05:15.706 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_restart_interval: 16 2024-04-25T00:05:15.706 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_block_restart_interval: 1 2024-04-25T00:05:15.706 INFO:journalctl@ceph.mon.a.smithi027.stdout: metadata_block_size: 4096 2024-04-25T00:05:15.706 INFO:journalctl@ceph.mon.a.smithi027.stdout: partition_filters: 0 2024-04-25T00:05:15.706 INFO:journalctl@ceph.mon.a.smithi027.stdout: use_delta_encoding: 1 2024-04-25T00:05:15.706 INFO:journalctl@ceph.mon.a.smithi027.stdout: filter_policy: bloomfilter 2024-04-25T00:05:15.706 INFO:journalctl@ceph.mon.a.smithi027.stdout: whole_key_filtering: 1 2024-04-25T00:05:15.706 INFO:journalctl@ceph.mon.a.smithi027.stdout: verify_compression: 0 2024-04-25T00:05:15.706 INFO:journalctl@ceph.mon.a.smithi027.stdout: read_amp_bytes_per_bit: 0 2024-04-25T00:05:15.707 INFO:journalctl@ceph.mon.a.smithi027.stdout: format_version: 5 2024-04-25T00:05:15.707 INFO:journalctl@ceph.mon.a.smithi027.stdout: enable_index_compression: 1 2024-04-25T00:05:15.707 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_align: 0 2024-04-25T00:05:15.707 INFO:journalctl@ceph.mon.a.smithi027.stdout: max_auto_readahead_size: 262144 2024-04-25T00:05:15.707 INFO:journalctl@ceph.mon.a.smithi027.stdout: prepopulate_block_cache: 0 2024-04-25T00:05:15.707 INFO:journalctl@ceph.mon.a.smithi027.stdout: initial_auto_readahead_size: 8192 2024-04-25T00:05:15.707 INFO:journalctl@ceph.mon.a.smithi027.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T00:05:15.707 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T00:05:15.707 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T00:05:15.707 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compression: NoCompression 2024-04-25T00:05:15.707 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T00:05:15.707 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T00:05:15.707 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T00:05:15.707 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.num_levels: 7 2024-04-25T00:05:15.707 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T00:05:15.708 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T00:05:15.708 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T00:05:15.708 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T00:05:15.708 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T00:05:15.709 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T00:05:15.709 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T00:05:15.709 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T00:05:15.709 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T00:05:15.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T00:05:15.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T00:05:15.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T00:05:15.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T00:05:15.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T00:05:15.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T00:05:15.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T00:05:15.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T00:05:15.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T00:05:15.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T00:05:15.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T00:05:15.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T00:05:15.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T00:05:15.712 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T00:05:15.712 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T00:05:15.712 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T00:05:15.712 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T00:05:15.712 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T00:05:15.712 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T00:05:15.712 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T00:05:15.712 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T00:05:15.712 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T00:05:15.712 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T00:05:15.712 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T00:05:15.712 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T00:05:15.712 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T00:05:15.712 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T00:05:15.713 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T00:05:15.713 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T00:05:15.713 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T00:05:15.713 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T00:05:15.713 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T00:05:15.713 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T00:05:15.713 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T00:05:15.713 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T00:05:15.713 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T00:05:15.713 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T00:05:15.713 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T00:05:15.714 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T00:05:15.714 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T00:05:15.714 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T00:05:15.714 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T00:05:15.714 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T00:05:15.714 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T00:05:15.714 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.table_properties_collectors: 2024-04-25T00:05:15.714 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.inplace_update_support: 0 2024-04-25T00:05:15.714 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T00:05:15.714 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T00:05:15.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T00:05:15.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T00:05:15.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.bloom_locality: 0 2024-04-25T00:05:15.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.max_successive_merges: 0 2024-04-25T00:05:15.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T00:05:15.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T00:05:15.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T00:05:15.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T00:05:15.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.ttl: 2592000 2024-04-25T00:05:15.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T00:05:15.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T00:05:15.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T00:05:15.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.enable_blob_files: false 2024-04-25T00:05:15.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.min_blob_size: 0 2024-04-25T00:05:15.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T00:05:15.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T00:05:15.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T00:05:15.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T00:05:15.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T00:05:15.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T00:05:15.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T00:05:15.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T00:05:15.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-25T00:05:15.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-25T00:05:15.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e205c20e-2004-40bb-a141-2f31e3632685 2024-04-25T00:05:15.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714003515456242, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-25T00:05:15.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-25T00:05:15.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714003515457246, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100741, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 330, "table_properties": {"data_size": 98661, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14423, "raw_average_key_size": 49, "raw_value_size": 90860, "raw_average_value_size": 314, "num_data_blocks": 12, "num_entries": 289, "num_filter_entries": 289, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714003515, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e205c20e-2004-40bb-a141-2f31e3632685", "db_session_id": "64CAV5WVOQ80ZY4RH8IH", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-25T00:05:15.717 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714003515457332, "job": 1, "event": "recovery_finished"} 2024-04-25T00:05:15.717 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-25T00:05:15.717 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T00:05:15.717 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f7fc41a000 2024-04-25T00:05:15.717 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: DB pointer 0x55f7fc404000 2024-04-25T00:05:15.717 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T00:05:15.717 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T00:05:15.717 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** DB Stats ** 2024-04-25T00:05:15.717 INFO:journalctl@ceph.mon.a.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T00:05:15.717 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T00:05:15.717 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T00:05:15.717 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T00:05:15.717 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T00:05:15.717 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T00:05:15.718 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T00:05:15.718 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T00:05:15.718 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-25T00:05:15.718 INFO:journalctl@ceph.mon.a.smithi027.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T00:05:15.718 INFO:journalctl@ceph.mon.a.smithi027.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T00:05:15.718 INFO:journalctl@ceph.mon.a.smithi027.stdout: L0 2/0 100.21 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T00:05:15.718 INFO:journalctl@ceph.mon.a.smithi027.stdout: Sum 2/0 100.21 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T00:05:15.718 INFO:journalctl@ceph.mon.a.smithi027.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T00:05:15.718 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T00:05:15.718 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-25T00:05:15.718 INFO:journalctl@ceph.mon.a.smithi027.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T00:05:15.718 INFO:journalctl@ceph.mon.a.smithi027.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T00:05:15.718 INFO:journalctl@ceph.mon.a.smithi027.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 160.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T00:05:15.719 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T00:05:15.719 INFO:journalctl@ceph.mon.a.smithi027.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T00:05:15.719 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T00:05:15.719 INFO:journalctl@ceph.mon.a.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T00:05:15.719 INFO:journalctl@ceph.mon.a.smithi027.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T00:05:15.719 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T00:05:15.719 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T00:05:15.719 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T00:05:15.719 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T00:05:15.719 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative compaction: 0.00 GB write, 31.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T00:05:15.719 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval compaction: 0.00 GB write, 31.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T00:05:15.719 INFO:journalctl@ceph.mon.a.smithi027.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T00:05:15.719 INFO:journalctl@ceph.mon.a.smithi027.stdout: Block cache BinnedLRUCache@0x55f7fc329090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-25T00:05:15.719 INFO:journalctl@ceph.mon.a.smithi027.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-25T00:05:15.719 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T00:05:15.720 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T00:05:15.720 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: starting mon.a rank 0 at public addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] at bind addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:05:15.720 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: mon.a@-1(???) e1 preinit fsid 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:05:15.720 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: mon.a@-1(???).mds e1 new map 2024-04-25T00:05:15.720 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: mon.a@-1(???).mds e1 print_map 2024-04-25T00:05:15.720 INFO:journalctl@ceph.mon.a.smithi027.stdout: e1 2024-04-25T00:05:15.720 INFO:journalctl@ceph.mon.a.smithi027.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T00:05:15.720 INFO:journalctl@ceph.mon.a.smithi027.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T00:05:15.720 INFO:journalctl@ceph.mon.a.smithi027.stdout: legacy client fscid: -1 2024-04-25T00:05:15.720 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T00:05:15.720 INFO:journalctl@ceph.mon.a.smithi027.stdout: No filesystems configured 2024-04-25T00:05:15.720 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T00:05:15.720 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T00:05:15.720 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T00:05:15.721 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T00:05:15.721 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-25T00:05:15.721 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 bash[29190]: f663a30b3ae1a8b72665aae6baae930fb2ff028b8c889d424da825be51b63cab 2024-04-25T00:05:15.721 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 systemd[1]: Started Ceph mon.a for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:05:15.721 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T00:05:15.721 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: monmap epoch 1 2024-04-25T00:05:15.721 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: fsid 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:05:15.721 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: last_changed 2024-04-25T00:05:07.441245+0000 2024-04-25T00:05:15.721 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: created 2024-04-25T00:05:07.441245+0000 2024-04-25T00:05:15.721 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: min_mon_release 19 (squid) 2024-04-25T00:05:15.721 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: election_strategy: 1 2024-04-25T00:05:15.721 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T00:05:15.721 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: fsmap 2024-04-25T00:05:15.721 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T00:05:15.722 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:15 smithi027 ceph-mon[29208]: mgrmap e1: no daemons active 2024-04-25T00:05:16.603 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.969+0000 7fac367a2700 1 Processor -- start 2024-04-25T00:05:16.603 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.970+0000 7fac367a2700 1 -- start start 2024-04-25T00:05:16.603 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.970+0000 7fac367a2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac30108910 0x7fac3010ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:16.603 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.970+0000 7fac367a2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fac3007be30 con 0x7fac30108910 2024-04-25T00:05:16.603 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.970+0000 7fac2ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac30108910 0x7fac3010ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:16.603 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.970+0000 7fac2ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac30108910 0x7fac3010ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52528/0 (socket says 172.21.15.27:52528) 2024-04-25T00:05:16.603 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.970+0000 7fac2ffff700 1 -- 172.21.15.27:0/2093687346 learned_addr learned my addr 172.21.15.27:0/2093687346 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:16.603 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.971+0000 7fac2ffff700 1 -- 172.21.15.27:0/2093687346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac3010b240 con 0x7fac30108910 2024-04-25T00:05:16.603 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.971+0000 7fac2ffff700 1 --2- 172.21.15.27:0/2093687346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac30108910 0x7fac3010ad00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fac200098a0 tx=0x7fac20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc7b50cd960acf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:16.604 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.971+0000 7fac2effd700 1 -- 172.21.15.27:0/2093687346 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac20004160 con 0x7fac30108910 2024-04-25T00:05:16.604 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.971+0000 7fac2effd700 1 -- 172.21.15.27:0/2093687346 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fac20004300 con 0x7fac30108910 2024-04-25T00:05:16.604 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.971+0000 7fac2effd700 1 -- 172.21.15.27:0/2093687346 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac20011b50 con 0x7fac30108910 2024-04-25T00:05:16.604 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.972+0000 7fac367a2700 1 -- 172.21.15.27:0/2093687346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac30108910 msgr2=0x7fac3010ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:16.604 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.972+0000 7fac367a2700 1 --2- 172.21.15.27:0/2093687346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac30108910 0x7fac3010ad00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fac200098a0 tx=0x7fac20009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:16.604 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.972+0000 7fac367a2700 1 -- 172.21.15.27:0/2093687346 shutdown_connections 2024-04-25T00:05:16.604 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.972+0000 7fac367a2700 1 --2- 172.21.15.27:0/2093687346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac30108910 0x7fac3010ad00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:16.604 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.972+0000 7fac367a2700 1 -- 172.21.15.27:0/2093687346 >> 172.21.15.27:0/2093687346 conn(0x7fac30102340 msgr2=0x7fac30104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:16.604 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.972+0000 7fac367a2700 1 -- 172.21.15.27:0/2093687346 shutdown_connections 2024-04-25T00:05:16.604 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.972+0000 7fac367a2700 1 -- 172.21.15.27:0/2093687346 wait complete. 2024-04-25T00:05:16.604 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.972+0000 7fac367a2700 1 Processor -- start 2024-04-25T00:05:16.604 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.972+0000 7fac367a2700 1 -- start start 2024-04-25T00:05:16.604 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.973+0000 7fac367a2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac30108910 0x7fac30108190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:16.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.973+0000 7fac367a2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fac3010c1f0 con 0x7fac30108910 2024-04-25T00:05:16.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.973+0000 7fac2ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac30108910 0x7fac30108190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:16.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.973+0000 7fac2ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac30108910 0x7fac30108190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52536/0 (socket says 172.21.15.27:52536) 2024-04-25T00:05:16.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.973+0000 7fac2ffff700 1 -- 172.21.15.27:0/2874053546 learned_addr learned my addr 172.21.15.27:0/2874053546 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:16.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.973+0000 7fac2ffff700 1 -- 172.21.15.27:0/2874053546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac301086d0 con 0x7fac30108910 2024-04-25T00:05:16.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.973+0000 7fac2ffff700 1 --2- 172.21.15.27:0/2874053546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac30108910 0x7fac30108190 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fac20011b60 tx=0x7fac20011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:16.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.973+0000 7fac2d7fa700 1 -- 172.21.15.27:0/2874053546 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac20003b20 con 0x7fac30108910 2024-04-25T00:05:16.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.973+0000 7fac2d7fa700 1 -- 172.21.15.27:0/2874053546 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fac20003cc0 con 0x7fac30108910 2024-04-25T00:05:16.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.973+0000 7fac367a2700 1 -- 172.21.15.27:0/2874053546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac301068c0 con 0x7fac30108910 2024-04-25T00:05:16.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.973+0000 7fac2d7fa700 1 -- 172.21.15.27:0/2874053546 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac20024840 con 0x7fac30108910 2024-04-25T00:05:16.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.973+0000 7fac367a2700 1 -- 172.21.15.27:0/2874053546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac30106e00 con 0x7fac30108910 2024-04-25T00:05:16.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.974+0000 7fac2d7fa700 1 -- 172.21.15.27:0/2874053546 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fac20022020 con 0x7fac30108910 2024-04-25T00:05:16.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.974+0000 7fac2d7fa700 1 -- 172.21.15.27:0/2874053546 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fac20011e00 con 0x7fac30108910 2024-04-25T00:05:16.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.974+0000 7fac367a2700 1 -- 172.21.15.27:0/2874053546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac30063890 con 0x7fac30108910 2024-04-25T00:05:16.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:15.977+0000 7fac2d7fa700 1 -- 172.21.15.27:0/2874053546 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fac20043dd0 con 0x7fac30108910 2024-04-25T00:05:16.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:16.049+0000 7fac367a2700 1 -- 172.21.15.27:0/2874053546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fac301070b0 con 0x7fac30108910 2024-04-25T00:05:16.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:16.051+0000 7fac2d7fa700 1 -- 172.21.15.27:0/2874053546 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fac20032030 con 0x7fac30108910 2024-04-25T00:05:16.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:16.051+0000 7fac367a2700 1 -- 172.21.15.27:0/2874053546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac30108910 msgr2=0x7fac30108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:16.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:16.051+0000 7fac367a2700 1 --2- 172.21.15.27:0/2874053546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac30108910 0x7fac30108190 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fac20011b60 tx=0x7fac20011c40 comp rx=0 tx=0).stop 2024-04-25T00:05:16.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:16.051+0000 7fac367a2700 1 -- 172.21.15.27:0/2874053546 shutdown_connections 2024-04-25T00:05:16.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:16.051+0000 7fac367a2700 1 --2- 172.21.15.27:0/2874053546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac30108910 0x7fac30108190 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:16.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:16.051+0000 7fac367a2700 1 -- 172.21.15.27:0/2874053546 >> 172.21.15.27:0/2874053546 conn(0x7fac30102340 msgr2=0x7fac30102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:16.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:16.052+0000 7fac367a2700 1 -- 172.21.15.27:0/2874053546 shutdown_connections 2024-04-25T00:05:16.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:16.052+0000 7fac367a2700 1 -- 172.21.15.27:0/2874053546 wait complete. 2024-04-25T00:05:16.606 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-25T00:05:16.621 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-25T00:05:16.621 INFO:teuthology.orchestra.run.smithi027.stdout:Creating mgr... 2024-04-25T00:05:16.621 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-25T00:05:16.622 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-25T00:05:16.881 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-518857f8-0297-11ef-bc93-c7b262605968@mgr.x 2024-04-25T00:05:16.881 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to reset failed state of unit ceph-518857f8-0297-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-518857f8-0297-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-25T00:05:17.031 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-518857f8-0297-11ef-bc93-c7b262605968.target.wants/ceph-518857f8-0297-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-518857f8-0297-11ef-bc93-c7b262605968@.service. 2024-04-25T00:05:17.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:17 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2874053546' entity='client.admin' 2024-04-25T00:05:17.192 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:17 smithi027 systemd[1]: Starting Ceph mgr.x for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:05:17.567 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:17 smithi027 podman[29426]: 2024-04-25 00:05:17.175555414 +0000 UTC m=+0.018151169 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:05:17.567 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:17 smithi027 podman[29426]: 2024-04-25 00:05:17.290196011 +0000 UTC m=+0.132791737 container create f7edeb76725eec46259d2c9af45d924ae873e77e218472c31ef5878a55f8b171 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-04-25T00:05:17.567 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:17 smithi027 podman[29426]: 2024-04-25 00:05:17.492307282 +0000 UTC m=+0.334903022 container init f7edeb76725eec46259d2c9af45d924ae873e77e218472c31ef5878a55f8b171 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T00:05:17.567 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:17 smithi027 podman[29426]: 2024-04-25 00:05:17.494971683 +0000 UTC m=+0.337567409 container start f7edeb76725eec46259d2c9af45d924ae873e77e218472c31ef5878a55f8b171 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T00:05:17.567 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:17 smithi027 bash[29426]: f7edeb76725eec46259d2c9af45d924ae873e77e218472c31ef5878a55f8b171 2024-04-25T00:05:17.580 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T00:05:17.580 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-04-25T00:05:17.587 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T00:05:17.588 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T00:05:17.588 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-04-25T00:05:17.588 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T00:05:17.596 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T00:05:17.596 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-04-25T00:05:17.603 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T00:05:17.604 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T00:05:17.604 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-04-25T00:05:17.604 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-25T00:05:17.604 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr to start... 2024-04-25T00:05:17.604 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr... 2024-04-25T00:05:17.842 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:17 smithi027 systemd[1]: Started Ceph mgr.x for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:05:17.842 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:17 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:17.651+0000 7f7dbf025200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T00:05:17.842 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:17 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:17.841+0000 7f7dbf025200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:05:18.122 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:17 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:17.949+0000 7f7dbf025200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T00:05:18.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:18 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2843367301' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T00:05:18.442 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:18 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:18.281+0000 7f7dbf025200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T00:05:18.442 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:18 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:18.369+0000 7f7dbf025200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T00:05:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T00:05:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T00:05:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "518857f8-0297-11ef-bc93-c7b262605968", 2024-04-25T00:05:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T00:05:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T00:05:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T00:05:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T00:05:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T00:05:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T00:05:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-04-25T00:05:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T00:05:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T00:05:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T00:05:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T00:05:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-25T00:05:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T00:05:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T00:05:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T00:05:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T00:05:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T00:05:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T00:05:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T00:05:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T00:05:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T00:05:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T00:05:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T00:05:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T00:05:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T00:05:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T00:05:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T00:05:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T00:05:18.839 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T00:05:18.839 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T00:05:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T00:05:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T00:05:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T00:05:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T00:05:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T00:05:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T00:05:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T00:05:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T00:05:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T00:05:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T00:05:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T00:05:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T00:05:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T00:05:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T00:05:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T00:05:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T00:05:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T00:05:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T00:05:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T00:05:10.376600+0000", 2024-04-25T00:05:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T00:05:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T00:05:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T00:05:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.150+0000 7f4df423e700 1 Processor -- start 2024-04-25T00:05:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.150+0000 7f4df423e700 1 -- start start 2024-04-25T00:05:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.150+0000 7f4df423e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dec107eb0 0x7f4dec1082d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.150+0000 7f4df423e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec108810 con 0x7f4dec107eb0 2024-04-25T00:05:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.151+0000 7f4df1fda700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dec107eb0 0x7f4dec1082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.151+0000 7f4df1fda700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dec107eb0 0x7f4dec1082d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49588/0 (socket says 172.21.15.27:49588) 2024-04-25T00:05:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.151+0000 7f4df1fda700 1 -- 172.21.15.27:0/3506764858 learned_addr learned my addr 172.21.15.27:0/3506764858 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.151+0000 7f4df1fda700 1 -- 172.21.15.27:0/3506764858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dec108f70 con 0x7f4dec107eb0 2024-04-25T00:05:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.151+0000 7f4df1fda700 1 --2- 172.21.15.27:0/3506764858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dec107eb0 0x7f4dec1082d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4de80098a0 tx=0x7f4de8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8388a9c974658336 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.152+0000 7f4df0fd8700 1 -- 172.21.15.27:0/3506764858 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4de8004160 con 0x7f4dec107eb0 2024-04-25T00:05:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.152+0000 7f4df0fd8700 1 -- 172.21.15.27:0/3506764858 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4de8004300 con 0x7f4dec107eb0 2024-04-25T00:05:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.152+0000 7f4df0fd8700 1 -- 172.21.15.27:0/3506764858 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4de8011b50 con 0x7f4dec107eb0 2024-04-25T00:05:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.152+0000 7f4df423e700 1 -- 172.21.15.27:0/3506764858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dec107eb0 msgr2=0x7f4dec1082d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.152+0000 7f4df423e700 1 --2- 172.21.15.27:0/3506764858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dec107eb0 0x7f4dec1082d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4de80098a0 tx=0x7f4de8009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.152+0000 7f4df423e700 1 -- 172.21.15.27:0/3506764858 shutdown_connections 2024-04-25T00:05:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.152+0000 7f4df423e700 1 --2- 172.21.15.27:0/3506764858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dec107eb0 0x7f4dec1082d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.152+0000 7f4df423e700 1 -- 172.21.15.27:0/3506764858 >> 172.21.15.27:0/3506764858 conn(0x7f4dec1034b0 msgr2=0x7f4dec1058d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.152+0000 7f4df423e700 1 -- 172.21.15.27:0/3506764858 shutdown_connections 2024-04-25T00:05:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.152+0000 7f4df423e700 1 -- 172.21.15.27:0/3506764858 wait complete. 2024-04-25T00:05:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.153+0000 7f4df423e700 1 Processor -- start 2024-04-25T00:05:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.153+0000 7f4df423e700 1 -- start start 2024-04-25T00:05:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.153+0000 7f4df423e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dec107eb0 0x7f4dec19bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.153+0000 7f4df423e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec10c230 con 0x7f4dec107eb0 2024-04-25T00:05:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.153+0000 7f4df1fda700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dec107eb0 0x7f4dec19bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.153+0000 7f4df1fda700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dec107eb0 0x7f4dec19bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49600/0 (socket says 172.21.15.27:49600) 2024-04-25T00:05:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.153+0000 7f4df1fda700 1 -- 172.21.15.27:0/2843367301 learned_addr learned my addr 172.21.15.27:0/2843367301 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.154+0000 7f4df1fda700 1 -- 172.21.15.27:0/2843367301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dec19c280 con 0x7f4dec107eb0 2024-04-25T00:05:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.154+0000 7f4df1fda700 1 --2- 172.21.15.27:0/2843367301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dec107eb0 0x7f4dec19bd40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4de8011b60 tx=0x7f4de8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.154+0000 7f4de2ffd700 1 -- 172.21.15.27:0/2843367301 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4de8003b20 con 0x7f4dec107eb0 2024-04-25T00:05:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.154+0000 7f4de2ffd700 1 -- 172.21.15.27:0/2843367301 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4de8003cc0 con 0x7f4dec107eb0 2024-04-25T00:05:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.154+0000 7f4df423e700 1 -- 172.21.15.27:0/2843367301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dec19c510 con 0x7f4dec107eb0 2024-04-25T00:05:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.154+0000 7f4de2ffd700 1 -- 172.21.15.27:0/2843367301 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4de8024840 con 0x7f4dec107eb0 2024-04-25T00:05:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.154+0000 7f4df423e700 1 -- 172.21.15.27:0/2843367301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dec19f200 con 0x7f4dec107eb0 2024-04-25T00:05:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.154+0000 7f4de2ffd700 1 -- 172.21.15.27:0/2843367301 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4de8022020 con 0x7f4dec107eb0 2024-04-25T00:05:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.154+0000 7f4de2ffd700 1 -- 172.21.15.27:0/2843367301 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4de8011e00 con 0x7f4dec107eb0 2024-04-25T00:05:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.155+0000 7f4df423e700 1 -- 172.21.15.27:0/2843367301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dec10c3b0 con 0x7f4dec107eb0 2024-04-25T00:05:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.157+0000 7f4de2ffd700 1 -- 172.21.15.27:0/2843367301 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4de8043dd0 con 0x7f4dec107eb0 2024-04-25T00:05:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.230+0000 7f4df423e700 1 -- 172.21.15.27:0/2843367301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f4dec19f3d0 con 0x7f4dec107eb0 2024-04-25T00:05:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.231+0000 7f4de2ffd700 1 -- 172.21.15.27:0/2843367301 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f4de8032020 con 0x7f4dec107eb0 2024-04-25T00:05:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.232+0000 7f4df423e700 1 -- 172.21.15.27:0/2843367301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dec107eb0 msgr2=0x7f4dec19bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.232+0000 7f4df423e700 1 --2- 172.21.15.27:0/2843367301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dec107eb0 0x7f4dec19bd40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4de8011b60 tx=0x7f4de8011c40 comp rx=0 tx=0).stop 2024-04-25T00:05:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.232+0000 7f4df423e700 1 -- 172.21.15.27:0/2843367301 shutdown_connections 2024-04-25T00:05:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.232+0000 7f4df423e700 1 --2- 172.21.15.27:0/2843367301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dec107eb0 0x7f4dec19bd40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.232+0000 7f4df423e700 1 -- 172.21.15.27:0/2843367301 >> 172.21.15.27:0/2843367301 conn(0x7f4dec1034b0 msgr2=0x7f4dec103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.232+0000 7f4df423e700 1 -- 172.21.15.27:0/2843367301 shutdown_connections 2024-04-25T00:05:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:18.232+0000 7f4df423e700 1 -- 172.21.15.27:0/2843367301 wait complete. 2024-04-25T00:05:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (1/15)... 2024-04-25T00:05:19.942 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:19 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:19.650+0000 7f7dbf025200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T00:05:19.942 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:19 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:19.826+0000 7f7dbf025200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T00:05:20.301 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:20 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:20.002+0000 7f7dbf025200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T00:05:20.363 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:20 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:20.300+0000 7f7dbf025200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T00:05:20.606 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:20 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:20.395+0000 7f7dbf025200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T00:05:20.606 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:20 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:20.605+0000 7f7dbf025200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T00:05:20.942 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:20 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:20.803+0000 7f7dbf025200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:05:21.692 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:21 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:21.440+0000 7f7dbf025200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T00:05:22.192 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:21 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:21.880+0000 7f7dbf025200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T00:05:22.192 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:21 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:21.980+0000 7f7dbf025200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T00:05:22.692 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:22 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:22.191+0000 7f7dbf025200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T00:05:23.064 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:22 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:22.741+0000 7f7dbf025200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T00:05:23.065 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:22 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:22.836+0000 7f7dbf025200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T00:05:23.336 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:23 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3585247469' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T00:05:23.336 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:23 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:23.090+0000 7f7dbf025200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T00:05:23.336 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:23 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:23.233+0000 7f7dbf025200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T00:05:23.336 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:23 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:23.335+0000 7f7dbf025200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T00:05:23.692 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:23 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:23.456+0000 7f7dbf025200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T00:05:23.692 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:23 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:23.551+0000 7f7dbf025200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T00:05:24.680 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:24 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:24.400+0000 7f7dbf025200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T00:05:24.680 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:24 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:24.624+0000 7f7dbf025200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T00:05:24.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:24 smithi027 ceph-mon[29208]: Activating manager daemon x 2024-04-25T00:05:24.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:24 smithi027 ceph-mon[29208]: mgrmap e2: x(active, starting, since 0.00299478s) 2024-04-25T00:05:24.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:24 smithi027 ceph-mon[29208]: from='mgr.14100 172.21.15.27:0/1013572367' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T00:05:24.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:24 smithi027 ceph-mon[29208]: from='mgr.14100 172.21.15.27:0/1013572367' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T00:05:24.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:24 smithi027 ceph-mon[29208]: from='mgr.14100 172.21.15.27:0/1013572367' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T00:05:24.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:24 smithi027 ceph-mon[29208]: from='mgr.14100 172.21.15.27:0/1013572367' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:05:24.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:24 smithi027 ceph-mon[29208]: from='mgr.14100 172.21.15.27:0/1013572367' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T00:05:24.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:24 smithi027 ceph-mon[29208]: Manager daemon x is now available 2024-04-25T00:05:24.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:24 smithi027 ceph-mon[29208]: from='mgr.14100 172.21.15.27:0/1013572367' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T00:05:24.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:24 smithi027 ceph-mon[29208]: from='mgr.14100 172.21.15.27:0/1013572367' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T00:05:24.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:24 smithi027 ceph-mon[29208]: from='mgr.14100 172.21.15.27:0/1013572367' entity='mgr.x' 2024-04-25T00:05:24.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:24 smithi027 ceph-mon[29208]: from='mgr.14100 172.21.15.27:0/1013572367' entity='mgr.x' 2024-04-25T00:05:24.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:24 smithi027 ceph-mon[29208]: from='mgr.14100 172.21.15.27:0/1013572367' entity='mgr.x' 2024-04-25T00:05:26.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:26 smithi027 ceph-mon[29208]: mgrmap e3: x(active, since 1.00763s) 2024-04-25T00:05:27.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:27 smithi027 ceph-mon[29208]: mgrmap e4: x(active, since 2s) 2024-04-25T00:05:29.252 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T00:05:29.252 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T00:05:29.252 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "518857f8-0297-11ef-bc93-c7b262605968", 2024-04-25T00:05:29.252 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T00:05:29.252 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T00:05:29.252 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T00:05:29.252 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T00:05:29.252 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T00:05:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T00:05:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-04-25T00:05:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T00:05:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T00:05:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T00:05:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T00:05:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-25T00:05:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T00:05:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T00:05:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T00:05:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T00:05:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:29.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T00:05:29.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T00:05:29.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T00:05:29.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T00:05:29.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T00:05:29.262 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T00:05:29.262 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T00:05:29.262 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T00:05:29.262 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:29.262 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T00:05:29.262 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T00:05:29.262 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T00:05:29.262 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T00:05:29.262 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T00:05:29.263 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T00:05:29.263 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T00:05:29.263 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T00:05:29.263 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T00:05:29.263 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:29.263 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T00:05:29.263 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T00:05:29.263 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T00:05:29.263 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T00:05:29.263 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:29.263 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T00:05:29.263 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T00:05:29.263 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T00:05:29.263 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T00:05:29.263 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T00:05:29.263 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T00:05:29.264 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T00:05:29.264 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T00:05:29.264 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T00:05:29.264 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:29.264 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T00:05:29.264 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T00:05:29.264 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T00:05:10.376600+0000", 2024-04-25T00:05:29.264 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T00:05:29.264 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:29.264 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T00:05:29.264 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T00:05:29.264 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.164+0000 7f442e159700 1 Processor -- start 2024-04-25T00:05:29.264 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.164+0000 7f442e159700 1 -- start start 2024-04-25T00:05:29.264 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.164+0000 7f442e159700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44281089f0 0x7f442810ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:29.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.164+0000 7f442e159700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f442807bd90 con 0x7f44281089f0 2024-04-25T00:05:29.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.165+0000 7f44277fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44281089f0 0x7f442810ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:29.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.165+0000 7f44277fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44281089f0 0x7f442810ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49606/0 (socket says 172.21.15.27:49606) 2024-04-25T00:05:29.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.165+0000 7f44277fe700 1 -- 172.21.15.27:0/3932867147 learned_addr learned my addr 172.21.15.27:0/3932867147 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:29.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.165+0000 7f44277fe700 1 -- 172.21.15.27:0/3932867147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f442810b320 con 0x7f44281089f0 2024-04-25T00:05:29.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.165+0000 7f44277fe700 1 --2- 172.21.15.27:0/3932867147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44281089f0 0x7f442810ade0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f44180098a0 tx=0x7f4418009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=147254566a34fb21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:29.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.165+0000 7f44267fc700 1 -- 172.21.15.27:0/3932867147 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4418004160 con 0x7f44281089f0 2024-04-25T00:05:29.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.165+0000 7f44267fc700 1 -- 172.21.15.27:0/3932867147 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4418004300 con 0x7f44281089f0 2024-04-25T00:05:29.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.165+0000 7f44267fc700 1 -- 172.21.15.27:0/3932867147 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4418011b50 con 0x7f44281089f0 2024-04-25T00:05:29.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.166+0000 7f442e159700 1 -- 172.21.15.27:0/3932867147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44281089f0 msgr2=0x7f442810ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:29.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.166+0000 7f442e159700 1 --2- 172.21.15.27:0/3932867147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44281089f0 0x7f442810ade0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f44180098a0 tx=0x7f4418009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:29.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.166+0000 7f442e159700 1 -- 172.21.15.27:0/3932867147 shutdown_connections 2024-04-25T00:05:29.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.166+0000 7f442e159700 1 --2- 172.21.15.27:0/3932867147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44281089f0 0x7f442810ade0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:29.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.166+0000 7f442e159700 1 -- 172.21.15.27:0/3932867147 >> 172.21.15.27:0/3932867147 conn(0x7f4428102360 msgr2=0x7f4428104780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:29.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.166+0000 7f442e159700 1 -- 172.21.15.27:0/3932867147 shutdown_connections 2024-04-25T00:05:29.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.166+0000 7f442e159700 1 -- 172.21.15.27:0/3932867147 wait complete. 2024-04-25T00:05:29.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.167+0000 7f442e159700 1 Processor -- start 2024-04-25T00:05:29.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.167+0000 7f442e159700 1 -- start start 2024-04-25T00:05:29.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.167+0000 7f442e159700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44281089f0 0x7f4428108270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:29.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.167+0000 7f442e159700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f442810c210 con 0x7f44281089f0 2024-04-25T00:05:29.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.167+0000 7f44277fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44281089f0 0x7f4428108270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:29.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.167+0000 7f44277fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44281089f0 0x7f4428108270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49622/0 (socket says 172.21.15.27:49622) 2024-04-25T00:05:29.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.167+0000 7f44277fe700 1 -- 172.21.15.27:0/3585247469 learned_addr learned my addr 172.21.15.27:0/3585247469 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:29.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.167+0000 7f44277fe700 1 -- 172.21.15.27:0/3585247469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44281087b0 con 0x7f44281089f0 2024-04-25T00:05:29.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.168+0000 7f44277fe700 1 --2- 172.21.15.27:0/3585247469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44281089f0 0x7f4428108270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4418009fb0 tx=0x7f4418011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:29.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.168+0000 7f4424ff9700 1 -- 172.21.15.27:0/3585247469 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44180038b0 con 0x7f44281089f0 2024-04-25T00:05:29.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.168+0000 7f4424ff9700 1 -- 172.21.15.27:0/3585247469 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4418011e50 con 0x7f44281089f0 2024-04-25T00:05:29.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.168+0000 7f442e159700 1 -- 172.21.15.27:0/3585247469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44281069a0 con 0x7f44281089f0 2024-04-25T00:05:29.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.168+0000 7f4424ff9700 1 -- 172.21.15.27:0/3585247469 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4418019420 con 0x7f44281089f0 2024-04-25T00:05:29.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.168+0000 7f442e159700 1 -- 172.21.15.27:0/3585247469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4428106ee0 con 0x7f44281089f0 2024-04-25T00:05:29.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.168+0000 7f4424ff9700 1 -- 172.21.15.27:0/3585247469 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4418019b30 con 0x7f44281089f0 2024-04-25T00:05:29.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.168+0000 7f4424ff9700 1 -- 172.21.15.27:0/3585247469 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4418024730 con 0x7f44281089f0 2024-04-25T00:05:29.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.169+0000 7f442e159700 1 -- 172.21.15.27:0/3585247469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4428063890 con 0x7f44281089f0 2024-04-25T00:05:29.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.171+0000 7f4424ff9700 1 -- 172.21.15.27:0/3585247469 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4418043dd0 con 0x7f44281089f0 2024-04-25T00:05:29.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.245+0000 7f442e159700 1 -- 172.21.15.27:0/3585247469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f44281073f0 con 0x7f44281089f0 2024-04-25T00:05:29.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.245+0000 7f4424ff9700 1 -- 172.21.15.27:0/3585247469 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f4418032030 con 0x7f44281089f0 2024-04-25T00:05:29.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.246+0000 7f442e159700 1 -- 172.21.15.27:0/3585247469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44281089f0 msgr2=0x7f4428108270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:29.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.246+0000 7f442e159700 1 --2- 172.21.15.27:0/3585247469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44281089f0 0x7f4428108270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4418009fb0 tx=0x7f4418011bf0 comp rx=0 tx=0).stop 2024-04-25T00:05:29.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.246+0000 7f442e159700 1 -- 172.21.15.27:0/3585247469 shutdown_connections 2024-04-25T00:05:29.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.246+0000 7f442e159700 1 --2- 172.21.15.27:0/3585247469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44281089f0 0x7f4428108270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:29.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.246+0000 7f442e159700 1 -- 172.21.15.27:0/3585247469 >> 172.21.15.27:0/3585247469 conn(0x7f4428102360 msgr2=0x7f4428102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:29.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.246+0000 7f442e159700 1 -- 172.21.15.27:0/3585247469 shutdown_connections 2024-04-25T00:05:29.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:23.246+0000 7f442e159700 1 -- 172.21.15.27:0/3585247469 wait complete. 2024-04-25T00:05:29.268 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (2/15)... 2024-04-25T00:05:33.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:32 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4136937152' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T00:05:33.455 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T00:05:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T00:05:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "518857f8-0297-11ef-bc93-c7b262605968", 2024-04-25T00:05:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T00:05:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T00:05:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T00:05:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T00:05:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T00:05:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T00:05:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-04-25T00:05:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T00:05:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T00:05:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T00:05:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T00:05:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 17, 2024-04-25T00:05:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T00:05:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T00:05:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T00:05:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T00:05:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T00:05:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T00:05:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T00:05:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T00:05:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T00:05:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T00:05:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T00:05:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T00:05:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T00:05:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T00:05:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T00:05:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T00:05:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T00:05:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T00:05:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T00:05:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T00:05:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T00:05:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T00:05:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T00:05:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T00:05:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T00:05:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T00:05:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T00:05:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T00:05:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T00:05:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T00:05:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T00:05:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T00:05:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T00:05:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T00:05:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T00:05:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T00:05:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T00:05:10.376600+0000", 2024-04-25T00:05:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T00:05:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T00:05:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T00:05:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T00:05:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.641+0000 7f273fa84700 1 Processor -- start 2024-04-25T00:05:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.641+0000 7f273fa84700 1 -- start start 2024-04-25T00:05:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.641+0000 7f273fa84700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738107e80 0x7f27381082a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.641+0000 7f273fa84700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f27381087e0 con 0x7f2738107e80 2024-04-25T00:05:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.642+0000 7f273d820700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738107e80 0x7f27381082a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.642+0000 7f273d820700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738107e80 0x7f27381082a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34962/0 (socket says 172.21.15.27:34962) 2024-04-25T00:05:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.642+0000 7f273d820700 1 -- 172.21.15.27:0/2298269491 learned_addr learned my addr 172.21.15.27:0/2298269491 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.642+0000 7f273d820700 1 -- 172.21.15.27:0/2298269491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2738108fe0 con 0x7f2738107e80 2024-04-25T00:05:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.642+0000 7f273d820700 1 --2- 172.21.15.27:0/2298269491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738107e80 0x7f27381082a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f27280098a0 tx=0x7f2728009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e146f9baf052589f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.642+0000 7f273c81e700 1 -- 172.21.15.27:0/2298269491 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2728004160 con 0x7f2738107e80 2024-04-25T00:05:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.642+0000 7f273c81e700 1 -- 172.21.15.27:0/2298269491 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2728004300 con 0x7f2738107e80 2024-04-25T00:05:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.642+0000 7f273c81e700 1 -- 172.21.15.27:0/2298269491 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2728011b50 con 0x7f2738107e80 2024-04-25T00:05:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.643+0000 7f273fa84700 1 -- 172.21.15.27:0/2298269491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738107e80 msgr2=0x7f27381082a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.643+0000 7f273fa84700 1 --2- 172.21.15.27:0/2298269491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738107e80 0x7f27381082a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f27280098a0 tx=0x7f2728009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.643+0000 7f273fa84700 1 -- 172.21.15.27:0/2298269491 shutdown_connections 2024-04-25T00:05:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.643+0000 7f273fa84700 1 --2- 172.21.15.27:0/2298269491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738107e80 0x7f27381082a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.643+0000 7f273fa84700 1 -- 172.21.15.27:0/2298269491 >> 172.21.15.27:0/2298269491 conn(0x7f2738103440 msgr2=0x7f2738105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.643+0000 7f273fa84700 1 -- 172.21.15.27:0/2298269491 shutdown_connections 2024-04-25T00:05:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.643+0000 7f273fa84700 1 -- 172.21.15.27:0/2298269491 wait complete. 2024-04-25T00:05:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.644+0000 7f273fa84700 1 Processor -- start 2024-04-25T00:05:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.644+0000 7f273fa84700 1 -- start start 2024-04-25T00:05:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.644+0000 7f273fa84700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738107e80 0x7f273807db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.644+0000 7f273fa84700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f273810c2a0 con 0x7f2738107e80 2024-04-25T00:05:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.644+0000 7f273d820700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738107e80 0x7f273807db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.644+0000 7f273d820700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738107e80 0x7f273807db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34966/0 (socket says 172.21.15.27:34966) 2024-04-25T00:05:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.644+0000 7f273d820700 1 -- 172.21.15.27:0/4136937152 learned_addr learned my addr 172.21.15.27:0/4136937152 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.644+0000 7f273d820700 1 -- 172.21.15.27:0/4136937152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f273807e070 con 0x7f2738107e80 2024-04-25T00:05:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.645+0000 7f273d820700 1 --2- 172.21.15.27:0/4136937152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738107e80 0x7f273807db30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2728009fb0 tx=0x7f2728011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.645+0000 7f272effd700 1 -- 172.21.15.27:0/4136937152 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27280038b0 con 0x7f2738107e80 2024-04-25T00:05:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.645+0000 7f273fa84700 1 -- 172.21.15.27:0/4136937152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f273807c260 con 0x7f2738107e80 2024-04-25T00:05:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.645+0000 7f273fa84700 1 -- 172.21.15.27:0/4136937152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f273807c7a0 con 0x7f2738107e80 2024-04-25T00:05:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.645+0000 7f272effd700 1 -- 172.21.15.27:0/4136937152 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2728011e50 con 0x7f2738107e80 2024-04-25T00:05:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.645+0000 7f272effd700 1 -- 172.21.15.27:0/4136937152 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2728019420 con 0x7f2738107e80 2024-04-25T00:05:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.646+0000 7f272effd700 1 -- 172.21.15.27:0/4136937152 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f2728019750 con 0x7f2738107e80 2024-04-25T00:05:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.646+0000 7f273fa84700 1 -- 172.21.15.27:0/4136937152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2738063890 con 0x7f2738107e80 2024-04-25T00:05:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.646+0000 7f272effd700 1 --2- 172.21.15.27:0/4136937152 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f2724039000 0x7f272403b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.646+0000 7f272effd700 1 -- 172.21.15.27:0/4136937152 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f272804d4d0 con 0x7f2738107e80 2024-04-25T00:05:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.646+0000 7f273d01f700 1 --2- 172.21.15.27:0/4136937152 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f2724039000 0x7f272403b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.647+0000 7f273d01f700 1 --2- 172.21.15.27:0/4136937152 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f2724039000 0x7f272403b4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2734006fd0 tx=0x7f2734006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.652+0000 7f272effd700 1 -- 172.21.15.27:0/4136937152 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f272804cef0 con 0x7f2738107e80 2024-04-25T00:05:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.944+0000 7f273fa84700 1 -- 172.21.15.27:0/4136937152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f273807ca20 con 0x7f2738107e80 2024-04-25T00:05:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.944+0000 7f272effd700 1 -- 172.21.15.27:0/4136937152 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f2728003ac0 con 0x7f2738107e80 2024-04-25T00:05:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.946+0000 7f273fa84700 1 -- 172.21.15.27:0/4136937152 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f2724039000 msgr2=0x7f272403b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.946+0000 7f273fa84700 1 --2- 172.21.15.27:0/4136937152 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f2724039000 0x7f272403b4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2734006fd0 tx=0x7f2734006e40 comp rx=0 tx=0).stop 2024-04-25T00:05:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.946+0000 7f273fa84700 1 -- 172.21.15.27:0/4136937152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738107e80 msgr2=0x7f273807db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.946+0000 7f273fa84700 1 --2- 172.21.15.27:0/4136937152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738107e80 0x7f273807db30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2728009fb0 tx=0x7f2728011bf0 comp rx=0 tx=0).stop 2024-04-25T00:05:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.946+0000 7f273fa84700 1 -- 172.21.15.27:0/4136937152 shutdown_connections 2024-04-25T00:05:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.946+0000 7f273fa84700 1 --2- 172.21.15.27:0/4136937152 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f2724039000 0x7f272403b4c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.946+0000 7f273fa84700 1 --2- 172.21.15.27:0/4136937152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738107e80 0x7f273807db30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.946+0000 7f273fa84700 1 -- 172.21.15.27:0/4136937152 >> 172.21.15.27:0/4136937152 conn(0x7f2738103440 msgr2=0x7f2738103fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.947+0000 7f273fa84700 1 -- 172.21.15.27:0/4136937152 shutdown_connections 2024-04-25T00:05:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:32.947+0000 7f273fa84700 1 -- 172.21.15.27:0/4136937152 wait complete. 2024-04-25T00:05:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:mgr is available 2024-04-25T00:05:34.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:34 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/248607444' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T00:05:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T00:05:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T00:05:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout fsid = 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:05:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T00:05:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T00:05:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T00:05:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T00:05:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T00:05:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T00:05:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T00:05:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T00:05:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T00:05:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-04-25T00:05:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T00:05:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T00:05:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T00:05:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T00:05:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T00:05:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.897+0000 7f4d862b2700 1 Processor -- start 2024-04-25T00:05:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.897+0000 7f4d862b2700 1 -- start start 2024-04-25T00:05:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.897+0000 7f4d862b2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80106150 0x7f4d80106570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.897+0000 7f4d862b2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4d80106b40 con 0x7f4d80106150 2024-04-25T00:05:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.897+0000 7f4d7ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80106150 0x7f4d80106570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.897+0000 7f4d7ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80106150 0x7f4d80106570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34970/0 (socket says 172.21.15.27:34970) 2024-04-25T00:05:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.897+0000 7f4d7ffff700 1 -- 172.21.15.27:0/1066056498 learned_addr learned my addr 172.21.15.27:0/1066056498 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.898+0000 7f4d7ffff700 1 -- 172.21.15.27:0/1066056498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d801090e0 con 0x7f4d80106150 2024-04-25T00:05:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.898+0000 7f4d7ffff700 1 --2- 172.21.15.27:0/1066056498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80106150 0x7f4d80106570 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4d700098a0 tx=0x7f4d70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440b32a46627dbcc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.898+0000 7f4d7effd700 1 -- 172.21.15.27:0/1066056498 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d70004160 con 0x7f4d80106150 2024-04-25T00:05:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.898+0000 7f4d7effd700 1 -- 172.21.15.27:0/1066056498 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4d70004300 con 0x7f4d80106150 2024-04-25T00:05:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.898+0000 7f4d7effd700 1 -- 172.21.15.27:0/1066056498 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d70011b50 con 0x7f4d80106150 2024-04-25T00:05:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.898+0000 7f4d862b2700 1 -- 172.21.15.27:0/1066056498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80106150 msgr2=0x7f4d80106570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.898+0000 7f4d862b2700 1 --2- 172.21.15.27:0/1066056498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80106150 0x7f4d80106570 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4d700098a0 tx=0x7f4d70009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:34.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.898+0000 7f4d862b2700 1 -- 172.21.15.27:0/1066056498 shutdown_connections 2024-04-25T00:05:34.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.898+0000 7f4d862b2700 1 --2- 172.21.15.27:0/1066056498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80106150 0x7f4d80106570 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:34.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.898+0000 7f4d862b2700 1 -- 172.21.15.27:0/1066056498 >> 172.21.15.27:0/1066056498 conn(0x7f4d80101ac0 msgr2=0x7f4d80103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:34.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.899+0000 7f4d862b2700 1 -- 172.21.15.27:0/1066056498 shutdown_connections 2024-04-25T00:05:34.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.899+0000 7f4d862b2700 1 -- 172.21.15.27:0/1066056498 wait complete. 2024-04-25T00:05:34.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.899+0000 7f4d862b2700 1 Processor -- start 2024-04-25T00:05:34.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.899+0000 7f4d862b2700 1 -- start start 2024-04-25T00:05:34.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.899+0000 7f4d862b2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80106150 0x7f4d80108960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:34.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.899+0000 7f4d862b2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8010c490 con 0x7f4d80106150 2024-04-25T00:05:34.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.900+0000 7f4d7ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80106150 0x7f4d80108960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:34.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.900+0000 7f4d7ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80106150 0x7f4d80108960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34980/0 (socket says 172.21.15.27:34980) 2024-04-25T00:05:34.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.900+0000 7f4d7ffff700 1 -- 172.21.15.27:0/248607444 learned_addr learned my addr 172.21.15.27:0/248607444 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:34.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.900+0000 7f4d7ffff700 1 -- 172.21.15.27:0/248607444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d80108ea0 con 0x7f4d80106150 2024-04-25T00:05:34.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.900+0000 7f4d7ffff700 1 --2- 172.21.15.27:0/248607444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80106150 0x7f4d80108960 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4d70011670 tx=0x7f4d70011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:34.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.900+0000 7f4d7d7fa700 1 -- 172.21.15.27:0/248607444 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d700039c0 con 0x7f4d80106150 2024-04-25T00:05:34.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.900+0000 7f4d862b2700 1 -- 172.21.15.27:0/248607444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d80107090 con 0x7f4d80106150 2024-04-25T00:05:34.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.900+0000 7f4d7d7fa700 1 -- 172.21.15.27:0/248607444 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4d70011e50 con 0x7f4d80106150 2024-04-25T00:05:34.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.900+0000 7f4d7d7fa700 1 -- 172.21.15.27:0/248607444 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d7001a5c0 con 0x7f4d80106150 2024-04-25T00:05:34.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.900+0000 7f4d862b2700 1 -- 172.21.15.27:0/248607444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d801075d0 con 0x7f4d80106150 2024-04-25T00:05:34.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.901+0000 7f4d7d7fa700 1 -- 172.21.15.27:0/248607444 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f4d7001a8c0 con 0x7f4d80106150 2024-04-25T00:05:34.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.901+0000 7f4d862b2700 1 -- 172.21.15.27:0/248607444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d80063890 con 0x7f4d80106150 2024-04-25T00:05:34.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.901+0000 7f4d7d7fa700 1 --2- 172.21.15.27:0/248607444 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f4d68038fb0 0x7f4d6803b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:34.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.901+0000 7f4d7d7fa700 1 -- 172.21.15.27:0/248607444 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4d7004d510 con 0x7f4d80106150 2024-04-25T00:05:34.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.902+0000 7f4d7f7fe700 1 --2- 172.21.15.27:0/248607444 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f4d68038fb0 0x7f4d6803b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:34.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.902+0000 7f4d7f7fe700 1 --2- 172.21.15.27:0/248607444 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f4d68038fb0 0x7f4d6803b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4d74006fd0 tx=0x7f4d74006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:34.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:33.906+0000 7f4d7d7fa700 1 -- 172.21.15.27:0/248607444 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4d7004cf30 con 0x7f4d80106150 2024-04-25T00:05:34.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:34.109+0000 7f4d862b2700 1 -- 172.21.15.27:0/248607444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f4d80107850 con 0x7f4d80106150 2024-04-25T00:05:34.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:34.110+0000 7f4d7d7fa700 1 -- 172.21.15.27:0/248607444 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+398 (secure 0 0 0) 0x7f4d7001ac60 con 0x7f4d80106150 2024-04-25T00:05:34.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:34.112+0000 7f4d862b2700 1 -- 172.21.15.27:0/248607444 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f4d68038fb0 msgr2=0x7f4d6803b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:34.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:34.112+0000 7f4d862b2700 1 --2- 172.21.15.27:0/248607444 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f4d68038fb0 0x7f4d6803b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4d74006fd0 tx=0x7f4d74006e40 comp rx=0 tx=0).stop 2024-04-25T00:05:34.655 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:34.112+0000 7f4d862b2700 1 -- 172.21.15.27:0/248607444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80106150 msgr2=0x7f4d80108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:34.655 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:34.112+0000 7f4d862b2700 1 --2- 172.21.15.27:0/248607444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80106150 0x7f4d80108960 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4d70011670 tx=0x7f4d70011b60 comp rx=0 tx=0).stop 2024-04-25T00:05:34.655 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:34.112+0000 7f4d862b2700 1 -- 172.21.15.27:0/248607444 shutdown_connections 2024-04-25T00:05:34.655 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:34.112+0000 7f4d862b2700 1 --2- 172.21.15.27:0/248607444 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f4d68038fb0 0x7f4d6803b470 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:34.655 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:34.112+0000 7f4d862b2700 1 --2- 172.21.15.27:0/248607444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80106150 0x7f4d80108960 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:34.655 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:34.112+0000 7f4d862b2700 1 -- 172.21.15.27:0/248607444 >> 172.21.15.27:0/248607444 conn(0x7f4d80101ac0 msgr2=0x7f4d801025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:34.655 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:34.113+0000 7f4d862b2700 1 -- 172.21.15.27:0/248607444 shutdown_connections 2024-04-25T00:05:34.655 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:34.113+0000 7f4d862b2700 1 -- 172.21.15.27:0/248607444 wait complete. 2024-04-25T00:05:34.655 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling cephadm module... 2024-04-25T00:05:35.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:35 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3790420092' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-25T00:05:35.692 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:35 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: ignoring --setuser ceph since I am not root 2024-04-25T00:05:35.692 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:35 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: ignoring --setgroup ceph since I am not root 2024-04-25T00:05:35.692 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:35 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:35.656+0000 7f7e3e2e3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T00:05:36.063 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.154+0000 7f3d5a1c7700 1 Processor -- start 2024-04-25T00:05:36.063 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.154+0000 7f3d5a1c7700 1 -- start start 2024-04-25T00:05:36.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.155+0000 7f3d5a1c7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d54106740 0x7f3d54106b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:36.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.155+0000 7f3d5a1c7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5407be10 con 0x7f3d54106740 2024-04-25T00:05:36.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.155+0000 7f3d537fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d54106740 0x7f3d54106b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:36.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.155+0000 7f3d537fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d54106740 0x7f3d54106b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34990/0 (socket says 172.21.15.27:34990) 2024-04-25T00:05:36.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.155+0000 7f3d537fe700 1 -- 172.21.15.27:0/3391077745 learned_addr learned my addr 172.21.15.27:0/3391077745 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:36.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.155+0000 7f3d537fe700 1 -- 172.21.15.27:0/3391077745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d541070a0 con 0x7f3d54106740 2024-04-25T00:05:36.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.155+0000 7f3d537fe700 1 --2- 172.21.15.27:0/3391077745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d54106740 0x7f3d54106b60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3d440098a0 tx=0x7f3d44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8528c1e3590188bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:36.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.155+0000 7f3d527fc700 1 -- 172.21.15.27:0/3391077745 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d44004160 con 0x7f3d54106740 2024-04-25T00:05:36.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.155+0000 7f3d527fc700 1 -- 172.21.15.27:0/3391077745 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3d44004300 con 0x7f3d54106740 2024-04-25T00:05:36.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.155+0000 7f3d527fc700 1 -- 172.21.15.27:0/3391077745 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d44011b50 con 0x7f3d54106740 2024-04-25T00:05:36.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.156+0000 7f3d5a1c7700 1 -- 172.21.15.27:0/3391077745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d54106740 msgr2=0x7f3d54106b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:36.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.156+0000 7f3d5a1c7700 1 --2- 172.21.15.27:0/3391077745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d54106740 0x7f3d54106b60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3d440098a0 tx=0x7f3d44009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:36.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.156+0000 7f3d5a1c7700 1 -- 172.21.15.27:0/3391077745 shutdown_connections 2024-04-25T00:05:36.065 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.156+0000 7f3d5a1c7700 1 --2- 172.21.15.27:0/3391077745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d54106740 0x7f3d54106b60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:36.065 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.156+0000 7f3d5a1c7700 1 -- 172.21.15.27:0/3391077745 >> 172.21.15.27:0/3391077745 conn(0x7f3d54102340 msgr2=0x7f3d54104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:36.065 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.156+0000 7f3d5a1c7700 1 -- 172.21.15.27:0/3391077745 shutdown_connections 2024-04-25T00:05:36.065 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.156+0000 7f3d5a1c7700 1 -- 172.21.15.27:0/3391077745 wait complete. 2024-04-25T00:05:36.065 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.157+0000 7f3d5a1c7700 1 Processor -- start 2024-04-25T00:05:36.065 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.157+0000 7f3d5a1c7700 1 -- start start 2024-04-25T00:05:36.065 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.157+0000 7f3d5a1c7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d54106740 0x7f3d5419df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:36.065 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.157+0000 7f3d5a1c7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3d541080c0 con 0x7f3d54106740 2024-04-25T00:05:36.065 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.157+0000 7f3d537fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d54106740 0x7f3d5419df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:36.065 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.157+0000 7f3d537fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d54106740 0x7f3d5419df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35006/0 (socket says 172.21.15.27:35006) 2024-04-25T00:05:36.065 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.157+0000 7f3d537fe700 1 -- 172.21.15.27:0/3790420092 learned_addr learned my addr 172.21.15.27:0/3790420092 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:36.065 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.157+0000 7f3d537fe700 1 -- 172.21.15.27:0/3790420092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5419e4a0 con 0x7f3d54106740 2024-04-25T00:05:36.065 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.158+0000 7f3d537fe700 1 --2- 172.21.15.27:0/3790420092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d54106740 0x7f3d5419df60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3d44011b60 tx=0x7f3d44011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:36.065 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.158+0000 7f3d50ff9700 1 -- 172.21.15.27:0/3790420092 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d44003b20 con 0x7f3d54106740 2024-04-25T00:05:36.065 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.158+0000 7f3d5a1c7700 1 -- 172.21.15.27:0/3790420092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d5419e730 con 0x7f3d54106740 2024-04-25T00:05:36.066 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.158+0000 7f3d50ff9700 1 -- 172.21.15.27:0/3790420092 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3d44003cc0 con 0x7f3d54106740 2024-04-25T00:05:36.066 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.158+0000 7f3d50ff9700 1 -- 172.21.15.27:0/3790420092 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d44024840 con 0x7f3d54106740 2024-04-25T00:05:36.066 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.158+0000 7f3d5a1c7700 1 -- 172.21.15.27:0/3790420092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d541a1420 con 0x7f3d54106740 2024-04-25T00:05:36.066 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.159+0000 7f3d50ff9700 1 -- 172.21.15.27:0/3790420092 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f3d44022020 con 0x7f3d54106740 2024-04-25T00:05:36.066 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.159+0000 7f3d5a1c7700 1 -- 172.21.15.27:0/3790420092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d54063890 con 0x7f3d54106740 2024-04-25T00:05:36.066 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.159+0000 7f3d50ff9700 1 --2- 172.21.15.27:0/3790420092 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f3d3c039000 0x7f3d3c03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:36.066 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.159+0000 7f3d50ff9700 1 -- 172.21.15.27:0/3790420092 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3d44011e00 con 0x7f3d54106740 2024-04-25T00:05:36.066 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.159+0000 7f3d52ffd700 1 --2- 172.21.15.27:0/3790420092 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f3d3c039000 0x7f3d3c03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:36.066 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.159+0000 7f3d52ffd700 1 --2- 172.21.15.27:0/3790420092 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f3d3c039000 0x7f3d3c03b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3d48006fd0 tx=0x7f3d48006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:36.066 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.164+0000 7f3d50ff9700 1 -- 172.21.15.27:0/3790420092 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3d440172b0 con 0x7f3d54106740 2024-04-25T00:05:36.066 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.432+0000 7f3d5a1c7700 1 -- 172.21.15.27:0/3790420092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f3d540008d0 con 0x7f3d54106740 2024-04-25T00:05:36.066 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.483+0000 7f3d50ff9700 1 -- 172.21.15.27:0/3790420092 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f3d4400fdd0 con 0x7f3d54106740 2024-04-25T00:05:36.067 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.485+0000 7f3d50ff9700 1 -- 172.21.15.27:0/3790420092 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f3d4402d770 con 0x7f3d54106740 2024-04-25T00:05:36.067 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.486+0000 7f3d5a1c7700 1 -- 172.21.15.27:0/3790420092 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f3d3c039000 msgr2=0x7f3d3c03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:36.067 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.486+0000 7f3d5a1c7700 1 --2- 172.21.15.27:0/3790420092 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f3d3c039000 0x7f3d3c03b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3d48006fd0 tx=0x7f3d48006e40 comp rx=0 tx=0).stop 2024-04-25T00:05:36.067 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.486+0000 7f3d5a1c7700 1 -- 172.21.15.27:0/3790420092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d54106740 msgr2=0x7f3d5419df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:36.067 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.486+0000 7f3d5a1c7700 1 --2- 172.21.15.27:0/3790420092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d54106740 0x7f3d5419df60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3d44011b60 tx=0x7f3d44011c40 comp rx=0 tx=0).stop 2024-04-25T00:05:36.067 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.486+0000 7f3d5a1c7700 1 -- 172.21.15.27:0/3790420092 shutdown_connections 2024-04-25T00:05:36.067 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.486+0000 7f3d5a1c7700 1 --2- 172.21.15.27:0/3790420092 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f3d3c039000 0x7f3d3c03b4c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:36.067 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.486+0000 7f3d5a1c7700 1 --2- 172.21.15.27:0/3790420092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d54106740 0x7f3d5419df60 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:36.067 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.486+0000 7f3d5a1c7700 1 -- 172.21.15.27:0/3790420092 >> 172.21.15.27:0/3790420092 conn(0x7f3d54102340 msgr2=0x7f3d54194c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:36.067 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.487+0000 7f3d5a1c7700 1 -- 172.21.15.27:0/3790420092 shutdown_connections 2024-04-25T00:05:36.067 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:35.487+0000 7f3d5a1c7700 1 -- 172.21.15.27:0/3790420092 wait complete. 2024-04-25T00:05:36.183 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:35 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:35.849+0000 7f7e3e2e3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:05:36.183 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:35 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:35.956+0000 7f7e3e2e3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T00:05:36.442 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:36 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:36.290+0000 7f7e3e2e3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T00:05:36.442 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:36 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:36.382+0000 7f7e3e2e3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T00:05:36.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:36 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3790420092' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-25T00:05:36.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:36 smithi027 ceph-mon[29208]: mgrmap e5: x(active, since 10s) 2024-04-25T00:05:37.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T00:05:37.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-25T00:05:37.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T00:05:37.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T00:05:37.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T00:05:37.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T00:05:37.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.493+0000 7fc6cd768700 1 Processor -- start 2024-04-25T00:05:37.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.493+0000 7fc6cd768700 1 -- start start 2024-04-25T00:05:37.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.493+0000 7fc6cd768700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c8106110 0x7fc6c8106530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:37.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.493+0000 7fc6cd768700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c8106b00 con 0x7fc6c8106110 2024-04-25T00:05:37.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.493+0000 7fc6c6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c8106110 0x7fc6c8106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:37.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.493+0000 7fc6c6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c8106110 0x7fc6c8106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35034/0 (socket says 172.21.15.27:35034) 2024-04-25T00:05:37.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.493+0000 7fc6c6ffd700 1 -- 172.21.15.27:0/1650562753 learned_addr learned my addr 172.21.15.27:0/1650562753 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:37.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.494+0000 7fc6c6ffd700 1 -- 172.21.15.27:0/1650562753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c81090a0 con 0x7fc6c8106110 2024-04-25T00:05:37.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.494+0000 7fc6c6ffd700 1 --2- 172.21.15.27:0/1650562753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c8106110 0x7fc6c8106530 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc6b80098a0 tx=0x7fc6b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b82b4e6f65b2fb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:37.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.494+0000 7fc6c5ffb700 1 -- 172.21.15.27:0/1650562753 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6b8004160 con 0x7fc6c8106110 2024-04-25T00:05:37.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.494+0000 7fc6c5ffb700 1 -- 172.21.15.27:0/1650562753 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc6b8004300 con 0x7fc6c8106110 2024-04-25T00:05:37.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.494+0000 7fc6c5ffb700 1 -- 172.21.15.27:0/1650562753 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6b8011b50 con 0x7fc6c8106110 2024-04-25T00:05:37.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.495+0000 7fc6cd768700 1 -- 172.21.15.27:0/1650562753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c8106110 msgr2=0x7fc6c8106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:37.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.495+0000 7fc6cd768700 1 --2- 172.21.15.27:0/1650562753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c8106110 0x7fc6c8106530 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc6b80098a0 tx=0x7fc6b8009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:37.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.495+0000 7fc6cd768700 1 -- 172.21.15.27:0/1650562753 shutdown_connections 2024-04-25T00:05:37.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.495+0000 7fc6cd768700 1 --2- 172.21.15.27:0/1650562753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c8106110 0x7fc6c8106530 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:37.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.495+0000 7fc6cd768700 1 -- 172.21.15.27:0/1650562753 >> 172.21.15.27:0/1650562753 conn(0x7fc6c8101a60 msgr2=0x7fc6c8103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:37.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.495+0000 7fc6cd768700 1 -- 172.21.15.27:0/1650562753 shutdown_connections 2024-04-25T00:05:37.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.495+0000 7fc6cd768700 1 -- 172.21.15.27:0/1650562753 wait complete. 2024-04-25T00:05:37.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.495+0000 7fc6cd768700 1 Processor -- start 2024-04-25T00:05:37.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.495+0000 7fc6cd768700 1 -- start start 2024-04-25T00:05:37.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.496+0000 7fc6cd768700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c8106110 0x7fc6c819e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:37.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.496+0000 7fc6cd768700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c810c450 con 0x7fc6c8106110 2024-04-25T00:05:37.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.496+0000 7fc6c6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c8106110 0x7fc6c819e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:37.431 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.496+0000 7fc6c6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c8106110 0x7fc6c819e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35046/0 (socket says 172.21.15.27:35046) 2024-04-25T00:05:37.431 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.496+0000 7fc6c6ffd700 1 -- 172.21.15.27:0/3787221105 learned_addr learned my addr 172.21.15.27:0/3787221105 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:37.431 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.496+0000 7fc6c6ffd700 1 -- 172.21.15.27:0/3787221105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c819e580 con 0x7fc6c8106110 2024-04-25T00:05:37.431 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.496+0000 7fc6c6ffd700 1 --2- 172.21.15.27:0/3787221105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c8106110 0x7fc6c819e040 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8011850 tx=0x7fc6b8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:37.431 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.496+0000 7fc6affff700 1 -- 172.21.15.27:0/3787221105 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6b8003af0 con 0x7fc6c8106110 2024-04-25T00:05:37.431 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.496+0000 7fc6affff700 1 -- 172.21.15.27:0/3787221105 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc6b8020070 con 0x7fc6c8106110 2024-04-25T00:05:37.431 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.496+0000 7fc6cd768700 1 -- 172.21.15.27:0/3787221105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6c819e810 con 0x7fc6c8106110 2024-04-25T00:05:37.431 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.496+0000 7fc6affff700 1 -- 172.21.15.27:0/3787221105 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6b80195a0 con 0x7fc6c8106110 2024-04-25T00:05:37.431 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.496+0000 7fc6cd768700 1 -- 172.21.15.27:0/3787221105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6c81a1500 con 0x7fc6c8106110 2024-04-25T00:05:37.431 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.497+0000 7fc6affff700 1 -- 172.21.15.27:0/3787221105 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7fc6b80198a0 con 0x7fc6c8106110 2024-04-25T00:05:37.431 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.497+0000 7fc6affff700 1 --2- 172.21.15.27:0/3787221105 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7fc6b0039050 0x7fc6b003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:37.431 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.497+0000 7fc6cd768700 1 -- 172.21.15.27:0/3787221105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6c8063890 con 0x7fc6c8106110 2024-04-25T00:05:37.431 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.498+0000 7fc6c67fc700 1 -- 172.21.15.27:0/3787221105 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7fc6b0039050 msgr2=0x7fc6b003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/3082151239 2024-04-25T00:05:37.431 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.498+0000 7fc6affff700 1 -- 172.21.15.27:0/3787221105 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc6b8011e00 con 0x7fc6c8106110 2024-04-25T00:05:37.432 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.498+0000 7fc6c67fc700 1 --2- 172.21.15.27:0/3787221105 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7fc6b0039050 0x7fc6b003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T00:05:37.432 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.503+0000 7fc6affff700 1 -- 172.21.15.27:0/3787221105 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc6b8003c90 con 0x7fc6c8106110 2024-04-25T00:05:37.432 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.699+0000 7fc6c67fc700 1 -- 172.21.15.27:0/3787221105 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7fc6b0039050 msgr2=0x7fc6b003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/3082151239 2024-04-25T00:05:37.432 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.699+0000 7fc6c67fc700 1 --2- 172.21.15.27:0/3787221105 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7fc6b0039050 0x7fc6b003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T00:05:37.432 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.770+0000 7fc6cd768700 1 -- 172.21.15.27:0/3787221105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fc6c81a16d0 con 0x7fc6c8106110 2024-04-25T00:05:37.432 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.770+0000 7fc6affff700 1 -- 172.21.15.27:0/3787221105 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fc6c81a16d0 con 0x7fc6c8106110 2024-04-25T00:05:37.432 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.773+0000 7fc6cd768700 1 -- 172.21.15.27:0/3787221105 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7fc6b0039050 msgr2=0x7fc6b003b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:05:37.432 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.773+0000 7fc6cd768700 1 --2- 172.21.15.27:0/3787221105 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7fc6b0039050 0x7fc6b003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:37.432 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.773+0000 7fc6cd768700 1 -- 172.21.15.27:0/3787221105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c8106110 msgr2=0x7fc6c819e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:37.432 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.773+0000 7fc6cd768700 1 --2- 172.21.15.27:0/3787221105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c8106110 0x7fc6c819e040 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8011850 tx=0x7fc6b8011c10 comp rx=0 tx=0).stop 2024-04-25T00:05:37.432 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.774+0000 7fc6cd768700 1 -- 172.21.15.27:0/3787221105 shutdown_connections 2024-04-25T00:05:37.432 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.774+0000 7fc6cd768700 1 --2- 172.21.15.27:0/3787221105 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7fc6b0039050 0x7fc6b003b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:37.432 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.774+0000 7fc6cd768700 1 --2- 172.21.15.27:0/3787221105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c8106110 0x7fc6c819e040 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:37.432 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.774+0000 7fc6cd768700 1 -- 172.21.15.27:0/3787221105 >> 172.21.15.27:0/3787221105 conn(0x7fc6c8101a60 msgr2=0x7fc6c8102570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:37.433 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.775+0000 7fc6cd768700 1 -- 172.21.15.27:0/3787221105 shutdown_connections 2024-04-25T00:05:37.433 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:36.775+0000 7fc6cd768700 1 -- 172.21.15.27:0/3787221105 wait complete. 2024-04-25T00:05:37.433 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for the mgr to restart... 2024-04-25T00:05:37.433 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr epoch 5... 2024-04-25T00:05:37.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:37 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3787221105' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T00:05:37.683 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:37 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:37.648+0000 7f7e3e2e3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T00:05:37.942 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:37 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:37.830+0000 7f7e3e2e3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T00:05:38.312 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:38 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:38.011+0000 7f7e3e2e3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T00:05:38.312 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:38 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:38.311+0000 7f7e3e2e3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T00:05:38.620 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:38 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:38.408+0000 7f7e3e2e3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T00:05:38.620 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:38 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:38.619+0000 7f7e3e2e3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T00:05:38.942 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:38 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:38.820+0000 7f7e3e2e3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:05:39.693 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:39 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:39.449+0000 7f7e3e2e3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T00:05:40.192 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:39 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:39.890+0000 7f7e3e2e3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T00:05:40.192 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:39 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:39.991+0000 7f7e3e2e3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T00:05:40.692 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:40 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:40.199+0000 7f7e3e2e3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T00:05:41.093 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:40 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:40.745+0000 7f7e3e2e3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T00:05:41.093 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:40 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:40.841+0000 7f7e3e2e3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T00:05:41.442 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:41 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:41.093+0000 7f7e3e2e3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T00:05:41.442 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:41 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:41.232+0000 7f7e3e2e3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T00:05:41.442 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:41 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:41.331+0000 7f7e3e2e3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T00:05:41.942 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:41 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:41.451+0000 7f7e3e2e3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T00:05:41.942 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:41 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:41.546+0000 7f7e3e2e3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T00:05:42.668 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:42 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:42.392+0000 7f7e3e2e3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T00:05:42.668 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:42 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:05:42.615+0000 7f7e3e2e3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T00:05:42.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:42 smithi027 ceph-mon[29208]: Active manager daemon x restarted 2024-04-25T00:05:42.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:42 smithi027 ceph-mon[29208]: Activating manager daemon x 2024-04-25T00:05:42.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:42 smithi027 ceph-mon[29208]: osdmap e2: 0 total, 0 up, 0 in 2024-04-25T00:05:42.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:42 smithi027 ceph-mon[29208]: mgrmap e6: x(active, starting, since 0.00377555s) 2024-04-25T00:05:42.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:42 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:05:42.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:42 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T00:05:42.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:42 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T00:05:42.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:42 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T00:05:42.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:42 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T00:05:42.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:42 smithi027 ceph-mon[29208]: Manager daemon x is now available 2024-04-25T00:05:42.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:42 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' 2024-04-25T00:05:42.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:42 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' 2024-04-25T00:05:42.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:42 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:05:42.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:42 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:05:42.946 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:42 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T00:05:43.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:43 smithi027 ceph-mon[29208]: Found migration_current of "None". Setting to last migration. 2024-04-25T00:05:43.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:43 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T00:05:43.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:43 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' 2024-04-25T00:05:43.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:43 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' 2024-04-25T00:05:43.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:43 smithi027 ceph-mon[29208]: [25/Apr/2024:00:05:43] ENGINE Bus STARTING 2024-04-25T00:05:43.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:43 smithi027 ceph-mon[29208]: mgrmap e7: x(active, since 1.00552s) 2024-04-25T00:05:44.161 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T00:05:44.161 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-25T00:05:44.161 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T00:05:44.161 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T00:05:44.161 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.875+0000 7f22c23b3700 1 Processor -- start 2024-04-25T00:05:44.161 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.876+0000 7f22c23b3700 1 -- start start 2024-04-25T00:05:44.161 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.876+0000 7f22c23b3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22bc1066f0 0x7f22bc106b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:44.161 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.876+0000 7f22c23b3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f22bc07be50 con 0x7f22bc1066f0 2024-04-25T00:05:44.162 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.876+0000 7f22bbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22bc1066f0 0x7f22bc106b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:44.162 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.876+0000 7f22bbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22bc1066f0 0x7f22bc106b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50362/0 (socket says 172.21.15.27:50362) 2024-04-25T00:05:44.162 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.876+0000 7f22bbfff700 1 -- 172.21.15.27:0/2469395484 learned_addr learned my addr 172.21.15.27:0/2469395484 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:44.162 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.876+0000 7f22bbfff700 1 -- 172.21.15.27:0/2469395484 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22bc107050 con 0x7f22bc1066f0 2024-04-25T00:05:44.162 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.876+0000 7f22bbfff700 1 --2- 172.21.15.27:0/2469395484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22bc1066f0 0x7f22bc106b10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f22ac0098a0 tx=0x7f22ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a32b6f306476686f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:44.162 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.877+0000 7f22baffd700 1 -- 172.21.15.27:0/2469395484 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22ac004160 con 0x7f22bc1066f0 2024-04-25T00:05:44.162 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.877+0000 7f22baffd700 1 -- 172.21.15.27:0/2469395484 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f22ac004300 con 0x7f22bc1066f0 2024-04-25T00:05:44.162 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.877+0000 7f22baffd700 1 -- 172.21.15.27:0/2469395484 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22ac011b50 con 0x7f22bc1066f0 2024-04-25T00:05:44.162 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.877+0000 7f22c23b3700 1 -- 172.21.15.27:0/2469395484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22bc1066f0 msgr2=0x7f22bc106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:44.162 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.877+0000 7f22c23b3700 1 --2- 172.21.15.27:0/2469395484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22bc1066f0 0x7f22bc106b10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f22ac0098a0 tx=0x7f22ac009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:44.162 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.877+0000 7f22c23b3700 1 -- 172.21.15.27:0/2469395484 shutdown_connections 2024-04-25T00:05:44.162 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.877+0000 7f22c23b3700 1 --2- 172.21.15.27:0/2469395484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22bc1066f0 0x7f22bc106b10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:44.162 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.877+0000 7f22c23b3700 1 -- 172.21.15.27:0/2469395484 >> 172.21.15.27:0/2469395484 conn(0x7f22bc102330 msgr2=0x7f22bc104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:44.162 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.878+0000 7f22c23b3700 1 -- 172.21.15.27:0/2469395484 shutdown_connections 2024-04-25T00:05:44.163 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.878+0000 7f22c23b3700 1 -- 172.21.15.27:0/2469395484 wait complete. 2024-04-25T00:05:44.163 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.878+0000 7f22c23b3700 1 Processor -- start 2024-04-25T00:05:44.163 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.878+0000 7f22c23b3700 1 -- start start 2024-04-25T00:05:44.163 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.879+0000 7f22c23b3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22bc1066f0 0x7f22bc19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:44.163 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.879+0000 7f22c23b3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f22bc108070 con 0x7f22bc1066f0 2024-04-25T00:05:44.163 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.879+0000 7f22bbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22bc1066f0 0x7f22bc19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:44.163 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.879+0000 7f22bbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22bc1066f0 0x7f22bc19dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50378/0 (socket says 172.21.15.27:50378) 2024-04-25T00:05:44.163 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.879+0000 7f22bbfff700 1 -- 172.21.15.27:0/3873135753 learned_addr learned my addr 172.21.15.27:0/3873135753 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:44.163 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.879+0000 7f22bbfff700 1 -- 172.21.15.27:0/3873135753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22bc19e520 con 0x7f22bc1066f0 2024-04-25T00:05:44.163 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.879+0000 7f22bbfff700 1 --2- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22bc1066f0 0x7f22bc19dfe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f22ac009fb0 tx=0x7f22ac011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:44.163 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.880+0000 7f22b97fa700 1 -- 172.21.15.27:0/3873135753 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22ac0038b0 con 0x7f22bc1066f0 2024-04-25T00:05:44.163 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.880+0000 7f22c23b3700 1 -- 172.21.15.27:0/3873135753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22bc19e7b0 con 0x7f22bc1066f0 2024-04-25T00:05:44.163 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.880+0000 7f22c23b3700 1 -- 172.21.15.27:0/3873135753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22bc1a14a0 con 0x7f22bc1066f0 2024-04-25T00:05:44.163 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.880+0000 7f22b97fa700 1 -- 172.21.15.27:0/3873135753 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f22ac011e50 con 0x7f22bc1066f0 2024-04-25T00:05:44.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.880+0000 7f22b97fa700 1 -- 172.21.15.27:0/3873135753 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22ac019550 con 0x7f22bc1066f0 2024-04-25T00:05:44.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.880+0000 7f22b97fa700 1 -- 172.21.15.27:0/3873135753 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f22ac019780 con 0x7f22bc1066f0 2024-04-25T00:05:44.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.881+0000 7f22b97fa700 1 --2- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f22a4039000 0x7f22a403b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:44.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.881+0000 7f22c23b3700 1 -- 172.21.15.27:0/3873135753 --> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f22a8000d40 con 0x7f22a4039000 2024-04-25T00:05:44.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.881+0000 7f22b97fa700 1 -- 172.21.15.27:0/3873135753 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f22ac04e2c0 con 0x7f22bc1066f0 2024-04-25T00:05:44.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.881+0000 7f22bb7fe700 1 -- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f22a4039000 msgr2=0x7f22a403b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/3082151239 2024-04-25T00:05:44.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:37.881+0000 7f22bb7fe700 1 --2- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f22a4039000 0x7f22a403b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T00:05:44.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:38.082+0000 7f22bb7fe700 1 -- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f22a4039000 msgr2=0x7f22a403b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/3082151239 2024-04-25T00:05:44.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:38.082+0000 7f22bb7fe700 1 --2- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f22a4039000 0x7f22a403b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T00:05:44.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:38.483+0000 7f22bb7fe700 1 -- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f22a4039000 msgr2=0x7f22a403b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/3082151239 2024-04-25T00:05:44.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:38.483+0000 7f22bb7fe700 1 --2- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f22a4039000 0x7f22a403b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T00:05:44.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:39.284+0000 7f22bb7fe700 1 -- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f22a4039000 msgr2=0x7f22a403b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/3082151239 2024-04-25T00:05:44.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:39.284+0000 7f22bb7fe700 1 --2- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f22a4039000 0x7f22a403b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T00:05:44.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:40.886+0000 7f22bb7fe700 1 -- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f22a4039000 msgr2=0x7f22a403b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/3082151239 2024-04-25T00:05:44.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:40.886+0000 7f22bb7fe700 1 --2- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f22a4039000 0x7f22a403b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T00:05:44.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:42.619+0000 7f22b97fa700 1 -- 172.21.15.27:0/3873135753 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7f22ac02d960 con 0x7f22bc1066f0 2024-04-25T00:05:44.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:42.619+0000 7f22b97fa700 1 -- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f22a4039000 msgr2=0x7f22a403b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:05:44.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:42.619+0000 7f22b97fa700 1 --2- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f22a4039000 0x7f22a403b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:44.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.620+0000 7f22b97fa700 1 -- 172.21.15.27:0/3873135753 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f22ac04f120 con 0x7f22bc1066f0 2024-04-25T00:05:44.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.620+0000 7f22b97fa700 1 --2- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f22a403c740 0x7f22a403eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:44.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.620+0000 7f22b97fa700 1 -- 172.21.15.27:0/3873135753 --> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f22ac027e80 con 0x7f22a403c740 2024-04-25T00:05:44.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.620+0000 7f22bb7fe700 1 --2- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f22a403c740 0x7f22a403eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:44.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.621+0000 7f22bb7fe700 1 --2- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f22a403c740 0x7f22a403eb30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f22b0003a10 tx=0x7f22b0009290 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:44.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.622+0000 7f22b97fa700 1 -- 172.21.15.27:0/3873135753 <== mgr.14118 v2:172.21.15.27:6800/1186428290 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f22ac027e80 con 0x7f22a403c740 2024-04-25T00:05:44.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.628+0000 7f22c23b3700 1 -- 172.21.15.27:0/3873135753 --> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f22a8002840 con 0x7f22a403c740 2024-04-25T00:05:44.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.629+0000 7f22b97fa700 1 -- 172.21.15.27:0/3873135753 <== mgr.14118 v2:172.21.15.27:6800/1186428290 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f22a8002840 con 0x7f22a403c740 2024-04-25T00:05:44.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.629+0000 7f22c23b3700 1 -- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f22a403c740 msgr2=0x7f22a403eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:44.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.629+0000 7f22c23b3700 1 --2- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f22a403c740 0x7f22a403eb30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f22b0003a10 tx=0x7f22b0009290 comp rx=0 tx=0).stop 2024-04-25T00:05:44.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.629+0000 7f22c23b3700 1 -- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22bc1066f0 msgr2=0x7f22bc19dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:44.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.629+0000 7f22c23b3700 1 --2- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22bc1066f0 0x7f22bc19dfe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f22ac009fb0 tx=0x7f22ac011bf0 comp rx=0 tx=0).stop 2024-04-25T00:05:44.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.629+0000 7f22c23b3700 1 -- 172.21.15.27:0/3873135753 shutdown_connections 2024-04-25T00:05:44.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.629+0000 7f22c23b3700 1 --2- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f22a403c740 0x7f22a403eb30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:44.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.629+0000 7f22c23b3700 1 --2- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22bc1066f0 0x7f22bc19dfe0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:44.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.629+0000 7f22c23b3700 1 --2- 172.21.15.27:0/3873135753 >> [v2:172.21.15.27:6800/3082151239,v1:172.21.15.27:6801/3082151239] conn(0x7f22a4039000 0x7f22a403b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:44.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.629+0000 7f22c23b3700 1 -- 172.21.15.27:0/3873135753 >> 172.21.15.27:0/3873135753 conn(0x7f22bc102330 msgr2=0x7f22bc194c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:44.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.629+0000 7f22c23b3700 1 -- 172.21.15.27:0/3873135753 shutdown_connections 2024-04-25T00:05:44.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:43.629+0000 7f22c23b3700 1 -- 172.21.15.27:0/3873135753 wait complete. 2024-04-25T00:05:44.166 INFO:teuthology.orchestra.run.smithi027.stdout:mgr epoch 5 is available 2024-04-25T00:05:44.166 INFO:teuthology.orchestra.run.smithi027.stdout:Setting orchestrator backend to cephadm... 2024-04-25T00:05:44.842 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:44 smithi027 ceph-mon[29208]: [25/Apr/2024:00:05:43] ENGINE Serving on https://172.21.15.27:7150 2024-04-25T00:05:44.842 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:44 smithi027 ceph-mon[29208]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T00:05:44.842 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:44 smithi027 ceph-mon[29208]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T00:05:44.842 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:44 smithi027 ceph-mon[29208]: [25/Apr/2024:00:05:43] ENGINE Serving on http://172.21.15.27:8765 2024-04-25T00:05:44.842 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:44 smithi027 ceph-mon[29208]: [25/Apr/2024:00:05:43] ENGINE Bus STARTED 2024-04-25T00:05:44.842 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:44 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:05:45.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.583+0000 7efc20b1b700 1 Processor -- start 2024-04-25T00:05:45.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.583+0000 7efc20b1b700 1 -- start start 2024-04-25T00:05:45.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.583+0000 7efc20b1b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc1c107ed0 0x7efc1c1082f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:45.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.583+0000 7efc20b1b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efc1c108830 con 0x7efc1c107ed0 2024-04-25T00:05:45.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.583+0000 7efc1a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc1c107ed0 0x7efc1c1082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:45.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.583+0000 7efc1a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc1c107ed0 0x7efc1c1082f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50444/0 (socket says 172.21.15.27:50444) 2024-04-25T00:05:45.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.583+0000 7efc1a59c700 1 -- 172.21.15.27:0/1764360376 learned_addr learned my addr 172.21.15.27:0/1764360376 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:45.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.584+0000 7efc1a59c700 1 -- 172.21.15.27:0/1764360376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc1c108fe0 con 0x7efc1c107ed0 2024-04-25T00:05:45.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.584+0000 7efc1a59c700 1 --2- 172.21.15.27:0/1764360376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc1c107ed0 0x7efc1c1082f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efc0c0098a0 tx=0x7efc0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e77e457e8bd09a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:45.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.584+0000 7efc1959a700 1 -- 172.21.15.27:0/1764360376 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc0c004160 con 0x7efc1c107ed0 2024-04-25T00:05:45.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.584+0000 7efc1959a700 1 -- 172.21.15.27:0/1764360376 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efc0c004300 con 0x7efc1c107ed0 2024-04-25T00:05:45.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.584+0000 7efc1959a700 1 -- 172.21.15.27:0/1764360376 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc0c011b50 con 0x7efc1c107ed0 2024-04-25T00:05:45.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.585+0000 7efc20b1b700 1 -- 172.21.15.27:0/1764360376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc1c107ed0 msgr2=0x7efc1c1082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:45.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.585+0000 7efc20b1b700 1 --2- 172.21.15.27:0/1764360376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc1c107ed0 0x7efc1c1082f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efc0c0098a0 tx=0x7efc0c009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:45.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.585+0000 7efc20b1b700 1 -- 172.21.15.27:0/1764360376 shutdown_connections 2024-04-25T00:05:45.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.585+0000 7efc20b1b700 1 --2- 172.21.15.27:0/1764360376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc1c107ed0 0x7efc1c1082f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:45.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.585+0000 7efc20b1b700 1 -- 172.21.15.27:0/1764360376 >> 172.21.15.27:0/1764360376 conn(0x7efc1c103490 msgr2=0x7efc1c1058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:45.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.585+0000 7efc20b1b700 1 -- 172.21.15.27:0/1764360376 shutdown_connections 2024-04-25T00:05:45.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.585+0000 7efc20b1b700 1 -- 172.21.15.27:0/1764360376 wait complete. 2024-04-25T00:05:45.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.586+0000 7efc20b1b700 1 Processor -- start 2024-04-25T00:05:45.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.586+0000 7efc20b1b700 1 -- start start 2024-04-25T00:05:45.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.586+0000 7efc20b1b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc1c107ed0 0x7efc1c19bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:45.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.586+0000 7efc20b1b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efc1c10c2a0 con 0x7efc1c107ed0 2024-04-25T00:05:45.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.586+0000 7efc1a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc1c107ed0 0x7efc1c19bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:45.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.586+0000 7efc1a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc1c107ed0 0x7efc1c19bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50458/0 (socket says 172.21.15.27:50458) 2024-04-25T00:05:45.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.586+0000 7efc1a59c700 1 -- 172.21.15.27:0/1283343451 learned_addr learned my addr 172.21.15.27:0/1283343451 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:45.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.586+0000 7efc1a59c700 1 -- 172.21.15.27:0/1283343451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc1c19c250 con 0x7efc1c107ed0 2024-04-25T00:05:45.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.586+0000 7efc1a59c700 1 --2- 172.21.15.27:0/1283343451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc1c107ed0 0x7efc1c19bd10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efc0c011670 tx=0x7efc0c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:45.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.586+0000 7efc0b7fe700 1 -- 172.21.15.27:0/1283343451 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc0c0039c0 con 0x7efc1c107ed0 2024-04-25T00:05:45.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.586+0000 7efc0b7fe700 1 -- 172.21.15.27:0/1283343451 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efc0c011e50 con 0x7efc1c107ed0 2024-04-25T00:05:45.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.587+0000 7efc20b1b700 1 -- 172.21.15.27:0/1283343451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc1c19c4e0 con 0x7efc1c107ed0 2024-04-25T00:05:45.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.587+0000 7efc0b7fe700 1 -- 172.21.15.27:0/1283343451 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc0c01a5c0 con 0x7efc1c107ed0 2024-04-25T00:05:45.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.587+0000 7efc20b1b700 1 -- 172.21.15.27:0/1283343451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc1c19f1c0 con 0x7efc1c107ed0 2024-04-25T00:05:45.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.587+0000 7efc0b7fe700 1 -- 172.21.15.27:0/1283343451 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7efc0c01a870 con 0x7efc1c107ed0 2024-04-25T00:05:45.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.587+0000 7efc20b1b700 1 -- 172.21.15.27:0/1283343451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc1c063890 con 0x7efc1c107ed0 2024-04-25T00:05:45.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.588+0000 7efc0b7fe700 1 --2- 172.21.15.27:0/1283343451 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7efc04038f40 0x7efc0403b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:45.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.588+0000 7efc0b7fe700 1 -- 172.21.15.27:0/1283343451 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7efc0c04d260 con 0x7efc1c107ed0 2024-04-25T00:05:45.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.588+0000 7efc19d9b700 1 --2- 172.21.15.27:0/1283343451 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7efc04038f40 0x7efc0403b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:45.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.588+0000 7efc19d9b700 1 --2- 172.21.15.27:0/1283343451 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7efc04038f40 0x7efc0403b400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efc10006fd0 tx=0x7efc10006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:45.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.594+0000 7efc0b7fe700 1 -- 172.21.15.27:0/1283343451 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efc0c00f2e0 con 0x7efc1c107ed0 2024-04-25T00:05:45.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.674+0000 7efc0b7fe700 1 -- 172.21.15.27:0/1283343451 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7efc0c024430 con 0x7efc1c107ed0 2024-04-25T00:05:45.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.822+0000 7efc20b1b700 1 -- 172.21.15.27:0/1283343451 --> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7efc1c104610 con 0x7efc04038f40 2024-04-25T00:05:45.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.828+0000 7efc0b7fe700 1 -- 172.21.15.27:0/1283343451 <== mgr.14118 v2:172.21.15.27:6800/1186428290 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7efc1c104610 con 0x7efc04038f40 2024-04-25T00:05:45.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.830+0000 7efc20b1b700 1 -- 172.21.15.27:0/1283343451 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7efc04038f40 msgr2=0x7efc0403b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:45.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.830+0000 7efc20b1b700 1 --2- 172.21.15.27:0/1283343451 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7efc04038f40 0x7efc0403b400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efc10006fd0 tx=0x7efc10006e40 comp rx=0 tx=0).stop 2024-04-25T00:05:45.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.830+0000 7efc20b1b700 1 -- 172.21.15.27:0/1283343451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc1c107ed0 msgr2=0x7efc1c19bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:45.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.830+0000 7efc20b1b700 1 --2- 172.21.15.27:0/1283343451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc1c107ed0 0x7efc1c19bd10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efc0c011670 tx=0x7efc0c011b60 comp rx=0 tx=0).stop 2024-04-25T00:05:45.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.830+0000 7efc20b1b700 1 -- 172.21.15.27:0/1283343451 shutdown_connections 2024-04-25T00:05:45.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.830+0000 7efc20b1b700 1 --2- 172.21.15.27:0/1283343451 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7efc04038f40 0x7efc0403b400 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:45.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.830+0000 7efc20b1b700 1 --2- 172.21.15.27:0/1283343451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc1c107ed0 0x7efc1c19bd10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:45.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.830+0000 7efc20b1b700 1 -- 172.21.15.27:0/1283343451 >> 172.21.15.27:0/1283343451 conn(0x7efc1c103490 msgr2=0x7efc1c103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:45.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.830+0000 7efc20b1b700 1 -- 172.21.15.27:0/1283343451 shutdown_connections 2024-04-25T00:05:45.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:44.830+0000 7efc20b1b700 1 -- 172.21.15.27:0/1283343451 wait complete. 2024-04-25T00:05:45.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:45 smithi027 ceph-mon[29208]: mgrmap e8: x(active, since 2s) 2024-04-25T00:05:45.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:45 smithi027 ceph-mon[29208]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:05:45.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:45 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' 2024-04-25T00:05:45.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:45 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:05:46.640 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-25T00:05:46.640 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.869+0000 7fcfdd3c8700 1 Processor -- start 2024-04-25T00:05:46.640 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.869+0000 7fcfdd3c8700 1 -- start start 2024-04-25T00:05:46.640 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.869+0000 7fcfdd3c8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd8107e50 0x7fcfd8108270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:46.640 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.869+0000 7fcfdd3c8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81087b0 con 0x7fcfd8107e50 2024-04-25T00:05:46.640 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.869+0000 7fcfd6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd8107e50 0x7fcfd8108270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:46.641 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.869+0000 7fcfd6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd8107e50 0x7fcfd8108270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50472/0 (socket says 172.21.15.27:50472) 2024-04-25T00:05:46.641 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.869+0000 7fcfd6ffd700 1 -- 172.21.15.27:0/3917257140 learned_addr learned my addr 172.21.15.27:0/3917257140 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:46.641 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.869+0000 7fcfd6ffd700 1 -- 172.21.15.27:0/3917257140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd8108fb0 con 0x7fcfd8107e50 2024-04-25T00:05:46.641 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.870+0000 7fcfd6ffd700 1 --2- 172.21.15.27:0/3917257140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd8107e50 0x7fcfd8108270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80098a0 tx=0x7fcfc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b02a64e09e056ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:46.641 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.870+0000 7fcfd5ffb700 1 -- 172.21.15.27:0/3917257140 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcfc8004160 con 0x7fcfd8107e50 2024-04-25T00:05:46.641 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.870+0000 7fcfd5ffb700 1 -- 172.21.15.27:0/3917257140 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcfc8004300 con 0x7fcfd8107e50 2024-04-25T00:05:46.641 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.870+0000 7fcfd5ffb700 1 -- 172.21.15.27:0/3917257140 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcfc8011b50 con 0x7fcfd8107e50 2024-04-25T00:05:46.641 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.870+0000 7fcfdd3c8700 1 -- 172.21.15.27:0/3917257140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd8107e50 msgr2=0x7fcfd8108270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:46.641 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.870+0000 7fcfdd3c8700 1 --2- 172.21.15.27:0/3917257140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd8107e50 0x7fcfd8108270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80098a0 tx=0x7fcfc8009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:46.641 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.870+0000 7fcfdd3c8700 1 -- 172.21.15.27:0/3917257140 shutdown_connections 2024-04-25T00:05:46.642 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.870+0000 7fcfdd3c8700 1 --2- 172.21.15.27:0/3917257140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd8107e50 0x7fcfd8108270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:46.642 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.870+0000 7fcfdd3c8700 1 -- 172.21.15.27:0/3917257140 >> 172.21.15.27:0/3917257140 conn(0x7fcfd81033f0 msgr2=0x7fcfd8105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:46.642 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.870+0000 7fcfdd3c8700 1 -- 172.21.15.27:0/3917257140 shutdown_connections 2024-04-25T00:05:46.642 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.870+0000 7fcfdd3c8700 1 -- 172.21.15.27:0/3917257140 wait complete. 2024-04-25T00:05:46.642 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.871+0000 7fcfdd3c8700 1 Processor -- start 2024-04-25T00:05:46.642 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.871+0000 7fcfdd3c8700 1 -- start start 2024-04-25T00:05:46.642 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.871+0000 7fcfdd3c8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd8107e50 0x7fcfd819bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:46.642 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.871+0000 7fcfdd3c8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd810c270 con 0x7fcfd8107e50 2024-04-25T00:05:46.642 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.871+0000 7fcfd6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd8107e50 0x7fcfd819bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:46.642 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.871+0000 7fcfd6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd8107e50 0x7fcfd819bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50482/0 (socket says 172.21.15.27:50482) 2024-04-25T00:05:46.642 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.872+0000 7fcfd6ffd700 1 -- 172.21.15.27:0/2316782325 learned_addr learned my addr 172.21.15.27:0/2316782325 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:46.642 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.872+0000 7fcfd6ffd700 1 -- 172.21.15.27:0/2316782325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd819c330 con 0x7fcfd8107e50 2024-04-25T00:05:46.642 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.872+0000 7fcfd6ffd700 1 --2- 172.21.15.27:0/2316782325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd8107e50 0x7fcfd819bdf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009fb0 tx=0x7fcfc8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:46.642 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.872+0000 7fcfbffff700 1 -- 172.21.15.27:0/2316782325 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcfc80038b0 con 0x7fcfd8107e50 2024-04-25T00:05:46.643 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.872+0000 7fcfbffff700 1 -- 172.21.15.27:0/2316782325 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcfc8011e50 con 0x7fcfd8107e50 2024-04-25T00:05:46.643 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.872+0000 7fcfdd3c8700 1 -- 172.21.15.27:0/2316782325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd819c5c0 con 0x7fcfd8107e50 2024-04-25T00:05:46.643 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.872+0000 7fcfbffff700 1 -- 172.21.15.27:0/2316782325 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcfc8019420 con 0x7fcfd8107e50 2024-04-25T00:05:46.643 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.872+0000 7fcfdd3c8700 1 -- 172.21.15.27:0/2316782325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd819f2b0 con 0x7fcfd8107e50 2024-04-25T00:05:46.643 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.873+0000 7fcfbffff700 1 -- 172.21.15.27:0/2316782325 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fcfc8019720 con 0x7fcfd8107e50 2024-04-25T00:05:46.643 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.873+0000 7fcfbffff700 1 --2- 172.21.15.27:0/2316782325 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fcfc0039000 0x7fcfc003b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:46.643 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.873+0000 7fcfbffff700 1 -- 172.21.15.27:0/2316782325 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcfc804d4a0 con 0x7fcfd8107e50 2024-04-25T00:05:46.643 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.873+0000 7fcfd67fc700 1 --2- 172.21.15.27:0/2316782325 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fcfc0039000 0x7fcfc003b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:46.643 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.873+0000 7fcfdd3c8700 1 -- 172.21.15.27:0/2316782325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd810c3f0 con 0x7fcfd8107e50 2024-04-25T00:05:46.643 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.874+0000 7fcfd67fc700 1 --2- 172.21.15.27:0/2316782325 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fcfc0039000 0x7fcfc003b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc006fd0 tx=0x7fcfcc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:46.643 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:45.879+0000 7fcfbffff700 1 -- 172.21.15.27:0/2316782325 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcfc8023c90 con 0x7fcfd8107e50 2024-04-25T00:05:46.643 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:46.087+0000 7fcfdd3c8700 1 -- 172.21.15.27:0/2316782325 --> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fcfd81046b0 con 0x7fcfc0039000 2024-04-25T00:05:46.643 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:46.087+0000 7fcfbffff700 1 -- 172.21.15.27:0/2316782325 <== mgr.14118 v2:172.21.15.27:6800/1186428290 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fcfd81046b0 con 0x7fcfc0039000 2024-04-25T00:05:46.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:46.089+0000 7fcfdd3c8700 1 -- 172.21.15.27:0/2316782325 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fcfc0039000 msgr2=0x7fcfc003b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:46.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:46.089+0000 7fcfdd3c8700 1 --2- 172.21.15.27:0/2316782325 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fcfc0039000 0x7fcfc003b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc006fd0 tx=0x7fcfcc006e40 comp rx=0 tx=0).stop 2024-04-25T00:05:46.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:46.089+0000 7fcfdd3c8700 1 -- 172.21.15.27:0/2316782325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd8107e50 msgr2=0x7fcfd819bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:46.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:46.089+0000 7fcfdd3c8700 1 --2- 172.21.15.27:0/2316782325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd8107e50 0x7fcfd819bdf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009fb0 tx=0x7fcfc8011bf0 comp rx=0 tx=0).stop 2024-04-25T00:05:46.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:46.090+0000 7fcfdd3c8700 1 -- 172.21.15.27:0/2316782325 shutdown_connections 2024-04-25T00:05:46.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:46.090+0000 7fcfdd3c8700 1 --2- 172.21.15.27:0/2316782325 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fcfc0039000 0x7fcfc003b4c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:46.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:46.090+0000 7fcfdd3c8700 1 --2- 172.21.15.27:0/2316782325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd8107e50 0x7fcfd819bdf0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:46.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:46.090+0000 7fcfdd3c8700 1 -- 172.21.15.27:0/2316782325 >> 172.21.15.27:0/2316782325 conn(0x7fcfd81033f0 msgr2=0x7fcfd8103fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:46.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:46.090+0000 7fcfdd3c8700 1 -- 172.21.15.27:0/2316782325 shutdown_connections 2024-04-25T00:05:46.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:46.090+0000 7fcfdd3c8700 1 -- 172.21.15.27:0/2316782325 wait complete. 2024-04-25T00:05:46.644 INFO:teuthology.orchestra.run.smithi027.stdout:Generating ssh key... 2024-04-25T00:05:46.933 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:46 smithi027 ceph-mon[29208]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:05:47.850 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: Generating public/private rsa key pair. 2024-04-25T00:05:47.850 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: Your identification has been saved in /tmp/tmpj3zw2yrk/key. 2024-04-25T00:05:47.850 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: Your public key has been saved in /tmp/tmpj3zw2yrk/key.pub. 2024-04-25T00:05:47.850 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: The key fingerprint is: 2024-04-25T00:05:47.851 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: SHA256:iROVVMBxFIpsxvu/6IpcCfg7hkHwJXiUB1GkjtmCD3E ceph-518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:05:47.851 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: The key's randomart image is: 2024-04-25T00:05:47.851 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: +---[RSA 3072]----+ 2024-04-25T00:05:47.851 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: | o+*o o=*=. | 2024-04-25T00:05:47.851 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: |o +.oo ooo | 2024-04-25T00:05:47.851 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: |.+E+ B . | 2024-04-25T00:05:47.851 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: |.Bo. o + . | 2024-04-25T00:05:47.851 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: |*.+ . + S | 2024-04-25T00:05:47.851 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: | +.. . + | 2024-04-25T00:05:47.851 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: | .o. o . | 2024-04-25T00:05:47.851 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: | ..o+ o | 2024-04-25T00:05:47.851 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: | .+..oo o. | 2024-04-25T00:05:47.851 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:05:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: +----[SHA256]-----+ 2024-04-25T00:05:48.106 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.036+0000 7fb92668a700 1 Processor -- start 2024-04-25T00:05:48.106 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.036+0000 7fb92668a700 1 -- start start 2024-04-25T00:05:48.106 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.036+0000 7fb92668a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb920107ed0 0x7fb9201082f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:48.106 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.036+0000 7fb92668a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb920108830 con 0x7fb920107ed0 2024-04-25T00:05:48.106 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.036+0000 7fb91ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb920107ed0 0x7fb9201082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:48.106 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.036+0000 7fb91ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb920107ed0 0x7fb9201082f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46006/0 (socket says 172.21.15.27:46006) 2024-04-25T00:05:48.107 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.036+0000 7fb91ffff700 1 -- 172.21.15.27:0/58796539 learned_addr learned my addr 172.21.15.27:0/58796539 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:48.107 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.037+0000 7fb91ffff700 1 -- 172.21.15.27:0/58796539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb920108fe0 con 0x7fb920107ed0 2024-04-25T00:05:48.107 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.037+0000 7fb91ffff700 1 --2- 172.21.15.27:0/58796539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb920107ed0 0x7fb9201082f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb9100098a0 tx=0x7fb910009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b35e0b66f92c301 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:48.107 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.037+0000 7fb91effd700 1 -- 172.21.15.27:0/58796539 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb910004160 con 0x7fb920107ed0 2024-04-25T00:05:48.107 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.037+0000 7fb91effd700 1 -- 172.21.15.27:0/58796539 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb910004300 con 0x7fb920107ed0 2024-04-25T00:05:48.107 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.037+0000 7fb91effd700 1 -- 172.21.15.27:0/58796539 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb910011b50 con 0x7fb920107ed0 2024-04-25T00:05:48.107 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.037+0000 7fb92668a700 1 -- 172.21.15.27:0/58796539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb920107ed0 msgr2=0x7fb9201082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:48.107 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.037+0000 7fb92668a700 1 --2- 172.21.15.27:0/58796539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb920107ed0 0x7fb9201082f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb9100098a0 tx=0x7fb910009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:48.107 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.037+0000 7fb92668a700 1 -- 172.21.15.27:0/58796539 shutdown_connections 2024-04-25T00:05:48.107 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.037+0000 7fb92668a700 1 --2- 172.21.15.27:0/58796539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb920107ed0 0x7fb9201082f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:48.107 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.037+0000 7fb92668a700 1 -- 172.21.15.27:0/58796539 >> 172.21.15.27:0/58796539 conn(0x7fb920103490 msgr2=0x7fb9201058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:48.107 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.038+0000 7fb92668a700 1 -- 172.21.15.27:0/58796539 shutdown_connections 2024-04-25T00:05:48.108 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.038+0000 7fb92668a700 1 -- 172.21.15.27:0/58796539 wait complete. 2024-04-25T00:05:48.108 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.038+0000 7fb92668a700 1 Processor -- start 2024-04-25T00:05:48.108 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.038+0000 7fb92668a700 1 -- start start 2024-04-25T00:05:48.108 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.039+0000 7fb92668a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb920107ed0 0x7fb92019bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:48.108 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.039+0000 7fb92668a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb92010c2a0 con 0x7fb920107ed0 2024-04-25T00:05:48.108 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.039+0000 7fb91ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb920107ed0 0x7fb92019bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:48.108 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.039+0000 7fb91ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb920107ed0 0x7fb92019bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46014/0 (socket says 172.21.15.27:46014) 2024-04-25T00:05:48.108 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.039+0000 7fb91ffff700 1 -- 172.21.15.27:0/609535736 learned_addr learned my addr 172.21.15.27:0/609535736 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:48.108 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.039+0000 7fb91ffff700 1 -- 172.21.15.27:0/609535736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92019c310 con 0x7fb920107ed0 2024-04-25T00:05:48.108 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.039+0000 7fb91ffff700 1 --2- 172.21.15.27:0/609535736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb920107ed0 0x7fb92019bdd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb910011b60 tx=0x7fb910011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:48.108 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.039+0000 7fb91d7fa700 1 -- 172.21.15.27:0/609535736 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb910003970 con 0x7fb920107ed0 2024-04-25T00:05:48.108 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.039+0000 7fb91d7fa700 1 -- 172.21.15.27:0/609535736 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb910003b10 con 0x7fb920107ed0 2024-04-25T00:05:48.108 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.039+0000 7fb91d7fa700 1 -- 172.21.15.27:0/609535736 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb910024610 con 0x7fb920107ed0 2024-04-25T00:05:48.109 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.039+0000 7fb92668a700 1 -- 172.21.15.27:0/609535736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb92019c5a0 con 0x7fb920107ed0 2024-04-25T00:05:48.109 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.039+0000 7fb92668a700 1 -- 172.21.15.27:0/609535736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb92019f290 con 0x7fb920107ed0 2024-04-25T00:05:48.109 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.040+0000 7fb91d7fa700 1 -- 172.21.15.27:0/609535736 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fb91001a9f0 con 0x7fb920107ed0 2024-04-25T00:05:48.109 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.041+0000 7fb92668a700 1 -- 172.21.15.27:0/609535736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb920063890 con 0x7fb920107ed0 2024-04-25T00:05:48.109 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.041+0000 7fb91d7fa700 1 --2- 172.21.15.27:0/609535736 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fb908039050 0x7fb90803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:48.109 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.041+0000 7fb91d7fa700 1 -- 172.21.15.27:0/609535736 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb910011e00 con 0x7fb920107ed0 2024-04-25T00:05:48.109 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.041+0000 7fb91f7fe700 1 --2- 172.21.15.27:0/609535736 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fb908039050 0x7fb90803b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:48.109 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.041+0000 7fb91f7fe700 1 --2- 172.21.15.27:0/609535736 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fb908039050 0x7fb90803b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb914006fd0 tx=0x7fb914006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:48.109 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.046+0000 7fb91d7fa700 1 -- 172.21.15.27:0/609535736 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb91000b3e0 con 0x7fb920107ed0 2024-04-25T00:05:48.109 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.253+0000 7fb92668a700 1 -- 172.21.15.27:0/609535736 --> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fb920104700 con 0x7fb908039050 2024-04-25T00:05:48.109 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.518+0000 7fb91d7fa700 1 -- 172.21.15.27:0/609535736 <== mgr.14118 v2:172.21.15.27:6800/1186428290 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb920104700 con 0x7fb908039050 2024-04-25T00:05:48.109 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.520+0000 7fb92668a700 1 -- 172.21.15.27:0/609535736 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fb908039050 msgr2=0x7fb90803b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:48.110 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.520+0000 7fb92668a700 1 --2- 172.21.15.27:0/609535736 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fb908039050 0x7fb90803b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb914006fd0 tx=0x7fb914006e40 comp rx=0 tx=0).stop 2024-04-25T00:05:48.110 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.520+0000 7fb92668a700 1 -- 172.21.15.27:0/609535736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb920107ed0 msgr2=0x7fb92019bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:48.110 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.520+0000 7fb92668a700 1 --2- 172.21.15.27:0/609535736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb920107ed0 0x7fb92019bdd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb910011b60 tx=0x7fb910011c40 comp rx=0 tx=0).stop 2024-04-25T00:05:48.110 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.520+0000 7fb92668a700 1 -- 172.21.15.27:0/609535736 shutdown_connections 2024-04-25T00:05:48.110 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.520+0000 7fb92668a700 1 --2- 172.21.15.27:0/609535736 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fb908039050 0x7fb90803b510 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:48.110 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.520+0000 7fb92668a700 1 --2- 172.21.15.27:0/609535736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb920107ed0 0x7fb92019bdd0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:48.110 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.520+0000 7fb92668a700 1 -- 172.21.15.27:0/609535736 >> 172.21.15.27:0/609535736 conn(0x7fb920103490 msgr2=0x7fb920103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:48.110 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.520+0000 7fb92668a700 1 -- 172.21.15.27:0/609535736 shutdown_connections 2024-04-25T00:05:48.110 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:47.520+0000 7fb92668a700 1 -- 172.21.15.27:0/609535736 wait complete. 2024-04-25T00:05:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:48 smithi027 ceph-mon[29208]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:05:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:48 smithi027 ceph-mon[29208]: Generating ssh key... 2024-04-25T00:05:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:48 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' 2024-04-25T00:05:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:48 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' 2024-04-25T00:05:49.277 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtfwAexjPUInlhRfXsAm8aPKQNnO60XcMzYnfFwWoJxjOCTYlR6/CnHw+QCbvwOg/6ZHW2vnFboj1yr67VrnQ53UrHz/YRNJVgd3aaXad971C8oEc5XONhHydKZvqtk96GAcKTb3eD9VZ7XML5yEA5vKReRS4pEWlmFpJHjeSPD9iBzYzw91tuaZEGGvfPRZSn0+ZaeR+FV2F7KeSZmlaHi4vyj7IAQdMgovstR9z4kG0kptn89IsNEbKuUvevpi8H5GwNOl6p+zi4XPwsxGbdGbYPrL/7xDk0FKLKvyHfOtSmlE0YWwBkD1TRzT0LWxpcmCYcra/cBxKbgvvmjcGwUb9IPhQTxAha5bW9elfNpJauskfFNORINMJ/bfgJSb7NmkCX6MPaPv/G5InHu1b6bjHXHc5tuD9JSvaQBqrlm3CY1U/RWp1s54ZN1CMfAR2Mt8YERDfMDCZXnvM2PyYXymXu1dUADwm9ILTWAiOSqo2csWRNPlRQMczceD+q+fM= ceph-518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:05:49.277 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.518+0000 7f34f7aef700 1 Processor -- start 2024-04-25T00:05:49.277 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.518+0000 7f34f7aef700 1 -- start start 2024-04-25T00:05:49.277 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.518+0000 7f34f7aef700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f0108900 0x7f34f010acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:49.277 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.518+0000 7f34f7aef700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f34f007be30 con 0x7f34f0108900 2024-04-25T00:05:49.277 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.518+0000 7f34f588b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f0108900 0x7f34f010acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:49.278 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.518+0000 7f34f588b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f0108900 0x7f34f010acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46026/0 (socket says 172.21.15.27:46026) 2024-04-25T00:05:49.278 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.518+0000 7f34f588b700 1 -- 172.21.15.27:0/383314157 learned_addr learned my addr 172.21.15.27:0/383314157 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:49.278 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.519+0000 7f34f588b700 1 -- 172.21.15.27:0/383314157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f010b230 con 0x7f34f0108900 2024-04-25T00:05:49.278 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.519+0000 7f34f588b700 1 --2- 172.21.15.27:0/383314157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f0108900 0x7f34f010acf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f34ec0098a0 tx=0x7f34ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d3bf57f2210c3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:49.278 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.519+0000 7f34f4889700 1 -- 172.21.15.27:0/383314157 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34ec004160 con 0x7f34f0108900 2024-04-25T00:05:49.278 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.519+0000 7f34f4889700 1 -- 172.21.15.27:0/383314157 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f34ec004300 con 0x7f34f0108900 2024-04-25T00:05:49.278 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.519+0000 7f34f4889700 1 -- 172.21.15.27:0/383314157 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34ec011b50 con 0x7f34f0108900 2024-04-25T00:05:49.278 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.519+0000 7f34f7aef700 1 -- 172.21.15.27:0/383314157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f0108900 msgr2=0x7f34f010acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:49.278 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.519+0000 7f34f7aef700 1 --2- 172.21.15.27:0/383314157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f0108900 0x7f34f010acf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f34ec0098a0 tx=0x7f34ec009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:49.278 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.520+0000 7f34f7aef700 1 -- 172.21.15.27:0/383314157 shutdown_connections 2024-04-25T00:05:49.278 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.520+0000 7f34f7aef700 1 --2- 172.21.15.27:0/383314157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f0108900 0x7f34f010acf0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:49.278 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.520+0000 7f34f7aef700 1 -- 172.21.15.27:0/383314157 >> 172.21.15.27:0/383314157 conn(0x7f34f01022d0 msgr2=0x7f34f0104730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:49.278 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.520+0000 7f34f7aef700 1 -- 172.21.15.27:0/383314157 shutdown_connections 2024-04-25T00:05:49.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.520+0000 7f34f7aef700 1 -- 172.21.15.27:0/383314157 wait complete. 2024-04-25T00:05:49.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.520+0000 7f34f7aef700 1 Processor -- start 2024-04-25T00:05:49.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.520+0000 7f34f7aef700 1 -- start start 2024-04-25T00:05:49.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.521+0000 7f34f7aef700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f0108900 0x7f34f019bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:49.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.521+0000 7f34f7aef700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f34f010c1e0 con 0x7f34f0108900 2024-04-25T00:05:49.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.521+0000 7f34f588b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f0108900 0x7f34f019bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:49.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.521+0000 7f34f588b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f0108900 0x7f34f019bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46038/0 (socket says 172.21.15.27:46038) 2024-04-25T00:05:49.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.521+0000 7f34f588b700 1 -- 172.21.15.27:0/2746703237 learned_addr learned my addr 172.21.15.27:0/2746703237 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:49.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.521+0000 7f34f588b700 1 -- 172.21.15.27:0/2746703237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f019c2f0 con 0x7f34f0108900 2024-04-25T00:05:49.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.521+0000 7f34f588b700 1 --2- 172.21.15.27:0/2746703237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f0108900 0x7f34f019bdb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009fb0 tx=0x7f34ec011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:49.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.521+0000 7f34e6ffd700 1 -- 172.21.15.27:0/2746703237 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34ec0038b0 con 0x7f34f0108900 2024-04-25T00:05:49.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.521+0000 7f34e6ffd700 1 -- 172.21.15.27:0/2746703237 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f34ec011e50 con 0x7f34f0108900 2024-04-25T00:05:49.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.521+0000 7f34f7aef700 1 -- 172.21.15.27:0/2746703237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34f019c580 con 0x7f34f0108900 2024-04-25T00:05:49.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.521+0000 7f34e6ffd700 1 -- 172.21.15.27:0/2746703237 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34ec019420 con 0x7f34f0108900 2024-04-25T00:05:49.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.521+0000 7f34f7aef700 1 -- 172.21.15.27:0/2746703237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34f019f270 con 0x7f34f0108900 2024-04-25T00:05:49.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.522+0000 7f34e6ffd700 1 -- 172.21.15.27:0/2746703237 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f34ec0196d0 con 0x7f34f0108900 2024-04-25T00:05:49.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.522+0000 7f34f7aef700 1 -- 172.21.15.27:0/2746703237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34f0063890 con 0x7f34f0108900 2024-04-25T00:05:49.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.522+0000 7f34e6ffd700 1 --2- 172.21.15.27:0/2746703237 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f34dc039050 0x7f34dc03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:49.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.522+0000 7f34e6ffd700 1 -- 172.21.15.27:0/2746703237 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f34ec04d420 con 0x7f34f0108900 2024-04-25T00:05:49.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.522+0000 7f34f508a700 1 --2- 172.21.15.27:0/2746703237 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f34dc039050 0x7f34dc03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:49.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.523+0000 7f34f508a700 1 --2- 172.21.15.27:0/2746703237 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f34dc039050 0x7f34dc03b510 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f34e0006fd0 tx=0x7f34e0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:49.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.527+0000 7f34e6ffd700 1 -- 172.21.15.27:0/2746703237 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f34ec00ba30 con 0x7f34f0108900 2024-04-25T00:05:49.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.733+0000 7f34f7aef700 1 -- 172.21.15.27:0/2746703237 --> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f34f0103500 con 0x7f34dc039050 2024-04-25T00:05:49.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.734+0000 7f34e6ffd700 1 -- 172.21.15.27:0/2746703237 <== mgr.14118 v2:172.21.15.27:6800/1186428290 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f34f0103500 con 0x7f34dc039050 2024-04-25T00:05:49.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.736+0000 7f34f7aef700 1 -- 172.21.15.27:0/2746703237 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f34dc039050 msgr2=0x7f34dc03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:49.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.736+0000 7f34f7aef700 1 --2- 172.21.15.27:0/2746703237 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f34dc039050 0x7f34dc03b510 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f34e0006fd0 tx=0x7f34e0006e40 comp rx=0 tx=0).stop 2024-04-25T00:05:49.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.736+0000 7f34f7aef700 1 -- 172.21.15.27:0/2746703237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f0108900 msgr2=0x7f34f019bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:49.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.736+0000 7f34f7aef700 1 --2- 172.21.15.27:0/2746703237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f0108900 0x7f34f019bdb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009fb0 tx=0x7f34ec011bf0 comp rx=0 tx=0).stop 2024-04-25T00:05:49.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.736+0000 7f34f7aef700 1 -- 172.21.15.27:0/2746703237 shutdown_connections 2024-04-25T00:05:49.281 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.736+0000 7f34f7aef700 1 --2- 172.21.15.27:0/2746703237 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f34dc039050 0x7f34dc03b510 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:49.281 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.736+0000 7f34f7aef700 1 --2- 172.21.15.27:0/2746703237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f0108900 0x7f34f019bdb0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:49.281 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.736+0000 7f34f7aef700 1 -- 172.21.15.27:0/2746703237 >> 172.21.15.27:0/2746703237 conn(0x7f34f01022d0 msgr2=0x7f34f0102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:49.281 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.736+0000 7f34f7aef700 1 -- 172.21.15.27:0/2746703237 shutdown_connections 2024-04-25T00:05:49.281 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:48.736+0000 7f34f7aef700 1 -- 172.21.15.27:0/2746703237 wait complete. 2024-04-25T00:05:49.281 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-25T00:05:49.281 INFO:teuthology.orchestra.run.smithi027.stdout:Adding key to root@localhost authorized_keys... 2024-04-25T00:05:49.281 INFO:teuthology.orchestra.run.smithi027.stdout:Adding host smithi027... 2024-04-25T00:05:49.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:49 smithi027 ceph-mon[29208]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:05:50.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:50 smithi027 ceph-mon[29208]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi027", "addr": "172.21.15.27", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:05:51.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:51 smithi027 ceph-mon[29208]: Deploying cephadm binary to smithi027 2024-04-25T00:05:54.826 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Added host 'smithi027' with addr '172.21.15.27' 2024-04-25T00:05:54.826 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.723+0000 7f7db2cec700 1 Processor -- start 2024-04-25T00:05:54.826 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.723+0000 7f7db2cec700 1 -- start start 2024-04-25T00:05:54.826 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.723+0000 7f7db2cec700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dac07ddf0 0x7f7dac07c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:54.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.723+0000 7f7db2cec700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7dac07c7e0 con 0x7f7dac07ddf0 2024-04-25T00:05:54.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.723+0000 7f7db0a88700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dac07ddf0 0x7f7dac07c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:54.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.723+0000 7f7db0a88700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dac07ddf0 0x7f7dac07c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46044/0 (socket says 172.21.15.27:46044) 2024-04-25T00:05:54.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.723+0000 7f7db0a88700 1 -- 172.21.15.27:0/3006741190 learned_addr learned my addr 172.21.15.27:0/3006741190 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:54.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.723+0000 7f7db0a88700 1 -- 172.21.15.27:0/3006741190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dac07c960 con 0x7f7dac07ddf0 2024-04-25T00:05:54.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.723+0000 7f7db0a88700 1 --2- 172.21.15.27:0/3006741190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dac07ddf0 0x7f7dac07c2a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7da00098a0 tx=0x7f7da0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d9082fb5f0691b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:54.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.724+0000 7f7dab7fe700 1 -- 172.21.15.27:0/3006741190 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7da0004160 con 0x7f7dac07ddf0 2024-04-25T00:05:54.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.724+0000 7f7dab7fe700 1 -- 172.21.15.27:0/3006741190 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7da0004300 con 0x7f7dac07ddf0 2024-04-25T00:05:54.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.724+0000 7f7dab7fe700 1 -- 172.21.15.27:0/3006741190 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7da0011b50 con 0x7f7dac07ddf0 2024-04-25T00:05:54.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.724+0000 7f7db2cec700 1 -- 172.21.15.27:0/3006741190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dac07ddf0 msgr2=0x7f7dac07c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:54.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.724+0000 7f7db2cec700 1 --2- 172.21.15.27:0/3006741190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dac07ddf0 0x7f7dac07c2a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7da00098a0 tx=0x7f7da0009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:54.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.724+0000 7f7db2cec700 1 -- 172.21.15.27:0/3006741190 shutdown_connections 2024-04-25T00:05:54.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.724+0000 7f7db2cec700 1 --2- 172.21.15.27:0/3006741190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dac07ddf0 0x7f7dac07c2a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:54.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.724+0000 7f7db2cec700 1 -- 172.21.15.27:0/3006741190 >> 172.21.15.27:0/3006741190 conn(0x7f7dac103490 msgr2=0x7f7dac1058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:54.828 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.724+0000 7f7db2cec700 1 -- 172.21.15.27:0/3006741190 shutdown_connections 2024-04-25T00:05:54.828 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.724+0000 7f7db2cec700 1 -- 172.21.15.27:0/3006741190 wait complete. 2024-04-25T00:05:54.828 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.725+0000 7f7db2cec700 1 Processor -- start 2024-04-25T00:05:54.828 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.725+0000 7f7db2cec700 1 -- start start 2024-04-25T00:05:54.828 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.725+0000 7f7db2cec700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dac07ddf0 0x7f7dac19df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:54.828 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.725+0000 7f7db2cec700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7dac10e370 con 0x7f7dac07ddf0 2024-04-25T00:05:54.828 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.725+0000 7f7db0a88700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dac07ddf0 0x7f7dac19df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:54.828 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.725+0000 7f7db0a88700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dac07ddf0 0x7f7dac19df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46056/0 (socket says 172.21.15.27:46056) 2024-04-25T00:05:54.828 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.725+0000 7f7db0a88700 1 -- 172.21.15.27:0/2872882932 learned_addr learned my addr 172.21.15.27:0/2872882932 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:54.828 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.725+0000 7f7db0a88700 1 -- 172.21.15.27:0/2872882932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dac19e450 con 0x7f7dac07ddf0 2024-04-25T00:05:54.828 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.726+0000 7f7db0a88700 1 --2- 172.21.15.27:0/2872882932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dac07ddf0 0x7f7dac19df10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7da0011b60 tx=0x7f7da0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:54.828 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.726+0000 7f7da9ffb700 1 -- 172.21.15.27:0/2872882932 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7da0003b20 con 0x7f7dac07ddf0 2024-04-25T00:05:54.828 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.726+0000 7f7da9ffb700 1 -- 172.21.15.27:0/2872882932 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7da0003cc0 con 0x7f7dac07ddf0 2024-04-25T00:05:54.828 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.726+0000 7f7db2cec700 1 -- 172.21.15.27:0/2872882932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dac19e6e0 con 0x7f7dac07ddf0 2024-04-25T00:05:54.829 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.726+0000 7f7da9ffb700 1 -- 172.21.15.27:0/2872882932 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7da0024840 con 0x7f7dac07ddf0 2024-04-25T00:05:54.829 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.726+0000 7f7db2cec700 1 -- 172.21.15.27:0/2872882932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dac1a13d0 con 0x7f7dac07ddf0 2024-04-25T00:05:54.829 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.727+0000 7f7da9ffb700 1 -- 172.21.15.27:0/2872882932 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f7da0022020 con 0x7f7dac07ddf0 2024-04-25T00:05:54.829 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.727+0000 7f7db2cec700 1 -- 172.21.15.27:0/2872882932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dac063890 con 0x7f7dac07ddf0 2024-04-25T00:05:54.829 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.727+0000 7f7da9ffb700 1 --2- 172.21.15.27:0/2872882932 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f7d9c0390a0 0x7f7d9c03b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:54.829 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.727+0000 7f7da9ffb700 1 -- 172.21.15.27:0/2872882932 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7da0011e00 con 0x7f7dac07ddf0 2024-04-25T00:05:54.829 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.727+0000 7f7dabfff700 1 --2- 172.21.15.27:0/2872882932 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f7d9c0390a0 0x7f7d9c03b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:54.829 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.728+0000 7f7dabfff700 1 --2- 172.21.15.27:0/2872882932 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f7d9c0390a0 0x7f7d9c03b560 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7d98006fd0 tx=0x7f7d98006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:54.829 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.732+0000 7f7da9ffb700 1 -- 172.21.15.27:0/2872882932 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7da000b9e0 con 0x7f7dac07ddf0 2024-04-25T00:05:54.829 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:49.941+0000 7f7db2cec700 1 -- 172.21.15.27:0/2872882932 --> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi027", "addr": "172.21.15.27", "target": ["mon-mgr", ""]}) -- 0x7f7dac104670 con 0x7f7d9c0390a0 2024-04-25T00:05:54.829 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:54.202+0000 7f7da9ffb700 1 -- 172.21.15.27:0/2872882932 <== mgr.14118 v2:172.21.15.27:6800/1186428290 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f7dac104670 con 0x7f7d9c0390a0 2024-04-25T00:05:54.829 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:54.204+0000 7f7db2cec700 1 -- 172.21.15.27:0/2872882932 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f7d9c0390a0 msgr2=0x7f7d9c03b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:54.830 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:54.204+0000 7f7db2cec700 1 --2- 172.21.15.27:0/2872882932 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f7d9c0390a0 0x7f7d9c03b560 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7d98006fd0 tx=0x7f7d98006e40 comp rx=0 tx=0).stop 2024-04-25T00:05:54.830 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:54.204+0000 7f7db2cec700 1 -- 172.21.15.27:0/2872882932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dac07ddf0 msgr2=0x7f7dac19df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:54.830 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:54.204+0000 7f7db2cec700 1 --2- 172.21.15.27:0/2872882932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dac07ddf0 0x7f7dac19df10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7da0011b60 tx=0x7f7da0011c40 comp rx=0 tx=0).stop 2024-04-25T00:05:54.830 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:54.204+0000 7f7db2cec700 1 -- 172.21.15.27:0/2872882932 shutdown_connections 2024-04-25T00:05:54.830 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:54.204+0000 7f7db2cec700 1 --2- 172.21.15.27:0/2872882932 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f7d9c0390a0 0x7f7d9c03b560 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:54.830 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:54.204+0000 7f7db2cec700 1 --2- 172.21.15.27:0/2872882932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dac07ddf0 0x7f7dac19df10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:54.830 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:54.204+0000 7f7db2cec700 1 -- 172.21.15.27:0/2872882932 >> 172.21.15.27:0/2872882932 conn(0x7f7dac103490 msgr2=0x7f7dac103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:54.830 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:54.204+0000 7f7db2cec700 1 -- 172.21.15.27:0/2872882932 shutdown_connections 2024-04-25T00:05:54.830 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:54.204+0000 7f7db2cec700 1 -- 172.21.15.27:0/2872882932 wait complete. 2024-04-25T00:05:54.830 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying unmanaged mon service... 2024-04-25T00:05:55.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:55 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' 2024-04-25T00:05:55.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:55 smithi027 ceph-mon[29208]: Added host smithi027 2024-04-25T00:05:55.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:55 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:05:56.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-25T00:05:56.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.210+0000 7f604e7f0700 1 Processor -- start 2024-04-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.210+0000 7f604e7f0700 1 -- start start 2024-04-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.210+0000 7f604e7f0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6048107ec0 0x7f60481082e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.210+0000 7f604e7f0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6048108820 con 0x7f6048107ec0 2024-04-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.211+0000 7f6047fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6048107ec0 0x7f60481082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.211+0000 7f6047fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6048107ec0 0x7f60481082e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46058/0 (socket says 172.21.15.27:46058) 2024-04-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.211+0000 7f6047fff700 1 -- 172.21.15.27:0/1652832838 learned_addr learned my addr 172.21.15.27:0/1652832838 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.211+0000 7f6047fff700 1 -- 172.21.15.27:0/1652832838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6048108fd0 con 0x7f6048107ec0 2024-04-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.211+0000 7f6047fff700 1 --2- 172.21.15.27:0/1652832838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6048107ec0 0x7f60481082e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f60300098a0 tx=0x7f6030009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=129dca7420363647 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.211+0000 7f6046ffd700 1 -- 172.21.15.27:0/1652832838 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6030004160 con 0x7f6048107ec0 2024-04-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.211+0000 7f6046ffd700 1 -- 172.21.15.27:0/1652832838 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6030004300 con 0x7f6048107ec0 2024-04-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.211+0000 7f6046ffd700 1 -- 172.21.15.27:0/1652832838 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6030011b50 con 0x7f6048107ec0 2024-04-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.212+0000 7f604e7f0700 1 -- 172.21.15.27:0/1652832838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6048107ec0 msgr2=0x7f60481082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.212+0000 7f604e7f0700 1 --2- 172.21.15.27:0/1652832838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6048107ec0 0x7f60481082e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f60300098a0 tx=0x7f6030009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:56.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.212+0000 7f604e7f0700 1 -- 172.21.15.27:0/1652832838 shutdown_connections 2024-04-25T00:05:56.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.212+0000 7f604e7f0700 1 --2- 172.21.15.27:0/1652832838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6048107ec0 0x7f60481082e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:56.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.212+0000 7f604e7f0700 1 -- 172.21.15.27:0/1652832838 >> 172.21.15.27:0/1652832838 conn(0x7f60481034c0 msgr2=0x7f60481058e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:56.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.212+0000 7f604e7f0700 1 -- 172.21.15.27:0/1652832838 shutdown_connections 2024-04-25T00:05:56.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.212+0000 7f604e7f0700 1 -- 172.21.15.27:0/1652832838 wait complete. 2024-04-25T00:05:56.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.213+0000 7f604e7f0700 1 Processor -- start 2024-04-25T00:05:56.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.213+0000 7f604e7f0700 1 -- start start 2024-04-25T00:05:56.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.213+0000 7f604e7f0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6048107ec0 0x7f604819bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:56.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.213+0000 7f604e7f0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f604810c290 con 0x7f6048107ec0 2024-04-25T00:05:56.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.213+0000 7f6047fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6048107ec0 0x7f604819bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:56.256 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.213+0000 7f6047fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6048107ec0 0x7f604819bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46074/0 (socket says 172.21.15.27:46074) 2024-04-25T00:05:56.256 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.213+0000 7f6047fff700 1 -- 172.21.15.27:0/1674742218 learned_addr learned my addr 172.21.15.27:0/1674742218 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:56.256 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.214+0000 7f6047fff700 1 -- 172.21.15.27:0/1674742218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f604819c290 con 0x7f6048107ec0 2024-04-25T00:05:56.256 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.214+0000 7f6047fff700 1 --2- 172.21.15.27:0/1674742218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6048107ec0 0x7f604819bd50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6030011670 tx=0x7f6030011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:56.256 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.214+0000 7f60457fa700 1 -- 172.21.15.27:0/1674742218 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60300039c0 con 0x7f6048107ec0 2024-04-25T00:05:56.256 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.214+0000 7f60457fa700 1 -- 172.21.15.27:0/1674742218 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6030011e50 con 0x7f6048107ec0 2024-04-25T00:05:56.256 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.214+0000 7f604e7f0700 1 -- 172.21.15.27:0/1674742218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f604819c520 con 0x7f6048107ec0 2024-04-25T00:05:56.256 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.214+0000 7f60457fa700 1 -- 172.21.15.27:0/1674742218 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f603001a5c0 con 0x7f6048107ec0 2024-04-25T00:05:56.256 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.214+0000 7f604e7f0700 1 -- 172.21.15.27:0/1674742218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f604819f210 con 0x7f6048107ec0 2024-04-25T00:05:56.256 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.215+0000 7f604e7f0700 1 -- 172.21.15.27:0/1674742218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6048063890 con 0x7f6048107ec0 2024-04-25T00:05:56.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.215+0000 7f60457fa700 1 -- 172.21.15.27:0/1674742218 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f603001a8c0 con 0x7f6048107ec0 2024-04-25T00:05:56.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.215+0000 7f60457fa700 1 --2- 172.21.15.27:0/1674742218 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f603403d410 0x7f603403f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:56.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.215+0000 7f60457fa700 1 -- 172.21.15.27:0/1674742218 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f603004d4c0 con 0x7f6048107ec0 2024-04-25T00:05:56.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.216+0000 7f60477fe700 1 --2- 172.21.15.27:0/1674742218 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f603403d410 0x7f603403f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:56.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.216+0000 7f60477fe700 1 --2- 172.21.15.27:0/1674742218 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f603403d410 0x7f603403f8d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6038006fd0 tx=0x7f6038006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:56.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.220+0000 7f60457fa700 1 -- 172.21.15.27:0/1674742218 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f603002d910 con 0x7f6048107ec0 2024-04-25T00:05:56.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.454+0000 7f604e7f0700 1 -- 172.21.15.27:0/1674742218 --> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f60481046a0 con 0x7f603403d410 2024-04-25T00:05:56.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.459+0000 7f60457fa700 1 -- 172.21.15.27:0/1674742218 <== mgr.14118 v2:172.21.15.27:6800/1186428290 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f60481046a0 con 0x7f603403d410 2024-04-25T00:05:56.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.461+0000 7f604e7f0700 1 -- 172.21.15.27:0/1674742218 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f603403d410 msgr2=0x7f603403f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:56.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.461+0000 7f604e7f0700 1 --2- 172.21.15.27:0/1674742218 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f603403d410 0x7f603403f8d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6038006fd0 tx=0x7f6038006e40 comp rx=0 tx=0).stop 2024-04-25T00:05:56.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.461+0000 7f604e7f0700 1 -- 172.21.15.27:0/1674742218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6048107ec0 msgr2=0x7f604819bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:56.258 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.461+0000 7f604e7f0700 1 --2- 172.21.15.27:0/1674742218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6048107ec0 0x7f604819bd50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6030011670 tx=0x7f6030011b60 comp rx=0 tx=0).stop 2024-04-25T00:05:56.258 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.461+0000 7f604e7f0700 1 -- 172.21.15.27:0/1674742218 shutdown_connections 2024-04-25T00:05:56.258 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.461+0000 7f604e7f0700 1 --2- 172.21.15.27:0/1674742218 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f603403d410 0x7f603403f8d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:56.258 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.461+0000 7f604e7f0700 1 --2- 172.21.15.27:0/1674742218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6048107ec0 0x7f604819bd50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:56.258 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.461+0000 7f604e7f0700 1 -- 172.21.15.27:0/1674742218 >> 172.21.15.27:0/1674742218 conn(0x7f60481034c0 msgr2=0x7f6048103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:56.258 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.461+0000 7f604e7f0700 1 -- 172.21.15.27:0/1674742218 shutdown_connections 2024-04-25T00:05:56.258 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:55.461+0000 7f604e7f0700 1 -- 172.21.15.27:0/1674742218 wait complete. 2024-04-25T00:05:56.258 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying unmanaged mgr service... 2024-04-25T00:05:56.538 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:56 smithi027 ceph-mon[29208]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:05:56.538 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:56 smithi027 ceph-mon[29208]: Saving service mon spec with placement count:5 2024-04-25T00:05:56.538 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:56 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' 2024-04-25T00:05:57.540 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-25T00:05:57.540 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.644+0000 7fe934228700 1 Processor -- start 2024-04-25T00:05:57.541 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.644+0000 7fe934228700 1 -- start start 2024-04-25T00:05:57.541 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.644+0000 7fe934228700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c1088d0 0x7fe92c10acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:57.541 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.644+0000 7fe934228700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe92c07be70 con 0x7fe92c1088d0 2024-04-25T00:05:57.541 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.644+0000 7fe931fc4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c1088d0 0x7fe92c10acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:57.541 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.644+0000 7fe931fc4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c1088d0 0x7fe92c10acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41964/0 (socket says 172.21.15.27:41964) 2024-04-25T00:05:57.541 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.644+0000 7fe931fc4700 1 -- 172.21.15.27:0/2961013314 learned_addr learned my addr 172.21.15.27:0/2961013314 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:57.541 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.645+0000 7fe931fc4700 1 -- 172.21.15.27:0/2961013314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe92c10b200 con 0x7fe92c1088d0 2024-04-25T00:05:57.541 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.645+0000 7fe931fc4700 1 --2- 172.21.15.27:0/2961013314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c1088d0 0x7fe92c10acc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe9280098a0 tx=0x7fe928009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe8e2734533e874 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:57.541 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.645+0000 7fe930fc2700 1 -- 172.21.15.27:0/2961013314 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe928004160 con 0x7fe92c1088d0 2024-04-25T00:05:57.542 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.645+0000 7fe930fc2700 1 -- 172.21.15.27:0/2961013314 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe928004300 con 0x7fe92c1088d0 2024-04-25T00:05:57.542 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.645+0000 7fe930fc2700 1 -- 172.21.15.27:0/2961013314 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe928011b50 con 0x7fe92c1088d0 2024-04-25T00:05:57.542 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.646+0000 7fe934228700 1 -- 172.21.15.27:0/2961013314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c1088d0 msgr2=0x7fe92c10acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:57.542 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.646+0000 7fe934228700 1 --2- 172.21.15.27:0/2961013314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c1088d0 0x7fe92c10acc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe9280098a0 tx=0x7fe928009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:57.542 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.646+0000 7fe934228700 1 -- 172.21.15.27:0/2961013314 shutdown_connections 2024-04-25T00:05:57.542 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.646+0000 7fe934228700 1 --2- 172.21.15.27:0/2961013314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c1088d0 0x7fe92c10acc0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:57.542 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.646+0000 7fe934228700 1 -- 172.21.15.27:0/2961013314 >> 172.21.15.27:0/2961013314 conn(0x7fe92c102340 msgr2=0x7fe92c104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:57.543 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.646+0000 7fe934228700 1 -- 172.21.15.27:0/2961013314 shutdown_connections 2024-04-25T00:05:57.543 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.646+0000 7fe934228700 1 -- 172.21.15.27:0/2961013314 wait complete. 2024-04-25T00:05:57.543 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.646+0000 7fe934228700 1 Processor -- start 2024-04-25T00:05:57.543 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.646+0000 7fe934228700 1 -- start start 2024-04-25T00:05:57.543 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.647+0000 7fe934228700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c1088d0 0x7fe92c19bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:57.543 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.647+0000 7fe934228700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe92c10c1c0 con 0x7fe92c1088d0 2024-04-25T00:05:57.544 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.647+0000 7fe931fc4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c1088d0 0x7fe92c19bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:57.544 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.647+0000 7fe931fc4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c1088d0 0x7fe92c19bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41976/0 (socket says 172.21.15.27:41976) 2024-04-25T00:05:57.544 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.647+0000 7fe931fc4700 1 -- 172.21.15.27:0/1893945566 learned_addr learned my addr 172.21.15.27:0/1893945566 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:57.544 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.647+0000 7fe931fc4700 1 -- 172.21.15.27:0/1893945566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe92c19c280 con 0x7fe92c1088d0 2024-04-25T00:05:57.544 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.647+0000 7fe931fc4700 1 --2- 172.21.15.27:0/1893945566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c1088d0 0x7fe92c19bd40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe928011670 tx=0x7fe928011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:57.544 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.647+0000 7fe922ffd700 1 -- 172.21.15.27:0/1893945566 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9280039c0 con 0x7fe92c1088d0 2024-04-25T00:05:57.544 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.647+0000 7fe922ffd700 1 -- 172.21.15.27:0/1893945566 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe928011e50 con 0x7fe92c1088d0 2024-04-25T00:05:57.544 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.647+0000 7fe922ffd700 1 -- 172.21.15.27:0/1893945566 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe92801a5c0 con 0x7fe92c1088d0 2024-04-25T00:05:57.544 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.647+0000 7fe934228700 1 -- 172.21.15.27:0/1893945566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe92c19c510 con 0x7fe92c1088d0 2024-04-25T00:05:57.544 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.647+0000 7fe934228700 1 -- 172.21.15.27:0/1893945566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe92c19f200 con 0x7fe92c1088d0 2024-04-25T00:05:57.544 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.648+0000 7fe922ffd700 1 -- 172.21.15.27:0/1893945566 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fe928024430 con 0x7fe92c1088d0 2024-04-25T00:05:57.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.648+0000 7fe934228700 1 -- 172.21.15.27:0/1893945566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe92c063890 con 0x7fe92c1088d0 2024-04-25T00:05:57.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.648+0000 7fe922ffd700 1 --2- 172.21.15.27:0/1893945566 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fe918039050 0x7fe91803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:57.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.648+0000 7fe922ffd700 1 -- 172.21.15.27:0/1893945566 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe92804d430 con 0x7fe92c1088d0 2024-04-25T00:05:57.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.649+0000 7fe9317c3700 1 --2- 172.21.15.27:0/1893945566 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fe918039050 0x7fe91803b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:57.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.649+0000 7fe9317c3700 1 --2- 172.21.15.27:0/1893945566 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fe918039050 0x7fe91803b510 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe91c006fd0 tx=0x7fe91c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:57.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.653+0000 7fe922ffd700 1 -- 172.21.15.27:0/1893945566 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe92802d3f0 con 0x7fe92c1088d0 2024-04-25T00:05:57.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.890+0000 7fe934228700 1 -- 172.21.15.27:0/1893945566 --> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fe92c1034c0 con 0x7fe918039050 2024-04-25T00:05:57.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.897+0000 7fe922ffd700 1 -- 172.21.15.27:0/1893945566 <== mgr.14118 v2:172.21.15.27:6800/1186428290 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe92c1034c0 con 0x7fe918039050 2024-04-25T00:05:57.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.899+0000 7fe934228700 1 -- 172.21.15.27:0/1893945566 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fe918039050 msgr2=0x7fe91803b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:57.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.899+0000 7fe934228700 1 --2- 172.21.15.27:0/1893945566 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fe918039050 0x7fe91803b510 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe91c006fd0 tx=0x7fe91c006e40 comp rx=0 tx=0).stop 2024-04-25T00:05:57.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.899+0000 7fe934228700 1 -- 172.21.15.27:0/1893945566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c1088d0 msgr2=0x7fe92c19bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:57.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.899+0000 7fe934228700 1 --2- 172.21.15.27:0/1893945566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c1088d0 0x7fe92c19bd40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe928011670 tx=0x7fe928011b60 comp rx=0 tx=0).stop 2024-04-25T00:05:57.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.900+0000 7fe934228700 1 -- 172.21.15.27:0/1893945566 shutdown_connections 2024-04-25T00:05:57.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.900+0000 7fe934228700 1 --2- 172.21.15.27:0/1893945566 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fe918039050 0x7fe91803b510 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:57.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.900+0000 7fe934228700 1 --2- 172.21.15.27:0/1893945566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92c1088d0 0x7fe92c19bd40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:57.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.900+0000 7fe934228700 1 -- 172.21.15.27:0/1893945566 >> 172.21.15.27:0/1893945566 conn(0x7fe92c102340 msgr2=0x7fe92c102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:57.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.900+0000 7fe934228700 1 -- 172.21.15.27:0/1893945566 shutdown_connections 2024-04-25T00:05:57.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:56.900+0000 7fe934228700 1 -- 172.21.15.27:0/1893945566 wait complete. 2024-04-25T00:05:58.048 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:57 smithi027 ceph-mon[29208]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:05:58.048 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:57 smithi027 ceph-mon[29208]: Saving service mgr spec with placement count:2 2024-04-25T00:05:58.048 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:57 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' 2024-04-25T00:05:59.621 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:59 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' 2024-04-25T00:05:59.622 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:59 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' 2024-04-25T00:05:59.622 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:05:59 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/842920942' entity='client.admin' 2024-04-25T00:05:59.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.791+0000 7f9b98909700 1 Processor -- start 2024-04-25T00:05:59.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.791+0000 7f9b98909700 1 -- start start 2024-04-25T00:05:59.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.792+0000 7f9b98909700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b94107ec0 0x7f9b941082e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:59.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.792+0000 7f9b98909700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9b94108820 con 0x7f9b94107ec0 2024-04-25T00:05:59.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.792+0000 7f9b9259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b94107ec0 0x7f9b941082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:59.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.792+0000 7f9b9259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b94107ec0 0x7f9b941082e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41992/0 (socket says 172.21.15.27:41992) 2024-04-25T00:05:59.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.792+0000 7f9b9259c700 1 -- 172.21.15.27:0/2844085754 learned_addr learned my addr 172.21.15.27:0/2844085754 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:59.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.792+0000 7f9b9259c700 1 -- 172.21.15.27:0/2844085754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b94108fd0 con 0x7f9b94107ec0 2024-04-25T00:05:59.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.793+0000 7f9b9259c700 1 --2- 172.21.15.27:0/2844085754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b94107ec0 0x7f9b941082e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9b840098a0 tx=0x7f9b84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d5d06a5b74b4e2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:59.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.793+0000 7f9b9159a700 1 -- 172.21.15.27:0/2844085754 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b84004160 con 0x7f9b94107ec0 2024-04-25T00:05:59.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.793+0000 7f9b9159a700 1 -- 172.21.15.27:0/2844085754 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b84004300 con 0x7f9b94107ec0 2024-04-25T00:05:59.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.793+0000 7f9b9159a700 1 -- 172.21.15.27:0/2844085754 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b84011b20 con 0x7f9b94107ec0 2024-04-25T00:05:59.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.793+0000 7f9b98909700 1 -- 172.21.15.27:0/2844085754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b94107ec0 msgr2=0x7f9b941082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:59.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.793+0000 7f9b98909700 1 --2- 172.21.15.27:0/2844085754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b94107ec0 0x7f9b941082e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9b840098a0 tx=0x7f9b84009bb0 comp rx=0 tx=0).stop 2024-04-25T00:05:59.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.794+0000 7f9b98909700 1 -- 172.21.15.27:0/2844085754 shutdown_connections 2024-04-25T00:05:59.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.794+0000 7f9b98909700 1 --2- 172.21.15.27:0/2844085754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b94107ec0 0x7f9b941082e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:59.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.794+0000 7f9b98909700 1 -- 172.21.15.27:0/2844085754 >> 172.21.15.27:0/2844085754 conn(0x7f9b941034c0 msgr2=0x7f9b941058e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:59.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.794+0000 7f9b98909700 1 -- 172.21.15.27:0/2844085754 shutdown_connections 2024-04-25T00:05:59.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.794+0000 7f9b98909700 1 -- 172.21.15.27:0/2844085754 wait complete. 2024-04-25T00:05:59.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.794+0000 7f9b98909700 1 Processor -- start 2024-04-25T00:05:59.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.794+0000 7f9b98909700 1 -- start start 2024-04-25T00:05:59.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.795+0000 7f9b98909700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b94107ec0 0x7f9b9419bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:59.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.795+0000 7f9b98909700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9410c410 con 0x7f9b94107ec0 2024-04-25T00:05:59.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.795+0000 7f9b9259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b94107ec0 0x7f9b9419bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:59.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.795+0000 7f9b9259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b94107ec0 0x7f9b9419bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42002/0 (socket says 172.21.15.27:42002) 2024-04-25T00:05:59.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.795+0000 7f9b9259c700 1 -- 172.21.15.27:0/842920942 learned_addr learned my addr 172.21.15.27:0/842920942 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:05:59.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.795+0000 7f9b9259c700 1 -- 172.21.15.27:0/842920942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9419c220 con 0x7f9b94107ec0 2024-04-25T00:05:59.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.795+0000 7f9b9259c700 1 --2- 172.21.15.27:0/842920942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b94107ec0 0x7f9b9419bce0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9b840117e0 tx=0x7f9b84011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:59.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.795+0000 7f9b837fe700 1 -- 172.21.15.27:0/842920942 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b84003ba0 con 0x7f9b94107ec0 2024-04-25T00:05:59.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.795+0000 7f9b837fe700 1 -- 172.21.15.27:0/842920942 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b8401a910 con 0x7f9b94107ec0 2024-04-25T00:05:59.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.796+0000 7f9b837fe700 1 -- 172.21.15.27:0/842920942 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b840248f0 con 0x7f9b94107ec0 2024-04-25T00:05:59.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.796+0000 7f9b98909700 1 -- 172.21.15.27:0/842920942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b9419c4b0 con 0x7f9b94107ec0 2024-04-25T00:05:59.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.796+0000 7f9b98909700 1 -- 172.21.15.27:0/842920942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b9419c910 con 0x7f9b94107ec0 2024-04-25T00:05:59.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.797+0000 7f9b98909700 1 -- 172.21.15.27:0/842920942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b94063890 con 0x7f9b94107ec0 2024-04-25T00:05:59.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.797+0000 7f9b837fe700 1 -- 172.21.15.27:0/842920942 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f9b84022020 con 0x7f9b94107ec0 2024-04-25T00:05:59.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.797+0000 7f9b837fe700 1 --2- 172.21.15.27:0/842920942 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f9b7c0390a0 0x7f9b7c03b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:05:59.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.797+0000 7f9b837fe700 1 -- 172.21.15.27:0/842920942 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9b8404d530 con 0x7f9b94107ec0 2024-04-25T00:05:59.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.797+0000 7f9b91d9b700 1 --2- 172.21.15.27:0/842920942 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f9b7c0390a0 0x7f9b7c03b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:05:59.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.798+0000 7f9b91d9b700 1 --2- 172.21.15.27:0/842920942 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f9b7c0390a0 0x7f9b7c03b560 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9b88006fd0 tx=0x7f9b88006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:05:59.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:58.802+0000 7f9b837fe700 1 -- 172.21.15.27:0/842920942 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9b8401ab70 con 0x7f9b94107ec0 2024-04-25T00:05:59.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.021+0000 7f9b98909700 1 -- 172.21.15.27:0/842920942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f9b9419f740 con 0x7f9b94107ec0 2024-04-25T00:05:59.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.024+0000 7f9b837fe700 1 -- 172.21.15.27:0/842920942 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f9b840212c0 con 0x7f9b94107ec0 2024-04-25T00:05:59.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.026+0000 7f9b98909700 1 -- 172.21.15.27:0/842920942 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f9b7c0390a0 msgr2=0x7f9b7c03b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:59.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.026+0000 7f9b98909700 1 --2- 172.21.15.27:0/842920942 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f9b7c0390a0 0x7f9b7c03b560 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9b88006fd0 tx=0x7f9b88006e40 comp rx=0 tx=0).stop 2024-04-25T00:05:59.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.026+0000 7f9b98909700 1 -- 172.21.15.27:0/842920942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b94107ec0 msgr2=0x7f9b9419bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:05:59.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.026+0000 7f9b98909700 1 --2- 172.21.15.27:0/842920942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b94107ec0 0x7f9b9419bce0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9b840117e0 tx=0x7f9b84011ba0 comp rx=0 tx=0).stop 2024-04-25T00:05:59.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.027+0000 7f9b98909700 1 -- 172.21.15.27:0/842920942 shutdown_connections 2024-04-25T00:05:59.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.027+0000 7f9b98909700 1 --2- 172.21.15.27:0/842920942 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f9b7c0390a0 0x7f9b7c03b560 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:59.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.027+0000 7f9b98909700 1 --2- 172.21.15.27:0/842920942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b94107ec0 0x7f9b9419bce0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:05:59.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.027+0000 7f9b98909700 1 -- 172.21.15.27:0/842920942 >> 172.21.15.27:0/842920942 conn(0x7f9b941034c0 msgr2=0x7f9b94103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:05:59.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.027+0000 7f9b98909700 1 -- 172.21.15.27:0/842920942 shutdown_connections 2024-04-25T00:05:59.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.027+0000 7f9b98909700 1 -- 172.21.15.27:0/842920942 wait complete. 2024-04-25T00:06:01.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:01 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/266024196' entity='client.admin' 2024-04-25T00:06:01.348 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.987+0000 7fbba1a7d700 1 Processor -- start 2024-04-25T00:06:01.348 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.987+0000 7fbba1a7d700 1 -- start start 2024-04-25T00:06:01.348 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.987+0000 7fbba1a7d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb9c107e20 0x7fbb9c108240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:01.348 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.987+0000 7fbba1a7d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9c108780 con 0x7fbb9c107e20 2024-04-25T00:06:01.348 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.987+0000 7fbb9b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb9c107e20 0x7fbb9c108240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:01.348 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.987+0000 7fbb9b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb9c107e20 0x7fbb9c108240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42012/0 (socket says 172.21.15.27:42012) 2024-04-25T00:06:01.348 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.987+0000 7fbb9b7fe700 1 -- 172.21.15.27:0/379747735 learned_addr learned my addr 172.21.15.27:0/379747735 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:01.348 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.988+0000 7fbb9b7fe700 1 -- 172.21.15.27:0/379747735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb9c108f80 con 0x7fbb9c107e20 2024-04-25T00:06:01.348 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.988+0000 7fbb9b7fe700 1 --2- 172.21.15.27:0/379747735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb9c107e20 0x7fbb9c108240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c0098a0 tx=0x7fbb8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77eb71da258b635 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:01.348 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.988+0000 7fbb9a7fc700 1 -- 172.21.15.27:0/379747735 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb8c004160 con 0x7fbb9c107e20 2024-04-25T00:06:01.349 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.988+0000 7fbb9a7fc700 1 -- 172.21.15.27:0/379747735 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb8c004300 con 0x7fbb9c107e20 2024-04-25T00:06:01.349 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.988+0000 7fbb9a7fc700 1 -- 172.21.15.27:0/379747735 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb8c011b20 con 0x7fbb9c107e20 2024-04-25T00:06:01.349 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.989+0000 7fbba1a7d700 1 -- 172.21.15.27:0/379747735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb9c107e20 msgr2=0x7fbb9c108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:01.349 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.989+0000 7fbba1a7d700 1 --2- 172.21.15.27:0/379747735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb9c107e20 0x7fbb9c108240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c0098a0 tx=0x7fbb8c009bb0 comp rx=0 tx=0).stop 2024-04-25T00:06:01.349 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.989+0000 7fbba1a7d700 1 -- 172.21.15.27:0/379747735 shutdown_connections 2024-04-25T00:06:01.349 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.989+0000 7fbba1a7d700 1 --2- 172.21.15.27:0/379747735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb9c107e20 0x7fbb9c108240 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:01.349 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.989+0000 7fbba1a7d700 1 -- 172.21.15.27:0/379747735 >> 172.21.15.27:0/379747735 conn(0x7fbb9c1033e0 msgr2=0x7fbb9c105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:01.349 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.989+0000 7fbba1a7d700 1 -- 172.21.15.27:0/379747735 shutdown_connections 2024-04-25T00:06:01.349 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.989+0000 7fbba1a7d700 1 -- 172.21.15.27:0/379747735 wait complete. 2024-04-25T00:06:01.349 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.989+0000 7fbba1a7d700 1 Processor -- start 2024-04-25T00:06:01.349 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.990+0000 7fbba1a7d700 1 -- start start 2024-04-25T00:06:01.349 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.990+0000 7fbba1a7d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb9c107e20 0x7fbb9c19bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:01.349 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.990+0000 7fbba1a7d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9c10c3c0 con 0x7fbb9c107e20 2024-04-25T00:06:01.349 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.990+0000 7fbb9b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb9c107e20 0x7fbb9c19bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:01.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.990+0000 7fbb9b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb9c107e20 0x7fbb9c19bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42016/0 (socket says 172.21.15.27:42016) 2024-04-25T00:06:01.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.990+0000 7fbb9b7fe700 1 -- 172.21.15.27:0/266024196 learned_addr learned my addr 172.21.15.27:0/266024196 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:01.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.990+0000 7fbb9b7fe700 1 -- 172.21.15.27:0/266024196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb9c19c240 con 0x7fbb9c107e20 2024-04-25T00:06:01.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.990+0000 7fbb9b7fe700 1 --2- 172.21.15.27:0/266024196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb9c107e20 0x7fbb9c19bd00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c011b60 tx=0x7fbb8c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:01.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.991+0000 7fbb98ff9700 1 -- 172.21.15.27:0/266024196 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb8c003b60 con 0x7fbb9c107e20 2024-04-25T00:06:01.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.991+0000 7fbba1a7d700 1 -- 172.21.15.27:0/266024196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb9c19c4d0 con 0x7fbb9c107e20 2024-04-25T00:06:01.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.991+0000 7fbba1a7d700 1 -- 172.21.15.27:0/266024196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb9c19c930 con 0x7fbb9c107e20 2024-04-25T00:06:01.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.991+0000 7fbb98ff9700 1 -- 172.21.15.27:0/266024196 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb8c004300 con 0x7fbb9c107e20 2024-04-25T00:06:01.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.991+0000 7fbb98ff9700 1 -- 172.21.15.27:0/266024196 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb8c024430 con 0x7fbb9c107e20 2024-04-25T00:06:01.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.992+0000 7fbba1a7d700 1 -- 172.21.15.27:0/266024196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb88005360 con 0x7fbb9c107e20 2024-04-25T00:06:01.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.992+0000 7fbb98ff9700 1 -- 172.21.15.27:0/266024196 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fbb8c022020 con 0x7fbb9c107e20 2024-04-25T00:06:01.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.992+0000 7fbb98ff9700 1 --2- 172.21.15.27:0/266024196 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fbb84039050 0x7fbb8403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:01.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.992+0000 7fbb98ff9700 1 -- 172.21.15.27:0/266024196 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbb8c04d230 con 0x7fbb9c107e20 2024-04-25T00:06:01.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.992+0000 7fbb9affd700 1 --2- 172.21.15.27:0/266024196 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fbb84039050 0x7fbb8403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:01.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.992+0000 7fbb9affd700 1 --2- 172.21.15.27:0/266024196 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fbb84039050 0x7fbb8403b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbb90006fd0 tx=0x7fbb90006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:01.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:05:59.998+0000 7fbb98ff9700 1 -- 172.21.15.27:0/266024196 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbb8c0172b0 con 0x7fbb9c107e20 2024-04-25T00:06:01.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:00.211+0000 7fbba1a7d700 1 -- 172.21.15.27:0/266024196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fbb88005650 con 0x7fbb9c107e20 2024-04-25T00:06:01.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:00.214+0000 7fbb98ff9700 1 -- 172.21.15.27:0/266024196 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fbb8c00fdd0 con 0x7fbb9c107e20 2024-04-25T00:06:01.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:00.216+0000 7fbba1a7d700 1 -- 172.21.15.27:0/266024196 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fbb84039050 msgr2=0x7fbb8403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:01.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:00.216+0000 7fbba1a7d700 1 --2- 172.21.15.27:0/266024196 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fbb84039050 0x7fbb8403b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbb90006fd0 tx=0x7fbb90006e40 comp rx=0 tx=0).stop 2024-04-25T00:06:01.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:00.216+0000 7fbba1a7d700 1 -- 172.21.15.27:0/266024196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb9c107e20 msgr2=0x7fbb9c19bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:01.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:00.216+0000 7fbba1a7d700 1 --2- 172.21.15.27:0/266024196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb9c107e20 0x7fbb9c19bd00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c011b60 tx=0x7fbb8c011c40 comp rx=0 tx=0).stop 2024-04-25T00:06:01.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:00.216+0000 7fbba1a7d700 1 -- 172.21.15.27:0/266024196 shutdown_connections 2024-04-25T00:06:01.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:00.216+0000 7fbba1a7d700 1 --2- 172.21.15.27:0/266024196 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fbb84039050 0x7fbb8403b510 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:01.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:00.216+0000 7fbba1a7d700 1 --2- 172.21.15.27:0/266024196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb9c107e20 0x7fbb9c19bd00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:01.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:00.216+0000 7fbba1a7d700 1 -- 172.21.15.27:0/266024196 >> 172.21.15.27:0/266024196 conn(0x7fbb9c1033e0 msgr2=0x7fbb9c103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:01.352 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:00.216+0000 7fbba1a7d700 1 -- 172.21.15.27:0/266024196 shutdown_connections 2024-04-25T00:06:01.352 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:00.217+0000 7fbba1a7d700 1 -- 172.21.15.27:0/266024196 wait complete. 2024-04-25T00:06:01.352 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling the dashboard module... 2024-04-25T00:06:02.546 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:02 smithi027 ceph-mon[29208]: from='mgr.14118 172.21.15.27:0/3113733454' entity='mgr.x' 2024-04-25T00:06:02.546 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:02 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2192474922' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-25T00:06:02.546 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:02 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: ignoring --setuser ceph since I am not root 2024-04-25T00:06:02.546 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:02 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: ignoring --setgroup ceph since I am not root 2024-04-25T00:06:02.546 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:02 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:02.504+0000 7f6a05afb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T00:06:02.811 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:02 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:02.702+0000 7f6a05afb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:06:02.863 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.766+0000 7fb0076b8700 1 Processor -- start 2024-04-25T00:06:02.863 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.766+0000 7fb0076b8700 1 -- start start 2024-04-25T00:06:02.863 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.766+0000 7fb0076b8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001088d0 0x7fb00010acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:02.863 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.766+0000 7fb0076b8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb00007be30 con 0x7fb0001088d0 2024-04-25T00:06:02.863 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.766+0000 7fb005454700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001088d0 0x7fb00010acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:02.864 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.767+0000 7fb005454700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001088d0 0x7fb00010acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42024/0 (socket says 172.21.15.27:42024) 2024-04-25T00:06:02.864 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.767+0000 7fb005454700 1 -- 172.21.15.27:0/1557113129 learned_addr learned my addr 172.21.15.27:0/1557113129 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:02.864 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.767+0000 7fb005454700 1 -- 172.21.15.27:0/1557113129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb00010b200 con 0x7fb0001088d0 2024-04-25T00:06:02.864 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.767+0000 7fb005454700 1 --2- 172.21.15.27:0/1557113129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001088d0 0x7fb00010acc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faffc0098a0 tx=0x7faffc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d0bc5852b7c319a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:02.864 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.767+0000 7faff7fff700 1 -- 172.21.15.27:0/1557113129 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faffc004160 con 0x7fb0001088d0 2024-04-25T00:06:02.864 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.767+0000 7faff7fff700 1 -- 172.21.15.27:0/1557113129 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faffc004300 con 0x7fb0001088d0 2024-04-25T00:06:02.864 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.767+0000 7faff7fff700 1 -- 172.21.15.27:0/1557113129 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faffc011b20 con 0x7fb0001088d0 2024-04-25T00:06:02.864 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.768+0000 7fb0076b8700 1 -- 172.21.15.27:0/1557113129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001088d0 msgr2=0x7fb00010acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:02.864 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.768+0000 7fb0076b8700 1 --2- 172.21.15.27:0/1557113129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001088d0 0x7fb00010acc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faffc0098a0 tx=0x7faffc009bb0 comp rx=0 tx=0).stop 2024-04-25T00:06:02.864 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.768+0000 7fb0076b8700 1 -- 172.21.15.27:0/1557113129 shutdown_connections 2024-04-25T00:06:02.864 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.768+0000 7fb0076b8700 1 --2- 172.21.15.27:0/1557113129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001088d0 0x7fb00010acc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:02.864 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.768+0000 7fb0076b8700 1 -- 172.21.15.27:0/1557113129 >> 172.21.15.27:0/1557113129 conn(0x7fb0001022c0 msgr2=0x7fb000104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:02.864 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.768+0000 7fb0076b8700 1 -- 172.21.15.27:0/1557113129 shutdown_connections 2024-04-25T00:06:02.864 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.768+0000 7fb0076b8700 1 -- 172.21.15.27:0/1557113129 wait complete. 2024-04-25T00:06:02.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.769+0000 7fb0076b8700 1 Processor -- start 2024-04-25T00:06:02.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.769+0000 7fb0076b8700 1 -- start start 2024-04-25T00:06:02.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.769+0000 7fb0076b8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001088d0 0x7fb000108150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:02.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.769+0000 7fb0076b8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb00010c260 con 0x7fb0001088d0 2024-04-25T00:06:02.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.769+0000 7fb005454700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001088d0 0x7fb000108150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:02.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.769+0000 7fb005454700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001088d0 0x7fb000108150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42038/0 (socket says 172.21.15.27:42038) 2024-04-25T00:06:02.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.769+0000 7fb005454700 1 -- 172.21.15.27:0/2192474922 learned_addr learned my addr 172.21.15.27:0/2192474922 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:02.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.769+0000 7fb005454700 1 -- 172.21.15.27:0/2192474922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb000108690 con 0x7fb0001088d0 2024-04-25T00:06:02.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.770+0000 7fb005454700 1 --2- 172.21.15.27:0/2192474922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001088d0 0x7fb000108150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faffc0116a0 tx=0x7faffc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:02.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.770+0000 7faff67fc700 1 -- 172.21.15.27:0/2192474922 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faffc003a40 con 0x7fb0001088d0 2024-04-25T00:06:02.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.770+0000 7faff67fc700 1 -- 172.21.15.27:0/2192474922 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faffc003be0 con 0x7fb0001088d0 2024-04-25T00:06:02.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.770+0000 7fb0076b8700 1 -- 172.21.15.27:0/2192474922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb000106880 con 0x7fb0001088d0 2024-04-25T00:06:02.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.770+0000 7faff67fc700 1 -- 172.21.15.27:0/2192474922 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faffc01a750 con 0x7fb0001088d0 2024-04-25T00:06:02.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.770+0000 7fb0076b8700 1 -- 172.21.15.27:0/2192474922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb000106d40 con 0x7fb0001088d0 2024-04-25T00:06:02.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.771+0000 7faff67fc700 1 -- 172.21.15.27:0/2192474922 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7faffc01aa00 con 0x7fb0001088d0 2024-04-25T00:06:02.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.771+0000 7faff67fc700 1 --2- 172.21.15.27:0/2192474922 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fafec039050 0x7fafec03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:02.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.771+0000 7faff67fc700 1 -- 172.21.15.27:0/2192474922 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7faffc04e4f0 con 0x7fb0001088d0 2024-04-25T00:06:02.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.771+0000 7fb004c53700 1 --2- 172.21.15.27:0/2192474922 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fafec039050 0x7fafec03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:02.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.771+0000 7fb0076b8700 1 -- 172.21.15.27:0/2192474922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb000063890 con 0x7fb0001088d0 2024-04-25T00:06:02.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.772+0000 7fb004c53700 1 --2- 172.21.15.27:0/2192474922 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fafec039050 0x7fafec03b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faff0006fd0 tx=0x7faff0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:02.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:01.776+0000 7faff67fc700 1 -- 172.21.15.27:0/2192474922 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faffc011e20 con 0x7fb0001088d0 2024-04-25T00:06:02.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:02.048+0000 7fb0076b8700 1 -- 172.21.15.27:0/2192474922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fb0001073c0 con 0x7fb0001088d0 2024-04-25T00:06:02.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:02.287+0000 7faff67fc700 1 -- 172.21.15.27:0/2192474922 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7faffc020240 con 0x7fb0001088d0 2024-04-25T00:06:02.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:02.289+0000 7faff67fc700 1 -- 172.21.15.27:0/2192474922 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7faffc02d910 con 0x7fb0001088d0 2024-04-25T00:06:02.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:02.291+0000 7fb0076b8700 1 -- 172.21.15.27:0/2192474922 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fafec039050 msgr2=0x7fafec03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:02.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:02.291+0000 7fb0076b8700 1 --2- 172.21.15.27:0/2192474922 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fafec039050 0x7fafec03b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faff0006fd0 tx=0x7faff0006e40 comp rx=0 tx=0).stop 2024-04-25T00:06:02.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:02.291+0000 7fb0076b8700 1 -- 172.21.15.27:0/2192474922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001088d0 msgr2=0x7fb000108150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:02.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:02.291+0000 7fb0076b8700 1 --2- 172.21.15.27:0/2192474922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001088d0 0x7fb000108150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faffc0116a0 tx=0x7faffc011ba0 comp rx=0 tx=0).stop 2024-04-25T00:06:02.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:02.291+0000 7fb0076b8700 1 -- 172.21.15.27:0/2192474922 shutdown_connections 2024-04-25T00:06:02.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:02.291+0000 7fb0076b8700 1 --2- 172.21.15.27:0/2192474922 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fafec039050 0x7fafec03b510 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:02.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:02.291+0000 7fb0076b8700 1 --2- 172.21.15.27:0/2192474922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001088d0 0x7fb000108150 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:02.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:02.291+0000 7fb0076b8700 1 -- 172.21.15.27:0/2192474922 >> 172.21.15.27:0/2192474922 conn(0x7fb0001022c0 msgr2=0x7fb000102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:02.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:02.291+0000 7fb0076b8700 1 -- 172.21.15.27:0/2192474922 shutdown_connections 2024-04-25T00:06:02.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:02.291+0000 7fb0076b8700 1 -- 172.21.15.27:0/2192474922 wait complete. 2024-04-25T00:06:03.134 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:02 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:02.810+0000 7f6a05afb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T00:06:03.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:03 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2192474922' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-25T00:06:03.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:03 smithi027 ceph-mon[29208]: mgrmap e9: x(active, since 19s) 2024-04-25T00:06:03.442 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:03 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:03.142+0000 7f6a05afb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T00:06:03.442 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:03 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:03.230+0000 7f6a05afb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T00:06:04.272 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T00:06:04.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-25T00:06:04.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T00:06:04.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T00:06:04.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T00:06:04.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T00:06:04.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.354+0000 7fc14f85f700 1 Processor -- start 2024-04-25T00:06:04.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.354+0000 7fc14f85f700 1 -- start start 2024-04-25T00:06:04.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.354+0000 7fc14f85f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc148106d90 0x7fc1481071b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:04.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.354+0000 7fc14f85f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc148107780 con 0x7fc148106d90 2024-04-25T00:06:04.279 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.354+0000 7fc14d5fb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc148106d90 0x7fc1481071b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:04.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.354+0000 7fc14d5fb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc148106d90 0x7fc1481071b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42066/0 (socket says 172.21.15.27:42066) 2024-04-25T00:06:04.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.354+0000 7fc14d5fb700 1 -- 172.21.15.27:0/3642040616 learned_addr learned my addr 172.21.15.27:0/3642040616 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:04.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.355+0000 7fc14d5fb700 1 -- 172.21.15.27:0/3642040616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc148107f90 con 0x7fc148106d90 2024-04-25T00:06:04.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.355+0000 7fc14d5fb700 1 --2- 172.21.15.27:0/3642040616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc148106d90 0x7fc1481071b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc1440098a0 tx=0x7fc144009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0cb4bfaa2a8ef0a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:04.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.355+0000 7fc13ffff700 1 -- 172.21.15.27:0/3642040616 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc144004160 con 0x7fc148106d90 2024-04-25T00:06:04.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.355+0000 7fc13ffff700 1 -- 172.21.15.27:0/3642040616 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc144004300 con 0x7fc148106d90 2024-04-25T00:06:04.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.355+0000 7fc13ffff700 1 -- 172.21.15.27:0/3642040616 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc144011b20 con 0x7fc148106d90 2024-04-25T00:06:04.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.356+0000 7fc14f85f700 1 -- 172.21.15.27:0/3642040616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc148106d90 msgr2=0x7fc1481071b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:04.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.356+0000 7fc14f85f700 1 --2- 172.21.15.27:0/3642040616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc148106d90 0x7fc1481071b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc1440098a0 tx=0x7fc144009bb0 comp rx=0 tx=0).stop 2024-04-25T00:06:04.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.356+0000 7fc14f85f700 1 -- 172.21.15.27:0/3642040616 shutdown_connections 2024-04-25T00:06:04.280 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.356+0000 7fc14f85f700 1 --2- 172.21.15.27:0/3642040616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc148106d90 0x7fc1481071b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:04.281 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.356+0000 7fc14f85f700 1 -- 172.21.15.27:0/3642040616 >> 172.21.15.27:0/3642040616 conn(0x7fc148102370 msgr2=0x7fc148104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:04.281 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.356+0000 7fc14f85f700 1 -- 172.21.15.27:0/3642040616 shutdown_connections 2024-04-25T00:06:04.281 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.356+0000 7fc14f85f700 1 -- 172.21.15.27:0/3642040616 wait complete. 2024-04-25T00:06:04.281 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.357+0000 7fc14f85f700 1 Processor -- start 2024-04-25T00:06:04.281 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.357+0000 7fc14f85f700 1 -- start start 2024-04-25T00:06:04.281 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.357+0000 7fc14f85f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc148106d90 0x7fc14819be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:04.281 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.357+0000 7fc14f85f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc14810b340 con 0x7fc148106d90 2024-04-25T00:06:04.281 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.357+0000 7fc14d5fb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc148106d90 0x7fc14819be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:04.281 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.357+0000 7fc14d5fb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc148106d90 0x7fc14819be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42080/0 (socket says 172.21.15.27:42080) 2024-04-25T00:06:04.281 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.357+0000 7fc14d5fb700 1 -- 172.21.15.27:0/2169800591 learned_addr learned my addr 172.21.15.27:0/2169800591 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:04.281 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.358+0000 7fc14d5fb700 1 -- 172.21.15.27:0/2169800591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14819c340 con 0x7fc148106d90 2024-04-25T00:06:04.282 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.358+0000 7fc14d5fb700 1 --2- 172.21.15.27:0/2169800591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc148106d90 0x7fc14819be00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc144000c00 tx=0x7fc144011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:04.282 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.358+0000 7fc13e7fc700 1 -- 172.21.15.27:0/2169800591 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc144003b60 con 0x7fc148106d90 2024-04-25T00:06:04.282 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.358+0000 7fc13e7fc700 1 -- 172.21.15.27:0/2169800591 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc14401a910 con 0x7fc148106d90 2024-04-25T00:06:04.282 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.358+0000 7fc14f85f700 1 -- 172.21.15.27:0/2169800591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc14819c5d0 con 0x7fc148106d90 2024-04-25T00:06:04.282 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.358+0000 7fc13e7fc700 1 -- 172.21.15.27:0/2169800591 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc144024930 con 0x7fc148106d90 2024-04-25T00:06:04.282 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.358+0000 7fc14f85f700 1 -- 172.21.15.27:0/2169800591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc14819ca30 con 0x7fc148106d90 2024-04-25T00:06:04.282 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.359+0000 7fc13e7fc700 1 -- 172.21.15.27:0/2169800591 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fc144022020 con 0x7fc148106d90 2024-04-25T00:06:04.282 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.359+0000 7fc14f85f700 1 -- 172.21.15.27:0/2169800591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc148063890 con 0x7fc148106d90 2024-04-25T00:06:04.282 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.359+0000 7fc13e7fc700 1 --2- 172.21.15.27:0/2169800591 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fc1340390a0 0x7fc13403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:04.282 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.360+0000 7fc13e7fc700 1 -- 172.21.15.27:0/2169800591 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc14404d2b0 con 0x7fc148106d90 2024-04-25T00:06:04.282 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.360+0000 7fc14cdfa700 1 -- 172.21.15.27:0/2169800591 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fc1340390a0 msgr2=0x7fc13403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1186428290 2024-04-25T00:06:04.283 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.360+0000 7fc14cdfa700 1 --2- 172.21.15.27:0/2169800591 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fc1340390a0 0x7fc13403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T00:06:04.283 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.364+0000 7fc13e7fc700 1 -- 172.21.15.27:0/2169800591 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc14401ab70 con 0x7fc148106d90 2024-04-25T00:06:04.283 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.561+0000 7fc14cdfa700 1 -- 172.21.15.27:0/2169800591 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fc1340390a0 msgr2=0x7fc13403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1186428290 2024-04-25T00:06:04.283 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.561+0000 7fc14cdfa700 1 --2- 172.21.15.27:0/2169800591 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fc1340390a0 0x7fc13403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T00:06:04.283 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.660+0000 7fc14f85f700 1 -- 172.21.15.27:0/2169800591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fc148075490 con 0x7fc148106d90 2024-04-25T00:06:04.283 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.661+0000 7fc13e7fc700 1 -- 172.21.15.27:0/2169800591 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fc14402d3f0 con 0x7fc148106d90 2024-04-25T00:06:04.283 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.665+0000 7fc14f85f700 1 -- 172.21.15.27:0/2169800591 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fc1340390a0 msgr2=0x7fc13403b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:06:04.283 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.665+0000 7fc14f85f700 1 --2- 172.21.15.27:0/2169800591 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fc1340390a0 0x7fc13403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:04.283 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.665+0000 7fc14f85f700 1 -- 172.21.15.27:0/2169800591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc148106d90 msgr2=0x7fc14819be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:04.283 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.665+0000 7fc14f85f700 1 --2- 172.21.15.27:0/2169800591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc148106d90 0x7fc14819be00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc144000c00 tx=0x7fc144011c10 comp rx=0 tx=0).stop 2024-04-25T00:06:04.283 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.665+0000 7fc14f85f700 1 -- 172.21.15.27:0/2169800591 shutdown_connections 2024-04-25T00:06:04.283 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.665+0000 7fc14f85f700 1 --2- 172.21.15.27:0/2169800591 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7fc1340390a0 0x7fc13403b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:04.283 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.665+0000 7fc14f85f700 1 --2- 172.21.15.27:0/2169800591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc148106d90 0x7fc14819be00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:04.283 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.665+0000 7fc14f85f700 1 -- 172.21.15.27:0/2169800591 >> 172.21.15.27:0/2169800591 conn(0x7fc148102370 msgr2=0x7fc1481031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:04.284 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.665+0000 7fc14f85f700 1 -- 172.21.15.27:0/2169800591 shutdown_connections 2024-04-25T00:06:04.284 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:03.665+0000 7fc14f85f700 1 -- 172.21.15.27:0/2169800591 wait complete. 2024-04-25T00:06:04.284 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for the mgr to restart... 2024-04-25T00:06:04.284 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr epoch 9... 2024-04-25T00:06:04.544 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:04 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2169800591' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T00:06:04.544 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:04 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:04.511+0000 7f6a05afb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T00:06:04.873 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:04 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:04.694+0000 7f6a05afb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T00:06:04.873 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:04 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:04.871+0000 7f6a05afb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T00:06:05.442 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:05 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:05.170+0000 7f6a05afb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T00:06:05.442 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:05 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:05.266+0000 7f6a05afb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T00:06:05.942 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:05 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:05.479+0000 7f6a05afb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T00:06:05.942 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:05 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:05.678+0000 7f6a05afb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:06:06.442 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:06 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:06.301+0000 7f6a05afb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T00:06:07.054 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:06 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:06.742+0000 7f6a05afb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T00:06:07.054 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:06 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:06.843+0000 7f6a05afb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T00:06:07.055 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:07 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:07.053+0000 7f6a05afb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T00:06:07.942 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:07 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:07.614+0000 7f6a05afb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T00:06:07.942 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:07 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:07.709+0000 7f6a05afb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T00:06:08.319 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:07 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:07.962+0000 7f6a05afb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T00:06:08.320 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:08 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:08.100+0000 7f6a05afb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T00:06:08.320 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:08 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:08.199+0000 7f6a05afb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T00:06:08.320 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:08 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:08.318+0000 7f6a05afb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T00:06:08.692 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:08 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:08.414+0000 7f6a05afb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T00:06:09.536 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:09 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:09.257+0000 7f6a05afb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T00:06:09.536 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:06:09 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:06:09.482+0000 7f6a05afb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T00:06:09.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:09 smithi027 ceph-mon[29208]: Active manager daemon x restarted 2024-04-25T00:06:09.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:09 smithi027 ceph-mon[29208]: Activating manager daemon x 2024-04-25T00:06:09.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:09 smithi027 ceph-mon[29208]: osdmap e3: 0 total, 0 up, 0 in 2024-04-25T00:06:09.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:09 smithi027 ceph-mon[29208]: mgrmap e10: x(active, starting, since 0.00321289s) 2024-04-25T00:06:09.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:09 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:06:09.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:09 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T00:06:09.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:09 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T00:06:09.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:09 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T00:06:09.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:09 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T00:06:09.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:09 smithi027 ceph-mon[29208]: Manager daemon x is now available 2024-04-25T00:06:09.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:09 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:06:09.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:09 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T00:06:10.827 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T00:06:10.827 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:10 smithi027 ceph-mon[29208]: [25/Apr/2024:00:06:09] ENGINE Bus STARTING 2024-04-25T00:06:10.827 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:10 smithi027 ceph-mon[29208]: [25/Apr/2024:00:06:10] ENGINE Serving on https://172.21.15.27:7150 2024-04-25T00:06:10.827 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:10 smithi027 ceph-mon[29208]: [25/Apr/2024:00:06:10] ENGINE Serving on http://172.21.15.27:8765 2024-04-25T00:06:10.828 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:10 smithi027 ceph-mon[29208]: [25/Apr/2024:00:06:10] ENGINE Bus STARTED 2024-04-25T00:06:10.828 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:10 smithi027 ceph-mon[29208]: mgrmap e11: x(active, since 1.00571s) 2024-04-25T00:06:11.114 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T00:06:11.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-25T00:06:11.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T00:06:11.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T00:06:11.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.717+0000 7f790ed8c700 1 Processor -- start 2024-04-25T00:06:11.125 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.717+0000 7f790ed8c700 1 -- start start 2024-04-25T00:06:11.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.717+0000 7f790ed8c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908107e80 0x7f79081082a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:11.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.717+0000 7f790ed8c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f79081087e0 con 0x7f7908107e80 2024-04-25T00:06:11.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.718+0000 7f790cb28700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908107e80 0x7f79081082a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:11.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.718+0000 7f790cb28700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908107e80 0x7f79081082a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42084/0 (socket says 172.21.15.27:42084) 2024-04-25T00:06:11.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.718+0000 7f790cb28700 1 -- 172.21.15.27:0/344712510 learned_addr learned my addr 172.21.15.27:0/344712510 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:11.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.718+0000 7f790cb28700 1 -- 172.21.15.27:0/344712510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7908108f90 con 0x7f7908107e80 2024-04-25T00:06:11.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.718+0000 7f790cb28700 1 --2- 172.21.15.27:0/344712510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908107e80 0x7f79081082a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f78fc0098a0 tx=0x7f78fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56bd2e5f616ba5ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:11.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.718+0000 7f79077fe700 1 -- 172.21.15.27:0/344712510 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78fc004160 con 0x7f7908107e80 2024-04-25T00:06:11.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.718+0000 7f79077fe700 1 -- 172.21.15.27:0/344712510 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78fc004300 con 0x7f7908107e80 2024-04-25T00:06:11.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.718+0000 7f79077fe700 1 -- 172.21.15.27:0/344712510 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78fc011b20 con 0x7f7908107e80 2024-04-25T00:06:11.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.719+0000 7f790ed8c700 1 -- 172.21.15.27:0/344712510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908107e80 msgr2=0x7f79081082a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:11.126 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.719+0000 7f790ed8c700 1 --2- 172.21.15.27:0/344712510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908107e80 0x7f79081082a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f78fc0098a0 tx=0x7f78fc009bb0 comp rx=0 tx=0).stop 2024-04-25T00:06:11.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.719+0000 7f790ed8c700 1 -- 172.21.15.27:0/344712510 shutdown_connections 2024-04-25T00:06:11.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.719+0000 7f790ed8c700 1 --2- 172.21.15.27:0/344712510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908107e80 0x7f79081082a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:11.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.719+0000 7f790ed8c700 1 -- 172.21.15.27:0/344712510 >> 172.21.15.27:0/344712510 conn(0x7f7908103420 msgr2=0x7f7908105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:11.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.719+0000 7f790ed8c700 1 -- 172.21.15.27:0/344712510 shutdown_connections 2024-04-25T00:06:11.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.719+0000 7f790ed8c700 1 -- 172.21.15.27:0/344712510 wait complete. 2024-04-25T00:06:11.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.720+0000 7f790ed8c700 1 Processor -- start 2024-04-25T00:06:11.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.720+0000 7f790ed8c700 1 -- start start 2024-04-25T00:06:11.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.720+0000 7f790ed8c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908107e80 0x7f790819bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:11.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.720+0000 7f790ed8c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f790810c3d0 con 0x7f7908107e80 2024-04-25T00:06:11.127 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.720+0000 7f790cb28700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908107e80 0x7f790819bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:11.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.720+0000 7f790cb28700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908107e80 0x7f790819bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42096/0 (socket says 172.21.15.27:42096) 2024-04-25T00:06:11.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.720+0000 7f790cb28700 1 -- 172.21.15.27:0/2177846486 learned_addr learned my addr 172.21.15.27:0/2177846486 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:11.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.720+0000 7f790cb28700 1 -- 172.21.15.27:0/2177846486 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f790819c260 con 0x7f7908107e80 2024-04-25T00:06:11.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.721+0000 7f790cb28700 1 --2- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908107e80 0x7f790819bd20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f78fc011b60 tx=0x7f78fc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:11.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.721+0000 7f7905ffb700 1 -- 172.21.15.27:0/2177846486 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78fc003b60 con 0x7f7908107e80 2024-04-25T00:06:11.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.721+0000 7f7905ffb700 1 -- 172.21.15.27:0/2177846486 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78fc01a910 con 0x7f7908107e80 2024-04-25T00:06:11.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.721+0000 7f790ed8c700 1 -- 172.21.15.27:0/2177846486 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f790819c4f0 con 0x7f7908107e80 2024-04-25T00:06:11.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.721+0000 7f7905ffb700 1 -- 172.21.15.27:0/2177846486 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78fc024930 con 0x7f7908107e80 2024-04-25T00:06:11.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.721+0000 7f790ed8c700 1 -- 172.21.15.27:0/2177846486 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f790819c950 con 0x7f7908107e80 2024-04-25T00:06:11.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.722+0000 7f7905ffb700 1 -- 172.21.15.27:0/2177846486 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f78fc022020 con 0x7f7908107e80 2024-04-25T00:06:11.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.722+0000 7f7905ffb700 1 --2- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f78f80390a0 0x7f78f803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:11.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.722+0000 7f7905ffb700 1 -- 172.21.15.27:0/2177846486 --> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f78f803bc70 con 0x7f78f80390a0 2024-04-25T00:06:11.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.722+0000 7f7907fff700 1 -- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f78f80390a0 msgr2=0x7f78f803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1186428290 2024-04-25T00:06:11.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.722+0000 7f7905ffb700 1 -- 172.21.15.27:0/2177846486 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f78fc04d270 con 0x7f7908107e80 2024-04-25T00:06:11.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.722+0000 7f7907fff700 1 --2- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f78f80390a0 0x7f78f803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T00:06:11.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.923+0000 7f7907fff700 1 -- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f78f80390a0 msgr2=0x7f78f803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1186428290 2024-04-25T00:06:11.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:04.923+0000 7f7907fff700 1 --2- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f78f80390a0 0x7f78f803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T00:06:11.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:05.324+0000 7f7907fff700 1 -- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f78f80390a0 msgr2=0x7f78f803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1186428290 2024-04-25T00:06:11.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:05.324+0000 7f7907fff700 1 --2- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f78f80390a0 0x7f78f803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T00:06:11.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:06.125+0000 7f7907fff700 1 -- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f78f80390a0 msgr2=0x7f78f803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1186428290 2024-04-25T00:06:11.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:06.125+0000 7f7907fff700 1 --2- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f78f80390a0 0x7f78f803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T00:06:11.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:07.726+0000 7f7907fff700 1 -- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f78f80390a0 msgr2=0x7f78f803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1186428290 2024-04-25T00:06:11.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:07.726+0000 7f7907fff700 1 --2- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f78f80390a0 0x7f78f803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T00:06:11.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:09.485+0000 7f7905ffb700 1 -- 172.21.15.27:0/2177846486 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7f78fc01a430 con 0x7f7908107e80 2024-04-25T00:06:11.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:09.485+0000 7f7905ffb700 1 -- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f78f80390a0 msgr2=0x7f78f803b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:06:11.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:09.485+0000 7f7905ffb700 1 --2- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f78f80390a0 0x7f78f803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:11.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.487+0000 7f7905ffb700 1 -- 172.21.15.27:0/2177846486 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f78fc04efc0 con 0x7f7908107e80 2024-04-25T00:06:11.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.487+0000 7f7905ffb700 1 --2- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78f803cb60 0x7f78f803ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:11.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.487+0000 7f7905ffb700 1 -- 172.21.15.27:0/2177846486 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f78fc001d30 con 0x7f78f803cb60 2024-04-25T00:06:11.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.487+0000 7f7907fff700 1 --2- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78f803cb60 0x7f78f803ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:11.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.488+0000 7f7907fff700 1 --2- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78f803cb60 0x7f78f803ef50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f78f4003a10 tx=0x7f78f40092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:11.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.489+0000 7f7905ffb700 1 -- 172.21.15.27:0/2177846486 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f78fc001d30 con 0x7f78f803cb60 2024-04-25T00:06:11.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.495+0000 7f790ed8c700 1 -- 172.21.15.27:0/2177846486 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f790810c550 con 0x7f78f803cb60 2024-04-25T00:06:11.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.496+0000 7f7905ffb700 1 -- 172.21.15.27:0/2177846486 <== mgr.14150 v2:172.21.15.27:6800/2161554778 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f790810c550 con 0x7f78f803cb60 2024-04-25T00:06:11.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.496+0000 7f790ed8c700 1 -- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78f803cb60 msgr2=0x7f78f803ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:11.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.496+0000 7f790ed8c700 1 --2- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78f803cb60 0x7f78f803ef50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f78f4003a10 tx=0x7f78f40092b0 comp rx=0 tx=0).stop 2024-04-25T00:06:11.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.496+0000 7f790ed8c700 1 -- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908107e80 msgr2=0x7f790819bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:11.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.496+0000 7f790ed8c700 1 --2- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908107e80 0x7f790819bd20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f78fc011b60 tx=0x7f78fc011c40 comp rx=0 tx=0).stop 2024-04-25T00:06:11.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.496+0000 7f790ed8c700 1 -- 172.21.15.27:0/2177846486 shutdown_connections 2024-04-25T00:06:11.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.496+0000 7f790ed8c700 1 --2- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78f803cb60 0x7f78f803ef50 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:11.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.496+0000 7f790ed8c700 1 --2- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908107e80 0x7f790819bd20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:11.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.496+0000 7f790ed8c700 1 --2- 172.21.15.27:0/2177846486 >> [v2:172.21.15.27:6800/1186428290,v1:172.21.15.27:6801/1186428290] conn(0x7f78f80390a0 0x7f78f803b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:11.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.496+0000 7f790ed8c700 1 -- 172.21.15.27:0/2177846486 >> 172.21.15.27:0/2177846486 conn(0x7f7908103420 msgr2=0x7f7908103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:11.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.496+0000 7f790ed8c700 1 -- 172.21.15.27:0/2177846486 shutdown_connections 2024-04-25T00:06:11.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:10.496+0000 7f790ed8c700 1 -- 172.21.15.27:0/2177846486 wait complete. 2024-04-25T00:06:11.131 INFO:teuthology.orchestra.run.smithi027.stdout:mgr epoch 9 is available 2024-04-25T00:06:11.132 INFO:teuthology.orchestra.run.smithi027.stdout:Generating a dashboard self-signed certificate... 2024-04-25T00:06:11.825 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:11 smithi027 ceph-mon[29208]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T00:06:11.825 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:11 smithi027 ceph-mon[29208]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T00:06:12.736 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:12 smithi027 ceph-mon[29208]: mgrmap e12: x(active, since 2s) 2024-04-25T00:06:12.736 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:12 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:12.736 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:12 smithi027 ceph-mon[29208]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:06:12.736 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:12 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:12.736 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:12 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:12.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-25T00:06:12.745 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.824+0000 7fd28ecf1700 1 Processor -- start 2024-04-25T00:06:12.745 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.824+0000 7fd28ecf1700 1 -- start start 2024-04-25T00:06:12.745 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.824+0000 7fd28ecf1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd288106d50 0x7fd288107170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:12.745 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.824+0000 7fd28ecf1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd288107740 con 0x7fd288106d50 2024-04-25T00:06:12.745 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.824+0000 7fd28ca8d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd288106d50 0x7fd288107170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:12.745 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.824+0000 7fd28ca8d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd288106d50 0x7fd288107170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52566/0 (socket says 172.21.15.27:52566) 2024-04-25T00:06:12.745 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.824+0000 7fd28ca8d700 1 -- 172.21.15.27:0/3831770264 learned_addr learned my addr 172.21.15.27:0/3831770264 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:12.745 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.825+0000 7fd28ca8d700 1 -- 172.21.15.27:0/3831770264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd288107f50 con 0x7fd288106d50 2024-04-25T00:06:12.745 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.825+0000 7fd28ca8d700 1 --2- 172.21.15.27:0/3831770264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd288106d50 0x7fd288107170 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd27c0098a0 tx=0x7fd27c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1b8b42563d2089 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:12.745 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.825+0000 7fd2877fe700 1 -- 172.21.15.27:0/3831770264 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd27c004160 con 0x7fd288106d50 2024-04-25T00:06:12.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.825+0000 7fd2877fe700 1 -- 172.21.15.27:0/3831770264 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd27c004300 con 0x7fd288106d50 2024-04-25T00:06:12.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.825+0000 7fd2877fe700 1 -- 172.21.15.27:0/3831770264 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd27c011b20 con 0x7fd288106d50 2024-04-25T00:06:12.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.826+0000 7fd28ecf1700 1 -- 172.21.15.27:0/3831770264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd288106d50 msgr2=0x7fd288107170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:12.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.826+0000 7fd28ecf1700 1 --2- 172.21.15.27:0/3831770264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd288106d50 0x7fd288107170 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd27c0098a0 tx=0x7fd27c009bb0 comp rx=0 tx=0).stop 2024-04-25T00:06:12.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.826+0000 7fd28ecf1700 1 -- 172.21.15.27:0/3831770264 shutdown_connections 2024-04-25T00:06:12.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.826+0000 7fd28ecf1700 1 --2- 172.21.15.27:0/3831770264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd288106d50 0x7fd288107170 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:12.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.826+0000 7fd28ecf1700 1 -- 172.21.15.27:0/3831770264 >> 172.21.15.27:0/3831770264 conn(0x7fd2881022d0 msgr2=0x7fd288104730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:12.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.826+0000 7fd28ecf1700 1 -- 172.21.15.27:0/3831770264 shutdown_connections 2024-04-25T00:06:12.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.826+0000 7fd28ecf1700 1 -- 172.21.15.27:0/3831770264 wait complete. 2024-04-25T00:06:12.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.826+0000 7fd28ecf1700 1 Processor -- start 2024-04-25T00:06:12.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.826+0000 7fd28ecf1700 1 -- start start 2024-04-25T00:06:12.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.826+0000 7fd28ecf1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd288106d50 0x7fd28819bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:12.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.826+0000 7fd28ecf1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd28807af90 con 0x7fd288106d50 2024-04-25T00:06:12.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.827+0000 7fd28ca8d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd288106d50 0x7fd28819bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:12.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.827+0000 7fd28ca8d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd288106d50 0x7fd28819bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52576/0 (socket says 172.21.15.27:52576) 2024-04-25T00:06:12.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.827+0000 7fd28ca8d700 1 -- 172.21.15.27:0/3902798943 learned_addr learned my addr 172.21.15.27:0/3902798943 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:12.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.827+0000 7fd28ca8d700 1 -- 172.21.15.27:0/3902798943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28819c2f0 con 0x7fd288106d50 2024-04-25T00:06:12.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.827+0000 7fd28ca8d700 1 --2- 172.21.15.27:0/3902798943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd288106d50 0x7fd28819bdb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd27c000c00 tx=0x7fd27c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:12.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.827+0000 7fd285ffb700 1 -- 172.21.15.27:0/3902798943 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd27c003b60 con 0x7fd288106d50 2024-04-25T00:06:12.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.827+0000 7fd28ecf1700 1 -- 172.21.15.27:0/3902798943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd28819c580 con 0x7fd288106d50 2024-04-25T00:06:12.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.827+0000 7fd285ffb700 1 -- 172.21.15.27:0/3902798943 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd27c01a910 con 0x7fd288106d50 2024-04-25T00:06:12.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.827+0000 7fd285ffb700 1 -- 172.21.15.27:0/3902798943 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd27c024930 con 0x7fd288106d50 2024-04-25T00:06:12.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.827+0000 7fd28ecf1700 1 -- 172.21.15.27:0/3902798943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd28819f230 con 0x7fd288106d50 2024-04-25T00:06:12.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.828+0000 7fd285ffb700 1 -- 172.21.15.27:0/3902798943 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7fd27c022020 con 0x7fd288106d50 2024-04-25T00:06:12.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.828+0000 7fd285ffb700 1 --2- 172.21.15.27:0/3902798943 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd2780390a0 0x7fd27803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:12.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.828+0000 7fd28ecf1700 1 -- 172.21.15.27:0/3902798943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd288063890 con 0x7fd288106d50 2024-04-25T00:06:12.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.828+0000 7fd285ffb700 1 -- 172.21.15.27:0/3902798943 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd27c04d4b0 con 0x7fd288106d50 2024-04-25T00:06:12.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.829+0000 7fd287fff700 1 --2- 172.21.15.27:0/3902798943 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd2780390a0 0x7fd27803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:12.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.829+0000 7fd287fff700 1 --2- 172.21.15.27:0/3902798943 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd2780390a0 0x7fd27803b560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd274006fd0 tx=0x7fd274006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:12.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:11.834+0000 7fd285ffb700 1 -- 172.21.15.27:0/3902798943 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd27c00b5a0 con 0x7fd288106d50 2024-04-25T00:06:12.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:12.048+0000 7fd28ecf1700 1 -- 172.21.15.27:0/3902798943 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fd28802dda0 con 0x7fd2780390a0 2024-04-25T00:06:12.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:12.126+0000 7fd285ffb700 1 -- 172.21.15.27:0/3902798943 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fd28802dda0 con 0x7fd2780390a0 2024-04-25T00:06:12.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:12.128+0000 7fd28ecf1700 1 -- 172.21.15.27:0/3902798943 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd2780390a0 msgr2=0x7fd27803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:12.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:12.128+0000 7fd28ecf1700 1 --2- 172.21.15.27:0/3902798943 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd2780390a0 0x7fd27803b560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd274006fd0 tx=0x7fd274006e40 comp rx=0 tx=0).stop 2024-04-25T00:06:12.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:12.128+0000 7fd28ecf1700 1 -- 172.21.15.27:0/3902798943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd288106d50 msgr2=0x7fd28819bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:12.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:12.128+0000 7fd28ecf1700 1 --2- 172.21.15.27:0/3902798943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd288106d50 0x7fd28819bdb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd27c000c00 tx=0x7fd27c011c10 comp rx=0 tx=0).stop 2024-04-25T00:06:12.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:12.128+0000 7fd28ecf1700 1 -- 172.21.15.27:0/3902798943 shutdown_connections 2024-04-25T00:06:12.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:12.128+0000 7fd28ecf1700 1 --2- 172.21.15.27:0/3902798943 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd2780390a0 0x7fd27803b560 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:12.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:12.128+0000 7fd28ecf1700 1 --2- 172.21.15.27:0/3902798943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd288106d50 0x7fd28819bdb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:12.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:12.128+0000 7fd28ecf1700 1 -- 172.21.15.27:0/3902798943 >> 172.21.15.27:0/3902798943 conn(0x7fd2881022d0 msgr2=0x7fd288102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:12.749 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:12.128+0000 7fd28ecf1700 1 -- 172.21.15.27:0/3902798943 shutdown_connections 2024-04-25T00:06:12.749 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:12.128+0000 7fd28ecf1700 1 -- 172.21.15.27:0/3902798943 wait complete. 2024-04-25T00:06:12.749 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial admin user... 2024-04-25T00:06:14.353 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:14 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:14.353 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:14 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:14.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$8XU7OsvsL6DEeL9/hfMSte3c7SSULiWBz0g5NViEVLB6n8boJw6jm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714003573, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-25T00:06:14.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.205+0000 7fc30d9a1700 1 Processor -- start 2024-04-25T00:06:14.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.205+0000 7fc30d9a1700 1 -- start start 2024-04-25T00:06:14.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.205+0000 7fc30d9a1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308108940 0x7fc30810ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:14.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.205+0000 7fc30d9a1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc30807be30 con 0x7fc308108940 2024-04-25T00:06:14.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.205+0000 7fc306ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308108940 0x7fc30810ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:14.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.205+0000 7fc306ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308108940 0x7fc30810ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52582/0 (socket says 172.21.15.27:52582) 2024-04-25T00:06:14.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.205+0000 7fc306ffd700 1 -- 172.21.15.27:0/2667367879 learned_addr learned my addr 172.21.15.27:0/2667367879 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:14.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.205+0000 7fc306ffd700 1 -- 172.21.15.27:0/2667367879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30810b270 con 0x7fc308108940 2024-04-25T00:06:14.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.206+0000 7fc306ffd700 1 --2- 172.21.15.27:0/2667367879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308108940 0x7fc30810ad30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc2f00098a0 tx=0x7fc2f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf88cde85ef54a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:14.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.206+0000 7fc305ffb700 1 -- 172.21.15.27:0/2667367879 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2f0004160 con 0x7fc308108940 2024-04-25T00:06:14.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.206+0000 7fc305ffb700 1 -- 172.21.15.27:0/2667367879 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2f0004300 con 0x7fc308108940 2024-04-25T00:06:14.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.206+0000 7fc305ffb700 1 -- 172.21.15.27:0/2667367879 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2f0011b20 con 0x7fc308108940 2024-04-25T00:06:14.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.206+0000 7fc30d9a1700 1 -- 172.21.15.27:0/2667367879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308108940 msgr2=0x7fc30810ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:14.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.206+0000 7fc30d9a1700 1 --2- 172.21.15.27:0/2667367879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308108940 0x7fc30810ad30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc2f00098a0 tx=0x7fc2f0009bb0 comp rx=0 tx=0).stop 2024-04-25T00:06:14.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.207+0000 7fc30d9a1700 1 -- 172.21.15.27:0/2667367879 shutdown_connections 2024-04-25T00:06:14.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.207+0000 7fc30d9a1700 1 --2- 172.21.15.27:0/2667367879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308108940 0x7fc30810ad30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:14.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.207+0000 7fc30d9a1700 1 -- 172.21.15.27:0/2667367879 >> 172.21.15.27:0/2667367879 conn(0x7fc308102370 msgr2=0x7fc308104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:14.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.207+0000 7fc30d9a1700 1 -- 172.21.15.27:0/2667367879 shutdown_connections 2024-04-25T00:06:14.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.207+0000 7fc30d9a1700 1 -- 172.21.15.27:0/2667367879 wait complete. 2024-04-25T00:06:14.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.207+0000 7fc30d9a1700 1 Processor -- start 2024-04-25T00:06:14.364 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.207+0000 7fc30d9a1700 1 -- start start 2024-04-25T00:06:14.364 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.208+0000 7fc30d9a1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308108940 0x7fc3081081c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:14.364 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.208+0000 7fc30d9a1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc30810c2d0 con 0x7fc308108940 2024-04-25T00:06:14.364 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.208+0000 7fc306ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308108940 0x7fc3081081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:14.364 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.208+0000 7fc306ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308108940 0x7fc3081081c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52588/0 (socket says 172.21.15.27:52588) 2024-04-25T00:06:14.364 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.208+0000 7fc306ffd700 1 -- 172.21.15.27:0/3344090610 learned_addr learned my addr 172.21.15.27:0/3344090610 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:14.364 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.208+0000 7fc306ffd700 1 -- 172.21.15.27:0/3344090610 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc308108700 con 0x7fc308108940 2024-04-25T00:06:14.364 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.208+0000 7fc306ffd700 1 --2- 172.21.15.27:0/3344090610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308108940 0x7fc3081081c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0011b60 tx=0x7fc2f0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:14.364 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.208+0000 7fc30c99f700 1 -- 172.21.15.27:0/3344090610 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2f0003b60 con 0x7fc308108940 2024-04-25T00:06:14.364 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.208+0000 7fc30d9a1700 1 -- 172.21.15.27:0/3344090610 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3081068f0 con 0x7fc308108940 2024-04-25T00:06:14.364 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.208+0000 7fc30c99f700 1 -- 172.21.15.27:0/3344090610 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2f001a910 con 0x7fc308108940 2024-04-25T00:06:14.364 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.208+0000 7fc30c99f700 1 -- 172.21.15.27:0/3344090610 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2f0024930 con 0x7fc308108940 2024-04-25T00:06:14.364 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.208+0000 7fc30d9a1700 1 -- 172.21.15.27:0/3344090610 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc308106db0 con 0x7fc308108940 2024-04-25T00:06:14.364 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.209+0000 7fc30c99f700 1 -- 172.21.15.27:0/3344090610 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7fc2f0022020 con 0x7fc308108940 2024-04-25T00:06:14.365 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.209+0000 7fc30c99f700 1 --2- 172.21.15.27:0/3344090610 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc2f4039100 0x7fc2f403b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:14.365 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.209+0000 7fc30c99f700 1 -- 172.21.15.27:0/3344090610 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc2f004e330 con 0x7fc308108940 2024-04-25T00:06:14.365 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.209+0000 7fc30d9a1700 1 -- 172.21.15.27:0/3344090610 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc308063890 con 0x7fc308108940 2024-04-25T00:06:14.365 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.210+0000 7fc3067fc700 1 --2- 172.21.15.27:0/3344090610 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc2f4039100 0x7fc2f403b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:14.365 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.210+0000 7fc3067fc700 1 --2- 172.21.15.27:0/3344090610 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc2f4039100 0x7fc2f403b5c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8006fd0 tx=0x7fc2f8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:14.365 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.215+0000 7fc30c99f700 1 -- 172.21.15.27:0/3344090610 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc2f000fdd0 con 0x7fc308108940 2024-04-25T00:06:14.365 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.445+0000 7fc30d9a1700 1 -- 172.21.15.27:0/3344090610 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fc308106fd0 con 0x7fc2f4039100 2024-04-25T00:06:14.365 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.692+0000 7fc30c99f700 1 -- 172.21.15.27:0/3344090610 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fc308106fd0 con 0x7fc2f4039100 2024-04-25T00:06:14.365 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.694+0000 7fc30d9a1700 1 -- 172.21.15.27:0/3344090610 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc2f4039100 msgr2=0x7fc2f403b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:14.366 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.694+0000 7fc30d9a1700 1 --2- 172.21.15.27:0/3344090610 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc2f4039100 0x7fc2f403b5c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8006fd0 tx=0x7fc2f8006e40 comp rx=0 tx=0).stop 2024-04-25T00:06:14.366 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.694+0000 7fc30d9a1700 1 -- 172.21.15.27:0/3344090610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308108940 msgr2=0x7fc3081081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:14.366 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.694+0000 7fc30d9a1700 1 --2- 172.21.15.27:0/3344090610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308108940 0x7fc3081081c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0011b60 tx=0x7fc2f0011c40 comp rx=0 tx=0).stop 2024-04-25T00:06:14.366 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.695+0000 7fc30d9a1700 1 -- 172.21.15.27:0/3344090610 shutdown_connections 2024-04-25T00:06:14.366 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.695+0000 7fc30d9a1700 1 --2- 172.21.15.27:0/3344090610 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc2f4039100 0x7fc2f403b5c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:14.366 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.695+0000 7fc30d9a1700 1 --2- 172.21.15.27:0/3344090610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308108940 0x7fc3081081c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:14.366 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.695+0000 7fc30d9a1700 1 -- 172.21.15.27:0/3344090610 >> 172.21.15.27:0/3344090610 conn(0x7fc308102370 msgr2=0x7fc308102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:14.366 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.695+0000 7fc30d9a1700 1 -- 172.21.15.27:0/3344090610 shutdown_connections 2024-04-25T00:06:14.366 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:13.695+0000 7fc30d9a1700 1 -- 172.21.15.27:0/3344090610 wait complete. 2024-04-25T00:06:14.366 INFO:teuthology.orchestra.run.smithi027.stdout:Fetching dashboard port number... 2024-04-25T00:06:15.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:15 smithi027 ceph-mon[29208]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:06:16.189 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 8443 2024-04-25T00:06:16.189 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.235+0000 7ff7cc6e4700 1 Processor -- start 2024-04-25T00:06:16.189 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.236+0000 7ff7cc6e4700 1 -- start start 2024-04-25T00:06:16.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.236+0000 7ff7cc6e4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c4107ed0 0x7ff7c41082f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:16.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.236+0000 7ff7cc6e4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c4108830 con 0x7ff7c4107ed0 2024-04-25T00:06:16.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.236+0000 7ff7ca480700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c4107ed0 0x7ff7c41082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:16.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.236+0000 7ff7ca480700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c4107ed0 0x7ff7c41082f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52600/0 (socket says 172.21.15.27:52600) 2024-04-25T00:06:16.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.236+0000 7ff7ca480700 1 -- 172.21.15.27:0/1547226892 learned_addr learned my addr 172.21.15.27:0/1547226892 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:16.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.237+0000 7ff7ca480700 1 -- 172.21.15.27:0/1547226892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c4108fe0 con 0x7ff7c4107ed0 2024-04-25T00:06:16.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.237+0000 7ff7ca480700 1 --2- 172.21.15.27:0/1547226892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c4107ed0 0x7ff7c41082f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00098a0 tx=0x7ff7c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76a030ab43379600 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:16.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.237+0000 7ff7c947e700 1 -- 172.21.15.27:0/1547226892 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7c0004160 con 0x7ff7c4107ed0 2024-04-25T00:06:16.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.237+0000 7ff7c947e700 1 -- 172.21.15.27:0/1547226892 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7c0004300 con 0x7ff7c4107ed0 2024-04-25T00:06:16.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.237+0000 7ff7c947e700 1 -- 172.21.15.27:0/1547226892 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7c0011b20 con 0x7ff7c4107ed0 2024-04-25T00:06:16.190 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.237+0000 7ff7cc6e4700 1 -- 172.21.15.27:0/1547226892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c4107ed0 msgr2=0x7ff7c41082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:16.191 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.237+0000 7ff7cc6e4700 1 --2- 172.21.15.27:0/1547226892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c4107ed0 0x7ff7c41082f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00098a0 tx=0x7ff7c0009bb0 comp rx=0 tx=0).stop 2024-04-25T00:06:16.191 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.237+0000 7ff7cc6e4700 1 -- 172.21.15.27:0/1547226892 shutdown_connections 2024-04-25T00:06:16.191 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.238+0000 7ff7cc6e4700 1 --2- 172.21.15.27:0/1547226892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c4107ed0 0x7ff7c41082f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:16.191 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.238+0000 7ff7cc6e4700 1 -- 172.21.15.27:0/1547226892 >> 172.21.15.27:0/1547226892 conn(0x7ff7c4103490 msgr2=0x7ff7c41058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:16.191 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.238+0000 7ff7cc6e4700 1 -- 172.21.15.27:0/1547226892 shutdown_connections 2024-04-25T00:06:16.191 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.238+0000 7ff7cc6e4700 1 -- 172.21.15.27:0/1547226892 wait complete. 2024-04-25T00:06:16.191 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.238+0000 7ff7cc6e4700 1 Processor -- start 2024-04-25T00:06:16.191 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.238+0000 7ff7cc6e4700 1 -- start start 2024-04-25T00:06:16.191 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.238+0000 7ff7cc6e4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c4107ed0 0x7ff7c419bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:16.191 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.238+0000 7ff7cc6e4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c410c420 con 0x7ff7c4107ed0 2024-04-25T00:06:16.191 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.238+0000 7ff7ca480700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c4107ed0 0x7ff7c419bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:16.192 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.239+0000 7ff7ca480700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c4107ed0 0x7ff7c419bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52616/0 (socket says 172.21.15.27:52616) 2024-04-25T00:06:16.192 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.239+0000 7ff7ca480700 1 -- 172.21.15.27:0/892517161 learned_addr learned my addr 172.21.15.27:0/892517161 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:16.192 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.239+0000 7ff7ca480700 1 -- 172.21.15.27:0/892517161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c419c1d0 con 0x7ff7c4107ed0 2024-04-25T00:06:16.192 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.239+0000 7ff7ca480700 1 --2- 172.21.15.27:0/892517161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c4107ed0 0x7ff7c419bc90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00117e0 tx=0x7ff7c0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:16.192 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.239+0000 7ff7bb7fe700 1 -- 172.21.15.27:0/892517161 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7c0003bf0 con 0x7ff7c4107ed0 2024-04-25T00:06:16.192 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.239+0000 7ff7bb7fe700 1 -- 172.21.15.27:0/892517161 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7c001a910 con 0x7ff7c4107ed0 2024-04-25T00:06:16.192 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.239+0000 7ff7cc6e4700 1 -- 172.21.15.27:0/892517161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7c419c460 con 0x7ff7c4107ed0 2024-04-25T00:06:16.192 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.239+0000 7ff7bb7fe700 1 -- 172.21.15.27:0/892517161 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7c0024970 con 0x7ff7c4107ed0 2024-04-25T00:06:16.192 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.239+0000 7ff7cc6e4700 1 -- 172.21.15.27:0/892517161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7c419c8c0 con 0x7ff7c4107ed0 2024-04-25T00:06:16.192 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.240+0000 7ff7bb7fe700 1 -- 172.21.15.27:0/892517161 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7ff7c0022020 con 0x7ff7c4107ed0 2024-04-25T00:06:16.192 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.240+0000 7ff7cc6e4700 1 -- 172.21.15.27:0/892517161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7c4063890 con 0x7ff7c4107ed0 2024-04-25T00:06:16.192 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.240+0000 7ff7bb7fe700 1 --2- 172.21.15.27:0/892517161 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff7b0039050 0x7ff7b003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:16.192 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.240+0000 7ff7bb7fe700 1 -- 172.21.15.27:0/892517161 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff7c004d530 con 0x7ff7c4107ed0 2024-04-25T00:06:16.192 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.240+0000 7ff7c9c7f700 1 --2- 172.21.15.27:0/892517161 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff7b0039050 0x7ff7b003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:16.193 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.241+0000 7ff7c9c7f700 1 --2- 172.21.15.27:0/892517161 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff7b0039050 0x7ff7b003b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4006fd0 tx=0x7ff7b4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:16.193 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.245+0000 7ff7bb7fe700 1 -- 172.21.15.27:0/892517161 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff7c0011e20 con 0x7ff7c4107ed0 2024-04-25T00:06:16.193 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.454+0000 7ff7cc6e4700 1 -- 172.21.15.27:0/892517161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7ff7c419f3f0 con 0x7ff7c4107ed0 2024-04-25T00:06:16.193 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.455+0000 7ff7bb7fe700 1 -- 172.21.15.27:0/892517161 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7ff7c000b3e0 con 0x7ff7c4107ed0 2024-04-25T00:06:16.193 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.457+0000 7ff7cc6e4700 1 -- 172.21.15.27:0/892517161 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff7b0039050 msgr2=0x7ff7b003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:16.193 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.457+0000 7ff7cc6e4700 1 --2- 172.21.15.27:0/892517161 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff7b0039050 0x7ff7b003b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4006fd0 tx=0x7ff7b4006e40 comp rx=0 tx=0).stop 2024-04-25T00:06:16.193 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.457+0000 7ff7cc6e4700 1 -- 172.21.15.27:0/892517161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c4107ed0 msgr2=0x7ff7c419bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:16.193 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.457+0000 7ff7cc6e4700 1 --2- 172.21.15.27:0/892517161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c4107ed0 0x7ff7c419bc90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00117e0 tx=0x7ff7c0011c10 comp rx=0 tx=0).stop 2024-04-25T00:06:16.193 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.457+0000 7ff7cc6e4700 1 -- 172.21.15.27:0/892517161 shutdown_connections 2024-04-25T00:06:16.193 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.457+0000 7ff7cc6e4700 1 --2- 172.21.15.27:0/892517161 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff7b0039050 0x7ff7b003b510 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:16.193 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.457+0000 7ff7cc6e4700 1 --2- 172.21.15.27:0/892517161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c4107ed0 0x7ff7c419bc90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:16.194 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.457+0000 7ff7cc6e4700 1 -- 172.21.15.27:0/892517161 >> 172.21.15.27:0/892517161 conn(0x7ff7c4103490 msgr2=0x7ff7c4103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:16.194 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.457+0000 7ff7cc6e4700 1 -- 172.21.15.27:0/892517161 shutdown_connections 2024-04-25T00:06:16.194 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:15.457+0000 7ff7cc6e4700 1 -- 172.21.15.27:0/892517161 wait complete. 2024-04-25T00:06:16.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:16 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/892517161' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-25T00:06:16.195 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T00:06:16.195 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-04-25T00:06:16.202 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T00:06:16.202 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T00:06:16.202 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-04-25T00:06:16.203 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-25T00:06:16.204 INFO:teuthology.orchestra.run.smithi027.stdout:Ceph Dashboard is now available at: 2024-04-25T00:06:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:06:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: URL: https://smithi027:8443/ 2024-04-25T00:06:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: User: admin 2024-04-25T00:06:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: Password: juyv84d1v0 2024-04-25T00:06:16.205 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:06:16.205 INFO:teuthology.orchestra.run.smithi027.stdout:Saving cluster configuration to /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config directory 2024-04-25T00:06:16.205 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling autotune for osd_memory_target 2024-04-25T00:06:17.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:17 smithi027 ceph-mon[29208]: mgrmap e13: x(active, since 6s) 2024-04-25T00:06:17.503 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.637+0000 7fdef83b1700 1 Processor -- start 2024-04-25T00:06:17.503 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.637+0000 7fdef83b1700 1 -- start start 2024-04-25T00:06:17.503 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.637+0000 7fdef83b1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef0107ed0 0x7fdef01082f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:17.503 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.637+0000 7fdef83b1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdef0108830 con 0x7fdef0107ed0 2024-04-25T00:06:17.503 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.638+0000 7fdef614d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef0107ed0 0x7fdef01082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:17.504 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.638+0000 7fdef614d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef0107ed0 0x7fdef01082f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49816/0 (socket says 172.21.15.27:49816) 2024-04-25T00:06:17.504 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.638+0000 7fdef614d700 1 -- 172.21.15.27:0/4195396129 learned_addr learned my addr 172.21.15.27:0/4195396129 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:17.504 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.638+0000 7fdef614d700 1 -- 172.21.15.27:0/4195396129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef0108fe0 con 0x7fdef0107ed0 2024-04-25T00:06:17.504 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.638+0000 7fdef614d700 1 --2- 172.21.15.27:0/4195396129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef0107ed0 0x7fdef01082f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdee8009e80 tx=0x7fdee800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d7095f11f7a9c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:17.504 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.638+0000 7fdef514b700 1 -- 172.21.15.27:0/4195396129 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdee8017070 con 0x7fdef0107ed0 2024-04-25T00:06:17.504 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.638+0000 7fdef514b700 1 -- 172.21.15.27:0/4195396129 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdee800b8c0 con 0x7fdef0107ed0 2024-04-25T00:06:17.504 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.638+0000 7fdef514b700 1 -- 172.21.15.27:0/4195396129 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdee80045c0 con 0x7fdef0107ed0 2024-04-25T00:06:17.504 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.639+0000 7fdef83b1700 1 -- 172.21.15.27:0/4195396129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef0107ed0 msgr2=0x7fdef01082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:17.504 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.639+0000 7fdef83b1700 1 --2- 172.21.15.27:0/4195396129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef0107ed0 0x7fdef01082f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdee8009e80 tx=0x7fdee800b330 comp rx=0 tx=0).stop 2024-04-25T00:06:17.504 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.639+0000 7fdef83b1700 1 -- 172.21.15.27:0/4195396129 shutdown_connections 2024-04-25T00:06:17.504 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.639+0000 7fdef83b1700 1 --2- 172.21.15.27:0/4195396129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef0107ed0 0x7fdef01082f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:17.504 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.639+0000 7fdef83b1700 1 -- 172.21.15.27:0/4195396129 >> 172.21.15.27:0/4195396129 conn(0x7fdef0103490 msgr2=0x7fdef01058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:17.504 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.639+0000 7fdef83b1700 1 -- 172.21.15.27:0/4195396129 shutdown_connections 2024-04-25T00:06:17.505 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.639+0000 7fdef83b1700 1 -- 172.21.15.27:0/4195396129 wait complete. 2024-04-25T00:06:17.505 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.640+0000 7fdef83b1700 1 Processor -- start 2024-04-25T00:06:17.505 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.640+0000 7fdef83b1700 1 -- start start 2024-04-25T00:06:17.505 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.640+0000 7fdef83b1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef0107ed0 0x7fdef019bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:17.505 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.640+0000 7fdef83b1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdef010c420 con 0x7fdef0107ed0 2024-04-25T00:06:17.505 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.640+0000 7fdef614d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef0107ed0 0x7fdef019bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:17.505 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.640+0000 7fdef614d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef0107ed0 0x7fdef019bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49820/0 (socket says 172.21.15.27:49820) 2024-04-25T00:06:17.505 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.640+0000 7fdef614d700 1 -- 172.21.15.27:0/3268522125 learned_addr learned my addr 172.21.15.27:0/3268522125 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:17.505 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.640+0000 7fdef614d700 1 -- 172.21.15.27:0/3268522125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef019c200 con 0x7fdef0107ed0 2024-04-25T00:06:17.505 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.641+0000 7fdef614d700 1 --2- 172.21.15.27:0/3268522125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef0107ed0 0x7fdef019bcc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdee800b7b0 tx=0x7fdee800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:17.505 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.641+0000 7fdeef7fe700 1 -- 172.21.15.27:0/3268522125 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdee8020020 con 0x7fdef0107ed0 2024-04-25T00:06:17.505 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.641+0000 7fdeef7fe700 1 -- 172.21.15.27:0/3268522125 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdee8003d90 con 0x7fdef0107ed0 2024-04-25T00:06:17.505 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.641+0000 7fdef83b1700 1 -- 172.21.15.27:0/3268522125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdef019c490 con 0x7fdef0107ed0 2024-04-25T00:06:17.505 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.641+0000 7fdeef7fe700 1 -- 172.21.15.27:0/3268522125 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdee8017040 con 0x7fdef0107ed0 2024-04-25T00:06:17.506 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.641+0000 7fdef83b1700 1 -- 172.21.15.27:0/3268522125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdef019f180 con 0x7fdef0107ed0 2024-04-25T00:06:17.506 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.642+0000 7fdeef7fe700 1 -- 172.21.15.27:0/3268522125 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fdee8023020 con 0x7fdef0107ed0 2024-04-25T00:06:17.506 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.642+0000 7fdeef7fe700 1 --2- 172.21.15.27:0/3268522125 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fded8039150 0x7fded803b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:17.506 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.642+0000 7fdeef7fe700 1 -- 172.21.15.27:0/3268522125 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdee800f050 con 0x7fdef0107ed0 2024-04-25T00:06:17.506 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.642+0000 7fdef83b1700 1 -- 172.21.15.27:0/3268522125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdef0063890 con 0x7fdef0107ed0 2024-04-25T00:06:17.506 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.642+0000 7fdef594c700 1 --2- 172.21.15.27:0/3268522125 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fded8039150 0x7fded803b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:17.506 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.642+0000 7fdef594c700 1 --2- 172.21.15.27:0/3268522125 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fded8039150 0x7fded803b610 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdee4006fd0 tx=0x7fdee4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:17.506 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.647+0000 7fdeef7fe700 1 -- 172.21.15.27:0/3268522125 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdee802a020 con 0x7fdef0107ed0 2024-04-25T00:06:17.506 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.855+0000 7fdef83b1700 1 -- 172.21.15.27:0/3268522125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fdef019f350 con 0x7fdef0107ed0 2024-04-25T00:06:17.506 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.855+0000 7fdeef7fe700 1 -- 172.21.15.27:0/3268522125 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fdee8007570 con 0x7fdef0107ed0 2024-04-25T00:06:17.506 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.857+0000 7fdef83b1700 1 -- 172.21.15.27:0/3268522125 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fded8039150 msgr2=0x7fded803b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:17.506 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.857+0000 7fdef83b1700 1 --2- 172.21.15.27:0/3268522125 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fded8039150 0x7fded803b610 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdee4006fd0 tx=0x7fdee4006e40 comp rx=0 tx=0).stop 2024-04-25T00:06:17.506 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.857+0000 7fdef83b1700 1 -- 172.21.15.27:0/3268522125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef0107ed0 msgr2=0x7fdef019bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:17.507 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.857+0000 7fdef83b1700 1 --2- 172.21.15.27:0/3268522125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef0107ed0 0x7fdef019bcc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdee800b7b0 tx=0x7fdee800bd50 comp rx=0 tx=0).stop 2024-04-25T00:06:17.507 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.857+0000 7fdef83b1700 1 -- 172.21.15.27:0/3268522125 shutdown_connections 2024-04-25T00:06:17.507 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.857+0000 7fdef83b1700 1 --2- 172.21.15.27:0/3268522125 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fded8039150 0x7fded803b610 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:17.507 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.857+0000 7fdef83b1700 1 --2- 172.21.15.27:0/3268522125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef0107ed0 0x7fdef019bcc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:17.507 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.857+0000 7fdef83b1700 1 -- 172.21.15.27:0/3268522125 >> 172.21.15.27:0/3268522125 conn(0x7fdef0103490 msgr2=0x7fdef0103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:17.507 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.858+0000 7fdef83b1700 1 -- 172.21.15.27:0/3268522125 shutdown_connections 2024-04-25T00:06:17.507 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:16.858+0000 7fdef83b1700 1 -- 172.21.15.27:0/3268522125 wait complete. 2024-04-25T00:06:18.822 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.910+0000 7f1e17e6f700 1 Processor -- start 2024-04-25T00:06:18.823 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.910+0000 7f1e17e6f700 1 -- start start 2024-04-25T00:06:18.823 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.910+0000 7f1e17e6f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1007ddf0 0x7f1e1007c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:18.823 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.910+0000 7f1e17e6f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1007c7e0 con 0x7f1e1007ddf0 2024-04-25T00:06:18.823 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.910+0000 7f1e15c0b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1007ddf0 0x7f1e1007c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:18.823 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.910+0000 7f1e15c0b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1007ddf0 0x7f1e1007c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49830/0 (socket says 172.21.15.27:49830) 2024-04-25T00:06:18.823 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.910+0000 7f1e15c0b700 1 -- 172.21.15.27:0/547457824 learned_addr learned my addr 172.21.15.27:0/547457824 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:18.823 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.911+0000 7f1e15c0b700 1 -- 172.21.15.27:0/547457824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e1007c960 con 0x7f1e1007ddf0 2024-04-25T00:06:18.823 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.911+0000 7f1e15c0b700 1 --2- 172.21.15.27:0/547457824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1007ddf0 0x7f1e1007c2a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c0098a0 tx=0x7f1e0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac561007db177847 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:18.823 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.911+0000 7f1e14c09700 1 -- 172.21.15.27:0/547457824 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e0c004160 con 0x7f1e1007ddf0 2024-04-25T00:06:18.823 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.911+0000 7f1e14c09700 1 -- 172.21.15.27:0/547457824 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e0c004300 con 0x7f1e1007ddf0 2024-04-25T00:06:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.911+0000 7f1e14c09700 1 -- 172.21.15.27:0/547457824 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e0c011b20 con 0x7f1e1007ddf0 2024-04-25T00:06:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.912+0000 7f1e17e6f700 1 -- 172.21.15.27:0/547457824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1007ddf0 msgr2=0x7f1e1007c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.912+0000 7f1e17e6f700 1 --2- 172.21.15.27:0/547457824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1007ddf0 0x7f1e1007c2a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c0098a0 tx=0x7f1e0c009bb0 comp rx=0 tx=0).stop 2024-04-25T00:06:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.912+0000 7f1e17e6f700 1 -- 172.21.15.27:0/547457824 shutdown_connections 2024-04-25T00:06:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.912+0000 7f1e17e6f700 1 --2- 172.21.15.27:0/547457824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1007ddf0 0x7f1e1007c2a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.912+0000 7f1e17e6f700 1 -- 172.21.15.27:0/547457824 >> 172.21.15.27:0/547457824 conn(0x7f1e10103490 msgr2=0x7f1e101058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.912+0000 7f1e17e6f700 1 -- 172.21.15.27:0/547457824 shutdown_connections 2024-04-25T00:06:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.912+0000 7f1e17e6f700 1 -- 172.21.15.27:0/547457824 wait complete. 2024-04-25T00:06:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.912+0000 7f1e17e6f700 1 Processor -- start 2024-04-25T00:06:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.912+0000 7f1e17e6f700 1 -- start start 2024-04-25T00:06:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.913+0000 7f1e17e6f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1007ddf0 0x7f1e1019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.913+0000 7f1e17e6f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1010e490 con 0x7f1e1007ddf0 2024-04-25T00:06:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.913+0000 7f1e15c0b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1007ddf0 0x7f1e1019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.913+0000 7f1e15c0b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1007ddf0 0x7f1e1019de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49840/0 (socket says 172.21.15.27:49840) 2024-04-25T00:06:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.913+0000 7f1e15c0b700 1 -- 172.21.15.27:0/3504087610 learned_addr learned my addr 172.21.15.27:0/3504087610 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.913+0000 7f1e15c0b700 1 -- 172.21.15.27:0/3504087610 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e1019e3c0 con 0x7f1e1007ddf0 2024-04-25T00:06:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.913+0000 7f1e15c0b700 1 --2- 172.21.15.27:0/3504087610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1007ddf0 0x7f1e1019de80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c0117e0 tx=0x7f1e0c011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.913+0000 7f1e06ffd700 1 -- 172.21.15.27:0/3504087610 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e0c01a430 con 0x7f1e1007ddf0 2024-04-25T00:06:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.913+0000 7f1e06ffd700 1 -- 172.21.15.27:0/3504087610 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e0c01aab0 con 0x7f1e1007ddf0 2024-04-25T00:06:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.913+0000 7f1e17e6f700 1 -- 172.21.15.27:0/3504087610 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e1019e650 con 0x7f1e1007ddf0 2024-04-25T00:06:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.913+0000 7f1e06ffd700 1 -- 172.21.15.27:0/3504087610 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e0c024a60 con 0x7f1e1007ddf0 2024-04-25T00:06:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.913+0000 7f1e17e6f700 1 -- 172.21.15.27:0/3504087610 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e1019eab0 con 0x7f1e1007ddf0 2024-04-25T00:06:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.914+0000 7f1e06ffd700 1 -- 172.21.15.27:0/3504087610 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f1e0c022020 con 0x7f1e1007ddf0 2024-04-25T00:06:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.914+0000 7f1e17e6f700 1 -- 172.21.15.27:0/3504087610 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e10063890 con 0x7f1e1007ddf0 2024-04-25T00:06:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.914+0000 7f1e06ffd700 1 --2- 172.21.15.27:0/3504087610 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1dfc0391a0 0x7f1dfc03b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.914+0000 7f1e06ffd700 1 -- 172.21.15.27:0/3504087610 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1e0c04da70 con 0x7f1e1007ddf0 2024-04-25T00:06:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.915+0000 7f1e1540a700 1 --2- 172.21.15.27:0/3504087610 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1dfc0391a0 0x7f1dfc03b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.915+0000 7f1e1540a700 1 --2- 172.21.15.27:0/3504087610 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1dfc0391a0 0x7f1dfc03b660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1e00006fd0 tx=0x7f1e00006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:17.919+0000 7f1e06ffd700 1 -- 172.21.15.27:0/3504087610 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1e0c02dde0 con 0x7f1e1007ddf0 2024-04-25T00:06:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:18.243+0000 7f1e17e6f700 1 -- 172.21.15.27:0/3504087610 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f1e101a1540 con 0x7f1e1007ddf0 2024-04-25T00:06:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:18.246+0000 7f1e06ffd700 1 -- 172.21.15.27:0/3504087610 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f1e0c02ad60 con 0x7f1e1007ddf0 2024-04-25T00:06:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:18.248+0000 7f1e17e6f700 1 -- 172.21.15.27:0/3504087610 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1dfc0391a0 msgr2=0x7f1dfc03b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:18.248+0000 7f1e17e6f700 1 --2- 172.21.15.27:0/3504087610 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1dfc0391a0 0x7f1dfc03b660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1e00006fd0 tx=0x7f1e00006e40 comp rx=0 tx=0).stop 2024-04-25T00:06:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:18.248+0000 7f1e17e6f700 1 -- 172.21.15.27:0/3504087610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1007ddf0 msgr2=0x7f1e1019de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:18.248+0000 7f1e17e6f700 1 --2- 172.21.15.27:0/3504087610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1007ddf0 0x7f1e1019de80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c0117e0 tx=0x7f1e0c011ce0 comp rx=0 tx=0).stop 2024-04-25T00:06:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:18.248+0000 7f1e17e6f700 1 -- 172.21.15.27:0/3504087610 shutdown_connections 2024-04-25T00:06:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:18.248+0000 7f1e17e6f700 1 --2- 172.21.15.27:0/3504087610 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1dfc0391a0 0x7f1dfc03b660 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:18.248+0000 7f1e17e6f700 1 --2- 172.21.15.27:0/3504087610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1007ddf0 0x7f1e1019de80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:18.248+0000 7f1e17e6f700 1 -- 172.21.15.27:0/3504087610 >> 172.21.15.27:0/3504087610 conn(0x7f1e10103490 msgr2=0x7f1e10103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:18.248+0000 7f1e17e6f700 1 -- 172.21.15.27:0/3504087610 shutdown_connections 2024-04-25T00:06:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T00:06:18.248+0000 7f1e17e6f700 1 -- 172.21.15.27:0/3504087610 wait complete. 2024-04-25T00:06:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-25T00:06:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-25T00:06:18.828 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:06:18.828 INFO:teuthology.orchestra.run.smithi027.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-25T00:06:18.828 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:06:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:Or, if you are only running a single cluster on this host: 2024-04-25T00:06:18.828 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:06:18.828 INFO:teuthology.orchestra.run.smithi027.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-25T00:06:18.828 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:06:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-25T00:06:18.828 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:06:18.828 INFO:teuthology.orchestra.run.smithi027.stdout: ceph telemetry on 2024-04-25T00:06:18.829 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:06:18.829 INFO:teuthology.orchestra.run.smithi027.stdout:For more information see: 2024-04-25T00:06:18.829 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:06:18.829 INFO:teuthology.orchestra.run.smithi027.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-25T00:06:18.829 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:06:18.829 INFO:teuthology.orchestra.run.smithi027.stdout:Bootstrap complete. 2024-04-25T00:06:18.858 INFO:tasks.cephadm:Fetching config... 2024-04-25T00:06:18.859 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:06:18.860 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-25T00:06:18.878 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-25T00:06:18.878 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:06:18.878 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-25T00:06:18.936 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-25T00:06:18.937 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:06:18.937 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-25T00:06:19.001 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-25T00:06:19.001 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:06:19.001 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-25T00:06:19.058 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-25T00:06:19.058 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtfwAexjPUInlhRfXsAm8aPKQNnO60XcMzYnfFwWoJxjOCTYlR6/CnHw+QCbvwOg/6ZHW2vnFboj1yr67VrnQ53UrHz/YRNJVgd3aaXad971C8oEc5XONhHydKZvqtk96GAcKTb3eD9VZ7XML5yEA5vKReRS4pEWlmFpJHjeSPD9iBzYzw91tuaZEGGvfPRZSn0+ZaeR+FV2F7KeSZmlaHi4vyj7IAQdMgovstR9z4kG0kptn89IsNEbKuUvevpi8H5GwNOl6p+zi4XPwsxGbdGbYPrL/7xDk0FKLKvyHfOtSmlE0YWwBkD1TRzT0LWxpcmCYcra/cBxKbgvvmjcGwUb9IPhQTxAha5bW9elfNpJauskfFNORINMJ/bfgJSb7NmkCX6MPaPv/G5InHu1b6bjHXHc5tuD9JSvaQBqrlm3CY1U/RWp1s54ZN1CMfAR2Mt8YERDfMDCZXnvM2PyYXymXu1dUADwm9ILTWAiOSqo2csWRNPlRQMczceD+q+fM= ceph-518857f8-0297-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T00:06:19.131 INFO:teuthology.orchestra.run.smithi027.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtfwAexjPUInlhRfXsAm8aPKQNnO60XcMzYnfFwWoJxjOCTYlR6/CnHw+QCbvwOg/6ZHW2vnFboj1yr67VrnQ53UrHz/YRNJVgd3aaXad971C8oEc5XONhHydKZvqtk96GAcKTb3eD9VZ7XML5yEA5vKReRS4pEWlmFpJHjeSPD9iBzYzw91tuaZEGGvfPRZSn0+ZaeR+FV2F7KeSZmlaHi4vyj7IAQdMgovstR9z4kG0kptn89IsNEbKuUvevpi8H5GwNOl6p+zi4XPwsxGbdGbYPrL/7xDk0FKLKvyHfOtSmlE0YWwBkD1TRzT0LWxpcmCYcra/cBxKbgvvmjcGwUb9IPhQTxAha5bW9elfNpJauskfFNORINMJ/bfgJSb7NmkCX6MPaPv/G5InHu1b6bjHXHc5tuD9JSvaQBqrlm3CY1U/RWp1s54ZN1CMfAR2Mt8YERDfMDCZXnvM2PyYXymXu1dUADwm9ILTWAiOSqo2csWRNPlRQMczceD+q+fM= ceph-518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:06:19.142 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtfwAexjPUInlhRfXsAm8aPKQNnO60XcMzYnfFwWoJxjOCTYlR6/CnHw+QCbvwOg/6ZHW2vnFboj1yr67VrnQ53UrHz/YRNJVgd3aaXad971C8oEc5XONhHydKZvqtk96GAcKTb3eD9VZ7XML5yEA5vKReRS4pEWlmFpJHjeSPD9iBzYzw91tuaZEGGvfPRZSn0+ZaeR+FV2F7KeSZmlaHi4vyj7IAQdMgovstR9z4kG0kptn89IsNEbKuUvevpi8H5GwNOl6p+zi4XPwsxGbdGbYPrL/7xDk0FKLKvyHfOtSmlE0YWwBkD1TRzT0LWxpcmCYcra/cBxKbgvvmjcGwUb9IPhQTxAha5bW9elfNpJauskfFNORINMJ/bfgJSb7NmkCX6MPaPv/G5InHu1b6bjHXHc5tuD9JSvaQBqrlm3CY1U/RWp1s54ZN1CMfAR2Mt8YERDfMDCZXnvM2PyYXymXu1dUADwm9ILTWAiOSqo2csWRNPlRQMczceD+q+fM= ceph-518857f8-0297-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T00:06:19.182 INFO:teuthology.orchestra.run.smithi139.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtfwAexjPUInlhRfXsAm8aPKQNnO60XcMzYnfFwWoJxjOCTYlR6/CnHw+QCbvwOg/6ZHW2vnFboj1yr67VrnQ53UrHz/YRNJVgd3aaXad971C8oEc5XONhHydKZvqtk96GAcKTb3eD9VZ7XML5yEA5vKReRS4pEWlmFpJHjeSPD9iBzYzw91tuaZEGGvfPRZSn0+ZaeR+FV2F7KeSZmlaHi4vyj7IAQdMgovstR9z4kG0kptn89IsNEbKuUvevpi8H5GwNOl6p+zi4XPwsxGbdGbYPrL/7xDk0FKLKvyHfOtSmlE0YWwBkD1TRzT0LWxpcmCYcra/cBxKbgvvmjcGwUb9IPhQTxAha5bW9elfNpJauskfFNORINMJ/bfgJSb7NmkCX6MPaPv/G5InHu1b6bjHXHc5tuD9JSvaQBqrlm3CY1U/RWp1s54ZN1CMfAR2Mt8YERDfMDCZXnvM2PyYXymXu1dUADwm9ILTWAiOSqo2csWRNPlRQMczceD+q+fM= ceph-518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:06:19.194 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtfwAexjPUInlhRfXsAm8aPKQNnO60XcMzYnfFwWoJxjOCTYlR6/CnHw+QCbvwOg/6ZHW2vnFboj1yr67VrnQ53UrHz/YRNJVgd3aaXad971C8oEc5XONhHydKZvqtk96GAcKTb3eD9VZ7XML5yEA5vKReRS4pEWlmFpJHjeSPD9iBzYzw91tuaZEGGvfPRZSn0+ZaeR+FV2F7KeSZmlaHi4vyj7IAQdMgovstR9z4kG0kptn89IsNEbKuUvevpi8H5GwNOl6p+zi4XPwsxGbdGbYPrL/7xDk0FKLKvyHfOtSmlE0YWwBkD1TRzT0LWxpcmCYcra/cBxKbgvvmjcGwUb9IPhQTxAha5bW9elfNpJauskfFNORINMJ/bfgJSb7NmkCX6MPaPv/G5InHu1b6bjHXHc5tuD9JSvaQBqrlm3CY1U/RWp1s54ZN1CMfAR2Mt8YERDfMDCZXnvM2PyYXymXu1dUADwm9ILTWAiOSqo2csWRNPlRQMczceD+q+fM= ceph-518857f8-0297-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T00:06:19.232 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtfwAexjPUInlhRfXsAm8aPKQNnO60XcMzYnfFwWoJxjOCTYlR6/CnHw+QCbvwOg/6ZHW2vnFboj1yr67VrnQ53UrHz/YRNJVgd3aaXad971C8oEc5XONhHydKZvqtk96GAcKTb3eD9VZ7XML5yEA5vKReRS4pEWlmFpJHjeSPD9iBzYzw91tuaZEGGvfPRZSn0+ZaeR+FV2F7KeSZmlaHi4vyj7IAQdMgovstR9z4kG0kptn89IsNEbKuUvevpi8H5GwNOl6p+zi4XPwsxGbdGbYPrL/7xDk0FKLKvyHfOtSmlE0YWwBkD1TRzT0LWxpcmCYcra/cBxKbgvvmjcGwUb9IPhQTxAha5bW9elfNpJauskfFNORINMJ/bfgJSb7NmkCX6MPaPv/G5InHu1b6bjHXHc5tuD9JSvaQBqrlm3CY1U/RWp1s54ZN1CMfAR2Mt8YERDfMDCZXnvM2PyYXymXu1dUADwm9ILTWAiOSqo2csWRNPlRQMczceD+q+fM= ceph-518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:06:19.244 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-25T00:06:19.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:19 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3504087610' entity='client.admin' 2024-04-25T00:06:19.501 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:06:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.920+0000 7f2ba8047700 1 -- 172.21.15.27:0/2586546279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba0101cd0 msgr2=0x7f2ba01020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.920+0000 7f2ba8047700 1 --2- 172.21.15.27:0/2586546279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba0101cd0 0x7f2ba01020f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c009910 tx=0x7f2b9c009c20 comp rx=0 tx=0).stop 2024-04-25T00:06:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.920+0000 7f2ba8047700 1 -- 172.21.15.27:0/2586546279 shutdown_connections 2024-04-25T00:06:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.920+0000 7f2ba8047700 1 --2- 172.21.15.27:0/2586546279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba0101cd0 0x7f2ba01020f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.920+0000 7f2ba8047700 1 -- 172.21.15.27:0/2586546279 >> 172.21.15.27:0/2586546279 conn(0x7f2ba00fd230 msgr2=0x7f2ba00ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.920+0000 7f2ba8047700 1 -- 172.21.15.27:0/2586546279 shutdown_connections 2024-04-25T00:06:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.920+0000 7f2ba8047700 1 -- 172.21.15.27:0/2586546279 wait complete. 2024-04-25T00:06:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.921+0000 7f2ba8047700 1 Processor -- start 2024-04-25T00:06:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.921+0000 7f2ba8047700 1 -- start start 2024-04-25T00:06:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.921+0000 7f2ba8047700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba0101cd0 0x7f2ba01995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.921+0000 7f2ba8047700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba0075970 con 0x7f2ba0101cd0 2024-04-25T00:06:21.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.921+0000 7f2ba5de3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba0101cd0 0x7f2ba01995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:21.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.921+0000 7f2ba5de3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba0101cd0 0x7f2ba01995c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49848/0 (socket says 172.21.15.27:49848) 2024-04-25T00:06:21.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.921+0000 7f2ba5de3700 1 -- 172.21.15.27:0/2634795043 learned_addr learned my addr 172.21.15.27:0/2634795043 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:21.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.921+0000 7f2ba5de3700 1 -- 172.21.15.27:0/2634795043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba0199b00 con 0x7f2ba0101cd0 2024-04-25T00:06:21.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.921+0000 7f2ba5de3700 1 --2- 172.21.15.27:0/2634795043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba0101cd0 0x7f2ba01995c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c006010 tx=0x7f2b9c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:21.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.922+0000 7f2b96ffd700 1 -- 172.21.15.27:0/2634795043 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b9c027070 con 0x7f2ba0101cd0 2024-04-25T00:06:21.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.922+0000 7f2ba8047700 1 -- 172.21.15.27:0/2634795043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba0199d90 con 0x7f2ba0101cd0 2024-04-25T00:06:21.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.922+0000 7f2b96ffd700 1 -- 172.21.15.27:0/2634795043 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b9c00be40 con 0x7f2ba0101cd0 2024-04-25T00:06:21.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.922+0000 7f2b96ffd700 1 -- 172.21.15.27:0/2634795043 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b9c00f460 con 0x7f2ba0101cd0 2024-04-25T00:06:21.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.922+0000 7f2ba8047700 1 -- 172.21.15.27:0/2634795043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ba019ca80 con 0x7f2ba0101cd0 2024-04-25T00:06:21.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.923+0000 7f2b96ffd700 1 -- 172.21.15.27:0/2634795043 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f2b9c02a070 con 0x7f2ba0101cd0 2024-04-25T00:06:21.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.923+0000 7f2ba8047700 1 -- 172.21.15.27:0/2634795043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ba00760f0 con 0x7f2ba0101cd0 2024-04-25T00:06:21.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.923+0000 7f2b96ffd700 1 --2- 172.21.15.27:0/2634795043 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2b8c0390f0 0x7f2b8c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:21.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.923+0000 7f2b96ffd700 1 -- 172.21.15.27:0/2634795043 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2b9c01e030 con 0x7f2ba0101cd0 2024-04-25T00:06:21.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.923+0000 7f2ba55e2700 1 --2- 172.21.15.27:0/2634795043 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2b8c0390f0 0x7f2b8c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:21.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.923+0000 7f2ba55e2700 1 --2- 172.21.15.27:0/2634795043 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2b8c0390f0 0x7f2b8c03b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2b90006fd0 tx=0x7f2b90006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:21.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:21.928+0000 7f2b96ffd700 1 -- 172.21.15.27:0/2634795043 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2b9c013070 con 0x7f2ba0101cd0 2024-04-25T00:06:22.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:22.134+0000 7f2ba8047700 1 -- 172.21.15.27:0/2634795043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f2ba002dc40 con 0x7f2ba0101cd0 2024-04-25T00:06:22.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:22.136+0000 7f2b96ffd700 1 -- 172.21.15.27:0/2634795043 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f2b9c031020 con 0x7f2ba0101cd0 2024-04-25T00:06:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:22.139+0000 7f2ba8047700 1 -- 172.21.15.27:0/2634795043 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2b8c0390f0 msgr2=0x7f2b8c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:22.139+0000 7f2ba8047700 1 --2- 172.21.15.27:0/2634795043 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2b8c0390f0 0x7f2b8c03b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2b90006fd0 tx=0x7f2b90006e40 comp rx=0 tx=0).stop 2024-04-25T00:06:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:22.139+0000 7f2ba8047700 1 -- 172.21.15.27:0/2634795043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba0101cd0 msgr2=0x7f2ba01995c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:22.139+0000 7f2ba8047700 1 --2- 172.21.15.27:0/2634795043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba0101cd0 0x7f2ba01995c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c006010 tx=0x7f2b9c005050 comp rx=0 tx=0).stop 2024-04-25T00:06:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:22.139+0000 7f2ba8047700 1 -- 172.21.15.27:0/2634795043 shutdown_connections 2024-04-25T00:06:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:22.139+0000 7f2ba8047700 1 --2- 172.21.15.27:0/2634795043 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2b8c0390f0 0x7f2b8c03b5b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:22.139+0000 7f2ba8047700 1 --2- 172.21.15.27:0/2634795043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba0101cd0 0x7f2ba01995c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:22.139+0000 7f2ba8047700 1 -- 172.21.15.27:0/2634795043 >> 172.21.15.27:0/2634795043 conn(0x7f2ba00fd230 msgr2=0x7f2ba00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:22.139+0000 7f2ba8047700 1 -- 172.21.15.27:0/2634795043 shutdown_connections 2024-04-25T00:06:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:22.139+0000 7f2ba8047700 1 -- 172.21.15.27:0/2634795043 wait complete. 2024-04-25T00:06:22.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:22.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:22.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:06:22.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:22.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:06:22.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:22.736 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-25T00:06:22.736 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-25T00:06:22.977 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:06:23.361 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:23 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2634795043' entity='client.admin' 2024-04-25T00:06:23.362 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:23 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:06:23.362 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:23 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.439+0000 7f9e7cdcc700 1 -- 172.21.15.27:0/1717941041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e78100510 msgr2=0x7f9e78102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.439+0000 7f9e7cdcc700 1 --2- 172.21.15.27:0/1717941041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e78100510 0x7f9e78102940 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009910 tx=0x7f9e68009c20 comp rx=0 tx=0).stop 2024-04-25T00:06:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.439+0000 7f9e7cdcc700 1 -- 172.21.15.27:0/1717941041 shutdown_connections 2024-04-25T00:06:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.439+0000 7f9e7cdcc700 1 --2- 172.21.15.27:0/1717941041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e78100510 0x7f9e78102940 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.439+0000 7f9e7cdcc700 1 -- 172.21.15.27:0/1717941041 >> 172.21.15.27:0/1717941041 conn(0x7f9e780fc130 msgr2=0x7f9e780fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.439+0000 7f9e7cdcc700 1 -- 172.21.15.27:0/1717941041 shutdown_connections 2024-04-25T00:06:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.439+0000 7f9e7cdcc700 1 -- 172.21.15.27:0/1717941041 wait complete. 2024-04-25T00:06:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.440+0000 7f9e7cdcc700 1 Processor -- start 2024-04-25T00:06:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.440+0000 7f9e7cdcc700 1 -- start start 2024-04-25T00:06:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.440+0000 7f9e7cdcc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e78100510 0x7f9e7819d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.440+0000 7f9e7cdcc700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9e78105bd0 con 0x7f9e78100510 2024-04-25T00:06:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.440+0000 7f9e7659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e78100510 0x7f9e7819d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.440+0000 7f9e7659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e78100510 0x7f9e7819d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49866/0 (socket says 172.21.15.27:49866) 2024-04-25T00:06:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.440+0000 7f9e7659c700 1 -- 172.21.15.27:0/2664646183 learned_addr learned my addr 172.21.15.27:0/2664646183 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.440+0000 7f9e7659c700 1 -- 172.21.15.27:0/2664646183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e7819dea0 con 0x7f9e78100510 2024-04-25T00:06:24.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.441+0000 7f9e7659c700 1 --2- 172.21.15.27:0/2664646183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e78100510 0x7f9e7819d960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9e68005950 tx=0x7f9e68005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.441+0000 7f9e677fe700 1 -- 172.21.15.27:0/2664646183 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e68026070 con 0x7f9e78100510 2024-04-25T00:06:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.441+0000 7f9e677fe700 1 -- 172.21.15.27:0/2664646183 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e6800bc30 con 0x7f9e78100510 2024-04-25T00:06:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.441+0000 7f9e7cdcc700 1 -- 172.21.15.27:0/2664646183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e7819e130 con 0x7f9e78100510 2024-04-25T00:06:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.441+0000 7f9e677fe700 1 -- 172.21.15.27:0/2664646183 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e68003b30 con 0x7f9e78100510 2024-04-25T00:06:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.441+0000 7f9e7cdcc700 1 -- 172.21.15.27:0/2664646183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e781a0e20 con 0x7f9e78100510 2024-04-25T00:06:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.442+0000 7f9e677fe700 1 -- 172.21.15.27:0/2664646183 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f9e68029070 con 0x7f9e78100510 2024-04-25T00:06:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.442+0000 7f9e7cdcc700 1 -- 172.21.15.27:0/2664646183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e78106350 con 0x7f9e78100510 2024-04-25T00:06:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.442+0000 7f9e677fe700 1 --2- 172.21.15.27:0/2664646183 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9e60039150 0x7f9e6003b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.442+0000 7f9e677fe700 1 -- 172.21.15.27:0/2664646183 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9e6801e070 con 0x7f9e78100510 2024-04-25T00:06:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.442+0000 7f9e75d9b700 1 --2- 172.21.15.27:0/2664646183 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9e60039150 0x7f9e6003b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.442+0000 7f9e75d9b700 1 --2- 172.21.15.27:0/2664646183 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9e60039150 0x7f9e6003b610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c006fd0 tx=0x7f9e6c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:24.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.447+0000 7f9e677fe700 1 -- 172.21.15.27:0/2664646183 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9e68013070 con 0x7f9e78100510 2024-04-25T00:06:24.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.658+0000 7f9e7cdcc700 1 -- 172.21.15.27:0/2664646183 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f9e7802dc40 con 0x7f9e60039150 2024-04-25T00:06:24.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.661+0000 7f9e677fe700 1 -- 172.21.15.27:0/2664646183 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9e7802dc40 con 0x7f9e60039150 2024-04-25T00:06:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.663+0000 7f9e7cdcc700 1 -- 172.21.15.27:0/2664646183 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9e60039150 msgr2=0x7f9e6003b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.663+0000 7f9e7cdcc700 1 --2- 172.21.15.27:0/2664646183 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9e60039150 0x7f9e6003b610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c006fd0 tx=0x7f9e6c006e40 comp rx=0 tx=0).stop 2024-04-25T00:06:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.664+0000 7f9e7cdcc700 1 -- 172.21.15.27:0/2664646183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e78100510 msgr2=0x7f9e7819d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.664+0000 7f9e7cdcc700 1 --2- 172.21.15.27:0/2664646183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e78100510 0x7f9e7819d960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9e68005950 tx=0x7f9e68005030 comp rx=0 tx=0).stop 2024-04-25T00:06:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.664+0000 7f9e7cdcc700 1 -- 172.21.15.27:0/2664646183 shutdown_connections 2024-04-25T00:06:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.664+0000 7f9e7cdcc700 1 --2- 172.21.15.27:0/2664646183 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9e60039150 0x7f9e6003b610 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.664+0000 7f9e7cdcc700 1 --2- 172.21.15.27:0/2664646183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e78100510 0x7f9e7819d960 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.664+0000 7f9e7cdcc700 1 -- 172.21.15.27:0/2664646183 >> 172.21.15.27:0/2664646183 conn(0x7f9e780fc130 msgr2=0x7f9e780fca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.664+0000 7f9e7cdcc700 1 -- 172.21.15.27:0/2664646183 shutdown_connections 2024-04-25T00:06:24.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:24.664+0000 7f9e7cdcc700 1 -- 172.21.15.27:0/2664646183 wait complete. 2024-04-25T00:06:25.353 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi139 2024-04-25T00:06:25.353 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T00:06:25.353 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/etc/ceph/ceph.conf 2024-04-25T00:06:25.371 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T00:06:25.371 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:06:25.429 INFO:tasks.cephadm:Adding host smithi139 to orchestrator... 2024-04-25T00:06:25.429 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch host add smithi139 2024-04-25T00:06:25.679 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:06:25.912 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:25 smithi027 ceph-mon[29208]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:06:25.912 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:25.912 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:06:25.912 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:06:25.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:06:25.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:25 smithi027 ceph-mon[29208]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T00:06:25.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:25 smithi027 ceph-mon[29208]: Updating smithi027:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:06:25.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:25 smithi027 ceph-mon[29208]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:06:25.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:25 smithi027 ceph-mon[29208]: Updating smithi027:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T00:06:25.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:25.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:25.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:27.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.249+0000 7f180e123700 1 -- 172.21.15.27:0/870341175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1808107cd0 msgr2=0x7f18081080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:27.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.249+0000 7f180e123700 1 --2- 172.21.15.27:0/870341175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1808107cd0 0x7f18081080b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f17f8009910 tx=0x7f17f8009c20 comp rx=0 tx=0).stop 2024-04-25T00:06:27.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.249+0000 7f180e123700 1 -- 172.21.15.27:0/870341175 shutdown_connections 2024-04-25T00:06:27.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.249+0000 7f180e123700 1 --2- 172.21.15.27:0/870341175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1808107cd0 0x7f18081080b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:27.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.249+0000 7f180e123700 1 -- 172.21.15.27:0/870341175 >> 172.21.15.27:0/870341175 conn(0x7f1808076690 msgr2=0x7f1808076aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:27.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.249+0000 7f180e123700 1 -- 172.21.15.27:0/870341175 shutdown_connections 2024-04-25T00:06:27.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.249+0000 7f180e123700 1 -- 172.21.15.27:0/870341175 wait complete. 2024-04-25T00:06:27.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.250+0000 7f180e123700 1 Processor -- start 2024-04-25T00:06:27.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.250+0000 7f180e123700 1 -- start start 2024-04-25T00:06:27.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.250+0000 7f180e123700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1808107cd0 0x7f180819d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:27.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.250+0000 7f180e123700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f180810c260 con 0x7f1808107cd0 2024-04-25T00:06:27.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.250+0000 7f18077fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1808107cd0 0x7f180819d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:27.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.250+0000 7f18077fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1808107cd0 0x7f180819d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39726/0 (socket says 172.21.15.27:39726) 2024-04-25T00:06:27.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.250+0000 7f18077fe700 1 -- 172.21.15.27:0/954058601 learned_addr learned my addr 172.21.15.27:0/954058601 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:27.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.251+0000 7f18077fe700 1 -- 172.21.15.27:0/954058601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180819d910 con 0x7f1808107cd0 2024-04-25T00:06:27.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.251+0000 7f18077fe700 1 --2- 172.21.15.27:0/954058601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1808107cd0 0x7f180819d3d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17f8005850 tx=0x7f17f8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:27.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.251+0000 7f1804ff9700 1 -- 172.21.15.27:0/954058601 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f17f8026070 con 0x7f1808107cd0 2024-04-25T00:06:27.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.251+0000 7f1804ff9700 1 -- 172.21.15.27:0/954058601 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17f800bbd0 con 0x7f1808107cd0 2024-04-25T00:06:27.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.251+0000 7f180e123700 1 -- 172.21.15.27:0/954058601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f180819dc00 con 0x7f1808107cd0 2024-04-25T00:06:27.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.251+0000 7f1804ff9700 1 -- 172.21.15.27:0/954058601 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f17f8003970 con 0x7f1808107cd0 2024-04-25T00:06:27.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.251+0000 7f180e123700 1 -- 172.21.15.27:0/954058601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1808197590 con 0x7f1808107cd0 2024-04-25T00:06:27.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.252+0000 7f1804ff9700 1 -- 172.21.15.27:0/954058601 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f17f8029070 con 0x7f1808107cd0 2024-04-25T00:06:27.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.252+0000 7f1804ff9700 1 --2- 172.21.15.27:0/954058601 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f17f0039150 0x7f17f003b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:27.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.252+0000 7f1804ff9700 1 -- 172.21.15.27:0/954058601 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f17f801e070 con 0x7f1808107cd0 2024-04-25T00:06:27.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.252+0000 7f180e123700 1 -- 172.21.15.27:0/954058601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1808103380 con 0x7f1808107cd0 2024-04-25T00:06:27.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.252+0000 7f1806ffd700 1 --2- 172.21.15.27:0/954058601 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f17f0039150 0x7f17f003b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:27.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.253+0000 7f1806ffd700 1 --2- 172.21.15.27:0/954058601 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f17f0039150 0x7f17f003b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f17fc006fd0 tx=0x7f17fc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:27.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.257+0000 7f1804ff9700 1 -- 172.21.15.27:0/954058601 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f17f8030070 con 0x7f1808107cd0 2024-04-25T00:06:27.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:27.472+0000 7f180e123700 1 -- 172.21.15.27:0/954058601 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi139", "target": ["mon-mgr", ""]}) -- 0x7f1808063890 con 0x7f17f0039150 2024-04-25T00:06:28.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:28 smithi027 ceph-mon[29208]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi139", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:06:28.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:28 smithi027 ceph-mon[29208]: Deploying cephadm binary to smithi139 2024-04-25T00:06:30.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:30 smithi027 ceph-mon[29208]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:31.656+0000 7f1804ff9700 1 -- 172.21.15.27:0/954058601 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f1808063890 con 0x7f17f0039150 2024-04-25T00:06:31.658 INFO:teuthology.orchestra.run.smithi027.stdout:Added host 'smithi139' with addr '172.21.15.139' 2024-04-25T00:06:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:31.660+0000 7f180e123700 1 -- 172.21.15.27:0/954058601 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f17f0039150 msgr2=0x7f17f003b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:31.660+0000 7f180e123700 1 --2- 172.21.15.27:0/954058601 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f17f0039150 0x7f17f003b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f17fc006fd0 tx=0x7f17fc006e40 comp rx=0 tx=0).stop 2024-04-25T00:06:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:31.660+0000 7f180e123700 1 -- 172.21.15.27:0/954058601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1808107cd0 msgr2=0x7f180819d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:31.660+0000 7f180e123700 1 --2- 172.21.15.27:0/954058601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1808107cd0 0x7f180819d3d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17f8005850 tx=0x7f17f8005030 comp rx=0 tx=0).stop 2024-04-25T00:06:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:31.660+0000 7f180e123700 1 -- 172.21.15.27:0/954058601 shutdown_connections 2024-04-25T00:06:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:31.660+0000 7f180e123700 1 --2- 172.21.15.27:0/954058601 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f17f0039150 0x7f17f003b610 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:31.660+0000 7f180e123700 1 --2- 172.21.15.27:0/954058601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1808107cd0 0x7f180819d3d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:31.660+0000 7f180e123700 1 -- 172.21.15.27:0/954058601 >> 172.21.15.27:0/954058601 conn(0x7f1808076690 msgr2=0x7f18080fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:31.660+0000 7f180e123700 1 -- 172.21.15.27:0/954058601 shutdown_connections 2024-04-25T00:06:31.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:31.660+0000 7f180e123700 1 -- 172.21.15.27:0/954058601 wait complete. 2024-04-25T00:06:32.280 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T00:06:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:06:32.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:32 smithi027 ceph-mon[29208]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:32.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:32 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:32.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:32 smithi027 ceph-mon[29208]: Added host smithi139 2024-04-25T00:06:32.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:32 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:06:32.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:32 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:33.818 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:33 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:34.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.032+0000 7f7cfa1dc700 1 -- 172.21.15.27:0/939612950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cf4102e40 msgr2=0x7f7cf4103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:34.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.032+0000 7f7cfa1dc700 1 --2- 172.21.15.27:0/939612950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cf4102e40 0x7f7cf4103220 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009910 tx=0x7f7ce8009c20 comp rx=0 tx=0).stop 2024-04-25T00:06:34.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.032+0000 7f7cfa1dc700 1 -- 172.21.15.27:0/939612950 shutdown_connections 2024-04-25T00:06:34.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.032+0000 7f7cfa1dc700 1 --2- 172.21.15.27:0/939612950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cf4102e40 0x7f7cf4103220 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:34.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.032+0000 7f7cfa1dc700 1 -- 172.21.15.27:0/939612950 >> 172.21.15.27:0/939612950 conn(0x7f7cf40fc690 msgr2=0x7f7cf40feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:34.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.032+0000 7f7cfa1dc700 1 -- 172.21.15.27:0/939612950 shutdown_connections 2024-04-25T00:06:34.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.032+0000 7f7cfa1dc700 1 -- 172.21.15.27:0/939612950 wait complete. 2024-04-25T00:06:34.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.033+0000 7f7cfa1dc700 1 Processor -- start 2024-04-25T00:06:34.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.033+0000 7f7cfa1dc700 1 -- start start 2024-04-25T00:06:34.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.033+0000 7f7cfa1dc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cf4102e40 0x7f7cf4100aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:34.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.033+0000 7f7cfa1dc700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf410c310 con 0x7f7cf4102e40 2024-04-25T00:06:34.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.034+0000 7f7cf37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cf4102e40 0x7f7cf4100aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.034+0000 7f7cf37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cf4102e40 0x7f7cf4100aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39748/0 (socket says 172.21.15.27:39748) 2024-04-25T00:06:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.034+0000 7f7cf37fe700 1 -- 172.21.15.27:0/145535672 learned_addr learned my addr 172.21.15.27:0/145535672 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.034+0000 7f7cf37fe700 1 -- 172.21.15.27:0/145535672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf4102990 con 0x7f7cf4102e40 2024-04-25T00:06:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.034+0000 7f7cf37fe700 1 --2- 172.21.15.27:0/145535672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cf4102e40 0x7f7cf4100aa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8006010 tx=0x7f7ce8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.034+0000 7f7cf0ff9700 1 -- 172.21.15.27:0/145535672 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ce8026070 con 0x7f7cf4102e40 2024-04-25T00:06:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.034+0000 7f7cf0ff9700 1 -- 172.21.15.27:0/145535672 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ce800bc30 con 0x7f7cf4102e40 2024-04-25T00:06:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.034+0000 7f7cfa1dc700 1 -- 172.21.15.27:0/145535672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf4100fe0 con 0x7f7cf4102e40 2024-04-25T00:06:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.034+0000 7f7cf0ff9700 1 -- 172.21.15.27:0/145535672 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ce80039a0 con 0x7f7cf4102e40 2024-04-25T00:06:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.034+0000 7f7cfa1dc700 1 -- 172.21.15.27:0/145535672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf41014a0 con 0x7f7cf4102e40 2024-04-25T00:06:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.035+0000 7f7cf0ff9700 1 -- 172.21.15.27:0/145535672 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f7ce8029070 con 0x7f7cf4102e40 2024-04-25T00:06:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.035+0000 7f7cfa1dc700 1 -- 172.21.15.27:0/145535672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cf406a980 con 0x7f7cf4102e40 2024-04-25T00:06:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.035+0000 7f7cf0ff9700 1 --2- 172.21.15.27:0/145535672 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7cdc039150 0x7f7cdc03b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.035+0000 7f7cf0ff9700 1 -- 172.21.15.27:0/145535672 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7ce801e070 con 0x7f7cf4102e40 2024-04-25T00:06:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.036+0000 7f7cf2ffd700 1 --2- 172.21.15.27:0/145535672 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7cdc039150 0x7f7cdc03b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:34.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.036+0000 7f7cf2ffd700 1 --2- 172.21.15.27:0/145535672 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7cdc039150 0x7f7cdc03b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4006fd0 tx=0x7f7ce4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:34.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.040+0000 7f7cf0ff9700 1 -- 172.21.15.27:0/145535672 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7ce8013070 con 0x7f7cf4102e40 2024-04-25T00:06:34.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.264+0000 7f7cfa1dc700 1 -- 172.21.15.27:0/145535672 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7cf41016d0 con 0x7f7cdc039150 2024-04-25T00:06:34.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.265+0000 7f7cf0ff9700 1 -- 172.21.15.27:0/145535672 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f7cf41016d0 con 0x7f7cdc039150 2024-04-25T00:06:34.265 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:06:34.265 INFO:teuthology.orchestra.run.smithi027.stdout:[{"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}, {"addr": "172.21.15.139", "hostname": "smithi139", "labels": [], "status": ""}] 2024-04-25T00:06:34.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.267+0000 7f7cfa1dc700 1 -- 172.21.15.27:0/145535672 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7cdc039150 msgr2=0x7f7cdc03b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:34.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.267+0000 7f7cfa1dc700 1 --2- 172.21.15.27:0/145535672 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7cdc039150 0x7f7cdc03b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4006fd0 tx=0x7f7ce4006e40 comp rx=0 tx=0).stop 2024-04-25T00:06:34.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.267+0000 7f7cfa1dc700 1 -- 172.21.15.27:0/145535672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cf4102e40 msgr2=0x7f7cf4100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:34.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.267+0000 7f7cfa1dc700 1 --2- 172.21.15.27:0/145535672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cf4102e40 0x7f7cf4100aa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8006010 tx=0x7f7ce8005030 comp rx=0 tx=0).stop 2024-04-25T00:06:34.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.267+0000 7f7cfa1dc700 1 -- 172.21.15.27:0/145535672 shutdown_connections 2024-04-25T00:06:34.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.267+0000 7f7cfa1dc700 1 --2- 172.21.15.27:0/145535672 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7cdc039150 0x7f7cdc03b610 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:34.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.267+0000 7f7cfa1dc700 1 --2- 172.21.15.27:0/145535672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cf4102e40 0x7f7cf4100aa0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:34.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.267+0000 7f7cfa1dc700 1 -- 172.21.15.27:0/145535672 >> 172.21.15.27:0/145535672 conn(0x7f7cf40fc690 msgr2=0x7f7cf4103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:34.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.268+0000 7f7cfa1dc700 1 -- 172.21.15.27:0/145535672 shutdown_connections 2024-04-25T00:06:34.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:34.268+0000 7f7cfa1dc700 1 -- 172.21.15.27:0/145535672 wait complete. 2024-04-25T00:06:34.861 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi153 2024-04-25T00:06:34.861 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T00:06:34.861 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.conf 2024-04-25T00:06:34.882 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T00:06:34.882 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:06:34.940 INFO:tasks.cephadm:Adding host smithi153 to orchestrator... 2024-04-25T00:06:34.940 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch host add smithi153 2024-04-25T00:06:35.184 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:06:35.559 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:35 smithi027 ceph-mon[29208]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:35.559 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:35 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:35.559 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:35 smithi027 ceph-mon[29208]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:06:36.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.630+0000 7f45a02ec700 1 -- 172.21.15.27:0/3483068406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4598101ea0 msgr2=0x7f4598102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:36.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.630+0000 7f45a02ec700 1 --2- 172.21.15.27:0/3483068406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4598101ea0 0x7f4598102280 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4594009910 tx=0x7f4594009c20 comp rx=0 tx=0).stop 2024-04-25T00:06:36.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.630+0000 7f45a02ec700 1 -- 172.21.15.27:0/3483068406 shutdown_connections 2024-04-25T00:06:36.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.630+0000 7f45a02ec700 1 --2- 172.21.15.27:0/3483068406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4598101ea0 0x7f4598102280 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:36.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.630+0000 7f45a02ec700 1 -- 172.21.15.27:0/3483068406 >> 172.21.15.27:0/3483068406 conn(0x7f45980fd770 msgr2=0x7f45980ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:36.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.630+0000 7f45a02ec700 1 -- 172.21.15.27:0/3483068406 shutdown_connections 2024-04-25T00:06:36.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.630+0000 7f45a02ec700 1 -- 172.21.15.27:0/3483068406 wait complete. 2024-04-25T00:06:36.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.631+0000 7f45a02ec700 1 Processor -- start 2024-04-25T00:06:36.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.631+0000 7f45a02ec700 1 -- start start 2024-04-25T00:06:36.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.631+0000 7f45a02ec700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4598101ea0 0x7f459819d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:36.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.631+0000 7f45a02ec700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f459810c270 con 0x7f4598101ea0 2024-04-25T00:06:36.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.631+0000 7f459e088700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4598101ea0 0x7f459819d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:36.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.631+0000 7f459e088700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4598101ea0 0x7f459819d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59578/0 (socket says 172.21.15.27:59578) 2024-04-25T00:06:36.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.631+0000 7f459e088700 1 -- 172.21.15.27:0/575392694 learned_addr learned my addr 172.21.15.27:0/575392694 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:36.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.631+0000 7f459e088700 1 -- 172.21.15.27:0/575392694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f459819dab0 con 0x7f4598101ea0 2024-04-25T00:06:36.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.632+0000 7f459e088700 1 --2- 172.21.15.27:0/575392694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4598101ea0 0x7f459819d570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4594006010 tx=0x7f4594005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:36.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.632+0000 7f458f7fe700 1 -- 172.21.15.27:0/575392694 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4594026070 con 0x7f4598101ea0 2024-04-25T00:06:36.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.632+0000 7f458f7fe700 1 -- 172.21.15.27:0/575392694 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f459400bbd0 con 0x7f4598101ea0 2024-04-25T00:06:36.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.632+0000 7f45a02ec700 1 -- 172.21.15.27:0/575392694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f459819dda0 con 0x7f4598101ea0 2024-04-25T00:06:36.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.632+0000 7f458f7fe700 1 -- 172.21.15.27:0/575392694 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4594003910 con 0x7f4598101ea0 2024-04-25T00:06:36.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.632+0000 7f45a02ec700 1 -- 172.21.15.27:0/575392694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4598197810 con 0x7f4598101ea0 2024-04-25T00:06:36.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.633+0000 7f458f7fe700 1 -- 172.21.15.27:0/575392694 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f4594029070 con 0x7f4598101ea0 2024-04-25T00:06:36.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.633+0000 7f45a02ec700 1 -- 172.21.15.27:0/575392694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4598103380 con 0x7f4598101ea0 2024-04-25T00:06:36.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.633+0000 7f458f7fe700 1 --2- 172.21.15.27:0/575392694 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f45840390f0 0x7f458403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:36.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.633+0000 7f458f7fe700 1 -- 172.21.15.27:0/575392694 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f459401e070 con 0x7f4598101ea0 2024-04-25T00:06:36.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.633+0000 7f459d887700 1 --2- 172.21.15.27:0/575392694 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f45840390f0 0x7f458403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:36.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.633+0000 7f459d887700 1 --2- 172.21.15.27:0/575392694 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f45840390f0 0x7f458403b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4588006fd0 tx=0x7f4588006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:36.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.638+0000 7f458f7fe700 1 -- 172.21.15.27:0/575392694 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4594013070 con 0x7f4598101ea0 2024-04-25T00:06:36.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:36.848+0000 7f45a02ec700 1 -- 172.21.15.27:0/575392694 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}) -- 0x7f4598198b70 con 0x7f45840390f0 2024-04-25T00:06:37.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:37 smithi027 ceph-mon[29208]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:38.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:38 smithi027 ceph-mon[29208]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:06:38.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:38.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:38.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:38.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:38.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi139", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:06:38.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:06:38.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:06:38.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:38 smithi027 ceph-mon[29208]: Updating smithi139:/etc/ceph/ceph.conf 2024-04-25T00:06:39.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:39 smithi027 ceph-mon[29208]: Deploying cephadm binary to smithi153 2024-04-25T00:06:39.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:39 smithi027 ceph-mon[29208]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:39.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:39 smithi027 ceph-mon[29208]: Updating smithi139:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:06:39.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:39 smithi027 ceph-mon[29208]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:06:41.138 INFO:teuthology.orchestra.run.smithi027.stdout:Added host 'smithi153' with addr '172.21.15.153' 2024-04-25T00:06:41.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:41.137+0000 7f458f7fe700 1 -- 172.21.15.27:0/575392694 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f4598198b70 con 0x7f45840390f0 2024-04-25T00:06:41.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:41.140+0000 7f45a02ec700 1 -- 172.21.15.27:0/575392694 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f45840390f0 msgr2=0x7f458403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:41.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:41.140+0000 7f45a02ec700 1 --2- 172.21.15.27:0/575392694 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f45840390f0 0x7f458403b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4588006fd0 tx=0x7f4588006e40 comp rx=0 tx=0).stop 2024-04-25T00:06:41.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:41.140+0000 7f45a02ec700 1 -- 172.21.15.27:0/575392694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4598101ea0 msgr2=0x7f459819d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:41.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:41.140+0000 7f45a02ec700 1 --2- 172.21.15.27:0/575392694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4598101ea0 0x7f459819d570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4594006010 tx=0x7f4594005030 comp rx=0 tx=0).stop 2024-04-25T00:06:41.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:41.140+0000 7f45a02ec700 1 -- 172.21.15.27:0/575392694 shutdown_connections 2024-04-25T00:06:41.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:41.140+0000 7f45a02ec700 1 --2- 172.21.15.27:0/575392694 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f45840390f0 0x7f458403b5b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:41.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:41.140+0000 7f45a02ec700 1 --2- 172.21.15.27:0/575392694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4598101ea0 0x7f459819d570 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:41.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:41.140+0000 7f45a02ec700 1 -- 172.21.15.27:0/575392694 >> 172.21.15.27:0/575392694 conn(0x7f45980fd770 msgr2=0x7f4598104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:41.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:41.140+0000 7f45a02ec700 1 -- 172.21.15.27:0/575392694 shutdown_connections 2024-04-25T00:06:41.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:41.140+0000 7f45a02ec700 1 -- 172.21.15.27:0/575392694 wait complete. 2024-04-25T00:06:41.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:41 smithi027 ceph-mon[29208]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:41.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:41 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:41.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:41 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:41.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:41 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:41.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:41 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:41.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:41 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:06:41.756 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T00:06:41.998 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:06:42.649 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:42 smithi027 ceph-mon[29208]: Updating smithi139:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T00:06:42.649 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:42 smithi027 ceph-mon[29208]: Added host smithi153 2024-04-25T00:06:42.649 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:42 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:42.649 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:42 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:43.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.382+0000 7f4bb4945700 1 -- 172.21.15.27:0/2413842380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb0101ea0 msgr2=0x7f4bb0102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:43.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.382+0000 7f4bb4945700 1 --2- 172.21.15.27:0/2413842380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb0101ea0 0x7f4bb0102280 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0009910 tx=0x7f4ba0009c20 comp rx=0 tx=0).stop 2024-04-25T00:06:43.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.382+0000 7f4bb4945700 1 -- 172.21.15.27:0/2413842380 shutdown_connections 2024-04-25T00:06:43.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.382+0000 7f4bb4945700 1 --2- 172.21.15.27:0/2413842380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb0101ea0 0x7f4bb0102280 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:43.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.382+0000 7f4bb4945700 1 -- 172.21.15.27:0/2413842380 >> 172.21.15.27:0/2413842380 conn(0x7f4bb00fd770 msgr2=0x7f4bb00ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:43.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.382+0000 7f4bb4945700 1 -- 172.21.15.27:0/2413842380 shutdown_connections 2024-04-25T00:06:43.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.382+0000 7f4bb4945700 1 -- 172.21.15.27:0/2413842380 wait complete. 2024-04-25T00:06:43.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.382+0000 7f4bb4945700 1 Processor -- start 2024-04-25T00:06:43.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.383+0000 7f4bb4945700 1 -- start start 2024-04-25T00:06:43.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.383+0000 7f4bb4945700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb0101ea0 0x7f4bb019d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:43.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.383+0000 7f4bb4945700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb010c270 con 0x7f4bb0101ea0 2024-04-25T00:06:43.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.383+0000 7f4bae59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb0101ea0 0x7f4bb019d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:43.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.383+0000 7f4bae59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb0101ea0 0x7f4bb019d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59598/0 (socket says 172.21.15.27:59598) 2024-04-25T00:06:43.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.383+0000 7f4bae59c700 1 -- 172.21.15.27:0/3676218438 learned_addr learned my addr 172.21.15.27:0/3676218438 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:43.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.383+0000 7f4bae59c700 1 -- 172.21.15.27:0/3676218438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb019dab0 con 0x7f4bb0101ea0 2024-04-25T00:06:43.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.383+0000 7f4bae59c700 1 --2- 172.21.15.27:0/3676218438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb0101ea0 0x7f4bb019d570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0005850 tx=0x7f4ba0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:43.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.383+0000 7f4b9f7fe700 1 -- 172.21.15.27:0/3676218438 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ba0026070 con 0x7f4bb0101ea0 2024-04-25T00:06:43.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.383+0000 7f4bb4945700 1 -- 172.21.15.27:0/3676218438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb019dda0 con 0x7f4bb0101ea0 2024-04-25T00:06:43.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.384+0000 7f4b9f7fe700 1 -- 172.21.15.27:0/3676218438 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ba000bbd0 con 0x7f4bb0101ea0 2024-04-25T00:06:43.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.384+0000 7f4b9f7fe700 1 -- 172.21.15.27:0/3676218438 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ba0003970 con 0x7f4bb0101ea0 2024-04-25T00:06:43.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.384+0000 7f4bb4945700 1 -- 172.21.15.27:0/3676218438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb0197810 con 0x7f4bb0101ea0 2024-04-25T00:06:43.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.384+0000 7f4b9f7fe700 1 -- 172.21.15.27:0/3676218438 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f4ba0029070 con 0x7f4bb0101ea0 2024-04-25T00:06:43.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.384+0000 7f4b9f7fe700 1 --2- 172.21.15.27:0/3676218438 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4b980390f0 0x7f4b9803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:43.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.385+0000 7f4b9f7fe700 1 -- 172.21.15.27:0/3676218438 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4ba001e070 con 0x7f4bb0101ea0 2024-04-25T00:06:43.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.385+0000 7f4badd9b700 1 --2- 172.21.15.27:0/3676218438 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4b980390f0 0x7f4b9803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:43.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.385+0000 7f4bb4945700 1 -- 172.21.15.27:0/3676218438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bb0103380 con 0x7f4bb0101ea0 2024-04-25T00:06:43.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.385+0000 7f4badd9b700 1 --2- 172.21.15.27:0/3676218438 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4b980390f0 0x7f4b9803b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4006fd0 tx=0x7f4ba4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:43.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.390+0000 7f4b9f7fe700 1 -- 172.21.15.27:0/3676218438 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4ba0030070 con 0x7f4bb0101ea0 2024-04-25T00:06:43.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.600+0000 7f4bb4945700 1 -- 172.21.15.27:0/3676218438 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4bb00008d0 con 0x7f4b980390f0 2024-04-25T00:06:43.601 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:43 smithi027 ceph-mon[29208]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:43.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.601+0000 7f4b9f7fe700 1 -- 172.21.15.27:0/3676218438 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f4bb00008d0 con 0x7f4b980390f0 2024-04-25T00:06:43.602 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:06:43.602 INFO:teuthology.orchestra.run.smithi027.stdout:[{"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}, {"addr": "172.21.15.139", "hostname": "smithi139", "labels": [], "status": ""}, {"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}] 2024-04-25T00:06:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.603+0000 7f4bb4945700 1 -- 172.21.15.27:0/3676218438 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4b980390f0 msgr2=0x7f4b9803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.603+0000 7f4bb4945700 1 --2- 172.21.15.27:0/3676218438 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4b980390f0 0x7f4b9803b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4006fd0 tx=0x7f4ba4006e40 comp rx=0 tx=0).stop 2024-04-25T00:06:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.603+0000 7f4bb4945700 1 -- 172.21.15.27:0/3676218438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb0101ea0 msgr2=0x7f4bb019d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.603+0000 7f4bb4945700 1 --2- 172.21.15.27:0/3676218438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb0101ea0 0x7f4bb019d570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0005850 tx=0x7f4ba0005030 comp rx=0 tx=0).stop 2024-04-25T00:06:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.603+0000 7f4bb4945700 1 -- 172.21.15.27:0/3676218438 shutdown_connections 2024-04-25T00:06:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.603+0000 7f4bb4945700 1 --2- 172.21.15.27:0/3676218438 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4b980390f0 0x7f4b9803b5b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.603+0000 7f4bb4945700 1 --2- 172.21.15.27:0/3676218438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bb0101ea0 0x7f4bb019d570 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.603+0000 7f4bb4945700 1 -- 172.21.15.27:0/3676218438 >> 172.21.15.27:0/3676218438 conn(0x7f4bb00fd770 msgr2=0x7f4bb0104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.603+0000 7f4bb4945700 1 -- 172.21.15.27:0/3676218438 shutdown_connections 2024-04-25T00:06:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:43.603+0000 7f4bb4945700 1 -- 172.21.15.27:0/3676218438 wait complete. 2024-04-25T00:06:44.198 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-25T00:06:44.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-25T00:06:44.437 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:06:44.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:44 smithi027 ceph-mon[29208]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:44.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:44 smithi027 ceph-mon[29208]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:06:44.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:44 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:45.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.944+0000 7feb34fd2700 1 -- 172.21.15.27:0/3932856842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb30106c00 msgr2=0x7feb30106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:45.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.944+0000 7feb34fd2700 1 --2- 172.21.15.27:0/3932856842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb30106c00 0x7feb30106fe0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7feb18009910 tx=0x7feb18009c20 comp rx=0 tx=0).stop 2024-04-25T00:06:45.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.944+0000 7feb34fd2700 1 -- 172.21.15.27:0/3932856842 shutdown_connections 2024-04-25T00:06:45.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.944+0000 7feb34fd2700 1 --2- 172.21.15.27:0/3932856842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb30106c00 0x7feb30106fe0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:45.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.944+0000 7feb34fd2700 1 -- 172.21.15.27:0/3932856842 >> 172.21.15.27:0/3932856842 conn(0x7feb30076690 msgr2=0x7feb30076aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:45.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.944+0000 7feb34fd2700 1 -- 172.21.15.27:0/3932856842 shutdown_connections 2024-04-25T00:06:45.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.944+0000 7feb34fd2700 1 -- 172.21.15.27:0/3932856842 wait complete. 2024-04-25T00:06:45.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.945+0000 7feb34fd2700 1 Processor -- start 2024-04-25T00:06:45.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.945+0000 7feb34fd2700 1 -- start start 2024-04-25T00:06:45.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.945+0000 7feb34fd2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb30106c00 0x7feb30198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:45.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.945+0000 7feb34fd2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feb3010b1c0 con 0x7feb30106c00 2024-04-25T00:06:45.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.945+0000 7feb2e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb30106c00 0x7feb30198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:45.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.946+0000 7feb2e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb30106c00 0x7feb30198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59624/0 (socket says 172.21.15.27:59624) 2024-04-25T00:06:45.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.946+0000 7feb2e59c700 1 -- 172.21.15.27:0/2628368798 learned_addr learned my addr 172.21.15.27:0/2628368798 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:06:45.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.946+0000 7feb2e59c700 1 -- 172.21.15.27:0/2628368798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb30198ab0 con 0x7feb30106c00 2024-04-25T00:06:45.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.946+0000 7feb2e59c700 1 --2- 172.21.15.27:0/2628368798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb30106c00 0x7feb30198570 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7feb18005850 tx=0x7feb18005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:45.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.946+0000 7feb277fe700 1 -- 172.21.15.27:0/2628368798 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb18026070 con 0x7feb30106c00 2024-04-25T00:06:45.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.946+0000 7feb34fd2700 1 -- 172.21.15.27:0/2628368798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb30198d40 con 0x7feb30106c00 2024-04-25T00:06:45.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.946+0000 7feb277fe700 1 -- 172.21.15.27:0/2628368798 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb1800bc30 con 0x7feb30106c00 2024-04-25T00:06:45.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.946+0000 7feb277fe700 1 -- 172.21.15.27:0/2628368798 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb18003a20 con 0x7feb30106c00 2024-04-25T00:06:45.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.946+0000 7feb34fd2700 1 -- 172.21.15.27:0/2628368798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb301991a0 con 0x7feb30106c00 2024-04-25T00:06:45.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.947+0000 7feb277fe700 1 -- 172.21.15.27:0/2628368798 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7feb18029070 con 0x7feb30106c00 2024-04-25T00:06:45.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.947+0000 7feb34fd2700 1 -- 172.21.15.27:0/2628368798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb30102200 con 0x7feb30106c00 2024-04-25T00:06:45.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.947+0000 7feb277fe700 1 --2- 172.21.15.27:0/2628368798 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7feb1c0390f0 0x7feb1c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:45.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.947+0000 7feb277fe700 1 -- 172.21.15.27:0/2628368798 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7feb1801e070 con 0x7feb30106c00 2024-04-25T00:06:45.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.947+0000 7feb2dd9b700 1 --2- 172.21.15.27:0/2628368798 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7feb1c0390f0 0x7feb1c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:45.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.948+0000 7feb2dd9b700 1 --2- 172.21.15.27:0/2628368798 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7feb1c0390f0 0x7feb1c03b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feb20006fd0 tx=0x7feb20006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:45.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:45.952+0000 7feb277fe700 1 -- 172.21.15.27:0/2628368798 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7feb18013070 con 0x7feb30106c00 2024-04-25T00:06:46.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:46.165+0000 7feb34fd2700 1 -- 172.21.15.27:0/2628368798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7feb30063890 con 0x7feb30106c00 2024-04-25T00:06:46.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:46.730+0000 7feb277fe700 1 -- 172.21.15.27:0/2628368798 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7feb18030090 con 0x7feb30106c00 2024-04-25T00:06:46.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:46.733+0000 7feb34fd2700 1 -- 172.21.15.27:0/2628368798 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7feb1c0390f0 msgr2=0x7feb1c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:46.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:46.733+0000 7feb34fd2700 1 --2- 172.21.15.27:0/2628368798 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7feb1c0390f0 0x7feb1c03b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feb20006fd0 tx=0x7feb20006e40 comp rx=0 tx=0).stop 2024-04-25T00:06:46.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:46.733+0000 7feb34fd2700 1 -- 172.21.15.27:0/2628368798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb30106c00 msgr2=0x7feb30198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:46.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:46.733+0000 7feb34fd2700 1 --2- 172.21.15.27:0/2628368798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb30106c00 0x7feb30198570 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7feb18005850 tx=0x7feb18005030 comp rx=0 tx=0).stop 2024-04-25T00:06:46.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:46.733+0000 7feb34fd2700 1 -- 172.21.15.27:0/2628368798 shutdown_connections 2024-04-25T00:06:46.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:46.733+0000 7feb34fd2700 1 --2- 172.21.15.27:0/2628368798 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7feb1c0390f0 0x7feb1c03b5b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:46.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:46.733+0000 7feb34fd2700 1 --2- 172.21.15.27:0/2628368798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb30106c00 0x7feb30198570 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:46.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:46.733+0000 7feb34fd2700 1 -- 172.21.15.27:0/2628368798 >> 172.21.15.27:0/2628368798 conn(0x7feb30076690 msgr2=0x7feb300ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:46.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:46.733+0000 7feb34fd2700 1 -- 172.21.15.27:0/2628368798 shutdown_connections 2024-04-25T00:06:46.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:06:46.733+0000 7feb34fd2700 1 -- 172.21.15.27:0/2628368798 wait complete. 2024-04-25T00:06:46.736 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2024-04-25T00:06:47.068 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:46 smithi027 ceph-mon[29208]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:47.068 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:46 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2628368798' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T00:06:47.422 INFO:tasks.cephadm:Adding mon.a on smithi027 2024-04-25T00:06:47.423 INFO:tasks.cephadm:Adding mon.b on smithi139 2024-04-25T00:06:47.423 INFO:tasks.cephadm:Adding mon.c on smithi153 2024-04-25T00:06:47.423 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi027:172.21.15.27=a;smithi139:172.21.15.139=b;smithi153:172.21.15.153=c' 2024-04-25T00:06:47.656 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:06:48.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:47 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2628368798' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T00:06:48.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:47 smithi027 ceph-mon[29208]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T00:06:48.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:48.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:48.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:48.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:48.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:06:48.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:06:48.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:06:48.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:47 smithi027 ceph-mon[29208]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-25T00:06:48.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:47 smithi027 ceph-mon[29208]: Updating smithi153:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:06:48.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:47 smithi027 ceph-mon[29208]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:06:48.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:48.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:48.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:48.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.980+0000 7f0354820700 1 -- 172.21.15.153:0/1457633395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350100da0 msgr2=0x7f0350101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:48.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.980+0000 7f0354820700 1 --2- 172.21.15.153:0/1457633395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350100da0 0x7f0350101180 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0340009910 tx=0x7f0340009c20 comp rx=0 tx=0).stop 2024-04-25T00:06:48.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.980+0000 7f0354820700 1 -- 172.21.15.153:0/1457633395 shutdown_connections 2024-04-25T00:06:48.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.980+0000 7f0354820700 1 --2- 172.21.15.153:0/1457633395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350100da0 0x7f0350101180 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:48.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.980+0000 7f0354820700 1 -- 172.21.15.153:0/1457633395 >> 172.21.15.153:0/1457633395 conn(0x7f03500fc630 msgr2=0x7f03500fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:48.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.980+0000 7f0354820700 1 -- 172.21.15.153:0/1457633395 shutdown_connections 2024-04-25T00:06:48.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.980+0000 7f0354820700 1 -- 172.21.15.153:0/1457633395 wait complete. 2024-04-25T00:06:48.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.981+0000 7f0354820700 1 Processor -- start 2024-04-25T00:06:48.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.981+0000 7f0354820700 1 -- start start 2024-04-25T00:06:48.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.981+0000 7f0354820700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350100da0 0x7f0350102840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:48.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.981+0000 7f0354820700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f035010b170 con 0x7f0350100da0 2024-04-25T00:06:48.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.982+0000 7f034e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350100da0 0x7f0350102840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:48.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.982+0000 7f034e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350100da0 0x7f0350102840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.153:58850/0 (socket says 172.21.15.153:58850) 2024-04-25T00:06:48.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.982+0000 7f034e59c700 1 -- 172.21.15.153:0/2728951863 learned_addr learned my addr 172.21.15.153:0/2728951863 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T00:06:48.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.982+0000 7f034e59c700 1 -- 172.21.15.153:0/2728951863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0350104730 con 0x7f0350100da0 2024-04-25T00:06:48.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.983+0000 7f034e59c700 1 --2- 172.21.15.153:0/2728951863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350100da0 0x7f0350102840 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0340005850 tx=0x7f0340005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:48.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.983+0000 7f0347fff700 1 -- 172.21.15.153:0/2728951863 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0340026070 con 0x7f0350100da0 2024-04-25T00:06:48.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.983+0000 7f0354820700 1 -- 172.21.15.153:0/2728951863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0350102de0 con 0x7f0350100da0 2024-04-25T00:06:48.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.983+0000 7f0347fff700 1 -- 172.21.15.153:0/2728951863 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f034000bc30 con 0x7f0350100da0 2024-04-25T00:06:48.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.983+0000 7f0347fff700 1 -- 172.21.15.153:0/2728951863 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0340003a20 con 0x7f0350100da0 2024-04-25T00:06:48.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.983+0000 7f0354820700 1 -- 172.21.15.153:0/2728951863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03501032a0 con 0x7f0350100da0 2024-04-25T00:06:48.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.984+0000 7f0354820700 1 -- 172.21.15.153:0/2728951863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0350102280 con 0x7f0350100da0 2024-04-25T00:06:48.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.984+0000 7f0347fff700 1 -- 172.21.15.153:0/2728951863 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f0340029070 con 0x7f0350100da0 2024-04-25T00:06:48.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.984+0000 7f0347fff700 1 --2- 172.21.15.153:0/2728951863 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f033c0390f0 0x7f033c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:48.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.984+0000 7f0347fff700 1 -- 172.21.15.153:0/2728951863 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f034001e070 con 0x7f0350100da0 2024-04-25T00:06:48.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.984+0000 7f0345d9b700 1 --2- 172.21.15.153:0/2728951863 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f033c0390f0 0x7f033c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:48.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.985+0000 7f0345d9b700 1 --2- 172.21.15.153:0/2728951863 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f033c0390f0 0x7f033c03b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0338006fd0 tx=0x7f0338006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:48.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:48.989+0000 7f0347fff700 1 -- 172.21.15.153:0/2728951863 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0340013070 con 0x7f0350100da0 2024-04-25T00:06:49.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:48 smithi027 ceph-mon[29208]: Updating smithi153:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T00:06:49.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:48 smithi027 ceph-mon[29208]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:49.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:49.198+0000 7f0354820700 1 -- 172.21.15.153:0/2728951863 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi139:172.21.15.139=b;smithi153:172.21.15.153=c", "target": ["mon-mgr", ""]}) -- 0x7f0350103d70 con 0x7f033c0390f0 2024-04-25T00:06:49.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:49.205+0000 7f0347fff700 1 -- 172.21.15.153:0/2728951863 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0350103d70 con 0x7f033c0390f0 2024-04-25T00:06:49.205 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mon update... 2024-04-25T00:06:49.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:49.207+0000 7f0354820700 1 -- 172.21.15.153:0/2728951863 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f033c0390f0 msgr2=0x7f033c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:49.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:49.207+0000 7f0354820700 1 --2- 172.21.15.153:0/2728951863 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f033c0390f0 0x7f033c03b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0338006fd0 tx=0x7f0338006e40 comp rx=0 tx=0).stop 2024-04-25T00:06:49.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:49.207+0000 7f0354820700 1 -- 172.21.15.153:0/2728951863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350100da0 msgr2=0x7f0350102840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:49.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:49.207+0000 7f0354820700 1 --2- 172.21.15.153:0/2728951863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350100da0 0x7f0350102840 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0340005850 tx=0x7f0340005030 comp rx=0 tx=0).stop 2024-04-25T00:06:49.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:49.208+0000 7f0354820700 1 -- 172.21.15.153:0/2728951863 shutdown_connections 2024-04-25T00:06:49.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:49.208+0000 7f0354820700 1 --2- 172.21.15.153:0/2728951863 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f033c0390f0 0x7f033c03b5b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:49.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:49.208+0000 7f0354820700 1 --2- 172.21.15.153:0/2728951863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350100da0 0x7f0350102840 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:49.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:49.208+0000 7f0354820700 1 -- 172.21.15.153:0/2728951863 >> 172.21.15.153:0/2728951863 conn(0x7f03500fc630 msgr2=0x7f035006a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:49.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:49.208+0000 7f0354820700 1 -- 172.21.15.153:0/2728951863 shutdown_connections 2024-04-25T00:06:49.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:49.208+0000 7f0354820700 1 -- 172.21.15.153:0/2728951863 wait complete. 2024-04-25T00:06:49.731 DEBUG:teuthology.orchestra.run.smithi139:mon.b> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.b.service 2024-04-25T00:06:49.734 DEBUG:teuthology.orchestra.run.smithi153:mon.c> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.c.service 2024-04-25T00:06:49.737 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T00:06:49.737 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T00:06:50.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:50 smithi027 ceph-mon[29208]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi139:172.21.15.139=b;smithi153:172.21.15.153=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:06:50.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:50 smithi027 ceph-mon[29208]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi139:172.21.15.139=b;smithi153:172.21.15.153=c;count:3 2024-04-25T00:06:50.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:50 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:50.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:50 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:06:50.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:50 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:06:50.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:50 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:06:50.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:50 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:50.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:50 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T00:06:50.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:50 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:06:50.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:50 smithi027 ceph-mon[29208]: Deploying daemon mon.c on smithi153 2024-04-25T00:06:50.905 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:06:51.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:51 smithi027 ceph-mon[29208]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:53.084 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:52 smithi153 systemd[1]: Starting Ceph mon.c for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:06:53.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.548+0000 7fd94466f700 1 -- 172.21.15.153:0/3606148982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd93c100da0 msgr2=0x7fd93c101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:53.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.548+0000 7fd94466f700 1 --2- 172.21.15.153:0/3606148982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd93c100da0 0x7fd93c101180 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd93400b0a0 tx=0x7fd93400b3b0 comp rx=0 tx=0).stop 2024-04-25T00:06:53.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.548+0000 7fd94466f700 1 -- 172.21.15.153:0/3606148982 shutdown_connections 2024-04-25T00:06:53.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.548+0000 7fd94466f700 1 --2- 172.21.15.153:0/3606148982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd93c100da0 0x7fd93c101180 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:53.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.548+0000 7fd94466f700 1 -- 172.21.15.153:0/3606148982 >> 172.21.15.153:0/3606148982 conn(0x7fd93c0fc650 msgr2=0x7fd93c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:53.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.548+0000 7fd94466f700 1 -- 172.21.15.153:0/3606148982 shutdown_connections 2024-04-25T00:06:53.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.548+0000 7fd94466f700 1 -- 172.21.15.153:0/3606148982 wait complete. 2024-04-25T00:06:53.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.549+0000 7fd94466f700 1 Processor -- start 2024-04-25T00:06:53.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.549+0000 7fd94466f700 1 -- start start 2024-04-25T00:06:53.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.549+0000 7fd94466f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd93c100da0 0x7fd93c1976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:53.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.549+0000 7fd94466f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd93c10b170 con 0x7fd93c100da0 2024-04-25T00:06:53.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.549+0000 7fd94240b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd93c100da0 0x7fd93c1976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:53.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.549+0000 7fd94240b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd93c100da0 0x7fd93c1976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.153:58862/0 (socket says 172.21.15.153:58862) 2024-04-25T00:06:53.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.549+0000 7fd94240b700 1 -- 172.21.15.153:0/3803863300 learned_addr learned my addr 172.21.15.153:0/3803863300 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T00:06:53.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.550+0000 7fd94240b700 1 -- 172.21.15.153:0/3803863300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd93c197c30 con 0x7fd93c100da0 2024-04-25T00:06:53.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.550+0000 7fd94240b700 1 --2- 172.21.15.153:0/3803863300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd93c100da0 0x7fd93c1976f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd934015040 tx=0x7fd934003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:53.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.550+0000 7fd93b7fe700 1 -- 172.21.15.153:0/3803863300 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd934017040 con 0x7fd93c100da0 2024-04-25T00:06:53.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.550+0000 7fd93b7fe700 1 -- 172.21.15.153:0/3803863300 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd934022070 con 0x7fd93c100da0 2024-04-25T00:06:53.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.550+0000 7fd93b7fe700 1 -- 172.21.15.153:0/3803863300 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd93401d5d0 con 0x7fd93c100da0 2024-04-25T00:06:53.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.550+0000 7fd94466f700 1 -- 172.21.15.153:0/3803863300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd93c197ec0 con 0x7fd93c100da0 2024-04-25T00:06:53.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.550+0000 7fd94466f700 1 -- 172.21.15.153:0/3803863300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd93c19bfc0 con 0x7fd93c100da0 2024-04-25T00:06:53.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.551+0000 7fd94466f700 1 -- 172.21.15.153:0/3803863300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd93c102280 con 0x7fd93c100da0 2024-04-25T00:06:53.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.551+0000 7fd93b7fe700 1 -- 172.21.15.153:0/3803863300 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fd934004000 con 0x7fd93c100da0 2024-04-25T00:06:53.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.551+0000 7fd93b7fe700 1 --2- 172.21.15.153:0/3803863300 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd9240390f0 0x7fd92403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:53.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.551+0000 7fd93b7fe700 1 -- 172.21.15.153:0/3803863300 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd93400f050 con 0x7fd93c100da0 2024-04-25T00:06:53.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.552+0000 7fd941c0a700 1 --2- 172.21.15.153:0/3803863300 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd9240390f0 0x7fd92403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:53.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.552+0000 7fd941c0a700 1 --2- 172.21.15.153:0/3803863300 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd9240390f0 0x7fd92403b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd930006fd0 tx=0x7fd930006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:53.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.557+0000 7fd93b7fe700 1 -- 172.21.15.153:0/3803863300 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd934007e20 con 0x7fd93c100da0 2024-04-25T00:06:53.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:53 smithi027 ceph-mon[29208]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:53.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.859+0000 7fd94466f700 1 -- 172.21.15.153:0/3803863300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fd93c198a50 con 0x7fd93c100da0 2024-04-25T00:06:53.860 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:06:53.860 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","modified":"2024-04-25T00:05:07.441245Z","created":"2024-04-25T00:05:07.441245Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-25T00:06:53.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.859+0000 7fd93b7fe700 1 -- 172.21.15.153:0/3803863300 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fd93400bcc0 con 0x7fd93c100da0 2024-04-25T00:06:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.862+0000 7fd94466f700 1 -- 172.21.15.153:0/3803863300 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd9240390f0 msgr2=0x7fd92403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.862+0000 7fd94466f700 1 --2- 172.21.15.153:0/3803863300 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd9240390f0 0x7fd92403b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd930006fd0 tx=0x7fd930006e40 comp rx=0 tx=0).stop 2024-04-25T00:06:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.862+0000 7fd94466f700 1 -- 172.21.15.153:0/3803863300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd93c100da0 msgr2=0x7fd93c1976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.862+0000 7fd94466f700 1 --2- 172.21.15.153:0/3803863300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd93c100da0 0x7fd93c1976f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd934015040 tx=0x7fd934003d50 comp rx=0 tx=0).stop 2024-04-25T00:06:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.862+0000 7fd94466f700 1 -- 172.21.15.153:0/3803863300 shutdown_connections 2024-04-25T00:06:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.862+0000 7fd94466f700 1 --2- 172.21.15.153:0/3803863300 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd9240390f0 0x7fd92403b5b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.862+0000 7fd94466f700 1 --2- 172.21.15.153:0/3803863300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd93c100da0 0x7fd93c1976f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.862+0000 7fd94466f700 1 -- 172.21.15.153:0/3803863300 >> 172.21.15.153:0/3803863300 conn(0x7fd93c0fc650 msgr2=0x7fd93c06a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.862+0000 7fd94466f700 1 -- 172.21.15.153:0/3803863300 shutdown_connections 2024-04-25T00:06:53.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:06:53.862+0000 7fd94466f700 1 -- 172.21.15.153:0/3803863300 wait complete. 2024-04-25T00:06:53.864 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-25T00:06:53.864 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 podman[32855]: 2024-04-25 00:06:53.517112437 +0000 UTC m=+0.150929372 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:06:53.864 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 podman[32855]: 2024-04-25 00:06:53.631346166 +0000 UTC m=+0.265163100 container create d2ab362c99894eec4a1a10c18d01b9a614ddd0a45c9d92d6e54d27e9d0e7853e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-c, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T00:06:53.864 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 podman[32855]: 2024-04-25 00:06:53.835013207 +0000 UTC m=+0.468830141 container init d2ab362c99894eec4a1a10c18d01b9a614ddd0a45c9d92d6e54d27e9d0e7853e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux ) 2024-04-25T00:06:53.864 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 podman[32855]: 2024-04-25 00:06:53.838256035 +0000 UTC m=+0.472072977 container start d2ab362c99894eec4a1a10c18d01b9a614ddd0a45c9d92d6e54d27e9d0e7853e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-25T00:06:53.864 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T00:06:53.865 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev), process ceph-mon, pid 2 2024-04-25T00:06:53.865 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: pidfile_write: ignore empty --pid-file 2024-04-25T00:06:54.256 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: load: jerasure load: lrc 2024-04-25T00:06:54.256 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: RocksDB version: 7.9.2 2024-04-25T00:06:54.256 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Git sha 0 2024-04-25T00:06:54.256 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Compile date 2024-04-24 16:41:23 2024-04-25T00:06:54.256 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: DB SUMMARY 2024-04-25T00:06:54.256 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: DB Session ID: QN67KC11VXWG61I7R6NA 2024-04-25T00:06:54.257 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: CURRENT file: CURRENT 2024-04-25T00:06:54.257 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T00:06:54.257 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-25T00:06:54.257 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-25T00:06:54.257 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-25T00:06:54.257 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.error_if_exists: 0 2024-04-25T00:06:54.257 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.create_if_missing: 0 2024-04-25T00:06:54.257 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.paranoid_checks: 1 2024-04-25T00:06:54.257 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T00:06:54.258 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T00:06:54.258 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T00:06:54.258 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.env: 0x55b6b6968c60 2024-04-25T00:06:54.258 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T00:06:54.258 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.info_log: 0x55b6b8723aa0 2024-04-25T00:06:54.258 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T00:06:54.258 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.statistics: (nil) 2024-04-25T00:06:54.259 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.use_fsync: 0 2024-04-25T00:06:54.259 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_log_file_size: 0 2024-04-25T00:06:54.259 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T00:06:54.259 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T00:06:54.259 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T00:06:54.259 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T00:06:54.259 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.allow_fallocate: 1 2024-04-25T00:06:54.259 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T00:06:54.260 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T00:06:54.260 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.use_direct_reads: 0 2024-04-25T00:06:54.260 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T00:06:54.260 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T00:06:54.260 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.db_log_dir: 2024-04-25T00:06:54.260 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.wal_dir: 2024-04-25T00:06:54.260 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T00:06:54.260 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T00:06:54.260 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T00:06:54.261 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T00:06:54.261 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T00:06:54.261 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T00:06:54.261 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T00:06:54.261 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T00:06:54.261 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.write_buffer_manager: 0x55b6b86f61e0 2024-04-25T00:06:54.261 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T00:06:54.261 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T00:06:54.262 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T00:06:54.262 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T00:06:54.262 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T00:06:54.262 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T00:06:54.262 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T00:06:54.262 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T00:06:54.262 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.unordered_write: 0 2024-04-25T00:06:54.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T00:06:54.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T00:06:54.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T00:06:54.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T00:06:54.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.row_cache: None 2024-04-25T00:06:54.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.wal_filter: None 2024-04-25T00:06:54.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T00:06:54.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T00:06:54.264 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.two_write_queues: 0 2024-04-25T00:06:54.264 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T00:06:54.264 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.wal_compression: 0 2024-04-25T00:06:54.264 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.atomic_flush: 0 2024-04-25T00:06:54.264 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T00:06:54.264 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T00:06:54.264 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T00:06:54.264 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.log_readahead_size: 0 2024-04-25T00:06:54.264 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T00:06:54.264 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T00:06:54.265 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T00:06:54.265 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T00:06:54.265 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T00:06:54.265 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T00:06:54.265 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T00:06:54.265 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_background_jobs: 2 2024-04-25T00:06:54.265 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_background_compactions: -1 2024-04-25T00:06:54.265 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_subcompactions: 1 2024-04-25T00:06:54.266 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T00:06:54.266 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T00:06:54.266 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T00:06:54.266 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T00:06:54.266 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T00:06:54.266 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T00:06:54.266 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T00:06:54.267 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T00:06:54.267 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_open_files: -1 2024-04-25T00:06:54.267 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T00:06:54.267 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T00:06:54.267 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T00:06:54.267 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T00:06:54.267 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_background_flushes: -1 2024-04-25T00:06:54.267 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Compression algorithms supported: 2024-04-25T00:06:54.267 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T00:06:54.268 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: kZSTD supported: 0 2024-04-25T00:06:54.268 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: kXpressCompression supported: 0 2024-04-25T00:06:54.268 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T00:06:54.268 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: kZlibCompression supported: 1 2024-04-25T00:06:54.268 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: kSnappyCompression supported: 1 2024-04-25T00:06:54.268 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: kLZ4Compression supported: 1 2024-04-25T00:06:54.268 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: kBZip2Compression supported: 0 2024-04-25T00:06:54.268 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T00:06:54.268 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T00:06:54.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-25T00:06:54.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T00:06:54.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T00:06:54.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.merge_operator: 2024-04-25T00:06:54.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compaction_filter: None 2024-04-25T00:06:54.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T00:06:54.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T00:06:54.269 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T00:06:54.270 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T00:06:54.270 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b6b8723ba0) 2024-04-25T00:06:54.270 INFO:journalctl@ceph.mon.c.smithi153.stdout: cache_index_and_filter_blocks: 1 2024-04-25T00:06:54.270 INFO:journalctl@ceph.mon.c.smithi153.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T00:06:54.270 INFO:journalctl@ceph.mon.c.smithi153.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T00:06:54.270 INFO:journalctl@ceph.mon.c.smithi153.stdout: pin_top_level_index_and_filter: 1 2024-04-25T00:06:54.270 INFO:journalctl@ceph.mon.c.smithi153.stdout: index_type: 0 2024-04-25T00:06:54.270 INFO:journalctl@ceph.mon.c.smithi153.stdout: data_block_index_type: 0 2024-04-25T00:06:54.271 INFO:journalctl@ceph.mon.c.smithi153.stdout: index_shortening: 1 2024-04-25T00:06:54.271 INFO:journalctl@ceph.mon.c.smithi153.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T00:06:54.271 INFO:journalctl@ceph.mon.c.smithi153.stdout: checksum: 4 2024-04-25T00:06:54.271 INFO:journalctl@ceph.mon.c.smithi153.stdout: no_block_cache: 0 2024-04-25T00:06:54.271 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_cache: 0x55b6b8727090 2024-04-25T00:06:54.271 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_cache_name: BinnedLRUCache 2024-04-25T00:06:54.271 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_cache_options: 2024-04-25T00:06:54.271 INFO:journalctl@ceph.mon.c.smithi153.stdout: capacity : 536870912 2024-04-25T00:06:54.271 INFO:journalctl@ceph.mon.c.smithi153.stdout: num_shard_bits : 4 2024-04-25T00:06:54.272 INFO:journalctl@ceph.mon.c.smithi153.stdout: strict_capacity_limit : 0 2024-04-25T00:06:54.272 INFO:journalctl@ceph.mon.c.smithi153.stdout: high_pri_pool_ratio: 0.000 2024-04-25T00:06:54.272 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_cache_compressed: (nil) 2024-04-25T00:06:54.272 INFO:journalctl@ceph.mon.c.smithi153.stdout: persistent_cache: (nil) 2024-04-25T00:06:54.272 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_size: 4096 2024-04-25T00:06:54.272 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_size_deviation: 10 2024-04-25T00:06:54.272 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_restart_interval: 16 2024-04-25T00:06:54.272 INFO:journalctl@ceph.mon.c.smithi153.stdout: index_block_restart_interval: 1 2024-04-25T00:06:54.272 INFO:journalctl@ceph.mon.c.smithi153.stdout: metadata_block_size: 4096 2024-04-25T00:06:54.273 INFO:journalctl@ceph.mon.c.smithi153.stdout: partition_filters: 0 2024-04-25T00:06:54.273 INFO:journalctl@ceph.mon.c.smithi153.stdout: use_delta_encoding: 1 2024-04-25T00:06:54.273 INFO:journalctl@ceph.mon.c.smithi153.stdout: filter_policy: bloomfilter 2024-04-25T00:06:54.273 INFO:journalctl@ceph.mon.c.smithi153.stdout: whole_key_filtering: 1 2024-04-25T00:06:54.273 INFO:journalctl@ceph.mon.c.smithi153.stdout: verify_compression: 0 2024-04-25T00:06:54.273 INFO:journalctl@ceph.mon.c.smithi153.stdout: read_amp_bytes_per_bit: 0 2024-04-25T00:06:54.273 INFO:journalctl@ceph.mon.c.smithi153.stdout: format_version: 5 2024-04-25T00:06:54.273 INFO:journalctl@ceph.mon.c.smithi153.stdout: enable_index_compression: 1 2024-04-25T00:06:54.273 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_align: 0 2024-04-25T00:06:54.273 INFO:journalctl@ceph.mon.c.smithi153.stdout: max_auto_readahead_size: 262144 2024-04-25T00:06:54.274 INFO:journalctl@ceph.mon.c.smithi153.stdout: prepopulate_block_cache: 0 2024-04-25T00:06:54.274 INFO:journalctl@ceph.mon.c.smithi153.stdout: initial_auto_readahead_size: 8192 2024-04-25T00:06:54.274 INFO:journalctl@ceph.mon.c.smithi153.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T00:06:54.274 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T00:06:54.274 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T00:06:54.274 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compression: NoCompression 2024-04-25T00:06:54.274 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T00:06:54.274 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T00:06:54.274 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T00:06:54.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.num_levels: 7 2024-04-25T00:06:54.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T00:06:54.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T00:06:54.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T00:06:54.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T00:06:54.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T00:06:54.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T00:06:54.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T00:06:54.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T00:06:54.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T00:06:54.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T00:06:54.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T00:06:54.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T00:06:54.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T00:06:54.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T00:06:54.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T00:06:54.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T00:06:54.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T00:06:54.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T00:06:54.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T00:06:54.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T00:06:54.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T00:06:54.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T00:06:54.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T00:06:54.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T00:06:54.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T00:06:54.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T00:06:54.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T00:06:54.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T00:06:54.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T00:06:54.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T00:06:54.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T00:06:54.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T00:06:54.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T00:06:54.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T00:06:54.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T00:06:54.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T00:06:54.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T00:06:54.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T00:06:54.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T00:06:54.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T00:06:54.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T00:06:54.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T00:06:54.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T00:06:54.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T00:06:54.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T00:06:54.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T00:06:54.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T00:06:54.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T00:06:54.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T00:06:54.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T00:06:54.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T00:06:54.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T00:06:54.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T00:06:54.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.table_properties_collectors: 2024-04-25T00:06:54.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.inplace_update_support: 0 2024-04-25T00:06:54.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T00:06:54.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T00:06:54.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T00:06:54.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T00:06:54.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.bloom_locality: 0 2024-04-25T00:06:54.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.max_successive_merges: 0 2024-04-25T00:06:54.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T00:06:54.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T00:06:54.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T00:06:54.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T00:06:54.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.ttl: 2592000 2024-04-25T00:06:54.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T00:06:54.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T00:06:54.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T00:06:54.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.enable_blob_files: false 2024-04-25T00:06:54.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.min_blob_size: 0 2024-04-25T00:06:54.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T00:06:54.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T00:06:54.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T00:06:54.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T00:06:54.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T00:06:54.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T00:06:54.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T00:06:54.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T00:06:54.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-25T00:06:54.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-25T00:06:54.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: bd0c2855-b96a-4047-9364-fd6706bc92c8 2024-04-25T00:06:54.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714003613868631, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-25T00:06:54.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-25T00:06:54.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714003613869046, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714003613, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bd0c2855-b96a-4047-9364-fd6706bc92c8", "db_session_id": "QN67KC11VXWG61I7R6NA", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-25T00:06:54.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714003613869127, "job": 1, "event": "recovery_finished"} 2024-04-25T00:06:54.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-25T00:06:54.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T00:06:54.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b6b8818000 2024-04-25T00:06:54.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: DB pointer 0x55b6b8802000 2024-04-25T00:06:54.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T00:06:54.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T00:06:54.287 INFO:journalctl@ceph.mon.c.smithi153.stdout: ** DB Stats ** 2024-04-25T00:06:54.287 INFO:journalctl@ceph.mon.c.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T00:06:54.287 INFO:journalctl@ceph.mon.c.smithi153.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T00:06:54.287 INFO:journalctl@ceph.mon.c.smithi153.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T00:06:54.287 INFO:journalctl@ceph.mon.c.smithi153.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T00:06:54.287 INFO:journalctl@ceph.mon.c.smithi153.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T00:06:54.287 INFO:journalctl@ceph.mon.c.smithi153.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T00:06:54.287 INFO:journalctl@ceph.mon.c.smithi153.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T00:06:54.288 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-04-25T00:06:54.288 INFO:journalctl@ceph.mon.c.smithi153.stdout: ** Compaction Stats [default] ** 2024-04-25T00:06:54.288 INFO:journalctl@ceph.mon.c.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T00:06:54.288 INFO:journalctl@ceph.mon.c.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T00:06:54.288 INFO:journalctl@ceph.mon.c.smithi153.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T00:06:54.288 INFO:journalctl@ceph.mon.c.smithi153.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T00:06:54.288 INFO:journalctl@ceph.mon.c.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T00:06:54.288 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-04-25T00:06:54.288 INFO:journalctl@ceph.mon.c.smithi153.stdout: ** Compaction Stats [default] ** 2024-04-25T00:06:54.288 INFO:journalctl@ceph.mon.c.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T00:06:54.289 INFO:journalctl@ceph.mon.c.smithi153.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T00:06:54.289 INFO:journalctl@ceph.mon.c.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T00:06:54.289 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-04-25T00:06:54.289 INFO:journalctl@ceph.mon.c.smithi153.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T00:06:54.289 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-04-25T00:06:54.289 INFO:journalctl@ceph.mon.c.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T00:06:54.289 INFO:journalctl@ceph.mon.c.smithi153.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T00:06:54.289 INFO:journalctl@ceph.mon.c.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T00:06:54.290 INFO:journalctl@ceph.mon.c.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T00:06:54.290 INFO:journalctl@ceph.mon.c.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T00:06:54.290 INFO:journalctl@ceph.mon.c.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T00:06:54.290 INFO:journalctl@ceph.mon.c.smithi153.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T00:06:54.290 INFO:journalctl@ceph.mon.c.smithi153.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T00:06:54.290 INFO:journalctl@ceph.mon.c.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T00:06:54.290 INFO:journalctl@ceph.mon.c.smithi153.stdout: Block cache BinnedLRUCache@0x55b6b8727090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-25T00:06:54.290 INFO:journalctl@ceph.mon.c.smithi153.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-25T00:06:54.291 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-04-25T00:06:54.291 INFO:journalctl@ceph.mon.c.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T00:06:54.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-25T00:06:54.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: using public_addr v2:172.21.15.153:0/0 -> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-04-25T00:06:54.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: starting mon.c rank -1 at public addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] at bind addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:06:54.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: mon.c@-1(???) e0 preinit fsid 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:06:54.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: mon.c@-1(synchronizing).mds e1 new map 2024-04-25T00:06:54.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-25T00:06:54.291 INFO:journalctl@ceph.mon.c.smithi153.stdout: e1 2024-04-25T00:06:54.292 INFO:journalctl@ceph.mon.c.smithi153.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T00:06:54.292 INFO:journalctl@ceph.mon.c.smithi153.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T00:06:54.292 INFO:journalctl@ceph.mon.c.smithi153.stdout: legacy client fscid: -1 2024-04-25T00:06:54.292 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-04-25T00:06:54.292 INFO:journalctl@ceph.mon.c.smithi153.stdout: No filesystems configured 2024-04-25T00:06:54.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T00:06:54.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T00:06:54.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T00:06:54.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T00:06:54.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T00:06:54.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T00:06:54.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T00:06:54.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T00:06:54.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T00:06:54.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T00:06:54.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:54.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:06:54.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi139", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:06:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:06:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:06:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: Updating smithi139:/etc/ceph/ceph.conf 2024-04-25T00:06:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: Deploying cephadm binary to smithi153 2024-04-25T00:06:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: Updating smithi139:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:06:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:06:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:06:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: Updating smithi139:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T00:06:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: Added host smithi153 2024-04-25T00:06:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:06:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2628368798' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T00:06:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2628368798' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T00:06:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T00:06:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:06:54.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:06:54.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:06:54.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-25T00:06:54.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: Updating smithi153:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:06:54.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:06:54.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: Updating smithi153:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T00:06:54.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:54.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi139:172.21.15.139=b;smithi153:172.21.15.153=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:06:54.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi139:172.21.15.139=b;smithi153:172.21.15.153=c;count:3 2024-04-25T00:06:54.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:06:54.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:06:54.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:06:54.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:54.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T00:06:54.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:06:54.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: Deploying daemon mon.c on smithi153 2024-04-25T00:06:54.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:54.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:54.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:53 smithi153 ceph-mon[32896]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T00:06:54.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:54 smithi153 bash[32855]: d2ab362c99894eec4a1a10c18d01b9a614ddd0a45c9d92d6e54d27e9d0e7853e 2024-04-25T00:06:54.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:54 smithi153 systemd[1]: Started Ceph mon.c for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:06:55.467 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T00:06:55.467 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T00:06:55.712 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T00:06:59.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:59.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:06:59.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:06:59.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: mon.a calling monitor election 2024-04-25T00:06:59.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:06:59.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:59.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:06:59.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: mon.c calling monitor election 2024-04-25T00:06:59.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:06:59.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:59.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:06:59.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:06:59.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T00:06:59.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: monmap epoch 2 2024-04-25T00:06:59.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: fsid 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:06:59.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: last_changed 2024-04-25T00:06:53.893616+0000 2024-04-25T00:06:59.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: created 2024-04-25T00:05:07.441245+0000 2024-04-25T00:06:59.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: min_mon_release 19 (squid) 2024-04-25T00:06:59.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: election_strategy: 1 2024-04-25T00:06:59.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T00:06:59.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-25T00:06:59.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: fsmap 2024-04-25T00:06:59.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T00:06:59.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: mgrmap e13: x(active, since 49s) 2024-04-25T00:06:59.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: overall HEALTH_OK 2024-04-25T00:06:59.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:59.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:06:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:06:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: mon.a calling monitor election 2024-04-25T00:06:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:06:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:06:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: mon.c calling monitor election 2024-04-25T00:06:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:06:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:06:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:06:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:06:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T00:06:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: monmap epoch 2 2024-04-25T00:06:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: fsid 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:06:59.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: last_changed 2024-04-25T00:06:53.893616+0000 2024-04-25T00:06:59.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: created 2024-04-25T00:05:07.441245+0000 2024-04-25T00:06:59.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: min_mon_release 19 (squid) 2024-04-25T00:06:59.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: election_strategy: 1 2024-04-25T00:06:59.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T00:06:59.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-25T00:06:59.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: fsmap 2024-04-25T00:06:59.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T00:06:59.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: mgrmap e13: x(active, since 49s) 2024-04-25T00:06:59.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: overall HEALTH_OK 2024-04-25T00:06:59.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:06:59.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:00.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.223+0000 7f7b4a8cf700 1 -- 172.21.15.153:0/324582141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b300035c0 msgr2=0x7f7b30005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:00.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.223+0000 7f7b4a8cf700 1 --2- 172.21.15.153:0/324582141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b300035c0 0x7f7b30005a50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7b44079470 tx=0x7f7b3400feb0 comp rx=0 tx=0).stop 2024-04-25T00:07:00.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.223+0000 7f7b4a8cf700 1 -- 172.21.15.153:0/324582141 shutdown_connections 2024-04-25T00:07:00.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.223+0000 7f7b4a8cf700 1 --2- 172.21.15.153:0/324582141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b300035c0 0x7f7b30005a50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:00.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.223+0000 7f7b4a8cf700 1 -- 172.21.15.153:0/324582141 >> 172.21.15.153:0/324582141 conn(0x7f7b440772e0 msgr2=0x7f7b440776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:00.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.223+0000 7f7b4a8cf700 1 -- 172.21.15.153:0/324582141 shutdown_connections 2024-04-25T00:07:00.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.223+0000 7f7b4a8cf700 1 -- 172.21.15.153:0/324582141 wait complete. 2024-04-25T00:07:00.225 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:00.225 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T00:07:00.225 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:00.225 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:59 smithi153 ceph-mon[32896]: Deploying daemon mon.b on smithi139 2024-04-25T00:07:00.225 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:06:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:07:00.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.224+0000 7f7b4a8cf700 1 Processor -- start 2024-04-25T00:07:00.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.224+0000 7f7b4a8cf700 1 -- start start 2024-04-25T00:07:00.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.225+0000 7f7b4a8cf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b300035c0 0x7f7b4419d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:00.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.225+0000 7f7b4a8cf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b4419dbd0 0x7f7b44197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:00.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.225+0000 7f7b4a8cf700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7b44109de0 con 0x7f7b4419dbd0 2024-04-25T00:07:00.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.225+0000 7f7b4a8cf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7b44109c60 con 0x7f7b300035c0 2024-04-25T00:07:00.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.225+0000 7f7b43fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b300035c0 0x7f7b4419d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:00.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.225+0000 7f7b43fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b300035c0 0x7f7b4419d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38282/0 (socket says 172.21.15.153:38282) 2024-04-25T00:07:00.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.225+0000 7f7b43fff700 1 -- 172.21.15.153:0/3164217050 learned_addr learned my addr 172.21.15.153:0/3164217050 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T00:07:00.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.225+0000 7f7b43fff700 1 -- 172.21.15.153:0/3164217050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b300035c0 msgr2=0x7f7b4419d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-25T00:07:00.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.225+0000 7f7b43fff700 1 -- 172.21.15.153:0/3164217050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b300035c0 msgr2=0x7f7b4419d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T00:07:00.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.225+0000 7f7b43fff700 1 --2- 172.21.15.153:0/3164217050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b300035c0 0x7f7b4419d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T00:07:00.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.225+0000 7f7b437fe700 1 --2- 172.21.15.153:0/3164217050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b4419dbd0 0x7f7b44197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:00.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.225+0000 7f7b43fff700 1 --2- 172.21.15.153:0/3164217050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b300035c0 0x7f7b4419d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T00:07:00.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.226+0000 7f7b437fe700 1 -- 172.21.15.153:0/3164217050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b300035c0 msgr2=0x7f7b4419d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:07:00.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.226+0000 7f7b437fe700 1 --2- 172.21.15.153:0/3164217050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b300035c0 0x7f7b4419d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:00.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.226+0000 7f7b437fe700 1 -- 172.21.15.153:0/3164217050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b4419e150 con 0x7f7b4419dbd0 2024-04-25T00:07:00.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.226+0000 7f7b437fe700 1 --2- 172.21.15.153:0/3164217050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b4419dbd0 0x7f7b44197760 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00b4d0 tx=0x7f7b2c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:00.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.226+0000 7f7b417fa700 1 -- 172.21.15.153:0/3164217050 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7b2c011630 con 0x7f7b4419dbd0 2024-04-25T00:07:00.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.226+0000 7f7b417fa700 1 -- 172.21.15.153:0/3164217050 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b2c011cb0 con 0x7f7b4419dbd0 2024-04-25T00:07:00.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.226+0000 7f7b4a8cf700 1 -- 172.21.15.153:0/3164217050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b44197d00 con 0x7f7b4419dbd0 2024-04-25T00:07:00.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.226+0000 7f7b417fa700 1 -- 172.21.15.153:0/3164217050 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7b2c010e60 con 0x7f7b4419dbd0 2024-04-25T00:07:00.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.226+0000 7f7b4a8cf700 1 -- 172.21.15.153:0/3164217050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b44198290 con 0x7f7b4419dbd0 2024-04-25T00:07:00.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.227+0000 7f7b417fa700 1 -- 172.21.15.153:0/3164217050 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f7b2c018020 con 0x7f7b4419dbd0 2024-04-25T00:07:00.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.227+0000 7f7b4a8cf700 1 -- 172.21.15.153:0/3164217050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b4406a190 con 0x7f7b4419dbd0 2024-04-25T00:07:00.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.227+0000 7f7b417fa700 1 --2- 172.21.15.153:0/3164217050 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7b24038e90 0x7f7b2403b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:00.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.227+0000 7f7b417fa700 1 -- 172.21.15.153:0/3164217050 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7b2c0117d0 con 0x7f7b4419dbd0 2024-04-25T00:07:00.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.228+0000 7f7b43fff700 1 --2- 172.21.15.153:0/3164217050 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7b24038e90 0x7f7b2403b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:00.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.228+0000 7f7b43fff700 1 --2- 172.21.15.153:0/3164217050 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7b24038e90 0x7f7b2403b350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400fa60 tx=0x7f7b34005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:00.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.233+0000 7f7b417fa700 1 -- 172.21.15.153:0/3164217050 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7b2c01f020 con 0x7f7b4419dbd0 2024-04-25T00:07:00.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:00.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T00:07:00.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:00.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:59 smithi027 ceph-mon[29208]: Deploying daemon mon.b on smithi139 2024-04-25T00:07:00.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:06:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:07:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.513+0000 7f7b4a8cf700 1 -- 172.21.15.153:0/3164217050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f7b44198600 con 0x7f7b4419dbd0 2024-04-25T00:07:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.514+0000 7f7b417fa700 1 -- 172.21.15.153:0/3164217050 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f7b2c018570 con 0x7f7b4419dbd0 2024-04-25T00:07:00.515 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:07:00.515 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":2,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","modified":"2024-04-25T00:06:53.893616Z","created":"2024-04-25T00:05:07.441245Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T00:07:00.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.516+0000 7f7b4a8cf700 1 -- 172.21.15.153:0/3164217050 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7b24038e90 msgr2=0x7f7b2403b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:00.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.516+0000 7f7b4a8cf700 1 --2- 172.21.15.153:0/3164217050 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7b24038e90 0x7f7b2403b350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400fa60 tx=0x7f7b34005960 comp rx=0 tx=0).stop 2024-04-25T00:07:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.516+0000 7f7b4a8cf700 1 -- 172.21.15.153:0/3164217050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b4419dbd0 msgr2=0x7f7b44197760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.516+0000 7f7b4a8cf700 1 --2- 172.21.15.153:0/3164217050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b4419dbd0 0x7f7b44197760 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00b4d0 tx=0x7f7b2c00b890 comp rx=0 tx=0).stop 2024-04-25T00:07:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.516+0000 7f7b4a8cf700 1 -- 172.21.15.153:0/3164217050 shutdown_connections 2024-04-25T00:07:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.516+0000 7f7b4a8cf700 1 --2- 172.21.15.153:0/3164217050 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7b24038e90 0x7f7b2403b350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.516+0000 7f7b4a8cf700 1 --2- 172.21.15.153:0/3164217050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b300035c0 0x7f7b4419d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.516+0000 7f7b4a8cf700 1 --2- 172.21.15.153:0/3164217050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b4419dbd0 0x7f7b44197760 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.516+0000 7f7b4a8cf700 1 -- 172.21.15.153:0/3164217050 >> 172.21.15.153:0/3164217050 conn(0x7f7b440772e0 msgr2=0x7f7b440fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.516+0000 7f7b4a8cf700 1 -- 172.21.15.153:0/3164217050 shutdown_connections 2024-04-25T00:07:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:00.516+0000 7f7b4a8cf700 1 -- 172.21.15.153:0/3164217050 wait complete. 2024-04-25T00:07:00.518 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 2 2024-04-25T00:07:01.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:00 smithi153 ceph-mon[32896]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:01.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:00 smithi153 ceph-mon[32896]: from='client.? 172.21.15.153:0/3164217050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T00:07:01.343 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:00 smithi027 ceph-mon[29208]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:01.343 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:00 smithi027 ceph-mon[29208]: from='client.? 172.21.15.153:0/3164217050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T00:07:02.097 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T00:07:02.097 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T00:07:02.341 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T00:07:03.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:02 smithi153 ceph-mon[32896]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:03.092 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:03 smithi139 ceph-mon[32833]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T00:07:03.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:02 smithi027 ceph-mon[29208]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:03.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:03 smithi139 bash[32814]: cb23fccae8acc7a35c7afd0e3837e82c5f9c73c7f8af24681dabf23273f60831 2024-04-25T00:07:03.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:03 smithi139 systemd[1]: Started Ceph mon.b for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:07:08.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.208+0000 7f00cb987700 1 -- 172.21.15.153:0/3636204848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0035c0 msgr2=0x7f00ac005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:08.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.208+0000 7f00cb987700 1 --2- 172.21.15.153:0/3636204848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0035c0 0x7f00ac005a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f00c4069bd0 tx=0x7f00bc00fe60 comp rx=0 tx=0).stop 2024-04-25T00:07:08.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.208+0000 7f00cb987700 1 -- 172.21.15.153:0/3636204848 shutdown_connections 2024-04-25T00:07:08.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.208+0000 7f00cb987700 1 --2- 172.21.15.153:0/3636204848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0035c0 0x7f00ac005a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:08.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.208+0000 7f00cb987700 1 -- 172.21.15.153:0/3636204848 >> 172.21.15.153:0/3636204848 conn(0x7f00c40fc710 msgr2=0x7f00c40feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:08.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.209+0000 7f00cb987700 1 -- 172.21.15.153:0/3636204848 shutdown_connections 2024-04-25T00:07:08.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.209+0000 7f00cb987700 1 -- 172.21.15.153:0/3636204848 wait complete. 2024-04-25T00:07:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.209+0000 7f00cb987700 1 Processor -- start 2024-04-25T00:07:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.209+0000 7f00cb987700 1 -- start start 2024-04-25T00:07:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00cb987700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0035c0 0x7f00c4100c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00cb987700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c4101140 0x7f00c41015c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00cb987700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00c41a3c00 0x7f00c41a5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00cb987700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f00c4109d60 con 0x7f00ac0035c0 2024-04-25T00:07:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00cb987700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f00c4109be0 con 0x7f00c41a3c00 2024-04-25T00:07:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00cb987700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f00c4109ee0 con 0x7f00c4101140 2024-04-25T00:07:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c8f22700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c4101140 0x7f00c41015c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:08.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c8f22700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c4101140 0x7f00c41015c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58908/0 (socket says 172.21.15.153:58908) 2024-04-25T00:07:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c8f22700 1 -- 172.21.15.153:0/3019581030 learned_addr learned my addr 172.21.15.153:0/3019581030 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T00:07:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c9723700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0035c0 0x7f00c4100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c9f24700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00c41a3c00 0x7f00c41a5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c8f22700 1 -- 172.21.15.153:0/3019581030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c4101140 msgr2=0x7f00c41015c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T00:07:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c8f22700 1 -- 172.21.15.153:0/3019581030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c4101140 msgr2=0x7f00c41015c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T00:07:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c8f22700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c4101140 0x7f00c41015c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T00:07:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c8f22700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c4101140 0x7f00c41015c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T00:07:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c9f24700 1 -- 172.21.15.153:0/3019581030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00c41a3c00 msgr2=0x7f00c41a5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-25T00:07:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c9f24700 1 -- 172.21.15.153:0/3019581030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00c41a3c00 msgr2=0x7f00c41a5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T00:07:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c9f24700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00c41a3c00 0x7f00c41a5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T00:07:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c9f24700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00c41a3c00 0x7f00c41a5ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T00:07:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c9723700 1 -- 172.21.15.153:0/3019581030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c4101140 msgr2=0x7f00c41015c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:07:08.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c9723700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c4101140 0x7f00c41015c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c9723700 1 -- 172.21.15.153:0/3019581030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00c41a3c00 msgr2=0x7f00c41a5ff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:07:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c9723700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00c41a3c00 0x7f00c41a5ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.210+0000 7f00c9723700 1 -- 172.21.15.153:0/3019581030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c41a6720 con 0x7f00ac0035c0 2024-04-25T00:07:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.211+0000 7f00c9723700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0035c0 0x7f00c4100c00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f00bc01f000 tx=0x7f00bc005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.211+0000 7f00c27fc700 1 -- 172.21.15.153:0/3019581030 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc02b020 con 0x7f00ac0035c0 2024-04-25T00:07:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.211+0000 7f00c27fc700 1 -- 172.21.15.153:0/3019581030 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00bc004110 con 0x7f00ac0035c0 2024-04-25T00:07:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.211+0000 7f00cb987700 1 -- 172.21.15.153:0/3019581030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c41a69b0 con 0x7f00ac0035c0 2024-04-25T00:07:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.211+0000 7f00c27fc700 1 -- 172.21.15.153:0/3019581030 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc022040 con 0x7f00ac0035c0 2024-04-25T00:07:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.211+0000 7f00cb987700 1 -- 172.21.15.153:0/3019581030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c41a6ef0 con 0x7f00ac0035c0 2024-04-25T00:07:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.212+0000 7f00c27fc700 1 -- 172.21.15.153:0/3019581030 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f00bc02e020 con 0x7f00ac0035c0 2024-04-25T00:07:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.212+0000 7f00c27fc700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f00b0039580 0x7f00b003ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.212+0000 7f00cb987700 1 -- 172.21.15.153:0/3019581030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00c4063890 con 0x7f00ac0035c0 2024-04-25T00:07:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.212+0000 7f00c27fc700 1 -- 172.21.15.153:0/3019581030 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f00bc04ec30 con 0x7f00ac0035c0 2024-04-25T00:07:08.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.212+0000 7f00c8f22700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f00b0039580 0x7f00b003ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:08.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.213+0000 7f00c8f22700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f00b0039580 0x7f00b003ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f00b8009400 tx=0x7f00b8007040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:08.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.218+0000 7f00c27fc700 1 -- 172.21.15.153:0/3019581030 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f00bc035020 con 0x7f00ac0035c0 2024-04-25T00:07:08.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:07:08.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: mon.a calling monitor election 2024-04-25T00:07:08.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: mon.c calling monitor election 2024-04-25T00:07:08.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:07:08.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:08.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: mon.b calling monitor election 2024-04-25T00:07:08.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:08.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T00:07:08.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: monmap epoch 3 2024-04-25T00:07:08.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: fsid 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:07:08.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: last_changed 2024-04-25T00:07:03.068854+0000 2024-04-25T00:07:08.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: created 2024-04-25T00:05:07.441245+0000 2024-04-25T00:07:08.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: min_mon_release 19 (squid) 2024-04-25T00:07:08.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: election_strategy: 1 2024-04-25T00:07:08.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T00:07:08.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-25T00:07:08.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: 2: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.b 2024-04-25T00:07:08.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: fsmap 2024-04-25T00:07:08.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T00:07:08.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: mgrmap e13: x(active, since 58s) 2024-04-25T00:07:08.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: overall HEALTH_OK 2024-04-25T00:07:08.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:08.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:08.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:08.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:08.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:07:08.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: mon.a calling monitor election 2024-04-25T00:07:08.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: mon.c calling monitor election 2024-04-25T00:07:08.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:07:08.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:08.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: mon.b calling monitor election 2024-04-25T00:07:08.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:08.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T00:07:08.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: monmap epoch 3 2024-04-25T00:07:08.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: fsid 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:07:08.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: last_changed 2024-04-25T00:07:03.068854+0000 2024-04-25T00:07:08.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: created 2024-04-25T00:05:07.441245+0000 2024-04-25T00:07:08.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: min_mon_release 19 (squid) 2024-04-25T00:07:08.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: election_strategy: 1 2024-04-25T00:07:08.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T00:07:08.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-25T00:07:08.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: 2: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.b 2024-04-25T00:07:08.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: fsmap 2024-04-25T00:07:08.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T00:07:08.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: mgrmap e13: x(active, since 58s) 2024-04-25T00:07:08.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: overall HEALTH_OK 2024-04-25T00:07:08.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:08.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:08.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:08.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:08.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.511+0000 7f00cb987700 1 -- 172.21.15.153:0/3019581030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f00c41a7190 con 0x7f00ac0035c0 2024-04-25T00:07:08.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.512+0000 7f00c27fc700 1 -- 172.21.15.153:0/3019581030 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1285 (secure 0 0 0) 0x7f00bc010c90 con 0x7f00ac0035c0 2024-04-25T00:07:08.512 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:07:08.513 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.513 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: mon.a calling monitor election 2024-04-25T00:07:08.513 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: mon.c calling monitor election 2024-04-25T00:07:08.513 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:07:08.513 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:08.513 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.513 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.513 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: mon.b calling monitor election 2024-04-25T00:07:08.513 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:08.513 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.513 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.513 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:08.513 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T00:07:08.514 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: monmap epoch 3 2024-04-25T00:07:08.514 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: fsid 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:07:08.514 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: last_changed 2024-04-25T00:07:03.068854+0000 2024-04-25T00:07:08.514 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: created 2024-04-25T00:05:07.441245+0000 2024-04-25T00:07:08.514 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: min_mon_release 19 (squid) 2024-04-25T00:07:08.514 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: election_strategy: 1 2024-04-25T00:07:08.514 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T00:07:08.514 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-25T00:07:08.514 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: 2: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.b 2024-04-25T00:07:08.514 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: fsmap 2024-04-25T00:07:08.514 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T00:07:08.514 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: mgrmap e13: x(active, since 58s) 2024-04-25T00:07:08.514 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: overall HEALTH_OK 2024-04-25T00:07:08.515 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:08.515 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:08.515 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:08.515 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:08.515 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T00:07:08.515 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":3,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","modified":"2024-04-25T00:07:03.068854Z","created":"2024-04-25T00:05:07.441245Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-25T00:07:08.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.514+0000 7f00cb987700 1 -- 172.21.15.153:0/3019581030 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f00b0039580 msgr2=0x7f00b003ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:08.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.514+0000 7f00cb987700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f00b0039580 0x7f00b003ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f00b8009400 tx=0x7f00b8007040 comp rx=0 tx=0).stop 2024-04-25T00:07:08.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.514+0000 7f00cb987700 1 -- 172.21.15.153:0/3019581030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0035c0 msgr2=0x7f00c4100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:08.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.514+0000 7f00cb987700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0035c0 0x7f00c4100c00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f00bc01f000 tx=0x7f00bc005720 comp rx=0 tx=0).stop 2024-04-25T00:07:08.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.514+0000 7f00cb987700 1 -- 172.21.15.153:0/3019581030 shutdown_connections 2024-04-25T00:07:08.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.514+0000 7f00cb987700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f00b0039580 0x7f00b003ba40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:08.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.514+0000 7f00cb987700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00c41a3c00 0x7f00c41a5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:08.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.514+0000 7f00cb987700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0035c0 0x7f00c4100c00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.514+0000 7f00cb987700 1 --2- 172.21.15.153:0/3019581030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c4101140 0x7f00c41015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.514+0000 7f00cb987700 1 -- 172.21.15.153:0/3019581030 >> 172.21.15.153:0/3019581030 conn(0x7f00c40fc710 msgr2=0x7f00c4103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.514+0000 7f00cb987700 1 -- 172.21.15.153:0/3019581030 shutdown_connections 2024-04-25T00:07:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:08.514+0000 7f00cb987700 1 -- 172.21.15.153:0/3019581030 wait complete. 2024-04-25T00:07:08.517 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 3 2024-04-25T00:07:09.104 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-25T00:07:09.105 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-25T00:07:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:07:09.406 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:09 smithi153 ceph-mon[32896]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:09.407 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:09 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:09.407 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:09 smithi153 ceph-mon[32896]: from='client.? 172.21.15.153:0/3019581030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T00:07:09.407 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:09 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:09.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:09 smithi027 ceph-mon[29208]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:09.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:09 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:09.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:09 smithi027 ceph-mon[29208]: from='client.? 172.21.15.153:0/3019581030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T00:07:09.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:09 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:09.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:09 smithi139 ceph-mon[32833]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:09.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:09 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:09.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:09 smithi139 ceph-mon[32833]: from='client.? 172.21.15.153:0/3019581030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T00:07:09.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:09 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:07:10.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:10.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:10.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:10 smithi139 ceph-mon[32833]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:10.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:10.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:10.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:10.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:10.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:10.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:10 smithi139 ceph-mon[32833]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T00:07:10.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:10 smithi139 ceph-mon[32833]: Updating smithi139:/etc/ceph/ceph.conf 2024-04-25T00:07:10.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:10 smithi139 ceph-mon[32833]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-25T00:07:10.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:10 smithi139 ceph-mon[32833]: Updating smithi027:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:07:10.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:10 smithi139 ceph-mon[32833]: Updating smithi139:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:07:10.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:10 smithi139 ceph-mon[32833]: Updating smithi153:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:07:10.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:10.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:10.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:10 smithi153 ceph-mon[32896]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:10.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:10.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:10.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:10.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:10.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:10.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:10 smithi153 ceph-mon[32896]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T00:07:10.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:10 smithi153 ceph-mon[32896]: Updating smithi139:/etc/ceph/ceph.conf 2024-04-25T00:07:10.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:10 smithi153 ceph-mon[32896]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-25T00:07:10.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:10 smithi153 ceph-mon[32896]: Updating smithi027:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:07:10.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:10 smithi153 ceph-mon[32896]: Updating smithi139:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:07:10.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:10 smithi153 ceph-mon[32896]: Updating smithi153:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:07:10.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:10.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:10.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:10 smithi027 ceph-mon[29208]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:10.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:10.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:10.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:10.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:10.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:10.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:10 smithi027 ceph-mon[29208]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T00:07:10.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:10 smithi027 ceph-mon[29208]: Updating smithi139:/etc/ceph/ceph.conf 2024-04-25T00:07:10.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:10 smithi027 ceph-mon[29208]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-25T00:07:10.816 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:10 smithi027 ceph-mon[29208]: Updating smithi027:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:07:10.816 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:10 smithi027 ceph-mon[29208]: Updating smithi139:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:07:10.816 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:10 smithi027 ceph-mon[29208]: Updating smithi153:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:07:11.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.126+0000 7fa658c07700 1 -- 172.21.15.27:0/1513702639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa654102b10 msgr2=0x7fa654102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:11.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.126+0000 7fa658c07700 1 --2- 172.21.15.27:0/1513702639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa654102b10 0x7fa654102ef0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa644009910 tx=0x7fa644009c20 comp rx=0 tx=0).stop 2024-04-25T00:07:11.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.126+0000 7fa658c07700 1 -- 172.21.15.27:0/1513702639 shutdown_connections 2024-04-25T00:07:11.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.126+0000 7fa658c07700 1 --2- 172.21.15.27:0/1513702639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa654102b10 0x7fa654102ef0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:11.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.126+0000 7fa658c07700 1 -- 172.21.15.27:0/1513702639 >> 172.21.15.27:0/1513702639 conn(0x7fa6540fbda0 msgr2=0x7fa6540fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:11.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.126+0000 7fa658c07700 1 -- 172.21.15.27:0/1513702639 shutdown_connections 2024-04-25T00:07:11.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.126+0000 7fa658c07700 1 -- 172.21.15.27:0/1513702639 wait complete. 2024-04-25T00:07:11.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.127+0000 7fa658c07700 1 Processor -- start 2024-04-25T00:07:11.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.127+0000 7fa658c07700 1 -- start start 2024-04-25T00:07:11.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.127+0000 7fa658c07700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa654102b10 0x7fa654198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:11.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.127+0000 7fa658c07700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa654198ea0 0x7fa654199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:11.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.127+0000 7fa658c07700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6541a19d0 0x7fa6541a3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:11.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.127+0000 7fa658c07700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa65410b300 con 0x7fa654102b10 2024-04-25T00:07:11.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.127+0000 7fa658c07700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa65410b180 con 0x7fa6541a19d0 2024-04-25T00:07:11.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.127+0000 7fa658c07700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa65410b480 con 0x7fa654198ea0 2024-04-25T00:07:11.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.127+0000 7fa65259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa654102b10 0x7fa654198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:11.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.127+0000 7fa65259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa654102b10 0x7fa654198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60108/0 (socket says 172.21.15.27:60108) 2024-04-25T00:07:11.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.127+0000 7fa65259c700 1 -- 172.21.15.27:0/4256851830 learned_addr learned my addr 172.21.15.27:0/4256851830 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:07:11.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.127+0000 7fa651d9b700 1 --2- 172.21.15.27:0/4256851830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa654198ea0 0x7fa654199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:11.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.128+0000 7fa652d9d700 1 --2- 172.21.15.27:0/4256851830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6541a19d0 0x7fa6541a3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:11.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.128+0000 7fa65259c700 1 -- 172.21.15.27:0/4256851830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa654198ea0 msgr2=0x7fa654199320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:11.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.128+0000 7fa65259c700 1 --2- 172.21.15.27:0/4256851830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa654198ea0 0x7fa654199320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:11.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.128+0000 7fa65259c700 1 -- 172.21.15.27:0/4256851830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6541a19d0 msgr2=0x7fa6541a3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:11.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.128+0000 7fa65259c700 1 --2- 172.21.15.27:0/4256851830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6541a19d0 0x7fa6541a3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:11.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.128+0000 7fa65259c700 1 -- 172.21.15.27:0/4256851830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6541a45d0 con 0x7fa654102b10 2024-04-25T00:07:11.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.128+0000 7fa65259c700 1 --2- 172.21.15.27:0/4256851830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa654102b10 0x7fa654198960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa64400bdc0 tx=0x7fa64400bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:11.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.128+0000 7fa6437fe700 1 -- 172.21.15.27:0/4256851830 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa644027070 con 0x7fa654102b10 2024-04-25T00:07:11.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.128+0000 7fa658c07700 1 -- 172.21.15.27:0/4256851830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6541a4860 con 0x7fa654102b10 2024-04-25T00:07:11.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.128+0000 7fa6437fe700 1 -- 172.21.15.27:0/4256851830 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6440050d0 con 0x7fa654102b10 2024-04-25T00:07:11.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.128+0000 7fa6437fe700 1 -- 172.21.15.27:0/4256851830 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa644022dc0 con 0x7fa654102b10 2024-04-25T00:07:11.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.128+0000 7fa658c07700 1 -- 172.21.15.27:0/4256851830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6541a4cd0 con 0x7fa654102b10 2024-04-25T00:07:11.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.129+0000 7fa6437fe700 1 -- 172.21.15.27:0/4256851830 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fa64402a070 con 0x7fa654102b10 2024-04-25T00:07:11.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.129+0000 7fa6437fe700 1 --2- 172.21.15.27:0/4256851830 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa63c039580 0x7fa63c03ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:11.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.129+0000 7fa6437fe700 1 -- 172.21.15.27:0/4256851830 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa644022470 con 0x7fa654102b10 2024-04-25T00:07:11.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.130+0000 7fa658c07700 1 -- 172.21.15.27:0/4256851830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa654063890 con 0x7fa654102b10 2024-04-25T00:07:11.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.130+0000 7fa651d9b700 1 --2- 172.21.15.27:0/4256851830 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa63c039580 0x7fa63c03ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:11.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.130+0000 7fa651d9b700 1 --2- 172.21.15.27:0/4256851830 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa63c039580 0x7fa63c03ba40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa648009ad0 tx=0x7fa648009210 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:11.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.135+0000 7fa6437fe700 1 -- 172.21.15.27:0/4256851830 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa644013070 con 0x7fa654102b10 2024-04-25T00:07:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.350+0000 7fa658c07700 1 -- 172.21.15.27:0/4256851830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fa6541a4f90 con 0x7fa654102b10 2024-04-25T00:07:11.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.351+0000 7fa6437fe700 1 -- 172.21.15.27:0/4256851830 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7fa644031020 con 0x7fa654102b10 2024-04-25T00:07:11.351 INFO:teuthology.orchestra.run.smithi027.stdout:# minimal ceph.conf for 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:07:11.351 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-04-25T00:07:11.351 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:07:11.351 INFO:teuthology.orchestra.run.smithi027.stdout: mon_host = [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-04-25T00:07:11.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.353+0000 7fa658c07700 1 -- 172.21.15.27:0/4256851830 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa63c039580 msgr2=0x7fa63c03ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:11.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.353+0000 7fa658c07700 1 --2- 172.21.15.27:0/4256851830 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa63c039580 0x7fa63c03ba40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa648009ad0 tx=0x7fa648009210 comp rx=0 tx=0).stop 2024-04-25T00:07:11.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.353+0000 7fa658c07700 1 -- 172.21.15.27:0/4256851830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa654102b10 msgr2=0x7fa654198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:11.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.353+0000 7fa658c07700 1 --2- 172.21.15.27:0/4256851830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa654102b10 0x7fa654198960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa64400bdc0 tx=0x7fa64400bdf0 comp rx=0 tx=0).stop 2024-04-25T00:07:11.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.353+0000 7fa658c07700 1 -- 172.21.15.27:0/4256851830 shutdown_connections 2024-04-25T00:07:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.353+0000 7fa658c07700 1 --2- 172.21.15.27:0/4256851830 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa63c039580 0x7fa63c03ba40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.353+0000 7fa658c07700 1 --2- 172.21.15.27:0/4256851830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6541a19d0 0x7fa6541a3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.353+0000 7fa658c07700 1 --2- 172.21.15.27:0/4256851830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa654102b10 0x7fa654198960 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.353+0000 7fa658c07700 1 --2- 172.21.15.27:0/4256851830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa654198ea0 0x7fa654199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.353+0000 7fa658c07700 1 -- 172.21.15.27:0/4256851830 >> 172.21.15.27:0/4256851830 conn(0x7fa6540fbda0 msgr2=0x7fa6541038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.353+0000 7fa658c07700 1 -- 172.21.15.27:0/4256851830 shutdown_connections 2024-04-25T00:07:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:11.353+0000 7fa658c07700 1 -- 172.21.15.27:0/4256851830 wait complete. 2024-04-25T00:07:11.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:11 smithi027 ceph-mon[29208]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T00:07:11.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T00:07:11.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T00:07:11.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:11.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:11 smithi027 ceph-mon[29208]: Reconfiguring daemon mon.a on smithi027 2024-04-25T00:07:11.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:11 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4256851830' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:11.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:11 smithi153 ceph-mon[32896]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T00:07:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T00:07:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T00:07:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:11 smithi153 ceph-mon[32896]: Reconfiguring daemon mon.a on smithi027 2024-04-25T00:07:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:11 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4256851830' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:12.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:12.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:12.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:12.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:12.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:12.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:12.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:12.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:12.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:12.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:12.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:12.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:11 smithi139 ceph-mon[32833]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T00:07:12.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T00:07:12.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T00:07:12.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:12.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:11 smithi139 ceph-mon[32833]: Reconfiguring daemon mon.a on smithi027 2024-04-25T00:07:12.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:11 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4256851830' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:12.389 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-25T00:07:12.389 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:07:12.389 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T00:07:12.419 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:07:12.420 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:07:12.486 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T00:07:12.487 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T00:07:12.518 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T00:07:12.518 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:07:12.587 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T00:07:12.587 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T00:07:12.618 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T00:07:12.618 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:07:12.686 INFO:tasks.cephadm:Adding mgr.x on smithi027 2024-04-25T00:07:12.686 INFO:tasks.cephadm:Adding mgr.y on smithi139 2024-04-25T00:07:12.686 INFO:tasks.cephadm:Adding mgr.z on smithi153 2024-04-25T00:07:12.686 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi027=x;smithi139=y;smithi153=z' 2024-04-25T00:07:12.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:12 smithi027 ceph-mon[29208]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:12.962 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T00:07:13.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:12 smithi139 ceph-mon[32833]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:13.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:12 smithi153 ceph-mon[32896]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:14.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:13 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:14.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:13 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:14.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:13 smithi139 ceph-mon[32833]: Reconfiguring mon.b (monmap changed)... 2024-04-25T00:07:14.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:13 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T00:07:14.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:13 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T00:07:14.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:13 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:14.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:13 smithi139 ceph-mon[32833]: Reconfiguring daemon mon.b on smithi139 2024-04-25T00:07:14.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:13 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:14.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:13 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:14.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:13 smithi153 ceph-mon[32896]: Reconfiguring mon.b (monmap changed)... 2024-04-25T00:07:14.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:13 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T00:07:14.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:13 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T00:07:14.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:13 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:14.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:13 smithi153 ceph-mon[32896]: Reconfiguring daemon mon.b on smithi139 2024-04-25T00:07:14.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:13 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:14.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:13 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:14.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:13 smithi027 ceph-mon[29208]: Reconfiguring mon.b (monmap changed)... 2024-04-25T00:07:14.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:13 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T00:07:14.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:13 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T00:07:14.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:13 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:14.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:13 smithi027 ceph-mon[29208]: Reconfiguring daemon mon.b on smithi139 2024-04-25T00:07:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.218+0000 7faff2917700 1 -- 172.21.15.153:0/431094506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafec102820 msgr2=0x7fafec104c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.218+0000 7faff2917700 1 --2- 172.21.15.153:0/431094506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafec102820 0x7fafec104c10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009910 tx=0x7fafdc009c20 comp rx=0 tx=0).stop 2024-04-25T00:07:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.218+0000 7faff2917700 1 -- 172.21.15.153:0/431094506 shutdown_connections 2024-04-25T00:07:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.218+0000 7faff2917700 1 --2- 172.21.15.153:0/431094506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafec102820 0x7fafec104c10 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.218+0000 7faff2917700 1 -- 172.21.15.153:0/431094506 >> 172.21.15.153:0/431094506 conn(0x7fafec0fc230 msgr2=0x7fafec0fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.218+0000 7faff2917700 1 -- 172.21.15.153:0/431094506 shutdown_connections 2024-04-25T00:07:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.218+0000 7faff2917700 1 -- 172.21.15.153:0/431094506 wait complete. 2024-04-25T00:07:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.219+0000 7faff2917700 1 Processor -- start 2024-04-25T00:07:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.219+0000 7faff2917700 1 -- start start 2024-04-25T00:07:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.219+0000 7faff2917700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fafec102820 0x7fafec102180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.219+0000 7faff2917700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafec1007d0 0x7fafec100c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.219+0000 7faff2917700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafec101190 0x7fafec1a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.219+0000 7faff2917700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fafec107d70 con 0x7fafec101190 2024-04-25T00:07:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.219+0000 7faff2917700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fafec107bf0 con 0x7fafec102820 2024-04-25T00:07:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.219+0000 7faff2917700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fafec107ef0 con 0x7fafec1007d0 2024-04-25T00:07:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.220+0000 7fafeb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafec1007d0 0x7fafec100c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.220+0000 7fafeb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafec1007d0 0x7fafec100c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58912/0 (socket says 172.21.15.153:58912) 2024-04-25T00:07:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.220+0000 7fafeb7fe700 1 -- 172.21.15.153:0/1057456537 learned_addr learned my addr 172.21.15.153:0/1057456537 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T00:07:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.220+0000 7fafebfff700 1 --2- 172.21.15.153:0/1057456537 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fafec102820 0x7fafec102180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.220+0000 7faff0eb4700 1 --2- 172.21.15.153:0/1057456537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafec101190 0x7fafec1a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.220+0000 7fafeb7fe700 1 -- 172.21.15.153:0/1057456537 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fafec102820 msgr2=0x7fafec102180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.220+0000 7fafeb7fe700 1 --2- 172.21.15.153:0/1057456537 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fafec102820 0x7fafec102180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.220+0000 7fafeb7fe700 1 -- 172.21.15.153:0/1057456537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafec101190 msgr2=0x7fafec1a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.220+0000 7fafeb7fe700 1 --2- 172.21.15.153:0/1057456537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafec101190 0x7fafec1a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.220+0000 7fafeb7fe700 1 -- 172.21.15.153:0/1057456537 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafec1a1b80 con 0x7fafec1007d0 2024-04-25T00:07:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.220+0000 7fafeb7fe700 1 --2- 172.21.15.153:0/1057456537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafec1007d0 0x7fafec100c50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fafe000b780 tx=0x7fafe000ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.220+0000 7fafe97fa700 1 -- 172.21.15.153:0/1057456537 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe0016070 con 0x7fafec1007d0 2024-04-25T00:07:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.220+0000 7fafe97fa700 1 -- 172.21.15.153:0/1057456537 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafe00036a0 con 0x7fafec1007d0 2024-04-25T00:07:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.220+0000 7fafe97fa700 1 -- 172.21.15.153:0/1057456537 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe0011af0 con 0x7fafec1007d0 2024-04-25T00:07:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.220+0000 7faff2917700 1 -- 172.21.15.153:0/1057456537 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafec1a1e70 con 0x7fafec1007d0 2024-04-25T00:07:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.220+0000 7faff2917700 1 -- 172.21.15.153:0/1057456537 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafec1a23f0 con 0x7fafec1007d0 2024-04-25T00:07:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.221+0000 7fafe97fa700 1 -- 172.21.15.153:0/1057456537 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fafe0019070 con 0x7fafec1007d0 2024-04-25T00:07:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.221+0000 7fafe97fa700 1 --2- 172.21.15.153:0/1057456537 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fafd4039580 0x7fafd403ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.221+0000 7fafe97fa700 1 -- 172.21.15.153:0/1057456537 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fafe004ad60 con 0x7fafec1007d0 2024-04-25T00:07:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.221+0000 7faff2917700 1 -- 172.21.15.153:0/1057456537 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafec063890 con 0x7fafec1007d0 2024-04-25T00:07:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.222+0000 7fafebfff700 1 --2- 172.21.15.153:0/1057456537 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fafd4039580 0x7fafd403ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.222+0000 7fafebfff700 1 --2- 172.21.15.153:0/1057456537 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fafd4039580 0x7fafd403ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fafdc005200 tx=0x7fafdc01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.227+0000 7fafe97fa700 1 -- 172.21.15.153:0/1057456537 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fafe0015170 con 0x7fafec1007d0 2024-04-25T00:07:14.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.438+0000 7faff2917700 1 -- 172.21.15.153:0/1057456537 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=x;smithi139=y;smithi153=z", "target": ["mon-mgr", ""]}) -- 0x7fafec062700 con 0x7fafd4039580 2024-04-25T00:07:14.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.445+0000 7fafe97fa700 1 -- 172.21.15.153:0/1057456537 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fafec062700 con 0x7fafd4039580 2024-04-25T00:07:14.446 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mgr update... 2024-04-25T00:07:14.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.447+0000 7faff2917700 1 -- 172.21.15.153:0/1057456537 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fafd4039580 msgr2=0x7fafd403ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:14.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.447+0000 7faff2917700 1 --2- 172.21.15.153:0/1057456537 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fafd4039580 0x7fafd403ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fafdc005200 tx=0x7fafdc01a040 comp rx=0 tx=0).stop 2024-04-25T00:07:14.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.448+0000 7faff2917700 1 -- 172.21.15.153:0/1057456537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafec1007d0 msgr2=0x7fafec100c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:14.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.448+0000 7faff2917700 1 --2- 172.21.15.153:0/1057456537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafec1007d0 0x7fafec100c50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fafe000b780 tx=0x7fafe000ba90 comp rx=0 tx=0).stop 2024-04-25T00:07:14.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.448+0000 7faff2917700 1 -- 172.21.15.153:0/1057456537 shutdown_connections 2024-04-25T00:07:14.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.448+0000 7faff2917700 1 --2- 172.21.15.153:0/1057456537 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fafd4039580 0x7fafd403ba40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:14.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.448+0000 7faff2917700 1 --2- 172.21.15.153:0/1057456537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafec101190 0x7fafec1a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:14.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.448+0000 7faff2917700 1 --2- 172.21.15.153:0/1057456537 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fafec102820 0x7fafec102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:14.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.448+0000 7faff2917700 1 --2- 172.21.15.153:0/1057456537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafec1007d0 0x7fafec100c50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:14.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.448+0000 7faff2917700 1 -- 172.21.15.153:0/1057456537 >> 172.21.15.153:0/1057456537 conn(0x7fafec0fc230 msgr2=0x7fafec0fceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:14.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.448+0000 7faff2917700 1 -- 172.21.15.153:0/1057456537 shutdown_connections 2024-04-25T00:07:14.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:07:14.448+0000 7faff2917700 1 -- 172.21.15.153:0/1057456537 wait complete. 2024-04-25T00:07:15.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:14 smithi153 ceph-mon[32896]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:15.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:14 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:15.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:14 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:15.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:14 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T00:07:15.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:14 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T00:07:15.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:14 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:15.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:14 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:15.079 DEBUG:teuthology.orchestra.run.smithi139:mgr.y> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mgr.y.service 2024-04-25T00:07:15.082 DEBUG:teuthology.orchestra.run.smithi153:mgr.z> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mgr.z.service 2024-04-25T00:07:15.084 INFO:tasks.cephadm:Deploying OSDs... 2024-04-25T00:07:15.084 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:07:15.084 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T00:07:15.104 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T00:07:15.104 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-04-25T00:07:15.105 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:14 smithi139 ceph-mon[32833]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:15.105 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:14 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:15.105 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:14 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:15.105 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:14 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T00:07:15.105 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:14 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T00:07:15.105 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:14 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:15.106 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:14 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:15.162 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T00:07:15.162 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:07:15.162 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-25T00:07:15.162 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:07:15.162 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T00:07:15.162 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-25 00:06:19.890377373 +0000 2024-04-25T00:07:15.162 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-25 00:00:30.913982531 +0000 2024-04-25T00:07:15.162 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-25 00:00:30.913982531 +0000 2024-04-25T00:07:15.162 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-25 00:00:30.913982531 +0000 2024-04-25T00:07:15.162 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T00:07:15.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:14 smithi027 ceph-mon[29208]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:15.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:14 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:15.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:14 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:15.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:14 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T00:07:15.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:14 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T00:07:15.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:14 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:15.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:14 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:15.220 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-25T00:07:15.221 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-25T00:07:15.221 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000246176 s, 2.1 MB/s 2024-04-25T00:07:15.222 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T00:07:15.281 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-04-25T00:07:15.338 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T00:07:15.338 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:07:15.338 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T00:07:15.338 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:07:15.338 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T00:07:15.339 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-25 00:06:19.918376670 +0000 2024-04-25T00:07:15.339 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-25 00:00:31.271973527 +0000 2024-04-25T00:07:15.339 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-25 00:00:31.271973527 +0000 2024-04-25T00:07:15.339 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-25 00:00:31.271973527 +0000 2024-04-25T00:07:15.339 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T00:07:15.404 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-25T00:07:15.404 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-25T00:07:15.405 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.00023155 s, 2.2 MB/s 2024-04-25T00:07:15.406 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T00:07:15.463 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-04-25T00:07:15.520 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T00:07:15.520 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:07:15.520 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-25T00:07:15.520 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:07:15.520 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T00:07:15.520 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-25 00:06:19.948375917 +0000 2024-04-25T00:07:15.520 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-25 00:00:31.622964699 +0000 2024-04-25T00:07:15.521 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-25 00:00:31.622964699 +0000 2024-04-25T00:07:15.521 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-25 00:00:31.622964699 +0000 2024-04-25T00:07:15.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T00:07:15.586 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-25T00:07:15.586 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-25T00:07:15.586 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000206823 s, 2.5 MB/s 2024-04-25T00:07:15.587 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T00:07:15.645 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-04-25T00:07:15.702 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T00:07:15.702 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:07:15.702 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-25T00:07:15.702 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:07:15.702 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T00:07:15.703 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-25 00:06:19.975375240 +0000 2024-04-25T00:07:15.703 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-25 00:00:31.965956072 +0000 2024-04-25T00:07:15.703 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-25 00:00:31.965956072 +0000 2024-04-25T00:07:15.703 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-25 00:00:31.965956072 +0000 2024-04-25T00:07:15.703 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T00:07:15.767 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-25T00:07:15.767 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-25T00:07:15.767 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.00019394 s, 2.6 MB/s 2024-04-25T00:07:15.769 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T00:07:15.826 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T00:07:15.826 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T00:07:15.845 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T00:07:15.845 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_1 2024-04-25T00:07:15.903 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T00:07:15.903 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:07:15.903 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-25T00:07:15.903 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:07:15.903 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T00:07:15.903 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-04-25 00:07:02.523386172 +0000 2024-04-25T00:07:15.903 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-04-25 00:00:58.261679821 +0000 2024-04-25T00:07:15.904 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-04-25 00:00:58.261679821 +0000 2024-04-25T00:07:15.904 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-04-25 00:00:58.261679821 +0000 2024-04-25T00:07:15.904 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T00:07:15.969 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-04-25T00:07:15.969 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-04-25T00:07:15.969 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.00028216 s, 1.8 MB/s 2024-04-25T00:07:15.970 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T00:07:16.028 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_2 2024-04-25T00:07:16.084 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T00:07:16.084 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:07:16.084 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T00:07:16.084 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:07:16.085 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T00:07:16.085 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-04-25 00:07:02.523386172 +0000 2024-04-25T00:07:16.085 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-04-25 00:00:58.608672450 +0000 2024-04-25T00:07:16.085 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-04-25 00:00:58.608672450 +0000 2024-04-25T00:07:16.085 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-04-25 00:00:58.608672450 +0000 2024-04-25T00:07:16.085 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T00:07:16.146 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:15 smithi139 ceph-mon[32833]: Reconfiguring mon.c (monmap changed)... 2024-04-25T00:07:16.146 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:15 smithi139 ceph-mon[32833]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=x;smithi139=y;smithi153=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:07:16.146 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:15 smithi139 ceph-mon[32833]: Saving service mgr spec with placement smithi027=x;smithi139=y;smithi153=z;count:3 2024-04-25T00:07:16.146 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:15 smithi139 ceph-mon[32833]: Reconfiguring daemon mon.c on smithi153 2024-04-25T00:07:16.148 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-04-25T00:07:16.148 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-04-25T00:07:16.148 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000264164 s, 1.9 MB/s 2024-04-25T00:07:16.150 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T00:07:16.181 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:15 smithi153 ceph-mon[32896]: Reconfiguring mon.c (monmap changed)... 2024-04-25T00:07:16.181 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:15 smithi153 ceph-mon[32896]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=x;smithi139=y;smithi153=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:07:16.181 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:15 smithi153 ceph-mon[32896]: Saving service mgr spec with placement smithi027=x;smithi139=y;smithi153=z;count:3 2024-04-25T00:07:16.181 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:15 smithi153 ceph-mon[32896]: Reconfiguring daemon mon.c on smithi153 2024-04-25T00:07:16.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:15 smithi027 ceph-mon[29208]: Reconfiguring mon.c (monmap changed)... 2024-04-25T00:07:16.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:15 smithi027 ceph-mon[29208]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=x;smithi139=y;smithi153=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:07:16.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:15 smithi027 ceph-mon[29208]: Saving service mgr spec with placement smithi027=x;smithi139=y;smithi153=z;count:3 2024-04-25T00:07:16.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:15 smithi027 ceph-mon[29208]: Reconfiguring daemon mon.c on smithi153 2024-04-25T00:07:16.207 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_3 2024-04-25T00:07:16.263 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T00:07:16.263 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:07:16.263 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-25T00:07:16.264 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:07:16.264 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T00:07:16.264 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-04-25 00:07:02.523386172 +0000 2024-04-25T00:07:16.264 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-04-25 00:00:58.950665184 +0000 2024-04-25T00:07:16.264 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-04-25 00:00:58.950665184 +0000 2024-04-25T00:07:16.264 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-04-25 00:00:58.950665184 +0000 2024-04-25T00:07:16.264 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T00:07:16.327 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-04-25T00:07:16.328 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-04-25T00:07:16.328 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000248209 s, 2.1 MB/s 2024-04-25T00:07:16.329 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T00:07:16.386 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_4 2024-04-25T00:07:16.441 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T00:07:16.442 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:07:16.442 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-25T00:07:16.442 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:07:16.442 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T00:07:16.442 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-04-25 00:07:02.524386151 +0000 2024-04-25T00:07:16.442 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-04-25 00:00:59.275658280 +0000 2024-04-25T00:07:16.442 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-04-25 00:00:59.275658280 +0000 2024-04-25T00:07:16.442 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-04-25 00:00:59.275658280 +0000 2024-04-25T00:07:16.442 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T00:07:16.505 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-04-25T00:07:16.505 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-04-25T00:07:16.505 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000200165 s, 2.6 MB/s 2024-04-25T00:07:16.506 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T00:07:16.562 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T00:07:16.563 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T00:07:16.582 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T00:07:16.582 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-04-25T00:07:16.639 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T00:07:16.639 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:07:16.640 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-25T00:07:16.640 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:07:16.640 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T00:07:16.640 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-25 00:06:52.893978726 +0000 2024-04-25T00:07:16.640 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-25 00:00:45.909422790 +0000 2024-04-25T00:07:16.640 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-25 00:00:45.909422790 +0000 2024-04-25T00:07:16.640 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-25 00:00:45.909422790 +0000 2024-04-25T00:07:16.640 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T00:07:16.707 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-25T00:07:16.707 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-25T00:07:16.707 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000290844 s, 1.8 MB/s 2024-04-25T00:07:16.709 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T00:07:16.767 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-04-25T00:07:16.822 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T00:07:16.822 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:07:16.823 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-25T00:07:16.823 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:07:16.823 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T00:07:16.823 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-25 00:06:52.894978708 +0000 2024-04-25T00:07:16.823 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-25 00:00:46.250416733 +0000 2024-04-25T00:07:16.823 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-25 00:00:46.250416733 +0000 2024-04-25T00:07:16.823 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-25 00:00:46.250416733 +0000 2024-04-25T00:07:16.823 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T00:07:16.885 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-25T00:07:16.885 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-25T00:07:16.885 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000220506 s, 2.3 MB/s 2024-04-25T00:07:16.887 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T00:07:16.942 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-04-25T00:07:16.997 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T00:07:16.997 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:07:16.997 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-25T00:07:16.997 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:07:16.997 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T00:07:16.997 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-25 00:06:52.894978708 +0000 2024-04-25T00:07:16.997 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-25 00:00:46.603410464 +0000 2024-04-25T00:07:16.998 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-25 00:00:46.603410464 +0000 2024-04-25T00:07:16.998 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-25 00:00:46.603410464 +0000 2024-04-25T00:07:16.998 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T00:07:17.060 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-25T00:07:17.061 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-25T00:07:17.061 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000210897 s, 2.4 MB/s 2024-04-25T00:07:17.062 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T00:07:17.119 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-04-25T00:07:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T00:07:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:07:17.174 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T00:07:17.174 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:07:17.175 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T00:07:17.175 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-25 00:06:52.894978708 +0000 2024-04-25T00:07:17.175 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-25 00:00:46.938404514 +0000 2024-04-25T00:07:17.175 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-25 00:00:46.938404514 +0000 2024-04-25T00:07:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-25 00:00:46.938404514 +0000 2024-04-25T00:07:17.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T00:07:17.238 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-25T00:07:17.238 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-25T00:07:17.238 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000193778 s, 2.6 MB/s 2024-04-25T00:07:17.239 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T00:07:17.295 INFO:tasks.cephadm:Deploying osd.0 on smithi027 with /dev/vg_nvme/lv_4... 2024-04-25T00:07:17.295 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T00:07:17.318 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:17 smithi027 ceph-mon[29208]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:17.318 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:17 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:17.318 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:17 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:17.318 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:17 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:17.318 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:17 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:17.318 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:17 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:17.318 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:17 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:17.318 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:17 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T00:07:17.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:17 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T00:07:17.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:17 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T00:07:17.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:17 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:17.376 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:17 smithi153 ceph-mon[32896]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:17.376 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:17 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:17.376 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:17 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:17.376 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:17 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:17.377 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:17 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:17.377 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:17 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:17.377 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:17 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:17.377 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:17 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T00:07:17.377 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:17 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T00:07:17.377 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:17 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T00:07:17.377 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:17 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:17.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:17 smithi139 ceph-mon[32833]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:17.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:17 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:17.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:17 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:17.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:17 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:17.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:17 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:17.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:17 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:17.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:17 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:17.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:17 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T00:07:17.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:17 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T00:07:17.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:17 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T00:07:17.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:17 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:07:18.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:18 smithi027 ceph-mon[29208]: Deploying daemon mgr.z on smithi153 2024-04-25T00:07:18.332 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:18 smithi153 ceph-mon[32896]: Deploying daemon mgr.z on smithi153 2024-04-25T00:07:18.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:18 smithi139 ceph-mon[32833]: Deploying daemon mgr.z on smithi153 2024-04-25T00:07:18.993 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:18 smithi153 systemd[1]: Starting Ceph mgr.z for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:07:19.261 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:19 smithi153 ceph-mon[32896]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:19.261 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:18 smithi153 podman[34763]: 2024-04-25 00:07:18.99261801 +0000 UTC m=+0.111963056 container create 1b5a5bdb31ff66bac4e8a2916d0ae0489f884895ea1ef3482a4cd49432ef38b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-25T00:07:19.262 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:18 smithi153 podman[34763]: 2024-04-25 00:07:18.898617152 +0000 UTC m=+0.017962206 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:07:19.262 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:19 smithi153 podman[34763]: 2024-04-25 00:07:19.186483108 +0000 UTC m=+0.305828156 container init 1b5a5bdb31ff66bac4e8a2916d0ae0489f884895ea1ef3482a4cd49432ef38b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-25T00:07:19.262 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:19 smithi153 podman[34763]: 2024-04-25 00:07:19.188937646 +0000 UTC m=+0.308282695 container start 1b5a5bdb31ff66bac4e8a2916d0ae0489f884895ea1ef3482a4cd49432ef38b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T00:07:19.262 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:19 smithi153 ceph-mgr[34782]: -- 172.21.15.153:0/1174192025 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56223b9c25a0 con 0x56223b997000 2024-04-25T00:07:19.262 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:19 smithi153 bash[34763]: 1b5a5bdb31ff66bac4e8a2916d0ae0489f884895ea1ef3482a4cd49432ef38b4 2024-04-25T00:07:19.354 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T00:07:19.354 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T00:07:19.355 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-25T00:07:19.355 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-25T00:07:19.355 INFO:teuthology.orchestra.run.smithi027.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112973 s, 928 MB/s 2024-04-25T00:07:19.355 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-25T00:07:19.355 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:19 smithi027 ceph-mon[29208]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:19.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:19 smithi139 ceph-mon[32833]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:19.526 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:19 smithi153 systemd[1]: Started Ceph mgr.z for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:07:19.526 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:19 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:19.342+0000 7f172c2ba200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T00:07:19.938 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:19 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:19.528+0000 7f172c2ba200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:07:19.938 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:19 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:19.630+0000 7f172c2ba200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T00:07:19.939 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:19 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:19.937+0000 7f172c2ba200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T00:07:19.940 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_4 2024-04-25T00:07:20.223 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:07:20.276 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:20 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:20.022+0000 7f172c2ba200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T00:07:20.629 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:20 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:20.629 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:20 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:20.629 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:20 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:20.629 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:20 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T00:07:20.629 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:20 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T00:07:20.629 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:20 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T00:07:20.629 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:20 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:20.656 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:20 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:20.656 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:20 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:20.656 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:20 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:20.656 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:20 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T00:07:20.656 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:20 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T00:07:20.656 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:20 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T00:07:20.656 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:20 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:20 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:20 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:20 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:20 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T00:07:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:20 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T00:07:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:20 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T00:07:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:20 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:21.492 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:21 smithi139 ceph-mon[32833]: Deploying daemon mgr.y on smithi139 2024-04-25T00:07:21.492 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:21 smithi139 ceph-mon[32833]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:21.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:21 smithi153 ceph-mon[32896]: Deploying daemon mgr.y on smithi139 2024-04-25T00:07:21.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:21 smithi153 ceph-mon[32896]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:21.526 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:21 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:21.260+0000 7f172c2ba200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T00:07:21.527 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:21 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:21.434+0000 7f172c2ba200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T00:07:21.670 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:21 smithi027 ceph-mon[29208]: Deploying daemon mgr.y on smithi139 2024-04-25T00:07:21.670 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:21 smithi027 ceph-mon[29208]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:21.753 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:21 smithi139 ceph-mgr[34218]: -- 172.21.15.139:0/3195584051 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5577970985a0 con 0x55779706d000 2024-04-25T00:07:21.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.776+0000 7f052f271700 1 -- 172.21.15.27:0/937972974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0528107f70 msgr2=0x7f052810a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:21.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.776+0000 7f052f271700 1 --2- 172.21.15.27:0/937972974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0528107f70 0x7f052810a400 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f052000b0a0 tx=0x7f052000b3b0 comp rx=0 tx=0).stop 2024-04-25T00:07:21.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.776+0000 7f052f271700 1 -- 172.21.15.27:0/937972974 shutdown_connections 2024-04-25T00:07:21.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.776+0000 7f052f271700 1 --2- 172.21.15.27:0/937972974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0528107f70 0x7f052810a400 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:21.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.776+0000 7f052f271700 1 --2- 172.21.15.27:0/937972974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05280ffef0 0x7f0528100310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:21.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.776+0000 7f052f271700 1 --2- 172.21.15.27:0/937972974 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05281008e0 0x7f0528107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:21.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.776+0000 7f052f271700 1 -- 172.21.15.27:0/937972974 >> 172.21.15.27:0/937972974 conn(0x7f05280fba80 msgr2=0x7f05280fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:21.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.777+0000 7f052f271700 1 -- 172.21.15.27:0/937972974 shutdown_connections 2024-04-25T00:07:21.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.777+0000 7f052f271700 1 -- 172.21.15.27:0/937972974 wait complete. 2024-04-25T00:07:21.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.777+0000 7f052f271700 1 Processor -- start 2024-04-25T00:07:21.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.777+0000 7f052f271700 1 -- start start 2024-04-25T00:07:21.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.777+0000 7f052f271700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05280ffef0 0x7f05281983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:21.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.777+0000 7f052f271700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05281008e0 0x7f0528198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:21.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.777+0000 7f052f271700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0528107f70 0x7f052819f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:21.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.777+0000 7f052f271700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f052810de80 con 0x7f05280ffef0 2024-04-25T00:07:21.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.777+0000 7f052f271700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f052810dd00 con 0x7f05281008e0 2024-04-25T00:07:21.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.777+0000 7f052f271700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f052810e000 con 0x7f0528107f70 2024-04-25T00:07:21.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.778+0000 7f052d00d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05280ffef0 0x7f05281983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:21.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.778+0000 7f052d00d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05280ffef0 0x7f05281983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41910/0 (socket says 172.21.15.27:41910) 2024-04-25T00:07:21.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.778+0000 7f052c80c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05281008e0 0x7f0528198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:21.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.778+0000 7f052d00d700 1 -- 172.21.15.27:0/1922907741 learned_addr learned my addr 172.21.15.27:0/1922907741 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:07:21.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.778+0000 7f052d80e700 1 --2- 172.21.15.27:0/1922907741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0528107f70 0x7f052819f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:21.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.778+0000 7f052d00d700 1 -- 172.21.15.27:0/1922907741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0528107f70 msgr2=0x7f052819f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:21.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.778+0000 7f052d00d700 1 --2- 172.21.15.27:0/1922907741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0528107f70 0x7f052819f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:21.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.778+0000 7f052d00d700 1 -- 172.21.15.27:0/1922907741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05281008e0 msgr2=0x7f0528198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:21.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.778+0000 7f052d00d700 1 --2- 172.21.15.27:0/1922907741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05281008e0 0x7f0528198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:21.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.778+0000 7f052d00d700 1 -- 172.21.15.27:0/1922907741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05281a0040 con 0x7f05280ffef0 2024-04-25T00:07:21.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.778+0000 7f052d00d700 1 --2- 172.21.15.27:0/1922907741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05280ffef0 0x7f05281983e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f052400d980 tx=0x7f052400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:21.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.778+0000 7f051e7fc700 1 -- 172.21.15.27:0/1922907741 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0524016070 con 0x7f05280ffef0 2024-04-25T00:07:21.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.778+0000 7f051e7fc700 1 -- 172.21.15.27:0/1922907741 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0524009db0 con 0x7f05280ffef0 2024-04-25T00:07:21.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.778+0000 7f051e7fc700 1 -- 172.21.15.27:0/1922907741 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052400f3c0 con 0x7f05280ffef0 2024-04-25T00:07:21.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.778+0000 7f052f271700 1 -- 172.21.15.27:0/1922907741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05281a0330 con 0x7f05280ffef0 2024-04-25T00:07:21.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.778+0000 7f052f271700 1 -- 172.21.15.27:0/1922907741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05281a08c0 con 0x7f05280ffef0 2024-04-25T00:07:21.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.779+0000 7f051e7fc700 1 -- 172.21.15.27:0/1922907741 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f0524019070 con 0x7f05280ffef0 2024-04-25T00:07:21.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.779+0000 7f052f271700 1 -- 172.21.15.27:0/1922907741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0528068f20 con 0x7f05280ffef0 2024-04-25T00:07:21.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.779+0000 7f051e7fc700 1 --2- 172.21.15.27:0/1922907741 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f0514039330 0x7f051403b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:21.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.779+0000 7f051e7fc700 1 -- 172.21.15.27:0/1922907741 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f052404bd90 con 0x7f05280ffef0 2024-04-25T00:07:21.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.780+0000 7f052c80c700 1 --2- 172.21.15.27:0/1922907741 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f0514039330 0x7f051403b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:21.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.780+0000 7f052c80c700 1 --2- 172.21.15.27:0/1922907741 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f0514039330 0x7f051403b7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0518007be0 tx=0x7f05180073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:21.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:21.784+0000 7f051e7fc700 1 -- 172.21.15.27:0/1922907741 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f052400ad10 con 0x7f05280ffef0 2024-04-25T00:07:21.901 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:21 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:21.611+0000 7f172c2ba200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T00:07:21.901 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:21 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:21.900+0000 7f172c2ba200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T00:07:22.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:22.016+0000 7f052f271700 1 -- 172.21.15.27:0/1922907741 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f0528062670 con 0x7f0514039330 2024-04-25T00:07:22.050 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:21 smithi139 bash[34199]: 9ed73c881dff1fa6ff4094912e5fb39041d780b0abafd42f7951bc64c7aaaafd 2024-04-25T00:07:22.051 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:21 smithi139 systemd[1]: Started Ceph mgr.y for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:07:22.051 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:21 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:21.861+0000 7fc0dd343200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T00:07:22.051 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:22 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:22.049+0000 7fc0dd343200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:07:22.206 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:21 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:21.994+0000 7f172c2ba200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T00:07:22.206 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:22 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:22.205+0000 7f172c2ba200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T00:07:22.324 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:22 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:22.163+0000 7fc0dd343200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T00:07:22.526 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:22 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:22.403+0000 7f172c2ba200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:07:22.598 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:22 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:22.499+0000 7fc0dd343200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T00:07:22.598 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:22 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:22.597+0000 7fc0dd343200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T00:07:22.870 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:22 smithi139 ceph-mon[32833]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:22.871 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:22 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:22.871 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:22 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:22.871 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:22 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:22.871 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:22 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:22.871 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:22 smithi139 ceph-mon[32833]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:07:22.871 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:22 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:07:22.871 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:22 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:07:22.871 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:22 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:22.871 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:22 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:22.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:22 smithi027 ceph-mon[29208]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:22.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:22.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:22.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:22.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:22.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:22 smithi027 ceph-mon[29208]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:07:22.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:07:22.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:07:22.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:22.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:23.277 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:23 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:23.041+0000 7f172c2ba200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T00:07:23.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:22 smithi153 ceph-mon[32896]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:23.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:22 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:23.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:22 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:23.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:22 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:23.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:22 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:23.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:22 smithi153 ceph-mon[32896]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:07:23.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:22 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:07:23.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:22 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:07:23.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:22 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:23.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:22 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:23.765 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:23 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:23.469+0000 7f172c2ba200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T00:07:23.765 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:23 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:23.565+0000 7f172c2ba200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T00:07:23.765 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:23 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:23.765+0000 7f172c2ba200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T00:07:24.199 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:23 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:23.857+0000 7fc0dd343200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T00:07:24.199 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:24 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:24.026+0000 7fc0dd343200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T00:07:24.199 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:24 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:24.197+0000 7fc0dd343200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T00:07:24.487 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:24 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:24.487 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:24 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:24.488 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:24 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:24.488 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:24 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:24.488 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:24 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:24.488 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:24 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:24.488 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:24 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:24.488 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:24 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T00:07:24.489 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:24 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T00:07:24.489 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:24 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:24.526 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:24 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:24.281+0000 7f172c2ba200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T00:07:24.526 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:24 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:24.370+0000 7f172c2ba200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T00:07:24.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:24 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:24.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:24 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:24.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:24 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:24.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:24 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:24.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:24 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:24.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:24 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:24.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:24 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:24.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:24 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T00:07:24.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:24 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T00:07:24.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:24 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:24.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:24 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:24.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:24 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:24.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:24 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:24.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:24 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:24.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:24 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:24.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:24 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:24.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:24 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:24.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:24 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T00:07:24.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:24 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T00:07:24.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:24 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:24.753 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:24 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:24.486+0000 7fc0dd343200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T00:07:24.753 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:24 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:24.578+0000 7fc0dd343200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T00:07:24.944 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:24 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:24.607+0000 7f172c2ba200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T00:07:24.944 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:24 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:24.738+0000 7f172c2ba200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T00:07:24.944 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:24 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:24.830+0000 7f172c2ba200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T00:07:24.944 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:24 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:24.943+0000 7f172c2ba200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T00:07:25.210 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:24 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:24.783+0000 7fc0dd343200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T00:07:25.210 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:24 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:24.972+0000 7fc0dd343200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:07:25.210 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:25 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:25.032+0000 7f172c2ba200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T00:07:25.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:25 smithi027 ceph-mon[29208]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:25.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:25 smithi027 ceph-mon[29208]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T00:07:25.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:25 smithi027 ceph-mon[29208]: Reconfiguring daemon mgr.x on smithi027 2024-04-25T00:07:25.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:25.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:25 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2038857478' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b523ee1d-7722-4c0d-ba18-82aa50b46b3c"}]: dispatch 2024-04-25T00:07:25.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:25 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2038857478' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b523ee1d-7722-4c0d-ba18-82aa50b46b3c"}]': finished 2024-04-25T00:07:25.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:25 smithi027 ceph-mon[29208]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T00:07:25.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:07:25.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:25 smithi139 ceph-mon[32833]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:25.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:25 smithi139 ceph-mon[32833]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T00:07:25.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:25 smithi139 ceph-mon[32833]: Reconfiguring daemon mgr.x on smithi027 2024-04-25T00:07:25.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:25 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:25.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:25 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2038857478' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b523ee1d-7722-4c0d-ba18-82aa50b46b3c"}]: dispatch 2024-04-25T00:07:25.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:25 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2038857478' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b523ee1d-7722-4c0d-ba18-82aa50b46b3c"}]': finished 2024-04-25T00:07:25.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:25 smithi139 ceph-mon[32833]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T00:07:25.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:25 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:07:25.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:25 smithi153 ceph-mon[32896]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:25.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:25 smithi153 ceph-mon[32896]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T00:07:25.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:25 smithi153 ceph-mon[32896]: Reconfiguring daemon mgr.x on smithi027 2024-04-25T00:07:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:25 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:25 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2038857478' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b523ee1d-7722-4c0d-ba18-82aa50b46b3c"}]: dispatch 2024-04-25T00:07:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:25 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2038857478' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b523ee1d-7722-4c0d-ba18-82aa50b46b3c"}]': finished 2024-04-25T00:07:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:25 smithi153 ceph-mon[32896]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T00:07:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:25 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:07:26.000 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:25 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:25.582+0000 7fc0dd343200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T00:07:26.219 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:25 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:25.834+0000 7f172c2ba200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T00:07:26.219 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:07:26 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:07:26.044+0000 7f172c2ba200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T00:07:26.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:26.221+0000 7f051e7fc700 1 -- 172.21.15.27:0/1922907741 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 14) ==== 92060+0+0 (secure 0 0 0) 0x7f0524052080 con 0x7f05280ffef0 2024-04-25T00:07:26.253 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:26 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:25.999+0000 7fc0dd343200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T00:07:26.253 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:26 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:26.094+0000 7fc0dd343200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T00:07:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:26 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3877580330' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:07:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:26 smithi139 ceph-mon[32833]: Standby manager daemon z started 2024-04-25T00:07:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:26 smithi139 ceph-mon[32833]: from='mgr.? 172.21.15.153:0/1793860903' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T00:07:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:26 smithi139 ceph-mon[32833]: from='mgr.? 172.21.15.153:0/1793860903' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T00:07:26.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:26 smithi139 ceph-mon[32833]: from='mgr.? 172.21.15.153:0/1793860903' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T00:07:26.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:26 smithi139 ceph-mon[32833]: from='mgr.? 172.21.15.153:0/1793860903' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T00:07:26.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:26 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3877580330' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:07:26.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:26 smithi153 ceph-mon[32896]: Standby manager daemon z started 2024-04-25T00:07:26.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:26 smithi153 ceph-mon[32896]: from='mgr.? 172.21.15.153:0/1793860903' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T00:07:26.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:26 smithi153 ceph-mon[32896]: from='mgr.? 172.21.15.153:0/1793860903' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T00:07:26.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:26 smithi153 ceph-mon[32896]: from='mgr.? 172.21.15.153:0/1793860903' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T00:07:26.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:26 smithi153 ceph-mon[32896]: from='mgr.? 172.21.15.153:0/1793860903' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T00:07:26.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:26 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3877580330' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:07:26.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:26 smithi027 ceph-mon[29208]: Standby manager daemon z started 2024-04-25T00:07:26.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:26 smithi027 ceph-mon[29208]: from='mgr.? 172.21.15.153:0/1793860903' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T00:07:26.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:26 smithi027 ceph-mon[29208]: from='mgr.? 172.21.15.153:0/1793860903' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T00:07:26.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:26 smithi027 ceph-mon[29208]: from='mgr.? 172.21.15.153:0/1793860903' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T00:07:26.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:26 smithi027 ceph-mon[29208]: from='mgr.? 172.21.15.153:0/1793860903' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T00:07:26.753 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:26 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:26.296+0000 7fc0dd343200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T00:07:27.165 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:26 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:26.837+0000 7fc0dd343200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T00:07:27.166 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:26 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:26.927+0000 7fc0dd343200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T00:07:27.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:27 smithi139 ceph-mon[32833]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:27.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:27 smithi139 ceph-mon[32833]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-25T00:07:27.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:27 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T00:07:27.503 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:27 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:27.164+0000 7fc0dd343200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T00:07:27.503 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:27 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:27.296+0000 7fc0dd343200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T00:07:27.503 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:27 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:27.389+0000 7fc0dd343200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T00:07:27.503 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:27 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:27.502+0000 7fc0dd343200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T00:07:27.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:27 smithi153 ceph-mon[32896]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:27.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:27 smithi153 ceph-mon[32896]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-25T00:07:27.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:27 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T00:07:27.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:27 smithi027 ceph-mon[29208]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:27.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:27 smithi027 ceph-mon[29208]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-25T00:07:27.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:27 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T00:07:28.003 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:27 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:27.592+0000 7fc0dd343200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T00:07:28.753 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:28 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:28.406+0000 7fc0dd343200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T00:07:28.753 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:07:28 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:07:28.617+0000 7fc0dd343200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T00:07:29.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:29.228+0000 7f051e7fc700 1 -- 172.21.15.27:0/1922907741 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f052401a3b0 con 0x7f05280ffef0 2024-04-25T00:07:29.337 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:29 smithi027 ceph-mon[29208]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:29.337 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:29 smithi027 ceph-mon[29208]: Standby manager daemon y started 2024-04-25T00:07:29.337 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:29 smithi027 ceph-mon[29208]: from='mgr.? 172.21.15.139:0/264322299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T00:07:29.337 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:29 smithi027 ceph-mon[29208]: from='mgr.? 172.21.15.139:0/264322299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T00:07:29.337 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:29 smithi027 ceph-mon[29208]: from='mgr.? 172.21.15.139:0/264322299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T00:07:29.337 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:29 smithi027 ceph-mon[29208]: from='mgr.? 172.21.15.139:0/264322299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T00:07:29.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:29 smithi139 ceph-mon[32833]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:29.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:29 smithi139 ceph-mon[32833]: Standby manager daemon y started 2024-04-25T00:07:29.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:29 smithi139 ceph-mon[32833]: from='mgr.? 172.21.15.139:0/264322299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T00:07:29.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:29 smithi139 ceph-mon[32833]: from='mgr.? 172.21.15.139:0/264322299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T00:07:29.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:29 smithi139 ceph-mon[32833]: from='mgr.? 172.21.15.139:0/264322299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T00:07:29.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:29 smithi139 ceph-mon[32833]: from='mgr.? 172.21.15.139:0/264322299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T00:07:29.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:29 smithi153 ceph-mon[32896]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:29.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:29 smithi153 ceph-mon[32896]: Standby manager daemon y started 2024-04-25T00:07:29.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:29 smithi153 ceph-mon[32896]: from='mgr.? 172.21.15.139:0/264322299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T00:07:29.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:29 smithi153 ceph-mon[32896]: from='mgr.? 172.21.15.139:0/264322299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T00:07:29.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:29 smithi153 ceph-mon[32896]: from='mgr.? 172.21.15.139:0/264322299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T00:07:29.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:29 smithi153 ceph-mon[32896]: from='mgr.? 172.21.15.139:0/264322299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T00:07:30.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:30 smithi139 ceph-mon[32833]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-25T00:07:30.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:30 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T00:07:30.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:30 smithi153 ceph-mon[32896]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-25T00:07:30.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:30 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T00:07:30.581 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:30 smithi027 ceph-mon[29208]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-25T00:07:30.581 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:30 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T00:07:31.634 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:31 smithi027 ceph-mon[29208]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:31.635 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:31 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:31.635 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:31 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:31.635 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:31 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:31.635 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:31 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:31.635 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:31 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:31.635 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:31 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:31.665 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:31 smithi153 ceph-mon[32896]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:31.665 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:31 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:31.665 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:31 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:31.665 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:31 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:31.665 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:31 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:31.665 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:31 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:31.665 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:31 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:31.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:31 smithi139 ceph-mon[32833]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:31.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:31 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:31.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:31 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:31.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:31 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:31.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:31 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:31.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:31 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:31.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:31 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:33.540 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:33 smithi027 ceph-mon[29208]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:33.541 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:33 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T00:07:33.541 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:33 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:33.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:33 smithi139 ceph-mon[32833]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:33.781 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:33 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T00:07:33.781 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:33 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:33.781 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:33 smithi153 ceph-mon[32896]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:33.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:33 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T00:07:33.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:33 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:34.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:34 smithi027 ceph-mon[29208]: Deploying daemon osd.0 on smithi027 2024-04-25T00:07:34.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:34 smithi139 ceph-mon[32833]: Deploying daemon osd.0 on smithi027 2024-04-25T00:07:34.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:34 smithi153 ceph-mon[32896]: Deploying daemon osd.0 on smithi027 2024-04-25T00:07:35.673 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:35 smithi027 ceph-mon[29208]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:35.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:35 smithi139 ceph-mon[32833]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:35.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:35 smithi153 ceph-mon[32896]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:36.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:36 smithi027 ceph-mon[29208]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:36.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:36 smithi139 ceph-mon[32833]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:36.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:36 smithi153 ceph-mon[32896]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:39.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:38 smithi027 ceph-mon[29208]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:39.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:39.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:39.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:39.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:38 smithi139 ceph-mon[32833]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:39.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:38 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:39.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:38 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:39.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:38 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:39.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:38 smithi153 ceph-mon[32896]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:39.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:39.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:39.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:41.011 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:40 smithi027 ceph-mon[29208]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:41.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:40 smithi139 ceph-mon[32833]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:41.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:40 smithi153 ceph-mon[32896]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:41.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:41.369+0000 7f051e7fc700 1 -- 172.21.15.27:0/1922907741 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0528062670 con 0x7f0514039330 2024-04-25T00:07:41.370 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 0 on host 'smithi027' 2024-04-25T00:07:41.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:41.372+0000 7f052f271700 1 -- 172.21.15.27:0/1922907741 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f0514039330 msgr2=0x7f051403b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:41.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:41.372+0000 7f052f271700 1 --2- 172.21.15.27:0/1922907741 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f0514039330 0x7f051403b7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0518007be0 tx=0x7f05180073d0 comp rx=0 tx=0).stop 2024-04-25T00:07:41.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:41.372+0000 7f052f271700 1 -- 172.21.15.27:0/1922907741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05280ffef0 msgr2=0x7f05281983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:41.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:41.372+0000 7f052f271700 1 --2- 172.21.15.27:0/1922907741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05280ffef0 0x7f05281983e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f052400d980 tx=0x7f052400dc90 comp rx=0 tx=0).stop 2024-04-25T00:07:41.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:41.372+0000 7f052f271700 1 -- 172.21.15.27:0/1922907741 shutdown_connections 2024-04-25T00:07:41.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:41.372+0000 7f052f271700 1 --2- 172.21.15.27:0/1922907741 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f0514039330 0x7f051403b7f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:41.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:41.372+0000 7f052f271700 1 --2- 172.21.15.27:0/1922907741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0528107f70 0x7f052819f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:41.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:41.372+0000 7f052f271700 1 --2- 172.21.15.27:0/1922907741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05280ffef0 0x7f05281983e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:41.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:41.372+0000 7f052f271700 1 --2- 172.21.15.27:0/1922907741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05281008e0 0x7f0528198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:41.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:41.372+0000 7f052f271700 1 -- 172.21.15.27:0/1922907741 >> 172.21.15.27:0/1922907741 conn(0x7f05280fba80 msgr2=0x7f05280fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:41.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:41.372+0000 7f052f271700 1 -- 172.21.15.27:0/1922907741 shutdown_connections 2024-04-25T00:07:41.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:07:41.372+0000 7f052f271700 1 -- 172.21.15.27:0/1922907741 wait complete. 2024-04-25T00:07:42.118 DEBUG:teuthology.orchestra.run.smithi027:osd.0> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.0.service 2024-04-25T00:07:42.121 INFO:tasks.cephadm:Deploying osd.1 on smithi139 with /dev/vg_nvme/lv_4... 2024-04-25T00:07:42.121 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T00:07:42.364 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T00:07:42.486 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:42 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:42.486 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:42 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:42.486 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:42 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:42.486 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:42 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:42.486 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:42 smithi139 ceph-mon[32833]: from='osd.0 [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T00:07:42.593 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:42 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:42.594 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:42 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:42.594 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:42 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:42.594 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:42 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:42.594 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:42 smithi027 ceph-mon[29208]: from='osd.0 [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T00:07:42.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:42 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:42.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:42 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:42.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:42 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:42.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:42 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:42.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:42 smithi153 ceph-mon[32896]: from='osd.0 [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T00:07:43.641 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:43 smithi027 ceph-mon[29208]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:43.641 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:43 smithi027 ceph-mon[29208]: from='osd.0 [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T00:07:43.641 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:43 smithi027 ceph-mon[29208]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T00:07:43.641 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:43 smithi027 ceph-mon[29208]: from='osd.0 [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T00:07:43.641 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:43 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:07:43.642 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:07:43 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-0[41329]: 2024-04-25T00:07:43.408+0000 7f7588e17700 -1 osd.0 0 waiting for initial osdmap 2024-04-25T00:07:43.642 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:07:43 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-0[41329]: 2024-04-25T00:07:43.412+0000 7f7584c0a700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:07:43.650 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:43 smithi139 ceph-mon[32833]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:43.650 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:43 smithi139 ceph-mon[32833]: from='osd.0 [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T00:07:43.650 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:43 smithi139 ceph-mon[32833]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T00:07:43.650 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:43 smithi139 ceph-mon[32833]: from='osd.0 [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T00:07:43.650 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:43 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:07:43.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:43 smithi153 ceph-mon[32896]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:43.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:43 smithi153 ceph-mon[32896]: from='osd.0 [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T00:07:43.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:43 smithi153 ceph-mon[32896]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T00:07:43.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:43 smithi153 ceph-mon[32896]: from='osd.0 [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T00:07:43.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:43 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:07:44.220 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T00:07:44.220 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T00:07:44.220 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2024-04-25T00:07:44.220 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2024-04-25T00:07:44.220 INFO:teuthology.orchestra.run.smithi139.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0139654 s, 751 MB/s 2024-04-25T00:07:44.220 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 2024-04-25T00:07:44.220 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2024-04-25T00:07:44.547 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:44 smithi139 ceph-mon[32833]: from='osd.0 [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T00:07:44.547 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:44 smithi139 ceph-mon[32833]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T00:07:44.547 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:44 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:07:44.547 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:44 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:07:44.547 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:44 smithi139 ceph-mon[32833]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:44.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:44 smithi027 ceph-mon[29208]: from='osd.0 [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T00:07:44.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:44 smithi027 ceph-mon[29208]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T00:07:44.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:44 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:07:44.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:44 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:07:44.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:44 smithi027 ceph-mon[29208]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:44.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:44 smithi153 ceph-mon[32896]: from='osd.0 [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T00:07:44.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:44 smithi153 ceph-mon[32896]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T00:07:44.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:44 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:07:44.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:44 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:07:44.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:44 smithi153 ceph-mon[32896]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T00:07:44.873 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi139:vg_nvme/lv_4 2024-04-25T00:07:45.114 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T00:07:45.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:45 smithi139 ceph-mon[32833]: purged_snaps scrub starts 2024-04-25T00:07:45.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:45 smithi139 ceph-mon[32833]: purged_snaps scrub ok 2024-04-25T00:07:45.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:45 smithi139 ceph-mon[32833]: osd.0 [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528] boot 2024-04-25T00:07:45.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:45 smithi139 ceph-mon[32833]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T00:07:45.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:45 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:07:45.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:45 smithi139 ceph-mon[32833]: Detected new or changed devices on smithi027 2024-04-25T00:07:45.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:45 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:45.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:45 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:45.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:45 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:07:45.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:45 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi027 to 17088M 2024-04-25T00:07:45.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:45 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:45.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:45 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:45.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:45 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:45.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:45 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:45.506 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:45 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:45.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:45 smithi027 ceph-mon[29208]: purged_snaps scrub starts 2024-04-25T00:07:45.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:45 smithi027 ceph-mon[29208]: purged_snaps scrub ok 2024-04-25T00:07:45.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:45 smithi027 ceph-mon[29208]: osd.0 [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528] boot 2024-04-25T00:07:45.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:45 smithi027 ceph-mon[29208]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T00:07:45.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:45 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:07:45.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:45 smithi027 ceph-mon[29208]: Detected new or changed devices on smithi027 2024-04-25T00:07:45.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:45 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:45.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:45 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:45.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:45 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:07:45.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:45 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi027 to 17088M 2024-04-25T00:07:45.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:45 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:45.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:45 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:45.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:45 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:45.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:45 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:45.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:45 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:45.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:45 smithi153 ceph-mon[32896]: purged_snaps scrub starts 2024-04-25T00:07:45.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:45 smithi153 ceph-mon[32896]: purged_snaps scrub ok 2024-04-25T00:07:45.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:45 smithi153 ceph-mon[32896]: osd.0 [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528] boot 2024-04-25T00:07:45.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:45 smithi153 ceph-mon[32896]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T00:07:45.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:45 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:07:45.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:45 smithi153 ceph-mon[32896]: Detected new or changed devices on smithi027 2024-04-25T00:07:45.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:45 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:45.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:45 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:45.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:45 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:07:45.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:45 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi027 to 17088M 2024-04-25T00:07:45.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:45 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:45.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:45 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:45.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:45 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:45.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:45 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:45.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:45 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:07:46.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.599+0000 7fe1fc822700 1 -- 172.21.15.139:0/2079192505 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1f8103e30 msgr2=0x7fe1f810a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:46.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.599+0000 7fe1fc822700 1 --2- 172.21.15.139:0/2079192505 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1f8103e30 0x7fe1f810a6e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe1f000b0d0 tx=0x7fe1f000b3e0 comp rx=0 tx=0).stop 2024-04-25T00:07:46.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.600+0000 7fe1fc822700 1 -- 172.21.15.139:0/2079192505 shutdown_connections 2024-04-25T00:07:46.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.600+0000 7fe1fc822700 1 --2- 172.21.15.139:0/2079192505 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1f8103e30 0x7fe1f810a6e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:46.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.600+0000 7fe1fc822700 1 --2- 172.21.15.139:0/2079192505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1f8102290 0x7fe1f81026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:46.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.600+0000 7fe1fc822700 1 --2- 172.21.15.139:0/2079192505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f8103490 0x7fe1f81038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:46.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.600+0000 7fe1fc822700 1 -- 172.21.15.139:0/2079192505 >> 172.21.15.139:0/2079192505 conn(0x7fe1f80fd830 msgr2=0x7fe1f80ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:46.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.600+0000 7fe1fc822700 1 -- 172.21.15.139:0/2079192505 shutdown_connections 2024-04-25T00:07:46.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.600+0000 7fe1fc822700 1 -- 172.21.15.139:0/2079192505 wait complete. 2024-04-25T00:07:46.607 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.600+0000 7fe1fc822700 1 Processor -- start 2024-04-25T00:07:46.607 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.600+0000 7fe1fc822700 1 -- start start 2024-04-25T00:07:46.607 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1fc822700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f8102290 0x7fe1f81985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:46.607 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1fc822700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1f8103490 0x7fe1f8198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:46.607 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1fc822700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1f8103e30 0x7fe1f819fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:46.607 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1fc822700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f810df70 con 0x7fe1f8102290 2024-04-25T00:07:46.607 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1fc822700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f810ddf0 con 0x7fe1f8103e30 2024-04-25T00:07:46.607 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1fc822700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f810e0f0 con 0x7fe1f8103490 2024-04-25T00:07:46.607 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1f6d9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1f8103e30 0x7fe1f819fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:46.608 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1f6d9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1f8103e30 0x7fe1f819fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:55124/0 (socket says 172.21.15.139:55124) 2024-04-25T00:07:46.608 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1f6d9d700 1 -- 172.21.15.139:0/2340692791 learned_addr learned my addr 172.21.15.139:0/2340692791 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-04-25T00:07:46.608 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1f659c700 1 --2- 172.21.15.139:0/2340692791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f8102290 0x7fe1f81985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:46.608 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1f5d9b700 1 --2- 172.21.15.139:0/2340692791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1f8103490 0x7fe1f8198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:46.608 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1f6d9d700 1 -- 172.21.15.139:0/2340692791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1f8103490 msgr2=0x7fe1f8198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:46.608 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1f6d9d700 1 --2- 172.21.15.139:0/2340692791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1f8103490 0x7fe1f8198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:46.609 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1f6d9d700 1 -- 172.21.15.139:0/2340692791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f8102290 msgr2=0x7fe1f81985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:46.609 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1f6d9d700 1 --2- 172.21.15.139:0/2340692791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f8102290 0x7fe1f81985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:46.609 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1f6d9d700 1 -- 172.21.15.139:0/2340692791 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f81a0210 con 0x7fe1f8103e30 2024-04-25T00:07:46.609 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1f659c700 1 --2- 172.21.15.139:0/2340692791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f8102290 0x7fe1f81985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:07:46.609 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1f5d9b700 1 --2- 172.21.15.139:0/2340692791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1f8103490 0x7fe1f8198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:07:46.609 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1f6d9d700 1 --2- 172.21.15.139:0/2340692791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1f8103e30 0x7fe1f819fb70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe1f00062a0 tx=0x7fe1f0007e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:46.609 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1e77fe700 1 -- 172.21.15.139:0/2340692791 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0003c00 con 0x7fe1f8103e30 2024-04-25T00:07:46.609 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1e77fe700 1 -- 172.21.15.139:0/2340692791 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1f0003da0 con 0x7fe1f8103e30 2024-04-25T00:07:46.610 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1fc822700 1 -- 172.21.15.139:0/2340692791 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1f81a04a0 con 0x7fe1f8103e30 2024-04-25T00:07:46.610 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.601+0000 7fe1e77fe700 1 -- 172.21.15.139:0/2340692791 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0005560 con 0x7fe1f8103e30 2024-04-25T00:07:46.610 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.602+0000 7fe1fc822700 1 -- 172.21.15.139:0/2340692791 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1f81a0a00 con 0x7fe1f8103e30 2024-04-25T00:07:46.610 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.602+0000 7fe1fc822700 1 -- 172.21.15.139:0/2340692791 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1f8068f20 con 0x7fe1f8103e30 2024-04-25T00:07:46.610 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.604+0000 7fe1e77fe700 1 -- 172.21.15.139:0/2340692791 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7fe1f0005700 con 0x7fe1f8103e30 2024-04-25T00:07:46.610 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.604+0000 7fe1e77fe700 1 --2- 172.21.15.139:0/2340692791 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1e00a2b90 0x7fe1e00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:46.610 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.604+0000 7fe1e77fe700 1 -- 172.21.15.139:0/2340692791 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fe1f000f020 con 0x7fe1f8103e30 2024-04-25T00:07:46.611 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.604+0000 7fe1f659c700 1 --2- 172.21.15.139:0/2340692791 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1e00a2b90 0x7fe1e00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:46.611 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.605+0000 7fe1f659c700 1 --2- 172.21.15.139:0/2340692791 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1e00a2b90 0x7fe1e00a5050 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe1f81032f0 tx=0x7fe1e8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:46.612 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.608+0000 7fe1e77fe700 1 -- 172.21.15.139:0/2340692791 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe1f002b020 con 0x7fe1f8103e30 2024-04-25T00:07:46.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:46 smithi027 ceph-mon[29208]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T00:07:46.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:46 smithi027 ceph-mon[29208]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:46.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:46 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:46.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:46 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:46.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:46 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:46.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:46 smithi139 ceph-mon[32833]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T00:07:46.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:46 smithi139 ceph-mon[32833]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:46.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:46 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:46.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:46 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:46.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:46 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:46.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:46 smithi153 ceph-mon[32896]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T00:07:46.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:46 smithi153 ceph-mon[32896]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:46.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:46 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:46.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:46 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:07:46.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:46 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:07:46.815 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:07:46.814+0000 7fe1fc822700 1 -- 172.21.15.139:0/2340692791 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fe1f8062670 con 0x7fe1e00a2b90 2024-04-25T00:07:47.683 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:47 smithi139 ceph-mon[32833]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:07:47.683 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:47 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:07:47.683 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:47 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:07:47.684 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:47 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:47.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:47 smithi027 ceph-mon[29208]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:07:47.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:07:47.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:07:47.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:47.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:47 smithi153 ceph-mon[32896]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:07:47.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:47 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:07:47.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:47 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:07:47.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:47 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:48.596 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:48 smithi139 ceph-mon[32833]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:48.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:48 smithi027 ceph-mon[29208]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:48.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:48 smithi153 ceph-mon[32896]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:49.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:49 smithi139 ceph-mon[32833]: from='client.? 172.21.15.139:0/4208854438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1317fe63-aa4c-4687-8cdb-47f9531f654f"}]: dispatch 2024-04-25T00:07:49.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:49 smithi139 ceph-mon[32833]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1317fe63-aa4c-4687-8cdb-47f9531f654f"}]: dispatch 2024-04-25T00:07:49.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:49 smithi139 ceph-mon[32833]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1317fe63-aa4c-4687-8cdb-47f9531f654f"}]': finished 2024-04-25T00:07:49.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:49 smithi139 ceph-mon[32833]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T00:07:49.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:49 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:07:49.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:49 smithi153 ceph-mon[32896]: from='client.? 172.21.15.139:0/4208854438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1317fe63-aa4c-4687-8cdb-47f9531f654f"}]: dispatch 2024-04-25T00:07:49.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:49 smithi153 ceph-mon[32896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1317fe63-aa4c-4687-8cdb-47f9531f654f"}]: dispatch 2024-04-25T00:07:49.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:49 smithi153 ceph-mon[32896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1317fe63-aa4c-4687-8cdb-47f9531f654f"}]': finished 2024-04-25T00:07:49.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:49 smithi153 ceph-mon[32896]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T00:07:49.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:49 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:07:49.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:49 smithi027 ceph-mon[29208]: from='client.? 172.21.15.139:0/4208854438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1317fe63-aa4c-4687-8cdb-47f9531f654f"}]: dispatch 2024-04-25T00:07:49.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:49 smithi027 ceph-mon[29208]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1317fe63-aa4c-4687-8cdb-47f9531f654f"}]: dispatch 2024-04-25T00:07:49.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:49 smithi027 ceph-mon[29208]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1317fe63-aa4c-4687-8cdb-47f9531f654f"}]': finished 2024-04-25T00:07:49.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:49 smithi027 ceph-mon[29208]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T00:07:49.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:49 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:07:50.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:50 smithi139 ceph-mon[32833]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:50.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:50 smithi139 ceph-mon[32833]: from='client.? 172.21.15.139:0/92228364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:07:50.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:50 smithi153 ceph-mon[32896]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:50.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:50 smithi153 ceph-mon[32896]: from='client.? 172.21.15.139:0/92228364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:07:50.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:50 smithi027 ceph-mon[29208]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:50.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:50 smithi027 ceph-mon[29208]: from='client.? 172.21.15.139:0/92228364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:07:52.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:52 smithi027 ceph-mon[29208]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:53.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:52 smithi139 ceph-mon[32833]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:53.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:52 smithi153 ceph-mon[32896]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:54.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:54 smithi027 ceph-mon[29208]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:55.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:54 smithi139 ceph-mon[32833]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:55.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:54 smithi153 ceph-mon[32896]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:56.923 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:56 smithi139 ceph-mon[32833]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:56.923 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:56 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T00:07:56.923 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:56 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:56.923 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:56 smithi139 ceph-mon[32833]: Deploying daemon osd.1 on smithi139 2024-04-25T00:07:56.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:56 smithi027 ceph-mon[29208]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:56.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:56 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T00:07:56.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:56 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:56.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:56 smithi027 ceph-mon[29208]: Deploying daemon osd.1 on smithi139 2024-04-25T00:07:57.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:56 smithi153 ceph-mon[32896]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:57.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:56 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T00:07:57.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:56 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:07:57.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:56 smithi153 ceph-mon[32896]: Deploying daemon osd.1 on smithi139 2024-04-25T00:07:58.716 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:07:58 smithi139 ceph-mon[32833]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:58.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:07:58 smithi027 ceph-mon[29208]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:07:59.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:07:58 smithi153 ceph-mon[32896]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:08:00.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:00 smithi153 ceph-mon[32896]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:08:00.858 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:00 smithi139 ceph-mon[32833]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:08:00.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:00 smithi027 ceph-mon[29208]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:08:02.744 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:02 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:02.744 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:02 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:02.744 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:02 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:02.744 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:02 smithi027 ceph-mon[29208]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:08:02.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:02 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:02.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:02 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:02.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:02 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:02.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:02 smithi139 ceph-mon[32833]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:08:02.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:02 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:02.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:02 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:02.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:02 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:02.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:02 smithi153 ceph-mon[32896]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:08:05.229 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 1 on host 'smithi139' 2024-04-25T00:08:05.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:08:05.228+0000 7fe1e77fe700 1 -- 172.21.15.139:0/2340692791 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe1f8062670 con 0x7fe1e00a2b90 2024-04-25T00:08:05.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:08:05.231+0000 7fe1fc822700 1 -- 172.21.15.139:0/2340692791 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1e00a2b90 msgr2=0x7fe1e00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:05.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:08:05.231+0000 7fe1fc822700 1 --2- 172.21.15.139:0/2340692791 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1e00a2b90 0x7fe1e00a5050 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe1f81032f0 tx=0x7fe1e8008040 comp rx=0 tx=0).stop 2024-04-25T00:08:05.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:08:05.231+0000 7fe1fc822700 1 -- 172.21.15.139:0/2340692791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1f8103e30 msgr2=0x7fe1f819fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:05.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:08:05.231+0000 7fe1fc822700 1 --2- 172.21.15.139:0/2340692791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1f8103e30 0x7fe1f819fb70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe1f00062a0 tx=0x7fe1f0007e40 comp rx=0 tx=0).stop 2024-04-25T00:08:05.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:08:05.231+0000 7fe1fc822700 1 -- 172.21.15.139:0/2340692791 shutdown_connections 2024-04-25T00:08:05.232 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:08:05.231+0000 7fe1fc822700 1 --2- 172.21.15.139:0/2340692791 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1e00a2b90 0x7fe1e00a5050 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:05.232 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:08:05.231+0000 7fe1fc822700 1 --2- 172.21.15.139:0/2340692791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1f8103e30 0x7fe1f819fb70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:05.232 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:08:05.231+0000 7fe1fc822700 1 --2- 172.21.15.139:0/2340692791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f8102290 0x7fe1f81985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:05.232 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:08:05.231+0000 7fe1fc822700 1 --2- 172.21.15.139:0/2340692791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1f8103490 0x7fe1f8198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:05.232 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:08:05.231+0000 7fe1fc822700 1 -- 172.21.15.139:0/2340692791 >> 172.21.15.139:0/2340692791 conn(0x7fe1f80fd830 msgr2=0x7fe1f81067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:08:05.232 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:08:05.231+0000 7fe1fc822700 1 -- 172.21.15.139:0/2340692791 shutdown_connections 2024-04-25T00:08:05.232 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:08:05.231+0000 7fe1fc822700 1 -- 172.21.15.139:0/2340692791 wait complete. 2024-04-25T00:08:05.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:05 smithi139 ceph-mon[32833]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:08:05.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:05 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:05.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:05 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:05.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:05 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:05.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:05 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:08:05.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:05 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:05.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:05 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:05.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:05 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:05.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:05 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:05.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:05 smithi027 ceph-mon[29208]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:08:05.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:05 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:05.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:05 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:05.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:05 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:05.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:05 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:08:05.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:05 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:05.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:05 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:05.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:05 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:05.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:05 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:05.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:05 smithi153 ceph-mon[32896]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:08:05.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:05 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:05.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:05 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:05.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:05 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:05.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:05 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:08:05.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:05 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:05.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:05 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:05.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:05 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:05.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:05 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:05.822 DEBUG:teuthology.orchestra.run.smithi139:osd.1> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.1.service 2024-04-25T00:08:05.824 INFO:tasks.cephadm:Deploying osd.2 on smithi153 with /dev/vg_nvme/lv_4... 2024-04-25T00:08:05.824 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T00:08:06.070 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T00:08:06.514 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:06 smithi153 ceph-mon[32896]: from='osd.1 [v2:172.21.15.139:6800/1346503461,v1:172.21.15.139:6801/1346503461]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T00:08:06.514 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:06 smithi153 ceph-mon[32896]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T00:08:06.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:06 smithi027 ceph-mon[29208]: from='osd.1 [v2:172.21.15.139:6800/1346503461,v1:172.21.15.139:6801/1346503461]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T00:08:06.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:06 smithi027 ceph-mon[29208]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T00:08:06.718 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:06 smithi139 ceph-mon[32833]: from='osd.1 [v2:172.21.15.139:6800/1346503461,v1:172.21.15.139:6801/1346503461]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T00:08:06.718 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:06 smithi139 ceph-mon[32833]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T00:08:07.419 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:07 smithi153 ceph-mon[32896]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:08:07.419 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:07 smithi153 ceph-mon[32896]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T00:08:07.419 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:07 smithi153 ceph-mon[32896]: from='osd.1 [v2:172.21.15.139:6800/1346503461,v1:172.21.15.139:6801/1346503461]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:08:07.419 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:07 smithi153 ceph-mon[32896]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T00:08:07.419 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:07 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:08:07.419 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:07 smithi153 ceph-mon[32896]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:08:07.422 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:07 smithi139 ceph-mon[32833]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:08:07.422 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:07 smithi139 ceph-mon[32833]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T00:08:07.422 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:07 smithi139 ceph-mon[32833]: from='osd.1 [v2:172.21.15.139:6800/1346503461,v1:172.21.15.139:6801/1346503461]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:08:07.422 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:07 smithi139 ceph-mon[32833]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T00:08:07.422 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:07 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:08:07.422 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:07 smithi139 ceph-mon[32833]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:08:07.422 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:08:07 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-1[37413]: 2024-04-25T00:08:07.348+0000 7f56f3dde700 -1 osd.1 0 waiting for initial osdmap 2024-04-25T00:08:07.423 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:08:07 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-1[37413]: 2024-04-25T00:08:07.352+0000 7f56efbd1700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:08:07.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:07 smithi027 ceph-mon[29208]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:08:07.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:07 smithi027 ceph-mon[29208]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T00:08:07.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:07 smithi027 ceph-mon[29208]: from='osd.1 [v2:172.21.15.139:6800/1346503461,v1:172.21.15.139:6801/1346503461]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:08:07.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:07 smithi027 ceph-mon[29208]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T00:08:07.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:07 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:08:07.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:07 smithi027 ceph-mon[29208]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:08:07.994 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T00:08:07.994 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T00:08:07.994 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-25T00:08:07.994 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-25T00:08:07.994 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120043 s, 873 MB/s 2024-04-25T00:08:07.994 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-25T00:08:08.640 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:08 smithi153 ceph-mon[32896]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T00:08:08.640 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:08 smithi153 ceph-mon[32896]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T00:08:08.640 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:08:08.640 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:08:08.668 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_4 2024-04-25T00:08:08.669 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:08 smithi139 ceph-mon[32833]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T00:08:08.669 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:08 smithi139 ceph-mon[32833]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T00:08:08.669 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:08:08.669 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:08:08.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:08 smithi027 ceph-mon[29208]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T00:08:08.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:08 smithi027 ceph-mon[29208]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T00:08:08.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:08:08.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:08:08.909 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T00:08:09.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:09 smithi027 ceph-mon[29208]: purged_snaps scrub starts 2024-04-25T00:08:09.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:09 smithi027 ceph-mon[29208]: purged_snaps scrub ok 2024-04-25T00:08:09.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:09 smithi027 ceph-mon[29208]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:08:09.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:09 smithi027 ceph-mon[29208]: osd.1 [v2:172.21.15.139:6800/1346503461,v1:172.21.15.139:6801/1346503461] boot 2024-04-25T00:08:09.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:09 smithi027 ceph-mon[29208]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T00:08:09.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:09 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:08:09.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:09 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:09.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:09 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:09.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:09 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:08:09.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:09 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:09.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:09 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:09.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:09 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:08:09.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:09 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:09.703 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:09 smithi153 ceph-mon[32896]: purged_snaps scrub starts 2024-04-25T00:08:09.703 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:09 smithi153 ceph-mon[32896]: purged_snaps scrub ok 2024-04-25T00:08:09.703 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:09 smithi153 ceph-mon[32896]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:08:09.703 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:09 smithi153 ceph-mon[32896]: osd.1 [v2:172.21.15.139:6800/1346503461,v1:172.21.15.139:6801/1346503461] boot 2024-04-25T00:08:09.703 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:09 smithi153 ceph-mon[32896]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T00:08:09.703 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:09 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:08:09.703 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:09 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:09.703 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:09 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:09.703 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:09 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:08:09.703 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:09 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:09.703 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:09 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:09.703 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:09 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:08:09.704 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:09 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:09.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:09 smithi139 ceph-mon[32833]: purged_snaps scrub starts 2024-04-25T00:08:09.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:09 smithi139 ceph-mon[32833]: purged_snaps scrub ok 2024-04-25T00:08:09.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:09 smithi139 ceph-mon[32833]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T00:08:09.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:09 smithi139 ceph-mon[32833]: osd.1 [v2:172.21.15.139:6800/1346503461,v1:172.21.15.139:6801/1346503461] boot 2024-04-25T00:08:09.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:09 smithi139 ceph-mon[32833]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T00:08:09.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:09 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:08:09.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:09 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:09.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:09 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:09.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:09 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:08:09.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:09 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:09.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:09 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:09.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:09 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:08:09.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:09 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.579+0000 7f7f1acb0700 1 -- 172.21.15.153:0/2879868243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f14102d10 msgr2=0x7f7f14105100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.579+0000 7f7f1acb0700 1 --2- 172.21.15.153:0/2879868243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f14102d10 0x7f7f14105100 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009990 tx=0x7f7f04009ca0 comp rx=0 tx=0).stop 2024-04-25T00:08:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.579+0000 7f7f1acb0700 1 -- 172.21.15.153:0/2879868243 shutdown_connections 2024-04-25T00:08:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.579+0000 7f7f1acb0700 1 --2- 172.21.15.153:0/2879868243 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f14107f70 0x7f7f1410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.579+0000 7f7f1acb0700 1 --2- 172.21.15.153:0/2879868243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f14102d10 0x7f7f14105100 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.579+0000 7f7f1acb0700 1 --2- 172.21.15.153:0/2879868243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f14105640 0x7f7f14107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.579+0000 7f7f1acb0700 1 -- 172.21.15.153:0/2879868243 >> 172.21.15.153:0/2879868243 conn(0x7f7f140fc720 msgr2=0x7f7f140feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:08:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.579+0000 7f7f1acb0700 1 -- 172.21.15.153:0/2879868243 shutdown_connections 2024-04-25T00:08:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.579+0000 7f7f1acb0700 1 -- 172.21.15.153:0/2879868243 wait complete. 2024-04-25T00:08:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f1acb0700 1 Processor -- start 2024-04-25T00:08:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f1acb0700 1 -- start start 2024-04-25T00:08:10.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f1acb0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f14102d10 0x7f7f14198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f1acb0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f14105640 0x7f7f14198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f1acb0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f14107f70 0x7f7f1419fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f1acb0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1410df20 con 0x7f7f14102d10 2024-04-25T00:08:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f1acb0700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1410dda0 con 0x7f7f14107f70 2024-04-25T00:08:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f1acb0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1410e0a0 con 0x7f7f14105640 2024-04-25T00:08:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f0bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f14105640 0x7f7f14198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f0bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f14105640 0x7f7f14198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36300/0 (socket says 172.21.15.153:36300) 2024-04-25T00:08:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f0bfff700 1 -- 172.21.15.153:0/734486636 learned_addr learned my addr 172.21.15.153:0/734486636 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T00:08:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f18a4c700 1 --2- 172.21.15.153:0/734486636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f14102d10 0x7f7f14198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f1924d700 1 --2- 172.21.15.153:0/734486636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f14107f70 0x7f7f1419fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f0bfff700 1 -- 172.21.15.153:0/734486636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f14107f70 msgr2=0x7f7f1419fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f0bfff700 1 --2- 172.21.15.153:0/734486636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f14107f70 0x7f7f1419fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f0bfff700 1 -- 172.21.15.153:0/734486636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f14102d10 msgr2=0x7f7f14198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:10.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f0bfff700 1 --2- 172.21.15.153:0/734486636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f14102d10 0x7f7f14198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f0bfff700 1 -- 172.21.15.153:0/734486636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f141a0160 con 0x7f7f14105640 2024-04-25T00:08:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f1924d700 1 --2- 172.21.15.153:0/734486636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f14107f70 0x7f7f1419fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:08:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.580+0000 7f7f0bfff700 1 --2- 172.21.15.153:0/734486636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f14105640 0x7f7f14198ab0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000c950 tx=0x7f7f1000cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.581+0000 7f7f09ffb700 1 -- 172.21.15.153:0/734486636 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f10007ac0 con 0x7f7f14105640 2024-04-25T00:08:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.581+0000 7f7f09ffb700 1 -- 172.21.15.153:0/734486636 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f10004d10 con 0x7f7f14105640 2024-04-25T00:08:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.581+0000 7f7f1acb0700 1 -- 172.21.15.153:0/734486636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f141a0450 con 0x7f7f14105640 2024-04-25T00:08:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.581+0000 7f7f09ffb700 1 -- 172.21.15.153:0/734486636 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1000f450 con 0x7f7f14105640 2024-04-25T00:08:10.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.581+0000 7f7f1acb0700 1 -- 172.21.15.153:0/734486636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f141a0a60 con 0x7f7f14105640 2024-04-25T00:08:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.582+0000 7f7f1acb0700 1 -- 172.21.15.153:0/734486636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f14068f20 con 0x7f7f14105640 2024-04-25T00:08:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.583+0000 7f7f09ffb700 1 -- 172.21.15.153:0/734486636 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f7f10004750 con 0x7f7f14105640 2024-04-25T00:08:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.583+0000 7f7f09ffb700 1 --2- 172.21.15.153:0/734486636 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7efc0a2c10 0x7f7efc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.583+0000 7f7f09ffb700 1 -- 172.21.15.153:0/734486636 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f7f100cd610 con 0x7f7f14105640 2024-04-25T00:08:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.584+0000 7f7f18a4c700 1 --2- 172.21.15.153:0/734486636 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7efc0a2c10 0x7f7efc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:10.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.584+0000 7f7f18a4c700 1 --2- 172.21.15.153:0/734486636 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7efc0a2c10 0x7f7efc0a50d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7f0400b5c0 tx=0x7f7f04005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:10.586 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:10 smithi153 ceph-mon[32896]: Detected new or changed devices on smithi139 2024-04-25T00:08:10.586 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:10 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi139 to 17088M 2024-04-25T00:08:10.586 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:10 smithi153 ceph-mon[32896]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T00:08:10.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.588+0000 7f7f09ffb700 1 -- 172.21.15.153:0/734486636 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7f10014030 con 0x7f7f14105640 2024-04-25T00:08:10.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:10 smithi027 ceph-mon[29208]: Detected new or changed devices on smithi139 2024-04-25T00:08:10.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:10 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi139 to 17088M 2024-04-25T00:08:10.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:10 smithi027 ceph-mon[29208]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T00:08:10.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:10 smithi139 ceph-mon[32833]: Detected new or changed devices on smithi139 2024-04-25T00:08:10.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:10 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi139 to 17088M 2024-04-25T00:08:10.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:10 smithi139 ceph-mon[32833]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T00:08:10.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:10.793+0000 7f7f1acb0700 1 -- 172.21.15.153:0/734486636 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f7f14062670 con 0x7f7efc0a2c10 2024-04-25T00:08:11.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:11 smithi027 ceph-mon[29208]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:11.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:08:11.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:08:11.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:11.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:11 smithi139 ceph-mon[32833]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:11.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:08:11.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:08:11.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:11.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:11 smithi153 ceph-mon[32896]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:11.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:08:11.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:08:11.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:12.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:12 smithi153 ceph-mon[32896]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:08:12.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:12 smithi027 ceph-mon[29208]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:08:12.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:12 smithi139 ceph-mon[32833]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:08:13.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:13 smithi027 ceph-mon[29208]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:13.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:13 smithi139 ceph-mon[32833]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:13.767 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:13 smithi153 ceph-mon[32896]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:14.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:14 smithi027 ceph-mon[29208]: from='client.? 172.21.15.153:0/3762666320' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbc82ce9-2c51-4c2d-91d0-84132845c085"}]: dispatch 2024-04-25T00:08:14.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:14 smithi027 ceph-mon[29208]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbc82ce9-2c51-4c2d-91d0-84132845c085"}]: dispatch 2024-04-25T00:08:14.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:14 smithi027 ceph-mon[29208]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fbc82ce9-2c51-4c2d-91d0-84132845c085"}]': finished 2024-04-25T00:08:14.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:14 smithi027 ceph-mon[29208]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T00:08:14.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:14 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:08:14.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:14 smithi027 ceph-mon[29208]: from='client.? 172.21.15.153:0/3803760419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:08:14.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:14 smithi139 ceph-mon[32833]: from='client.? 172.21.15.153:0/3762666320' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbc82ce9-2c51-4c2d-91d0-84132845c085"}]: dispatch 2024-04-25T00:08:14.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:14 smithi139 ceph-mon[32833]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbc82ce9-2c51-4c2d-91d0-84132845c085"}]: dispatch 2024-04-25T00:08:14.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:14 smithi139 ceph-mon[32833]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fbc82ce9-2c51-4c2d-91d0-84132845c085"}]': finished 2024-04-25T00:08:14.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:14 smithi139 ceph-mon[32833]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T00:08:14.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:14 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:08:14.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:14 smithi139 ceph-mon[32833]: from='client.? 172.21.15.153:0/3803760419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:08:14.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:14 smithi153 ceph-mon[32896]: from='client.? 172.21.15.153:0/3762666320' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbc82ce9-2c51-4c2d-91d0-84132845c085"}]: dispatch 2024-04-25T00:08:14.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:14 smithi153 ceph-mon[32896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbc82ce9-2c51-4c2d-91d0-84132845c085"}]: dispatch 2024-04-25T00:08:14.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:14 smithi153 ceph-mon[32896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fbc82ce9-2c51-4c2d-91d0-84132845c085"}]': finished 2024-04-25T00:08:14.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:14 smithi153 ceph-mon[32896]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T00:08:14.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:14 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:08:14.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:14 smithi153 ceph-mon[32896]: from='client.? 172.21.15.153:0/3803760419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:08:15.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:15 smithi027 ceph-mon[29208]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:15.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:15 smithi139 ceph-mon[32833]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:15.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:15 smithi153 ceph-mon[32896]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:16.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:16 smithi027 ceph-mon[29208]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:16.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:16 smithi139 ceph-mon[32833]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:16.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:16 smithi153 ceph-mon[32896]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:18.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:18 smithi153 ceph-mon[32896]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:18.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:18 smithi027 ceph-mon[29208]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:19.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:18 smithi139 ceph-mon[32833]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:20.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:20 smithi153 ceph-mon[32896]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:20.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:20 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T00:08:20.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:20 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:20.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:20 smithi027 ceph-mon[29208]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:20.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:20 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T00:08:20.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:20 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:21.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:20 smithi139 ceph-mon[32833]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:21.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:20 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T00:08:21.007 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:20 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:21.746 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:21 smithi153 ceph-mon[32896]: Deploying daemon osd.2 on smithi153 2024-04-25T00:08:21.826 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:21 smithi139 ceph-mon[32833]: Deploying daemon osd.2 on smithi153 2024-04-25T00:08:21.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:21 smithi027 ceph-mon[29208]: Deploying daemon osd.2 on smithi153 2024-04-25T00:08:22.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:22 smithi153 ceph-mon[32896]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:22.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:22 smithi027 ceph-mon[29208]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:22.992 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:22 smithi139 ceph-mon[32833]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:24.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:24 smithi153 ceph-mon[32896]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:24.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:24 smithi027 ceph-mon[29208]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:25.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:24 smithi139 ceph-mon[32833]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:27.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:26 smithi027 ceph-mon[29208]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:27.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:26 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:27.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:26 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:27.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:26 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:27.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:26 smithi139 ceph-mon[32833]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:27.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:26 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:27.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:26 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:27.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:26 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:27.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:26 smithi153 ceph-mon[32896]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:27.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:26 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:27.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:26 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:27.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:26 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:29.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:28 smithi027 ceph-mon[29208]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:29.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:28 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:29.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:28 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:29.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:28 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:29.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:28 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:08:29.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:28 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:29.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:28 smithi139 ceph-mon[32833]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:29.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:28 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:29.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:28 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:29.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:28 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:29.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:28 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:08:29.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:28 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:29.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:28 smithi153 ceph-mon[32896]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:28 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:28 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:28 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:28 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:08:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:28 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:29.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:29.613+0000 7f7f09ffb700 1 -- 172.21.15.153:0/734486636 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7f14062670 con 0x7f7efc0a2c10 2024-04-25T00:08:29.615 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 2 on host 'smithi153' 2024-04-25T00:08:29.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:29.615+0000 7f7f1acb0700 1 -- 172.21.15.153:0/734486636 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7efc0a2c10 msgr2=0x7f7efc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:29.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:29.615+0000 7f7f1acb0700 1 --2- 172.21.15.153:0/734486636 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7efc0a2c10 0x7f7efc0a50d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7f0400b5c0 tx=0x7f7f04005fb0 comp rx=0 tx=0).stop 2024-04-25T00:08:29.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:29.615+0000 7f7f1acb0700 1 -- 172.21.15.153:0/734486636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f14105640 msgr2=0x7f7f14198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:29.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:29.615+0000 7f7f1acb0700 1 --2- 172.21.15.153:0/734486636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f14105640 0x7f7f14198ab0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000c950 tx=0x7f7f1000cc60 comp rx=0 tx=0).stop 2024-04-25T00:08:29.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:29.615+0000 7f7f1acb0700 1 -- 172.21.15.153:0/734486636 shutdown_connections 2024-04-25T00:08:29.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:29.615+0000 7f7f1acb0700 1 --2- 172.21.15.153:0/734486636 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7efc0a2c10 0x7f7efc0a50d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:29.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:29.615+0000 7f7f1acb0700 1 --2- 172.21.15.153:0/734486636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f14107f70 0x7f7f1419fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:29.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:29.615+0000 7f7f1acb0700 1 --2- 172.21.15.153:0/734486636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f14102d10 0x7f7f14198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:29.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:29.615+0000 7f7f1acb0700 1 --2- 172.21.15.153:0/734486636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f14105640 0x7f7f14198ab0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:29.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:29.615+0000 7f7f1acb0700 1 -- 172.21.15.153:0/734486636 >> 172.21.15.153:0/734486636 conn(0x7f7f140fc720 msgr2=0x7f7f141036b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:08:29.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:29.615+0000 7f7f1acb0700 1 -- 172.21.15.153:0/734486636 shutdown_connections 2024-04-25T00:08:29.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:08:29.615+0000 7f7f1acb0700 1 -- 172.21.15.153:0/734486636 wait complete. 2024-04-25T00:08:30.267 DEBUG:teuthology.orchestra.run.smithi153:osd.2> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.2.service 2024-04-25T00:08:30.269 INFO:tasks.cephadm:Deploying osd.3 on smithi027 with /dev/vg_nvme/lv_3... 2024-04-25T00:08:30.270 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T00:08:30.526 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:08:30.881 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:30 smithi027 ceph-mon[29208]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:30.881 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:30 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:30.881 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:30 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:30.881 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:30 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:30.881 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:30 smithi027 ceph-mon[29208]: from='osd.2 [v2:172.21.15.153:6800/662156492,v1:172.21.15.153:6801/662156492]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T00:08:30.882 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:30 smithi027 ceph-mon[29208]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T00:08:31.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:30 smithi139 ceph-mon[32833]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:31.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:30 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:31.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:30 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:31.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:30 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:31.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:30 smithi139 ceph-mon[32833]: from='osd.2 [v2:172.21.15.153:6800/662156492,v1:172.21.15.153:6801/662156492]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T00:08:31.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:30 smithi139 ceph-mon[32833]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T00:08:31.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:30 smithi153 ceph-mon[32896]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:31.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:30 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:31.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:30 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:31.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:30 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:31.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:30 smithi153 ceph-mon[32896]: from='osd.2 [v2:172.21.15.153:6800/662156492,v1:172.21.15.153:6801/662156492]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T00:08:31.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:30 smithi153 ceph-mon[32896]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T00:08:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:31 smithi153 ceph-mon[32896]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T00:08:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:31 smithi153 ceph-mon[32896]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T00:08:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:31 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:08:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:31 smithi153 ceph-mon[32896]: from='osd.2 [v2:172.21.15.153:6800/662156492,v1:172.21.15.153:6801/662156492]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:08:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:31 smithi153 ceph-mon[32896]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:08:31.807 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:08:31 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-2[38088]: 2024-04-25T00:08:31.589+0000 7f468f277700 -1 osd.2 0 waiting for initial osdmap 2024-04-25T00:08:31.807 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:08:31 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-2[38088]: 2024-04-25T00:08:31.594+0000 7f468b06a700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:08:31.889 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:31 smithi027 ceph-mon[29208]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T00:08:31.890 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:31 smithi027 ceph-mon[29208]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T00:08:31.890 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:31 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:08:31.890 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:31 smithi027 ceph-mon[29208]: from='osd.2 [v2:172.21.15.153:6800/662156492,v1:172.21.15.153:6801/662156492]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:08:31.890 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:31 smithi027 ceph-mon[29208]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:08:32.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:31 smithi139 ceph-mon[32833]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T00:08:32.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:31 smithi139 ceph-mon[32833]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T00:08:32.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:31 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:08:32.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:31 smithi139 ceph-mon[32833]: from='osd.2 [v2:172.21.15.153:6800/662156492,v1:172.21.15.153:6801/662156492]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:08:32.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:31 smithi139 ceph-mon[32833]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:08:32.690 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T00:08:32.690 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T00:08:32.690 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-25T00:08:32.690 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-25T00:08:32.690 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112382 s, 933 MB/s 2024-04-25T00:08:32.690 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-25T00:08:32.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:32 smithi139 ceph-mon[32833]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:32.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:32 smithi139 ceph-mon[32833]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T00:08:32.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:32 smithi139 ceph-mon[32833]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T00:08:32.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:32 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:08:32.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:32 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:08:32.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:32 smithi027 ceph-mon[29208]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:32.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:32 smithi027 ceph-mon[29208]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T00:08:32.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:32 smithi027 ceph-mon[29208]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T00:08:32.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:32 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:08:32.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:32 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:08:33.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:32 smithi153 ceph-mon[32896]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:33.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:32 smithi153 ceph-mon[32896]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T00:08:33.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:32 smithi153 ceph-mon[32896]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T00:08:33.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:32 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:08:33.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:32 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:08:33.395 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_3 2024-04-25T00:08:33.650 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:08:34.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:34 smithi027 ceph-mon[29208]: purged_snaps scrub starts 2024-04-25T00:08:34.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:34 smithi027 ceph-mon[29208]: purged_snaps scrub ok 2024-04-25T00:08:34.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:34 smithi027 ceph-mon[29208]: osd.2 [v2:172.21.15.153:6800/662156492,v1:172.21.15.153:6801/662156492] boot 2024-04-25T00:08:34.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:34 smithi027 ceph-mon[29208]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T00:08:34.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:08:34.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:34.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:34.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:08:34.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:34.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:34.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:08:34.096 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:34.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:34 smithi153 ceph-mon[32896]: purged_snaps scrub starts 2024-04-25T00:08:34.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:34 smithi153 ceph-mon[32896]: purged_snaps scrub ok 2024-04-25T00:08:34.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:34 smithi153 ceph-mon[32896]: osd.2 [v2:172.21.15.153:6800/662156492,v1:172.21.15.153:6801/662156492] boot 2024-04-25T00:08:34.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:34 smithi153 ceph-mon[32896]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T00:08:34.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:08:34.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:34.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:34.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:08:34.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:34.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:34.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:08:34.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:34 smithi139 ceph-mon[32833]: purged_snaps scrub starts 2024-04-25T00:08:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:34 smithi139 ceph-mon[32833]: purged_snaps scrub ok 2024-04-25T00:08:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:34 smithi139 ceph-mon[32833]: osd.2 [v2:172.21.15.153:6800/662156492,v1:172.21.15.153:6801/662156492] boot 2024-04-25T00:08:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:34 smithi139 ceph-mon[32833]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T00:08:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:08:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:34.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:34.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:08:34.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:34.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:34.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:08:34.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:35.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:35 smithi153 ceph-mon[32896]: Detected new or changed devices on smithi153 2024-04-25T00:08:35.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:35 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi153 to 17088M 2024-04-25T00:08:35.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:35 smithi153 ceph-mon[32896]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:35.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:35 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T00:08:35.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:35 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T00:08:35.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:35 smithi153 ceph-mon[32896]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T00:08:35.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:35 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T00:08:35.302 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:35 smithi027 ceph-mon[29208]: Detected new or changed devices on smithi153 2024-04-25T00:08:35.302 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:35 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi153 to 17088M 2024-04-25T00:08:35.302 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:35 smithi027 ceph-mon[29208]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:35.302 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:35 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T00:08:35.302 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:35 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T00:08:35.302 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:35 smithi027 ceph-mon[29208]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T00:08:35.302 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:35 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T00:08:35.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.306+0000 7f167492e700 1 -- 172.21.15.27:0/1143278584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16701022c0 msgr2=0x7f16701026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:35.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.306+0000 7f167492e700 1 --2- 172.21.15.27:0/1143278584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16701022c0 0x7f16701026e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f16600099c0 tx=0x7f1660009cd0 comp rx=0 tx=0).stop 2024-04-25T00:08:35.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.306+0000 7f167492e700 1 -- 172.21.15.27:0/1143278584 shutdown_connections 2024-04-25T00:08:35.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.306+0000 7f167492e700 1 --2- 172.21.15.27:0/1143278584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1670103e60 0x7f167010a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:35.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.306+0000 7f167492e700 1 --2- 172.21.15.27:0/1143278584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16701022c0 0x7f16701026e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:35.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.306+0000 7f167492e700 1 --2- 172.21.15.27:0/1143278584 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16701034c0 0x7f1670103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:35.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.306+0000 7f167492e700 1 -- 172.21.15.27:0/1143278584 >> 172.21.15.27:0/1143278584 conn(0x7f16700fd840 msgr2=0x7f16700ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:08:35.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.306+0000 7f167492e700 1 -- 172.21.15.27:0/1143278584 shutdown_connections 2024-04-25T00:08:35.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.306+0000 7f167492e700 1 -- 172.21.15.27:0/1143278584 wait complete. 2024-04-25T00:08:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.307+0000 7f167492e700 1 Processor -- start 2024-04-25T00:08:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.307+0000 7f167492e700 1 -- start start 2024-04-25T00:08:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.307+0000 7f167492e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16701022c0 0x7f16701984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.307+0000 7f167492e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16701034c0 0x7f1670198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.307+0000 7f167492e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1670103e60 0x7f167019fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.307+0000 7f167492e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f167010de80 con 0x7f16701034c0 2024-04-25T00:08:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.307+0000 7f167492e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f167010dd00 con 0x7f16701022c0 2024-04-25T00:08:35.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.307+0000 7f167492e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f167010e000 con 0x7f1670103e60 2024-04-25T00:08:35.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.308+0000 7f166e59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16701022c0 0x7f16701984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:35.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.308+0000 7f166dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16701034c0 0x7f1670198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:35.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.308+0000 7f166e59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16701022c0 0x7f16701984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:40882/0 (socket says 172.21.15.27:40882) 2024-04-25T00:08:35.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.308+0000 7f166e59c700 1 -- 172.21.15.27:0/2670453581 learned_addr learned my addr 172.21.15.27:0/2670453581 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:08:35.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.308+0000 7f166ed9d700 1 --2- 172.21.15.27:0/2670453581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1670103e60 0x7f167019fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:35.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.308+0000 7f166e59c700 1 -- 172.21.15.27:0/2670453581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1670103e60 msgr2=0x7f167019fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:35.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.308+0000 7f166e59c700 1 --2- 172.21.15.27:0/2670453581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1670103e60 0x7f167019fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:35.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.308+0000 7f166e59c700 1 -- 172.21.15.27:0/2670453581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16701034c0 msgr2=0x7f1670198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:35.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.308+0000 7f166e59c700 1 --2- 172.21.15.27:0/2670453581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16701034c0 0x7f1670198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:35.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.308+0000 7f166e59c700 1 -- 172.21.15.27:0/2670453581 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16701a0120 con 0x7f16701022c0 2024-04-25T00:08:35.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.308+0000 7f166dd9b700 1 --2- 172.21.15.27:0/2670453581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16701034c0 0x7f1670198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:08:35.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.308+0000 7f166e59c700 1 --2- 172.21.15.27:0/2670453581 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16701022c0 0x7f16701984c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f166000bf60 tx=0x7f166000bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:35.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.308+0000 7f165f7fe700 1 -- 172.21.15.27:0/2670453581 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660009e60 con 0x7f16701022c0 2024-04-25T00:08:35.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.308+0000 7f167492e700 1 -- 172.21.15.27:0/2670453581 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16701a03b0 con 0x7f16701022c0 2024-04-25T00:08:35.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.308+0000 7f165f7fe700 1 -- 172.21.15.27:0/2670453581 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f166000f460 con 0x7f16701022c0 2024-04-25T00:08:35.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.308+0000 7f165f7fe700 1 -- 172.21.15.27:0/2670453581 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660017630 con 0x7f16701022c0 2024-04-25T00:08:35.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.309+0000 7f167492e700 1 -- 172.21.15.27:0/2670453581 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16701a0890 con 0x7f16701022c0 2024-04-25T00:08:35.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.310+0000 7f167492e700 1 -- 172.21.15.27:0/2670453581 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1670068f20 con 0x7f16701022c0 2024-04-25T00:08:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.311+0000 7f165f7fe700 1 -- 172.21.15.27:0/2670453581 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f166000f610 con 0x7f16701022c0 2024-04-25T00:08:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.311+0000 7f165f7fe700 1 --2- 172.21.15.27:0/2670453581 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16580a2b90 0x7f16580a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.312+0000 7f165f7fe700 1 -- 172.21.15.27:0/2670453581 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7f16600ce940 con 0x7f16701022c0 2024-04-25T00:08:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.311+0000 7f166dd9b700 1 --2- 172.21.15.27:0/2670453581 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16580a2b90 0x7f16580a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.312+0000 7f166dd9b700 1 --2- 172.21.15.27:0/2670453581 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16580a2b90 0x7f16580a5050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1664011490 tx=0x7f1664009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.315+0000 7f165f7fe700 1 -- 172.21.15.27:0/2670453581 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1660013070 con 0x7f16701022c0 2024-04-25T00:08:35.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:35 smithi139 ceph-mon[32833]: Detected new or changed devices on smithi153 2024-04-25T00:08:35.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:35 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi153 to 17088M 2024-04-25T00:08:35.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:35 smithi139 ceph-mon[32833]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T00:08:35.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:35 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T00:08:35.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:35 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T00:08:35.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:35 smithi139 ceph-mon[32833]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T00:08:35.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:35 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T00:08:35.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:35.535+0000 7f167492e700 1 -- 172.21.15.27:0/2670453581 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1670062670 con 0x7f16580a2b90 2024-04-25T00:08:35.942 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:08:35 smithi027 sudo[45626]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T00:08:35.942 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:08:35 smithi027 sudo[45626]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T00:08:35.942 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:08:35 smithi027 sudo[45626]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T00:08:35.942 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:08:35 smithi027 sudo[45626]: pam_unix(sudo:session): session closed for user root 2024-04-25T00:08:36.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:36.030+0000 7f165f7fe700 1 -- 172.21.15.27:0/2670453581 <== mon.2 v2:172.21.15.139:3300/0 7 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1660098cb0 con 0x7f16701022c0 2024-04-25T00:08:36.299 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:08:36 smithi027 sudo[45687]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T00:08:36.299 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:08:36 smithi027 sudo[45687]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T00:08:36.299 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:08:36 smithi027 sudo[45687]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T00:08:36.299 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:08:36 smithi027 sudo[45687]: pam_unix(sudo:session): session closed for user root 2024-04-25T00:08:36.299 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:36 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T00:08:36.300 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:36 smithi027 ceph-mon[29208]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T00:08:36.300 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:36 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:08:36.300 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:36 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:08:36.300 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:36 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:36.325 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:36 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T00:08:36.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:36 smithi139 ceph-mon[32833]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T00:08:36.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:36 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:08:36.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:36 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:08:36.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:36 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:36.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:36 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T00:08:36.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:36 smithi153 ceph-mon[32896]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T00:08:36.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:36 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:08:36.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:36 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:08:36.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:36 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:36.579 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:08:36 smithi139 sudo[41377]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T00:08:36.579 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:08:36 smithi139 sudo[41377]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T00:08:36.580 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:08:36 smithi139 sudo[41377]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T00:08:36.580 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:08:36 smithi139 sudo[41377]: pam_unix(sudo:session): session closed for user root 2024-04-25T00:08:37.003 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:08:36 smithi139 sudo[41380]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T00:08:37.003 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:08:36 smithi139 sudo[41380]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T00:08:37.003 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:08:36 smithi139 sudo[41380]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T00:08:37.003 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:08:36 smithi139 sudo[41380]: pam_unix(sudo:session): session closed for user root 2024-04-25T00:08:37.098 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:08:36 smithi153 sudo[41827]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T00:08:37.098 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:08:36 smithi153 sudo[41827]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T00:08:37.098 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:08:36 smithi153 sudo[41827]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T00:08:37.098 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:08:36 smithi153 sudo[41827]: pam_unix(sudo:session): session closed for user root 2024-04-25T00:08:37.099 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:37 smithi153 ceph-mon[32896]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:37.099 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:37 smithi153 ceph-mon[32896]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:08:37.099 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:37 smithi153 ceph-mon[32896]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T00:08:37.099 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:37 smithi153 ceph-mon[32896]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T00:08:37.099 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:37 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:37.099 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:37 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:37.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:37 smithi027 ceph-mon[29208]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:37.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:37 smithi027 ceph-mon[29208]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:08:37.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:37 smithi027 ceph-mon[29208]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T00:08:37.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:37 smithi027 ceph-mon[29208]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T00:08:37.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:37 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:37.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:37 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:37.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:37 smithi139 ceph-mon[32833]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:37.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:37 smithi139 ceph-mon[32833]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:08:37.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:37 smithi139 ceph-mon[32833]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T00:08:37.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:37 smithi139 ceph-mon[32833]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T00:08:37.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:37 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:37.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:37 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:37.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:08:37 smithi153 sudo[41830]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T00:08:37.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:08:37 smithi153 sudo[41830]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T00:08:37.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:08:37 smithi153 sudo[41830]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T00:08:37.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:08:37 smithi153 sudo[41830]: pam_unix(sudo:session): session closed for user root 2024-04-25T00:08:37.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:37 smithi027 sudo[45753]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T00:08:37.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:37 smithi027 sudo[45753]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T00:08:37.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:37 smithi027 sudo[45753]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T00:08:37.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:37 smithi027 sudo[45753]: pam_unix(sudo:session): session closed for user root 2024-04-25T00:08:38.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:38 smithi027 sudo[45778]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T00:08:38.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:38 smithi027 sudo[45778]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T00:08:38.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:38 smithi027 sudo[45778]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T00:08:38.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:38 smithi027 sudo[45778]: pam_unix(sudo:session): session closed for user root 2024-04-25T00:08:38.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:38 smithi153 sudo[41833]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T00:08:38.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:38 smithi153 sudo[41833]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T00:08:38.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:38 smithi153 sudo[41833]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T00:08:38.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:38 smithi153 sudo[41833]: pam_unix(sudo:session): session closed for user root 2024-04-25T00:08:38.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:38 smithi153 sudo[41836]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T00:08:38.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:38 smithi153 sudo[41836]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T00:08:38.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:38 smithi153 sudo[41836]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T00:08:38.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:38 smithi153 sudo[41836]: pam_unix(sudo:session): session closed for user root 2024-04-25T00:08:38.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:38.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:38 smithi153 ceph-mon[32896]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T00:08:38.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:38 smithi153 ceph-mon[32896]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T00:08:38.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:08:38.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:08:38.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:08:38.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:08:38.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:08:38.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:08:38.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:38.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:38 smithi027 ceph-mon[29208]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T00:08:38.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:38 smithi027 ceph-mon[29208]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T00:08:38.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:08:38.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:08:38.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:08:38.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:08:38.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:08:38.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:08:39.017 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:38 smithi139 sudo[41383]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T00:08:39.017 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:38 smithi139 sudo[41383]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T00:08:39.017 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:38 smithi139 sudo[41383]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T00:08:39.017 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:38 smithi139 sudo[41383]: pam_unix(sudo:session): session closed for user root 2024-04-25T00:08:39.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 sudo[41386]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T00:08:39.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 sudo[41386]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T00:08:39.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 sudo[41386]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T00:08:39.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 sudo[41386]: pam_unix(sudo:session): session closed for user root 2024-04-25T00:08:39.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:39.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T00:08:39.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T00:08:39.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:08:39.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:08:39.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:08:39.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:08:39.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:08:39.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:08:39.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:39 smithi027 ceph-mon[29208]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:39.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:39 smithi027 ceph-mon[29208]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T00:08:39.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:39 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1025177066' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a5e90823-a44c-47e7-bef7-6274de4d59ee"}]: dispatch 2024-04-25T00:08:39.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:39 smithi027 ceph-mon[29208]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T00:08:39.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:39 smithi027 ceph-mon[29208]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T00:08:39.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:39 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:08:39.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:39 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1025177066' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a5e90823-a44c-47e7-bef7-6274de4d59ee"}]': finished 2024-04-25T00:08:39.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:39 smithi027 ceph-mon[29208]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T00:08:39.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:39 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:08:39.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:39 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:08:39.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:39 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:08:39.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:39 smithi027 ceph-mon[29208]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T00:08:39.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:39 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3481453368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:08:40.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:40.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T00:08:40.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1025177066' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a5e90823-a44c-47e7-bef7-6274de4d59ee"}]: dispatch 2024-04-25T00:08:40.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T00:08:40.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T00:08:40.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:08:40.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1025177066' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a5e90823-a44c-47e7-bef7-6274de4d59ee"}]': finished 2024-04-25T00:08:40.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T00:08:40.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:08:40.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:08:40.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:08:40.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T00:08:40.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:39 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3481453368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:08:40.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:39 smithi153 ceph-mon[32896]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:40.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:39 smithi153 ceph-mon[32896]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T00:08:40.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:39 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1025177066' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a5e90823-a44c-47e7-bef7-6274de4d59ee"}]: dispatch 2024-04-25T00:08:40.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:39 smithi153 ceph-mon[32896]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T00:08:40.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:39 smithi153 ceph-mon[32896]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T00:08:40.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:39 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:08:40.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:39 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1025177066' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a5e90823-a44c-47e7-bef7-6274de4d59ee"}]': finished 2024-04-25T00:08:40.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:39 smithi153 ceph-mon[32896]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T00:08:40.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:39 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:08:40.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:39 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:08:40.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:39 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:08:40.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:39 smithi153 ceph-mon[32896]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T00:08:40.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:39 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3481453368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:08:40.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:40 smithi027 ceph-mon[29208]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:41.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:40 smithi139 ceph-mon[32833]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:41.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:40 smithi153 ceph-mon[32896]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:42.937 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:42 smithi027 ceph-mon[29208]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:43.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:42 smithi139 ceph-mon[32833]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:43.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:42 smithi153 ceph-mon[32896]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:44.801 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:44 smithi027 ceph-mon[29208]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:44 smithi139 ceph-mon[32833]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:45.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:44 smithi153 ceph-mon[32896]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:46.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:46 smithi027 ceph-mon[29208]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:46.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:46 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T00:08:46.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:46 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:46.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:46 smithi027 ceph-mon[29208]: Deploying daemon osd.3 on smithi027 2024-04-25T00:08:47.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:46 smithi139 ceph-mon[32833]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:47.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:46 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T00:08:47.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:46 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:47.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:46 smithi139 ceph-mon[32833]: Deploying daemon osd.3 on smithi027 2024-04-25T00:08:47.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:46 smithi153 ceph-mon[32896]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:47.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:46 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T00:08:47.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:46 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:47.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:46 smithi153 ceph-mon[32896]: Deploying daemon osd.3 on smithi027 2024-04-25T00:08:48.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:48 smithi027 ceph-mon[29208]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:49.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:48 smithi139 ceph-mon[32833]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:49.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:48 smithi153 ceph-mon[32896]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:50.805 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:50 smithi139 ceph-mon[32833]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:50.828 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:50 smithi153 ceph-mon[32896]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:50.918 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:50 smithi027 ceph-mon[29208]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:53.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:52 smithi027 ceph-mon[29208]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:53.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:52 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:53.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:52 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:53.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:52 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:53.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:52 smithi139 ceph-mon[32833]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:53.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:52 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:53.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:52 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:53.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:52 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:53.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:52 smithi153 ceph-mon[32896]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:53.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:52 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:53.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:52 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:53.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:52 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:55.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:54 smithi027 ceph-mon[29208]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:55.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:54 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:55.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:54 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:55.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:54 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:55.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:54 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:08:55.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:54 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:55.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:54 smithi139 ceph-mon[32833]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:55.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:54 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:55.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:54 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:55.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:54 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:55.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:54 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:08:55.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:54 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:55.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:54 smithi153 ceph-mon[32896]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:55.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:54 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:55.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:54 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:55.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:54 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:55.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:54 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:08:55.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:54 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:55.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:55.746+0000 7f165f7fe700 1 -- 172.21.15.27:0/2670453581 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1670062670 con 0x7f16580a2b90 2024-04-25T00:08:55.748 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 3 on host 'smithi027' 2024-04-25T00:08:55.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:55.749+0000 7f167492e700 1 -- 172.21.15.27:0/2670453581 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16580a2b90 msgr2=0x7f16580a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:55.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:55.749+0000 7f167492e700 1 --2- 172.21.15.27:0/2670453581 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16580a2b90 0x7f16580a5050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1664011490 tx=0x7f1664009260 comp rx=0 tx=0).stop 2024-04-25T00:08:55.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:55.749+0000 7f167492e700 1 -- 172.21.15.27:0/2670453581 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16701022c0 msgr2=0x7f16701984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:55.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:55.749+0000 7f167492e700 1 --2- 172.21.15.27:0/2670453581 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16701022c0 0x7f16701984c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f166000bf60 tx=0x7f166000bf90 comp rx=0 tx=0).stop 2024-04-25T00:08:55.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:55.749+0000 7f167492e700 1 -- 172.21.15.27:0/2670453581 shutdown_connections 2024-04-25T00:08:55.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:55.749+0000 7f167492e700 1 --2- 172.21.15.27:0/2670453581 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16580a2b90 0x7f16580a5050 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:55.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:55.749+0000 7f167492e700 1 --2- 172.21.15.27:0/2670453581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1670103e60 0x7f167019fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:55.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:55.749+0000 7f167492e700 1 --2- 172.21.15.27:0/2670453581 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16701022c0 0x7f16701984c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:55.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:55.749+0000 7f167492e700 1 --2- 172.21.15.27:0/2670453581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16701034c0 0x7f1670198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:55.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:55.749+0000 7f167492e700 1 -- 172.21.15.27:0/2670453581 >> 172.21.15.27:0/2670453581 conn(0x7f16700fd840 msgr2=0x7f16701067c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:08:55.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:55.749+0000 7f167492e700 1 -- 172.21.15.27:0/2670453581 shutdown_connections 2024-04-25T00:08:55.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:08:55.749+0000 7f167492e700 1 -- 172.21.15.27:0/2670453581 wait complete. 2024-04-25T00:08:55.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:55 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:55.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:55 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:55.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:55 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:55.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:55 smithi027 ceph-mon[29208]: from='osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T00:08:56.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:55 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:56.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:55 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:56.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:55 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:56.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:55 smithi139 ceph-mon[32833]: from='osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T00:08:56.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:55 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:08:56.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:55 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:56.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:55 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:56.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:55 smithi153 ceph-mon[32896]: from='osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T00:08:56.427 DEBUG:teuthology.orchestra.run.smithi027:osd.3> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.3.service 2024-04-25T00:08:56.430 INFO:tasks.cephadm:Deploying osd.4 on smithi139 with /dev/vg_nvme/lv_3... 2024-04-25T00:08:56.430 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T00:08:56.675 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T00:08:57.049 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:56 smithi139 ceph-mon[32833]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:57.049 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:56 smithi139 ceph-mon[32833]: from='osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T00:08:57.049 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:56 smithi139 ceph-mon[32833]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T00:08:57.049 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:56 smithi139 ceph-mon[32833]: from='osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T00:08:57.049 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:56 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:08:57.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:56 smithi027 ceph-mon[29208]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:57.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:56 smithi027 ceph-mon[29208]: from='osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T00:08:57.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:56 smithi027 ceph-mon[29208]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T00:08:57.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:56 smithi027 ceph-mon[29208]: from='osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T00:08:57.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:56 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:08:57.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:56 smithi153 ceph-mon[32896]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:57.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:56 smithi153 ceph-mon[32896]: from='osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T00:08:57.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:56 smithi153 ceph-mon[32896]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T00:08:57.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:56 smithi153 ceph-mon[32896]: from='osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T00:08:57.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:56 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:08:58.442 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 00:08:58 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-3[47836]: 2024-04-25T00:08:58.174+0000 7fe7abaad700 -1 osd.3 0 waiting for initial osdmap 2024-04-25T00:08:58.443 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 00:08:58 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-3[47836]: 2024-04-25T00:08:58.179+0000 7fe7a3898700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:08:58.655 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T00:08:58.655 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T00:08:58.655 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2024-04-25T00:08:58.655 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2024-04-25T00:08:58.655 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112847 s, 929 MB/s 2024-04-25T00:08:58.655 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2024-04-25T00:08:59.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:58 smithi139 ceph-mon[32833]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:59.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:58 smithi139 ceph-mon[32833]: from='osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T00:08:59.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:58 smithi139 ceph-mon[32833]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T00:08:59.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:58 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:08:59.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:58 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:08:59.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:58 smithi139 ceph-mon[32833]: from='osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629]' entity='osd.3' 2024-04-25T00:08:59.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:58 smithi153 ceph-mon[32896]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:59.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:58 smithi153 ceph-mon[32896]: from='osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T00:08:59.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:58 smithi153 ceph-mon[32896]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T00:08:59.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:08:59.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:08:59.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:58 smithi153 ceph-mon[32896]: from='osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629]' entity='osd.3' 2024-04-25T00:08:59.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:58 smithi027 ceph-mon[29208]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T00:08:59.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:58 smithi027 ceph-mon[29208]: from='osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T00:08:59.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:58 smithi027 ceph-mon[29208]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T00:08:59.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:08:59.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:08:59.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:58 smithi027 ceph-mon[29208]: from='osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629]' entity='osd.3' 2024-04-25T00:08:59.339 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi139:vg_nvme/lv_3 2024-04-25T00:08:59.584 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T00:08:59.912 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:59 smithi139 ceph-mon[32833]: purged_snaps scrub starts 2024-04-25T00:08:59.913 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:59 smithi139 ceph-mon[32833]: purged_snaps scrub ok 2024-04-25T00:08:59.913 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:59 smithi139 ceph-mon[32833]: osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629] boot 2024-04-25T00:08:59.913 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:59 smithi139 ceph-mon[32833]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T00:08:59.913 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:08:59.913 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:59.913 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:59.913 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:08:59.913 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:08:59.913 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:08:59.913 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:08:59.913 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:08:59.913 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:08:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:00.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:59 smithi153 ceph-mon[32896]: purged_snaps scrub starts 2024-04-25T00:09:00.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:59 smithi153 ceph-mon[32896]: purged_snaps scrub ok 2024-04-25T00:09:00.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:59 smithi153 ceph-mon[32896]: osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629] boot 2024-04-25T00:09:00.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:59 smithi153 ceph-mon[32896]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T00:09:00.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:09:00.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:00.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:00.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:09:00.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:09:00.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:00.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:00.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:09:00.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:08:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:00.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:59 smithi027 ceph-mon[29208]: purged_snaps scrub starts 2024-04-25T00:09:00.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:59 smithi027 ceph-mon[29208]: purged_snaps scrub ok 2024-04-25T00:09:00.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:59 smithi027 ceph-mon[29208]: osd.3 [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629] boot 2024-04-25T00:09:00.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:59 smithi027 ceph-mon[29208]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T00:09:00.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:09:00.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:00.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:00.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:09:00.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:09:00.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:00.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:00.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:09:00.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:08:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:00.928 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:00 smithi139 ceph-mon[32833]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T00:09:00.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:00 smithi139 ceph-mon[32833]: Detected new or changed devices on smithi027 2024-04-25T00:09:00.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:00 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi027 to 8544M 2024-04-25T00:09:00.929 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:00 smithi139 ceph-mon[32833]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T00:09:01.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:00 smithi153 ceph-mon[32896]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T00:09:01.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:00 smithi153 ceph-mon[32896]: Detected new or changed devices on smithi027 2024-04-25T00:09:01.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:00 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi027 to 8544M 2024-04-25T00:09:01.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:00 smithi153 ceph-mon[32896]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T00:09:01.159 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.155+0000 7f7064491700 1 -- 172.21.15.139:0/2414413165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f705c100ca0 msgr2=0x7f705c10a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:01.159 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.155+0000 7f7064491700 1 --2- 172.21.15.139:0/2414413165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f705c100ca0 0x7f705c10a870 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f705400b0a0 tx=0x7f705400b3b0 comp rx=0 tx=0).stop 2024-04-25T00:09:01.159 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.155+0000 7f7064491700 1 -- 172.21.15.139:0/2414413165 shutdown_connections 2024-04-25T00:09:01.159 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.155+0000 7f7064491700 1 --2- 172.21.15.139:0/2414413165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f705c100ca0 0x7f705c10a870 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:01.159 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.155+0000 7f7064491700 1 --2- 172.21.15.139:0/2414413165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705c103880 0x7f705c103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:01.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.155+0000 7f7064491700 1 --2- 172.21.15.139:0/2414413165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f705c100300 0x7f705c100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:01.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.156+0000 7f7064491700 1 -- 172.21.15.139:0/2414413165 >> 172.21.15.139:0/2414413165 conn(0x7f705c0fbea0 msgr2=0x7f705c0fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:01.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.156+0000 7f7064491700 1 -- 172.21.15.139:0/2414413165 shutdown_connections 2024-04-25T00:09:01.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.156+0000 7f7064491700 1 -- 172.21.15.139:0/2414413165 wait complete. 2024-04-25T00:09:01.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.156+0000 7f7064491700 1 Processor -- start 2024-04-25T00:09:01.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.156+0000 7f7064491700 1 -- start start 2024-04-25T00:09:01.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.156+0000 7f7064491700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f705c100300 0x7f705c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:01.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.156+0000 7f7064491700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705c100ca0 0x7f705c1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:01.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.156+0000 7f7064491700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f705c103880 0x7f705c19fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:01.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.156+0000 7f7064491700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f705c10dee0 con 0x7f705c100ca0 2024-04-25T00:09:01.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.156+0000 7f7064491700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f705c10dd60 con 0x7f705c100300 2024-04-25T00:09:01.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.156+0000 7f7064491700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f705c10e060 con 0x7f705c103880 2024-04-25T00:09:01.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f706222d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f705c100300 0x7f705c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:01.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f706222d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f705c100300 0x7f705c198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:44810/0 (socket says 172.21.15.139:44810) 2024-04-25T00:09:01.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f706222d700 1 -- 172.21.15.139:0/4070988402 learned_addr learned my addr 172.21.15.139:0/4070988402 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-04-25T00:09:01.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f7062a2e700 1 --2- 172.21.15.139:0/4070988402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f705c103880 0x7f705c19fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:01.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f706222d700 1 -- 172.21.15.139:0/4070988402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f705c103880 msgr2=0x7f705c19fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:01.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f706222d700 1 --2- 172.21.15.139:0/4070988402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f705c103880 0x7f705c19fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:01.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f706222d700 1 -- 172.21.15.139:0/4070988402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705c100ca0 msgr2=0x7f705c1989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:01.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f7061a2c700 1 --2- 172.21.15.139:0/4070988402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705c100ca0 0x7f705c1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:01.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f706222d700 1 --2- 172.21.15.139:0/4070988402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705c100ca0 0x7f705c1989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:01.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f706222d700 1 -- 172.21.15.139:0/4070988402 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f705c1a00f0 con 0x7f705c100300 2024-04-25T00:09:01.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f7062a2e700 1 --2- 172.21.15.139:0/4070988402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f705c103880 0x7f705c19fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:09:01.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f7061a2c700 1 --2- 172.21.15.139:0/4070988402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705c100ca0 0x7f705c1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:09:01.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f706222d700 1 --2- 172.21.15.139:0/4070988402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f705c100300 0x7f705c198490 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f705800c970 tx=0x7f705800cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:01.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f70537fe700 1 -- 172.21.15.139:0/4070988402 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7058004220 con 0x7f705c100300 2024-04-25T00:09:01.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f7064491700 1 -- 172.21.15.139:0/4070988402 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f705c1a03e0 con 0x7f705c100300 2024-04-25T00:09:01.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f70537fe700 1 -- 172.21.15.139:0/4070988402 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7058007d80 con 0x7f705c100300 2024-04-25T00:09:01.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f70537fe700 1 -- 172.21.15.139:0/4070988402 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705800f450 con 0x7f705c100300 2024-04-25T00:09:01.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.157+0000 7f7064491700 1 -- 172.21.15.139:0/4070988402 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f705c1a09f0 con 0x7f705c100300 2024-04-25T00:09:01.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.158+0000 7f7064491700 1 -- 172.21.15.139:0/4070988402 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f705c068f20 con 0x7f705c100300 2024-04-25T00:09:01.163 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.159+0000 7f70537fe700 1 -- 172.21.15.139:0/4070988402 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f70580078a0 con 0x7f705c100300 2024-04-25T00:09:01.163 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.160+0000 7f70537fe700 1 --2- 172.21.15.139:0/4070988402 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f70480a2ca0 0x7f70480a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:01.163 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.160+0000 7f70537fe700 1 -- 172.21.15.139:0/4070988402 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7f70580cde60 con 0x7f705c100300 2024-04-25T00:09:01.163 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.160+0000 7f7061a2c700 1 --2- 172.21.15.139:0/4070988402 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f70480a2ca0 0x7f70480a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:01.163 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.161+0000 7f7061a2c700 1 --2- 172.21.15.139:0/4070988402 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f70480a2ca0 0x7f70480a5160 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f704c007900 tx=0x7f704c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:01.164 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.164+0000 7f70537fe700 1 -- 172.21.15.139:0/4070988402 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7058014030 con 0x7f705c100300 2024-04-25T00:09:01.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:00 smithi027 ceph-mon[29208]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T00:09:01.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:00 smithi027 ceph-mon[29208]: Detected new or changed devices on smithi027 2024-04-25T00:09:01.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:00 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi027 to 8544M 2024-04-25T00:09:01.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:00 smithi027 ceph-mon[29208]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T00:09:01.370 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:01.369+0000 7f7064491700 1 -- 172.21.15.139:0/4070988402 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f705c062670 con 0x7f70480a2ca0 2024-04-25T00:09:01.994 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:01 smithi139 ceph-mon[32833]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T00:09:01.995 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:01 smithi139 ceph-mon[32833]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:09:01.995 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:01 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:09:01.995 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:01 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:09:01.995 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:01 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:02.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:01 smithi153 ceph-mon[32896]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T00:09:02.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:01 smithi153 ceph-mon[32896]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:09:02.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:01 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:09:02.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:01 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:09:02.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:01 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:02.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:01 smithi027 ceph-mon[29208]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T00:09:02.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:01 smithi027 ceph-mon[29208]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:09:02.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:01 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:09:02.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:01 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:09:02.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:01 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:02.933 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:02 smithi139 ceph-mon[32833]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T00:09:02.933 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:02 smithi139 ceph-mon[32833]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T00:09:03.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:02 smithi153 ceph-mon[32896]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T00:09:03.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:02 smithi153 ceph-mon[32896]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T00:09:03.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:02 smithi027 ceph-mon[29208]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T00:09:03.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:02 smithi027 ceph-mon[29208]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T00:09:04.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:03 smithi139 ceph-mon[32833]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T00:09:04.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:03 smithi153 ceph-mon[32896]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T00:09:04.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:03 smithi027 ceph-mon[29208]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T00:09:05.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:04 smithi139 ceph-mon[32833]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T00:09:05.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:04 smithi139 ceph-mon[32833]: from='client.? 172.21.15.139:0/3680776036' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be18fdcd-c36d-4176-8292-90b0aaa85cb8"}]: dispatch 2024-04-25T00:09:05.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:04 smithi139 ceph-mon[32833]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be18fdcd-c36d-4176-8292-90b0aaa85cb8"}]: dispatch 2024-04-25T00:09:05.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:04 smithi139 ceph-mon[32833]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be18fdcd-c36d-4176-8292-90b0aaa85cb8"}]': finished 2024-04-25T00:09:05.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:04 smithi139 ceph-mon[32833]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T00:09:05.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:04 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:09:05.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:04 smithi139 ceph-mon[32833]: from='client.? 172.21.15.139:0/1818105885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:09:05.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:04 smithi153 ceph-mon[32896]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T00:09:05.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:04 smithi153 ceph-mon[32896]: from='client.? 172.21.15.139:0/3680776036' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be18fdcd-c36d-4176-8292-90b0aaa85cb8"}]: dispatch 2024-04-25T00:09:05.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:04 smithi153 ceph-mon[32896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be18fdcd-c36d-4176-8292-90b0aaa85cb8"}]: dispatch 2024-04-25T00:09:05.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:04 smithi153 ceph-mon[32896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be18fdcd-c36d-4176-8292-90b0aaa85cb8"}]': finished 2024-04-25T00:09:05.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:04 smithi153 ceph-mon[32896]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T00:09:05.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:04 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:09:05.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:04 smithi153 ceph-mon[32896]: from='client.? 172.21.15.139:0/1818105885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:09:05.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:04 smithi027 ceph-mon[29208]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T00:09:05.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:04 smithi027 ceph-mon[29208]: from='client.? 172.21.15.139:0/3680776036' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be18fdcd-c36d-4176-8292-90b0aaa85cb8"}]: dispatch 2024-04-25T00:09:05.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:04 smithi027 ceph-mon[29208]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be18fdcd-c36d-4176-8292-90b0aaa85cb8"}]: dispatch 2024-04-25T00:09:05.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:04 smithi027 ceph-mon[29208]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be18fdcd-c36d-4176-8292-90b0aaa85cb8"}]': finished 2024-04-25T00:09:05.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:04 smithi027 ceph-mon[29208]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T00:09:05.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:04 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:09:05.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:04 smithi027 ceph-mon[29208]: from='client.? 172.21.15.139:0/1818105885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:09:07.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:06 smithi139 ceph-mon[32833]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-25T00:09:07.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:06 smithi153 ceph-mon[32896]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-25T00:09:07.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:06 smithi027 ceph-mon[29208]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-25T00:09:09.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:08 smithi153 ceph-mon[32896]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T00:09:09.081 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:08 smithi139 ceph-mon[32833]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T00:09:09.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:08 smithi027 ceph-mon[29208]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T00:09:11.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:10 smithi139 ceph-mon[32833]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-25T00:09:11.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:10 smithi153 ceph-mon[32896]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-25T00:09:11.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:10 smithi027 ceph-mon[29208]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-25T00:09:12.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T00:09:12.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:12.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T00:09:12.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:12.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T00:09:12.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:13.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:12 smithi153 ceph-mon[32896]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-25T00:09:13.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:12 smithi153 ceph-mon[32896]: Deploying daemon osd.4 on smithi139 2024-04-25T00:09:13.185 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:12 smithi139 ceph-mon[32833]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-25T00:09:13.186 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:12 smithi139 ceph-mon[32833]: Deploying daemon osd.4 on smithi139 2024-04-25T00:09:13.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:12 smithi027 ceph-mon[29208]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-25T00:09:13.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:12 smithi027 ceph-mon[29208]: Deploying daemon osd.4 on smithi139 2024-04-25T00:09:15.017 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:14 smithi139 ceph-mon[32833]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T00:09:15.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:14 smithi153 ceph-mon[32896]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T00:09:15.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:14 smithi027 ceph-mon[29208]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T00:09:16.949 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:16 smithi139 ceph-mon[32833]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-25T00:09:17.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:16 smithi153 ceph-mon[32896]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-25T00:09:17.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:16 smithi027 ceph-mon[29208]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-25T00:09:18.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:18 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:18.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:18 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:18.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:18 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:09:18.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:18 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:18.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:18 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:18.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:18 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:09:18.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:18 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:18.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:18 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:18.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:18 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:09:19.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:19 smithi139 ceph-mon[32833]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T00:09:19.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:19 smithi153 ceph-mon[32896]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T00:09:19.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:19 smithi027 ceph-mon[29208]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T00:09:20.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:20.786+0000 7f70537fe700 1 -- 172.21.15.139:0/4070988402 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f705c062670 con 0x7f70480a2ca0 2024-04-25T00:09:20.789 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 4 on host 'smithi139' 2024-04-25T00:09:20.789 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:20.789+0000 7f7064491700 1 -- 172.21.15.139:0/4070988402 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f70480a2ca0 msgr2=0x7f70480a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:20.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:20.789+0000 7f7064491700 1 --2- 172.21.15.139:0/4070988402 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f70480a2ca0 0x7f70480a5160 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f704c007900 tx=0x7f704c008040 comp rx=0 tx=0).stop 2024-04-25T00:09:20.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:20.789+0000 7f7064491700 1 -- 172.21.15.139:0/4070988402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f705c100300 msgr2=0x7f705c198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:20.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:20.789+0000 7f7064491700 1 --2- 172.21.15.139:0/4070988402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f705c100300 0x7f705c198490 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f705800c970 tx=0x7f705800cd30 comp rx=0 tx=0).stop 2024-04-25T00:09:20.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:20.789+0000 7f7064491700 1 -- 172.21.15.139:0/4070988402 shutdown_connections 2024-04-25T00:09:20.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:20.789+0000 7f7064491700 1 --2- 172.21.15.139:0/4070988402 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f70480a2ca0 0x7f70480a5160 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:20.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:20.789+0000 7f7064491700 1 --2- 172.21.15.139:0/4070988402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f705c103880 0x7f705c19fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:20.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:20.789+0000 7f7064491700 1 --2- 172.21.15.139:0/4070988402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f705c100300 0x7f705c198490 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:20.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:20.789+0000 7f7064491700 1 --2- 172.21.15.139:0/4070988402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705c100ca0 0x7f705c1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:20.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:20.789+0000 7f7064491700 1 -- 172.21.15.139:0/4070988402 >> 172.21.15.139:0/4070988402 conn(0x7f705c0fbea0 msgr2=0x7f705c106850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:20.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:20.789+0000 7f7064491700 1 -- 172.21.15.139:0/4070988402 shutdown_connections 2024-04-25T00:09:20.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:09:20.789+0000 7f7064491700 1 -- 172.21.15.139:0/4070988402 wait complete. 2024-04-25T00:09:21.364 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:21 smithi139 ceph-mon[32833]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T00:09:21.365 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:21 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:21.365 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:21 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:21.365 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:21 smithi139 ceph-mon[32833]: from='osd.4 [v2:172.21.15.139:6808/856291222,v1:172.21.15.139:6809/856291222]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T00:09:21.365 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:21 smithi139 ceph-mon[32833]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T00:09:21.390 DEBUG:teuthology.orchestra.run.smithi139:osd.4> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.4.service 2024-04-25T00:09:21.392 INFO:tasks.cephadm:Deploying osd.5 on smithi153 with /dev/vg_nvme/lv_3... 2024-04-25T00:09:21.393 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T00:09:21.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:21 smithi153 ceph-mon[32896]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T00:09:21.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:21 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:21.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:21 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:21.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:21 smithi153 ceph-mon[32896]: from='osd.4 [v2:172.21.15.139:6808/856291222,v1:172.21.15.139:6809/856291222]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T00:09:21.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:21 smithi153 ceph-mon[32896]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T00:09:21.642 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T00:09:21.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:21 smithi027 ceph-mon[29208]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T00:09:21.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:21 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:21.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:21 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:21.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:21 smithi027 ceph-mon[29208]: from='osd.4 [v2:172.21.15.139:6808/856291222,v1:172.21.15.139:6809/856291222]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T00:09:21.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:21 smithi027 ceph-mon[29208]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T00:09:22.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:22 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:22.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:22 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:22.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:22 smithi139 ceph-mon[32833]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T00:09:22.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:22 smithi139 ceph-mon[32833]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T00:09:22.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:22 smithi139 ceph-mon[32833]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T00:09:22.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:22 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:09:22.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:22 smithi139 ceph-mon[32833]: from='osd.4 [v2:172.21.15.139:6808/856291222,v1:172.21.15.139:6809/856291222]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:09:22.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:22 smithi139 ceph-mon[32833]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:09:22.764 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:22 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:22.764 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:22 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:22.764 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:22 smithi153 ceph-mon[32896]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T00:09:22.764 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:22 smithi153 ceph-mon[32896]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T00:09:22.764 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:22 smithi153 ceph-mon[32896]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T00:09:22.764 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:22 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:09:22.764 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:22 smithi153 ceph-mon[32896]: from='osd.4 [v2:172.21.15.139:6808/856291222,v1:172.21.15.139:6809/856291222]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:09:22.764 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:22 smithi153 ceph-mon[32896]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:09:22.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:22.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:22.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:22 smithi027 ceph-mon[29208]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T00:09:22.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:22 smithi027 ceph-mon[29208]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T00:09:22.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:22 smithi027 ceph-mon[29208]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T00:09:22.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:09:22.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:22 smithi027 ceph-mon[29208]: from='osd.4 [v2:172.21.15.139:6808/856291222,v1:172.21.15.139:6809/856291222]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:09:22.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:22 smithi027 ceph-mon[29208]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:09:23.131 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 00:09:22 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-4[44194]: 2024-04-25T00:09:22.758+0000 7f5af9444700 -1 osd.4 0 waiting for initial osdmap 2024-04-25T00:09:23.131 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 00:09:22 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-4[44194]: 2024-04-25T00:09:22.763+0000 7f5af5237700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:09:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T00:09:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T00:09:23.599 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-25T00:09:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-25T00:09:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0117042 s, 896 MB/s 2024-04-25T00:09:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-25T00:09:23.982 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:23 smithi153 ceph-mon[32896]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T00:09:23.982 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:23 smithi153 ceph-mon[32896]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T00:09:23.982 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:23 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:09:23.982 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:23 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:09:24.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:23 smithi027 ceph-mon[29208]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T00:09:24.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:23 smithi027 ceph-mon[29208]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T00:09:24.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:23 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:09:24.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:23 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:09:24.211 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:23 smithi139 ceph-mon[32833]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T00:09:24.211 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:23 smithi139 ceph-mon[32833]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T00:09:24.212 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:23 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:09:24.212 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:23 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:09:24.326 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_3 2024-04-25T00:09:24.561 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T00:09:24.946 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:24 smithi139 ceph-mon[32833]: purged_snaps scrub starts 2024-04-25T00:09:24.946 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:24 smithi139 ceph-mon[32833]: purged_snaps scrub ok 2024-04-25T00:09:24.946 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:24 smithi139 ceph-mon[32833]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T00:09:24.946 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:24 smithi139 ceph-mon[32833]: osd.4 [v2:172.21.15.139:6808/856291222,v1:172.21.15.139:6809/856291222] boot 2024-04-25T00:09:24.946 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:24 smithi139 ceph-mon[32833]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T00:09:24.946 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:24 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:09:24.961 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:24 smithi153 ceph-mon[32896]: purged_snaps scrub starts 2024-04-25T00:09:24.961 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:24 smithi153 ceph-mon[32896]: purged_snaps scrub ok 2024-04-25T00:09:24.961 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:24 smithi153 ceph-mon[32896]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T00:09:24.961 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:24 smithi153 ceph-mon[32896]: osd.4 [v2:172.21.15.139:6808/856291222,v1:172.21.15.139:6809/856291222] boot 2024-04-25T00:09:24.961 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:24 smithi153 ceph-mon[32896]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T00:09:24.961 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:24 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:09:25.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:24 smithi027 ceph-mon[29208]: purged_snaps scrub starts 2024-04-25T00:09:25.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:24 smithi027 ceph-mon[29208]: purged_snaps scrub ok 2024-04-25T00:09:25.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:24 smithi027 ceph-mon[29208]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T00:09:25.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:24 smithi027 ceph-mon[29208]: osd.4 [v2:172.21.15.139:6808/856291222,v1:172.21.15.139:6809/856291222] boot 2024-04-25T00:09:25.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:24 smithi027 ceph-mon[29208]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T00:09:25.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:24 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:09:25.892 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:25 smithi153 ceph-mon[32896]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T00:09:25.892 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:25 smithi153 ceph-mon[32896]: Detected new or changed devices on smithi139 2024-04-25T00:09:25.892 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:25 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:25.892 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:25 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:25.892 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:25 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:09:25.892 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:25 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:09:25.892 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:25 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi139 to 8544M 2024-04-25T00:09:25.892 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:25 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:25.892 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:25 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:25.893 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:25 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:09:25.893 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:25 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:25.893 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:25 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:09:25.893 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:25 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:25.893 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:25 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:09:25.893 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:25 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:26.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.095+0000 7f7c6516f700 1 -- 172.21.15.153:0/3219077640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c60107f80 msgr2=0x7f7c6010a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:26.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.095+0000 7f7c6516f700 1 --2- 172.21.15.153:0/3219077640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c60107f80 0x7f7c6010a3f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800b0a0 tx=0x7f7c5800b3b0 comp rx=0 tx=0).stop 2024-04-25T00:09:26.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.095+0000 7f7c6516f700 1 -- 172.21.15.153:0/3219077640 shutdown_connections 2024-04-25T00:09:26.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.095+0000 7f7c6516f700 1 --2- 172.21.15.153:0/3219077640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c60107f80 0x7f7c6010a3f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:26.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.095+0000 7f7c6516f700 1 --2- 172.21.15.153:0/3219077640 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c60102d20 0x7f7c60105110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:26.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.095+0000 7f7c6516f700 1 --2- 172.21.15.153:0/3219077640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c60105650 0x7f7c60107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:26.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.095+0000 7f7c6516f700 1 -- 172.21.15.153:0/3219077640 >> 172.21.15.153:0/3219077640 conn(0x7f7c600fc730 msgr2=0x7f7c600feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:26.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.095+0000 7f7c6516f700 1 -- 172.21.15.153:0/3219077640 shutdown_connections 2024-04-25T00:09:26.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.095+0000 7f7c6516f700 1 -- 172.21.15.153:0/3219077640 wait complete. 2024-04-25T00:09:26.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c6516f700 1 Processor -- start 2024-04-25T00:09:26.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c6516f700 1 -- start start 2024-04-25T00:09:26.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c6516f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c60102d20 0x7f7c60198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:26.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c6516f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c60105650 0x7f7c601989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:26.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c6516f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c60107f80 0x7f7c6019fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:26.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c6516f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6010dea0 con 0x7f7c60105650 2024-04-25T00:09:26.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c6516f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6010dd20 con 0x7f7c60102d20 2024-04-25T00:09:26.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c6516f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6010e020 con 0x7f7c60107f80 2024-04-25T00:09:26.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c5f59e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c60107f80 0x7f7c6019fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:26.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c5f59e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c60107f80 0x7f7c6019fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55996/0 (socket says 172.21.15.153:55996) 2024-04-25T00:09:26.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c5f59e700 1 -- 172.21.15.153:0/2311145997 learned_addr learned my addr 172.21.15.153:0/2311145997 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T00:09:26.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c57fff700 1 --2- 172.21.15.153:0/2311145997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c60105650 0x7f7c601989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:26.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c5ed9d700 1 --2- 172.21.15.153:0/2311145997 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c60102d20 0x7f7c60198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:26.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c5f59e700 1 -- 172.21.15.153:0/2311145997 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c60102d20 msgr2=0x7f7c60198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:26.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c5f59e700 1 --2- 172.21.15.153:0/2311145997 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c60102d20 0x7f7c60198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:26.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c5f59e700 1 -- 172.21.15.153:0/2311145997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c60105650 msgr2=0x7f7c601989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:26.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c5f59e700 1 --2- 172.21.15.153:0/2311145997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c60105650 0x7f7c601989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:26.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c5f59e700 1 -- 172.21.15.153:0/2311145997 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c601a00e0 con 0x7f7c60107f80 2024-04-25T00:09:26.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.096+0000 7f7c57fff700 1 --2- 172.21.15.153:0/2311145997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c60105650 0x7f7c601989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:09:26.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.097+0000 7f7c5f59e700 1 --2- 172.21.15.153:0/2311145997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c60107f80 0x7f7c6019fa40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800d010 tx=0x7f7c58004dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:26.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.097+0000 7f7c5cd99700 1 -- 172.21.15.153:0/2311145997 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c58021070 con 0x7f7c60107f80 2024-04-25T00:09:26.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.097+0000 7f7c5cd99700 1 -- 172.21.15.153:0/2311145997 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c58007c00 con 0x7f7c60107f80 2024-04-25T00:09:26.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.097+0000 7f7c5cd99700 1 -- 172.21.15.153:0/2311145997 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5801d450 con 0x7f7c60107f80 2024-04-25T00:09:26.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.097+0000 7f7c6516f700 1 -- 172.21.15.153:0/2311145997 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c601a0370 con 0x7f7c60107f80 2024-04-25T00:09:26.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.097+0000 7f7c6516f700 1 -- 172.21.15.153:0/2311145997 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c601a0820 con 0x7f7c60107f80 2024-04-25T00:09:26.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.098+0000 7f7c6516f700 1 -- 172.21.15.153:0/2311145997 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c60068f20 con 0x7f7c60107f80 2024-04-25T00:09:26.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.099+0000 7f7c5cd99700 1 -- 172.21.15.153:0/2311145997 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7c58004000 con 0x7f7c60107f80 2024-04-25T00:09:26.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.100+0000 7f7c5cd99700 1 --2- 172.21.15.153:0/2311145997 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7c4c0a2c20 0x7f7c4c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:26.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.100+0000 7f7c5cd99700 1 -- 172.21.15.153:0/2311145997 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f7c5800f020 con 0x7f7c60107f80 2024-04-25T00:09:26.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.100+0000 7f7c5ed9d700 1 --2- 172.21.15.153:0/2311145997 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7c4c0a2c20 0x7f7c4c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:26.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.100+0000 7f7c5ed9d700 1 --2- 172.21.15.153:0/2311145997 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7c4c0a2c20 0x7f7c4c0a50e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7c50005950 tx=0x7f7c5000a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:26.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.103+0000 7f7c5cd99700 1 -- 172.21.15.153:0/2311145997 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7c5802a020 con 0x7f7c60107f80 2024-04-25T00:09:26.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:25 smithi027 ceph-mon[29208]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T00:09:26.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:25 smithi027 ceph-mon[29208]: Detected new or changed devices on smithi139 2024-04-25T00:09:26.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:26.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:26.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:09:26.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:09:26.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:25 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi139 to 8544M 2024-04-25T00:09:26.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:26.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:26.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:09:26.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:26.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:09:26.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:26.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:09:26.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:25 smithi139 ceph-mon[32833]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T00:09:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:25 smithi139 ceph-mon[32833]: Detected new or changed devices on smithi139 2024-04-25T00:09:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:25 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:25 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:25 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:09:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:25 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:09:26.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:25 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi139 to 8544M 2024-04-25T00:09:26.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:25 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:26.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:25 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:26.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:25 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:09:26.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:25 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:26.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:25 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:09:26.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:25 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:26.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:25 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:09:26.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:25 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:26.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:26.309+0000 7f7c6516f700 1 -- 172.21.15.153:0/2311145997 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f7c60062670 con 0x7f7c4c0a2c20 2024-04-25T00:09:27.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:27 smithi139 ceph-mon[32833]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:27.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:27 smithi139 ceph-mon[32833]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:09:27.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:27 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:09:27.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:27 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:09:27.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:27 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:27.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:27 smithi153 ceph-mon[32896]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:27.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:27 smithi153 ceph-mon[32896]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:09:27.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:27 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:09:27.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:27 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:09:27.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:27 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:27.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:27 smithi027 ceph-mon[29208]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:27.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:27 smithi027 ceph-mon[29208]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:09:27.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:27 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:09:27.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:27 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:09:27.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:27 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:29.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:29 smithi153 ceph-mon[32896]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:29.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:29 smithi153 ceph-mon[32896]: from='client.? 172.21.15.153:0/2343769056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35ff5087-0d52-48f2-8417-a373b85a47e8"}]: dispatch 2024-04-25T00:09:29.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:29 smithi153 ceph-mon[32896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35ff5087-0d52-48f2-8417-a373b85a47e8"}]: dispatch 2024-04-25T00:09:29.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:29 smithi153 ceph-mon[32896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35ff5087-0d52-48f2-8417-a373b85a47e8"}]': finished 2024-04-25T00:09:29.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:29 smithi153 ceph-mon[32896]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T00:09:29.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:29 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:09:29.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:29 smithi027 ceph-mon[29208]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:29.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:29 smithi027 ceph-mon[29208]: from='client.? 172.21.15.153:0/2343769056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35ff5087-0d52-48f2-8417-a373b85a47e8"}]: dispatch 2024-04-25T00:09:29.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:29 smithi027 ceph-mon[29208]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35ff5087-0d52-48f2-8417-a373b85a47e8"}]: dispatch 2024-04-25T00:09:29.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:29 smithi027 ceph-mon[29208]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35ff5087-0d52-48f2-8417-a373b85a47e8"}]': finished 2024-04-25T00:09:29.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:29 smithi027 ceph-mon[29208]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T00:09:29.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:29 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:09:29.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:29 smithi139 ceph-mon[32833]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:29.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:29 smithi139 ceph-mon[32833]: from='client.? 172.21.15.153:0/2343769056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35ff5087-0d52-48f2-8417-a373b85a47e8"}]: dispatch 2024-04-25T00:09:29.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:29 smithi139 ceph-mon[32833]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35ff5087-0d52-48f2-8417-a373b85a47e8"}]: dispatch 2024-04-25T00:09:29.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:29 smithi139 ceph-mon[32833]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35ff5087-0d52-48f2-8417-a373b85a47e8"}]': finished 2024-04-25T00:09:29.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:29 smithi139 ceph-mon[32833]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T00:09:29.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:29 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:09:30.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:30 smithi153 ceph-mon[32896]: from='client.? 172.21.15.153:0/2899301101' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:09:30.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:30 smithi027 ceph-mon[29208]: from='client.? 172.21.15.153:0/2899301101' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:09:30.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:30 smithi139 ceph-mon[32833]: from='client.? 172.21.15.153:0/2899301101' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:09:31.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:31 smithi153 ceph-mon[32896]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:31.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:31 smithi027 ceph-mon[29208]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:31.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:31 smithi139 ceph-mon[32833]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:33.392 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:33 smithi153 ceph-mon[32896]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:33 smithi027 ceph-mon[29208]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:33.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:33 smithi139 ceph-mon[32833]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:35.334 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:35 smithi153 ceph-mon[32896]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:35.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:35 smithi027 ceph-mon[29208]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:35.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:35 smithi139 ceph-mon[32833]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:37.392 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:37 smithi153 ceph-mon[32896]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:37.392 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:37 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T00:09:37.392 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:37 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:37.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:37 smithi027 ceph-mon[29208]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:37.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:37 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T00:09:37.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:37 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:37.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:37 smithi139 ceph-mon[32833]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:37.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:37 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T00:09:37.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:37 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:38.392 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:38 smithi153 ceph-mon[32896]: Deploying daemon osd.5 on smithi153 2024-04-25T00:09:38.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:38 smithi027 ceph-mon[29208]: Deploying daemon osd.5 on smithi153 2024-04-25T00:09:38.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:38 smithi139 ceph-mon[32833]: Deploying daemon osd.5 on smithi153 2024-04-25T00:09:39.679 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:39 smithi153 ceph-mon[32896]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:39.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:39 smithi027 ceph-mon[29208]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:39.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:39 smithi139 ceph-mon[32833]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:41.453 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:41 smithi153 ceph-mon[32896]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:41.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:41 smithi027 ceph-mon[29208]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:41.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:41 smithi139 ceph-mon[32833]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:43.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:43 smithi153 ceph-mon[32896]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:43.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:43 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:43.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:43 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:43.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:43 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:09:43.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:43 smithi027 ceph-mon[29208]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:43.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:43 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:43.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:43 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:43.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:43 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:09:43.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:43 smithi139 ceph-mon[32833]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:43.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:43 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:43.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:43 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:43.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:43 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:09:45.182 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:45 smithi153 ceph-mon[32896]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:45.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:45 smithi027 ceph-mon[29208]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:45.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:45 smithi139 ceph-mon[32833]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:45.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:45.531+0000 7f7c5cd99700 1 -- 172.21.15.153:0/2311145997 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7c60062670 con 0x7f7c4c0a2c20 2024-04-25T00:09:45.533 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 5 on host 'smithi153' 2024-04-25T00:09:45.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:45.534+0000 7f7c6516f700 1 -- 172.21.15.153:0/2311145997 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7c4c0a2c20 msgr2=0x7f7c4c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:45.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:45.534+0000 7f7c6516f700 1 --2- 172.21.15.153:0/2311145997 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7c4c0a2c20 0x7f7c4c0a50e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7c50005950 tx=0x7f7c5000a300 comp rx=0 tx=0).stop 2024-04-25T00:09:45.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:45.534+0000 7f7c6516f700 1 -- 172.21.15.153:0/2311145997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c60107f80 msgr2=0x7f7c6019fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:45.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:45.534+0000 7f7c6516f700 1 --2- 172.21.15.153:0/2311145997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c60107f80 0x7f7c6019fa40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800d010 tx=0x7f7c58004dd0 comp rx=0 tx=0).stop 2024-04-25T00:09:45.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:45.534+0000 7f7c6516f700 1 -- 172.21.15.153:0/2311145997 shutdown_connections 2024-04-25T00:09:45.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:45.534+0000 7f7c6516f700 1 --2- 172.21.15.153:0/2311145997 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7c4c0a2c20 0x7f7c4c0a50e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:45.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:45.534+0000 7f7c6516f700 1 --2- 172.21.15.153:0/2311145997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c60107f80 0x7f7c6019fa40 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:45.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:45.534+0000 7f7c6516f700 1 --2- 172.21.15.153:0/2311145997 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c60102d20 0x7f7c60198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:45.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:45.534+0000 7f7c6516f700 1 --2- 172.21.15.153:0/2311145997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c60105650 0x7f7c601989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:45.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:45.534+0000 7f7c6516f700 1 -- 172.21.15.153:0/2311145997 >> 172.21.15.153:0/2311145997 conn(0x7f7c600fc730 msgr2=0x7f7c60103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:45.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:45.534+0000 7f7c6516f700 1 -- 172.21.15.153:0/2311145997 shutdown_connections 2024-04-25T00:09:45.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:09:45.534+0000 7f7c6516f700 1 -- 172.21.15.153:0/2311145997 wait complete. 2024-04-25T00:09:46.304 DEBUG:teuthology.orchestra.run.smithi153:osd.5> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.5.service 2024-04-25T00:09:46.307 INFO:tasks.cephadm:Deploying osd.6 on smithi027 with /dev/vg_nvme/lv_2... 2024-04-25T00:09:46.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T00:09:46.566 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:09:46.634 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:46 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:46.635 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:46 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:46.635 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:46 smithi153 ceph-mon[32896]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:46.635 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:46 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:46.635 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:46 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:46.635 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:46 smithi153 ceph-mon[32896]: from='osd.5 [v2:172.21.15.153:6808/3798123772,v1:172.21.15.153:6809/3798123772]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T00:09:46.635 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:46 smithi153 ceph-mon[32896]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T00:09:46.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:46 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:46.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:46 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:46.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:46 smithi027 ceph-mon[29208]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:46.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:46 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:46.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:46 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:46.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:46 smithi027 ceph-mon[29208]: from='osd.5 [v2:172.21.15.153:6808/3798123772,v1:172.21.15.153:6809/3798123772]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T00:09:46.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:46 smithi027 ceph-mon[29208]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T00:09:46.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:46 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:46.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:46 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:46.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:46 smithi139 ceph-mon[32833]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:46.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:46 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:46.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:46 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:46.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:46 smithi139 ceph-mon[32833]: from='osd.5 [v2:172.21.15.153:6808/3798123772,v1:172.21.15.153:6809/3798123772]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T00:09:46.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:46 smithi139 ceph-mon[32833]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T00:09:47.764 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:47 smithi027 ceph-mon[29208]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T00:09:47.765 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:47 smithi027 ceph-mon[29208]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T00:09:47.765 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:09:47.765 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:47 smithi027 ceph-mon[29208]: from='osd.5 [v2:172.21.15.153:6808/3798123772,v1:172.21.15.153:6809/3798123772]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:09:47.765 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:47 smithi027 ceph-mon[29208]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:09:47.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:47 smithi153 ceph-mon[32896]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T00:09:47.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:47 smithi153 ceph-mon[32896]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T00:09:47.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:47 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:09:47.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:47 smithi153 ceph-mon[32896]: from='osd.5 [v2:172.21.15.153:6808/3798123772,v1:172.21.15.153:6809/3798123772]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:09:47.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:47 smithi153 ceph-mon[32896]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:09:48.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:47 smithi139 ceph-mon[32833]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T00:09:48.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:47 smithi139 ceph-mon[32833]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T00:09:48.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:47 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:09:48.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:47 smithi139 ceph-mon[32833]: from='osd.5 [v2:172.21.15.153:6808/3798123772,v1:172.21.15.153:6809/3798123772]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:09:48.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:47 smithi139 ceph-mon[32833]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:09:48.277 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 00:09:48 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-5[44710]: 2024-04-25T00:09:48.029+0000 7fca34173700 -1 osd.5 0 waiting for initial osdmap 2024-04-25T00:09:48.277 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 00:09:48 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-5[44710]: 2024-04-25T00:09:48.036+0000 7fca2bf5e700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:09:48.692 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T00:09:48.692 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T00:09:48.693 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-25T00:09:48.693 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-25T00:09:48.693 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011446 s, 916 MB/s 2024-04-25T00:09:48.693 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-25T00:09:48.718 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:48 smithi027 ceph-mon[29208]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:48.719 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:48 smithi027 ceph-mon[29208]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T00:09:48.719 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:48 smithi027 ceph-mon[29208]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T00:09:48.719 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:48 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:09:48.719 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:48 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:09:48.719 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:48 smithi027 ceph-mon[29208]: from='osd.5 ' entity='osd.5' 2024-04-25T00:09:48.967 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:48 smithi153 ceph-mon[32896]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:48.967 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:48 smithi153 ceph-mon[32896]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T00:09:48.968 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:48 smithi153 ceph-mon[32896]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T00:09:48.968 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:48 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:09:48.968 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:48 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:09:48.968 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:48 smithi153 ceph-mon[32896]: from='osd.5 ' entity='osd.5' 2024-04-25T00:09:49.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:48 smithi139 ceph-mon[32833]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T00:09:49.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:48 smithi139 ceph-mon[32833]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T00:09:49.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:48 smithi139 ceph-mon[32833]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T00:09:49.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:48 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:09:49.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:48 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:09:49.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:48 smithi139 ceph-mon[32833]: from='osd.5 ' entity='osd.5' 2024-04-25T00:09:49.418 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_2 2024-04-25T00:09:49.673 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:09:49.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: purged_snaps scrub starts 2024-04-25T00:09:49.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: purged_snaps scrub ok 2024-04-25T00:09:49.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: osd.5 [v2:172.21.15.153:6808/3798123772,v1:172.21.15.153:6809/3798123772] boot 2024-04-25T00:09:49.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T00:09:49.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:09:49.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: Detected new or changed devices on smithi153 2024-04-25T00:09:49.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:49.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:49.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:09:49.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:09:49.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi153 to 8544M 2024-04-25T00:09:49.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:49.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:49.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:09:49.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:49.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:09:49.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:49.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:09:49.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:49 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:50.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: purged_snaps scrub starts 2024-04-25T00:09:50.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: purged_snaps scrub ok 2024-04-25T00:09:50.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: osd.5 [v2:172.21.15.153:6808/3798123772,v1:172.21.15.153:6809/3798123772] boot 2024-04-25T00:09:50.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T00:09:50.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:09:50.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: Detected new or changed devices on smithi153 2024-04-25T00:09:50.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:50.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:50.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:09:50.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:09:50.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi153 to 8544M 2024-04-25T00:09:50.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:50.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:50.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:09:50.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:50.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:09:50.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:50.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:09:50.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:49 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:50.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: purged_snaps scrub starts 2024-04-25T00:09:50.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: purged_snaps scrub ok 2024-04-25T00:09:50.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: osd.5 [v2:172.21.15.153:6808/3798123772,v1:172.21.15.153:6809/3798123772] boot 2024-04-25T00:09:50.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T00:09:50.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:09:50.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: Detected new or changed devices on smithi153 2024-04-25T00:09:50.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:50.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:50.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:09:50.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:09:50.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi153 to 8544M 2024-04-25T00:09:50.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:50.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:50.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:09:50.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:50.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:09:50.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:50.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:09:50.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:49 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:09:50.788 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:50 smithi027 ceph-mon[29208]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 205 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:09:50.789 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:50 smithi027 ceph-mon[29208]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T00:09:50.904 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:50 smithi139 ceph-mon[32833]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 205 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:09:50.904 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:50 smithi139 ceph-mon[32833]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T00:09:51.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:50 smithi153 ceph-mon[32896]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 205 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:09:51.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:50 smithi153 ceph-mon[32896]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T00:09:51.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.387+0000 7f6fa2691700 1 -- 172.21.15.27:0/3023691962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f9c1014d0 msgr2=0x7f6f9c101950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:51.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.387+0000 7f6fa2691700 1 --2- 172.21.15.27:0/3023691962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f9c1014d0 0x7f6f9c101950 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009990 tx=0x7f6f84009ca0 comp rx=0 tx=0).stop 2024-04-25T00:09:51.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.387+0000 7f6fa2691700 1 -- 172.21.15.27:0/3023691962 shutdown_connections 2024-04-25T00:09:51.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.387+0000 7f6fa2691700 1 --2- 172.21.15.27:0/3023691962 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f9c101e90 0x7f6f9c102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:51.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.387+0000 7f6fa2691700 1 --2- 172.21.15.27:0/3023691962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f9c100b70 0x7f6f9c100f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:51.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.387+0000 7f6fa2691700 1 --2- 172.21.15.27:0/3023691962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f9c1014d0 0x7f6f9c101950 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.387+0000 7f6fa2691700 1 -- 172.21.15.27:0/3023691962 >> 172.21.15.27:0/3023691962 conn(0x7f6f9c0fc730 msgr2=0x7f6f9c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.387+0000 7f6fa2691700 1 -- 172.21.15.27:0/3023691962 shutdown_connections 2024-04-25T00:09:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.387+0000 7f6fa2691700 1 -- 172.21.15.27:0/3023691962 wait complete. 2024-04-25T00:09:51.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.388+0000 7f6fa2691700 1 Processor -- start 2024-04-25T00:09:51.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.388+0000 7f6fa2691700 1 -- start start 2024-04-25T00:09:51.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.388+0000 7f6fa2691700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f9c100b70 0x7f6f9c19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:51.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.388+0000 7f6fa2691700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f9c1014d0 0x7f6f9c19ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:51.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.388+0000 7f6fa2691700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f9c101e90 0x7f6f9c1a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:51.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.388+0000 7f6fa2691700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c110060 con 0x7f6f9c100b70 2024-04-25T00:09:51.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.388+0000 7f6fa2691700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c10fee0 con 0x7f6f9c101e90 2024-04-25T00:09:51.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.388+0000 7f6fa2691700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c1101e0 con 0x7f6f9c1014d0 2024-04-25T00:09:51.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.388+0000 7f6f9bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f9c100b70 0x7f6f9c19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.388+0000 7f6f9bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f9c100b70 0x7f6f9c19a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56312/0 (socket says 172.21.15.27:56312) 2024-04-25T00:09:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.388+0000 7f6f9bfff700 1 -- 172.21.15.27:0/1785551768 learned_addr learned my addr 172.21.15.27:0/1785551768 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:09:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.388+0000 7f6f9b7fe700 1 --2- 172.21.15.27:0/1785551768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f9c1014d0 0x7f6f9c19ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.388+0000 7f6fa0c2e700 1 --2- 172.21.15.27:0/1785551768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f9c101e90 0x7f6f9c1a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.389+0000 7f6f9bfff700 1 -- 172.21.15.27:0/1785551768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f9c1014d0 msgr2=0x7f6f9c19ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.389+0000 7f6f9bfff700 1 --2- 172.21.15.27:0/1785551768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f9c1014d0 0x7f6f9c19ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.389+0000 7f6f9bfff700 1 -- 172.21.15.27:0/1785551768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f9c101e90 msgr2=0x7f6f9c1a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.389+0000 7f6f9bfff700 1 --2- 172.21.15.27:0/1785551768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f9c101e90 0x7f6f9c1a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.389+0000 7f6f9bfff700 1 -- 172.21.15.27:0/1785551768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f9c1a2400 con 0x7f6f9c100b70 2024-04-25T00:09:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.389+0000 7f6f9bfff700 1 --2- 172.21.15.27:0/1785551768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f9c100b70 0x7f6f9c19a7a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c00d9b0 tx=0x7f6f8c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.389+0000 7f6f997fa700 1 -- 172.21.15.27:0/1785551768 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c017070 con 0x7f6f9c100b70 2024-04-25T00:09:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.389+0000 7f6fa2691700 1 -- 172.21.15.27:0/1785551768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f9c1a26f0 con 0x7f6f9c100b70 2024-04-25T00:09:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.389+0000 7f6f997fa700 1 -- 172.21.15.27:0/1785551768 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f8c005020 con 0x7f6f9c100b70 2024-04-25T00:09:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.389+0000 7f6fa2691700 1 -- 172.21.15.27:0/1785551768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f9c1a2be0 con 0x7f6f9c100b70 2024-04-25T00:09:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.389+0000 7f6f997fa700 1 -- 172.21.15.27:0/1785551768 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c009bd0 con 0x7f6f9c100b70 2024-04-25T00:09:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.390+0000 7f6fa2691700 1 -- 172.21.15.27:0/1785551768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f9c068f20 con 0x7f6f9c100b70 2024-04-25T00:09:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.391+0000 7f6f997fa700 1 -- 172.21.15.27:0/1785551768 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6f8c0051d0 con 0x7f6f9c100b70 2024-04-25T00:09:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.392+0000 7f6f997fa700 1 --2- 172.21.15.27:0/1785551768 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6f880a2c20 0x7f6f880a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.392+0000 7f6f9b7fe700 1 --2- 172.21.15.27:0/1785551768 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6f880a2c20 0x7f6f880a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.392+0000 7f6f997fa700 1 -- 172.21.15.27:0/1785551768 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f6f8c0ce0d0 con 0x7f6f9c100b70 2024-04-25T00:09:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.392+0000 7f6f9b7fe700 1 --2- 172.21.15.27:0/1785551768 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6f880a2c20 0x7f6f880a50e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6f8400b5c0 tx=0x7f6f84003960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:51.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.397+0000 7f6f997fa700 1 -- 172.21.15.27:0/1785551768 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6f8c020020 con 0x7f6f9c100b70 2024-04-25T00:09:51.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:09:51.613+0000 7f6fa2691700 1 -- 172.21.15.27:0/1785551768 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f6f9c062670 con 0x7f6f880a2c20 2024-04-25T00:09:51.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:51 smithi027 ceph-mon[29208]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T00:09:51.922 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:51 smithi139 ceph-mon[32833]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T00:09:51.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:51 smithi153 ceph-mon[32896]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T00:09:52.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:52 smithi027 ceph-mon[29208]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 205 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:09:52.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:52 smithi027 ceph-mon[29208]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T00:09:52.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:52 smithi027 ceph-mon[29208]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:09:52.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:52 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:09:52.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:52 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:09:52.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:52 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:53.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:52 smithi139 ceph-mon[32833]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 205 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:09:53.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:52 smithi139 ceph-mon[32833]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T00:09:53.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:52 smithi139 ceph-mon[32833]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:09:53.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:52 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:09:53.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:52 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:09:53.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:52 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:53.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:52 smithi153 ceph-mon[32896]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 205 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:09:53.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:52 smithi153 ceph-mon[32896]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T00:09:53.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:52 smithi153 ceph-mon[32896]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:09:53.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:52 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:09:53.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:52 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:09:53.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:52 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:09:53.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:53 smithi027 ceph-mon[29208]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T00:09:54.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:53 smithi139 ceph-mon[32833]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T00:09:54.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:53 smithi153 ceph-mon[32896]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T00:09:54.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:54 smithi027 ceph-mon[29208]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 205 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:09:54.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:54 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3030656329' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "923eafd7-9819-4ef8-9008-0c5ff2a611c2"}]: dispatch 2024-04-25T00:09:54.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:54 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3030656329' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "923eafd7-9819-4ef8-9008-0c5ff2a611c2"}]': finished 2024-04-25T00:09:54.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:54 smithi027 ceph-mon[29208]: osdmap e43: 7 total, 6 up, 7 in 2024-04-25T00:09:54.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:54 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:09:55.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:54 smithi139 ceph-mon[32833]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 205 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:09:55.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:54 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3030656329' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "923eafd7-9819-4ef8-9008-0c5ff2a611c2"}]: dispatch 2024-04-25T00:09:55.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:54 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3030656329' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "923eafd7-9819-4ef8-9008-0c5ff2a611c2"}]': finished 2024-04-25T00:09:55.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:54 smithi139 ceph-mon[32833]: osdmap e43: 7 total, 6 up, 7 in 2024-04-25T00:09:55.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:54 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:09:55.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:54 smithi153 ceph-mon[32896]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 205 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:09:55.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:54 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3030656329' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "923eafd7-9819-4ef8-9008-0c5ff2a611c2"}]: dispatch 2024-04-25T00:09:55.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:54 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3030656329' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "923eafd7-9819-4ef8-9008-0c5ff2a611c2"}]': finished 2024-04-25T00:09:55.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:54 smithi153 ceph-mon[32896]: osdmap e43: 7 total, 6 up, 7 in 2024-04-25T00:09:55.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:54 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:09:55.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:55 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2633919676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:09:56.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:55 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2633919676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:09:56.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:55 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2633919676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:09:56.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:56 smithi027 ceph-mon[29208]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-25T00:09:57.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:56 smithi139 ceph-mon[32833]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-25T00:09:57.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:56 smithi153 ceph-mon[32896]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-25T00:09:58.659 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:09:58 smithi027 ceph-mon[29208]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T00:09:59.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:09:58 smithi139 ceph-mon[32833]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T00:09:59.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:09:58 smithi153 ceph-mon[32896]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T00:10:00.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:00 smithi027 ceph-mon[29208]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T00:10:00.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:00 smithi027 ceph-mon[29208]: overall HEALTH_OK 2024-04-25T00:10:01.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:00 smithi139 ceph-mon[32833]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T00:10:01.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:00 smithi139 ceph-mon[32833]: overall HEALTH_OK 2024-04-25T00:10:01.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:00 smithi153 ceph-mon[32896]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T00:10:01.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:00 smithi153 ceph-mon[32896]: overall HEALTH_OK 2024-04-25T00:10:02.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:02 smithi027 ceph-mon[29208]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-25T00:10:02.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:02 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T00:10:02.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:02 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:02.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:02 smithi027 ceph-mon[29208]: Deploying daemon osd.6 on smithi027 2024-04-25T00:10:03.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:02 smithi139 ceph-mon[32833]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-25T00:10:03.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:02 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T00:10:03.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:02 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:03.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:02 smithi139 ceph-mon[32833]: Deploying daemon osd.6 on smithi027 2024-04-25T00:10:03.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:02 smithi153 ceph-mon[32896]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-25T00:10:03.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:02 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T00:10:03.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:02 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:03.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:02 smithi153 ceph-mon[32896]: Deploying daemon osd.6 on smithi027 2024-04-25T00:10:04.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:04 smithi027 ceph-mon[29208]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T00:10:05.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:04 smithi139 ceph-mon[32833]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T00:10:05.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:04 smithi153 ceph-mon[32896]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T00:10:06.818 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:06 smithi027 ceph-mon[29208]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-25T00:10:07.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:06 smithi139 ceph-mon[32833]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-25T00:10:07.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:06 smithi153 ceph-mon[32896]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-25T00:10:09.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:08 smithi027 ceph-mon[29208]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:10:09.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:09.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:09.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:10:09.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:08 smithi139 ceph-mon[32833]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:10:09.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:09.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:09.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:10:09.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:08 smithi153 ceph-mon[32896]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:10:09.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:09.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:09.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:10:10.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:10 smithi027 ceph-mon[29208]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:10:11.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:10 smithi139 ceph-mon[32833]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:10:11.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:10 smithi153 ceph-mon[32896]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:10:11.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:10:11.722+0000 7f6f997fa700 1 -- 172.21.15.27:0/1785551768 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6f9c062670 con 0x7f6f880a2c20 2024-04-25T00:10:11.723 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 6 on host 'smithi027' 2024-04-25T00:10:11.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:10:11.724+0000 7f6fa2691700 1 -- 172.21.15.27:0/1785551768 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6f880a2c20 msgr2=0x7f6f880a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:11.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:10:11.725+0000 7f6fa2691700 1 --2- 172.21.15.27:0/1785551768 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6f880a2c20 0x7f6f880a50e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6f8400b5c0 tx=0x7f6f84003960 comp rx=0 tx=0).stop 2024-04-25T00:10:11.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:10:11.725+0000 7f6fa2691700 1 -- 172.21.15.27:0/1785551768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f9c100b70 msgr2=0x7f6f9c19a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:11.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:10:11.725+0000 7f6fa2691700 1 --2- 172.21.15.27:0/1785551768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f9c100b70 0x7f6f9c19a7a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c00d9b0 tx=0x7f6f8c00dcc0 comp rx=0 tx=0).stop 2024-04-25T00:10:11.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:10:11.725+0000 7f6fa2691700 1 -- 172.21.15.27:0/1785551768 shutdown_connections 2024-04-25T00:10:11.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:10:11.725+0000 7f6fa2691700 1 --2- 172.21.15.27:0/1785551768 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6f880a2c20 0x7f6f880a50e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:11.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:10:11.725+0000 7f6fa2691700 1 --2- 172.21.15.27:0/1785551768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f9c101e90 0x7f6f9c1a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:11.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:10:11.725+0000 7f6fa2691700 1 --2- 172.21.15.27:0/1785551768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f9c100b70 0x7f6f9c19a7a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:11.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:10:11.725+0000 7f6fa2691700 1 --2- 172.21.15.27:0/1785551768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f9c1014d0 0x7f6f9c19ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:11.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:10:11.725+0000 7f6fa2691700 1 -- 172.21.15.27:0/1785551768 >> 172.21.15.27:0/1785551768 conn(0x7f6f9c0fc730 msgr2=0x7f6f9c0fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:11.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:10:11.725+0000 7f6fa2691700 1 -- 172.21.15.27:0/1785551768 shutdown_connections 2024-04-25T00:10:11.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:10:11.725+0000 7f6fa2691700 1 -- 172.21.15.27:0/1785551768 wait complete. 2024-04-25T00:10:12.348 DEBUG:teuthology.orchestra.run.smithi027:osd.6> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.6.service 2024-04-25T00:10:12.351 INFO:tasks.cephadm:Deploying osd.7 on smithi139 with /dev/vg_nvme/lv_2... 2024-04-25T00:10:12.351 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T00:10:12.597 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T00:10:12.934 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:12 smithi139 ceph-mon[32833]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:10:12.934 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:12 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:12.934 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:12 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:12.934 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:12 smithi139 ceph-mon[32833]: from='osd.6 [v2:172.21.15.27:6818/769569119,v1:172.21.15.27:6819/769569119]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T00:10:12.934 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:12 smithi139 ceph-mon[32833]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T00:10:12.935 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:12 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:12.935 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:12 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:12.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:12 smithi027 ceph-mon[29208]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:10:12.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:12 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:12.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:12 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:12.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:12 smithi027 ceph-mon[29208]: from='osd.6 [v2:172.21.15.27:6818/769569119,v1:172.21.15.27:6819/769569119]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T00:10:12.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:12 smithi027 ceph-mon[29208]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T00:10:12.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:12 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:12.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:12 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:13.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:12 smithi153 ceph-mon[32896]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:10:13.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:12 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:13.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:12 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:13.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:12 smithi153 ceph-mon[32896]: from='osd.6 [v2:172.21.15.27:6818/769569119,v1:172.21.15.27:6819/769569119]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T00:10:13.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:12 smithi153 ceph-mon[32896]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T00:10:13.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:12 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:13.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:12 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:13.709 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:13 smithi027 ceph-mon[29208]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T00:10:13.709 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:13 smithi027 ceph-mon[29208]: osdmap e44: 7 total, 6 up, 7 in 2024-04-25T00:10:13.709 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:13 smithi027 ceph-mon[29208]: from='osd.6 [v2:172.21.15.27:6818/769569119,v1:172.21.15.27:6819/769569119]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T00:10:13.709 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:13 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:10:13.709 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:13 smithi027 ceph-mon[29208]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T00:10:13.710 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 00:10:13 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-6[54396]: 2024-04-25T00:10:13.699+0000 7fa1053ad700 -1 osd.6 0 waiting for initial osdmap 2024-04-25T00:10:13.710 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 00:10:13 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-6[54396]: 2024-04-25T00:10:13.708+0000 7fa1011a0700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:10:14.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:13 smithi139 ceph-mon[32833]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T00:10:14.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:13 smithi139 ceph-mon[32833]: osdmap e44: 7 total, 6 up, 7 in 2024-04-25T00:10:14.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:13 smithi139 ceph-mon[32833]: from='osd.6 [v2:172.21.15.27:6818/769569119,v1:172.21.15.27:6819/769569119]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T00:10:14.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:13 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:10:14.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:13 smithi139 ceph-mon[32833]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T00:10:14.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:13 smithi153 ceph-mon[32896]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T00:10:14.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:13 smithi153 ceph-mon[32896]: osdmap e44: 7 total, 6 up, 7 in 2024-04-25T00:10:14.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:13 smithi153 ceph-mon[32896]: from='osd.6 [v2:172.21.15.27:6818/769569119,v1:172.21.15.27:6819/769569119]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T00:10:14.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:13 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:10:14.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:13 smithi153 ceph-mon[32896]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T00:10:14.925 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T00:10:14.925 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T00:10:14.925 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2024-04-25T00:10:14.925 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2024-04-25T00:10:14.926 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129879 s, 807 MB/s 2024-04-25T00:10:14.926 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2024-04-25T00:10:14.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:14 smithi027 ceph-mon[29208]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:10:14.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:14 smithi027 ceph-mon[29208]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T00:10:14.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:14 smithi027 ceph-mon[29208]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T00:10:14.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:14 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:10:14.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:14 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:10:15.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:14 smithi139 ceph-mon[32833]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:10:15.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:14 smithi139 ceph-mon[32833]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T00:10:15.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:14 smithi139 ceph-mon[32833]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T00:10:15.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:14 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:10:15.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:14 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:10:15.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:14 smithi153 ceph-mon[32896]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T00:10:15.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:14 smithi153 ceph-mon[32896]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T00:10:15.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:14 smithi153 ceph-mon[32896]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T00:10:15.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:14 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:10:15.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:14 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:10:15.562 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi139:vg_nvme/lv_2 2024-04-25T00:10:15.807 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T00:10:15.845 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:15 smithi139 ceph-mon[32833]: purged_snaps scrub starts 2024-04-25T00:10:15.845 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:15 smithi139 ceph-mon[32833]: purged_snaps scrub ok 2024-04-25T00:10:15.845 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:15 smithi139 ceph-mon[32833]: osd.6 [v2:172.21.15.27:6818/769569119,v1:172.21.15.27:6819/769569119] boot 2024-04-25T00:10:15.845 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:15 smithi139 ceph-mon[32833]: osdmap e46: 7 total, 7 up, 7 in 2024-04-25T00:10:15.845 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:15 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:10:15.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:15 smithi027 ceph-mon[29208]: purged_snaps scrub starts 2024-04-25T00:10:15.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:15 smithi027 ceph-mon[29208]: purged_snaps scrub ok 2024-04-25T00:10:15.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:15 smithi027 ceph-mon[29208]: osd.6 [v2:172.21.15.27:6818/769569119,v1:172.21.15.27:6819/769569119] boot 2024-04-25T00:10:15.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:15 smithi027 ceph-mon[29208]: osdmap e46: 7 total, 7 up, 7 in 2024-04-25T00:10:15.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:15 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:10:16.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:15 smithi153 ceph-mon[32896]: purged_snaps scrub starts 2024-04-25T00:10:16.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:15 smithi153 ceph-mon[32896]: purged_snaps scrub ok 2024-04-25T00:10:16.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:15 smithi153 ceph-mon[32896]: osd.6 [v2:172.21.15.27:6818/769569119,v1:172.21.15.27:6819/769569119] boot 2024-04-25T00:10:16.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:15 smithi153 ceph-mon[32896]: osdmap e46: 7 total, 7 up, 7 in 2024-04-25T00:10:16.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:15 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:10:16.802 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:16 smithi139 ceph-mon[32833]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:16.802 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:16 smithi139 ceph-mon[32833]: osdmap e47: 7 total, 7 up, 7 in 2024-04-25T00:10:16.802 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:16.803 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:16.803 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:16.803 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:16.803 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:16.803 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:16.803 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:16.803 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:10:16.803 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:16.803 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:10:16.803 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:16.803 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:10:16.803 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:17.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:16 smithi153 ceph-mon[32896]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:17.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:16 smithi153 ceph-mon[32896]: osdmap e47: 7 total, 7 up, 7 in 2024-04-25T00:10:17.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:17.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:17.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:17.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:17.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:17.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:17.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:17.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:10:17.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:17.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:10:17.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:17.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:10:17.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:17.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:16 smithi027 ceph-mon[29208]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:17.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:16 smithi027 ceph-mon[29208]: osdmap e47: 7 total, 7 up, 7 in 2024-04-25T00:10:17.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:17.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:17.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:17.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:17.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:17.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:17.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:17.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:10:17.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:17.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:10:17.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:17.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:10:17.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:17.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.387+0000 7fb4e4ebd700 1 -- 172.21.15.139:0/147993172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4e0103e20 msgr2=0x7fb4e010a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:17.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.388+0000 7fb4e4ebd700 1 --2- 172.21.15.139:0/147993172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4e0103e20 0x7fb4e010a6d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb4d800b0d0 tx=0x7fb4d800b3e0 comp rx=0 tx=0).stop 2024-04-25T00:10:17.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.388+0000 7fb4e4ebd700 1 -- 172.21.15.139:0/147993172 shutdown_connections 2024-04-25T00:10:17.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.388+0000 7fb4e4ebd700 1 --2- 172.21.15.139:0/147993172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4e0103e20 0x7fb4e010a6d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:17.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.388+0000 7fb4e4ebd700 1 --2- 172.21.15.139:0/147993172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4e0102280 0x7fb4e01026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:17.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.388+0000 7fb4e4ebd700 1 --2- 172.21.15.139:0/147993172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4e0103480 0x7fb4e01038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:17.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.388+0000 7fb4e4ebd700 1 -- 172.21.15.139:0/147993172 >> 172.21.15.139:0/147993172 conn(0x7fb4e00fd820 msgr2=0x7fb4e00ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:17.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.388+0000 7fb4e4ebd700 1 -- 172.21.15.139:0/147993172 shutdown_connections 2024-04-25T00:10:17.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.388+0000 7fb4e4ebd700 1 -- 172.21.15.139:0/147993172 wait complete. 2024-04-25T00:10:17.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.388+0000 7fb4e4ebd700 1 Processor -- start 2024-04-25T00:10:17.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4e4ebd700 1 -- start start 2024-04-25T00:10:17.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4e4ebd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4e0102280 0x7fb4e0198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:17.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4e4ebd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4e0103480 0x7fb4e01988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:17.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4e4ebd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4e0103e20 0x7fb4e019f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:17.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4e4ebd700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e010de40 con 0x7fb4e0103e20 2024-04-25T00:10:17.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4e4ebd700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e010dcc0 con 0x7fb4e0102280 2024-04-25T00:10:17.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4e4ebd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e010dfc0 con 0x7fb4e0103480 2024-04-25T00:10:17.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4de59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4e0102280 0x7fb4e0198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:17.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4de59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4e0102280 0x7fb4e0198360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:44678/0 (socket says 172.21.15.139:44678) 2024-04-25T00:10:17.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4de59c700 1 -- 172.21.15.139:0/3556489867 learned_addr learned my addr 172.21.15.139:0/3556489867 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-04-25T00:10:17.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4ded9d700 1 --2- 172.21.15.139:0/3556489867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4e0103e20 0x7fb4e019f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:17.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4de59c700 1 -- 172.21.15.139:0/3556489867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4e0103480 msgr2=0x7fb4e01988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:17.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4de59c700 1 --2- 172.21.15.139:0/3556489867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4e0103480 0x7fb4e01988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:17.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4de59c700 1 -- 172.21.15.139:0/3556489867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4e0103e20 msgr2=0x7fb4e019f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:17.393 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4de59c700 1 --2- 172.21.15.139:0/3556489867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4e0103e20 0x7fb4e019f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:17.393 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4de59c700 1 -- 172.21.15.139:0/3556489867 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4e019ffc0 con 0x7fb4e0102280 2024-04-25T00:10:17.393 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4de59c700 1 --2- 172.21.15.139:0/3556489867 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4e0102280 0x7fb4e0198360 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb4d000c990 tx=0x7fb4d000cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:17.393 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4cf7fe700 1 -- 172.21.15.139:0/3556489867 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0007b00 con 0x7fb4e0102280 2024-04-25T00:10:17.393 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4cf7fe700 1 -- 172.21.15.139:0/3556489867 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4d0007ca0 con 0x7fb4e0102280 2024-04-25T00:10:17.393 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.389+0000 7fb4cf7fe700 1 -- 172.21.15.139:0/3556489867 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d00186f0 con 0x7fb4e0102280 2024-04-25T00:10:17.393 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.390+0000 7fb4e4ebd700 1 -- 172.21.15.139:0/3556489867 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4e01a02b0 con 0x7fb4e0102280 2024-04-25T00:10:17.393 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.390+0000 7fb4e4ebd700 1 -- 172.21.15.139:0/3556489867 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4e01a07f0 con 0x7fb4e0102280 2024-04-25T00:10:17.394 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.391+0000 7fb4e4ebd700 1 -- 172.21.15.139:0/3556489867 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4e0068f20 con 0x7fb4e0102280 2024-04-25T00:10:17.394 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.392+0000 7fb4cf7fe700 1 -- 172.21.15.139:0/3556489867 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb4d0018890 con 0x7fb4e0102280 2024-04-25T00:10:17.394 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.392+0000 7fb4cf7fe700 1 --2- 172.21.15.139:0/3556489867 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb4c80a2c20 0x7fb4c80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:17.394 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.392+0000 7fb4cf7fe700 1 -- 172.21.15.139:0/3556489867 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fb4d00ce290 con 0x7fb4e0102280 2024-04-25T00:10:17.394 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.392+0000 7fb4ddd9b700 1 --2- 172.21.15.139:0/3556489867 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb4c80a2c20 0x7fb4c80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:17.394 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.393+0000 7fb4ddd9b700 1 --2- 172.21.15.139:0/3556489867 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb4c80a2c20 0x7fb4c80a50e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4005950 tx=0x7fb4d40058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:17.397 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.397+0000 7fb4cf7fe700 1 -- 172.21.15.139:0/3556489867 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb4d0014030 con 0x7fb4e0102280 2024-04-25T00:10:17.601 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:17.601+0000 7fb4e4ebd700 1 -- 172.21.15.139:0/3556489867 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fb4e0062670 con 0x7fb4c80a2c20 2024-04-25T00:10:18.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:17 smithi139 ceph-mon[32833]: Detected new or changed devices on smithi027 2024-04-25T00:10:18.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:17 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi027 to 5696M 2024-04-25T00:10:18.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:17 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:10:18.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:17 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:10:18.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:17 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:18.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:17 smithi153 ceph-mon[32896]: Detected new or changed devices on smithi027 2024-04-25T00:10:18.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:17 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi027 to 5696M 2024-04-25T00:10:18.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:17 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:10:18.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:17 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:10:18.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:17 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:18.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:17 smithi027 ceph-mon[29208]: Detected new or changed devices on smithi027 2024-04-25T00:10:18.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:17 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi027 to 5696M 2024-04-25T00:10:18.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:17 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:10:18.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:17 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:10:18.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:17 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:19.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:18 smithi139 ceph-mon[32833]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:19.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:18 smithi139 ceph-mon[32833]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:10:19.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:18 smithi153 ceph-mon[32896]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:19.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:18 smithi153 ceph-mon[32896]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:10:19.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:18 smithi027 ceph-mon[29208]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:19.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:18 smithi027 ceph-mon[29208]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:10:21.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:20 smithi139 ceph-mon[32833]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:21.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:20 smithi139 ceph-mon[32833]: from='client.? 172.21.15.139:0/3184103827' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a71b99a8-c31e-4ac9-91fc-61735cd196b2"}]: dispatch 2024-04-25T00:10:21.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:20 smithi139 ceph-mon[32833]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a71b99a8-c31e-4ac9-91fc-61735cd196b2"}]: dispatch 2024-04-25T00:10:21.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:20 smithi139 ceph-mon[32833]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a71b99a8-c31e-4ac9-91fc-61735cd196b2"}]': finished 2024-04-25T00:10:21.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:20 smithi139 ceph-mon[32833]: osdmap e48: 8 total, 7 up, 8 in 2024-04-25T00:10:21.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:20 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:10:21.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:20 smithi153 ceph-mon[32896]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:21.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:20 smithi153 ceph-mon[32896]: from='client.? 172.21.15.139:0/3184103827' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a71b99a8-c31e-4ac9-91fc-61735cd196b2"}]: dispatch 2024-04-25T00:10:21.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:20 smithi153 ceph-mon[32896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a71b99a8-c31e-4ac9-91fc-61735cd196b2"}]: dispatch 2024-04-25T00:10:21.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:20 smithi153 ceph-mon[32896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a71b99a8-c31e-4ac9-91fc-61735cd196b2"}]': finished 2024-04-25T00:10:21.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:20 smithi153 ceph-mon[32896]: osdmap e48: 8 total, 7 up, 8 in 2024-04-25T00:10:21.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:20 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:10:21.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:20 smithi027 ceph-mon[29208]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:21.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:20 smithi027 ceph-mon[29208]: from='client.? 172.21.15.139:0/3184103827' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a71b99a8-c31e-4ac9-91fc-61735cd196b2"}]: dispatch 2024-04-25T00:10:21.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:20 smithi027 ceph-mon[29208]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a71b99a8-c31e-4ac9-91fc-61735cd196b2"}]: dispatch 2024-04-25T00:10:21.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:20 smithi027 ceph-mon[29208]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a71b99a8-c31e-4ac9-91fc-61735cd196b2"}]': finished 2024-04-25T00:10:21.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:20 smithi027 ceph-mon[29208]: osdmap e48: 8 total, 7 up, 8 in 2024-04-25T00:10:21.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:20 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:10:22.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:21 smithi139 ceph-mon[32833]: from='client.? 172.21.15.139:0/61794804' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:10:22.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:21 smithi153 ceph-mon[32896]: from='client.? 172.21.15.139:0/61794804' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:10:22.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:21 smithi027 ceph-mon[29208]: from='client.? 172.21.15.139:0/61794804' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:10:23.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:22 smithi139 ceph-mon[32833]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:23.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:22 smithi153 ceph-mon[32896]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:23.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:22 smithi027 ceph-mon[29208]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:25.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:24 smithi139 ceph-mon[32833]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:25.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:24 smithi153 ceph-mon[32896]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:25.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:24 smithi027 ceph-mon[29208]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:27.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:26 smithi139 ceph-mon[32833]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:27.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:26 smithi153 ceph-mon[32896]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:27.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:26 smithi027 ceph-mon[29208]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:28.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:27 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T00:10:28.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:27 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:28.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:27 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T00:10:28.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:27 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:28.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:27 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T00:10:28.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:27 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:29.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:28 smithi139 ceph-mon[32833]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:29.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:28 smithi139 ceph-mon[32833]: Deploying daemon osd.7 on smithi139 2024-04-25T00:10:29.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:28 smithi153 ceph-mon[32896]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:29.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:28 smithi153 ceph-mon[32896]: Deploying daemon osd.7 on smithi139 2024-04-25T00:10:29.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:28 smithi027 ceph-mon[29208]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:29.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:28 smithi027 ceph-mon[29208]: Deploying daemon osd.7 on smithi139 2024-04-25T00:10:31.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:30 smithi139 ceph-mon[32833]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:31.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:30 smithi153 ceph-mon[32896]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:31.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:30 smithi027 ceph-mon[29208]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:32.949 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:32 smithi139 ceph-mon[32833]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:33.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:32 smithi153 ceph-mon[32896]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:33.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:32 smithi027 ceph-mon[29208]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:10:34.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:34.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:34.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:10:34.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:34.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:34.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:10:35.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:35 smithi139 ceph-mon[32833]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:35.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:35 smithi027 ceph-mon[29208]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:35.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:35 smithi153 ceph-mon[32896]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:37.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:37.116+0000 7fb4cf7fe700 1 -- 172.21.15.139:0/3556489867 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb4e0062670 con 0x7fb4c80a2c20 2024-04-25T00:10:37.118 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 7 on host 'smithi139' 2024-04-25T00:10:37.120 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:37.119+0000 7fb4e4ebd700 1 -- 172.21.15.139:0/3556489867 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb4c80a2c20 msgr2=0x7fb4c80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:37.120 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:37.119+0000 7fb4e4ebd700 1 --2- 172.21.15.139:0/3556489867 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb4c80a2c20 0x7fb4c80a50e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4005950 tx=0x7fb4d40058e0 comp rx=0 tx=0).stop 2024-04-25T00:10:37.120 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:37.119+0000 7fb4e4ebd700 1 -- 172.21.15.139:0/3556489867 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4e0102280 msgr2=0x7fb4e0198360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:37.121 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:37.119+0000 7fb4e4ebd700 1 --2- 172.21.15.139:0/3556489867 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4e0102280 0x7fb4e0198360 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb4d000c990 tx=0x7fb4d000cca0 comp rx=0 tx=0).stop 2024-04-25T00:10:37.121 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:37.119+0000 7fb4e4ebd700 1 -- 172.21.15.139:0/3556489867 shutdown_connections 2024-04-25T00:10:37.121 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:37.119+0000 7fb4e4ebd700 1 --2- 172.21.15.139:0/3556489867 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb4c80a2c20 0x7fb4c80a50e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:37.121 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:37.119+0000 7fb4e4ebd700 1 --2- 172.21.15.139:0/3556489867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4e0103e20 0x7fb4e019f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:37.121 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:37.119+0000 7fb4e4ebd700 1 --2- 172.21.15.139:0/3556489867 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4e0102280 0x7fb4e0198360 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:37.121 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:37.119+0000 7fb4e4ebd700 1 --2- 172.21.15.139:0/3556489867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4e0103480 0x7fb4e01988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:37.121 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:37.119+0000 7fb4e4ebd700 1 -- 172.21.15.139:0/3556489867 >> 172.21.15.139:0/3556489867 conn(0x7fb4e00fd820 msgr2=0x7fb4e0106720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:37.121 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:37.119+0000 7fb4e4ebd700 1 -- 172.21.15.139:0/3556489867 shutdown_connections 2024-04-25T00:10:37.121 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:10:37.119+0000 7fb4e4ebd700 1 -- 172.21.15.139:0/3556489867 wait complete. 2024-04-25T00:10:37.440 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:37 smithi139 ceph-mon[32833]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:37.441 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:37 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:37.441 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:37 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:37.441 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:37 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:37.441 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:37 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:10:37.441 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:37 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:37.441 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:37 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:37.441 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:37 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:37.441 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:37 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:10:37.441 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:37 smithi139 ceph-mon[32833]: from='osd.7 [v2:172.21.15.139:6816/4023269486,v1:172.21.15.139:6817/4023269486]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T00:10:37.441 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:37 smithi139 ceph-mon[32833]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T00:10:37.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:37 smithi027 ceph-mon[29208]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:37.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:37 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:37.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:37 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:37.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:37 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:37.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:37 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:10:37.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:37 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:37.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:37 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:37.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:37 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:37.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:37 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:10:37.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:37 smithi027 ceph-mon[29208]: from='osd.7 [v2:172.21.15.139:6816/4023269486,v1:172.21.15.139:6817/4023269486]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T00:10:37.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:37 smithi027 ceph-mon[29208]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T00:10:37.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:37 smithi153 ceph-mon[32896]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:37.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:37 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:37.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:37 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:37.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:37 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:37.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:37 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:10:37.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:37 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:37.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:37 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:37.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:37 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:37.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:37 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:10:37.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:37 smithi153 ceph-mon[32896]: from='osd.7 [v2:172.21.15.139:6816/4023269486,v1:172.21.15.139:6817/4023269486]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T00:10:37.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:37 smithi153 ceph-mon[32896]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T00:10:37.793 DEBUG:teuthology.orchestra.run.smithi139:osd.7> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.7.service 2024-04-25T00:10:37.795 INFO:tasks.cephadm:Deploying osd.8 on smithi153 with /dev/vg_nvme/lv_2... 2024-04-25T00:10:37.796 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T00:10:38.042 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T00:10:39.354 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:39 smithi139 ceph-mon[32833]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:39.354 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:39 smithi139 ceph-mon[32833]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T00:10:39.354 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:39 smithi139 ceph-mon[32833]: osdmap e49: 8 total, 7 up, 8 in 2024-04-25T00:10:39.355 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:39 smithi139 ceph-mon[32833]: from='osd.7 [v2:172.21.15.139:6816/4023269486,v1:172.21.15.139:6817/4023269486]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:10:39.355 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:39 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:10:39.355 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:39 smithi139 ceph-mon[32833]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:10:39.355 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 00:10:39 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-7[51188]: 2024-04-25T00:10:39.125+0000 7fd1ef805700 -1 osd.7 0 waiting for initial osdmap 2024-04-25T00:10:39.355 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 00:10:39 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-7[51188]: 2024-04-25T00:10:39.133+0000 7fd1eb5f8700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:10:39.433 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:39 smithi153 ceph-mon[32896]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:39.433 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:39 smithi153 ceph-mon[32896]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T00:10:39.433 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:39 smithi153 ceph-mon[32896]: osdmap e49: 8 total, 7 up, 8 in 2024-04-25T00:10:39.433 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:39 smithi153 ceph-mon[32896]: from='osd.7 [v2:172.21.15.139:6816/4023269486,v1:172.21.15.139:6817/4023269486]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:10:39.433 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:39 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:10:39.433 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:39 smithi153 ceph-mon[32896]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:10:39.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:39 smithi027 ceph-mon[29208]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:39.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:39 smithi027 ceph-mon[29208]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T00:10:39.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:39 smithi027 ceph-mon[29208]: osdmap e49: 8 total, 7 up, 8 in 2024-04-25T00:10:39.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:39 smithi027 ceph-mon[29208]: from='osd.7 [v2:172.21.15.139:6816/4023269486,v1:172.21.15.139:6817/4023269486]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:10:39.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:39 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:10:39.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:39 smithi027 ceph-mon[29208]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:10:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T00:10:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T00:10:40.143 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-25T00:10:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-25T00:10:40.143 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122426 s, 857 MB/s 2024-04-25T00:10:40.143 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-25T00:10:40.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:40 smithi027 ceph-mon[29208]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T00:10:40.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:40 smithi027 ceph-mon[29208]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T00:10:40.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:40 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:10:40.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:40 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:10:40.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:40 smithi153 ceph-mon[32896]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T00:10:40.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:40 smithi153 ceph-mon[32896]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T00:10:40.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:40 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:10:40.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:40 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:10:40.455 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:40 smithi139 ceph-mon[32833]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T00:10:40.455 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:40 smithi139 ceph-mon[32833]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T00:10:40.455 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:40 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:10:40.455 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:40 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:10:40.818 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_2 2024-04-25T00:10:41.061 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T00:10:41.403 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:41 smithi153 ceph-mon[32896]: purged_snaps scrub starts 2024-04-25T00:10:41.403 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:41 smithi153 ceph-mon[32896]: purged_snaps scrub ok 2024-04-25T00:10:41.403 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:41 smithi153 ceph-mon[32896]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:41.403 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:41 smithi153 ceph-mon[32896]: osd.7 [v2:172.21.15.139:6816/4023269486,v1:172.21.15.139:6817/4023269486] boot 2024-04-25T00:10:41.403 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:41 smithi153 ceph-mon[32896]: osdmap e51: 8 total, 8 up, 8 in 2024-04-25T00:10:41.403 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:41 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:10:41.403 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:41 smithi153 ceph-mon[32896]: osdmap e52: 8 total, 8 up, 8 in 2024-04-25T00:10:41.403 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:41 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:41.403 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:41 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:41.403 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:41 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:41.403 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:41 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:41.404 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:41 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:41.404 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:41 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:41.404 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:41 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:41.404 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:41 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:10:41.404 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:41 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:41.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:41 smithi027 ceph-mon[29208]: purged_snaps scrub starts 2024-04-25T00:10:41.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:41 smithi027 ceph-mon[29208]: purged_snaps scrub ok 2024-04-25T00:10:41.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:41 smithi027 ceph-mon[29208]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:41.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:41 smithi027 ceph-mon[29208]: osd.7 [v2:172.21.15.139:6816/4023269486,v1:172.21.15.139:6817/4023269486] boot 2024-04-25T00:10:41.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:41 smithi027 ceph-mon[29208]: osdmap e51: 8 total, 8 up, 8 in 2024-04-25T00:10:41.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:41 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:10:41.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:41 smithi027 ceph-mon[29208]: osdmap e52: 8 total, 8 up, 8 in 2024-04-25T00:10:41.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:41 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:41.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:41 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:41.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:41 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:41.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:41 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:41.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:41 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:41.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:41 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:41.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:41 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:41.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:41 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:10:41.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:41 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:41.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:41 smithi139 ceph-mon[32833]: purged_snaps scrub starts 2024-04-25T00:10:41.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:41 smithi139 ceph-mon[32833]: purged_snaps scrub ok 2024-04-25T00:10:41.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:41 smithi139 ceph-mon[32833]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T00:10:41.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:41 smithi139 ceph-mon[32833]: osd.7 [v2:172.21.15.139:6816/4023269486,v1:172.21.15.139:6817/4023269486] boot 2024-04-25T00:10:41.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:41 smithi139 ceph-mon[32833]: osdmap e51: 8 total, 8 up, 8 in 2024-04-25T00:10:41.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:41 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:10:41.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:41 smithi139 ceph-mon[32833]: osdmap e52: 8 total, 8 up, 8 in 2024-04-25T00:10:41.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:41 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:41.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:41 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:41.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:41 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:41.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:41 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:41.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:41 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:10:41.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:41 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:41.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:41 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:41.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:41 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:10:41.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:41 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:42.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:42 smithi153 ceph-mon[32896]: Detected new or changed devices on smithi139 2024-04-25T00:10:42.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:42 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi139 to 5696M 2024-04-25T00:10:42.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:42 smithi153 ceph-mon[32896]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T00:10:42.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:42 smithi027 ceph-mon[29208]: Detected new or changed devices on smithi139 2024-04-25T00:10:42.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:42 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi139 to 5696M 2024-04-25T00:10:42.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:42 smithi027 ceph-mon[29208]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T00:10:42.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:42 smithi139 ceph-mon[32833]: Detected new or changed devices on smithi139 2024-04-25T00:10:42.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:42 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi139 to 5696M 2024-04-25T00:10:42.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:42 smithi139 ceph-mon[32833]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T00:10:42.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.741+0000 7f4a9b3b8700 1 -- 172.21.15.153:0/1954418679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a941056a0 msgr2=0x7f4a94107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:42.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.741+0000 7f4a9b3b8700 1 --2- 172.21.15.153:0/1954418679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a941056a0 0x7f4a94107a90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4a84009990 tx=0x7f4a84009ca0 comp rx=0 tx=0).stop 2024-04-25T00:10:42.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.741+0000 7f4a9b3b8700 1 -- 172.21.15.153:0/1954418679 shutdown_connections 2024-04-25T00:10:42.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.741+0000 7f4a9b3b8700 1 --2- 172.21.15.153:0/1954418679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a94107fd0 0x7f4a9410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:42.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.741+0000 7f4a9b3b8700 1 --2- 172.21.15.153:0/1954418679 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a94102d70 0x7f4a94105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:42.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.741+0000 7f4a9b3b8700 1 --2- 172.21.15.153:0/1954418679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a941056a0 0x7f4a94107a90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:42.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.741+0000 7f4a9b3b8700 1 -- 172.21.15.153:0/1954418679 >> 172.21.15.153:0/1954418679 conn(0x7f4a940fc720 msgr2=0x7f4a940feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:42.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.741+0000 7f4a9b3b8700 1 -- 172.21.15.153:0/1954418679 shutdown_connections 2024-04-25T00:10:42.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.741+0000 7f4a9b3b8700 1 -- 172.21.15.153:0/1954418679 wait complete. 2024-04-25T00:10:42.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.742+0000 7f4a9b3b8700 1 Processor -- start 2024-04-25T00:10:42.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.742+0000 7f4a9b3b8700 1 -- start start 2024-04-25T00:10:42.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.742+0000 7f4a9b3b8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a94102d70 0x7f4a941983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:42.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.742+0000 7f4a9b3b8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a941056a0 0x7f4a94198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:42.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.742+0000 7f4a9b3b8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a94107fd0 0x7f4a9419f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:42.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.742+0000 7f4a9b3b8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9410de90 con 0x7f4a94107fd0 2024-04-25T00:10:42.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.742+0000 7f4a9b3b8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9410dd10 con 0x7f4a941056a0 2024-04-25T00:10:42.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.742+0000 7f4a9b3b8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9410e010 con 0x7f4a94102d70 2024-04-25T00:10:42.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.742+0000 7f4a99154700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a94102d70 0x7f4a941983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:42.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.742+0000 7f4a99154700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a94102d70 0x7f4a941983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54140/0 (socket says 172.21.15.153:54140) 2024-04-25T00:10:42.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.742+0000 7f4a99154700 1 -- 172.21.15.153:0/2445072441 learned_addr learned my addr 172.21.15.153:0/2445072441 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T00:10:42.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.743+0000 7f4a99955700 1 --2- 172.21.15.153:0/2445072441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a94107fd0 0x7f4a9419f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:42.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.743+0000 7f4a99154700 1 -- 172.21.15.153:0/2445072441 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a941056a0 msgr2=0x7f4a94198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:42.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.743+0000 7f4a99154700 1 --2- 172.21.15.153:0/2445072441 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a941056a0 0x7f4a94198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:42.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.743+0000 7f4a99154700 1 -- 172.21.15.153:0/2445072441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a94107fd0 msgr2=0x7f4a9419f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:42.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.743+0000 7f4a99154700 1 --2- 172.21.15.153:0/2445072441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a94107fd0 0x7f4a9419f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:42.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.743+0000 7f4a99154700 1 -- 172.21.15.153:0/2445072441 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a941a0040 con 0x7f4a94102d70 2024-04-25T00:10:42.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.743+0000 7f4a99955700 1 --2- 172.21.15.153:0/2445072441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a94107fd0 0x7f4a9419f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:10:42.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.743+0000 7f4a99154700 1 --2- 172.21.15.153:0/2445072441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a94102d70 0x7f4a941983e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000add0 tx=0x7f4a9000c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:42.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.743+0000 7f4a8a7fc700 1 -- 172.21.15.153:0/2445072441 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90017020 con 0x7f4a94102d70 2024-04-25T00:10:42.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.743+0000 7f4a8a7fc700 1 -- 172.21.15.153:0/2445072441 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a9000ce20 con 0x7f4a94102d70 2024-04-25T00:10:42.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.743+0000 7f4a9b3b8700 1 -- 172.21.15.153:0/2445072441 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a941a0330 con 0x7f4a94102d70 2024-04-25T00:10:42.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.743+0000 7f4a8a7fc700 1 -- 172.21.15.153:0/2445072441 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90005520 con 0x7f4a94102d70 2024-04-25T00:10:42.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.743+0000 7f4a9b3b8700 1 -- 172.21.15.153:0/2445072441 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a941a0870 con 0x7f4a94102d70 2024-04-25T00:10:42.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.744+0000 7f4a9b3b8700 1 -- 172.21.15.153:0/2445072441 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a94068f20 con 0x7f4a94102d70 2024-04-25T00:10:42.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.745+0000 7f4a8a7fc700 1 -- 172.21.15.153:0/2445072441 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4a90004ad0 con 0x7f4a94102d70 2024-04-25T00:10:42.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.745+0000 7f4a8a7fc700 1 --2- 172.21.15.153:0/2445072441 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a800a2c20 0x7f4a800a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:42.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.746+0000 7f4a8a7fc700 1 -- 172.21.15.153:0/2445072441 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5397+0+0 (secure 0 0 0) 0x7f4a900cdde0 con 0x7f4a94102d70 2024-04-25T00:10:42.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.746+0000 7f4a98953700 1 --2- 172.21.15.153:0/2445072441 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a800a2c20 0x7f4a800a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:42.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.746+0000 7f4a98953700 1 --2- 172.21.15.153:0/2445072441 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a800a2c20 0x7f4a800a50e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4a84005f50 tx=0x7f4a84005e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:42.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.750+0000 7f4a8a7fc700 1 -- 172.21.15.153:0/2445072441 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4a90021020 con 0x7f4a94102d70 2024-04-25T00:10:42.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:10:42.955+0000 7f4a9b3b8700 1 -- 172.21.15.153:0/2445072441 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f4a94062670 con 0x7f4a800a2c20 2024-04-25T00:10:43.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:43 smithi027 ceph-mon[29208]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:43.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:43 smithi027 ceph-mon[29208]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T00:10:43.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:43 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:10:43.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:43 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:10:43.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:43 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:43.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:43 smithi139 ceph-mon[32833]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:43.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:43 smithi139 ceph-mon[32833]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T00:10:43.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:43 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:10:43.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:43 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:10:43.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:43 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:43.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:43 smithi153 ceph-mon[32896]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:43.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:43 smithi153 ceph-mon[32896]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T00:10:43.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:43 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:10:43.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:43 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:10:43.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:43 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:44.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:44 smithi153 ceph-mon[32896]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:10:44.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:44 smithi153 ceph-mon[32896]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T00:10:44.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:44 smithi027 ceph-mon[29208]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:10:44.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:44 smithi027 ceph-mon[29208]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T00:10:44.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:44 smithi139 ceph-mon[32833]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:10:44.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:44 smithi139 ceph-mon[32833]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T00:10:45.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:45 smithi027 ceph-mon[29208]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:45.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:45 smithi139 ceph-mon[32833]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:45.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:45 smithi153 ceph-mon[32896]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:46.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:46 smithi027 ceph-mon[29208]: from='client.? 172.21.15.153:0/955324080' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b73934bc-9fd5-4da3-8c9b-047a2d116d19"}]: dispatch 2024-04-25T00:10:46.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:46 smithi027 ceph-mon[29208]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b73934bc-9fd5-4da3-8c9b-047a2d116d19"}]: dispatch 2024-04-25T00:10:46.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:46 smithi027 ceph-mon[29208]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b73934bc-9fd5-4da3-8c9b-047a2d116d19"}]': finished 2024-04-25T00:10:46.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:46 smithi027 ceph-mon[29208]: osdmap e56: 9 total, 8 up, 9 in 2024-04-25T00:10:46.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:46 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:10:46.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:46 smithi139 ceph-mon[32833]: from='client.? 172.21.15.153:0/955324080' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b73934bc-9fd5-4da3-8c9b-047a2d116d19"}]: dispatch 2024-04-25T00:10:46.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:46 smithi139 ceph-mon[32833]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b73934bc-9fd5-4da3-8c9b-047a2d116d19"}]: dispatch 2024-04-25T00:10:46.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:46 smithi139 ceph-mon[32833]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b73934bc-9fd5-4da3-8c9b-047a2d116d19"}]': finished 2024-04-25T00:10:46.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:46 smithi139 ceph-mon[32833]: osdmap e56: 9 total, 8 up, 9 in 2024-04-25T00:10:46.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:46 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:10:46.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:46 smithi153 ceph-mon[32896]: from='client.? 172.21.15.153:0/955324080' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b73934bc-9fd5-4da3-8c9b-047a2d116d19"}]: dispatch 2024-04-25T00:10:46.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:46 smithi153 ceph-mon[32896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b73934bc-9fd5-4da3-8c9b-047a2d116d19"}]: dispatch 2024-04-25T00:10:46.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:46 smithi153 ceph-mon[32896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b73934bc-9fd5-4da3-8c9b-047a2d116d19"}]': finished 2024-04-25T00:10:46.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:46 smithi153 ceph-mon[32896]: osdmap e56: 9 total, 8 up, 9 in 2024-04-25T00:10:46.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:46 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:10:47.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:47 smithi027 ceph-mon[29208]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:47.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:47 smithi027 ceph-mon[29208]: from='client.? 172.21.15.153:0/2740344394' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:10:47.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:47 smithi139 ceph-mon[32833]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:47.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:47 smithi139 ceph-mon[32833]: from='client.? 172.21.15.153:0/2740344394' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:10:47.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:47 smithi153 ceph-mon[32896]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:47.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:47 smithi153 ceph-mon[32896]: from='client.? 172.21.15.153:0/2740344394' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:10:49.412 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:49 smithi153 ceph-mon[32896]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:49.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:49 smithi027 ceph-mon[29208]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:49.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:49 smithi139 ceph-mon[32833]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:51.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:51 smithi153 ceph-mon[32896]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:51.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:51 smithi027 ceph-mon[29208]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:51.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:51 smithi139 ceph-mon[32833]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:53.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:53 smithi027 ceph-mon[29208]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:53.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:53 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T00:10:53.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:53 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:53.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:53 smithi139 ceph-mon[32833]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:53.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:53 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T00:10:53.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:53 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:53.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:53 smithi153 ceph-mon[32896]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:53.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T00:10:53.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:53 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:10:54.430 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:54 smithi153 ceph-mon[32896]: Deploying daemon osd.8 on smithi153 2024-04-25T00:10:54.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:54 smithi027 ceph-mon[29208]: Deploying daemon osd.8 on smithi153 2024-04-25T00:10:54.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:54 smithi139 ceph-mon[32833]: Deploying daemon osd.8 on smithi153 2024-04-25T00:10:55.403 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:55 smithi153 ceph-mon[32896]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:55.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:55 smithi027 ceph-mon[29208]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:55.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:55 smithi139 ceph-mon[32833]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:57.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:57 smithi027 ceph-mon[29208]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:57.478 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:57 smithi153 ceph-mon[32896]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:57.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:57 smithi139 ceph-mon[32833]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:59.389 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:59 smithi153 ceph-mon[32896]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:59.389 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:59.389 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:10:59.389 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:10:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:59.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:59 smithi027 ceph-mon[29208]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:59.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:59.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:10:59.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:10:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:59.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:59 smithi139 ceph-mon[32833]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:10:59.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:10:59.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:10:59.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:10:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:01.424 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:01 smithi153 ceph-mon[32896]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:11:01.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:01 smithi027 ceph-mon[29208]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:11:01.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:01 smithi139 ceph-mon[32833]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:11:02.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:11:02.326+0000 7f4a8a7fc700 1 -- 172.21.15.153:0/2445072441 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4a94062670 con 0x7f4a800a2c20 2024-04-25T00:11:02.327 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 8 on host 'smithi153' 2024-04-25T00:11:02.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:11:02.329+0000 7f4a9b3b8700 1 -- 172.21.15.153:0/2445072441 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a800a2c20 msgr2=0x7f4a800a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:02.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:11:02.329+0000 7f4a9b3b8700 1 --2- 172.21.15.153:0/2445072441 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a800a2c20 0x7f4a800a50e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4a84005f50 tx=0x7f4a84005e80 comp rx=0 tx=0).stop 2024-04-25T00:11:02.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:11:02.329+0000 7f4a9b3b8700 1 -- 172.21.15.153:0/2445072441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a94102d70 msgr2=0x7f4a941983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:02.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:11:02.329+0000 7f4a9b3b8700 1 --2- 172.21.15.153:0/2445072441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a94102d70 0x7f4a941983e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000add0 tx=0x7f4a9000c330 comp rx=0 tx=0).stop 2024-04-25T00:11:02.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:11:02.329+0000 7f4a9b3b8700 1 -- 172.21.15.153:0/2445072441 shutdown_connections 2024-04-25T00:11:02.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:11:02.329+0000 7f4a9b3b8700 1 --2- 172.21.15.153:0/2445072441 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a800a2c20 0x7f4a800a50e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:11:02.329+0000 7f4a9b3b8700 1 --2- 172.21.15.153:0/2445072441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a94107fd0 0x7f4a9419f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:11:02.329+0000 7f4a9b3b8700 1 --2- 172.21.15.153:0/2445072441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a94102d70 0x7f4a941983e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:11:02.329+0000 7f4a9b3b8700 1 --2- 172.21.15.153:0/2445072441 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a941056a0 0x7f4a94198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:11:02.329+0000 7f4a9b3b8700 1 -- 172.21.15.153:0/2445072441 >> 172.21.15.153:0/2445072441 conn(0x7f4a940fc720 msgr2=0x7f4a941036b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:02.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:11:02.329+0000 7f4a9b3b8700 1 -- 172.21.15.153:0/2445072441 shutdown_connections 2024-04-25T00:11:02.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:11:02.329+0000 7f4a9b3b8700 1 -- 172.21.15.153:0/2445072441 wait complete. 2024-04-25T00:11:03.051 DEBUG:teuthology.orchestra.run.smithi153:osd.8> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.8.service 2024-04-25T00:11:03.054 INFO:tasks.cephadm:Deploying osd.9 on smithi027 with /dev/vg_nvme/lv_1... 2024-04-25T00:11:03.054 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T00:11:03.314 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:11:03.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:03 smithi027 ceph-mon[29208]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:11:03.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:03 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:03.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:03 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:03.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:03 smithi027 ceph-mon[29208]: from='osd.8 [v2:172.21.15.153:6816/2251501067,v1:172.21.15.153:6817/2251501067]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T00:11:03.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:03 smithi027 ceph-mon[29208]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T00:11:03.439 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:03 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:03.439 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:03 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:03.576 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:03 smithi139 ceph-mon[32833]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:11:03.576 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:03 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:03.576 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:03 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:03.576 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:03 smithi139 ceph-mon[32833]: from='osd.8 [v2:172.21.15.153:6816/2251501067,v1:172.21.15.153:6817/2251501067]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T00:11:03.576 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:03 smithi139 ceph-mon[32833]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T00:11:03.576 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:03 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:03.576 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:03 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:03.748 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:03 smithi153 ceph-mon[32896]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:11:03.749 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:03 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:03.749 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:03 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:03.749 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:03 smithi153 ceph-mon[32896]: from='osd.8 [v2:172.21.15.153:6816/2251501067,v1:172.21.15.153:6817/2251501067]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T00:11:03.749 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:03 smithi153 ceph-mon[32896]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T00:11:03.749 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:03 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:03.749 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:03 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:04.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:04 smithi027 ceph-mon[29208]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T00:11:04.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:04 smithi027 ceph-mon[29208]: from='osd.8 [v2:172.21.15.153:6816/2251501067,v1:172.21.15.153:6817/2251501067]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:11:04.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:04 smithi027 ceph-mon[29208]: osdmap e57: 9 total, 8 up, 9 in 2024-04-25T00:11:04.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:04 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:11:04.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:04 smithi027 ceph-mon[29208]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:11:04.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:04 smithi139 ceph-mon[32833]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T00:11:04.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:04 smithi139 ceph-mon[32833]: from='osd.8 [v2:172.21.15.153:6816/2251501067,v1:172.21.15.153:6817/2251501067]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:11:04.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:04 smithi139 ceph-mon[32833]: osdmap e57: 9 total, 8 up, 9 in 2024-04-25T00:11:04.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:04 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:11:04.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:04 smithi139 ceph-mon[32833]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:11:04.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:04 smithi153 ceph-mon[32896]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T00:11:04.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:04 smithi153 ceph-mon[32896]: from='osd.8 [v2:172.21.15.153:6816/2251501067,v1:172.21.15.153:6817/2251501067]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:11:04.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:04 smithi153 ceph-mon[32896]: osdmap e57: 9 total, 8 up, 9 in 2024-04-25T00:11:04.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:04 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:11:04.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:04 smithi153 ceph-mon[32896]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:11:04.777 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 00:11:04 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-8[51595]: 2024-04-25T00:11:04.312+0000 7fda34190700 -1 osd.8 0 waiting for initial osdmap 2024-04-25T00:11:04.777 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 00:11:04 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-8[51595]: 2024-04-25T00:11:04.322+0000 7fda2ff83700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:11:05.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:05 smithi027 ceph-mon[29208]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:11:05.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:05 smithi027 ceph-mon[29208]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T00:11:05.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:05 smithi027 ceph-mon[29208]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T00:11:05.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:05 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:11:05.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:05 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:11:05.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:05 smithi153 ceph-mon[32896]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:11:05.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:05 smithi153 ceph-mon[32896]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T00:11:05.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:05 smithi153 ceph-mon[32896]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T00:11:05.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:05 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:11:05.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:05 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:11:05.562 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T00:11:05.562 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T00:11:05.562 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-25T00:11:05.562 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-25T00:11:05.562 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124039 s, 845 MB/s 2024-04-25T00:11:05.562 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-25T00:11:05.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:05 smithi139 ceph-mon[32833]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-25T00:11:05.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:05 smithi139 ceph-mon[32833]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T00:11:05.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:05 smithi139 ceph-mon[32833]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T00:11:05.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:05 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:11:05.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:05 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:11:06.245 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_1 2024-04-25T00:11:06.497 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:11:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:06 smithi027 ceph-mon[29208]: purged_snaps scrub starts 2024-04-25T00:11:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:06 smithi027 ceph-mon[29208]: purged_snaps scrub ok 2024-04-25T00:11:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:06 smithi027 ceph-mon[29208]: osd.8 [v2:172.21.15.153:6816/2251501067,v1:172.21.15.153:6817/2251501067] boot 2024-04-25T00:11:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:06 smithi027 ceph-mon[29208]: osdmap e59: 9 total, 9 up, 9 in 2024-04-25T00:11:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:06 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:11:06.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:06 smithi027 ceph-mon[29208]: osdmap e60: 9 total, 9 up, 9 in 2024-04-25T00:11:06.624 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:06 smithi153 ceph-mon[32896]: purged_snaps scrub starts 2024-04-25T00:11:06.624 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:06 smithi153 ceph-mon[32896]: purged_snaps scrub ok 2024-04-25T00:11:06.624 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:06 smithi153 ceph-mon[32896]: osd.8 [v2:172.21.15.153:6816/2251501067,v1:172.21.15.153:6817/2251501067] boot 2024-04-25T00:11:06.625 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:06 smithi153 ceph-mon[32896]: osdmap e59: 9 total, 9 up, 9 in 2024-04-25T00:11:06.625 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:06 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:11:06.625 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:06 smithi153 ceph-mon[32896]: osdmap e60: 9 total, 9 up, 9 in 2024-04-25T00:11:06.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:06 smithi139 ceph-mon[32833]: purged_snaps scrub starts 2024-04-25T00:11:06.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:06 smithi139 ceph-mon[32833]: purged_snaps scrub ok 2024-04-25T00:11:06.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:06 smithi139 ceph-mon[32833]: osd.8 [v2:172.21.15.153:6816/2251501067,v1:172.21.15.153:6817/2251501067] boot 2024-04-25T00:11:06.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:06 smithi139 ceph-mon[32833]: osdmap e59: 9 total, 9 up, 9 in 2024-04-25T00:11:06.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:06 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:11:06.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:06 smithi139 ceph-mon[32833]: osdmap e60: 9 total, 9 up, 9 in 2024-04-25T00:11:07.573 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:07 smithi027 ceph-mon[29208]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:07.573 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:07 smithi027 ceph-mon[29208]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T00:11:07.573 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:07 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:07.573 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:07 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:07.573 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:07 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:07.573 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:07 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:07.573 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:07 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:07.573 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:07 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:07.574 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:07 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:07.574 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:07 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:07.574 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:07 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:07.574 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:07 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:11:07.574 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:07 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:07.574 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:07 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:07.574 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:07 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:07.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:07 smithi139 ceph-mon[32833]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:07.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:07 smithi139 ceph-mon[32833]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T00:11:07.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:07 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:07.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:07 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:07.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:07 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:07.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:07 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:07.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:07 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:07.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:07 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:07.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:07 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:07.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:07 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:07.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:07 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:07.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:07 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:11:07.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:07 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:07.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:07 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:07.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:07 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:07.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:07 smithi153 ceph-mon[32896]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:07.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:07 smithi153 ceph-mon[32896]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T00:11:07.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:07 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:07.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:07 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:07.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:07 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:07.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:07 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:07.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:07 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:07.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:07 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:07.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:07 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:07.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:07 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:07.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:07 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:07.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:07 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:11:07.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:07 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:07.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:07 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:07.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:07 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:08.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.060+0000 7f5e1c037700 1 -- 172.21.15.27:0/3832361229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e141056b0 msgr2=0x7f5e14107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:08.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.060+0000 7f5e1c037700 1 --2- 172.21.15.27:0/3832361229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e141056b0 0x7f5e14107aa0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5e04009990 tx=0x7f5e04009ca0 comp rx=0 tx=0).stop 2024-04-25T00:11:08.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.060+0000 7f5e1c037700 1 -- 172.21.15.27:0/3832361229 shutdown_connections 2024-04-25T00:11:08.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.060+0000 7f5e1c037700 1 --2- 172.21.15.27:0/3832361229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e14107fe0 0x7f5e1410a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:08.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.060+0000 7f5e1c037700 1 --2- 172.21.15.27:0/3832361229 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e14102d80 0x7f5e14105170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:08.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.060+0000 7f5e1c037700 1 --2- 172.21.15.27:0/3832361229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e141056b0 0x7f5e14107aa0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:08.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.060+0000 7f5e1c037700 1 -- 172.21.15.27:0/3832361229 >> 172.21.15.27:0/3832361229 conn(0x7f5e140fc730 msgr2=0x7f5e140feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:08.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.060+0000 7f5e1c037700 1 -- 172.21.15.27:0/3832361229 shutdown_connections 2024-04-25T00:11:08.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.060+0000 7f5e1c037700 1 -- 172.21.15.27:0/3832361229 wait complete. 2024-04-25T00:11:08.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.061+0000 7f5e1c037700 1 Processor -- start 2024-04-25T00:11:08.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.061+0000 7f5e1c037700 1 -- start start 2024-04-25T00:11:08.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.061+0000 7f5e1c037700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e14102d80 0x7f5e14198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:08.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.061+0000 7f5e1c037700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e141056b0 0x7f5e14198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:08.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.061+0000 7f5e1c037700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e14107fe0 0x7f5e1419fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:08.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.061+0000 7f5e1c037700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5e1410df00 con 0x7f5e14107fe0 2024-04-25T00:11:08.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.061+0000 7f5e1c037700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5e1410dd80 con 0x7f5e14102d80 2024-04-25T00:11:08.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.061+0000 7f5e1c037700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5e1410e080 con 0x7f5e141056b0 2024-04-25T00:11:08.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.061+0000 7f5e1a5d4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e14107fe0 0x7f5e1419fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:08.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.061+0000 7f5e1a5d4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e14107fe0 0x7f5e1419fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35922/0 (socket says 172.21.15.27:35922) 2024-04-25T00:11:08.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.061+0000 7f5e1a5d4700 1 -- 172.21.15.27:0/1348241790 learned_addr learned my addr 172.21.15.27:0/1348241790 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:11:08.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.061+0000 7f5e195d2700 1 --2- 172.21.15.27:0/1348241790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e141056b0 0x7f5e14198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:08.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.061+0000 7f5e19dd3700 1 --2- 172.21.15.27:0/1348241790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e14102d80 0x7f5e14198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:08.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.062+0000 7f5e1a5d4700 1 -- 172.21.15.27:0/1348241790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e141056b0 msgr2=0x7f5e14198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.062+0000 7f5e1a5d4700 1 --2- 172.21.15.27:0/1348241790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e141056b0 0x7f5e14198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.062+0000 7f5e1a5d4700 1 -- 172.21.15.27:0/1348241790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e14102d80 msgr2=0x7f5e14198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.062+0000 7f5e1a5d4700 1 --2- 172.21.15.27:0/1348241790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e14102d80 0x7f5e14198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.062+0000 7f5e1a5d4700 1 -- 172.21.15.27:0/1348241790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e141a01d0 con 0x7f5e14107fe0 2024-04-25T00:11:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.062+0000 7f5e19dd3700 1 --2- 172.21.15.27:0/1348241790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e14102d80 0x7f5e14198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:11:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.062+0000 7f5e1a5d4700 1 --2- 172.21.15.27:0/1348241790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e14107fe0 0x7f5e1419fb30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c007ee0 tx=0x7f5e0c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.062+0000 7f5e0affd700 1 -- 172.21.15.27:0/1348241790 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c004db0 con 0x7f5e14107fe0 2024-04-25T00:11:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.062+0000 7f5e0affd700 1 -- 172.21.15.27:0/1348241790 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e0c00dbd0 con 0x7f5e14107fe0 2024-04-25T00:11:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.062+0000 7f5e0affd700 1 -- 172.21.15.27:0/1348241790 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c015660 con 0x7f5e14107fe0 2024-04-25T00:11:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.062+0000 7f5e1c037700 1 -- 172.21.15.27:0/1348241790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e141a04c0 con 0x7f5e14107fe0 2024-04-25T00:11:08.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.062+0000 7f5e1c037700 1 -- 172.21.15.27:0/1348241790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e141a0a30 con 0x7f5e14107fe0 2024-04-25T00:11:08.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.063+0000 7f5e1c037700 1 -- 172.21.15.27:0/1348241790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e14068f20 con 0x7f5e14107fe0 2024-04-25T00:11:08.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.064+0000 7f5e0affd700 1 -- 172.21.15.27:0/1348241790 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5e0c005000 con 0x7f5e14107fe0 2024-04-25T00:11:08.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.065+0000 7f5e0affd700 1 --2- 172.21.15.27:0/1348241790 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f5e000a2c20 0x7f5e000a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:08.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.065+0000 7f5e19dd3700 1 --2- 172.21.15.27:0/1348241790 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f5e000a2c20 0x7f5e000a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:08.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.065+0000 7f5e0affd700 1 -- 172.21.15.27:0/1348241790 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5829+0+0 (secure 0 0 0) 0x7f5e0c0ce970 con 0x7f5e14107fe0 2024-04-25T00:11:08.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.065+0000 7f5e19dd3700 1 --2- 172.21.15.27:0/1348241790 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f5e000a2c20 0x7f5e000a50e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5e100099d0 tx=0x7f5e10008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:08.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.068+0000 7f5e0affd700 1 -- 172.21.15.27:0/1348241790 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5e0c010070 con 0x7f5e14107fe0 2024-04-25T00:11:08.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:08.285+0000 7f5e1c037700 1 -- 172.21.15.27:0/1348241790 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f5e14062670 con 0x7f5e000a2c20 2024-04-25T00:11:08.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:08 smithi027 ceph-mon[29208]: Detected new or changed devices on smithi153 2024-04-25T00:11:08.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:08 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi153 to 5696M 2024-04-25T00:11:08.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:08 smithi027 ceph-mon[29208]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T00:11:08.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:11:08.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:11:08.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:08.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:08 smithi139 ceph-mon[32833]: Detected new or changed devices on smithi153 2024-04-25T00:11:08.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:08 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi153 to 5696M 2024-04-25T00:11:08.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:08 smithi139 ceph-mon[32833]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T00:11:08.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:11:08.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:11:08.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:08.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:08 smithi153 ceph-mon[32896]: Detected new or changed devices on smithi153 2024-04-25T00:11:08.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:08 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi153 to 5696M 2024-04-25T00:11:08.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:08 smithi153 ceph-mon[32896]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T00:11:08.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:11:08.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:11:08.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:09.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:09 smithi027 ceph-mon[29208]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:09.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:09 smithi027 ceph-mon[29208]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:11:09.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:09 smithi027 ceph-mon[29208]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T00:11:09.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:09 smithi139 ceph-mon[32833]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:09.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:09 smithi139 ceph-mon[32833]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:11:09.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:09 smithi139 ceph-mon[32833]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T00:11:09.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:09 smithi153 ceph-mon[32896]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:09.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:09 smithi153 ceph-mon[32896]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:11:09.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:09 smithi153 ceph-mon[32896]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T00:11:11.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:11 smithi027 ceph-mon[29208]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:11.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:11 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2391652268' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "375ebd2b-ed51-4e03-b8bb-9725b30d81b5"}]: dispatch 2024-04-25T00:11:11.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:11 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2391652268' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "375ebd2b-ed51-4e03-b8bb-9725b30d81b5"}]': finished 2024-04-25T00:11:11.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:11 smithi027 ceph-mon[29208]: osdmap e64: 10 total, 9 up, 10 in 2024-04-25T00:11:11.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:11:11.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:11 smithi139 ceph-mon[32833]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:11.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:11 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2391652268' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "375ebd2b-ed51-4e03-b8bb-9725b30d81b5"}]: dispatch 2024-04-25T00:11:11.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:11 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2391652268' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "375ebd2b-ed51-4e03-b8bb-9725b30d81b5"}]': finished 2024-04-25T00:11:11.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:11 smithi139 ceph-mon[32833]: osdmap e64: 10 total, 9 up, 10 in 2024-04-25T00:11:11.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:11:11.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:11 smithi153 ceph-mon[32896]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:11.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:11 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2391652268' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "375ebd2b-ed51-4e03-b8bb-9725b30d81b5"}]: dispatch 2024-04-25T00:11:11.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:11 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2391652268' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "375ebd2b-ed51-4e03-b8bb-9725b30d81b5"}]': finished 2024-04-25T00:11:11.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:11 smithi153 ceph-mon[32896]: osdmap e64: 10 total, 9 up, 10 in 2024-04-25T00:11:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:11:12.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:12 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3224916704' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:11:12.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:12 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3224916704' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:11:12.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:12 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3224916704' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:11:13.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:13 smithi027 ceph-mon[29208]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-25T00:11:13.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:13 smithi139 ceph-mon[32833]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-25T00:11:13.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:13 smithi153 ceph-mon[32896]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-25T00:11:15.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:15 smithi027 ceph-mon[29208]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T00:11:15.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:15 smithi139 ceph-mon[32833]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T00:11:15.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:15 smithi153 ceph-mon[32896]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T00:11:17.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:17 smithi027 ceph-mon[29208]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T00:11:17.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:17 smithi139 ceph-mon[32833]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T00:11:17.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:17 smithi153 ceph-mon[32896]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T00:11:19.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:19 smithi027 ceph-mon[29208]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-25T00:11:19.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:19 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T00:11:19.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:19 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:19.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:19 smithi139 ceph-mon[32833]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-25T00:11:19.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:19 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T00:11:19.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:19 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:19.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:19 smithi153 ceph-mon[32896]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-25T00:11:19.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:19 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T00:11:19.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:19 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:20.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:20 smithi027 ceph-mon[29208]: Deploying daemon osd.9 on smithi027 2024-04-25T00:11:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:20 smithi139 ceph-mon[32833]: Deploying daemon osd.9 on smithi027 2024-04-25T00:11:20.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:20 smithi153 ceph-mon[32896]: Deploying daemon osd.9 on smithi027 2024-04-25T00:11:21.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:21 smithi027 ceph-mon[29208]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T00:11:21.655 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:21 smithi139 ceph-mon[32833]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T00:11:21.678 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:21 smithi153 ceph-mon[32896]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T00:11:23.648 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:23 smithi027 ceph-mon[29208]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:23.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:23 smithi139 ceph-mon[32833]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:23.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:23 smithi153 ceph-mon[32896]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:25.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:25 smithi027 ceph-mon[29208]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:25.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:25 smithi139 ceph-mon[32833]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:25.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:25 smithi153 ceph-mon[32896]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:26.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:26 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:26.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:26 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:11:26.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:26 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:26.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:26 smithi027 ceph-mon[29208]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:26.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:26 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:26.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:26 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:11:26.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:26 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:26.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:26 smithi139 ceph-mon[32833]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:26.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:26 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:26.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:26 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:11:26.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:26 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:26.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:26 smithi153 ceph-mon[32896]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:28.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:28 smithi027 ceph-mon[29208]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:29.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:28 smithi139 ceph-mon[32833]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:29.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:28 smithi153 ceph-mon[32896]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:29.141 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 9 on host 'smithi027' 2024-04-25T00:11:29.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:29.140+0000 7f5e0affd700 1 -- 172.21.15.27:0/1348241790 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5e14062670 con 0x7f5e000a2c20 2024-04-25T00:11:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:29.142+0000 7f5e1c037700 1 -- 172.21.15.27:0/1348241790 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f5e000a2c20 msgr2=0x7f5e000a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:29.142+0000 7f5e1c037700 1 --2- 172.21.15.27:0/1348241790 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f5e000a2c20 0x7f5e000a50e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5e100099d0 tx=0x7f5e10008040 comp rx=0 tx=0).stop 2024-04-25T00:11:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:29.142+0000 7f5e1c037700 1 -- 172.21.15.27:0/1348241790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e14107fe0 msgr2=0x7f5e1419fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:29.142+0000 7f5e1c037700 1 --2- 172.21.15.27:0/1348241790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e14107fe0 0x7f5e1419fb30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c007ee0 tx=0x7f5e0c00d3b0 comp rx=0 tx=0).stop 2024-04-25T00:11:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:29.142+0000 7f5e1c037700 1 -- 172.21.15.27:0/1348241790 shutdown_connections 2024-04-25T00:11:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:29.142+0000 7f5e1c037700 1 --2- 172.21.15.27:0/1348241790 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f5e000a2c20 0x7f5e000a50e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:29.142+0000 7f5e1c037700 1 --2- 172.21.15.27:0/1348241790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e14107fe0 0x7f5e1419fb30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:29.142+0000 7f5e1c037700 1 --2- 172.21.15.27:0/1348241790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e14102d80 0x7f5e14198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:29.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:29.142+0000 7f5e1c037700 1 --2- 172.21.15.27:0/1348241790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e141056b0 0x7f5e14198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:29.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:29.142+0000 7f5e1c037700 1 -- 172.21.15.27:0/1348241790 >> 172.21.15.27:0/1348241790 conn(0x7f5e140fc730 msgr2=0x7f5e14103840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:29.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:29.142+0000 7f5e1c037700 1 -- 172.21.15.27:0/1348241790 shutdown_connections 2024-04-25T00:11:29.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:11:29.142+0000 7f5e1c037700 1 -- 172.21.15.27:0/1348241790 wait complete. 2024-04-25T00:11:29.836 DEBUG:teuthology.orchestra.run.smithi027:osd.9> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.9.service 2024-04-25T00:11:29.838 INFO:tasks.cephadm:Deploying osd.10 on smithi139 with /dev/vg_nvme/lv_1... 2024-04-25T00:11:29.838 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T00:11:30.084 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T00:11:30.207 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:30 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:30.207 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:30 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:30.207 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:30 smithi139 ceph-mon[32833]: from='osd.9 [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T00:11:30.207 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:30 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:30.207 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:30 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:30.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:30 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:30.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:30 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:30.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:30 smithi027 ceph-mon[29208]: from='osd.9 [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T00:11:30.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:30 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:30.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:30 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:30.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:30 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:30.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:30 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:30.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:30 smithi153 ceph-mon[32896]: from='osd.9 [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T00:11:30.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:30 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:30.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:30 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:31.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:31 smithi027 ceph-mon[29208]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:31.362 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:31 smithi027 ceph-mon[29208]: from='osd.9 [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T00:11:31.362 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:31 smithi027 ceph-mon[29208]: osdmap e65: 10 total, 9 up, 10 in 2024-04-25T00:11:31.362 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:31 smithi027 ceph-mon[29208]: from='osd.9 [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T00:11:31.362 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:31 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:11:31.363 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:11:31 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:11:31.114+0000 7f10e28f0700 -1 osd.9 0 waiting for initial osdmap 2024-04-25T00:11:31.363 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:11:31 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:11:31.128+0000 7f10de6e3700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:11:31.452 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:31 smithi139 ceph-mon[32833]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:31.452 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:31 smithi139 ceph-mon[32833]: from='osd.9 [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T00:11:31.452 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:31 smithi139 ceph-mon[32833]: osdmap e65: 10 total, 9 up, 10 in 2024-04-25T00:11:31.452 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:31 smithi139 ceph-mon[32833]: from='osd.9 [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T00:11:31.452 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:31 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:11:31.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:31 smithi153 ceph-mon[32896]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:31.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:31 smithi153 ceph-mon[32896]: from='osd.9 [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T00:11:31.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:31 smithi153 ceph-mon[32896]: osdmap e65: 10 total, 9 up, 10 in 2024-04-25T00:11:31.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:31 smithi153 ceph-mon[32896]: from='osd.9 [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T00:11:31.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:31 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:11:32.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:32 smithi027 ceph-mon[29208]: from='osd.9 [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T00:11:32.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:32 smithi027 ceph-mon[29208]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T00:11:32.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:32 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:11:32.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:32 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:11:32.385 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:32 smithi153 ceph-mon[32896]: from='osd.9 [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T00:11:32.385 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:32 smithi153 ceph-mon[32896]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T00:11:32.385 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:32 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:11:32.385 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:32 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:11:32.396 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:32 smithi139 ceph-mon[32833]: from='osd.9 [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T00:11:32.396 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:32 smithi139 ceph-mon[32833]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T00:11:32.396 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:32 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:11:32.397 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:32 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:11:32.397 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T00:11:32.397 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T00:11:32.397 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2024-04-25T00:11:32.397 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2024-04-25T00:11:32.397 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115019 s, 912 MB/s 2024-04-25T00:11:32.397 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2024-04-25T00:11:33.188 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi139:vg_nvme/lv_1 2024-04-25T00:11:33.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:33 smithi153 ceph-mon[32896]: purged_snaps scrub starts 2024-04-25T00:11:33.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:33 smithi153 ceph-mon[32896]: purged_snaps scrub ok 2024-04-25T00:11:33.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:33 smithi153 ceph-mon[32896]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:33.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:33 smithi153 ceph-mon[32896]: osd.9 [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594] boot 2024-04-25T00:11:33.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:33 smithi153 ceph-mon[32896]: osdmap e67: 10 total, 10 up, 10 in 2024-04-25T00:11:33.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:33 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:11:33.332 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:33 smithi027 ceph-mon[29208]: purged_snaps scrub starts 2024-04-25T00:11:33.332 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:33 smithi027 ceph-mon[29208]: purged_snaps scrub ok 2024-04-25T00:11:33.332 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:33 smithi027 ceph-mon[29208]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:33.332 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:33 smithi027 ceph-mon[29208]: osd.9 [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594] boot 2024-04-25T00:11:33.332 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:33 smithi027 ceph-mon[29208]: osdmap e67: 10 total, 10 up, 10 in 2024-04-25T00:11:33.332 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:33 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:11:33.429 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T00:11:33.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:33 smithi139 ceph-mon[32833]: purged_snaps scrub starts 2024-04-25T00:11:33.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:33 smithi139 ceph-mon[32833]: purged_snaps scrub ok 2024-04-25T00:11:33.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:33 smithi139 ceph-mon[32833]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-25T00:11:33.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:33 smithi139 ceph-mon[32833]: osd.9 [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594] boot 2024-04-25T00:11:33.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:33 smithi139 ceph-mon[32833]: osdmap e67: 10 total, 10 up, 10 in 2024-04-25T00:11:33.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:33 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:11:34.409 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:34 smithi027 ceph-mon[29208]: osdmap e68: 10 total, 10 up, 10 in 2024-04-25T00:11:34.409 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:34.409 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:34.409 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:34.409 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:34.409 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:34.409 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:34.409 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:34.409 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:34.409 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:34.410 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:34.410 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:11:34.410 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:34.410 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:34.410 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:34 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:34.502 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:34 smithi139 ceph-mon[32833]: osdmap e68: 10 total, 10 up, 10 in 2024-04-25T00:11:34.502 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:34.502 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:34.502 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:11:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:34 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:34.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:34 smithi153 ceph-mon[32896]: osdmap e68: 10 total, 10 up, 10 in 2024-04-25T00:11:34.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:34.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:34.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:34.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:34.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:34.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:34.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:34.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:34.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:34.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:34.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:11:34.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:34.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:34.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:34 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:35.002 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:34.999+0000 7fe657677700 1 -- 172.21.15.139:0/1052732466 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe650101190 msgr2=0x7fe6501015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:35.002 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:34.999+0000 7fe657677700 1 --2- 172.21.15.139:0/1052732466 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe650101190 0x7fe6501015b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe64c009920 tx=0x7fe64c009c30 comp rx=0 tx=0).stop 2024-04-25T00:11:35.002 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:34.999+0000 7fe657677700 1 -- 172.21.15.139:0/1052732466 shutdown_connections 2024-04-25T00:11:35.002 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:34.999+0000 7fe657677700 1 --2- 172.21.15.139:0/1052732466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe650102d30 0x7fe6501095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.002 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:34.999+0000 7fe657677700 1 --2- 172.21.15.139:0/1052732466 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe650101190 0x7fe6501015b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.002 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:34.999+0000 7fe657677700 1 --2- 172.21.15.139:0/1052732466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe650102390 0x7fe6501027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:34.999+0000 7fe657677700 1 -- 172.21.15.139:0/1052732466 >> 172.21.15.139:0/1052732466 conn(0x7fe6500fc730 msgr2=0x7fe6500feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:35.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:34.999+0000 7fe657677700 1 -- 172.21.15.139:0/1052732466 shutdown_connections 2024-04-25T00:11:35.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:34.999+0000 7fe657677700 1 -- 172.21.15.139:0/1052732466 wait complete. 2024-04-25T00:11:35.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:34.999+0000 7fe657677700 1 Processor -- start 2024-04-25T00:11:35.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:34.999+0000 7fe657677700 1 -- start start 2024-04-25T00:11:35.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:34.999+0000 7fe657677700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe650101190 0x7fe650105310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:35.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:34.999+0000 7fe657677700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe650102390 0x7fe650103960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:35.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe657677700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe650102d30 0x7fe650103ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:35.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe657677700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe65010cd70 con 0x7fe650102390 2024-04-25T00:11:35.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe657677700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe65010cbf0 con 0x7fe650102d30 2024-04-25T00:11:35.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe657677700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe65010cef0 con 0x7fe650101190 2024-04-25T00:11:35.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe655c14700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe650102d30 0x7fe650103ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:35.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe655c14700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe650102d30 0x7fe650103ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:51364/0 (socket says 172.21.15.139:51364) 2024-04-25T00:11:35.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe655c14700 1 -- 172.21.15.139:0/4011208709 learned_addr learned my addr 172.21.15.139:0/4011208709 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-04-25T00:11:35.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe654c12700 1 --2- 172.21.15.139:0/4011208709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe650102390 0x7fe650103960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:35.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe655413700 1 --2- 172.21.15.139:0/4011208709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe650101190 0x7fe650105310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:35.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe655c14700 1 -- 172.21.15.139:0/4011208709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe650101190 msgr2=0x7fe650105310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:35.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe655c14700 1 --2- 172.21.15.139:0/4011208709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe650101190 0x7fe650105310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe655c14700 1 -- 172.21.15.139:0/4011208709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe650102390 msgr2=0x7fe650103960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:35.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe655c14700 1 --2- 172.21.15.139:0/4011208709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe650102390 0x7fe650103960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe655c14700 1 -- 172.21.15.139:0/4011208709 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe650104750 con 0x7fe650102d30 2024-04-25T00:11:35.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe654c12700 1 --2- 172.21.15.139:0/4011208709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe650102390 0x7fe650103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:11:35.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe655c14700 1 --2- 172.21.15.139:0/4011208709 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe650102d30 0x7fe650103ea0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe64800c110 tx=0x7fe64800c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:35.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe6467fc700 1 -- 172.21.15.139:0/4011208709 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6480092e0 con 0x7fe650102d30 2024-04-25T00:11:35.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe6467fc700 1 -- 172.21.15.139:0/4011208709 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe648004bb0 con 0x7fe650102d30 2024-04-25T00:11:35.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe657677700 1 -- 172.21.15.139:0/4011208709 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6501a2700 con 0x7fe650102d30 2024-04-25T00:11:35.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe6467fc700 1 -- 172.21.15.139:0/4011208709 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe648007d10 con 0x7fe650102d30 2024-04-25T00:11:35.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.000+0000 7fe657677700 1 -- 172.21.15.139:0/4011208709 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6501a2bc0 con 0x7fe650102d30 2024-04-25T00:11:35.005 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.001+0000 7fe657677700 1 -- 172.21.15.139:0/4011208709 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe650068f20 con 0x7fe650102d30 2024-04-25T00:11:35.005 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.003+0000 7fe6467fc700 1 -- 172.21.15.139:0/4011208709 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe64801a020 con 0x7fe650102d30 2024-04-25T00:11:35.005 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.004+0000 7fe6467fc700 1 --2- 172.21.15.139:0/4011208709 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe63c0a2c20 0x7fe63c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:35.006 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.004+0000 7fe6467fc700 1 -- 172.21.15.139:0/4011208709 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7fe6480ced90 con 0x7fe650102d30 2024-04-25T00:11:35.006 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.004+0000 7fe655413700 1 --2- 172.21.15.139:0/4011208709 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe63c0a2c20 0x7fe63c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:35.006 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.004+0000 7fe655413700 1 --2- 172.21.15.139:0/4011208709 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe63c0a2c20 0x7fe63c0a50e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00a010 tx=0x7fe64c019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:35.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.007+0000 7fe6467fc700 1 -- 172.21.15.139:0/4011208709 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe648010040 con 0x7fe650102d30 2024-04-25T00:11:35.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:35.213+0000 7fe657677700 1 -- 172.21.15.139:0/4011208709 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe65006b090 con 0x7fe63c0a2c20 2024-04-25T00:11:35.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:35 smithi027 ceph-mon[29208]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:35.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:35 smithi027 ceph-mon[29208]: Detected new or changed devices on smithi027 2024-04-25T00:11:35.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:35 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi027 to 4272M 2024-04-25T00:11:35.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:35 smithi139 ceph-mon[32833]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:35.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:35 smithi139 ceph-mon[32833]: Detected new or changed devices on smithi027 2024-04-25T00:11:35.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:35 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi027 to 4272M 2024-04-25T00:11:35.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:35 smithi153 ceph-mon[32896]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:35.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:35 smithi153 ceph-mon[32896]: Detected new or changed devices on smithi027 2024-04-25T00:11:35.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:35 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi027 to 4272M 2024-04-25T00:11:36.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:36 smithi027 ceph-mon[29208]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:11:36.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:36 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:11:36.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:36 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:11:36.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:36 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:36.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:36 smithi139 ceph-mon[32833]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:11:36.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:36 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:11:36.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:36 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:11:36.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:36 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:36.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:36 smithi153 ceph-mon[32896]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:11:36.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:36 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:11:36.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:36 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:11:36.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:36 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:37.367 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:37 smithi139 ceph-mon[32833]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:37.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:37 smithi027 ceph-mon[29208]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:37.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:37 smithi153 ceph-mon[32896]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:38.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:38 smithi139 ceph-mon[32833]: from='client.? 172.21.15.139:0/3431896777' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea5bca6d-3675-453d-a0f2-cae08531fe5f"}]: dispatch 2024-04-25T00:11:38.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:38 smithi139 ceph-mon[32833]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea5bca6d-3675-453d-a0f2-cae08531fe5f"}]: dispatch 2024-04-25T00:11:38.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:38 smithi139 ceph-mon[32833]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea5bca6d-3675-453d-a0f2-cae08531fe5f"}]': finished 2024-04-25T00:11:38.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:38 smithi139 ceph-mon[32833]: osdmap e69: 11 total, 10 up, 11 in 2024-04-25T00:11:38.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:38 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:11:38.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:38 smithi027 ceph-mon[29208]: from='client.? 172.21.15.139:0/3431896777' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea5bca6d-3675-453d-a0f2-cae08531fe5f"}]: dispatch 2024-04-25T00:11:38.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:38 smithi027 ceph-mon[29208]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea5bca6d-3675-453d-a0f2-cae08531fe5f"}]: dispatch 2024-04-25T00:11:38.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:38 smithi027 ceph-mon[29208]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea5bca6d-3675-453d-a0f2-cae08531fe5f"}]': finished 2024-04-25T00:11:38.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:38 smithi027 ceph-mon[29208]: osdmap e69: 11 total, 10 up, 11 in 2024-04-25T00:11:38.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:11:38.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:38 smithi153 ceph-mon[32896]: from='client.? 172.21.15.139:0/3431896777' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea5bca6d-3675-453d-a0f2-cae08531fe5f"}]: dispatch 2024-04-25T00:11:38.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:38 smithi153 ceph-mon[32896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea5bca6d-3675-453d-a0f2-cae08531fe5f"}]: dispatch 2024-04-25T00:11:38.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:38 smithi153 ceph-mon[32896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea5bca6d-3675-453d-a0f2-cae08531fe5f"}]': finished 2024-04-25T00:11:38.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:38 smithi153 ceph-mon[32896]: osdmap e69: 11 total, 10 up, 11 in 2024-04-25T00:11:38.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:11:39.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:39 smithi027 ceph-mon[29208]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:39.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:39 smithi027 ceph-mon[29208]: from='client.? 172.21.15.139:0/814724757' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:11:39.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:39 smithi139 ceph-mon[32833]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:39.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:39 smithi139 ceph-mon[32833]: from='client.? 172.21.15.139:0/814724757' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:11:39.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:39 smithi153 ceph-mon[32896]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:39.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:39 smithi153 ceph-mon[32896]: from='client.? 172.21.15.139:0/814724757' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:11:41.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:41 smithi027 ceph-mon[29208]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:41.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:41 smithi139 ceph-mon[32833]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:41.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:41 smithi153 ceph-mon[32896]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:43.413 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:43 smithi139 ceph-mon[32833]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:43.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:43 smithi027 ceph-mon[29208]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:43.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:43 smithi153 ceph-mon[32896]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:45.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:45 smithi027 ceph-mon[29208]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:45.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:45 smithi139 ceph-mon[32833]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:45.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:45 smithi153 ceph-mon[32896]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:46.416 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:46 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T00:11:46.416 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:46 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:46.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:46 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T00:11:46.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:46 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:46.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:46 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T00:11:46.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:46 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:47.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:47 smithi027 ceph-mon[29208]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:47.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:47 smithi027 ceph-mon[29208]: Deploying daemon osd.10 on smithi139 2024-04-25T00:11:47.493 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:47 smithi139 ceph-mon[32833]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:47.493 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:47 smithi139 ceph-mon[32833]: Deploying daemon osd.10 on smithi139 2024-04-25T00:11:47.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:47 smithi153 ceph-mon[32896]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:47.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:47 smithi153 ceph-mon[32896]: Deploying daemon osd.10 on smithi139 2024-04-25T00:11:49.435 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:49 smithi139 ceph-mon[32833]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:49.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:49 smithi027 ceph-mon[29208]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:49.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:49 smithi153 ceph-mon[32896]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:51.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:51 smithi027 ceph-mon[29208]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:51.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:51 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:11:51.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:51 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:51.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:51 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:51.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:51 smithi139 ceph-mon[32833]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:51.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:51 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:11:51.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:51 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:51.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:51 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:51.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:51 smithi153 ceph-mon[32896]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:51.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:51 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:11:51.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:51 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:51.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:51 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:53.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:53 smithi027 ceph-mon[29208]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:53.486 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:53 smithi139 ceph-mon[32833]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:53.521 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:53 smithi153 ceph-mon[32896]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:55.033 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:55.032+0000 7fe6467fc700 1 -- 172.21.15.139:0/4011208709 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fe65006b090 con 0x7fe63c0a2c20 2024-04-25T00:11:55.033 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 10 on host 'smithi139' 2024-04-25T00:11:55.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:55.035+0000 7fe657677700 1 -- 172.21.15.139:0/4011208709 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe63c0a2c20 msgr2=0x7fe63c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:55.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:55.035+0000 7fe657677700 1 --2- 172.21.15.139:0/4011208709 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe63c0a2c20 0x7fe63c0a50e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00a010 tx=0x7fe64c019040 comp rx=0 tx=0).stop 2024-04-25T00:11:55.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:55.035+0000 7fe657677700 1 -- 172.21.15.139:0/4011208709 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe650102d30 msgr2=0x7fe650103ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:55.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:55.035+0000 7fe657677700 1 --2- 172.21.15.139:0/4011208709 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe650102d30 0x7fe650103ea0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe64800c110 tx=0x7fe64800c420 comp rx=0 tx=0).stop 2024-04-25T00:11:55.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:55.035+0000 7fe657677700 1 -- 172.21.15.139:0/4011208709 shutdown_connections 2024-04-25T00:11:55.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:55.035+0000 7fe657677700 1 --2- 172.21.15.139:0/4011208709 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe63c0a2c20 0x7fe63c0a50e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:55.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:55.035+0000 7fe657677700 1 --2- 172.21.15.139:0/4011208709 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe650102d30 0x7fe650103ea0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:55.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:55.035+0000 7fe657677700 1 --2- 172.21.15.139:0/4011208709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe650101190 0x7fe650105310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:55.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:55.035+0000 7fe657677700 1 --2- 172.21.15.139:0/4011208709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe650102390 0x7fe650103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:55.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:55.035+0000 7fe657677700 1 -- 172.21.15.139:0/4011208709 >> 172.21.15.139:0/4011208709 conn(0x7fe6500fc730 msgr2=0x7fe65006a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:55.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:55.035+0000 7fe657677700 1 -- 172.21.15.139:0/4011208709 shutdown_connections 2024-04-25T00:11:55.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T00:11:55.035+0000 7fe657677700 1 -- 172.21.15.139:0/4011208709 wait complete. 2024-04-25T00:11:55.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:55 smithi139 ceph-mon[32833]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:55.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:55 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:55.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:55 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:55.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:55 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:55.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:55 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:55.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:55 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:55.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:55 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:55.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:55 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:55.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:55 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:11:55.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:55 smithi027 ceph-mon[29208]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:55.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:55 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:55.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:55 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:55.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:55 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:55.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:55 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:55.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:55 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:55.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:55 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:55.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:55 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:55.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:55 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:11:55.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:55 smithi153 ceph-mon[32896]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:55.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:55 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:55.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:55 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:55.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:55 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:55.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:55 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:55.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:55 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:55.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:55 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:55.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:55 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:55.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:55 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:11:55.775 DEBUG:teuthology.orchestra.run.smithi139:osd.10> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.10.service 2024-04-25T00:11:55.777 INFO:tasks.cephadm:Deploying osd.11 on smithi153 with /dev/vg_nvme/lv_1... 2024-04-25T00:11:55.777 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T00:11:56.026 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T00:11:56.427 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:56 smithi153 ceph-mon[32896]: from='osd.10 [v2:172.21.15.139:6824/3154318186,v1:172.21.15.139:6825/3154318186]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T00:11:56.427 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:56 smithi153 ceph-mon[32896]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T00:11:56.427 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:56 smithi153 ceph-mon[32896]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T00:11:56.427 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:56 smithi153 ceph-mon[32896]: from='osd.10 [v2:172.21.15.139:6824/3154318186,v1:172.21.15.139:6825/3154318186]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:11:56.428 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:56 smithi153 ceph-mon[32896]: osdmap e70: 11 total, 10 up, 11 in 2024-04-25T00:11:56.428 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:56 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:11:56.428 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:56 smithi153 ceph-mon[32896]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:11:56.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:56 smithi027 ceph-mon[29208]: from='osd.10 [v2:172.21.15.139:6824/3154318186,v1:172.21.15.139:6825/3154318186]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T00:11:56.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:56 smithi027 ceph-mon[29208]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T00:11:56.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:56 smithi027 ceph-mon[29208]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T00:11:56.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:56 smithi027 ceph-mon[29208]: from='osd.10 [v2:172.21.15.139:6824/3154318186,v1:172.21.15.139:6825/3154318186]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:11:56.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:56 smithi027 ceph-mon[29208]: osdmap e70: 11 total, 10 up, 11 in 2024-04-25T00:11:56.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:56 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:11:56.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:56 smithi027 ceph-mon[29208]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:11:56.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:56 smithi139 ceph-mon[32833]: from='osd.10 [v2:172.21.15.139:6824/3154318186,v1:172.21.15.139:6825/3154318186]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T00:11:56.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:56 smithi139 ceph-mon[32833]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T00:11:56.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:56 smithi139 ceph-mon[32833]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T00:11:56.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:56 smithi139 ceph-mon[32833]: from='osd.10 [v2:172.21.15.139:6824/3154318186,v1:172.21.15.139:6825/3154318186]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:11:56.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:56 smithi139 ceph-mon[32833]: osdmap e70: 11 total, 10 up, 11 in 2024-04-25T00:11:56.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:56 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:11:56.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:56 smithi139 ceph-mon[32833]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T00:11:57.198 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:57 smithi139 ceph-mon[32833]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:57.198 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:57 smithi139 ceph-mon[32833]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T00:11:57.199 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:57 smithi139 ceph-mon[32833]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T00:11:57.199 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:57 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:11:57.199 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:57 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:11:57.199 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:11:56 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:11:56.994+0000 7f858256f700 -1 osd.10 0 waiting for initial osdmap 2024-04-25T00:11:57.199 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:11:57 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:11:57.005+0000 7f857e362700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:11:57.337 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:57 smithi153 ceph-mon[32896]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:57.338 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:57 smithi153 ceph-mon[32896]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T00:11:57.338 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:57 smithi153 ceph-mon[32896]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T00:11:57.338 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:57 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:11:57.338 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:57 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:11:57.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:57 smithi027 ceph-mon[29208]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:57.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:57 smithi027 ceph-mon[29208]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T00:11:57.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:57 smithi027 ceph-mon[29208]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T00:11:57.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:57 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:11:57.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:57 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:11:58.270 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T00:11:58.270 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T00:11:58.271 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-25T00:11:58.271 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-25T00:11:58.271 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131447 s, 798 MB/s 2024-04-25T00:11:58.271 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-25T00:11:58.930 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_1 2024-04-25T00:11:59.167 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T00:11:59.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:58 smithi139 ceph-mon[32833]: purged_snaps scrub starts 2024-04-25T00:11:59.282 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:58 smithi139 ceph-mon[32833]: purged_snaps scrub ok 2024-04-25T00:11:59.282 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:58 smithi139 ceph-mon[32833]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:59.282 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:58 smithi139 ceph-mon[32833]: osd.10 [v2:172.21.15.139:6824/3154318186,v1:172.21.15.139:6825/3154318186] boot 2024-04-25T00:11:59.282 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:58 smithi139 ceph-mon[32833]: osdmap e72: 11 total, 11 up, 11 in 2024-04-25T00:11:59.282 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:58 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:11:59.282 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:58 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:59.282 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:58 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:59.282 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:58 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:59.282 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:58 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:59.282 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:58 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:59.283 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:58 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:59.283 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:58 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:59.283 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:58 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:59.283 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:58 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:59.283 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:11:58 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:59.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:58 smithi153 ceph-mon[32896]: purged_snaps scrub starts 2024-04-25T00:11:59.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:58 smithi153 ceph-mon[32896]: purged_snaps scrub ok 2024-04-25T00:11:59.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:58 smithi153 ceph-mon[32896]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:59.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:58 smithi153 ceph-mon[32896]: osd.10 [v2:172.21.15.139:6824/3154318186,v1:172.21.15.139:6825/3154318186] boot 2024-04-25T00:11:59.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:58 smithi153 ceph-mon[32896]: osdmap e72: 11 total, 11 up, 11 in 2024-04-25T00:11:59.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:11:59.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:59.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:59.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:59.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:59.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:59.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:59.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:59.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:59.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:59.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:11:58 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:58 smithi027 ceph-mon[29208]: purged_snaps scrub starts 2024-04-25T00:11:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:58 smithi027 ceph-mon[29208]: purged_snaps scrub ok 2024-04-25T00:11:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:58 smithi027 ceph-mon[29208]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-04-25T00:11:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:58 smithi027 ceph-mon[29208]: osd.10 [v2:172.21.15.139:6824/3154318186,v1:172.21.15.139:6825/3154318186] boot 2024-04-25T00:11:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:58 smithi027 ceph-mon[29208]: osdmap e72: 11 total, 11 up, 11 in 2024-04-25T00:11:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:11:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:11:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:11:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:11:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:11:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:11:58 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:00.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:00 smithi139 ceph-mon[32833]: Detected new or changed devices on smithi139 2024-04-25T00:12:00.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:00 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi139 to 4272M 2024-04-25T00:12:00.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:00 smithi139 ceph-mon[32833]: osdmap e73: 11 total, 11 up, 11 in 2024-04-25T00:12:00.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:00 smithi153 ceph-mon[32896]: Detected new or changed devices on smithi139 2024-04-25T00:12:00.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:00 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi139 to 4272M 2024-04-25T00:12:00.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:00 smithi153 ceph-mon[32896]: osdmap e73: 11 total, 11 up, 11 in 2024-04-25T00:12:00.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:00 smithi027 ceph-mon[29208]: Detected new or changed devices on smithi139 2024-04-25T00:12:00.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:00 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi139 to 4272M 2024-04-25T00:12:00.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:00 smithi027 ceph-mon[29208]: osdmap e73: 11 total, 11 up, 11 in 2024-04-25T00:12:00.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.939+0000 7f3c80ed1700 1 -- 172.21.15.153:0/667805262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c7c101180 msgr2=0x7f3c7c1015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:00.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.939+0000 7f3c80ed1700 1 --2- 172.21.15.153:0/667805262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c7c101180 0x7f3c7c1015a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c009990 tx=0x7f3c6c009ca0 comp rx=0 tx=0).stop 2024-04-25T00:12:00.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.940+0000 7f3c80ed1700 1 -- 172.21.15.153:0/667805262 shutdown_connections 2024-04-25T00:12:00.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.940+0000 7f3c80ed1700 1 --2- 172.21.15.153:0/667805262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7c102d20 0x7f3c7c1095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:00.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.940+0000 7f3c80ed1700 1 --2- 172.21.15.153:0/667805262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c7c101180 0x7f3c7c1015a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:00.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.940+0000 7f3c80ed1700 1 --2- 172.21.15.153:0/667805262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c7c102380 0x7f3c7c1027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:00.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.940+0000 7f3c80ed1700 1 -- 172.21.15.153:0/667805262 >> 172.21.15.153:0/667805262 conn(0x7f3c7c0fc720 msgr2=0x7f3c7c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:00.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.940+0000 7f3c80ed1700 1 -- 172.21.15.153:0/667805262 shutdown_connections 2024-04-25T00:12:00.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.940+0000 7f3c80ed1700 1 -- 172.21.15.153:0/667805262 wait complete. 2024-04-25T00:12:00.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.940+0000 7f3c80ed1700 1 Processor -- start 2024-04-25T00:12:00.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.940+0000 7f3c80ed1700 1 -- start start 2024-04-25T00:12:00.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c80ed1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7c101180 0x7f3c7c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:00.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c80ed1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c7c102380 0x7f3c7c198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:00.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c80ed1700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c7c102d20 0x7f3c7c19fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:00.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c80ed1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c10ce90 con 0x7f3c7c101180 2024-04-25T00:12:00.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c80ed1700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c10cd10 con 0x7f3c7c102d20 2024-04-25T00:12:00.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c80ed1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c10d010 con 0x7f3c7c102380 2024-04-25T00:12:00.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c79d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c7c102380 0x7f3c7c198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:00.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c7a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7c101180 0x7f3c7c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:00.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c79d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c7c102380 0x7f3c7c198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37648/0 (socket says 172.21.15.153:37648) 2024-04-25T00:12:00.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c79d9b700 1 -- 172.21.15.153:0/1338777062 learned_addr learned my addr 172.21.15.153:0/1338777062 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T00:12:00.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c7ad9d700 1 --2- 172.21.15.153:0/1338777062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c7c102d20 0x7f3c7c19fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:00.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c79d9b700 1 -- 172.21.15.153:0/1338777062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c7c102d20 msgr2=0x7f3c7c19fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c79d9b700 1 --2- 172.21.15.153:0/1338777062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c7c102d20 0x7f3c7c19fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c79d9b700 1 -- 172.21.15.153:0/1338777062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7c101180 msgr2=0x7f3c7c198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c79d9b700 1 --2- 172.21.15.153:0/1338777062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7c101180 0x7f3c7c198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c79d9b700 1 -- 172.21.15.153:0/1338777062 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c7c1a02f0 con 0x7f3c7c102380 2024-04-25T00:12:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c7a59c700 1 --2- 172.21.15.153:0/1338777062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7c101180 0x7f3c7c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:12:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.941+0000 7f3c79d9b700 1 --2- 172.21.15.153:0/1338777062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c7c102380 0x7f3c7c198bd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000b560 tx=0x7f3c7000b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.942+0000 7f3c6b7fe700 1 -- 172.21.15.153:0/1338777062 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c70015020 con 0x7f3c7c102380 2024-04-25T00:12:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.942+0000 7f3c6b7fe700 1 -- 172.21.15.153:0/1338777062 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c70011d10 con 0x7f3c7c102380 2024-04-25T00:12:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.942+0000 7f3c6b7fe700 1 -- 172.21.15.153:0/1338777062 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7000f2e0 con 0x7f3c7c102380 2024-04-25T00:12:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.942+0000 7f3c80ed1700 1 -- 172.21.15.153:0/1338777062 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c7c1a05e0 con 0x7f3c7c102380 2024-04-25T00:12:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.942+0000 7f3c80ed1700 1 -- 172.21.15.153:0/1338777062 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c7c1a0ba0 con 0x7f3c7c102380 2024-04-25T00:12:00.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.943+0000 7f3c80ed1700 1 -- 172.21.15.153:0/1338777062 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c7c068f20 con 0x7f3c7c102380 2024-04-25T00:12:00.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.944+0000 7f3c6b7fe700 1 -- 172.21.15.153:0/1338777062 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3c70018020 con 0x7f3c7c102380 2024-04-25T00:12:00.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.944+0000 7f3c6b7fe700 1 --2- 172.21.15.153:0/1338777062 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3c640a2ca0 0x7f3c640a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:00.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.944+0000 7f3c6b7fe700 1 -- 172.21.15.153:0/1338777062 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7f3c700ce0c0 con 0x7f3c7c102380 2024-04-25T00:12:00.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.944+0000 7f3c7a59c700 1 --2- 172.21.15.153:0/1338777062 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3c640a2ca0 0x7f3c640a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:00.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.945+0000 7f3c7a59c700 1 --2- 172.21.15.153:0/1338777062 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3c640a2ca0 0x7f3c640a5160 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c003890 tx=0x7f3c6c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:00.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:00.949+0000 7f3c6b7fe700 1 -- 172.21.15.153:0/1338777062 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3c7001f020 con 0x7f3c7c102380 2024-04-25T00:12:01.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:01.155+0000 7f3c80ed1700 1 -- 172.21.15.153:0/1338777062 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3c7c1a0e60 con 0x7f3c640a2ca0 2024-04-25T00:12:01.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:01 smithi139 ceph-mon[32833]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:01.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:01 smithi139 ceph-mon[32833]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T00:12:01.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:01 smithi139 ceph-mon[32833]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T00:12:01.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:01 smithi153 ceph-mon[32896]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:01.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:01 smithi153 ceph-mon[32896]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T00:12:01.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:01 smithi153 ceph-mon[32896]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T00:12:01.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:01 smithi027 ceph-mon[29208]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:01.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:01 smithi027 ceph-mon[29208]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T00:12:01.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:01 smithi027 ceph-mon[29208]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T00:12:02.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:02 smithi153 ceph-mon[32896]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:12:02.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:02 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:12:02.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:02 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:12:02.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:02 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:02.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:02 smithi153 ceph-mon[32896]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T00:12:02.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:02 smithi027 ceph-mon[29208]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:12:02.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:02 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:12:02.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:02 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:12:02.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:02 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:02.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:02 smithi027 ceph-mon[29208]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T00:12:02.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:02 smithi139 ceph-mon[32833]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:12:02.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:02 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T00:12:02.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:02 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T00:12:02.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:02 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:02.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:02 smithi139 ceph-mon[32833]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T00:12:03.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:03 smithi153 ceph-mon[32896]: pgmap v242: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail; 162 KiB/s, 0 objects/s recovering 2024-04-25T00:12:03.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:03 smithi139 ceph-mon[32833]: pgmap v242: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail; 162 KiB/s, 0 objects/s recovering 2024-04-25T00:12:03.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:03 smithi027 ceph-mon[29208]: pgmap v242: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail; 162 KiB/s, 0 objects/s recovering 2024-04-25T00:12:05.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:05 smithi153 ceph-mon[32896]: pgmap v243: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-25T00:12:05.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:05 smithi153 ceph-mon[32896]: from='client.? 172.21.15.153:0/3920810512' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a15ea59-44bf-4e78-b60a-8cc5944ec151"}]: dispatch 2024-04-25T00:12:05.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:05 smithi153 ceph-mon[32896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a15ea59-44bf-4e78-b60a-8cc5944ec151"}]: dispatch 2024-04-25T00:12:05.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:05 smithi153 ceph-mon[32896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a15ea59-44bf-4e78-b60a-8cc5944ec151"}]': finished 2024-04-25T00:12:05.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:05 smithi153 ceph-mon[32896]: osdmap e77: 12 total, 11 up, 12 in 2024-04-25T00:12:05.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:05 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:05.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:05 smithi153 ceph-mon[32896]: from='client.? 172.21.15.153:0/2194310694' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:12:05.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:05 smithi027 ceph-mon[29208]: pgmap v243: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-25T00:12:05.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:05 smithi027 ceph-mon[29208]: from='client.? 172.21.15.153:0/3920810512' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a15ea59-44bf-4e78-b60a-8cc5944ec151"}]: dispatch 2024-04-25T00:12:05.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:05 smithi027 ceph-mon[29208]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a15ea59-44bf-4e78-b60a-8cc5944ec151"}]: dispatch 2024-04-25T00:12:05.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:05 smithi027 ceph-mon[29208]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a15ea59-44bf-4e78-b60a-8cc5944ec151"}]': finished 2024-04-25T00:12:05.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:05 smithi027 ceph-mon[29208]: osdmap e77: 12 total, 11 up, 12 in 2024-04-25T00:12:05.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:05 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:05.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:05 smithi027 ceph-mon[29208]: from='client.? 172.21.15.153:0/2194310694' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:12:05.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:05 smithi139 ceph-mon[32833]: pgmap v243: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-25T00:12:05.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:05 smithi139 ceph-mon[32833]: from='client.? 172.21.15.153:0/3920810512' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a15ea59-44bf-4e78-b60a-8cc5944ec151"}]: dispatch 2024-04-25T00:12:05.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:05 smithi139 ceph-mon[32833]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a15ea59-44bf-4e78-b60a-8cc5944ec151"}]: dispatch 2024-04-25T00:12:05.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:05 smithi139 ceph-mon[32833]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a15ea59-44bf-4e78-b60a-8cc5944ec151"}]': finished 2024-04-25T00:12:05.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:05 smithi139 ceph-mon[32833]: osdmap e77: 12 total, 11 up, 12 in 2024-04-25T00:12:05.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:05 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:05.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:05 smithi139 ceph-mon[32833]: from='client.? 172.21.15.153:0/2194310694' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T00:12:07.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:07 smithi153 ceph-mon[32896]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-25T00:12:07.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:07 smithi027 ceph-mon[29208]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-25T00:12:07.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:07 smithi139 ceph-mon[32833]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-25T00:12:09.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:09 smithi153 ceph-mon[32896]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:09.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:09 smithi027 ceph-mon[29208]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:09.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:09 smithi139 ceph-mon[32833]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:11.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:11 smithi153 ceph-mon[32896]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:11.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:11 smithi027 ceph-mon[29208]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:11.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:11 smithi139 ceph-mon[32833]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:12.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:12 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T00:12:12.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:12 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:12.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:12 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T00:12:12.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:12 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:12.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:12 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T00:12:12.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:12 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:13.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:13 smithi153 ceph-mon[32896]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:13.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:13 smithi153 ceph-mon[32896]: Deploying daemon osd.11 on smithi153 2024-04-25T00:12:13.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:13 smithi027 ceph-mon[29208]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:13.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:13 smithi027 ceph-mon[29208]: Deploying daemon osd.11 on smithi153 2024-04-25T00:12:13.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:13 smithi139 ceph-mon[32833]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:13.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:13 smithi139 ceph-mon[32833]: Deploying daemon osd.11 on smithi153 2024-04-25T00:12:15.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:15 smithi153 ceph-mon[32896]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:15.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:15 smithi027 ceph-mon[29208]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:15.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:15 smithi139 ceph-mon[32833]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:17.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:17 smithi153 ceph-mon[32896]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:17.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:17 smithi027 ceph-mon[29208]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:17.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:17 smithi139 ceph-mon[32833]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:19.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:19 smithi027 ceph-mon[29208]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:19.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:19 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:19.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:19 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:19.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:19 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:12:19.490 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:19 smithi153 ceph-mon[32896]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:19.491 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:19 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:19.491 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:19 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:19.491 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:19 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:12:19.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:19 smithi139 ceph-mon[32833]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:19.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:19 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:19.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:19 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:19.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:19 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:12:21.305 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:21 smithi139 ceph-mon[32833]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:21.306 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:21 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:21.306 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:21 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:21.306 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:21 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:21.306 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:21 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:12:21.306 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:21 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:21.337 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:21 smithi153 ceph-mon[32896]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:21.337 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:21 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:21.337 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:21 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:21.337 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:21 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:21.337 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:21 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:12:21.337 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:21 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:21.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:21 smithi027 ceph-mon[29208]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:21.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:21 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:21.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:21 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:21.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:21 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:21.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:21 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:12:21.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:21 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:21.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:21.976+0000 7f3c6b7fe700 1 -- 172.21.15.153:0/1338777062 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f3c7c1a0e60 con 0x7f3c640a2ca0 2024-04-25T00:12:21.976 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 11 on host 'smithi153' 2024-04-25T00:12:21.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:21.980+0000 7f3c80ed1700 1 -- 172.21.15.153:0/1338777062 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3c640a2ca0 msgr2=0x7f3c640a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:21.980+0000 7f3c80ed1700 1 --2- 172.21.15.153:0/1338777062 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3c640a2ca0 0x7f3c640a5160 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c003890 tx=0x7f3c6c005fd0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:21.980+0000 7f3c80ed1700 1 -- 172.21.15.153:0/1338777062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c7c102380 msgr2=0x7f3c7c198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:21.980+0000 7f3c80ed1700 1 --2- 172.21.15.153:0/1338777062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c7c102380 0x7f3c7c198bd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000b560 tx=0x7f3c7000b870 comp rx=0 tx=0).stop 2024-04-25T00:12:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:21.980+0000 7f3c80ed1700 1 -- 172.21.15.153:0/1338777062 shutdown_connections 2024-04-25T00:12:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:21.980+0000 7f3c80ed1700 1 --2- 172.21.15.153:0/1338777062 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3c640a2ca0 0x7f3c640a5160 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:21.980+0000 7f3c80ed1700 1 --2- 172.21.15.153:0/1338777062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c7c102d20 0x7f3c7c19fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:21.980+0000 7f3c80ed1700 1 --2- 172.21.15.153:0/1338777062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7c101180 0x7f3c7c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:21.980+0000 7f3c80ed1700 1 --2- 172.21.15.153:0/1338777062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c7c102380 0x7f3c7c198bd0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:21.980+0000 7f3c80ed1700 1 -- 172.21.15.153:0/1338777062 >> 172.21.15.153:0/1338777062 conn(0x7f3c7c0fc720 msgr2=0x7f3c7c06a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:21.980+0000 7f3c80ed1700 1 -- 172.21.15.153:0/1338777062 shutdown_connections 2024-04-25T00:12:21.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:21.980+0000 7f3c80ed1700 1 -- 172.21.15.153:0/1338777062 wait complete. 2024-04-25T00:12:22.726 DEBUG:teuthology.orchestra.run.smithi153:osd.11> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.11.service 2024-04-25T00:12:22.728 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-25T00:12:22.728 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T00:12:22.999 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:12:23.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:22 smithi153 ceph-mon[32896]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:23.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:22 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:23.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:22 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:23.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:22 smithi153 ceph-mon[32896]: from='osd.11 [v2:172.21.15.153:6824/2357170710,v1:172.21.15.153:6825/2357170710]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T00:12:23.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:22 smithi153 ceph-mon[32896]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T00:12:23.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:22 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:12:23.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:22 smithi027 ceph-mon[29208]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:23.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:23.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:23.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:22 smithi027 ceph-mon[29208]: from='osd.11 [v2:172.21.15.153:6824/2357170710,v1:172.21.15.153:6825/2357170710]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T00:12:23.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:22 smithi027 ceph-mon[29208]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T00:12:23.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:22 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:12:23.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:22 smithi139 ceph-mon[32833]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:23.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:22 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:23.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:22 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:23.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:22 smithi139 ceph-mon[32833]: from='osd.11 [v2:172.21.15.153:6824/2357170710,v1:172.21.15.153:6825/2357170710]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T00:12:23.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:22 smithi139 ceph-mon[32833]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T00:12:23.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:22 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:12:24.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:23 smithi139 ceph-mon[32833]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T00:12:24.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:23 smithi139 ceph-mon[32833]: osdmap e78: 12 total, 11 up, 12 in 2024-04-25T00:12:24.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:23 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:24.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:23 smithi139 ceph-mon[32833]: from='osd.11 [v2:172.21.15.153:6824/2357170710,v1:172.21.15.153:6825/2357170710]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:12:24.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:23 smithi139 ceph-mon[32833]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:12:24.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:23 smithi153 ceph-mon[32896]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T00:12:24.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:23 smithi153 ceph-mon[32896]: osdmap e78: 12 total, 11 up, 12 in 2024-04-25T00:12:24.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:23 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:24.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:23 smithi153 ceph-mon[32896]: from='osd.11 [v2:172.21.15.153:6824/2357170710,v1:172.21.15.153:6825/2357170710]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:12:24.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:23 smithi153 ceph-mon[32896]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:12:24.331 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:23 smithi027 ceph-mon[29208]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T00:12:24.331 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:23 smithi027 ceph-mon[29208]: osdmap e78: 12 total, 11 up, 12 in 2024-04-25T00:12:24.331 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:23 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:24.331 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:23 smithi027 ceph-mon[29208]: from='osd.11 [v2:172.21.15.153:6824/2357170710,v1:172.21.15.153:6825/2357170710]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:12:24.331 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:23 smithi027 ceph-mon[29208]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T00:12:24.527 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:12:24 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:12:24.411+0000 7fc094896700 -1 osd.11 0 waiting for initial osdmap 2024-04-25T00:12:24.527 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:12:24 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:12:24.428+0000 7fc08c681700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:12:24.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.884+0000 7fed6af28700 1 -- 172.21.15.27:0/2457639369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed64102d70 msgr2=0x7fed64105160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:24.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.884+0000 7fed6af28700 1 --2- 172.21.15.27:0/2457639369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed64102d70 0x7fed64105160 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fed58009990 tx=0x7fed58009ca0 comp rx=0 tx=0).stop 2024-04-25T00:12:24.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.884+0000 7fed6af28700 1 -- 172.21.15.27:0/2457639369 shutdown_connections 2024-04-25T00:12:24.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.884+0000 7fed6af28700 1 --2- 172.21.15.27:0/2457639369 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed64107fd0 0x7fed6410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:24.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.884+0000 7fed6af28700 1 --2- 172.21.15.27:0/2457639369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed64102d70 0x7fed64105160 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:24.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.884+0000 7fed6af28700 1 --2- 172.21.15.27:0/2457639369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed641056a0 0x7fed64107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:24.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.884+0000 7fed6af28700 1 -- 172.21.15.27:0/2457639369 >> 172.21.15.27:0/2457639369 conn(0x7fed640fc720 msgr2=0x7fed640feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:24.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.884+0000 7fed6af28700 1 -- 172.21.15.27:0/2457639369 shutdown_connections 2024-04-25T00:12:24.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.884+0000 7fed6af28700 1 -- 172.21.15.27:0/2457639369 wait complete. 2024-04-25T00:12:24.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed6af28700 1 Processor -- start 2024-04-25T00:12:24.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed6af28700 1 -- start start 2024-04-25T00:12:24.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed6af28700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed64102d70 0x7fed641026d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:24.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed6af28700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed641056a0 0x7fed64100d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:24.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed6af28700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed64107fd0 0x7fed64101260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:24.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed6af28700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fed6410de90 con 0x7fed64107fd0 2024-04-25T00:12:24.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed6af28700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fed6410dd10 con 0x7fed64102d70 2024-04-25T00:12:24.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed6af28700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fed6410e010 con 0x7fed641056a0 2024-04-25T00:12:24.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed694c5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed64107fd0 0x7fed64101260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:24.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed63fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed641056a0 0x7fed64100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:24.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed694c5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed64107fd0 0x7fed64101260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55362/0 (socket says 172.21.15.27:55362) 2024-04-25T00:12:24.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed694c5700 1 -- 172.21.15.27:0/4119653871 learned_addr learned my addr 172.21.15.27:0/4119653871 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:12:24.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed68cc4700 1 --2- 172.21.15.27:0/4119653871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed64102d70 0x7fed641026d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:24.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed694c5700 1 -- 172.21.15.27:0/4119653871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed641056a0 msgr2=0x7fed64100d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:24.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed694c5700 1 --2- 172.21.15.27:0/4119653871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed641056a0 0x7fed64100d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:24.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed694c5700 1 -- 172.21.15.27:0/4119653871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed64102d70 msgr2=0x7fed641026d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:24.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed694c5700 1 --2- 172.21.15.27:0/4119653871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed64102d70 0x7fed641026d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:24.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed694c5700 1 -- 172.21.15.27:0/4119653871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed64101b10 con 0x7fed64107fd0 2024-04-25T00:12:24.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.885+0000 7fed694c5700 1 --2- 172.21.15.27:0/4119653871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed64107fd0 0x7fed64101260 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0094f0 tx=0x7fed5c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:24.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.886+0000 7fed61ffb700 1 -- 172.21.15.27:0/4119653871 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5c004db0 con 0x7fed64107fd0 2024-04-25T00:12:24.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.886+0000 7fed61ffb700 1 -- 172.21.15.27:0/4119653871 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed5c009e20 con 0x7fed64107fd0 2024-04-25T00:12:24.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.886+0000 7fed6af28700 1 -- 172.21.15.27:0/4119653871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed641a2640 con 0x7fed64107fd0 2024-04-25T00:12:24.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.886+0000 7fed61ffb700 1 -- 172.21.15.27:0/4119653871 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5c015400 con 0x7fed64107fd0 2024-04-25T00:12:24.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.886+0000 7fed6af28700 1 -- 172.21.15.27:0/4119653871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed641a2b50 con 0x7fed64107fd0 2024-04-25T00:12:24.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.887+0000 7fed6af28700 1 -- 172.21.15.27:0/4119653871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed64068f20 con 0x7fed64107fd0 2024-04-25T00:12:24.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.888+0000 7fed61ffb700 1 -- 172.21.15.27:0/4119653871 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fed5c00db20 con 0x7fed64107fd0 2024-04-25T00:12:24.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.888+0000 7fed61ffb700 1 --2- 172.21.15.27:0/4119653871 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fed540a2ca0 0x7fed540a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:24.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.889+0000 7fed68cc4700 1 --2- 172.21.15.27:0/4119653871 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fed540a2ca0 0x7fed540a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:24.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.889+0000 7fed61ffb700 1 -- 172.21.15.27:0/4119653871 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7fed5c0ce420 con 0x7fed64107fd0 2024-04-25T00:12:24.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.889+0000 7fed68cc4700 1 --2- 172.21.15.27:0/4119653871 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fed540a2ca0 0x7fed540a5160 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fed5800b2e0 tx=0x7fed58005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:24.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:24.893+0000 7fed61ffb700 1 -- 172.21.15.27:0/4119653871 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fed5c010070 con 0x7fed64107fd0 2024-04-25T00:12:25.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:25.103+0000 7fed6af28700 1 -- 172.21.15.27:0/4119653871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fed64062e40 con 0x7fed64107fd0 2024-04-25T00:12:25.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:25.103+0000 7fed61ffb700 1 -- 172.21.15.27:0/4119653871 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) ==== 74+0+133 (secure 0 0 0) 0x7fed5c09b940 con 0x7fed64107fd0 2024-04-25T00:12:25.103 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:12:25.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:25.105+0000 7fed6af28700 1 -- 172.21.15.27:0/4119653871 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fed540a2ca0 msgr2=0x7fed540a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:25.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:25.105+0000 7fed6af28700 1 --2- 172.21.15.27:0/4119653871 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fed540a2ca0 0x7fed540a5160 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fed5800b2e0 tx=0x7fed58005fd0 comp rx=0 tx=0).stop 2024-04-25T00:12:25.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:25.105+0000 7fed6af28700 1 -- 172.21.15.27:0/4119653871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed64107fd0 msgr2=0x7fed64101260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:25.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:25.105+0000 7fed6af28700 1 --2- 172.21.15.27:0/4119653871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed64107fd0 0x7fed64101260 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0094f0 tx=0x7fed5c00d3b0 comp rx=0 tx=0).stop 2024-04-25T00:12:25.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:25.105+0000 7fed6af28700 1 -- 172.21.15.27:0/4119653871 shutdown_connections 2024-04-25T00:12:25.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:25.105+0000 7fed6af28700 1 --2- 172.21.15.27:0/4119653871 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fed540a2ca0 0x7fed540a5160 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:25.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:25.105+0000 7fed6af28700 1 --2- 172.21.15.27:0/4119653871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed64107fd0 0x7fed64101260 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:25.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:25.105+0000 7fed6af28700 1 --2- 172.21.15.27:0/4119653871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed64102d70 0x7fed641026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:25.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:25.105+0000 7fed6af28700 1 --2- 172.21.15.27:0/4119653871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed641056a0 0x7fed64100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:25.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:25.105+0000 7fed6af28700 1 -- 172.21.15.27:0/4119653871 >> 172.21.15.27:0/4119653871 conn(0x7fed640fc720 msgr2=0x7fed641037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:25.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:25.105+0000 7fed6af28700 1 -- 172.21.15.27:0/4119653871 shutdown_connections 2024-04-25T00:12:25.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:25.105+0000 7fed6af28700 1 -- 172.21.15.27:0/4119653871 wait complete. 2024-04-25T00:12:25.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:24 smithi027 ceph-mon[29208]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:25.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:24 smithi027 ceph-mon[29208]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T00:12:25.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:24 smithi027 ceph-mon[29208]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T00:12:25.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:24 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:25.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:24 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:25.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:24 smithi027 ceph-mon[29208]: from='osd.11 ' entity='osd.11' 2024-04-25T00:12:25.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:24 smithi139 ceph-mon[32833]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:25.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:24 smithi139 ceph-mon[32833]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T00:12:25.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:24 smithi139 ceph-mon[32833]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T00:12:25.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:24 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:25.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:24 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:25.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:24 smithi139 ceph-mon[32833]: from='osd.11 ' entity='osd.11' 2024-04-25T00:12:25.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:24 smithi153 ceph-mon[32896]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:25.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:24 smithi153 ceph-mon[32896]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T00:12:25.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:24 smithi153 ceph-mon[32896]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T00:12:25.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:24 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:25.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:24 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:25.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:24 smithi153 ceph-mon[32896]: from='osd.11 ' entity='osd.11' 2024-04-25T00:12:25.792 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1714003917,"num_in_osds":12,"osd_in_since":1714003924,"num_remapped_pgs":0} 2024-04-25T00:12:26.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:25 smithi027 ceph-mon[29208]: purged_snaps scrub starts 2024-04-25T00:12:26.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:25 smithi027 ceph-mon[29208]: purged_snaps scrub ok 2024-04-25T00:12:26.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:26.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:25 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4119653871' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T00:12:26.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:25 smithi027 ceph-mon[29208]: osd.11 [v2:172.21.15.153:6824/2357170710,v1:172.21.15.153:6825/2357170710] boot 2024-04-25T00:12:26.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:25 smithi027 ceph-mon[29208]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T00:12:26.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:25 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:25 smithi139 ceph-mon[32833]: purged_snaps scrub starts 2024-04-25T00:12:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:25 smithi139 ceph-mon[32833]: purged_snaps scrub ok 2024-04-25T00:12:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:25 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:25 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4119653871' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T00:12:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:25 smithi139 ceph-mon[32833]: osd.11 [v2:172.21.15.153:6824/2357170710,v1:172.21.15.153:6825/2357170710] boot 2024-04-25T00:12:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:25 smithi139 ceph-mon[32833]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T00:12:26.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:25 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:26.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:25 smithi153 ceph-mon[32896]: purged_snaps scrub starts 2024-04-25T00:12:26.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:25 smithi153 ceph-mon[32896]: purged_snaps scrub ok 2024-04-25T00:12:26.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:25 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:26.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:25 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4119653871' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T00:12:26.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:25 smithi153 ceph-mon[32896]: osd.11 [v2:172.21.15.153:6824/2357170710,v1:172.21.15.153:6825/2357170710] boot 2024-04-25T00:12:26.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:25 smithi153 ceph-mon[32896]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T00:12:26.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:25 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:12:26.794 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T00:12:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:12:27.165 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:26 smithi027 ceph-mon[29208]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:27.166 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:26 smithi027 ceph-mon[29208]: Detected new or changed devices on smithi153 2024-04-25T00:12:27.166 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:26 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:27.166 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:26 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:27.166 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:26 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:12:27.166 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:26 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:12:27.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:26 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:12:27.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:26 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:12:27.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:26 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi153 to 4272M 2024-04-25T00:12:27.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:26 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:27.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:26 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:27.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:26 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:12:27.168 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:26 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:27.168 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:26 smithi027 ceph-mon[29208]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T00:12:27.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:26 smithi139 ceph-mon[32833]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:27.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:26 smithi139 ceph-mon[32833]: Detected new or changed devices on smithi153 2024-04-25T00:12:27.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:26 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:27.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:26 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:27.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:26 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:12:27.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:26 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:12:27.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:26 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:12:27.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:26 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:12:27.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:26 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi153 to 4272M 2024-04-25T00:12:27.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:26 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:27.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:26 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:27.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:26 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:12:27.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:26 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:27.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:26 smithi139 ceph-mon[32833]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T00:12:27.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:26 smithi153 ceph-mon[32896]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-04-25T00:12:27.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:26 smithi153 ceph-mon[32896]: Detected new or changed devices on smithi153 2024-04-25T00:12:27.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:26 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:27.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:26 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:27.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:26 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:12:27.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:26 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:12:27.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:26 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:12:27.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:26 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:12:27.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:26 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi153 to 4272M 2024-04-25T00:12:27.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:26 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:27.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:26 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:27.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:26 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:12:27.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:26 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:27.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:26 smithi153 ceph-mon[32896]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T00:12:28.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.807+0000 7f3ccbfff700 1 -- 172.21.15.27:0/3306104002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ccc102c50 msgr2=0x7f3ccc109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:28.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.807+0000 7f3ccbfff700 1 --2- 172.21.15.27:0/3306104002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ccc102c50 0x7f3ccc109500 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc0099c0 tx=0x7f3cbc009cd0 comp rx=0 tx=0).stop 2024-04-25T00:12:28.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.807+0000 7f3ccbfff700 1 -- 172.21.15.27:0/3306104002 shutdown_connections 2024-04-25T00:12:28.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.807+0000 7f3ccbfff700 1 --2- 172.21.15.27:0/3306104002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ccc102c50 0x7f3ccc109500 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:28.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.807+0000 7f3ccbfff700 1 --2- 172.21.15.27:0/3306104002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ccc1010b0 0x7f3ccc1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:28.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.807+0000 7f3ccbfff700 1 --2- 172.21.15.27:0/3306104002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ccc1022b0 0x7f3ccc102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:28.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.807+0000 7f3ccbfff700 1 -- 172.21.15.27:0/3306104002 >> 172.21.15.27:0/3306104002 conn(0x7f3ccc0fc650 msgr2=0x7f3ccc0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:28.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.807+0000 7f3ccbfff700 1 -- 172.21.15.27:0/3306104002 shutdown_connections 2024-04-25T00:12:28.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.807+0000 7f3ccbfff700 1 -- 172.21.15.27:0/3306104002 wait complete. 2024-04-25T00:12:28.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.807+0000 7f3ccbfff700 1 Processor -- start 2024-04-25T00:12:28.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.808+0000 7f3ccbfff700 1 -- start start 2024-04-25T00:12:28.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.808+0000 7f3ccbfff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ccc1010b0 0x7f3ccc105040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:28.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.808+0000 7f3ccbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ccc1022b0 0x7f3ccc103690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:28.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.808+0000 7f3ccbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ccc102c50 0x7f3ccc103bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:28.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.808+0000 7f3ccbfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3ccc10cd30 con 0x7f3ccc102c50 2024-04-25T00:12:28.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.808+0000 7f3ccbfff700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3ccc10cbb0 con 0x7f3ccc1010b0 2024-04-25T00:12:28.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.808+0000 7f3ccbfff700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3ccc10ceb0 con 0x7f3ccc1022b0 2024-04-25T00:12:28.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.808+0000 7f3ccb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ccc102c50 0x7f3ccc103bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:28.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.808+0000 7f3ccb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ccc102c50 0x7f3ccc103bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50240/0 (socket says 172.21.15.27:50240) 2024-04-25T00:12:28.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.808+0000 7f3ccb7fe700 1 -- 172.21.15.27:0/3457350663 learned_addr learned my addr 172.21.15.27:0/3457350663 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:12:28.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.808+0000 7f3ccaffd700 1 --2- 172.21.15.27:0/3457350663 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ccc1010b0 0x7f3ccc105040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:28.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.808+0000 7f3cca7fc700 1 --2- 172.21.15.27:0/3457350663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ccc1022b0 0x7f3ccc103690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:28.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.808+0000 7f3ccb7fe700 1 -- 172.21.15.27:0/3457350663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ccc1022b0 msgr2=0x7f3ccc103690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:28.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.808+0000 7f3ccb7fe700 1 --2- 172.21.15.27:0/3457350663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ccc1022b0 0x7f3ccc103690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:28.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.808+0000 7f3ccb7fe700 1 -- 172.21.15.27:0/3457350663 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ccc1010b0 msgr2=0x7f3ccc105040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:28.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.808+0000 7f3ccb7fe700 1 --2- 172.21.15.27:0/3457350663 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ccc1010b0 0x7f3ccc105040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:28.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.808+0000 7f3ccb7fe700 1 -- 172.21.15.27:0/3457350663 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ccc104480 con 0x7f3ccc102c50 2024-04-25T00:12:28.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.809+0000 7f3ccb7fe700 1 --2- 172.21.15.27:0/3457350663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ccc102c50 0x7f3ccc103bd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc0039c0 tx=0x7f3cbc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:28.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.809+0000 7f3cd09fe700 1 -- 172.21.15.27:0/3457350663 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc027070 con 0x7f3ccc102c50 2024-04-25T00:12:28.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.809+0000 7f3ccbfff700 1 -- 172.21.15.27:0/3457350663 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ccc1a24d0 con 0x7f3ccc102c50 2024-04-25T00:12:28.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.809+0000 7f3cd09fe700 1 -- 172.21.15.27:0/3457350663 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cbc004150 con 0x7f3ccc102c50 2024-04-25T00:12:28.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.809+0000 7f3cd09fe700 1 -- 172.21.15.27:0/3457350663 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc00f460 con 0x7f3ccc102c50 2024-04-25T00:12:28.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.809+0000 7f3ccbfff700 1 -- 172.21.15.27:0/3457350663 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ccc1a2950 con 0x7f3ccc102c50 2024-04-25T00:12:28.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.810+0000 7f3ccbfff700 1 -- 172.21.15.27:0/3457350663 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ccc068f20 con 0x7f3ccc102c50 2024-04-25T00:12:28.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.811+0000 7f3cd09fe700 1 -- 172.21.15.27:0/3457350663 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3cbc02a070 con 0x7f3ccc102c50 2024-04-25T00:12:28.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.812+0000 7f3cd09fe700 1 --2- 172.21.15.27:0/3457350663 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3cb40a2c20 0x7f3cb40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:28.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.812+0000 7f3ccaffd700 1 --2- 172.21.15.27:0/3457350663 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3cb40a2c20 0x7f3cb40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:28.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.812+0000 7f3cd09fe700 1 -- 172.21.15.27:0/3457350663 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3cbc0d0310 con 0x7f3ccc102c50 2024-04-25T00:12:28.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.812+0000 7f3ccaffd700 1 --2- 172.21.15.27:0/3457350663 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3cb40a2c20 0x7f3cb40a50e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc102110 tx=0x7f3cc40094f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:28.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:28.816+0000 7f3cd09fe700 1 -- 172.21.15.27:0/3457350663 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3cbc031070 con 0x7f3ccc102c50 2024-04-25T00:12:29.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:29.022+0000 7f3ccbfff700 1 -- 172.21.15.27:0/3457350663 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f3ccc1a2d70 con 0x7f3ccc102c50 2024-04-25T00:12:29.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:29.023+0000 7f3cd09fe700 1 -- 172.21.15.27:0/3457350663 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f3cbc013070 con 0x7f3ccc102c50 2024-04-25T00:12:29.023 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:12:29.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:29.025+0000 7f3ccbfff700 1 -- 172.21.15.27:0/3457350663 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3cb40a2c20 msgr2=0x7f3cb40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:29.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:29.025+0000 7f3ccbfff700 1 --2- 172.21.15.27:0/3457350663 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3cb40a2c20 0x7f3cb40a50e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc102110 tx=0x7f3cc40094f0 comp rx=0 tx=0).stop 2024-04-25T00:12:29.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:29.025+0000 7f3ccbfff700 1 -- 172.21.15.27:0/3457350663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ccc102c50 msgr2=0x7f3ccc103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:29.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:29.025+0000 7f3ccbfff700 1 --2- 172.21.15.27:0/3457350663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ccc102c50 0x7f3ccc103bd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc0039c0 tx=0x7f3cbc0047b0 comp rx=0 tx=0).stop 2024-04-25T00:12:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:29.025+0000 7f3ccbfff700 1 -- 172.21.15.27:0/3457350663 shutdown_connections 2024-04-25T00:12:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:29.025+0000 7f3ccbfff700 1 --2- 172.21.15.27:0/3457350663 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3cb40a2c20 0x7f3cb40a50e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:29.025+0000 7f3ccbfff700 1 --2- 172.21.15.27:0/3457350663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ccc102c50 0x7f3ccc103bd0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:29.025+0000 7f3ccbfff700 1 --2- 172.21.15.27:0/3457350663 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ccc1010b0 0x7f3ccc105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:29.025+0000 7f3ccbfff700 1 --2- 172.21.15.27:0/3457350663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ccc1022b0 0x7f3ccc103690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:29.025+0000 7f3ccbfff700 1 -- 172.21.15.27:0/3457350663 >> 172.21.15.27:0/3457350663 conn(0x7f3ccc0fc650 msgr2=0x7f3ccc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:29.025+0000 7f3ccbfff700 1 -- 172.21.15.27:0/3457350663 shutdown_connections 2024-04-25T00:12:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:29.025+0000 7f3ccbfff700 1 -- 172.21.15.27:0/3457350663 wait complete. 2024-04-25T00:12:29.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:28 smithi027 ceph-mon[29208]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:29.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:28 smithi139 ceph-mon[32833]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:29.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:28 smithi153 ceph-mon[32896]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:29.795 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1714003945,"num_in_osds":12,"osd_in_since":1714003924,"num_remapped_pgs":0} 2024-04-25T00:12:29.796 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T00:12:30.045 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:12:30.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:29 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3457350663' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T00:12:30.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:29 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3457350663' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T00:12:30.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:29 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3457350663' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T00:12:31.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:30 smithi027 ceph-mon[29208]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:31.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:30 smithi139 ceph-mon[32833]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:31.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:30 smithi153 ceph-mon[32896]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:31.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.796+0000 7fd2e36d5700 1 -- 172.21.15.27:0/2591922967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2dc102e90 msgr2=0x7fd2dc106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:31.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.796+0000 7fd2e36d5700 1 --2- 172.21.15.27:0/2591922967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2dc102e90 0x7fd2dc106e90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0009990 tx=0x7fd2d0009ca0 comp rx=0 tx=0).stop 2024-04-25T00:12:31.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.796+0000 7fd2e36d5700 1 -- 172.21.15.27:0/2591922967 shutdown_connections 2024-04-25T00:12:31.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.796+0000 7fd2e36d5700 1 --2- 172.21.15.27:0/2591922967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2dc107570 0x7fd2dc10f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:31.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.796+0000 7fd2e36d5700 1 --2- 172.21.15.27:0/2591922967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2dc1024e0 0x7fd2dc1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:31.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.796+0000 7fd2e36d5700 1 --2- 172.21.15.27:0/2591922967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2dc102e90 0x7fd2dc106e90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:31.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.796+0000 7fd2e36d5700 1 -- 172.21.15.27:0/2591922967 >> 172.21.15.27:0/2591922967 conn(0x7fd2dc079390 msgr2=0x7fd2dc0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:31.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.797+0000 7fd2e36d5700 1 -- 172.21.15.27:0/2591922967 shutdown_connections 2024-04-25T00:12:31.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.797+0000 7fd2e36d5700 1 -- 172.21.15.27:0/2591922967 wait complete. 2024-04-25T00:12:31.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.797+0000 7fd2e36d5700 1 Processor -- start 2024-04-25T00:12:31.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.797+0000 7fd2e36d5700 1 -- start start 2024-04-25T00:12:31.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.797+0000 7fd2e36d5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2dc1024e0 0x7fd2dc19e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:31.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.797+0000 7fd2e36d5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2dc102e90 0x7fd2dc19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:31.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.797+0000 7fd2e36d5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2dc107570 0x7fd2dc1988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:31.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2e36d5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd2dc112610 con 0x7fd2dc102e90 2024-04-25T00:12:31.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2e36d5700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd2dc112490 con 0x7fd2dc107570 2024-04-25T00:12:31.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2e36d5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd2dc112790 con 0x7fd2dc1024e0 2024-04-25T00:12:31.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2e0c70700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2dc102e90 0x7fd2dc19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:31.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2e0c70700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2dc102e90 0x7fd2dc19ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50252/0 (socket says 172.21.15.27:50252) 2024-04-25T00:12:31.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2e0c70700 1 -- 172.21.15.27:0/3698470412 learned_addr learned my addr 172.21.15.27:0/3698470412 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:12:31.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2e1471700 1 --2- 172.21.15.27:0/3698470412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2dc1024e0 0x7fd2dc19e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:31.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2e1c72700 1 --2- 172.21.15.27:0/3698470412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2dc107570 0x7fd2dc1988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:31.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2e0c70700 1 -- 172.21.15.27:0/3698470412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2dc1024e0 msgr2=0x7fd2dc19e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:31.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2e0c70700 1 --2- 172.21.15.27:0/3698470412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2dc1024e0 0x7fd2dc19e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:31.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2e0c70700 1 -- 172.21.15.27:0/3698470412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2dc107570 msgr2=0x7fd2dc1988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:31.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2e0c70700 1 --2- 172.21.15.27:0/3698470412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2dc107570 0x7fd2dc1988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:31.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2e0c70700 1 -- 172.21.15.27:0/3698470412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2dc199050 con 0x7fd2dc102e90 2024-04-25T00:12:31.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2e0c70700 1 --2- 172.21.15.27:0/3698470412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2dc102e90 0x7fd2dc19ec70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0004580 tx=0x7fd2d00045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:31.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2ce7fc700 1 -- 172.21.15.27:0/3698470412 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0026070 con 0x7fd2dc102e90 2024-04-25T00:12:31.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2ce7fc700 1 -- 172.21.15.27:0/3698470412 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2d0004ce0 con 0x7fd2dc102e90 2024-04-25T00:12:31.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2ce7fc700 1 -- 172.21.15.27:0/3698470412 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0005320 con 0x7fd2dc102e90 2024-04-25T00:12:31.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.798+0000 7fd2e36d5700 1 -- 172.21.15.27:0/3698470412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2dc1992e0 con 0x7fd2dc102e90 2024-04-25T00:12:31.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.799+0000 7fd2e36d5700 1 -- 172.21.15.27:0/3698470412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2dc1a5600 con 0x7fd2dc102e90 2024-04-25T00:12:31.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.799+0000 7fd2e36d5700 1 -- 172.21.15.27:0/3698470412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2dc103970 con 0x7fd2dc102e90 2024-04-25T00:12:31.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.801+0000 7fd2ce7fc700 1 -- 172.21.15.27:0/3698470412 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd2d0029070 con 0x7fd2dc102e90 2024-04-25T00:12:31.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.802+0000 7fd2ce7fc700 1 --2- 172.21.15.27:0/3698470412 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd2c80a2ca0 0x7fd2c80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:31.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.802+0000 7fd2ce7fc700 1 -- 172.21.15.27:0/3698470412 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd2d00cfa50 con 0x7fd2dc102e90 2024-04-25T00:12:31.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.802+0000 7fd2e1471700 1 --2- 172.21.15.27:0/3698470412 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd2c80a2ca0 0x7fd2c80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:31.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.802+0000 7fd2e1471700 1 --2- 172.21.15.27:0/3698470412 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd2c80a2ca0 0x7fd2c80a5160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc199a30 tx=0x7fd2d8009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:31.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:31.805+0000 7fd2ce7fc700 1 -- 172.21.15.27:0/3698470412 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd2d0030020 con 0x7fd2dc102e90 2024-04-25T00:12:32.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:32.021+0000 7fd2e36d5700 1 -- 172.21.15.27:0/3698470412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2dc199e20 con 0x7fd2dc102e90 2024-04-25T00:12:32.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:32.022+0000 7fd2ce7fc700 1 -- 172.21.15.27:0/3698470412 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18445 (secure 0 0 0) 0x7fd2d0013070 con 0x7fd2dc102e90 2024-04-25T00:12:32.026 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:12:32.028 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":81,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:12:26.417275+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:12:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:32.024+0000 7fd2e36d5700 1 -- 172.21.15.27:0/3698470412 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd2c80a2ca0 msgr2=0x7fd2c80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:32.024+0000 7fd2e36d5700 1 --2- 172.21.15.27:0/3698470412 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd2c80a2ca0 0x7fd2c80a5160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc199a30 tx=0x7fd2d8009190 comp rx=0 tx=0).stop 2024-04-25T00:12:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:32.024+0000 7fd2e36d5700 1 -- 172.21.15.27:0/3698470412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2dc102e90 msgr2=0x7fd2dc19ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:32.024+0000 7fd2e36d5700 1 --2- 172.21.15.27:0/3698470412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2dc102e90 0x7fd2dc19ec70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0004580 tx=0x7fd2d00045b0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:32.024+0000 7fd2e36d5700 1 -- 172.21.15.27:0/3698470412 shutdown_connections 2024-04-25T00:12:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:32.024+0000 7fd2e36d5700 1 --2- 172.21.15.27:0/3698470412 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd2c80a2ca0 0x7fd2c80a5160 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:32.024+0000 7fd2e36d5700 1 --2- 172.21.15.27:0/3698470412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2dc107570 0x7fd2dc1988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:32.024+0000 7fd2e36d5700 1 --2- 172.21.15.27:0/3698470412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2dc1024e0 0x7fd2dc19e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:32.024+0000 7fd2e36d5700 1 --2- 172.21.15.27:0/3698470412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2dc102e90 0x7fd2dc19ec70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:32.024+0000 7fd2e36d5700 1 -- 172.21.15.27:0/3698470412 >> 172.21.15.27:0/3698470412 conn(0x7fd2dc079390 msgr2=0x7fd2dc101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:32.024+0000 7fd2e36d5700 1 -- 172.21.15.27:0/3698470412 shutdown_connections 2024-04-25T00:12:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:32.024+0000 7fd2e36d5700 1 -- 172.21.15.27:0/3698470412 wait complete. 2024-04-25T00:12:32.731 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:08:34.006973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T00:12:32.731 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-25T00:12:32.983 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:12:33.256 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:33 smithi139 ceph-mon[32833]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:33.256 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:33 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3698470412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:12:33.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:33 smithi153 ceph-mon[32896]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:33.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:33 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3698470412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:12:33.344 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:33 smithi027 ceph-mon[29208]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:33.344 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:33 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3698470412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:12:34.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.759+0000 7f997ab36700 1 -- 172.21.15.27:0/241045388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99741071e0 msgr2=0x7f99741075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:34.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.759+0000 7f997ab36700 1 --2- 172.21.15.27:0/241045388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99741071e0 0x7f99741075c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9968009990 tx=0x7f9968009ca0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.759+0000 7f997ab36700 1 -- 172.21.15.27:0/241045388 shutdown_connections 2024-04-25T00:12:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.759+0000 7f997ab36700 1 --2- 172.21.15.27:0/241045388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9974101b20 0x7f997410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.759+0000 7f997ab36700 1 --2- 172.21.15.27:0/241045388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99741071e0 0x7f99741075c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.759+0000 7f997ab36700 1 --2- 172.21.15.27:0/241045388 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9974101160 0x7f99741015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.759+0000 7f997ab36700 1 -- 172.21.15.27:0/241045388 >> 172.21.15.27:0/241045388 conn(0x7f99740771a0 msgr2=0x7f99740775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.759+0000 7f997ab36700 1 -- 172.21.15.27:0/241045388 shutdown_connections 2024-04-25T00:12:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.759+0000 7f997ab36700 1 -- 172.21.15.27:0/241045388 wait complete. 2024-04-25T00:12:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.760+0000 7f997ab36700 1 Processor -- start 2024-04-25T00:12:34.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.760+0000 7f997ab36700 1 -- start start 2024-04-25T00:12:34.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.760+0000 7f997ab36700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9974101160 0x7f99741998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:34.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.760+0000 7f997ab36700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9974101b20 0x7f9974199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:34.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.760+0000 7f997ab36700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99741071e0 0x7f99741a4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:34.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.760+0000 7f997ab36700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9974111420 con 0x7f9974101b20 2024-04-25T00:12:34.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.760+0000 7f997ab36700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f99741112a0 con 0x7f9974101160 2024-04-25T00:12:34.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.760+0000 7f997ab36700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f99741115a0 con 0x7f99741071e0 2024-04-25T00:12:34.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.761+0000 7f9973fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9974101b20 0x7f9974199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:34.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.761+0000 7f9973fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9974101b20 0x7f9974199de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50270/0 (socket says 172.21.15.27:50270) 2024-04-25T00:12:34.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.761+0000 7f9973fff700 1 -- 172.21.15.27:0/1041298260 learned_addr learned my addr 172.21.15.27:0/1041298260 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:12:34.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.761+0000 7f99790d3700 1 --2- 172.21.15.27:0/1041298260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99741071e0 0x7f99741a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:34.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.761+0000 7f99788d2700 1 --2- 172.21.15.27:0/1041298260 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9974101160 0x7f99741998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:34.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.761+0000 7f9973fff700 1 -- 172.21.15.27:0/1041298260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99741071e0 msgr2=0x7f99741a4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:34.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.761+0000 7f9973fff700 1 --2- 172.21.15.27:0/1041298260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99741071e0 0x7f99741a4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.761+0000 7f9973fff700 1 -- 172.21.15.27:0/1041298260 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9974101160 msgr2=0x7f99741998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:34.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.761+0000 7f9973fff700 1 --2- 172.21.15.27:0/1041298260 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9974101160 0x7f99741998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.761+0000 7f9973fff700 1 -- 172.21.15.27:0/1041298260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99741a4e60 con 0x7f9974101b20 2024-04-25T00:12:34.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.761+0000 7f99790d3700 1 --2- 172.21.15.27:0/1041298260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99741071e0 0x7f99741a4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:12:34.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.761+0000 7f9973fff700 1 --2- 172.21.15.27:0/1041298260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9974101b20 0x7f9974199de0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f996000d950 tx=0x7f996000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:34.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.761+0000 7f9971ffb700 1 -- 172.21.15.27:0/1041298260 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9960016070 con 0x7f9974101b20 2024-04-25T00:12:34.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.761+0000 7f9971ffb700 1 -- 172.21.15.27:0/1041298260 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9960009b40 con 0x7f9974101b20 2024-04-25T00:12:34.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.761+0000 7f997ab36700 1 -- 172.21.15.27:0/1041298260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99741a5090 con 0x7f9974101b20 2024-04-25T00:12:34.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.761+0000 7f9971ffb700 1 -- 172.21.15.27:0/1041298260 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996000f620 con 0x7f9974101b20 2024-04-25T00:12:34.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.762+0000 7f997ab36700 1 -- 172.21.15.27:0/1041298260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99741a55d0 con 0x7f9974101b20 2024-04-25T00:12:34.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.762+0000 7f997ab36700 1 -- 172.21.15.27:0/1041298260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99741096a0 con 0x7f9974101b20 2024-04-25T00:12:34.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.764+0000 7f9971ffb700 1 -- 172.21.15.27:0/1041298260 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9960019070 con 0x7f9974101b20 2024-04-25T00:12:34.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.764+0000 7f9971ffb700 1 --2- 172.21.15.27:0/1041298260 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f99640a2c20 0x7f99640a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:34.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.764+0000 7f99788d2700 1 --2- 172.21.15.27:0/1041298260 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f99640a2c20 0x7f99640a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:34.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.764+0000 7f9971ffb700 1 -- 172.21.15.27:0/1041298260 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f99600cebe0 con 0x7f9974101b20 2024-04-25T00:12:34.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.765+0000 7f99788d2700 1 --2- 172.21.15.27:0/1041298260 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f99640a2c20 0x7f99640a50e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9968006010 tx=0x7f996800b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:34.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.769+0000 7f9971ffb700 1 -- 172.21.15.27:0/1041298260 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f996009bfe0 con 0x7f9974101b20 2024-04-25T00:12:34.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.973+0000 7f997ab36700 1 -- 172.21.15.27:0/1041298260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f997406afb0 con 0x7f9974101b20 2024-04-25T00:12:34.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.973+0000 7f9971ffb700 1 -- 172.21.15.27:0/1041298260 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f996009b8e0 con 0x7f9974101b20 2024-04-25T00:12:34.978 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T00:12:34.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.975+0000 7f997ab36700 1 -- 172.21.15.27:0/1041298260 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f99640a2c20 msgr2=0x7f99640a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:34.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.975+0000 7f997ab36700 1 --2- 172.21.15.27:0/1041298260 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f99640a2c20 0x7f99640a50e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9968006010 tx=0x7f996800b540 comp rx=0 tx=0).stop 2024-04-25T00:12:34.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.975+0000 7f997ab36700 1 -- 172.21.15.27:0/1041298260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9974101b20 msgr2=0x7f9974199de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:34.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.975+0000 7f997ab36700 1 --2- 172.21.15.27:0/1041298260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9974101b20 0x7f9974199de0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f996000d950 tx=0x7f996000dd10 comp rx=0 tx=0).stop 2024-04-25T00:12:34.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.975+0000 7f997ab36700 1 -- 172.21.15.27:0/1041298260 shutdown_connections 2024-04-25T00:12:34.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.975+0000 7f997ab36700 1 --2- 172.21.15.27:0/1041298260 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f99640a2c20 0x7f99640a50e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.975+0000 7f997ab36700 1 --2- 172.21.15.27:0/1041298260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99741071e0 0x7f99741a4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.975+0000 7f997ab36700 1 --2- 172.21.15.27:0/1041298260 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9974101160 0x7f99741998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.975+0000 7f997ab36700 1 --2- 172.21.15.27:0/1041298260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9974101b20 0x7f9974199de0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.975+0000 7f997ab36700 1 -- 172.21.15.27:0/1041298260 >> 172.21.15.27:0/1041298260 conn(0x7f99740771a0 msgr2=0x7f99740fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:34.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.975+0000 7f997ab36700 1 -- 172.21.15.27:0/1041298260 shutdown_connections 2024-04-25T00:12:34.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:34.975+0000 7f997ab36700 1 -- 172.21.15.27:0/1041298260 wait complete. 2024-04-25T00:12:35.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:35 smithi153 ceph-mon[32896]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:35.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:35 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1041298260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:12:35.348 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:35 smithi027 ceph-mon[29208]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:35.348 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:35 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1041298260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:12:35.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:35 smithi139 ceph-mon[32833]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:35.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:35 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1041298260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:12:35.780 INFO:tasks.cephadm:Adding mds.a on smithi027 2024-04-25T00:12:35.781 INFO:tasks.cephadm:Adding mds.d on smithi027 2024-04-25T00:12:35.781 INFO:tasks.cephadm:Adding mds.g on smithi027 2024-04-25T00:12:35.781 INFO:tasks.cephadm:Adding mds.j on smithi027 2024-04-25T00:12:35.781 INFO:tasks.cephadm:Adding mds.b on smithi139 2024-04-25T00:12:35.781 INFO:tasks.cephadm:Adding mds.e on smithi139 2024-04-25T00:12:35.781 INFO:tasks.cephadm:Adding mds.h on smithi139 2024-04-25T00:12:35.781 INFO:tasks.cephadm:Adding mds.k on smithi139 2024-04-25T00:12:35.781 INFO:tasks.cephadm:Adding mds.c on smithi153 2024-04-25T00:12:35.781 INFO:tasks.cephadm:Adding mds.f on smithi153 2024-04-25T00:12:35.781 INFO:tasks.cephadm:Adding mds.i on smithi153 2024-04-25T00:12:35.781 INFO:tasks.cephadm:Adding mds.l on smithi153 2024-04-25T00:12:35.781 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l' 2024-04-25T00:12:36.035 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T00:12:37.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:37 smithi153 ceph-mon[32896]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:37.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:37 smithi027 ceph-mon[29208]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:37.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:37 smithi139 ceph-mon[32833]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:37.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.695+0000 7fd37bf03700 1 -- 172.21.15.153:0/2594315921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd374101190 msgr2=0x7fd3741015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:37.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.695+0000 7fd37bf03700 1 --2- 172.21.15.153:0/2594315921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd374101190 0x7fd3741015b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd370009990 tx=0x7fd370009ca0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.695+0000 7fd37bf03700 1 -- 172.21.15.153:0/2594315921 shutdown_connections 2024-04-25T00:12:37.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.695+0000 7fd37bf03700 1 --2- 172.21.15.153:0/2594315921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd374102d30 0x7fd3741095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.695+0000 7fd37bf03700 1 --2- 172.21.15.153:0/2594315921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd374101190 0x7fd3741015b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.695+0000 7fd37bf03700 1 --2- 172.21.15.153:0/2594315921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd374102390 0x7fd3741027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.695+0000 7fd37bf03700 1 -- 172.21.15.153:0/2594315921 >> 172.21.15.153:0/2594315921 conn(0x7fd3740fc730 msgr2=0x7fd3740feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:37.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.695+0000 7fd37bf03700 1 -- 172.21.15.153:0/2594315921 shutdown_connections 2024-04-25T00:12:37.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.695+0000 7fd37bf03700 1 -- 172.21.15.153:0/2594315921 wait complete. 2024-04-25T00:12:37.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd37bf03700 1 Processor -- start 2024-04-25T00:12:37.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd37bf03700 1 -- start start 2024-04-25T00:12:37.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd37bf03700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd374101190 0x7fd374198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:37.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd37bf03700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd374102390 0x7fd374198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:37.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd37bf03700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd374102d30 0x7fd37419fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:37.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd37bf03700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd37410cdd0 con 0x7fd374102d30 2024-04-25T00:12:37.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd37bf03700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd37410cc50 con 0x7fd374101190 2024-04-25T00:12:37.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd37bf03700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd37410cf50 con 0x7fd374102390 2024-04-25T00:12:37.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd37949e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd374102390 0x7fd374198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:37.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd37949e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd374102390 0x7fd374198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53130/0 (socket says 172.21.15.153:53130) 2024-04-25T00:12:37.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd37949e700 1 -- 172.21.15.153:0/1607363335 learned_addr learned my addr 172.21.15.153:0/1607363335 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T00:12:37.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd37a4a0700 1 --2- 172.21.15.153:0/1607363335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd374102d30 0x7fd37419fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:37.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd379c9f700 1 --2- 172.21.15.153:0/1607363335 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd374101190 0x7fd374198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:37.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd37949e700 1 -- 172.21.15.153:0/1607363335 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd374101190 msgr2=0x7fd374198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:37.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd37949e700 1 --2- 172.21.15.153:0/1607363335 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd374101190 0x7fd374198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd37949e700 1 -- 172.21.15.153:0/1607363335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd374102d30 msgr2=0x7fd37419fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:37.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd37949e700 1 --2- 172.21.15.153:0/1607363335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd374102d30 0x7fd37419fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.696+0000 7fd37949e700 1 -- 172.21.15.153:0/1607363335 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3741a01f0 con 0x7fd374102390 2024-04-25T00:12:37.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.697+0000 7fd37949e700 1 --2- 172.21.15.153:0/1607363335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd374102390 0x7fd374198ad0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd36400d9b0 tx=0x7fd36400dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:37.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.697+0000 7fd36affd700 1 -- 172.21.15.153:0/1607363335 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd364016070 con 0x7fd374102390 2024-04-25T00:12:37.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.697+0000 7fd37bf03700 1 -- 172.21.15.153:0/1607363335 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3741a04e0 con 0x7fd374102390 2024-04-25T00:12:37.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.697+0000 7fd36affd700 1 -- 172.21.15.153:0/1607363335 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd364004d10 con 0x7fd374102390 2024-04-25T00:12:37.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.697+0000 7fd37bf03700 1 -- 172.21.15.153:0/1607363335 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3741a0a70 con 0x7fd374102390 2024-04-25T00:12:37.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.697+0000 7fd36affd700 1 -- 172.21.15.153:0/1607363335 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd364005530 con 0x7fd374102390 2024-04-25T00:12:37.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.698+0000 7fd37bf03700 1 -- 172.21.15.153:0/1607363335 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd374068f20 con 0x7fd374102390 2024-04-25T00:12:37.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.699+0000 7fd36affd700 1 -- 172.21.15.153:0/1607363335 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd364004750 con 0x7fd374102390 2024-04-25T00:12:37.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.699+0000 7fd36affd700 1 --2- 172.21.15.153:0/1607363335 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd3600a2c20 0x7fd3600a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:37.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.699+0000 7fd36affd700 1 -- 172.21.15.153:0/1607363335 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd3640ce9a0 con 0x7fd374102390 2024-04-25T00:12:37.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.700+0000 7fd379c9f700 1 --2- 172.21.15.153:0/1607363335 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd3600a2c20 0x7fd3600a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:37.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.700+0000 7fd379c9f700 1 --2- 172.21.15.153:0/1607363335 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd3600a2c20 0x7fd3600a50e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd37000b5c0 tx=0x7fd370005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:37.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.704+0000 7fd36affd700 1 -- 172.21.15.153:0/1607363335 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd36401f020 con 0x7fd374102390 2024-04-25T00:12:37.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.909+0000 7fd37bf03700 1 -- 172.21.15.153:0/1607363335 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l", "target": ["mon-mgr", ""]}) -- 0x7fd37406a720 con 0x7fd3600a2c20 2024-04-25T00:12:37.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.915+0000 7fd36affd700 1 -- 172.21.15.153:0/1607363335 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fd37406a720 con 0x7fd3600a2c20 2024-04-25T00:12:37.916 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mds.all update... 2024-04-25T00:12:37.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.917+0000 7fd37bf03700 1 -- 172.21.15.153:0/1607363335 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd3600a2c20 msgr2=0x7fd3600a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:37.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.917+0000 7fd37bf03700 1 --2- 172.21.15.153:0/1607363335 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd3600a2c20 0x7fd3600a50e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd37000b5c0 tx=0x7fd370005fd0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.917+0000 7fd37bf03700 1 -- 172.21.15.153:0/1607363335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd374102390 msgr2=0x7fd374198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:37.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.918+0000 7fd37bf03700 1 --2- 172.21.15.153:0/1607363335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd374102390 0x7fd374198ad0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd36400d9b0 tx=0x7fd36400dcc0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.918+0000 7fd37bf03700 1 -- 172.21.15.153:0/1607363335 shutdown_connections 2024-04-25T00:12:37.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.918+0000 7fd37bf03700 1 --2- 172.21.15.153:0/1607363335 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd3600a2c20 0x7fd3600a50e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.918+0000 7fd37bf03700 1 --2- 172.21.15.153:0/1607363335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd374102d30 0x7fd37419fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.918+0000 7fd37bf03700 1 --2- 172.21.15.153:0/1607363335 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd374101190 0x7fd374198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.918+0000 7fd37bf03700 1 --2- 172.21.15.153:0/1607363335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd374102390 0x7fd374198ad0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.918+0000 7fd37bf03700 1 -- 172.21.15.153:0/1607363335 >> 172.21.15.153:0/1607363335 conn(0x7fd3740fc730 msgr2=0x7fd374069f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:37.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.918+0000 7fd37bf03700 1 -- 172.21.15.153:0/1607363335 shutdown_connections 2024-04-25T00:12:37.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T00:12:37.918+0000 7fd37bf03700 1 -- 172.21.15.153:0/1607363335 wait complete. 2024-04-25T00:12:38.598 DEBUG:teuthology.orchestra.run.smithi027:mds.a> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.a.service 2024-04-25T00:12:38.601 DEBUG:teuthology.orchestra.run.smithi027:mds.d> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.d.service 2024-04-25T00:12:38.645 DEBUG:teuthology.orchestra.run.smithi027:mds.g> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.g.service 2024-04-25T00:12:38.688 DEBUG:teuthology.orchestra.run.smithi027:mds.j> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.j.service 2024-04-25T00:12:38.732 DEBUG:teuthology.orchestra.run.smithi139:mds.b> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.b.service 2024-04-25T00:12:38.735 DEBUG:teuthology.orchestra.run.smithi139:mds.e> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.e.service 2024-04-25T00:12:38.778 DEBUG:teuthology.orchestra.run.smithi139:mds.h> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.h.service 2024-04-25T00:12:38.820 DEBUG:teuthology.orchestra.run.smithi139:mds.k> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.k.service 2024-04-25T00:12:38.863 DEBUG:teuthology.orchestra.run.smithi153:mds.c> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.c.service 2024-04-25T00:12:38.866 DEBUG:teuthology.orchestra.run.smithi153:mds.f> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.f.service 2024-04-25T00:12:38.908 DEBUG:teuthology.orchestra.run.smithi153:mds.i> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.i.service 2024-04-25T00:12:38.951 DEBUG:teuthology.orchestra.run.smithi153:mds.l> sudo journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.l.service 2024-04-25T00:12:38.994 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-25T00:12:38.995 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:12:39.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.115+0000 7f8b8f534640 1 -- 172.21.15.27:0/449103388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b88104b50 msgr2=0x7f8b88106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.115+0000 7f8b8f534640 1 --2- 172.21.15.27:0/449103388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b88104b50 0x7f8b88106f40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8b78009900 tx=0x7f8b7802a410 comp rx=0 tx=0).stop 2024-04-25T00:12:39.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.115+0000 7f8b8f534640 1 -- 172.21.15.27:0/449103388 shutdown_connections 2024-04-25T00:12:39.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.115+0000 7f8b8f534640 1 --2- 172.21.15.27:0/449103388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b88107480 0x7f8b881098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.115+0000 7f8b8f534640 1 --2- 172.21.15.27:0/449103388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b88104b50 0x7f8b88106f40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.115+0000 7f8b8f534640 1 --2- 172.21.15.27:0/449103388 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b88102220 0x7f8b88104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.115+0000 7f8b8f534640 1 -- 172.21.15.27:0/449103388 >> 172.21.15.27:0/449103388 conn(0x7f8b880fbc30 msgr2=0x7f8b880fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.115+0000 7f8b8f534640 1 -- 172.21.15.27:0/449103388 shutdown_connections 2024-04-25T00:12:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.115+0000 7f8b8f534640 1 -- 172.21.15.27:0/449103388 wait complete. 2024-04-25T00:12:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.116+0000 7f8b8f534640 1 Processor -- start 2024-04-25T00:12:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.116+0000 7f8b8f534640 1 -- start start 2024-04-25T00:12:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.116+0000 7f8b8f534640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b88102220 0x7f8b88197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.116+0000 7f8b8f534640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b88104b50 0x7f8b88198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.116+0000 7f8b8f534640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b88107480 0x7f8b8819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.116+0000 7f8b8f534640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8810d410 con 0x7f8b88104b50 2024-04-25T00:12:39.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.116+0000 7f8b8f534640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8810d290 con 0x7f8b88107480 2024-04-25T00:12:39.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.116+0000 7f8b8f534640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8810d590 con 0x7f8b88102220 2024-04-25T00:12:39.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.116+0000 7f8b8cacf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b88104b50 0x7f8b88198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.116+0000 7f8b8cacf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b88104b50 0x7f8b88198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47314/0 (socket says 172.21.15.27:47314) 2024-04-25T00:12:39.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.116+0000 7f8b8cacf640 1 -- 172.21.15.27:0/785446088 learned_addr learned my addr 172.21.15.27:0/785446088 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:12:39.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.116+0000 7f8b8d2d0640 1 --2- 172.21.15.27:0/785446088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b88102220 0x7f8b88197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.116+0000 7f8b8dad1640 1 --2- 172.21.15.27:0/785446088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b88107480 0x7f8b8819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.117+0000 7f8b8cacf640 1 -- 172.21.15.27:0/785446088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b88102220 msgr2=0x7f8b88197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.117+0000 7f8b8cacf640 1 --2- 172.21.15.27:0/785446088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b88102220 0x7f8b88197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.117+0000 7f8b8cacf640 1 -- 172.21.15.27:0/785446088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b88107480 msgr2=0x7f8b8819f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.117+0000 7f8b8cacf640 1 --2- 172.21.15.27:0/785446088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b88107480 0x7f8b8819f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.117+0000 7f8b8cacf640 1 -- 172.21.15.27:0/785446088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b8819f730 con 0x7f8b88104b50 2024-04-25T00:12:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.117+0000 7f8b8dad1640 1 --2- 172.21.15.27:0/785446088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b88107480 0x7f8b8819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:12:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.117+0000 7f8b8cacf640 1 --2- 172.21.15.27:0/785446088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b88104b50 0x7f8b88198050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8b78031610 tx=0x7f8b78004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.117+0000 7f8b7e7fc640 1 -- 172.21.15.27:0/785446088 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7803f030 con 0x7f8b88104b50 2024-04-25T00:12:39.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.117+0000 7f8b7e7fc640 1 -- 172.21.15.27:0/785446088 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b78003ad0 con 0x7f8b88104b50 2024-04-25T00:12:39.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.117+0000 7f8b8f534640 1 -- 172.21.15.27:0/785446088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b8819f9c0 con 0x7f8b88104b50 2024-04-25T00:12:39.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.117+0000 7f8b7e7fc640 1 -- 172.21.15.27:0/785446088 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b78035070 con 0x7f8b88104b50 2024-04-25T00:12:39.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.117+0000 7f8b8f534640 1 -- 172.21.15.27:0/785446088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b8819ff00 con 0x7f8b88104b50 2024-04-25T00:12:39.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.118+0000 7f8b8f534640 1 -- 172.21.15.27:0/785446088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b880fd720 con 0x7f8b88104b50 2024-04-25T00:12:39.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.119+0000 7f8b7e7fc640 1 -- 172.21.15.27:0/785446088 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8b78042030 con 0x7f8b88104b50 2024-04-25T00:12:39.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.120+0000 7f8b7e7fc640 1 --2- 172.21.15.27:0/785446088 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8b740a2c10 0x7f8b740a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.120+0000 7f8b8d2d0640 1 --2- 172.21.15.27:0/785446088 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8b740a2c10 0x7f8b740a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.120+0000 7f8b7e7fc640 1 -- 172.21.15.27:0/785446088 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8b780ef490 con 0x7f8b88104b50 2024-04-25T00:12:39.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.120+0000 7f8b8d2d0640 1 --2- 172.21.15.27:0/785446088 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8b740a2c10 0x7f8b740a50d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8b8400bbe0 tx=0x7f8b8400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:39.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.123+0000 7f8b7e7fc640 1 -- 172.21.15.27:0/785446088 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8b780bc890 con 0x7f8b88104b50 2024-04-25T00:12:39.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:38 smithi027 ceph-mon[29208]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:39.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:38 smithi027 ceph-mon[29208]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:12:39.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:38 smithi027 ceph-mon[29208]: Saving service mds.all spec with placement smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l;count:12 2024-04-25T00:12:39.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:39.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:12:39.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:39.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:12:39.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:39.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:39.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:39.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:38 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:39.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:38 smithi139 ceph-mon[32833]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:39.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:38 smithi139 ceph-mon[32833]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:12:39.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:38 smithi139 ceph-mon[32833]: Saving service mds.all spec with placement smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l;count:12 2024-04-25T00:12:39.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:38 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:39.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:38 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:12:39.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:38 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:39.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:38 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:12:39.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:38 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:39.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:38 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:39.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:38 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:39.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:38 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:39.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:38 smithi153 ceph-mon[32896]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:39.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:38 smithi153 ceph-mon[32896]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:12:39.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:38 smithi153 ceph-mon[32896]: Saving service mds.all spec with placement smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l;count:12 2024-04-25T00:12:39.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:39.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:12:39.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:39.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:12:39.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:39.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:39.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:39.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:38 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:39.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.310+0000 7f8b8f534640 1 -- 172.21.15.27:0/785446088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b8802d3c0 con 0x7f8b88104b50 2024-04-25T00:12:39.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.311+0000 7f8b7e7fc640 1 -- 172.21.15.27:0/785446088 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18445 (secure 0 0 0) 0x7f8b780bc190 con 0x7f8b88104b50 2024-04-25T00:12:39.312 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:12:39.312 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":81,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:12:26.417275+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:12:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.312+0000 7f8b8f534640 1 -- 172.21.15.27:0/785446088 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8b740a2c10 msgr2=0x7f8b740a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.312+0000 7f8b8f534640 1 --2- 172.21.15.27:0/785446088 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8b740a2c10 0x7f8b740a50d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8b8400bbe0 tx=0x7f8b8400b390 comp rx=0 tx=0).stop 2024-04-25T00:12:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.312+0000 7f8b8f534640 1 -- 172.21.15.27:0/785446088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b88104b50 msgr2=0x7f8b88198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.312+0000 7f8b8f534640 1 --2- 172.21.15.27:0/785446088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b88104b50 0x7f8b88198050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8b78031610 tx=0x7f8b78004270 comp rx=0 tx=0).stop 2024-04-25T00:12:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.313+0000 7f8b8f534640 1 -- 172.21.15.27:0/785446088 shutdown_connections 2024-04-25T00:12:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.313+0000 7f8b8f534640 1 --2- 172.21.15.27:0/785446088 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8b740a2c10 0x7f8b740a50d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.313+0000 7f8b8f534640 1 --2- 172.21.15.27:0/785446088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b88107480 0x7f8b8819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.313+0000 7f8b8f534640 1 --2- 172.21.15.27:0/785446088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b88104b50 0x7f8b88198050 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.313+0000 7f8b8f534640 1 --2- 172.21.15.27:0/785446088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b88102220 0x7f8b88197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.313+0000 7f8b8f534640 1 -- 172.21.15.27:0/785446088 >> 172.21.15.27:0/785446088 conn(0x7f8b880fbc30 msgr2=0x7f8b88102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:39.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.313+0000 7f8b8f534640 1 -- 172.21.15.27:0/785446088 shutdown_connections 2024-04-25T00:12:39.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.313+0000 7f8b8f534640 1 -- 172.21.15.27:0/785446088 wait complete. 2024-04-25T00:12:39.324 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:08:34.006973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T00:12:39.325 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:12:39.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.429+0000 7f30e38b3640 1 -- 172.21.15.27:0/3442233270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30dc102250 msgr2=0x7f30dc108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.429+0000 7f30e38b3640 1 --2- 172.21.15.27:0/3442233270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30dc102250 0x7f30dc108ae0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009ca0 tx=0x7f30d402a8c0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.429+0000 7f30e38b3640 1 -- 172.21.15.27:0/3442233270 shutdown_connections 2024-04-25T00:12:39.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.429+0000 7f30e38b3640 1 --2- 172.21.15.27:0/3442233270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30dc102250 0x7f30dc108ae0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.429+0000 7f30e38b3640 1 --2- 172.21.15.27:0/3442233270 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30dc101890 0x7f30dc101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.429+0000 7f30e38b3640 1 --2- 172.21.15.27:0/3442233270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30dc1006a0 0x7f30dc100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.429+0000 7f30e38b3640 1 -- 172.21.15.27:0/3442233270 >> 172.21.15.27:0/3442233270 conn(0x7f30dc0fbc30 msgr2=0x7f30dc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:39.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.429+0000 7f30e38b3640 1 -- 172.21.15.27:0/3442233270 shutdown_connections 2024-04-25T00:12:39.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.429+0000 7f30e38b3640 1 -- 172.21.15.27:0/3442233270 wait complete. 2024-04-25T00:12:39.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.429+0000 7f30e38b3640 1 Processor -- start 2024-04-25T00:12:39.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e38b3640 1 -- start start 2024-04-25T00:12:39.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e38b3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30dc1006a0 0x7f30dc197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e38b3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30dc101890 0x7f30dc197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e38b3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30dc102250 0x7f30dc19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e38b3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f30dc10d4a0 con 0x7f30dc101890 2024-04-25T00:12:39.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e38b3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f30dc10d320 con 0x7f30dc102250 2024-04-25T00:12:39.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e38b3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f30dc10d620 con 0x7f30dc1006a0 2024-04-25T00:12:39.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e164f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30dc1006a0 0x7f30dc197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e0e4e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30dc101890 0x7f30dc197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e0e4e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30dc101890 0x7f30dc197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47342/0 (socket says 172.21.15.27:47342) 2024-04-25T00:12:39.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e164f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30dc1006a0 0x7f30dc197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:35282/0 (socket says 172.21.15.27:35282) 2024-04-25T00:12:39.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e0e4e640 1 -- 172.21.15.27:0/3682361029 learned_addr learned my addr 172.21.15.27:0/3682361029 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:12:39.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e1e50640 1 --2- 172.21.15.27:0/3682361029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30dc102250 0x7f30dc19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e0e4e640 1 -- 172.21.15.27:0/3682361029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30dc1006a0 msgr2=0x7f30dc197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e0e4e640 1 --2- 172.21.15.27:0/3682361029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30dc1006a0 0x7f30dc197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e0e4e640 1 -- 172.21.15.27:0/3682361029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30dc102250 msgr2=0x7f30dc19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e0e4e640 1 --2- 172.21.15.27:0/3682361029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30dc102250 0x7f30dc19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e0e4e640 1 -- 172.21.15.27:0/3682361029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30dc19f660 con 0x7f30dc101890 2024-04-25T00:12:39.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e1e50640 1 --2- 172.21.15.27:0/3682361029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30dc102250 0x7f30dc19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:12:39.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e0e4e640 1 --2- 172.21.15.27:0/3682361029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30dc101890 0x7f30dc197f80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f30cc00e9f0 tx=0x7f30cc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:39.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30d27fc640 1 -- 172.21.15.27:0/3682361029 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30cc00cd80 con 0x7f30dc101890 2024-04-25T00:12:39.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30d27fc640 1 -- 172.21.15.27:0/3682361029 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30cc010910 con 0x7f30dc101890 2024-04-25T00:12:39.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e38b3640 1 -- 172.21.15.27:0/3682361029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30dc19f950 con 0x7f30dc101890 2024-04-25T00:12:39.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30d27fc640 1 -- 172.21.15.27:0/3682361029 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30cc018990 con 0x7f30dc101890 2024-04-25T00:12:39.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.430+0000 7f30e38b3640 1 -- 172.21.15.27:0/3682361029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30dc19fef0 con 0x7f30dc101890 2024-04-25T00:12:39.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.431+0000 7f30e38b3640 1 -- 172.21.15.27:0/3682361029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30dc107960 con 0x7f30dc101890 2024-04-25T00:12:39.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.432+0000 7f30d27fc640 1 -- 172.21.15.27:0/3682361029 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f30cc010ac0 con 0x7f30dc101890 2024-04-25T00:12:39.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.433+0000 7f30d27fc640 1 --2- 172.21.15.27:0/3682361029 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f30c80a2c10 0x7f30c80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.433+0000 7f30d27fc640 1 -- 172.21.15.27:0/3682361029 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f30cc0cf2b0 con 0x7f30dc101890 2024-04-25T00:12:39.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.433+0000 7f30e164f640 1 --2- 172.21.15.27:0/3682361029 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f30c80a2c10 0x7f30c80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.434+0000 7f30e164f640 1 --2- 172.21.15.27:0/3682361029 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f30c80a2c10 0x7f30c80a50d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f30dc1016f0 tx=0x7f30d8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:39.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.436+0000 7f30d27fc640 1 -- 172.21.15.27:0/3682361029 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f30cc09c6b0 con 0x7f30dc101890 2024-04-25T00:12:39.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.624+0000 7f30e38b3640 1 -- 172.21.15.27:0/3682361029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f30dc02d3c0 con 0x7f30dc101890 2024-04-25T00:12:39.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.624+0000 7f30d27fc640 1 -- 172.21.15.27:0/3682361029 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f30cc09bfb0 con 0x7f30dc101890 2024-04-25T00:12:39.625 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T00:12:39.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.626+0000 7f30e38b3640 1 -- 172.21.15.27:0/3682361029 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f30c80a2c10 msgr2=0x7f30c80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.626+0000 7f30e38b3640 1 --2- 172.21.15.27:0/3682361029 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f30c80a2c10 0x7f30c80a50d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f30dc1016f0 tx=0x7f30d8005f90 comp rx=0 tx=0).stop 2024-04-25T00:12:39.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.626+0000 7f30e38b3640 1 -- 172.21.15.27:0/3682361029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30dc101890 msgr2=0x7f30dc197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.626+0000 7f30e38b3640 1 --2- 172.21.15.27:0/3682361029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30dc101890 0x7f30dc197f80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f30cc00e9f0 tx=0x7f30cc00eeb0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.626+0000 7f30e38b3640 1 -- 172.21.15.27:0/3682361029 shutdown_connections 2024-04-25T00:12:39.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.626+0000 7f30e38b3640 1 --2- 172.21.15.27:0/3682361029 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f30c80a2c10 0x7f30c80a50d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.626+0000 7f30e38b3640 1 --2- 172.21.15.27:0/3682361029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30dc102250 0x7f30dc19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.626+0000 7f30e38b3640 1 --2- 172.21.15.27:0/3682361029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30dc101890 0x7f30dc197f80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.626+0000 7f30e38b3640 1 --2- 172.21.15.27:0/3682361029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30dc1006a0 0x7f30dc197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.626+0000 7f30e38b3640 1 -- 172.21.15.27:0/3682361029 >> 172.21.15.27:0/3682361029 conn(0x7f30dc0fbc30 msgr2=0x7f30dc104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:39.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.626+0000 7f30e38b3640 1 -- 172.21.15.27:0/3682361029 shutdown_connections 2024-04-25T00:12:39.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.626+0000 7f30e38b3640 1 -- 172.21.15.27:0/3682361029 wait complete. 2024-04-25T00:12:39.638 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:39.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.739+0000 7f99e57b6640 1 -- 172.21.15.27:0/1257826495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e01029a0 msgr2=0x7f99e0102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.739+0000 7f99e57b6640 1 --2- 172.21.15.27:0/1257826495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e01029a0 0x7f99e0102e20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f99c8009900 tx=0x7f99c802a410 comp rx=0 tx=0).stop 2024-04-25T00:12:39.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.739+0000 7f99e57b6640 1 -- 172.21.15.27:0/1257826495 shutdown_connections 2024-04-25T00:12:39.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.739+0000 7f99e57b6640 1 --2- 172.21.15.27:0/1257826495 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99e0103360 0x7f99e0109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.739+0000 7f99e57b6640 1 --2- 172.21.15.27:0/1257826495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e01029a0 0x7f99e0102e20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.739+0000 7f99e57b6640 1 --2- 172.21.15.27:0/1257826495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e01017b0 0x7f99e0101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.739+0000 7f99e57b6640 1 -- 172.21.15.27:0/1257826495 >> 172.21.15.27:0/1257826495 conn(0x7f99e00fcd40 msgr2=0x7f99e00ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:39.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.739+0000 7f99e57b6640 1 -- 172.21.15.27:0/1257826495 shutdown_connections 2024-04-25T00:12:39.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.739+0000 7f99e57b6640 1 -- 172.21.15.27:0/1257826495 wait complete. 2024-04-25T00:12:39.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99e57b6640 1 Processor -- start 2024-04-25T00:12:39.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99e57b6640 1 -- start start 2024-04-25T00:12:39.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99e57b6640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99e01017b0 0x7f99e0199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99e57b6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e01029a0 0x7f99e019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99e57b6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e0103360 0x7f99e01a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99e57b6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f99e010d430 con 0x7f99e01029a0 2024-04-25T00:12:39.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99e57b6640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f99e010d2b0 con 0x7f99e01017b0 2024-04-25T00:12:39.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99e57b6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f99e010d5b0 con 0x7f99e0103360 2024-04-25T00:12:39.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99d7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e01029a0 0x7f99e019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99d7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e01029a0 0x7f99e019a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47364/0 (socket says 172.21.15.27:47364) 2024-04-25T00:12:39.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99d7fff640 1 -- 172.21.15.27:0/4206936789 learned_addr learned my addr 172.21.15.27:0/4206936789 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:12:39.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99df7fe640 1 --2- 172.21.15.27:0/4206936789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e0103360 0x7f99e01a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99deffd640 1 --2- 172.21.15.27:0/4206936789 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99e01017b0 0x7f99e0199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99d7fff640 1 -- 172.21.15.27:0/4206936789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e0103360 msgr2=0x7f99e01a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99d7fff640 1 --2- 172.21.15.27:0/4206936789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e0103360 0x7f99e01a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99d7fff640 1 -- 172.21.15.27:0/4206936789 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99e01017b0 msgr2=0x7f99e0199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99d7fff640 1 --2- 172.21.15.27:0/4206936789 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99e01017b0 0x7f99e0199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.740+0000 7f99d7fff640 1 -- 172.21.15.27:0/4206936789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99e01a1960 con 0x7f99e01029a0 2024-04-25T00:12:39.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.741+0000 7f99d7fff640 1 --2- 172.21.15.27:0/4206936789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e01029a0 0x7f99e019a280 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f99c8009230 tx=0x7f99c8031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:39.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.741+0000 7f99dcff9640 1 -- 172.21.15.27:0/4206936789 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c803f030 con 0x7f99e01029a0 2024-04-25T00:12:39.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.741+0000 7f99dcff9640 1 -- 172.21.15.27:0/4206936789 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99c8003ad0 con 0x7f99e01029a0 2024-04-25T00:12:39.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.741+0000 7f99e57b6640 1 -- 172.21.15.27:0/4206936789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99e01a1bf0 con 0x7f99e01029a0 2024-04-25T00:12:39.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.741+0000 7f99dcff9640 1 -- 172.21.15.27:0/4206936789 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c8035070 con 0x7f99e01029a0 2024-04-25T00:12:39.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.741+0000 7f99e57b6640 1 -- 172.21.15.27:0/4206936789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99e01a20e0 con 0x7f99e01029a0 2024-04-25T00:12:39.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.742+0000 7f99e57b6640 1 -- 172.21.15.27:0/4206936789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99e01a22b0 con 0x7f99e01029a0 2024-04-25T00:12:39.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.743+0000 7f99dcff9640 1 -- 172.21.15.27:0/4206936789 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f99c8042030 con 0x7f99e01029a0 2024-04-25T00:12:39.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.744+0000 7f99dcff9640 1 --2- 172.21.15.27:0/4206936789 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f99cc0a2b90 0x7f99cc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.744+0000 7f99deffd640 1 --2- 172.21.15.27:0/4206936789 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f99cc0a2b90 0x7f99cc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.744+0000 7f99dcff9640 1 -- 172.21.15.27:0/4206936789 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f99c80ef150 con 0x7f99e01029a0 2024-04-25T00:12:39.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.744+0000 7f99deffd640 1 --2- 172.21.15.27:0/4206936789 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f99cc0a2b90 0x7f99cc0a5050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f99e0102800 tx=0x7f99d0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:39.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.748+0000 7f99dcff9640 1 -- 172.21.15.27:0/4206936789 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f99c80bc550 con 0x7f99e01029a0 2024-04-25T00:12:39.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.972+0000 7f99e57b6640 1 -- 172.21.15.27:0/4206936789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99e01a2640 con 0x7f99e01029a0 2024-04-25T00:12:39.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.972+0000 7f99dcff9640 1 -- 172.21.15.27:0/4206936789 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f99c80bbe50 con 0x7f99e01029a0 2024-04-25T00:12:39.974 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:12:39.974 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-25T00:12:39.974 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 1 2024-04-25T00:12:39.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.975+0000 7f99e57b6640 1 -- 172.21.15.27:0/4206936789 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f99cc0a2b90 msgr2=0x7f99cc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.975+0000 7f99e57b6640 1 --2- 172.21.15.27:0/4206936789 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f99cc0a2b90 0x7f99cc0a5050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f99e0102800 tx=0x7f99d0005f90 comp rx=0 tx=0).stop 2024-04-25T00:12:39.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.975+0000 7f99e57b6640 1 -- 172.21.15.27:0/4206936789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e01029a0 msgr2=0x7f99e019a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.975+0000 7f99e57b6640 1 --2- 172.21.15.27:0/4206936789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e01029a0 0x7f99e019a280 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f99c8009230 tx=0x7f99c8031b60 comp rx=0 tx=0).stop 2024-04-25T00:12:39.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.975+0000 7f99e57b6640 1 -- 172.21.15.27:0/4206936789 shutdown_connections 2024-04-25T00:12:39.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.975+0000 7f99e57b6640 1 --2- 172.21.15.27:0/4206936789 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f99cc0a2b90 0x7f99cc0a5050 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.975+0000 7f99e57b6640 1 --2- 172.21.15.27:0/4206936789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e0103360 0x7f99e01a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.975+0000 7f99e57b6640 1 --2- 172.21.15.27:0/4206936789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e01029a0 0x7f99e019a280 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.975+0000 7f99e57b6640 1 --2- 172.21.15.27:0/4206936789 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99e01017b0 0x7f99e0199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.975+0000 7f99e57b6640 1 -- 172.21.15.27:0/4206936789 >> 172.21.15.27:0/4206936789 conn(0x7f99e00fcd40 msgr2=0x7f99e0073970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:39.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.975+0000 7f99e57b6640 1 -- 172.21.15.27:0/4206936789 shutdown_connections 2024-04-25T00:12:39.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:39.975+0000 7f99e57b6640 1 -- 172.21.15.27:0/4206936789 wait complete. 2024-04-25T00:12:40.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:39 smithi153 ceph-mon[32896]: Deploying daemon mds.l on smithi153 2024-04-25T00:12:40.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:39 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/785446088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:12:40.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:39 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3682361029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:12:40.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:39 smithi027 ceph-mon[29208]: Deploying daemon mds.l on smithi153 2024-04-25T00:12:40.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:39 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/785446088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:12:40.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:39 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3682361029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:12:40.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:39 smithi139 ceph-mon[32833]: Deploying daemon mds.l on smithi153 2024-04-25T00:12:40.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:39 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/785446088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:12:40.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:39 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3682361029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:12:41.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:40 smithi027 ceph-mon[29208]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:41.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:40 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4206936789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:41.235 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:40 smithi153 ceph-mon[32896]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:41.235 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:40 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4206936789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:41.235 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:12:41 smithi153 systemd[1]: Starting Ceph mds.l for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:12:41.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:40 smithi139 ceph-mon[32833]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:41.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:40 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4206936789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:41.519 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:12:41 smithi153 podman[63015]: 2024-04-25 00:12:41.137446686 +0000 UTC m=+0.017630740 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:12:41.519 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:12:41 smithi153 podman[63015]: 2024-04-25 00:12:41.270714617 +0000 UTC m=+0.150898659 container create c058c58a68b601d0ec629950f6e70b581c1e485d43b0a47cdae667600b23ffd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-25T00:12:41.776 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:12:41 smithi153 podman[63015]: 2024-04-25 00:12:41.519773915 +0000 UTC m=+0.399957956 container init c058c58a68b601d0ec629950f6e70b581c1e485d43b0a47cdae667600b23ffd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9) 2024-04-25T00:12:41.776 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:12:41 smithi153 podman[63015]: 2024-04-25 00:12:41.522398498 +0000 UTC m=+0.402582539 container start c058c58a68b601d0ec629950f6e70b581c1e485d43b0a47cdae667600b23ffd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T00:12:41.777 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:12:41 smithi153 ceph-mds[63034]: -- 172.21.15.153:0/931771974 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563857421a40 con 0x5638573f4c00 2024-04-25T00:12:41.777 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:12:41 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-l[63030]: starting mds.l at 2024-04-25T00:12:41.777 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:12:41 smithi153 bash[63015]: c058c58a68b601d0ec629950f6e70b581c1e485d43b0a47cdae667600b23ffd5 2024-04-25T00:12:41.777 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:12:41 smithi153 systemd[1]: Started Ceph mds.l for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:12:41.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:42.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.101+0000 7f555182c640 1 -- 172.21.15.27:0/4189725929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f554c1006a0 msgr2=0x7f554c100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:42.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.101+0000 7f555182c640 1 --2- 172.21.15.27:0/4189725929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f554c1006a0 0x7f554c100ac0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5534009900 tx=0x7f553402a410 comp rx=0 tx=0).stop 2024-04-25T00:12:42.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.101+0000 7f555182c640 1 -- 172.21.15.27:0/4189725929 shutdown_connections 2024-04-25T00:12:42.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.101+0000 7f555182c640 1 --2- 172.21.15.27:0/4189725929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f554c102250 0x7f554c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:42.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.101+0000 7f555182c640 1 --2- 172.21.15.27:0/4189725929 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f554c101890 0x7f554c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:42.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.101+0000 7f555182c640 1 --2- 172.21.15.27:0/4189725929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f554c1006a0 0x7f554c100ac0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:42.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.101+0000 7f555182c640 1 -- 172.21.15.27:0/4189725929 >> 172.21.15.27:0/4189725929 conn(0x7f554c0fbc30 msgr2=0x7f554c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:42.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.101+0000 7f555182c640 1 -- 172.21.15.27:0/4189725929 shutdown_connections 2024-04-25T00:12:42.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.101+0000 7f555182c640 1 -- 172.21.15.27:0/4189725929 wait complete. 2024-04-25T00:12:42.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.101+0000 7f555182c640 1 Processor -- start 2024-04-25T00:12:42.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f555182c640 1 -- start start 2024-04-25T00:12:42.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f555182c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f554c1006a0 0x7f554c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f555182c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f554c101890 0x7f554c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f555182c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f554c102250 0x7f554c19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f555182c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f554c10c320 con 0x7f554c1006a0 2024-04-25T00:12:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f555182c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f554c10c1a0 con 0x7f554c101890 2024-04-25T00:12:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f555182c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f554c10c4a0 con 0x7f554c102250 2024-04-25T00:12:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f554affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f554c1006a0 0x7f554c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f554affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f554c1006a0 0x7f554c197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47388/0 (socket says 172.21.15.27:47388) 2024-04-25T00:12:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f554affd640 1 -- 172.21.15.27:0/2580615332 learned_addr learned my addr 172.21.15.27:0/2580615332 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:12:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f554a7fc640 1 --2- 172.21.15.27:0/2580615332 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f554c101890 0x7f554c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f554b7fe640 1 --2- 172.21.15.27:0/2580615332 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f554c102250 0x7f554c19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f554affd640 1 -- 172.21.15.27:0/2580615332 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f554c102250 msgr2=0x7f554c19f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f554affd640 1 --2- 172.21.15.27:0/2580615332 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f554c102250 0x7f554c19f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f554affd640 1 -- 172.21.15.27:0/2580615332 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f554c101890 msgr2=0x7f554c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f554affd640 1 --2- 172.21.15.27:0/2580615332 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f554c101890 0x7f554c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f554affd640 1 -- 172.21.15.27:0/2580615332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f554c19f700 con 0x7f554c1006a0 2024-04-25T00:12:42.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.102+0000 7f554affd640 1 --2- 172.21.15.27:0/2580615332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f554c1006a0 0x7f554c197ae0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5534031610 tx=0x7f55340043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:42.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.103+0000 7f555082a640 1 -- 172.21.15.27:0/2580615332 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553403f030 con 0x7f554c1006a0 2024-04-25T00:12:42.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.103+0000 7f555082a640 1 -- 172.21.15.27:0/2580615332 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5534003ce0 con 0x7f554c1006a0 2024-04-25T00:12:42.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.103+0000 7f555182c640 1 -- 172.21.15.27:0/2580615332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f554c19f990 con 0x7f554c1006a0 2024-04-25T00:12:42.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.103+0000 7f555082a640 1 -- 172.21.15.27:0/2580615332 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5534035070 con 0x7f554c1006a0 2024-04-25T00:12:42.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.103+0000 7f555182c640 1 -- 172.21.15.27:0/2580615332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f554c19fed0 con 0x7f554c1006a0 2024-04-25T00:12:42.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.104+0000 7f555182c640 1 -- 172.21.15.27:0/2580615332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f554c1a00a0 con 0x7f554c1006a0 2024-04-25T00:12:42.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.105+0000 7f555082a640 1 -- 172.21.15.27:0/2580615332 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5534042030 con 0x7f554c1006a0 2024-04-25T00:12:42.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.105+0000 7f555082a640 1 --2- 172.21.15.27:0/2580615332 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f55380a2c10 0x7f55380a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:42.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.106+0000 7f554a7fc640 1 --2- 172.21.15.27:0/2580615332 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f55380a2c10 0x7f55380a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:42.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.106+0000 7f555082a640 1 -- 172.21.15.27:0/2580615332 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f55340ef040 con 0x7f554c1006a0 2024-04-25T00:12:42.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.106+0000 7f554a7fc640 1 --2- 172.21.15.27:0/2580615332 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f55380a2c10 0x7f55380a50d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f554c198ff0 tx=0x7f553c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:42.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.110+0000 7f555082a640 1 -- 172.21.15.27:0/2580615332 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f55340bbdf0 con 0x7f554c1006a0 2024-04-25T00:12:42.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.334+0000 7f555182c640 1 -- 172.21.15.27:0/2580615332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f554c1a0430 con 0x7f554c1006a0 2024-04-25T00:12:42.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.334+0000 7f555082a640 1 -- 172.21.15.27:0/2580615332 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f55340c00f0 con 0x7f554c1006a0 2024-04-25T00:12:42.335 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:12:42.335 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T00:12:42.336 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 2 2024-04-25T00:12:42.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.336+0000 7f555182c640 1 -- 172.21.15.27:0/2580615332 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f55380a2c10 msgr2=0x7f55380a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:42.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.336+0000 7f555182c640 1 --2- 172.21.15.27:0/2580615332 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f55380a2c10 0x7f55380a50d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f554c198ff0 tx=0x7f553c00b390 comp rx=0 tx=0).stop 2024-04-25T00:12:42.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.336+0000 7f555182c640 1 -- 172.21.15.27:0/2580615332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f554c1006a0 msgr2=0x7f554c197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:42.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.336+0000 7f555182c640 1 --2- 172.21.15.27:0/2580615332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f554c1006a0 0x7f554c197ae0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5534031610 tx=0x7f55340043b0 comp rx=0 tx=0).stop 2024-04-25T00:12:42.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.336+0000 7f555182c640 1 -- 172.21.15.27:0/2580615332 shutdown_connections 2024-04-25T00:12:42.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.336+0000 7f555182c640 1 --2- 172.21.15.27:0/2580615332 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f55380a2c10 0x7f55380a50d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:42.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.336+0000 7f555182c640 1 --2- 172.21.15.27:0/2580615332 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f554c102250 0x7f554c19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:42.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.336+0000 7f555182c640 1 --2- 172.21.15.27:0/2580615332 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f554c101890 0x7f554c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.336+0000 7f555182c640 1 --2- 172.21.15.27:0/2580615332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f554c1006a0 0x7f554c197ae0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.336+0000 7f555182c640 1 -- 172.21.15.27:0/2580615332 >> 172.21.15.27:0/2580615332 conn(0x7f554c0fbc30 msgr2=0x7f554c069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.336+0000 7f555182c640 1 -- 172.21.15.27:0/2580615332 shutdown_connections 2024-04-25T00:12:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:42.336+0000 7f555182c640 1 -- 172.21.15.27:0/2580615332 wait complete. 2024-04-25T00:12:42.639 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:42 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] up:boot 2024-04-25T00:12:42.640 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:42 smithi139 ceph-mon[32833]: fsmap 1 up:standby 2024-04-25T00:12:42.640 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:42 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T00:12:42.640 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:42 smithi139 ceph-mon[32833]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:42.640 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:42 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:42.640 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:42 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:42.640 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:42 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:42.640 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:42 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:42.640 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:42 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:42.640 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:42 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:42.640 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:42 smithi139 ceph-mon[32833]: Deploying daemon mds.k on smithi139 2024-04-25T00:12:42.640 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:42 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2580615332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:42.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:42 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] up:boot 2024-04-25T00:12:42.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:42 smithi027 ceph-mon[29208]: fsmap 1 up:standby 2024-04-25T00:12:42.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:42 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T00:12:42.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:42 smithi027 ceph-mon[29208]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:42.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:42 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:42.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:42 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:42.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:42 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:42.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:42 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:42.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:42 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:42.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:42 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:42.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:42 smithi027 ceph-mon[29208]: Deploying daemon mds.k on smithi139 2024-04-25T00:12:42.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:42 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2580615332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:43.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:42 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] up:boot 2024-04-25T00:12:43.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:42 smithi153 ceph-mon[32896]: fsmap 1 up:standby 2024-04-25T00:12:43.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:42 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T00:12:43.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:42 smithi153 ceph-mon[32896]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:43.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:42 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:43.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:42 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:43.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:42 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:43.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:42 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:43.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:42 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:43.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:42 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:43.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:42 smithi153 ceph-mon[32896]: Deploying daemon mds.k on smithi139 2024-04-25T00:12:43.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:42 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2580615332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:44.204 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:12:43 smithi139 systemd[1]: Starting Ceph mds.k for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:12:44.348 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:44.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.461+0000 7f8d27509640 1 -- 172.21.15.27:0/1368303951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d20102250 msgr2=0x7f8d20108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:44.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.461+0000 7f8d27509640 1 --2- 172.21.15.27:0/1368303951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d20102250 0x7f8d20108ae0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8d18009ca0 tx=0x7f8d1802a8c0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.461+0000 7f8d27509640 1 -- 172.21.15.27:0/1368303951 shutdown_connections 2024-04-25T00:12:44.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.461+0000 7f8d27509640 1 --2- 172.21.15.27:0/1368303951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d20102250 0x7f8d20108ae0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.461+0000 7f8d27509640 1 --2- 172.21.15.27:0/1368303951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d20101890 0x7f8d20101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.461+0000 7f8d27509640 1 --2- 172.21.15.27:0/1368303951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d201006a0 0x7f8d20100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.461+0000 7f8d27509640 1 -- 172.21.15.27:0/1368303951 >> 172.21.15.27:0/1368303951 conn(0x7f8d200fbc30 msgr2=0x7f8d200fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:44.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.461+0000 7f8d27509640 1 -- 172.21.15.27:0/1368303951 shutdown_connections 2024-04-25T00:12:44.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.461+0000 7f8d27509640 1 -- 172.21.15.27:0/1368303951 wait complete. 2024-04-25T00:12:44.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d27509640 1 Processor -- start 2024-04-25T00:12:44.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d27509640 1 -- start start 2024-04-25T00:12:44.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d27509640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d201006a0 0x7f8d201045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:44.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d27509640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d20101890 0x7f8d20102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:44.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d27509640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d20102250 0x7f8d201031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:44.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d27509640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2010c320 con 0x7f8d201006a0 2024-04-25T00:12:44.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d27509640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2010c1a0 con 0x7f8d20101890 2024-04-25T00:12:44.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d27509640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2010c4a0 con 0x7f8d20102250 2024-04-25T00:12:44.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d252a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d201006a0 0x7f8d201045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:44.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d252a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d201006a0 0x7f8d201045e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47416/0 (socket says 172.21.15.27:47416) 2024-04-25T00:12:44.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d252a5640 1 -- 172.21.15.27:0/3101897140 learned_addr learned my addr 172.21.15.27:0/3101897140 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:12:44.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d25aa6640 1 --2- 172.21.15.27:0/3101897140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d20102250 0x7f8d201031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:44.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d24aa4640 1 --2- 172.21.15.27:0/3101897140 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d20101890 0x7f8d20102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:44.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d252a5640 1 -- 172.21.15.27:0/3101897140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d20102250 msgr2=0x7f8d201031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:44.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d252a5640 1 --2- 172.21.15.27:0/3101897140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d20102250 0x7f8d201031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d252a5640 1 -- 172.21.15.27:0/3101897140 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d20101890 msgr2=0x7f8d20102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:44.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d252a5640 1 --2- 172.21.15.27:0/3101897140 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d20101890 0x7f8d20102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d252a5640 1 -- 172.21.15.27:0/3101897140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d20103a70 con 0x7f8d201006a0 2024-04-25T00:12:44.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.462+0000 7f8d25aa6640 1 --2- 172.21.15.27:0/3101897140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d20102250 0x7f8d201031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:12:44.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.463+0000 7f8d252a5640 1 --2- 172.21.15.27:0/3101897140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d201006a0 0x7f8d201045e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400e9c0 tx=0x7f8d1400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:44.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.463+0000 7f8d127fc640 1 -- 172.21.15.27:0/3101897140 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1400cd80 con 0x7f8d201006a0 2024-04-25T00:12:44.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.463+0000 7f8d127fc640 1 -- 172.21.15.27:0/3101897140 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d140035f0 con 0x7f8d201006a0 2024-04-25T00:12:44.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.463+0000 7f8d27509640 1 -- 172.21.15.27:0/3101897140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d201a1bb0 con 0x7f8d201006a0 2024-04-25T00:12:44.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.463+0000 7f8d127fc640 1 -- 172.21.15.27:0/3101897140 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d14010690 con 0x7f8d201006a0 2024-04-25T00:12:44.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.463+0000 7f8d27509640 1 -- 172.21.15.27:0/3101897140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d201a2080 con 0x7f8d201006a0 2024-04-25T00:12:44.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.464+0000 7f8d27509640 1 -- 172.21.15.27:0/3101897140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d201a22b0 con 0x7f8d201006a0 2024-04-25T00:12:44.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.465+0000 7f8d127fc640 1 -- 172.21.15.27:0/3101897140 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8d140037a0 con 0x7f8d201006a0 2024-04-25T00:12:44.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.465+0000 7f8d127fc640 1 --2- 172.21.15.27:0/3101897140 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d0c0a2b90 0x7f8d0c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:44.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.466+0000 7f8d24aa4640 1 --2- 172.21.15.27:0/3101897140 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d0c0a2b90 0x7f8d0c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:44.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.466+0000 7f8d127fc640 1 -- 172.21.15.27:0/3101897140 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8d140cf120 con 0x7f8d201006a0 2024-04-25T00:12:44.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.466+0000 7f8d24aa4640 1 --2- 172.21.15.27:0/3101897140 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d0c0a2b90 0x7f8d0c0a5050 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8d20104350 tx=0x7f8d1c00b590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:44.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.469+0000 7f8d127fc640 1 -- 172.21.15.27:0/3101897140 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8d1409c520 con 0x7f8d201006a0 2024-04-25T00:12:44.503 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:12:44 smithi139 podman[62028]: 2024-04-25 00:12:44.107538227 +0000 UTC m=+0.018660288 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:12:44.503 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:12:44 smithi139 podman[62028]: 2024-04-25 00:12:44.230403773 +0000 UTC m=+0.141525857 container create 7024280520b2158248c5b1fe3b149d309f87206bb66e7aef0585dbb72718fd98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, ceph=True, CEPH_POINT_RELEASE=) 2024-04-25T00:12:44.503 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:12:44 smithi139 podman[62028]: 2024-04-25 00:12:44.417379878 +0000 UTC m=+0.328501915 container init 7024280520b2158248c5b1fe3b149d309f87206bb66e7aef0585dbb72718fd98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T00:12:44.503 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:12:44 smithi139 podman[62028]: 2024-04-25 00:12:44.420113887 +0000 UTC m=+0.331235923 container start 7024280520b2158248c5b1fe3b149d309f87206bb66e7aef0585dbb72718fd98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:12:44.504 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:12:44 smithi139 ceph-mds[62048]: -- 172.21.15.139:0/2052569595 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ca37127a40 con 0x55ca370fa800 2024-04-25T00:12:44.504 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:12:44 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-k[62044]: starting mds.k at 2024-04-25T00:12:44.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:44 smithi027 ceph-mon[29208]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:44.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:44 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:44.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:44 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:44.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.720+0000 7f8d27509640 1 -- 172.21.15.27:0/3101897140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d201a2480 con 0x7f8d201006a0 2024-04-25T00:12:44.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.721+0000 7f8d127fc640 1 -- 172.21.15.27:0/3101897140 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f8d1409be20 con 0x7f8d201006a0 2024-04-25T00:12:44.722 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:12:44.722 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T00:12:44.722 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 3 2024-04-25T00:12:44.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.723+0000 7f8d27509640 1 -- 172.21.15.27:0/3101897140 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d0c0a2b90 msgr2=0x7f8d0c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:44.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.723+0000 7f8d27509640 1 --2- 172.21.15.27:0/3101897140 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d0c0a2b90 0x7f8d0c0a5050 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8d20104350 tx=0x7f8d1c00b590 comp rx=0 tx=0).stop 2024-04-25T00:12:44.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.723+0000 7f8d27509640 1 -- 172.21.15.27:0/3101897140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d201006a0 msgr2=0x7f8d201045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:44.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.723+0000 7f8d27509640 1 --2- 172.21.15.27:0/3101897140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d201006a0 0x7f8d201045e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400e9c0 tx=0x7f8d1400ee80 comp rx=0 tx=0).stop 2024-04-25T00:12:44.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.723+0000 7f8d27509640 1 -- 172.21.15.27:0/3101897140 shutdown_connections 2024-04-25T00:12:44.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.723+0000 7f8d27509640 1 --2- 172.21.15.27:0/3101897140 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d0c0a2b90 0x7f8d0c0a5050 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.723+0000 7f8d27509640 1 --2- 172.21.15.27:0/3101897140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d20102250 0x7f8d201031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.723+0000 7f8d27509640 1 --2- 172.21.15.27:0/3101897140 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d20101890 0x7f8d20102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.723+0000 7f8d27509640 1 --2- 172.21.15.27:0/3101897140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d201006a0 0x7f8d201045e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.723+0000 7f8d27509640 1 -- 172.21.15.27:0/3101897140 >> 172.21.15.27:0/3101897140 conn(0x7f8d200fbc30 msgr2=0x7f8d200fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:44.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.723+0000 7f8d27509640 1 -- 172.21.15.27:0/3101897140 shutdown_connections 2024-04-25T00:12:44.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:44.723+0000 7f8d27509640 1 -- 172.21.15.27:0/3101897140 wait complete. 2024-04-25T00:12:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:44 smithi139 ceph-mon[32833]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:44 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:44 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:45.003 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:12:44 smithi139 bash[62028]: 7024280520b2158248c5b1fe3b149d309f87206bb66e7aef0585dbb72718fd98 2024-04-25T00:12:45.003 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:12:44 smithi139 systemd[1]: Started Ceph mds.k for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:12:45.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:44 smithi153 ceph-mon[32896]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:45.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:44 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:44 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:45.849 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:45 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] up:boot 2024-04-25T00:12:45.850 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:45 smithi139 ceph-mon[32833]: fsmap 2 up:standby 2024-04-25T00:12:45.850 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:45 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T00:12:45.850 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:45 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:45.850 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:45 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:45.850 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:45 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:45.850 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:45 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:45.850 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:45 smithi139 ceph-mon[32833]: Deploying daemon mds.j on smithi027 2024-04-25T00:12:45.850 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:45 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3101897140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:45.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:45 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] up:boot 2024-04-25T00:12:45.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:45 smithi027 ceph-mon[29208]: fsmap 2 up:standby 2024-04-25T00:12:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:45 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T00:12:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:45 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:45 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:45 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:45 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:45 smithi027 ceph-mon[29208]: Deploying daemon mds.j on smithi027 2024-04-25T00:12:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:45 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3101897140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:45 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] up:boot 2024-04-25T00:12:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:45 smithi153 ceph-mon[32896]: fsmap 2 up:standby 2024-04-25T00:12:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:45 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T00:12:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:45 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:45 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:45 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:45 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:46.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:45 smithi153 ceph-mon[32896]: Deploying daemon mds.j on smithi027 2024-04-25T00:12:46.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:45 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3101897140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:46.737 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.934+0000 7fc4e99a9640 1 -- 172.21.15.27:0/1885881988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4e4101890 msgr2=0x7fc4e4101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.934+0000 7fc4e99a9640 1 --2- 172.21.15.27:0/1885881988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4e4101890 0x7fc4e4101d10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8009900 tx=0x7fc4d802a410 comp rx=0 tx=0).stop 2024-04-25T00:12:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.934+0000 7fc4e99a9640 1 -- 172.21.15.27:0/1885881988 shutdown_connections 2024-04-25T00:12:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.934+0000 7fc4e99a9640 1 --2- 172.21.15.27:0/1885881988 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc4e4102250 0x7fc4e4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.934+0000 7fc4e99a9640 1 --2- 172.21.15.27:0/1885881988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4e4101890 0x7fc4e4101d10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.934+0000 7fc4e99a9640 1 --2- 172.21.15.27:0/1885881988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4e41006a0 0x7fc4e4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.934+0000 7fc4e99a9640 1 -- 172.21.15.27:0/1885881988 >> 172.21.15.27:0/1885881988 conn(0x7fc4e40fbc30 msgr2=0x7fc4e40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.934+0000 7fc4e99a9640 1 -- 172.21.15.27:0/1885881988 shutdown_connections 2024-04-25T00:12:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.934+0000 7fc4e99a9640 1 -- 172.21.15.27:0/1885881988 wait complete. 2024-04-25T00:12:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.934+0000 7fc4e99a9640 1 Processor -- start 2024-04-25T00:12:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e99a9640 1 -- start start 2024-04-25T00:12:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e99a9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc4e41006a0 0x7fc4e4197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e99a9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4e4101890 0x7fc4e4198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e99a9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4e4102250 0x7fc4e419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e99a9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e410c370 con 0x7fc4e4102250 2024-04-25T00:12:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e99a9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e410c1f0 con 0x7fc4e41006a0 2024-04-25T00:12:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e99a9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e410c4f0 con 0x7fc4e4101890 2024-04-25T00:12:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4e4102250 0x7fc4e419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4e4102250 0x7fc4e419f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46408/0 (socket says 172.21.15.27:46408) 2024-04-25T00:12:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e37fe640 1 -- 172.21.15.27:0/4075352359 learned_addr learned my addr 172.21.15.27:0/4075352359 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:12:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e2ffd640 1 --2- 172.21.15.27:0/4075352359 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc4e41006a0 0x7fc4e4197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e37fe640 1 -- 172.21.15.27:0/4075352359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4e4101890 msgr2=0x7fc4e4198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e27fc640 1 --2- 172.21.15.27:0/4075352359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4e4101890 0x7fc4e4198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e37fe640 1 --2- 172.21.15.27:0/4075352359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4e4101890 0x7fc4e4198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e37fe640 1 -- 172.21.15.27:0/4075352359 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc4e41006a0 msgr2=0x7fc4e4197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e37fe640 1 --2- 172.21.15.27:0/4075352359 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc4e41006a0 0x7fc4e4197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e37fe640 1 -- 172.21.15.27:0/4075352359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4e419f700 con 0x7fc4e4102250 2024-04-25T00:12:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e27fc640 1 --2- 172.21.15.27:0/4075352359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4e4101890 0x7fc4e4198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:12:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e37fe640 1 --2- 172.21.15.27:0/4075352359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4e4102250 0x7fc4e419f090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00eb10 tx=0x7fc4dc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.935+0000 7fc4e89a7640 1 -- 172.21.15.27:0/4075352359 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc004b20 con 0x7fc4e4102250 2024-04-25T00:12:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.936+0000 7fc4e89a7640 1 -- 172.21.15.27:0/4075352359 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4dc004cc0 con 0x7fc4e4102250 2024-04-25T00:12:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.936+0000 7fc4e99a9640 1 -- 172.21.15.27:0/4075352359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4e419f9f0 con 0x7fc4e4102250 2024-04-25T00:12:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.936+0000 7fc4e89a7640 1 -- 172.21.15.27:0/4075352359 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc010760 con 0x7fc4e4102250 2024-04-25T00:12:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.936+0000 7fc4e99a9640 1 -- 172.21.15.27:0/4075352359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4e419fef0 con 0x7fc4e4102250 2024-04-25T00:12:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.937+0000 7fc4e99a9640 1 -- 172.21.15.27:0/4075352359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4e41a0120 con 0x7fc4e4102250 2024-04-25T00:12:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.938+0000 7fc4e89a7640 1 -- 172.21.15.27:0/4075352359 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc4dc005020 con 0x7fc4e4102250 2024-04-25T00:12:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.938+0000 7fc4e89a7640 1 --2- 172.21.15.27:0/4075352359 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc4cc0a2b90 0x7fc4cc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.938+0000 7fc4e89a7640 1 -- 172.21.15.27:0/4075352359 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc4dc0cf200 con 0x7fc4e4102250 2024-04-25T00:12:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.938+0000 7fc4e2ffd640 1 --2- 172.21.15.27:0/4075352359 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc4cc0a2b90 0x7fc4cc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.939+0000 7fc4e2ffd640 1 --2- 172.21.15.27:0/4075352359 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc4cc0a2b90 0x7fc4cc0a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc4e41016f0 tx=0x7fc4d4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:46.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:46 smithi027 ceph-mon[29208]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:46.942 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 00:12:46 smithi027 systemd[1]: Starting Ceph mds.j for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:12:46.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:46.943+0000 7fc4e89a7640 1 -- 172.21.15.27:0/4075352359 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc4dc09c600 con 0x7fc4e4102250 2024-04-25T00:12:47.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:46 smithi139 ceph-mon[32833]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:47.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:46 smithi153 ceph-mon[32896]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:47.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:47.181+0000 7fc4e99a9640 1 -- 172.21.15.27:0/4075352359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4e41a02f0 con 0x7fc4e4102250 2024-04-25T00:12:47.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:47.182+0000 7fc4e89a7640 1 -- 172.21.15.27:0/4075352359 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7fc4dc09bf00 con 0x7fc4e4102250 2024-04-25T00:12:47.183 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:12:47.183 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T00:12:47.183 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 3 2024-04-25T00:12:47.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:47.184+0000 7fc4e99a9640 1 -- 172.21.15.27:0/4075352359 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc4cc0a2b90 msgr2=0x7fc4cc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:47.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:47.184+0000 7fc4e99a9640 1 --2- 172.21.15.27:0/4075352359 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc4cc0a2b90 0x7fc4cc0a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc4e41016f0 tx=0x7fc4d4005f90 comp rx=0 tx=0).stop 2024-04-25T00:12:47.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:47.184+0000 7fc4e99a9640 1 -- 172.21.15.27:0/4075352359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4e4102250 msgr2=0x7fc4e419f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:47.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:47.184+0000 7fc4e99a9640 1 --2- 172.21.15.27:0/4075352359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4e4102250 0x7fc4e419f090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00eb10 tx=0x7fc4dc00c2d0 comp rx=0 tx=0).stop 2024-04-25T00:12:47.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:47.184+0000 7fc4e99a9640 1 -- 172.21.15.27:0/4075352359 shutdown_connections 2024-04-25T00:12:47.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:47.184+0000 7fc4e99a9640 1 --2- 172.21.15.27:0/4075352359 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc4cc0a2b90 0x7fc4cc0a5050 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:47.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:47.184+0000 7fc4e99a9640 1 --2- 172.21.15.27:0/4075352359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4e4102250 0x7fc4e419f090 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:47.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:47.184+0000 7fc4e99a9640 1 --2- 172.21.15.27:0/4075352359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4e4101890 0x7fc4e4198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:47.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:47.184+0000 7fc4e99a9640 1 --2- 172.21.15.27:0/4075352359 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc4e41006a0 0x7fc4e4197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:47.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:47.184+0000 7fc4e99a9640 1 -- 172.21.15.27:0/4075352359 >> 172.21.15.27:0/4075352359 conn(0x7fc4e40fbc30 msgr2=0x7fc4e40fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:47.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:47.184+0000 7fc4e99a9640 1 -- 172.21.15.27:0/4075352359 shutdown_connections 2024-04-25T00:12:47.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:47.184+0000 7fc4e99a9640 1 -- 172.21.15.27:0/4075352359 wait complete. 2024-04-25T00:12:47.367 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 00:12:47 smithi027 podman[66532]: 2024-04-25 00:12:47.007744612 +0000 UTC m=+0.022949451 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:12:47.367 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 00:12:47 smithi027 podman[66532]: 2024-04-25 00:12:47.131074203 +0000 UTC m=+0.146279027 container create 9926742db09d993558e2a1b8f4d83c9b6c9b0f256d71f5d9875311fb5201cacd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True) 2024-04-25T00:12:47.367 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 00:12:47 smithi027 podman[66532]: 2024-04-25 00:12:47.367267264 +0000 UTC m=+0.382472090 container init 9926742db09d993558e2a1b8f4d83c9b6c9b0f256d71f5d9875311fb5201cacd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0) 2024-04-25T00:12:47.637 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 00:12:47 smithi027 podman[66532]: 2024-04-25 00:12:47.37005054 +0000 UTC m=+0.385255373 container start 9926742db09d993558e2a1b8f4d83c9b6c9b0f256d71f5d9875311fb5201cacd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-04-25T00:12:47.637 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 00:12:47 smithi027 ceph-mds[66553]: -- 172.21.15.27:0/3563549097 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55637b4e7a40 con 0x55637b4ba800 2024-04-25T00:12:47.638 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 00:12:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-j[66549]: starting mds.j at 2024-04-25T00:12:47.638 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 00:12:47 smithi027 bash[66532]: 9926742db09d993558e2a1b8f4d83c9b6c9b0f256d71f5d9875311fb5201cacd 2024-04-25T00:12:47.638 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 00:12:47 smithi027 systemd[1]: Started Ceph mds.j for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:12:47.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:47 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4075352359' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:47.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:47.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:47.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:47.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:47.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:47.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:47 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:48.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:47 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4075352359' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:48.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:47 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:48.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:47 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:48.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:47 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:48.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:47 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:48.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:47 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:48.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:47 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:48.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:47 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4075352359' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:48.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:47 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:48.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:47 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:48.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:47 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:48.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:47 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:48.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:47 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:48.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:47 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:48.916 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:48 smithi027 ceph-mon[29208]: Deploying daemon mds.d on smithi027 2024-04-25T00:12:48.916 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:48 smithi027 ceph-mon[29208]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:48.916 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:48 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6834/2280519367,v1:172.21.15.27:6835/2280519367] up:boot 2024-04-25T00:12:48.916 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:48 smithi027 ceph-mon[29208]: fsmap 3 up:standby 2024-04-25T00:12:48.916 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:48 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T00:12:49.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:48 smithi139 ceph-mon[32833]: Deploying daemon mds.d on smithi027 2024-04-25T00:12:49.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:48 smithi139 ceph-mon[32833]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:49.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:48 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6834/2280519367,v1:172.21.15.27:6835/2280519367] up:boot 2024-04-25T00:12:49.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:48 smithi139 ceph-mon[32833]: fsmap 3 up:standby 2024-04-25T00:12:49.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:48 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T00:12:49.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:48 smithi153 ceph-mon[32896]: Deploying daemon mds.d on smithi027 2024-04-25T00:12:49.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:48 smithi153 ceph-mon[32896]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:49.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:48 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6834/2280519367,v1:172.21.15.27:6835/2280519367] up:boot 2024-04-25T00:12:49.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:48 smithi153 ceph-mon[32896]: fsmap 3 up:standby 2024-04-25T00:12:49.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:48 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T00:12:49.197 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:49.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.311+0000 7fc73b390640 1 -- 172.21.15.27:0/3561704861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7341006a0 msgr2=0x7fc734100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:49.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.311+0000 7fc73b390640 1 --2- 172.21.15.27:0/3561704861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7341006a0 0x7fc734100ac0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc730009900 tx=0x7fc73002a410 comp rx=0 tx=0).stop 2024-04-25T00:12:49.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.311+0000 7fc73b390640 1 -- 172.21.15.27:0/3561704861 shutdown_connections 2024-04-25T00:12:49.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.311+0000 7fc73b390640 1 --2- 172.21.15.27:0/3561704861 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc734102250 0x7fc734108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.311+0000 7fc73b390640 1 --2- 172.21.15.27:0/3561704861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc734101890 0x7fc734101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.311+0000 7fc73b390640 1 --2- 172.21.15.27:0/3561704861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7341006a0 0x7fc734100ac0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.311+0000 7fc73b390640 1 -- 172.21.15.27:0/3561704861 >> 172.21.15.27:0/3561704861 conn(0x7fc7340fbc30 msgr2=0x7fc7340fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:49.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.311+0000 7fc73b390640 1 -- 172.21.15.27:0/3561704861 shutdown_connections 2024-04-25T00:12:49.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.311+0000 7fc73b390640 1 -- 172.21.15.27:0/3561704861 wait complete. 2024-04-25T00:12:49.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73b390640 1 Processor -- start 2024-04-25T00:12:49.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73b390640 1 -- start start 2024-04-25T00:12:49.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73b390640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7341006a0 0x7fc734104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73b390640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc734101890 0x7fc734102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73b390640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc734102250 0x7fc7341031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73b390640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc73410c320 con 0x7fc7341006a0 2024-04-25T00:12:49.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73b390640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc73410c1a0 con 0x7fc734101890 2024-04-25T00:12:49.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73b390640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc73410c4a0 con 0x7fc734102250 2024-04-25T00:12:49.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73912c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7341006a0 0x7fc734104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73912c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7341006a0 0x7fc734104630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46442/0 (socket says 172.21.15.27:46442) 2024-04-25T00:12:49.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73912c640 1 -- 172.21.15.27:0/1129338260 learned_addr learned my addr 172.21.15.27:0/1129338260 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:12:49.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73992d640 1 --2- 172.21.15.27:0/1129338260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc734102250 0x7fc7341031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73892b640 1 --2- 172.21.15.27:0/1129338260 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc734101890 0x7fc734102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73912c640 1 -- 172.21.15.27:0/1129338260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc734102250 msgr2=0x7fc7341031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:49.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73912c640 1 --2- 172.21.15.27:0/1129338260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc734102250 0x7fc7341031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73912c640 1 -- 172.21.15.27:0/1129338260 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc734101890 msgr2=0x7fc734102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:49.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73912c640 1 --2- 172.21.15.27:0/1129338260 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc734101890 0x7fc734102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.312+0000 7fc73912c640 1 -- 172.21.15.27:0/1129338260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc734103a40 con 0x7fc7341006a0 2024-04-25T00:12:49.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.313+0000 7fc73912c640 1 --2- 172.21.15.27:0/1129338260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7341006a0 0x7fc734104630 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc730031610 tx=0x7fc7300043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:49.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.313+0000 7fc72a7fc640 1 -- 172.21.15.27:0/1129338260 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73003f030 con 0x7fc7341006a0 2024-04-25T00:12:49.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.313+0000 7fc73b390640 1 -- 172.21.15.27:0/1129338260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7341a1bb0 con 0x7fc7341006a0 2024-04-25T00:12:49.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.313+0000 7fc72a7fc640 1 -- 172.21.15.27:0/1129338260 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc730003ce0 con 0x7fc7341006a0 2024-04-25T00:12:49.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.313+0000 7fc72a7fc640 1 -- 172.21.15.27:0/1129338260 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc730035070 con 0x7fc7341006a0 2024-04-25T00:12:49.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.313+0000 7fc73b390640 1 -- 172.21.15.27:0/1129338260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7341a2100 con 0x7fc7341006a0 2024-04-25T00:12:49.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.314+0000 7fc73b390640 1 -- 172.21.15.27:0/1129338260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7341a22d0 con 0x7fc7341006a0 2024-04-25T00:12:49.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.315+0000 7fc72a7fc640 1 -- 172.21.15.27:0/1129338260 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc730042030 con 0x7fc7341006a0 2024-04-25T00:12:49.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.316+0000 7fc72a7fc640 1 --2- 172.21.15.27:0/1129338260 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc7200a2b90 0x7fc7200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.316+0000 7fc73892b640 1 --2- 172.21.15.27:0/1129338260 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc7200a2b90 0x7fc7200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.316+0000 7fc72a7fc640 1 -- 172.21.15.27:0/1129338260 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc7300ef3b0 con 0x7fc7341006a0 2024-04-25T00:12:49.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.316+0000 7fc73892b640 1 --2- 172.21.15.27:0/1129338260 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc7200a2b90 0x7fc7200a5050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc724005fd0 tx=0x7fc724005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:49.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.319+0000 7fc72a7fc640 1 -- 172.21.15.27:0/1129338260 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc7300bc7b0 con 0x7fc7341006a0 2024-04-25T00:12:49.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.564+0000 7fc73b390640 1 -- 172.21.15.27:0/1129338260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7341a24a0 con 0x7fc7341006a0 2024-04-25T00:12:49.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.565+0000 7fc72a7fc640 1 -- 172.21.15.27:0/1129338260 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7fc7300bc0b0 con 0x7fc7341006a0 2024-04-25T00:12:49.567 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:12:49.567 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T00:12:49.567 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 4 2024-04-25T00:12:49.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.567+0000 7fc73b390640 1 -- 172.21.15.27:0/1129338260 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc7200a2b90 msgr2=0x7fc7200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:49.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.567+0000 7fc73b390640 1 --2- 172.21.15.27:0/1129338260 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc7200a2b90 0x7fc7200a5050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc724005fd0 tx=0x7fc724005850 comp rx=0 tx=0).stop 2024-04-25T00:12:49.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.567+0000 7fc73b390640 1 -- 172.21.15.27:0/1129338260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7341006a0 msgr2=0x7fc734104630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:49.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.567+0000 7fc73b390640 1 --2- 172.21.15.27:0/1129338260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7341006a0 0x7fc734104630 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc730031610 tx=0x7fc7300043b0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.567+0000 7fc73b390640 1 -- 172.21.15.27:0/1129338260 shutdown_connections 2024-04-25T00:12:49.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.567+0000 7fc73b390640 1 --2- 172.21.15.27:0/1129338260 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc7200a2b90 0x7fc7200a5050 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.567+0000 7fc73b390640 1 --2- 172.21.15.27:0/1129338260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc734102250 0x7fc7341031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.567+0000 7fc73b390640 1 --2- 172.21.15.27:0/1129338260 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc734101890 0x7fc734102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.567+0000 7fc73b390640 1 --2- 172.21.15.27:0/1129338260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7341006a0 0x7fc734104630 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.567+0000 7fc73b390640 1 -- 172.21.15.27:0/1129338260 >> 172.21.15.27:0/1129338260 conn(0x7fc7340fbc30 msgr2=0x7fc7340fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:49.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.567+0000 7fc73b390640 1 -- 172.21.15.27:0/1129338260 shutdown_connections 2024-04-25T00:12:49.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:49.567+0000 7fc73b390640 1 -- 172.21.15.27:0/1129338260 wait complete. 2024-04-25T00:12:49.875 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:49 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1129338260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:49.875 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 00:12:49 smithi027 systemd[1]: Starting Ceph mds.d for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:12:49.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:49 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1129338260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:50.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:49 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1129338260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:50.192 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 00:12:50 smithi027 podman[66883]: 2024-04-25 00:12:49.998404733 +0000 UTC m=+0.018240228 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:12:50.192 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 00:12:50 smithi027 podman[66883]: 2024-04-25 00:12:50.119490718 +0000 UTC m=+0.139326186 container create def632097920c29825e5d2b372618bc00719ec2d7f5df6310bdb1b4e60816061 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-d, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-25T00:12:50.649 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 00:12:50 smithi027 podman[66883]: 2024-04-25 00:12:50.363867817 +0000 UTC m=+0.383703290 container init def632097920c29825e5d2b372618bc00719ec2d7f5df6310bdb1b4e60816061 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-d, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9) 2024-04-25T00:12:50.649 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 00:12:50 smithi027 podman[66883]: 2024-04-25 00:12:50.366537535 +0000 UTC m=+0.386373015 container start def632097920c29825e5d2b372618bc00719ec2d7f5df6310bdb1b4e60816061 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-d, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T00:12:50.649 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 00:12:50 smithi027 ceph-mds[66902]: -- 172.21.15.27:0/1602313167 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55acc879ba40 con 0x55acc876f000 2024-04-25T00:12:50.649 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 00:12:50 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-d[66898]: starting mds.d at 2024-04-25T00:12:50.649 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 00:12:50 smithi027 bash[66883]: def632097920c29825e5d2b372618bc00719ec2d7f5df6310bdb1b4e60816061 2024-04-25T00:12:50.650 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 00:12:50 smithi027 systemd[1]: Started Ceph mds.d for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:12:50.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:50 smithi027 ceph-mon[29208]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:50.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:50 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:50.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:50 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:50.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:50 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:50.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:50 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:50.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:50 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:50.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:50 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:51.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:50 smithi139 ceph-mon[32833]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:51.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:50 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:51.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:50 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:51.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:50 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:51.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:50 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:51.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:50 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:51.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:50 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:51.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:50 smithi153 ceph-mon[32896]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:51.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:50 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:51.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:50 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:51.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:50 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:51.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:50 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:51.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:50 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:51.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:50 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:51.581 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.699+0000 7fe34d3ce640 1 -- 172.21.15.27:0/3045814764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3481017b0 msgr2=0x7fe348101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.699+0000 7fe34d3ce640 1 --2- 172.21.15.27:0/3045814764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3481017b0 0x7fe348101bd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe338009900 tx=0x7fe33802a410 comp rx=0 tx=0).stop 2024-04-25T00:12:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.699+0000 7fe34d3ce640 1 -- 172.21.15.27:0/3045814764 shutdown_connections 2024-04-25T00:12:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.699+0000 7fe34d3ce640 1 --2- 172.21.15.27:0/3045814764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe348103360 0x7fe348109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.699+0000 7fe34d3ce640 1 --2- 172.21.15.27:0/3045814764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3481029a0 0x7fe348102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.699+0000 7fe34d3ce640 1 --2- 172.21.15.27:0/3045814764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3481017b0 0x7fe348101bd0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.699+0000 7fe34d3ce640 1 -- 172.21.15.27:0/3045814764 >> 172.21.15.27:0/3045814764 conn(0x7fe3480fcd40 msgr2=0x7fe3480ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.699+0000 7fe34d3ce640 1 -- 172.21.15.27:0/3045814764 shutdown_connections 2024-04-25T00:12:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.699+0000 7fe34d3ce640 1 -- 172.21.15.27:0/3045814764 wait complete. 2024-04-25T00:12:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.700+0000 7fe34d3ce640 1 Processor -- start 2024-04-25T00:12:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.700+0000 7fe34d3ce640 1 -- start start 2024-04-25T00:12:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.700+0000 7fe34d3ce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3481017b0 0x7fe3480754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.700+0000 7fe34d3ce640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3481029a0 0x7fe348073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.700+0000 7fe34d3ce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348103360 0x7fe348074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.700+0000 7fe34d3ce640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe34810d470 con 0x7fe348103360 2024-04-25T00:12:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.700+0000 7fe34d3ce640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe34810d2f0 con 0x7fe3481029a0 2024-04-25T00:12:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.700+0000 7fe34d3ce640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe34810d5f0 con 0x7fe3481017b0 2024-04-25T00:12:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.700+0000 7fe3477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348103360 0x7fe348074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.700+0000 7fe346ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3481017b0 0x7fe3480754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.700+0000 7fe3477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348103360 0x7fe348074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46492/0 (socket says 172.21.15.27:46492) 2024-04-25T00:12:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.700+0000 7fe3477fe640 1 -- 172.21.15.27:0/3837447088 learned_addr learned my addr 172.21.15.27:0/3837447088 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:12:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.700+0000 7fe33ffff640 1 --2- 172.21.15.27:0/3837447088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3481029a0 0x7fe348073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.701+0000 7fe3477fe640 1 -- 172.21.15.27:0/3837447088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3481017b0 msgr2=0x7fe3480754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.701+0000 7fe3477fe640 1 --2- 172.21.15.27:0/3837447088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3481017b0 0x7fe3480754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.701+0000 7fe3477fe640 1 -- 172.21.15.27:0/3837447088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3481029a0 msgr2=0x7fe348073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.701+0000 7fe3477fe640 1 --2- 172.21.15.27:0/3837447088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3481029a0 0x7fe348073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.701+0000 7fe3477fe640 1 -- 172.21.15.27:0/3837447088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe348074900 con 0x7fe348103360 2024-04-25T00:12:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.701+0000 7fe346ffd640 1 --2- 172.21.15.27:0/3837447088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3481017b0 0x7fe3480754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:12:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.701+0000 7fe33ffff640 1 --2- 172.21.15.27:0/3837447088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3481029a0 0x7fe348073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:12:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.701+0000 7fe3477fe640 1 --2- 172.21.15.27:0/3837447088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348103360 0x7fe348074070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe34000eaa0 tx=0x7fe34000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.701+0000 7fe344ff9640 1 -- 172.21.15.27:0/3837447088 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34000cdb0 con 0x7fe348103360 2024-04-25T00:12:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.701+0000 7fe34d3ce640 1 -- 172.21.15.27:0/3837447088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3481a1c30 con 0x7fe348103360 2024-04-25T00:12:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.701+0000 7fe344ff9640 1 -- 172.21.15.27:0/3837447088 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe340005020 con 0x7fe348103360 2024-04-25T00:12:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.701+0000 7fe344ff9640 1 -- 172.21.15.27:0/3837447088 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3400116b0 con 0x7fe348103360 2024-04-25T00:12:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.701+0000 7fe34d3ce640 1 -- 172.21.15.27:0/3837447088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3481a2180 con 0x7fe348103360 2024-04-25T00:12:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.702+0000 7fe34d3ce640 1 -- 172.21.15.27:0/3837447088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe348108b20 con 0x7fe348103360 2024-04-25T00:12:51.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.703+0000 7fe344ff9640 1 -- 172.21.15.27:0/3837447088 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe340005560 con 0x7fe348103360 2024-04-25T00:12:51.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.704+0000 7fe344ff9640 1 --2- 172.21.15.27:0/3837447088 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe3340a2b90 0x7fe3340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:51.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.704+0000 7fe346ffd640 1 --2- 172.21.15.27:0/3837447088 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe3340a2b90 0x7fe3340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:51.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.704+0000 7fe344ff9640 1 -- 172.21.15.27:0/3837447088 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe3400cee90 con 0x7fe348103360 2024-04-25T00:12:51.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.704+0000 7fe346ffd640 1 --2- 172.21.15.27:0/3837447088 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe3340a2b90 0x7fe3340a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe33802aa00 tx=0x7fe3380315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:51.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.708+0000 7fe344ff9640 1 -- 172.21.15.27:0/3837447088 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe340016030 con 0x7fe348103360 2024-04-25T00:12:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:51 smithi139 ceph-mon[32833]: Deploying daemon mds.e on smithi139 2024-04-25T00:12:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:51 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6836/590015015,v1:172.21.15.27:6837/590015015] up:boot 2024-04-25T00:12:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:51 smithi139 ceph-mon[32833]: fsmap 4 up:standby 2024-04-25T00:12:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:51 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T00:12:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.932+0000 7fe34d3ce640 1 -- 172.21.15.27:0/3837447088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe34802d370 con 0x7fe348103360 2024-04-25T00:12:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.933+0000 7fe344ff9640 1 -- 172.21.15.27:0/3837447088 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7fe34009c290 con 0x7fe348103360 2024-04-25T00:12:51.934 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:12:51.934 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T00:12:51.934 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 5 2024-04-25T00:12:51.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.935+0000 7fe34d3ce640 1 -- 172.21.15.27:0/3837447088 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe3340a2b90 msgr2=0x7fe3340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:51.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.935+0000 7fe34d3ce640 1 --2- 172.21.15.27:0/3837447088 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe3340a2b90 0x7fe3340a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe33802aa00 tx=0x7fe3380315b0 comp rx=0 tx=0).stop 2024-04-25T00:12:51.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.935+0000 7fe34d3ce640 1 -- 172.21.15.27:0/3837447088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348103360 msgr2=0x7fe348074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:51.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.935+0000 7fe34d3ce640 1 --2- 172.21.15.27:0/3837447088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348103360 0x7fe348074070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe34000eaa0 tx=0x7fe34000ef60 comp rx=0 tx=0).stop 2024-04-25T00:12:51.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.935+0000 7fe34d3ce640 1 -- 172.21.15.27:0/3837447088 shutdown_connections 2024-04-25T00:12:51.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.935+0000 7fe34d3ce640 1 --2- 172.21.15.27:0/3837447088 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe3340a2b90 0x7fe3340a5050 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:51.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.935+0000 7fe34d3ce640 1 --2- 172.21.15.27:0/3837447088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348103360 0x7fe348074070 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:51.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.935+0000 7fe34d3ce640 1 --2- 172.21.15.27:0/3837447088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3481029a0 0x7fe348073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:51.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.935+0000 7fe34d3ce640 1 --2- 172.21.15.27:0/3837447088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3481017b0 0x7fe3480754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:51.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.935+0000 7fe34d3ce640 1 -- 172.21.15.27:0/3837447088 >> 172.21.15.27:0/3837447088 conn(0x7fe3480fcd40 msgr2=0x7fe348105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:51.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.935+0000 7fe34d3ce640 1 -- 172.21.15.27:0/3837447088 shutdown_connections 2024-04-25T00:12:51.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:51.935+0000 7fe34d3ce640 1 -- 172.21.15.27:0/3837447088 wait complete. 2024-04-25T00:12:51.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:51 smithi027 ceph-mon[29208]: Deploying daemon mds.e on smithi139 2024-04-25T00:12:51.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:51 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6836/590015015,v1:172.21.15.27:6837/590015015] up:boot 2024-04-25T00:12:51.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:51 smithi027 ceph-mon[29208]: fsmap 4 up:standby 2024-04-25T00:12:51.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:51 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T00:12:52.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:51 smithi153 ceph-mon[32896]: Deploying daemon mds.e on smithi139 2024-04-25T00:12:52.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:51 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6836/590015015,v1:172.21.15.27:6837/590015015] up:boot 2024-04-25T00:12:52.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:51 smithi153 ceph-mon[32896]: fsmap 4 up:standby 2024-04-25T00:12:52.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:51 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T00:12:52.939 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:52 smithi139 ceph-mon[32833]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:52.939 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:52 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3837447088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:52.939 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:12:52 smithi139 systemd[1]: Starting Ceph mds.e for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:12:52.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:52 smithi027 ceph-mon[29208]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:52.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:52 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3837447088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:53.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:52 smithi153 ceph-mon[32896]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:53.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:52 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3837447088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:53.205 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:12:52 smithi139 podman[62352]: 2024-04-25 00:12:52.840909091 +0000 UTC m=+0.017990581 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:12:53.205 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:12:52 smithi139 podman[62352]: 2024-04-25 00:12:52.960667445 +0000 UTC m=+0.137748964 container create 8faab5e9836d4a198e7a50570f709197e79195cd9b5fecc8d843c623251c5388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-25T00:12:53.205 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:12:53 smithi139 podman[62352]: 2024-04-25 00:12:53.180116727 +0000 UTC m=+0.357198204 container init 8faab5e9836d4a198e7a50570f709197e79195cd9b5fecc8d843c623251c5388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-25T00:12:53.206 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:12:53 smithi139 podman[62352]: 2024-04-25 00:12:53.182583282 +0000 UTC m=+0.359664759 container start 8faab5e9836d4a198e7a50570f709197e79195cd9b5fecc8d843c623251c5388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-25T00:12:53.206 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:12:53 smithi139 ceph-mds[62372]: -- 172.21.15.139:0/4231639518 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559614d19a40 con 0x559614ced000 2024-04-25T00:12:53.503 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:12:53 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-e[62368]: starting mds.e at 2024-04-25T00:12:53.503 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:12:53 smithi139 bash[62352]: 8faab5e9836d4a198e7a50570f709197e79195cd9b5fecc8d843c623251c5388 2024-04-25T00:12:53.503 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:12:53 smithi139 systemd[1]: Started Ceph mds.e for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:12:53.947 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:54.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.061+0000 7f4a3409e640 1 -- 172.21.15.27:0/4083086035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a2c101930 msgr2=0x7f4a2c108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:54.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.061+0000 7f4a3409e640 1 --2- 172.21.15.27:0/4083086035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a2c101930 0x7f4a2c108250 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4a24009ca0 tx=0x7f4a2402a8c0 comp rx=0 tx=0).stop 2024-04-25T00:12:54.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.061+0000 7f4a3409e640 1 -- 172.21.15.27:0/4083086035 shutdown_connections 2024-04-25T00:12:54.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.061+0000 7f4a3409e640 1 --2- 172.21.15.27:0/4083086035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a2c101930 0x7f4a2c108250 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:54.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.061+0000 7f4a3409e640 1 --2- 172.21.15.27:0/4083086035 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a2c100f70 0x7f4a2c1013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:54.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.061+0000 7f4a3409e640 1 --2- 172.21.15.27:0/4083086035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a2c0ffe20 0x7f4a2c100240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:54.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.061+0000 7f4a3409e640 1 -- 172.21.15.27:0/4083086035 >> 172.21.15.27:0/4083086035 conn(0x7f4a2c0fb3b0 msgr2=0x7f4a2c0fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:54.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.061+0000 7f4a3409e640 1 -- 172.21.15.27:0/4083086035 shutdown_connections 2024-04-25T00:12:54.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.061+0000 7f4a3409e640 1 -- 172.21.15.27:0/4083086035 wait complete. 2024-04-25T00:12:54.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a3409e640 1 Processor -- start 2024-04-25T00:12:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a3409e640 1 -- start start 2024-04-25T00:12:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a3409e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a2c0ffe20 0x7f4a2c1097b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a3409e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a2c100f70 0x7f4a2c19c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a3409e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a2c101930 0x7f4a2c19edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a3409e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2c10c350 con 0x7f4a2c100f70 2024-04-25T00:12:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a3409e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2c10c1d0 con 0x7f4a2c0ffe20 2024-04-25T00:12:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a3409e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2c10c4d0 con 0x7f4a2c101930 2024-04-25T00:12:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a31639640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a2c100f70 0x7f4a2c19c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a31639640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a2c100f70 0x7f4a2c19c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46522/0 (socket says 172.21.15.27:46522) 2024-04-25T00:12:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a31639640 1 -- 172.21.15.27:0/3681983500 learned_addr learned my addr 172.21.15.27:0/3681983500 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:12:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a3263b640 1 --2- 172.21.15.27:0/3681983500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a2c101930 0x7f4a2c19edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a31e3a640 1 --2- 172.21.15.27:0/3681983500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a2c0ffe20 0x7f4a2c1097b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a31639640 1 -- 172.21.15.27:0/3681983500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a2c101930 msgr2=0x7f4a2c19edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a31639640 1 --2- 172.21.15.27:0/3681983500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a2c101930 0x7f4a2c19edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a31639640 1 -- 172.21.15.27:0/3681983500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a2c0ffe20 msgr2=0x7f4a2c1097b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a31639640 1 --2- 172.21.15.27:0/3681983500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a2c0ffe20 0x7f4a2c1097b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a31639640 1 -- 172.21.15.27:0/3681983500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a2c19f6e0 con 0x7f4a2c100f70 2024-04-25T00:12:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.062+0000 7f4a31e3a640 1 --2- 172.21.15.27:0/3681983500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a2c0ffe20 0x7f4a2c1097b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:12:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.063+0000 7f4a31639640 1 --2- 172.21.15.27:0/3681983500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a2c100f70 0x7f4a2c19c7b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00d950 tx=0x7f4a1c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.063+0000 7f4a22ffd640 1 -- 172.21.15.27:0/3681983500 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c017020 con 0x7f4a2c100f70 2024-04-25T00:12:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.063+0000 7f4a22ffd640 1 -- 172.21.15.27:0/3681983500 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a1c009c30 con 0x7f4a2c100f70 2024-04-25T00:12:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.063+0000 7f4a22ffd640 1 -- 172.21.15.27:0/3681983500 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c00f720 con 0x7f4a2c100f70 2024-04-25T00:12:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.063+0000 7f4a3409e640 1 -- 172.21.15.27:0/3681983500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a2c19f970 con 0x7f4a2c100f70 2024-04-25T00:12:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.063+0000 7f4a3409e640 1 -- 172.21.15.27:0/3681983500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a2c19ff10 con 0x7f4a2c100f70 2024-04-25T00:12:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.064+0000 7f4a3409e640 1 -- 172.21.15.27:0/3681983500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a2c1070d0 con 0x7f4a2c100f70 2024-04-25T00:12:54.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.065+0000 7f4a22ffd640 1 -- 172.21.15.27:0/3681983500 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4a1c01a020 con 0x7f4a2c100f70 2024-04-25T00:12:54.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.066+0000 7f4a22ffd640 1 --2- 172.21.15.27:0/3681983500 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a180a2c10 0x7f4a180a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:54.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.066+0000 7f4a31e3a640 1 --2- 172.21.15.27:0/3681983500 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a180a2c10 0x7f4a180a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:54.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.066+0000 7f4a22ffd640 1 -- 172.21.15.27:0/3681983500 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4a1c0cea80 con 0x7f4a2c100f70 2024-04-25T00:12:54.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.066+0000 7f4a31e3a640 1 --2- 172.21.15.27:0/3681983500 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a180a2c10 0x7f4a180a50d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4a28009b50 tx=0x7f4a28005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:54.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.070+0000 7f4a22ffd640 1 -- 172.21.15.27:0/3681983500 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4a1c021020 con 0x7f4a2c100f70 2024-04-25T00:12:54.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.293+0000 7f4a3409e640 1 -- 172.21.15.27:0/3681983500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a2c02d170 con 0x7f4a2c100f70 2024-04-25T00:12:54.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.294+0000 7f4a22ffd640 1 -- 172.21.15.27:0/3681983500 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4699 (secure 0 0 0) 0x7f4a1c009de0 con 0x7f4a2c100f70 2024-04-25T00:12:54.295 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:12:54.295 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-25T00:12:54.296 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 6 2024-04-25T00:12:54.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.295+0000 7f4a3409e640 1 -- 172.21.15.27:0/3681983500 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a180a2c10 msgr2=0x7f4a180a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:54.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.295+0000 7f4a3409e640 1 --2- 172.21.15.27:0/3681983500 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a180a2c10 0x7f4a180a50d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4a28009b50 tx=0x7f4a28005f90 comp rx=0 tx=0).stop 2024-04-25T00:12:54.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.296+0000 7f4a3409e640 1 -- 172.21.15.27:0/3681983500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a2c100f70 msgr2=0x7f4a2c19c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:54.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.296+0000 7f4a3409e640 1 --2- 172.21.15.27:0/3681983500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a2c100f70 0x7f4a2c19c7b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00d950 tx=0x7f4a1c00de10 comp rx=0 tx=0).stop 2024-04-25T00:12:54.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.296+0000 7f4a3409e640 1 -- 172.21.15.27:0/3681983500 shutdown_connections 2024-04-25T00:12:54.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.296+0000 7f4a3409e640 1 --2- 172.21.15.27:0/3681983500 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a180a2c10 0x7f4a180a50d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:54.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.296+0000 7f4a3409e640 1 --2- 172.21.15.27:0/3681983500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a2c101930 0x7f4a2c19edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:54.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.296+0000 7f4a3409e640 1 --2- 172.21.15.27:0/3681983500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a2c100f70 0x7f4a2c19c7b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:54.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.296+0000 7f4a3409e640 1 --2- 172.21.15.27:0/3681983500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a2c0ffe20 0x7f4a2c1097b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:54.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.296+0000 7f4a3409e640 1 -- 172.21.15.27:0/3681983500 >> 172.21.15.27:0/3681983500 conn(0x7f4a2c0fb3b0 msgr2=0x7f4a2c1042a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:54.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.296+0000 7f4a3409e640 1 -- 172.21.15.27:0/3681983500 shutdown_connections 2024-04-25T00:12:54.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:54.296+0000 7f4a3409e640 1 -- 172.21.15.27:0/3681983500 wait complete. 2024-04-25T00:12:54.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:54 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:54.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:54 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:54.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:54 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:54.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:54 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:54.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:54 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:54.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:54 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:54.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:54 smithi027 ceph-mon[29208]: Deploying daemon mds.f on smithi153 2024-04-25T00:12:54.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:54 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6834/3731284556,v1:172.21.15.139:6835/3731284556] up:boot 2024-04-25T00:12:54.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:54 smithi027 ceph-mon[29208]: fsmap 5 up:standby 2024-04-25T00:12:54.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:54 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T00:12:54.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:54 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3681983500' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:54.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:54 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:54.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:54 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:54.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:54 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:54.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:54 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:54.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:54 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:54.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:54 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:54.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:54 smithi139 ceph-mon[32833]: Deploying daemon mds.f on smithi153 2024-04-25T00:12:54.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:54 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6834/3731284556,v1:172.21.15.139:6835/3731284556] up:boot 2024-04-25T00:12:54.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:54 smithi139 ceph-mon[32833]: fsmap 5 up:standby 2024-04-25T00:12:54.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:54 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T00:12:54.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:54 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3681983500' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:54.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:54 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:54.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:54 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:54.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:54 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:54.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:54 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:54.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:54 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:54.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:54 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:54.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:54 smithi153 ceph-mon[32896]: Deploying daemon mds.f on smithi153 2024-04-25T00:12:54.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:54 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6834/3731284556,v1:172.21.15.139:6835/3731284556] up:boot 2024-04-25T00:12:54.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:54 smithi153 ceph-mon[32896]: fsmap 5 up:standby 2024-04-25T00:12:54.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:54 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T00:12:54.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:54 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3681983500' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:55.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:55 smithi027 ceph-mon[29208]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:55.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:55 smithi139 ceph-mon[32833]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:55.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:55 smithi153 ceph-mon[32896]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:55.777 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:12:55 smithi153 systemd[1]: Starting Ceph mds.f for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:12:56.098 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:12:55 smithi153 podman[63340]: 2024-04-25 00:12:55.68766916 +0000 UTC m=+0.018853504 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:12:56.099 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:12:55 smithi153 podman[63340]: 2024-04-25 00:12:55.824017426 +0000 UTC m=+0.155201770 container create fe42e8756c601342a29b81dd6154e1e625be3453b81afabbd5741faa749009a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T00:12:56.099 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:12:56 smithi153 podman[63340]: 2024-04-25 00:12:56.098896546 +0000 UTC m=+0.430080893 container init fe42e8756c601342a29b81dd6154e1e625be3453b81afabbd5741faa749009a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-25T00:12:56.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:56.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.424+0000 7f7eae2d4640 1 -- 172.21.15.27:0/2418563705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ea81017b0 msgr2=0x7f7ea8101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:56.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.424+0000 7f7eae2d4640 1 --2- 172.21.15.27:0/2418563705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ea81017b0 0x7f7ea8101bd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7e98009900 tx=0x7f7e9802a410 comp rx=0 tx=0).stop 2024-04-25T00:12:56.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.424+0000 7f7eae2d4640 1 -- 172.21.15.27:0/2418563705 shutdown_connections 2024-04-25T00:12:56.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.424+0000 7f7eae2d4640 1 --2- 172.21.15.27:0/2418563705 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ea8103360 0x7f7ea8109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:56.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.424+0000 7f7eae2d4640 1 --2- 172.21.15.27:0/2418563705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ea81029a0 0x7f7ea8102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:56.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.424+0000 7f7eae2d4640 1 --2- 172.21.15.27:0/2418563705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ea81017b0 0x7f7ea8101bd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:56.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.424+0000 7f7eae2d4640 1 -- 172.21.15.27:0/2418563705 >> 172.21.15.27:0/2418563705 conn(0x7f7ea80fcd40 msgr2=0x7f7ea80ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:56.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.424+0000 7f7eae2d4640 1 -- 172.21.15.27:0/2418563705 shutdown_connections 2024-04-25T00:12:56.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.424+0000 7f7eae2d4640 1 -- 172.21.15.27:0/2418563705 wait complete. 2024-04-25T00:12:56.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.425+0000 7f7eae2d4640 1 Processor -- start 2024-04-25T00:12:56.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.425+0000 7f7eae2d4640 1 -- start start 2024-04-25T00:12:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.425+0000 7f7eae2d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ea81017b0 0x7f7ea8197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.425+0000 7f7eae2d4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ea81029a0 0x7f7ea8197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.425+0000 7f7eae2d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ea8103360 0x7f7ea819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.425+0000 7f7eae2d4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea810d420 con 0x7f7ea8103360 2024-04-25T00:12:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.425+0000 7f7eae2d4640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea810d2a0 con 0x7f7ea81029a0 2024-04-25T00:12:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.425+0000 7f7eae2d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea810d5a0 con 0x7f7ea81017b0 2024-04-25T00:12:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.425+0000 7f7eac871640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ea8103360 0x7f7ea819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.425+0000 7f7eac871640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ea8103360 0x7f7ea819f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46528/0 (socket says 172.21.15.27:46528) 2024-04-25T00:12:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.425+0000 7f7eac871640 1 -- 172.21.15.27:0/2264580898 learned_addr learned my addr 172.21.15.27:0/2264580898 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:12:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.425+0000 7f7ea7fff640 1 --2- 172.21.15.27:0/2264580898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ea81017b0 0x7f7ea8197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.426+0000 7f7ea77fe640 1 --2- 172.21.15.27:0/2264580898 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ea81029a0 0x7f7ea8197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.426+0000 7f7eac871640 1 -- 172.21.15.27:0/2264580898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ea81017b0 msgr2=0x7f7ea8197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.426+0000 7f7eac871640 1 --2- 172.21.15.27:0/2264580898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ea81017b0 0x7f7ea8197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.426+0000 7f7eac871640 1 -- 172.21.15.27:0/2264580898 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ea81029a0 msgr2=0x7f7ea8197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.426+0000 7f7eac871640 1 --2- 172.21.15.27:0/2264580898 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ea81029a0 0x7f7ea8197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.426+0000 7f7eac871640 1 -- 172.21.15.27:0/2264580898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ea819f6d0 con 0x7f7ea8103360 2024-04-25T00:12:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.426+0000 7f7eac871640 1 --2- 172.21.15.27:0/2264580898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ea8103360 0x7f7ea819f060 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7ea000eb10 tx=0x7f7ea000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.426+0000 7f7ea57fa640 1 -- 172.21.15.27:0/2264580898 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0004b20 con 0x7f7ea8103360 2024-04-25T00:12:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.426+0000 7f7ea57fa640 1 -- 172.21.15.27:0/2264580898 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ea0004cc0 con 0x7f7ea8103360 2024-04-25T00:12:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.426+0000 7f7eae2d4640 1 -- 172.21.15.27:0/2264580898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ea819f9c0 con 0x7f7ea8103360 2024-04-25T00:12:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.426+0000 7f7ea57fa640 1 -- 172.21.15.27:0/2264580898 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0010760 con 0x7f7ea8103360 2024-04-25T00:12:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.426+0000 7f7eae2d4640 1 -- 172.21.15.27:0/2264580898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ea819fec0 con 0x7f7ea8103360 2024-04-25T00:12:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.427+0000 7f7eae2d4640 1 -- 172.21.15.27:0/2264580898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ea8108b70 con 0x7f7ea8103360 2024-04-25T00:12:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.428+0000 7f7ea57fa640 1 -- 172.21.15.27:0/2264580898 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7ea0005000 con 0x7f7ea8103360 2024-04-25T00:12:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.429+0000 7f7ea57fa640 1 --2- 172.21.15.27:0/2264580898 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7e900a2b90 0x7f7e900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.429+0000 7f7ea7fff640 1 --2- 172.21.15.27:0/2264580898 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7e900a2b90 0x7f7e900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.429+0000 7f7ea57fa640 1 -- 172.21.15.27:0/2264580898 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7ea00cf3f0 con 0x7f7ea8103360 2024-04-25T00:12:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.430+0000 7f7ea7fff640 1 --2- 172.21.15.27:0/2264580898 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7e900a2b90 0x7f7e900a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7e98005ab0 tx=0x7f7e98000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:56.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.433+0000 7f7ea57fa640 1 -- 172.21.15.27:0/2264580898 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7ea0053740 con 0x7f7ea8103360 2024-04-25T00:12:56.527 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:12:56 smithi153 podman[63340]: 2024-04-25 00:12:56.10157163 +0000 UTC m=+0.432755983 container start fe42e8756c601342a29b81dd6154e1e625be3453b81afabbd5741faa749009a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-25T00:12:56.527 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:12:56 smithi153 ceph-mds[63359]: -- 172.21.15.153:0/3500950901 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56034f27fa40 con 0x56034f252c00 2024-04-25T00:12:56.527 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:12:56 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-f[63355]: starting mds.f at 2024-04-25T00:12:56.527 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:12:56 smithi153 bash[63340]: fe42e8756c601342a29b81dd6154e1e625be3453b81afabbd5741faa749009a6 2024-04-25T00:12:56.527 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:12:56 smithi153 systemd[1]: Started Ceph mds.f for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:12:56.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.658+0000 7f7eae2d4640 1 -- 172.21.15.27:0/2264580898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ea802d3c0 con 0x7f7ea8103360 2024-04-25T00:12:56.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.659+0000 7f7ea57fa640 1 -- 172.21.15.27:0/2264580898 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7f7ea009c7f0 con 0x7f7ea8103360 2024-04-25T00:12:56.660 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:12:56.660 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-25T00:12:56.660 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2024-04-25T00:12:56.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.661+0000 7f7eae2d4640 1 -- 172.21.15.27:0/2264580898 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7e900a2b90 msgr2=0x7f7e900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:56.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.661+0000 7f7eae2d4640 1 --2- 172.21.15.27:0/2264580898 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7e900a2b90 0x7f7e900a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7e98005ab0 tx=0x7f7e98000bc0 comp rx=0 tx=0).stop 2024-04-25T00:12:56.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.661+0000 7f7eae2d4640 1 -- 172.21.15.27:0/2264580898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ea8103360 msgr2=0x7f7ea819f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:56.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.661+0000 7f7eae2d4640 1 --2- 172.21.15.27:0/2264580898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ea8103360 0x7f7ea819f060 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7ea000eb10 tx=0x7f7ea000c2d0 comp rx=0 tx=0).stop 2024-04-25T00:12:56.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.661+0000 7f7eae2d4640 1 -- 172.21.15.27:0/2264580898 shutdown_connections 2024-04-25T00:12:56.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.661+0000 7f7eae2d4640 1 --2- 172.21.15.27:0/2264580898 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7e900a2b90 0x7f7e900a5050 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:56.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.661+0000 7f7eae2d4640 1 --2- 172.21.15.27:0/2264580898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ea8103360 0x7f7ea819f060 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:56.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.661+0000 7f7eae2d4640 1 --2- 172.21.15.27:0/2264580898 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ea81029a0 0x7f7ea8197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:56.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.661+0000 7f7eae2d4640 1 --2- 172.21.15.27:0/2264580898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ea81017b0 0x7f7ea8197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:56.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.661+0000 7f7eae2d4640 1 -- 172.21.15.27:0/2264580898 >> 172.21.15.27:0/2264580898 conn(0x7f7ea80fcd40 msgr2=0x7f7ea8105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:56.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.661+0000 7f7eae2d4640 1 -- 172.21.15.27:0/2264580898 shutdown_connections 2024-04-25T00:12:56.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:56.661+0000 7f7eae2d4640 1 -- 172.21.15.27:0/2264580898 wait complete. 2024-04-25T00:12:57.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:57 smithi139 ceph-mon[32833]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:57.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:57 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:57.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:57 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:57.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:57 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:57.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:57 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:57.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:57 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:57.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:57 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:57.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:57 smithi139 ceph-mon[32833]: Deploying daemon mds.c on smithi153 2024-04-25T00:12:57.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:57 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.153:6834/584718807,v1:172.21.15.153:6835/584718807] up:boot 2024-04-25T00:12:57.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:57 smithi139 ceph-mon[32833]: fsmap 6 up:standby 2024-04-25T00:12:57.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:57 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T00:12:57.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:57 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2264580898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:57.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:57 smithi153 ceph-mon[32896]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:57.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:57 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:57.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:57 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:57.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:57 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:57.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:57 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:57.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:57 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:57.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:57 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:57.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:57 smithi153 ceph-mon[32896]: Deploying daemon mds.c on smithi153 2024-04-25T00:12:57.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:57 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.153:6834/584718807,v1:172.21.15.153:6835/584718807] up:boot 2024-04-25T00:12:57.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:57 smithi153 ceph-mon[32896]: fsmap 6 up:standby 2024-04-25T00:12:57.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:57 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T00:12:57.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:57 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2264580898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:57.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:57 smithi027 ceph-mon[29208]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:57.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:57 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:57.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:57 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:57.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:57 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:57.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:57 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:57.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:57 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:57.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:57 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:57.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:57 smithi027 ceph-mon[29208]: Deploying daemon mds.c on smithi153 2024-04-25T00:12:57.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:57 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.153:6834/584718807,v1:172.21.15.153:6835/584718807] up:boot 2024-04-25T00:12:57.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:57 smithi027 ceph-mon[29208]: fsmap 6 up:standby 2024-04-25T00:12:57.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:57 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T00:12:57.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:57 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2264580898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:58.526 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:12:58 smithi153 systemd[1]: Starting Ceph mds.c for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:12:58.674 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:58.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.789+0000 7fb530c1b640 1 -- 172.21.15.27:0/2087770088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c101890 msgr2=0x7fb52c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:58.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.789+0000 7fb530c1b640 1 --2- 172.21.15.27:0/2087770088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c101890 0x7fb52c101d10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb514009900 tx=0x7fb51402a410 comp rx=0 tx=0).stop 2024-04-25T00:12:58.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.789+0000 7fb530c1b640 1 -- 172.21.15.27:0/2087770088 shutdown_connections 2024-04-25T00:12:58.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.789+0000 7fb530c1b640 1 --2- 172.21.15.27:0/2087770088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c102250 0x7fb52c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:58.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.789+0000 7fb530c1b640 1 --2- 172.21.15.27:0/2087770088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c101890 0x7fb52c101d10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:58.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.789+0000 7fb530c1b640 1 --2- 172.21.15.27:0/2087770088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb52c1006a0 0x7fb52c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:58.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.789+0000 7fb530c1b640 1 -- 172.21.15.27:0/2087770088 >> 172.21.15.27:0/2087770088 conn(0x7fb52c0fbc30 msgr2=0x7fb52c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:58.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.789+0000 7fb530c1b640 1 -- 172.21.15.27:0/2087770088 shutdown_connections 2024-04-25T00:12:58.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.789+0000 7fb530c1b640 1 -- 172.21.15.27:0/2087770088 wait complete. 2024-04-25T00:12:58.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.790+0000 7fb530c1b640 1 Processor -- start 2024-04-25T00:12:58.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.790+0000 7fb530c1b640 1 -- start start 2024-04-25T00:12:58.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.790+0000 7fb530c1b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c1006a0 0x7fb52c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.790+0000 7fb530c1b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c101890 0x7fb52c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.790+0000 7fb530c1b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb52c102250 0x7fb52c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.790+0000 7fb530c1b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c10c370 con 0x7fb52c1006a0 2024-04-25T00:12:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.790+0000 7fb530c1b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c10c1f0 con 0x7fb52c102250 2024-04-25T00:12:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.790+0000 7fb530c1b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c10c4f0 con 0x7fb52c101890 2024-04-25T00:12:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.790+0000 7fb52a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c1006a0 0x7fb52c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.790+0000 7fb52a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c1006a0 0x7fb52c197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47938/0 (socket says 172.21.15.27:47938) 2024-04-25T00:12:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.790+0000 7fb52a59c640 1 -- 172.21.15.27:0/3223764305 learned_addr learned my addr 172.21.15.27:0/3223764305 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:12:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.790+0000 7fb529d9b640 1 --2- 172.21.15.27:0/3223764305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c101890 0x7fb52c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.790+0000 7fb52a59c640 1 -- 172.21.15.27:0/3223764305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c101890 msgr2=0x7fb52c197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.790+0000 7fb52a59c640 1 --2- 172.21.15.27:0/3223764305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c101890 0x7fb52c197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.790+0000 7fb52a59c640 1 -- 172.21.15.27:0/3223764305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb52c102250 msgr2=0x7fb52c19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:58.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.790+0000 7fb52ad9d640 1 --2- 172.21.15.27:0/3223764305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb52c102250 0x7fb52c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:58.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.791+0000 7fb52a59c640 1 --2- 172.21.15.27:0/3223764305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb52c102250 0x7fb52c19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:58.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.791+0000 7fb52a59c640 1 -- 172.21.15.27:0/3223764305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb52c19f6c0 con 0x7fb52c1006a0 2024-04-25T00:12:58.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.791+0000 7fb52ad9d640 1 --2- 172.21.15.27:0/3223764305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb52c102250 0x7fb52c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:12:58.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.791+0000 7fb52a59c640 1 --2- 172.21.15.27:0/3223764305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c1006a0 0x7fb52c197aa0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb51c00d770 tx=0x7fb51c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:58.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.791+0000 7fb5237fe640 1 -- 172.21.15.27:0/3223764305 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c017070 con 0x7fb52c1006a0 2024-04-25T00:12:58.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.791+0000 7fb530c1b640 1 -- 172.21.15.27:0/3223764305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb52c19f9b0 con 0x7fb52c1006a0 2024-04-25T00:12:58.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.791+0000 7fb5237fe640 1 -- 172.21.15.27:0/3223764305 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb51c009c30 con 0x7fb52c1006a0 2024-04-25T00:12:58.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.791+0000 7fb5237fe640 1 -- 172.21.15.27:0/3223764305 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c003dd0 con 0x7fb52c1006a0 2024-04-25T00:12:58.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.791+0000 7fb530c1b640 1 -- 172.21.15.27:0/3223764305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb52c19fed0 con 0x7fb52c1006a0 2024-04-25T00:12:58.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.792+0000 7fb530c1b640 1 -- 172.21.15.27:0/3223764305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb52c1a00a0 con 0x7fb52c1006a0 2024-04-25T00:12:58.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.793+0000 7fb5237fe640 1 -- 172.21.15.27:0/3223764305 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb51c003f70 con 0x7fb52c1006a0 2024-04-25T00:12:58.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.794+0000 7fb5237fe640 1 --2- 172.21.15.27:0/3223764305 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb5180a2b90 0x7fb5180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:58.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.794+0000 7fb529d9b640 1 --2- 172.21.15.27:0/3223764305 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb5180a2b90 0x7fb5180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:58.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.794+0000 7fb5237fe640 1 -- 172.21.15.27:0/3223764305 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb51c0ce970 con 0x7fb52c1006a0 2024-04-25T00:12:58.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.794+0000 7fb529d9b640 1 --2- 172.21.15.27:0/3223764305 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb5180a2b90 0x7fb5180a5050 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb514031440 tx=0x7fb5140315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:58.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:58.797+0000 7fb5237fe640 1 -- 172.21.15.27:0/3223764305 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb51c009de0 con 0x7fb52c1006a0 2024-04-25T00:12:58.861 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:12:58 smithi153 podman[63647]: 2024-04-25 00:12:58.473522314 +0000 UTC m=+0.018132741 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:12:58.861 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:12:58 smithi153 podman[63647]: 2024-04-25 00:12:58.592820369 +0000 UTC m=+0.137430833 container create 1da4ca3353f4fa644d85e23fff5e12eeca3a8c3b821020bfb1020d7364099f41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T00:12:58.861 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:12:58 smithi153 podman[63647]: 2024-04-25 00:12:58.861915892 +0000 UTC m=+0.406526311 container init 1da4ca3353f4fa644d85e23fff5e12eeca3a8c3b821020bfb1020d7364099f41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=) 2024-04-25T00:12:59.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:59.028+0000 7fb530c1b640 1 -- 172.21.15.27:0/3223764305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb52c1a0270 con 0x7fb52c1006a0 2024-04-25T00:12:59.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:59.028+0000 7fb5237fe640 1 -- 172.21.15.27:0/3223764305 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7fb51c09bd70 con 0x7fb52c1006a0 2024-04-25T00:12:59.029 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:12:59.029 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-25T00:12:59.030 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2024-04-25T00:12:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:59.031+0000 7fb530c1b640 1 -- 172.21.15.27:0/3223764305 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb5180a2b90 msgr2=0x7fb5180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:59.031+0000 7fb530c1b640 1 --2- 172.21.15.27:0/3223764305 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb5180a2b90 0x7fb5180a5050 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb514031440 tx=0x7fb5140315b0 comp rx=0 tx=0).stop 2024-04-25T00:12:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:59.031+0000 7fb530c1b640 1 -- 172.21.15.27:0/3223764305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c1006a0 msgr2=0x7fb52c197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:59.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:59.031+0000 7fb530c1b640 1 --2- 172.21.15.27:0/3223764305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c1006a0 0x7fb52c197aa0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb51c00d770 tx=0x7fb51c00dc30 comp rx=0 tx=0).stop 2024-04-25T00:12:59.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:59.031+0000 7fb530c1b640 1 -- 172.21.15.27:0/3223764305 shutdown_connections 2024-04-25T00:12:59.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:59.031+0000 7fb530c1b640 1 --2- 172.21.15.27:0/3223764305 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb5180a2b90 0x7fb5180a5050 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:59.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:59.031+0000 7fb530c1b640 1 --2- 172.21.15.27:0/3223764305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb52c102250 0x7fb52c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:59.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:59.031+0000 7fb530c1b640 1 --2- 172.21.15.27:0/3223764305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c101890 0x7fb52c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:59.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:59.031+0000 7fb530c1b640 1 --2- 172.21.15.27:0/3223764305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c1006a0 0x7fb52c197aa0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:59.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:59.031+0000 7fb530c1b640 1 -- 172.21.15.27:0/3223764305 >> 172.21.15.27:0/3223764305 conn(0x7fb52c0fbc30 msgr2=0x7fb52c0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:59.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:59.031+0000 7fb530c1b640 1 -- 172.21.15.27:0/3223764305 shutdown_connections 2024-04-25T00:12:59.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:12:59.031+0000 7fb530c1b640 1 -- 172.21.15.27:0/3223764305 wait complete. 2024-04-25T00:12:59.245 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:12:58 smithi153 podman[63647]: 2024-04-25 00:12:58.864644986 +0000 UTC m=+0.409255414 container start 1da4ca3353f4fa644d85e23fff5e12eeca3a8c3b821020bfb1020d7364099f41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-25T00:12:59.246 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:12:58 smithi153 ceph-mds[63666]: -- 172.21.15.153:0/4087543309 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560fd3877a40 con 0x560fd384a800 2024-04-25T00:12:59.246 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:12:58 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-c[63662]: starting mds.c at 2024-04-25T00:12:59.246 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:12:58 smithi153 bash[63647]: 1da4ca3353f4fa644d85e23fff5e12eeca3a8c3b821020bfb1020d7364099f41 2024-04-25T00:12:59.246 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:12:58 smithi153 systemd[1]: Started Ceph mds.c for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:12:59.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:59 smithi027 ceph-mon[29208]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:59.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:59.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:59 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3223764305' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:12:59 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:59.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:59 smithi139 ceph-mon[32833]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:59.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:59.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:59.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:59 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3223764305' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:59.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:59.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:59.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:59.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:12:59 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:12:59.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:59 smithi153 ceph-mon[32896]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:12:59.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:59.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:59.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:59 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3223764305' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:12:59.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:12:59.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:12:59.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:12:59.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:12:59 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:00.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:00 smithi139 ceph-mon[32833]: Deploying daemon mds.h on smithi139 2024-04-25T00:13:00.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:00 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] up:boot 2024-04-25T00:13:00.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:00 smithi139 ceph-mon[32833]: fsmap 7 up:standby 2024-04-25T00:13:00.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:00 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T00:13:00.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:00 smithi153 ceph-mon[32896]: Deploying daemon mds.h on smithi139 2024-04-25T00:13:00.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:00 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] up:boot 2024-04-25T00:13:00.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:00 smithi153 ceph-mon[32896]: fsmap 7 up:standby 2024-04-25T00:13:00.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:00 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T00:13:00.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:00 smithi027 ceph-mon[29208]: Deploying daemon mds.h on smithi139 2024-04-25T00:13:00.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:00 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] up:boot 2024-04-25T00:13:00.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:00 smithi027 ceph-mon[29208]: fsmap 7 up:standby 2024-04-25T00:13:00.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:00 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T00:13:01.043 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:01.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.157+0000 7f6374c46640 1 -- 172.21.15.27:0/631358965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6370101890 msgr2=0x7f6370101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:01.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.157+0000 7f6374c46640 1 --2- 172.21.15.27:0/631358965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6370101890 0x7f6370101d10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6364009900 tx=0x7f636402a410 comp rx=0 tx=0).stop 2024-04-25T00:13:01.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.157+0000 7f6374c46640 1 -- 172.21.15.27:0/631358965 shutdown_connections 2024-04-25T00:13:01.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.157+0000 7f6374c46640 1 --2- 172.21.15.27:0/631358965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6370102250 0x7f6370108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:01.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.157+0000 7f6374c46640 1 --2- 172.21.15.27:0/631358965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6370101890 0x7f6370101d10 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:01.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.157+0000 7f6374c46640 1 --2- 172.21.15.27:0/631358965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f63701006a0 0x7f6370100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:01.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.157+0000 7f6374c46640 1 -- 172.21.15.27:0/631358965 >> 172.21.15.27:0/631358965 conn(0x7f63700fbc30 msgr2=0x7f63700fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:01.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.157+0000 7f6374c46640 1 -- 172.21.15.27:0/631358965 shutdown_connections 2024-04-25T00:13:01.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.157+0000 7f6374c46640 1 -- 172.21.15.27:0/631358965 wait complete. 2024-04-25T00:13:01.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.158+0000 7f6374c46640 1 Processor -- start 2024-04-25T00:13:01.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.158+0000 7f6374c46640 1 -- start start 2024-04-25T00:13:01.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.158+0000 7f6374c46640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63701006a0 0x7f637006a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:01.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.158+0000 7f6374c46640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6370101890 0x7f637019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:01.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.158+0000 7f6374c46640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6370102250 0x7f637019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:01.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.158+0000 7f6374c46640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f637010c380 con 0x7f63701006a0 2024-04-25T00:13:01.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.158+0000 7f6374c46640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f637010c200 con 0x7f6370101890 2024-04-25T00:13:01.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.158+0000 7f6374c46640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f637010c500 con 0x7f6370102250 2024-04-25T00:13:01.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.158+0000 7f636e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63701006a0 0x7f637006a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:01.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.159+0000 7f636e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63701006a0 0x7f637006a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47948/0 (socket says 172.21.15.27:47948) 2024-04-25T00:13:01.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.159+0000 7f636e59c640 1 -- 172.21.15.27:0/888934507 learned_addr learned my addr 172.21.15.27:0/888934507 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:01.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.159+0000 7f636dd9b640 1 --2- 172.21.15.27:0/888934507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6370101890 0x7f637019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:01.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.159+0000 7f636ed9d640 1 --2- 172.21.15.27:0/888934507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6370102250 0x7f637019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:01.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.159+0000 7f636e59c640 1 -- 172.21.15.27:0/888934507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6370102250 msgr2=0x7f637019edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:01.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.159+0000 7f636e59c640 1 --2- 172.21.15.27:0/888934507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6370102250 0x7f637019edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:01.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.159+0000 7f636e59c640 1 -- 172.21.15.27:0/888934507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6370101890 msgr2=0x7f637019c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:01.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.159+0000 7f636e59c640 1 --2- 172.21.15.27:0/888934507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6370101890 0x7f637019c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:01.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.159+0000 7f636e59c640 1 -- 172.21.15.27:0/888934507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637019f690 con 0x7f63701006a0 2024-04-25T00:13:01.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.159+0000 7f636e59c640 1 --2- 172.21.15.27:0/888934507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63701006a0 0x7f637006a930 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f636000c970 tx=0x7f636000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:01.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.159+0000 7f635f7fe640 1 -- 172.21.15.27:0/888934507 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6360007c10 con 0x7f63701006a0 2024-04-25T00:13:01.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.159+0000 7f6374c46640 1 -- 172.21.15.27:0/888934507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f637019f980 con 0x7f63701006a0 2024-04-25T00:13:01.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.159+0000 7f635f7fe640 1 -- 172.21.15.27:0/888934507 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6360007db0 con 0x7f63701006a0 2024-04-25T00:13:01.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.159+0000 7f635f7fe640 1 -- 172.21.15.27:0/888934507 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636000f940 con 0x7f63701006a0 2024-04-25T00:13:01.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.159+0000 7f6374c46640 1 -- 172.21.15.27:0/888934507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f637019ff20 con 0x7f63701006a0 2024-04-25T00:13:01.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.160+0000 7f6374c46640 1 -- 172.21.15.27:0/888934507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63701079c0 con 0x7f63701006a0 2024-04-25T00:13:01.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.162+0000 7f635f7fe640 1 -- 172.21.15.27:0/888934507 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f636000fae0 con 0x7f63701006a0 2024-04-25T00:13:01.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.162+0000 7f635f7fe640 1 --2- 172.21.15.27:0/888934507 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f63580a2b90 0x7f63580a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:01.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.163+0000 7f636dd9b640 1 --2- 172.21.15.27:0/888934507 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f63580a2b90 0x7f63580a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:01.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.163+0000 7f635f7fe640 1 -- 172.21.15.27:0/888934507 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f63600cef90 con 0x7f63701006a0 2024-04-25T00:13:01.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.163+0000 7f636dd9b640 1 --2- 172.21.15.27:0/888934507 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f63580a2b90 0x7f63580a5050 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f63640314e0 tx=0x7f6364000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:01.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.167+0000 7f635f7fe640 1 -- 172.21.15.27:0/888934507 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f636009c420 con 0x7f63701006a0 2024-04-25T00:13:01.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.389+0000 7f6374c46640 1 -- 172.21.15.27:0/888934507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6370062340 con 0x7f63701006a0 2024-04-25T00:13:01.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.390+0000 7f635f7fe640 1 -- 172.21.15.27:0/888934507 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7f636009bd20 con 0x7f63701006a0 2024-04-25T00:13:01.390 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:01.390 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-25T00:13:01.390 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-04-25T00:13:01.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.392+0000 7f6374c46640 1 -- 172.21.15.27:0/888934507 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f63580a2b90 msgr2=0x7f63580a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:01.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.392+0000 7f6374c46640 1 --2- 172.21.15.27:0/888934507 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f63580a2b90 0x7f63580a5050 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f63640314e0 tx=0x7f6364000bc0 comp rx=0 tx=0).stop 2024-04-25T00:13:01.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.392+0000 7f6374c46640 1 -- 172.21.15.27:0/888934507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63701006a0 msgr2=0x7f637006a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:01.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.392+0000 7f6374c46640 1 --2- 172.21.15.27:0/888934507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63701006a0 0x7f637006a930 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f636000c970 tx=0x7f636000ce30 comp rx=0 tx=0).stop 2024-04-25T00:13:01.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.392+0000 7f6374c46640 1 -- 172.21.15.27:0/888934507 shutdown_connections 2024-04-25T00:13:01.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.392+0000 7f6374c46640 1 --2- 172.21.15.27:0/888934507 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f63580a2b90 0x7f63580a5050 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:01.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.392+0000 7f6374c46640 1 --2- 172.21.15.27:0/888934507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6370102250 0x7f637019edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:01.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.392+0000 7f6374c46640 1 --2- 172.21.15.27:0/888934507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6370101890 0x7f637019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:01.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.392+0000 7f6374c46640 1 --2- 172.21.15.27:0/888934507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63701006a0 0x7f637006a930 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:01.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.392+0000 7f6374c46640 1 -- 172.21.15.27:0/888934507 >> 172.21.15.27:0/888934507 conn(0x7f63700fbc30 msgr2=0x7f6370104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:01.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.392+0000 7f6374c46640 1 -- 172.21.15.27:0/888934507 shutdown_connections 2024-04-25T00:13:01.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:01.392+0000 7f6374c46640 1 -- 172.21.15.27:0/888934507 wait complete. 2024-04-25T00:13:01.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:01 smithi027 ceph-mon[29208]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:01.461 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:01 smithi139 ceph-mon[32833]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:01.462 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:13:01 smithi139 systemd[1]: Starting Ceph mds.h for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:13:01.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:01 smithi153 ceph-mon[32896]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:01.753 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:13:01 smithi139 podman[62664]: 2024-04-25 00:13:01.414737841 +0000 UTC m=+0.017806533 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:13:01.753 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:13:01 smithi139 podman[62664]: 2024-04-25 00:13:01.541120372 +0000 UTC m=+0.144189093 container create 82223188b6cc765aff24143766a63beef588e22b935024d3b8883cf276923378 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-25T00:13:02.253 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:13:01 smithi139 podman[62664]: 2024-04-25 00:13:01.81029899 +0000 UTC m=+0.413367671 container init 82223188b6cc765aff24143766a63beef588e22b935024d3b8883cf276923378 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T00:13:02.253 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:13:01 smithi139 podman[62664]: 2024-04-25 00:13:01.813012088 +0000 UTC m=+0.416080781 container start 82223188b6cc765aff24143766a63beef588e22b935024d3b8883cf276923378 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T00:13:02.253 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:13:01 smithi139 ceph-mds[62683]: -- 172.21.15.139:0/4202680918 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fe71f89a40 con 0x55fe71f5d000 2024-04-25T00:13:02.254 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:13:01 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-h[62679]: starting mds.h at 2024-04-25T00:13:02.254 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:13:01 smithi139 bash[62664]: 82223188b6cc765aff24143766a63beef588e22b935024d3b8883cf276923378 2024-04-25T00:13:02.254 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:13:01 smithi139 systemd[1]: Started Ceph mds.h for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:13:02.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:02 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/888934507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:02.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:02 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:02.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:02 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:02.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:02 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:02.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:02 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:13:02.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:02 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:13:02.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:02 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:02.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:02 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/888934507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:02.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:02 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:02.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:02 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:02.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:02 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:02.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:02 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:13:02.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:02 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:13:02.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:02 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:02.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:02 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/888934507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:02.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:02 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:02.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:02 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:02.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:02 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:02.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:02 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:13:02.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:02 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:13:02.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:02 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:03.404 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.520+0000 7f037ff70640 1 -- 172.21.15.27:0/3443201553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03781006a0 msgr2=0x7f0378100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.520+0000 7f037ff70640 1 --2- 172.21.15.27:0/3443201553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03781006a0 0x7f0378100ac0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0368009900 tx=0x7f036802a410 comp rx=0 tx=0).stop 2024-04-25T00:13:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.520+0000 7f037ff70640 1 -- 172.21.15.27:0/3443201553 shutdown_connections 2024-04-25T00:13:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.520+0000 7f037ff70640 1 --2- 172.21.15.27:0/3443201553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0378102250 0x7f0378108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.520+0000 7f037ff70640 1 --2- 172.21.15.27:0/3443201553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0378101890 0x7f0378101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.520+0000 7f037ff70640 1 --2- 172.21.15.27:0/3443201553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03781006a0 0x7f0378100ac0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.520+0000 7f037ff70640 1 -- 172.21.15.27:0/3443201553 >> 172.21.15.27:0/3443201553 conn(0x7f03780fbc30 msgr2=0x7f03780fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.520+0000 7f037ff70640 1 -- 172.21.15.27:0/3443201553 shutdown_connections 2024-04-25T00:13:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.520+0000 7f037ff70640 1 -- 172.21.15.27:0/3443201553 wait complete. 2024-04-25T00:13:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037ff70640 1 Processor -- start 2024-04-25T00:13:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037ff70640 1 -- start start 2024-04-25T00:13:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037ff70640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03781006a0 0x7f0378197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037ff70640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0378101890 0x7f0378197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037ff70640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0378102250 0x7f037819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037ff70640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f037810c320 con 0x7f0378101890 2024-04-25T00:13:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037ff70640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f037810c1a0 con 0x7f0378102250 2024-04-25T00:13:03.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037ff70640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f037810c4a0 con 0x7f03781006a0 2024-04-25T00:13:03.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037d50b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0378101890 0x7f0378197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:03.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037d50b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0378101890 0x7f0378197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47956/0 (socket says 172.21.15.27:47956) 2024-04-25T00:13:03.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037d50b640 1 -- 172.21.15.27:0/1604854529 learned_addr learned my addr 172.21.15.27:0/1604854529 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:03.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037dd0c640 1 --2- 172.21.15.27:0/1604854529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03781006a0 0x7f0378197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:03.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037e50d640 1 --2- 172.21.15.27:0/1604854529 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0378102250 0x7f037819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:03.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037d50b640 1 -- 172.21.15.27:0/1604854529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03781006a0 msgr2=0x7f0378197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:03.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037d50b640 1 --2- 172.21.15.27:0/1604854529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03781006a0 0x7f0378197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:03.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037d50b640 1 -- 172.21.15.27:0/1604854529 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0378102250 msgr2=0x7f037819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:03.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037d50b640 1 --2- 172.21.15.27:0/1604854529 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0378102250 0x7f037819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:03.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.521+0000 7f037d50b640 1 -- 172.21.15.27:0/1604854529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037819f6d0 con 0x7f0378101890 2024-04-25T00:13:03.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.522+0000 7f037d50b640 1 --2- 172.21.15.27:0/1604854529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0378101890 0x7f0378197ff0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f037400e9f0 tx=0x7f037400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:03.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.522+0000 7f036effd640 1 -- 172.21.15.27:0/1604854529 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037400cd80 con 0x7f0378101890 2024-04-25T00:13:03.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.522+0000 7f036effd640 1 -- 172.21.15.27:0/1604854529 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0374010910 con 0x7f0378101890 2024-04-25T00:13:03.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.522+0000 7f037ff70640 1 -- 172.21.15.27:0/1604854529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f037819f9c0 con 0x7f0378101890 2024-04-25T00:13:03.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.522+0000 7f036effd640 1 -- 172.21.15.27:0/1604854529 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0374018990 con 0x7f0378101890 2024-04-25T00:13:03.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.522+0000 7f037ff70640 1 -- 172.21.15.27:0/1604854529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f037819ff60 con 0x7f0378101890 2024-04-25T00:13:03.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.523+0000 7f037ff70640 1 -- 172.21.15.27:0/1604854529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03781a0190 con 0x7f0378101890 2024-04-25T00:13:03.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.524+0000 7f036effd640 1 -- 172.21.15.27:0/1604854529 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0374010ac0 con 0x7f0378101890 2024-04-25T00:13:03.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.525+0000 7f036effd640 1 --2- 172.21.15.27:0/1604854529 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03640a2c10 0x7f03640a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:03.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.525+0000 7f036effd640 1 -- 172.21.15.27:0/1604854529 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f03740cee10 con 0x7f0378101890 2024-04-25T00:13:03.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.525+0000 7f037dd0c640 1 --2- 172.21.15.27:0/1604854529 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03640a2c10 0x7f03640a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:03.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:03 smithi153 ceph-mon[32896]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:03.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:03 smithi153 ceph-mon[32896]: Deploying daemon mds.i on smithi153 2024-04-25T00:13:03.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:03 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:boot 2024-04-25T00:13:03.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:03 smithi153 ceph-mon[32896]: fsmap 8 up:standby 2024-04-25T00:13:03.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:03 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T00:13:03.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.525+0000 7f037dd0c640 1 --2- 172.21.15.27:0/1604854529 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03640a2c10 0x7f03640a50d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0368005ab0 tx=0x7f0368000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:03.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.529+0000 7f036effd640 1 -- 172.21.15.27:0/1604854529 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f037409bbc0 con 0x7f0378101890 2024-04-25T00:13:03.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:03 smithi027 ceph-mon[29208]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:03.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:03 smithi027 ceph-mon[29208]: Deploying daemon mds.i on smithi153 2024-04-25T00:13:03.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:03 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:boot 2024-04-25T00:13:03.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:03 smithi027 ceph-mon[29208]: fsmap 8 up:standby 2024-04-25T00:13:03.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:03 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T00:13:03.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.753+0000 7f037ff70640 1 -- 172.21.15.27:0/1604854529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03781a0360 con 0x7f0378101890 2024-04-25T00:13:03.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:03 smithi139 ceph-mon[32833]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:03.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:03 smithi139 ceph-mon[32833]: Deploying daemon mds.i on smithi153 2024-04-25T00:13:03.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:03 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:boot 2024-04-25T00:13:03.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:03 smithi139 ceph-mon[32833]: fsmap 8 up:standby 2024-04-25T00:13:03.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:03 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T00:13:03.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.753+0000 7f036effd640 1 -- 172.21.15.27:0/1604854529 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7180 (secure 0 0 0) 0x7f037409fec0 con 0x7f0378101890 2024-04-25T00:13:03.754 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:03.754 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-25T00:13:03.754 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 9 2024-04-25T00:13:03.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.755+0000 7f037ff70640 1 -- 172.21.15.27:0/1604854529 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03640a2c10 msgr2=0x7f03640a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:03.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.755+0000 7f037ff70640 1 --2- 172.21.15.27:0/1604854529 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03640a2c10 0x7f03640a50d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0368005ab0 tx=0x7f0368000bc0 comp rx=0 tx=0).stop 2024-04-25T00:13:03.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.755+0000 7f037ff70640 1 -- 172.21.15.27:0/1604854529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0378101890 msgr2=0x7f0378197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:03.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.755+0000 7f037ff70640 1 --2- 172.21.15.27:0/1604854529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0378101890 0x7f0378197ff0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f037400e9f0 tx=0x7f037400eeb0 comp rx=0 tx=0).stop 2024-04-25T00:13:03.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.755+0000 7f037ff70640 1 -- 172.21.15.27:0/1604854529 shutdown_connections 2024-04-25T00:13:03.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.755+0000 7f037ff70640 1 --2- 172.21.15.27:0/1604854529 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03640a2c10 0x7f03640a50d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:03.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.755+0000 7f037ff70640 1 --2- 172.21.15.27:0/1604854529 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0378102250 0x7f037819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:03.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.755+0000 7f037ff70640 1 --2- 172.21.15.27:0/1604854529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0378101890 0x7f0378197ff0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:03.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.755+0000 7f037ff70640 1 --2- 172.21.15.27:0/1604854529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03781006a0 0x7f0378197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:03.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.755+0000 7f037ff70640 1 -- 172.21.15.27:0/1604854529 >> 172.21.15.27:0/1604854529 conn(0x7f03780fbc30 msgr2=0x7f03780fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:03.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.756+0000 7f037ff70640 1 -- 172.21.15.27:0/1604854529 shutdown_connections 2024-04-25T00:13:03.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:03.756+0000 7f037ff70640 1 -- 172.21.15.27:0/1604854529 wait complete. 2024-04-25T00:13:04.264 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:13:04 smithi153 systemd[1]: Starting Ceph mds.i for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:13:04.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:04 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1604854529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:04.522 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:13:04 smithi153 podman[63954]: 2024-04-25 00:13:04.267296302 +0000 UTC m=+0.018348772 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:13:04.522 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:13:04 smithi153 podman[63954]: 2024-04-25 00:13:04.387819285 +0000 UTC m=+0.138871736 container create 4a115f16103e9e64e5a69a59c161aa21f323a121d19946a3a42de572e80940c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T00:13:04.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:04 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1604854529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:04.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:04 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1604854529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:05.026 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:13:04 smithi153 podman[63954]: 2024-04-25 00:13:04.700737129 +0000 UTC m=+0.451789580 container init 4a115f16103e9e64e5a69a59c161aa21f323a121d19946a3a42de572e80940c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T00:13:05.027 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:13:04 smithi153 podman[63954]: 2024-04-25 00:13:04.703469508 +0000 UTC m=+0.454521962 container start 4a115f16103e9e64e5a69a59c161aa21f323a121d19946a3a42de572e80940c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T00:13:05.027 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:13:04 smithi153 ceph-mds[63973]: -- 172.21.15.153:0/1460262008 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558170e3da40 con 0x558170e11000 2024-04-25T00:13:05.027 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:13:04 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-i[63969]: starting mds.i at 2024-04-25T00:13:05.027 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:13:04 smithi153 bash[63954]: 4a115f16103e9e64e5a69a59c161aa21f323a121d19946a3a42de572e80940c3 2024-04-25T00:13:05.027 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:13:04 smithi153 systemd[1]: Started Ceph mds.i for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:13:05.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:05 smithi153 ceph-mon[32896]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:05.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:05 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:05.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:05 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:05.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:05 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:05.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:05 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:13:05.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:05 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:13:05.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:05 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:05.544 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:05 smithi027 ceph-mon[29208]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:05.544 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:05 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:05.544 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:05 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:05.544 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:05 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:05.544 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:05 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:13:05.544 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:05 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:13:05.544 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:05 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:05.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:05 smithi139 ceph-mon[32833]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:05.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:05 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:05.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:05 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:05.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:05 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:05.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:05 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:13:05.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:05 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:13:05.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:05 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:05.767 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.876+0000 7f414cdf9640 1 -- 172.21.15.27:0/1610621730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148101870 msgr2=0x7f4148101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.877+0000 7f414cdf9640 1 --2- 172.21.15.27:0/1610621730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148101870 0x7f4148101cf0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f413c009900 tx=0x7f413c02a410 comp rx=0 tx=0).stop 2024-04-25T00:13:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.877+0000 7f414cdf9640 1 -- 172.21.15.27:0/1610621730 shutdown_connections 2024-04-25T00:13:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.877+0000 7f414cdf9640 1 --2- 172.21.15.27:0/1610621730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4148102230 0x7f4148108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.877+0000 7f414cdf9640 1 --2- 172.21.15.27:0/1610621730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148101870 0x7f4148101cf0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.877+0000 7f414cdf9640 1 --2- 172.21.15.27:0/1610621730 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4148100680 0x7f4148100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.877+0000 7f414cdf9640 1 -- 172.21.15.27:0/1610621730 >> 172.21.15.27:0/1610621730 conn(0x7f41480fbbf0 msgr2=0x7f41480fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.877+0000 7f414cdf9640 1 -- 172.21.15.27:0/1610621730 shutdown_connections 2024-04-25T00:13:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.877+0000 7f414cdf9640 1 -- 172.21.15.27:0/1610621730 wait complete. 2024-04-25T00:13:05.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.877+0000 7f414cdf9640 1 Processor -- start 2024-04-25T00:13:05.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f414cdf9640 1 -- start start 2024-04-25T00:13:05.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f414cdf9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4148100680 0x7f4148197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:05.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f414cdf9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148101870 0x7f4148197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:05.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f414cdf9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4148102230 0x7f414819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:05.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f414cdf9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f414810c2e0 con 0x7f4148101870 2024-04-25T00:13:05.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f414cdf9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f414810c160 con 0x7f4148100680 2024-04-25T00:13:05.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f414cdf9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f414810c460 con 0x7f4148102230 2024-04-25T00:13:05.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f4145d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148101870 0x7f4148197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:05.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f4145d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148101870 0x7f4148197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47980/0 (socket says 172.21.15.27:47980) 2024-04-25T00:13:05.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f4145d9b640 1 -- 172.21.15.27:0/3596860293 learned_addr learned my addr 172.21.15.27:0/3596860293 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f414659c640 1 --2- 172.21.15.27:0/3596860293 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4148100680 0x7f4148197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f4146d9d640 1 --2- 172.21.15.27:0/3596860293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4148102230 0x7f414819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f4145d9b640 1 -- 172.21.15.27:0/3596860293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4148102230 msgr2=0x7f414819f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f4145d9b640 1 --2- 172.21.15.27:0/3596860293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4148102230 0x7f414819f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f4145d9b640 1 -- 172.21.15.27:0/3596860293 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4148100680 msgr2=0x7f4148197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f4145d9b640 1 --2- 172.21.15.27:0/3596860293 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4148100680 0x7f4148197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f4145d9b640 1 -- 172.21.15.27:0/3596860293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f414819f670 con 0x7f4148101870 2024-04-25T00:13:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f414659c640 1 --2- 172.21.15.27:0/3596860293 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4148100680 0x7f4148197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.878+0000 7f4146d9d640 1 --2- 172.21.15.27:0/3596860293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4148102230 0x7f414819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.879+0000 7f4145d9b640 1 --2- 172.21.15.27:0/3596860293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148101870 0x7f4148197f90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f413c009230 tx=0x7f413c004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.879+0000 7f41377fe640 1 -- 172.21.15.27:0/3596860293 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413c03f030 con 0x7f4148101870 2024-04-25T00:13:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.879+0000 7f414cdf9640 1 -- 172.21.15.27:0/3596860293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f414819f900 con 0x7f4148101870 2024-04-25T00:13:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.879+0000 7f41377fe640 1 -- 172.21.15.27:0/3596860293 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f413c003ce0 con 0x7f4148101870 2024-04-25T00:13:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.879+0000 7f41377fe640 1 -- 172.21.15.27:0/3596860293 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413c035070 con 0x7f4148101870 2024-04-25T00:13:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.879+0000 7f414cdf9640 1 -- 172.21.15.27:0/3596860293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f414819fe40 con 0x7f4148101870 2024-04-25T00:13:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.880+0000 7f414cdf9640 1 -- 172.21.15.27:0/3596860293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41481a0070 con 0x7f4148101870 2024-04-25T00:13:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.882+0000 7f41377fe640 1 -- 172.21.15.27:0/3596860293 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f413c042030 con 0x7f4148101870 2024-04-25T00:13:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.883+0000 7f41377fe640 1 --2- 172.21.15.27:0/3596860293 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f41300a2c10 0x7f41300a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.883+0000 7f41377fe640 1 -- 172.21.15.27:0/3596860293 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f413c0ef5c0 con 0x7f4148101870 2024-04-25T00:13:05.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.883+0000 7f414659c640 1 --2- 172.21.15.27:0/3596860293 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f41300a2c10 0x7f41300a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:05.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.883+0000 7f414659c640 1 --2- 172.21.15.27:0/3596860293 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f41300a2c10 0x7f41300a50d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f41481016d0 tx=0x7f4138005e70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:05.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:05.886+0000 7f41377fe640 1 -- 172.21.15.27:0/3596860293 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f413c0bc9c0 con 0x7f4148101870 2024-04-25T00:13:06.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:06.114+0000 7f414cdf9640 1 -- 172.21.15.27:0/3596860293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41481a0240 con 0x7f4148101870 2024-04-25T00:13:06.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:06.114+0000 7f41377fe640 1 -- 172.21.15.27:0/3596860293 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8010 (secure 0 0 0) 0x7f413c0bc2c0 con 0x7f4148101870 2024-04-25T00:13:06.115 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:06.115 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24514,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T00:13:06.115 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 10 2024-04-25T00:13:06.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:06.116+0000 7f414cdf9640 1 -- 172.21.15.27:0/3596860293 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f41300a2c10 msgr2=0x7f41300a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:06.116+0000 7f414cdf9640 1 --2- 172.21.15.27:0/3596860293 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f41300a2c10 0x7f41300a50d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f41481016d0 tx=0x7f4138005e70 comp rx=0 tx=0).stop 2024-04-25T00:13:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:06.116+0000 7f414cdf9640 1 -- 172.21.15.27:0/3596860293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148101870 msgr2=0x7f4148197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:06.116+0000 7f414cdf9640 1 --2- 172.21.15.27:0/3596860293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148101870 0x7f4148197f90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f413c009230 tx=0x7f413c004360 comp rx=0 tx=0).stop 2024-04-25T00:13:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:06.116+0000 7f414cdf9640 1 -- 172.21.15.27:0/3596860293 shutdown_connections 2024-04-25T00:13:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:06.116+0000 7f414cdf9640 1 --2- 172.21.15.27:0/3596860293 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f41300a2c10 0x7f41300a50d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:06.116+0000 7f414cdf9640 1 --2- 172.21.15.27:0/3596860293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4148102230 0x7f414819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:06.116+0000 7f414cdf9640 1 --2- 172.21.15.27:0/3596860293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148101870 0x7f4148197f90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:06.116+0000 7f414cdf9640 1 --2- 172.21.15.27:0/3596860293 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4148100680 0x7f4148197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:06.116+0000 7f414cdf9640 1 -- 172.21.15.27:0/3596860293 >> 172.21.15.27:0/3596860293 conn(0x7f41480fbbf0 msgr2=0x7f41480688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:06.117+0000 7f414cdf9640 1 -- 172.21.15.27:0/3596860293 shutdown_connections 2024-04-25T00:13:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:06.117+0000 7f414cdf9640 1 -- 172.21.15.27:0/3596860293 wait complete. 2024-04-25T00:13:06.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:06 smithi027 ceph-mon[29208]: Deploying daemon mds.g on smithi027 2024-04-25T00:13:06.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:06 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] up:boot 2024-04-25T00:13:06.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:06 smithi027 ceph-mon[29208]: fsmap 9 up:standby 2024-04-25T00:13:06.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:06 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T00:13:06.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:06 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3596860293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:06.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:06 smithi153 ceph-mon[32896]: Deploying daemon mds.g on smithi027 2024-04-25T00:13:06.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:06 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] up:boot 2024-04-25T00:13:06.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:06 smithi153 ceph-mon[32896]: fsmap 9 up:standby 2024-04-25T00:13:06.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:06 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T00:13:06.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:06 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3596860293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:06.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:06 smithi139 ceph-mon[32833]: Deploying daemon mds.g on smithi027 2024-04-25T00:13:06.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:06 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] up:boot 2024-04-25T00:13:06.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:06 smithi139 ceph-mon[32833]: fsmap 9 up:standby 2024-04-25T00:13:06.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:06 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T00:13:06.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:06 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3596860293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:07.192 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:13:07 smithi027 systemd[1]: Starting Ceph mds.g for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:13:07.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:07 smithi027 ceph-mon[29208]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:07.576 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:13:07 smithi027 podman[67489]: 2024-04-25 00:13:07.189142073 +0000 UTC m=+0.018858999 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:13:07.577 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:13:07 smithi027 podman[67489]: 2024-04-25 00:13:07.307310822 +0000 UTC m=+0.137027792 container create 8ec929729bb67d74985e98123c026f82740492c4083966af08110725f203f701 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-g, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T00:13:07.577 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:13:07 smithi027 podman[67489]: 2024-04-25 00:13:07.576824782 +0000 UTC m=+0.406541706 container init 8ec929729bb67d74985e98123c026f82740492c4083966af08110725f203f701 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-g, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-25T00:13:07.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:07 smithi139 ceph-mon[32833]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:07.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:07 smithi153 ceph-mon[32896]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:07.942 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:13:07 smithi027 podman[67489]: 2024-04-25 00:13:07.579858345 +0000 UTC m=+0.409575275 container start 8ec929729bb67d74985e98123c026f82740492c4083966af08110725f203f701 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:13:07.942 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:13:07 smithi027 ceph-mds[67508]: -- 172.21.15.27:0/1324641449 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5577f65ffa40 con 0x5577f65d3000 2024-04-25T00:13:07.942 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:13:07 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-g[67504]: starting mds.g at 2024-04-25T00:13:07.943 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:13:07 smithi027 bash[67489]: 8ec929729bb67d74985e98123c026f82740492c4083966af08110725f203f701 2024-04-25T00:13:07.943 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:13:07 smithi027 systemd[1]: Started Ceph mds.g for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:13:08.128 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.237+0000 7f8135f09640 1 -- 172.21.15.27:0/3683090099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8130100a30 msgr2=0x7f8130100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:08.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.237+0000 7f8135f09640 1 --2- 172.21.15.27:0/3683090099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8130100a30 0x7f8130100e90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8124009900 tx=0x7f812402a410 comp rx=0 tx=0).stop 2024-04-25T00:13:08.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.237+0000 7f8135f09640 1 -- 172.21.15.27:0/3683090099 shutdown_connections 2024-04-25T00:13:08.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.237+0000 7f8135f09640 1 --2- 172.21.15.27:0/3683090099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81301013d0 0x7f8130101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:08.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.237+0000 7f8135f09640 1 --2- 172.21.15.27:0/3683090099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8130100a30 0x7f8130100e90 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:08.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.237+0000 7f8135f09640 1 --2- 172.21.15.27:0/3683090099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f81301000d0 0x7f81301004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:08.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.237+0000 7f8135f09640 1 -- 172.21.15.27:0/3683090099 >> 172.21.15.27:0/3683090099 conn(0x7f81300fbc90 msgr2=0x7f81300fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:08.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.237+0000 7f8135f09640 1 -- 172.21.15.27:0/3683090099 shutdown_connections 2024-04-25T00:13:08.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.237+0000 7f8135f09640 1 -- 172.21.15.27:0/3683090099 wait complete. 2024-04-25T00:13:08.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.238+0000 7f8135f09640 1 Processor -- start 2024-04-25T00:13:08.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.238+0000 7f8135f09640 1 -- start start 2024-04-25T00:13:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.238+0000 7f8135f09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81301000d0 0x7f8130199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.238+0000 7f8135f09640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8130100a30 0x7f813019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.238+0000 7f8135f09640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81301013d0 0x7f81301a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.238+0000 7f8135f09640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f813010f600 con 0x7f81301013d0 2024-04-25T00:13:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.238+0000 7f8135f09640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f813010f480 con 0x7f8130100a30 2024-04-25T00:13:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.238+0000 7f8135f09640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f813010f780 con 0x7f81301000d0 2024-04-25T00:13:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.238+0000 7f812ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81301013d0 0x7f81301a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.238+0000 7f812ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81301013d0 0x7f81301a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59668/0 (socket says 172.21.15.27:59668) 2024-04-25T00:13:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.238+0000 7f812ffff640 1 -- 172.21.15.27:0/3948248922 learned_addr learned my addr 172.21.15.27:0/3948248922 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.238+0000 7f812effd640 1 --2- 172.21.15.27:0/3948248922 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8130100a30 0x7f813019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.239+0000 7f812f7fe640 1 --2- 172.21.15.27:0/3948248922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81301000d0 0x7f8130199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.239+0000 7f812ffff640 1 -- 172.21.15.27:0/3948248922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81301000d0 msgr2=0x7f8130199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.239+0000 7f812ffff640 1 --2- 172.21.15.27:0/3948248922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81301000d0 0x7f8130199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.239+0000 7f812ffff640 1 -- 172.21.15.27:0/3948248922 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8130100a30 msgr2=0x7f813019a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.239+0000 7f812ffff640 1 --2- 172.21.15.27:0/3948248922 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8130100a30 0x7f813019a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.239+0000 7f812ffff640 1 -- 172.21.15.27:0/3948248922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81301a18b0 con 0x7f81301013d0 2024-04-25T00:13:08.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.239+0000 7f812effd640 1 --2- 172.21.15.27:0/3948248922 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8130100a30 0x7f813019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:13:08.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.239+0000 7f812ffff640 1 --2- 172.21.15.27:0/3948248922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81301013d0 0x7f81301a1210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f812800eaa0 tx=0x7f812800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:08.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.239+0000 7f812cff9640 1 -- 172.21.15.27:0/3948248922 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812800cdb0 con 0x7f81301013d0 2024-04-25T00:13:08.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.239+0000 7f812cff9640 1 -- 172.21.15.27:0/3948248922 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8128005020 con 0x7f81301013d0 2024-04-25T00:13:08.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.239+0000 7f8135f09640 1 -- 172.21.15.27:0/3948248922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81301a1ba0 con 0x7f81301013d0 2024-04-25T00:13:08.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.239+0000 7f812cff9640 1 -- 172.21.15.27:0/3948248922 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81280116b0 con 0x7f81301013d0 2024-04-25T00:13:08.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.239+0000 7f8135f09640 1 -- 172.21.15.27:0/3948248922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81301a20c0 con 0x7f81301013d0 2024-04-25T00:13:08.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.240+0000 7f8135f09640 1 -- 172.21.15.27:0/3948248922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f813010afb0 con 0x7f81301013d0 2024-04-25T00:13:08.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.241+0000 7f812cff9640 1 -- 172.21.15.27:0/3948248922 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8128005560 con 0x7f81301013d0 2024-04-25T00:13:08.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.242+0000 7f812cff9640 1 --2- 172.21.15.27:0/3948248922 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f81180a2b90 0x7f81180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:08.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.242+0000 7f812f7fe640 1 --2- 172.21.15.27:0/3948248922 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f81180a2b90 0x7f81180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:08.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.242+0000 7f812cff9640 1 -- 172.21.15.27:0/3948248922 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f81280ceea0 con 0x7f81301013d0 2024-04-25T00:13:08.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.243+0000 7f812f7fe640 1 --2- 172.21.15.27:0/3948248922 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f81180a2b90 0x7f81180a5050 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8120005fd0 tx=0x7f8120005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:08.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.247+0000 7f812cff9640 1 -- 172.21.15.27:0/3948248922 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8128016030 con 0x7f81301013d0 2024-04-25T00:13:08.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.476+0000 7f8135f09640 1 -- 172.21.15.27:0/3948248922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f813002d170 con 0x7f81301013d0 2024-04-25T00:13:08.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.477+0000 7f812cff9640 1 -- 172.21.15.27:0/3948248922 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8836 (secure 0 0 0) 0x7f812809c2a0 con 0x7f81301013d0 2024-04-25T00:13:08.477 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:08.477 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24514,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T00:13:08.478 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 11 2024-04-25T00:13:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.479+0000 7f8135f09640 1 -- 172.21.15.27:0/3948248922 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f81180a2b90 msgr2=0x7f81180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.479+0000 7f8135f09640 1 --2- 172.21.15.27:0/3948248922 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f81180a2b90 0x7f81180a5050 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8120005fd0 tx=0x7f8120005850 comp rx=0 tx=0).stop 2024-04-25T00:13:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.479+0000 7f8135f09640 1 -- 172.21.15.27:0/3948248922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81301013d0 msgr2=0x7f81301a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.479+0000 7f8135f09640 1 --2- 172.21.15.27:0/3948248922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81301013d0 0x7f81301a1210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f812800eaa0 tx=0x7f812800ef60 comp rx=0 tx=0).stop 2024-04-25T00:13:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.479+0000 7f8135f09640 1 -- 172.21.15.27:0/3948248922 shutdown_connections 2024-04-25T00:13:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.479+0000 7f8135f09640 1 --2- 172.21.15.27:0/3948248922 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f81180a2b90 0x7f81180a5050 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.479+0000 7f8135f09640 1 --2- 172.21.15.27:0/3948248922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81301013d0 0x7f81301a1210 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:08.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.479+0000 7f8135f09640 1 --2- 172.21.15.27:0/3948248922 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8130100a30 0x7f813019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:08.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.479+0000 7f8135f09640 1 --2- 172.21.15.27:0/3948248922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81301000d0 0x7f8130199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:08.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.479+0000 7f8135f09640 1 -- 172.21.15.27:0/3948248922 >> 172.21.15.27:0/3948248922 conn(0x7f81300fbc90 msgr2=0x7f81300fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:08.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.479+0000 7f8135f09640 1 -- 172.21.15.27:0/3948248922 shutdown_connections 2024-04-25T00:13:08.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:08.479+0000 7f8135f09640 1 -- 172.21.15.27:0/3948248922 wait complete. 2024-04-25T00:13:08.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:08 smithi027 ceph-mon[29208]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:08.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:08.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:08.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:08.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:13:08.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:13:08.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:08.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:08 smithi027 ceph-mon[29208]: Deploying daemon mds.a on smithi027 2024-04-25T00:13:08.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:08 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:boot 2024-04-25T00:13:08.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:08 smithi027 ceph-mon[29208]: fsmap 10 up:standby 2024-04-25T00:13:08.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:08 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T00:13:08.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:08 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3948248922' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:09.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:08 smithi139 ceph-mon[32833]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:09.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:09.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:09.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:09.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:13:09.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:13:09.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:09.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:08 smithi139 ceph-mon[32833]: Deploying daemon mds.a on smithi027 2024-04-25T00:13:09.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:08 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:boot 2024-04-25T00:13:09.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:08 smithi139 ceph-mon[32833]: fsmap 10 up:standby 2024-04-25T00:13:09.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:08 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T00:13:09.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:08 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3948248922' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:09.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:08 smithi153 ceph-mon[32896]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:09.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:09.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:09.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:09.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:13:09.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:13:09.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:09.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:08 smithi153 ceph-mon[32896]: Deploying daemon mds.a on smithi027 2024-04-25T00:13:09.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:08 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:boot 2024-04-25T00:13:09.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:08 smithi153 ceph-mon[32896]: fsmap 10 up:standby 2024-04-25T00:13:09.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:08 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T00:13:09.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:08 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3948248922' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:09.942 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:13:09 smithi027 systemd[1]: Starting Ceph mds.a for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:13:10.309 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:13:10 smithi027 podman[67838]: 2024-04-25 00:13:09.961480561 +0000 UTC m=+0.018406488 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:13:10.309 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:13:10 smithi027 podman[67838]: 2024-04-25 00:13:10.078645263 +0000 UTC m=+0.135571223 container create 08bfbb52e555de6afb24f7f55187f64dffc302c3b383dae47d0af62d8f1f0fbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-25T00:13:10.309 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:13:10 smithi027 podman[67838]: 2024-04-25 00:13:10.306312438 +0000 UTC m=+0.363238356 container init 08bfbb52e555de6afb24f7f55187f64dffc302c3b383dae47d0af62d8f1f0fbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-25T00:13:10.491 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:10.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.599+0000 7fe4d57f9640 1 -- 172.21.15.27:0/3402548573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d0104b30 msgr2=0x7fe4d0106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:10.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.599+0000 7fe4d57f9640 1 --2- 172.21.15.27:0/3402548573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d0104b30 0x7fe4d0106f20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4009900 tx=0x7fe4c402a410 comp rx=0 tx=0).stop 2024-04-25T00:13:10.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.599+0000 7fe4d57f9640 1 -- 172.21.15.27:0/3402548573 shutdown_connections 2024-04-25T00:13:10.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.599+0000 7fe4d57f9640 1 --2- 172.21.15.27:0/3402548573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4d0107460 0x7fe4d01098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:10.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.599+0000 7fe4d57f9640 1 --2- 172.21.15.27:0/3402548573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d0104b30 0x7fe4d0106f20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:10.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.599+0000 7fe4d57f9640 1 --2- 172.21.15.27:0/3402548573 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4d0102200 0x7fe4d01045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:10.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.599+0000 7fe4d57f9640 1 -- 172.21.15.27:0/3402548573 >> 172.21.15.27:0/3402548573 conn(0x7fe4d00fbbf0 msgr2=0x7fe4d00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:10.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.599+0000 7fe4d57f9640 1 -- 172.21.15.27:0/3402548573 shutdown_connections 2024-04-25T00:13:10.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.599+0000 7fe4d57f9640 1 -- 172.21.15.27:0/3402548573 wait complete. 2024-04-25T00:13:10.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.599+0000 7fe4d57f9640 1 Processor -- start 2024-04-25T00:13:10.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4d57f9640 1 -- start start 2024-04-25T00:13:10.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4d57f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d0102200 0x7fe4d0197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:10.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4d57f9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4d0104b30 0x7fe4d0198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:10.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4d57f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4d0107460 0x7fe4d019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:10.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4d57f9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d010d400 con 0x7fe4d0102200 2024-04-25T00:13:10.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4d57f9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d010d280 con 0x7fe4d0104b30 2024-04-25T00:13:10.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4d57f9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d010d580 con 0x7fe4d0107460 2024-04-25T00:13:10.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4ceffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d0102200 0x7fe4d0197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:10.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4ceffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d0102200 0x7fe4d0197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59716/0 (socket says 172.21.15.27:59716) 2024-04-25T00:13:10.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4ceffd640 1 -- 172.21.15.27:0/2298357734 learned_addr learned my addr 172.21.15.27:0/2298357734 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:10.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4ce7fc640 1 --2- 172.21.15.27:0/2298357734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4d0104b30 0x7fe4d0198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:10.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4cf7fe640 1 --2- 172.21.15.27:0/2298357734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4d0107460 0x7fe4d019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:10.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4ceffd640 1 -- 172.21.15.27:0/2298357734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4d0107460 msgr2=0x7fe4d019f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:10.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4ceffd640 1 --2- 172.21.15.27:0/2298357734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4d0107460 0x7fe4d019f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:10.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4ceffd640 1 -- 172.21.15.27:0/2298357734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4d0104b30 msgr2=0x7fe4d0198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:10.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4ceffd640 1 --2- 172.21.15.27:0/2298357734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4d0104b30 0x7fe4d0198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:10.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4ceffd640 1 -- 172.21.15.27:0/2298357734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d019f6f0 con 0x7fe4d0102200 2024-04-25T00:13:10.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4ceffd640 1 --2- 172.21.15.27:0/2298357734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d0102200 0x7fe4d0197ad0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000e9f0 tx=0x7fe4c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:10.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.600+0000 7fe4b7fff640 1 -- 172.21.15.27:0/2298357734 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c000cd80 con 0x7fe4d0102200 2024-04-25T00:13:10.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.601+0000 7fe4b7fff640 1 -- 172.21.15.27:0/2298357734 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4c0010910 con 0x7fe4d0102200 2024-04-25T00:13:10.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.601+0000 7fe4d57f9640 1 -- 172.21.15.27:0/2298357734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4d019f9e0 con 0x7fe4d0102200 2024-04-25T00:13:10.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.601+0000 7fe4b7fff640 1 -- 172.21.15.27:0/2298357734 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0018990 con 0x7fe4d0102200 2024-04-25T00:13:10.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.601+0000 7fe4d57f9640 1 -- 172.21.15.27:0/2298357734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4d019ff00 con 0x7fe4d0102200 2024-04-25T00:13:10.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.602+0000 7fe4d57f9640 1 -- 172.21.15.27:0/2298357734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4d00fd6e0 con 0x7fe4d0102200 2024-04-25T00:13:10.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.603+0000 7fe4b7fff640 1 -- 172.21.15.27:0/2298357734 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe4c0010ac0 con 0x7fe4d0102200 2024-04-25T00:13:10.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.603+0000 7fe4b7fff640 1 --2- 172.21.15.27:0/2298357734 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe4b80a2b90 0x7fe4b80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:10.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.603+0000 7fe4ce7fc640 1 --2- 172.21.15.27:0/2298357734 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe4b80a2b90 0x7fe4b80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:10.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.603+0000 7fe4b7fff640 1 -- 172.21.15.27:0/2298357734 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe4c00ceeb0 con 0x7fe4d0102200 2024-04-25T00:13:10.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.604+0000 7fe4ce7fc640 1 --2- 172.21.15.27:0/2298357734 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe4b80a2b90 0x7fe4b80a5050 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4005ab0 tx=0x7fe4c4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:10.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.608+0000 7fe4b7fff640 1 -- 172.21.15.27:0/2298357734 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe4c009bc60 con 0x7fe4d0102200 2024-04-25T00:13:10.692 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:13:10 smithi027 podman[67838]: 2024-04-25 00:13:10.308924595 +0000 UTC m=+0.365850522 container start 08bfbb52e555de6afb24f7f55187f64dffc302c3b383dae47d0af62d8f1f0fbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T00:13:10.692 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:13:10 smithi027 ceph-mds[67857]: -- 172.21.15.27:0/3545040390 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ca94bc9a40 con 0x55ca94b9d000 2024-04-25T00:13:10.692 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:13:10 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-a[67853]: starting mds.a at 2024-04-25T00:13:10.692 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:13:10 smithi027 bash[67838]: 08bfbb52e555de6afb24f7f55187f64dffc302c3b383dae47d0af62d8f1f0fbd 2024-04-25T00:13:10.693 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:13:10 smithi027 systemd[1]: Started Ceph mds.a for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:13:10.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.833+0000 7fe4d57f9640 1 -- 172.21.15.27:0/2298357734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4d002d170 con 0x7fe4d0102200 2024-04-25T00:13:10.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.833+0000 7fe4b7fff640 1 -- 172.21.15.27:0/2298357734 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9662 (secure 0 0 0) 0x7fe4c009ff60 con 0x7fe4d0102200 2024-04-25T00:13:10.834 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:10.834 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24514,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T00:13:10.835 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 12 2024-04-25T00:13:10.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.836+0000 7fe4d57f9640 1 -- 172.21.15.27:0/2298357734 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe4b80a2b90 msgr2=0x7fe4b80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:10.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.836+0000 7fe4d57f9640 1 --2- 172.21.15.27:0/2298357734 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe4b80a2b90 0x7fe4b80a5050 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4005ab0 tx=0x7fe4c4000bc0 comp rx=0 tx=0).stop 2024-04-25T00:13:10.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.836+0000 7fe4d57f9640 1 -- 172.21.15.27:0/2298357734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d0102200 msgr2=0x7fe4d0197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:10.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.836+0000 7fe4d57f9640 1 --2- 172.21.15.27:0/2298357734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d0102200 0x7fe4d0197ad0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000e9f0 tx=0x7fe4c000eeb0 comp rx=0 tx=0).stop 2024-04-25T00:13:10.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.836+0000 7fe4d57f9640 1 -- 172.21.15.27:0/2298357734 shutdown_connections 2024-04-25T00:13:10.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.836+0000 7fe4d57f9640 1 --2- 172.21.15.27:0/2298357734 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe4b80a2b90 0x7fe4b80a5050 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:10.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.836+0000 7fe4d57f9640 1 --2- 172.21.15.27:0/2298357734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4d0107460 0x7fe4d019f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:10.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.836+0000 7fe4d57f9640 1 --2- 172.21.15.27:0/2298357734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4d0104b30 0x7fe4d0198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:10.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.836+0000 7fe4d57f9640 1 --2- 172.21.15.27:0/2298357734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d0102200 0x7fe4d0197ad0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:10.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.836+0000 7fe4d57f9640 1 -- 172.21.15.27:0/2298357734 >> 172.21.15.27:0/2298357734 conn(0x7fe4d00fbbf0 msgr2=0x7fe4d0102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:10.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.836+0000 7fe4d57f9640 1 -- 172.21.15.27:0/2298357734 shutdown_connections 2024-04-25T00:13:10.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:10.836+0000 7fe4d57f9640 1 -- 172.21.15.27:0/2298357734 wait complete. 2024-04-25T00:13:11.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:10 smithi139 ceph-mon[32833]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:11.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:11.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:11.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:11.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:13:11.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:13:11.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:11.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:10 smithi153 ceph-mon[32896]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:11.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:11.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:11.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:11.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:13:11.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:13:11.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:11.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:10 smithi027 ceph-mon[29208]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:11.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:11.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:11.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:11.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T00:13:11.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T00:13:11.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:12.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:11 smithi139 ceph-mon[32833]: Deploying daemon mds.b on smithi139 2024-04-25T00:13:12.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:11 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6840/2271898034,v1:172.21.15.27:6841/2271898034] up:boot 2024-04-25T00:13:12.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:11 smithi139 ceph-mon[32833]: fsmap 11 up:standby 2024-04-25T00:13:12.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T00:13:12.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:11 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2298357734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:12.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:11 smithi153 ceph-mon[32896]: Deploying daemon mds.b on smithi139 2024-04-25T00:13:12.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:11 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6840/2271898034,v1:172.21.15.27:6841/2271898034] up:boot 2024-04-25T00:13:12.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:11 smithi153 ceph-mon[32896]: fsmap 11 up:standby 2024-04-25T00:13:12.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T00:13:12.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:11 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2298357734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:12.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:11 smithi027 ceph-mon[29208]: Deploying daemon mds.b on smithi139 2024-04-25T00:13:12.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:11 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6840/2271898034,v1:172.21.15.27:6841/2271898034] up:boot 2024-04-25T00:13:12.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:11 smithi027 ceph-mon[29208]: fsmap 11 up:standby 2024-04-25T00:13:12.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T00:13:12.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:11 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2298357734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:12.848 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:12.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.964+0000 7f6d2da72640 1 -- 172.21.15.27:0/3964578814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d281017b0 msgr2=0x7f6d28101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:12.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.964+0000 7f6d2da72640 1 --2- 172.21.15.27:0/3964578814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d281017b0 0x7f6d28101bd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009900 tx=0x7f6d1802a410 comp rx=0 tx=0).stop 2024-04-25T00:13:12.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.964+0000 7f6d2da72640 1 -- 172.21.15.27:0/3964578814 shutdown_connections 2024-04-25T00:13:12.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.964+0000 7f6d2da72640 1 --2- 172.21.15.27:0/3964578814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d28103360 0x7f6d28109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:12.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.964+0000 7f6d2da72640 1 --2- 172.21.15.27:0/3964578814 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d281029a0 0x7f6d28102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:12.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.964+0000 7f6d2da72640 1 --2- 172.21.15.27:0/3964578814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d281017b0 0x7f6d28101bd0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.964+0000 7f6d2da72640 1 -- 172.21.15.27:0/3964578814 >> 172.21.15.27:0/3964578814 conn(0x7f6d280fcd40 msgr2=0x7f6d280ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.964+0000 7f6d2da72640 1 -- 172.21.15.27:0/3964578814 shutdown_connections 2024-04-25T00:13:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.964+0000 7f6d2da72640 1 -- 172.21.15.27:0/3964578814 wait complete. 2024-04-25T00:13:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.964+0000 7f6d2da72640 1 Processor -- start 2024-04-25T00:13:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d2da72640 1 -- start start 2024-04-25T00:13:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d2da72640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d281017b0 0x7f6d28197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d2da72640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d281029a0 0x7f6d28198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d2da72640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d28103360 0x7f6d2819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d2da72640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2810d400 con 0x7f6d281017b0 2024-04-25T00:13:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d2da72640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2810d280 con 0x7f6d28103360 2024-04-25T00:13:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d2da72640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2810d580 con 0x7f6d281029a0 2024-04-25T00:13:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d277fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d281017b0 0x7f6d28197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d277fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d281017b0 0x7f6d28197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59736/0 (socket says 172.21.15.27:59736) 2024-04-25T00:13:12.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d277fe640 1 -- 172.21.15.27:0/2357188480 learned_addr learned my addr 172.21.15.27:0/2357188480 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:12.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d27fff640 1 --2- 172.21.15.27:0/2357188480 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d28103360 0x7f6d2819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:12.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d26ffd640 1 --2- 172.21.15.27:0/2357188480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d281029a0 0x7f6d28198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:12.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d277fe640 1 -- 172.21.15.27:0/2357188480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d281029a0 msgr2=0x7f6d28198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:12.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d277fe640 1 --2- 172.21.15.27:0/2357188480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d281029a0 0x7f6d28198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:12.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d277fe640 1 -- 172.21.15.27:0/2357188480 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d28103360 msgr2=0x7f6d2819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:12.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d277fe640 1 --2- 172.21.15.27:0/2357188480 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d28103360 0x7f6d2819f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:12.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d277fe640 1 -- 172.21.15.27:0/2357188480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2819f6f0 con 0x7f6d281017b0 2024-04-25T00:13:12.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.965+0000 7f6d26ffd640 1 --2- 172.21.15.27:0/2357188480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d281029a0 0x7f6d28198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:13:12.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.966+0000 7f6d277fe640 1 --2- 172.21.15.27:0/2357188480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d281017b0 0x7f6d28197ad0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6d18031610 tx=0x7f6d180043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:12.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.966+0000 7f6d24ff9640 1 -- 172.21.15.27:0/2357188480 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1803f030 con 0x7f6d281017b0 2024-04-25T00:13:12.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.966+0000 7f6d24ff9640 1 -- 172.21.15.27:0/2357188480 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d18003ce0 con 0x7f6d281017b0 2024-04-25T00:13:12.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.966+0000 7f6d2da72640 1 -- 172.21.15.27:0/2357188480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d2819f980 con 0x7f6d281017b0 2024-04-25T00:13:12.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.966+0000 7f6d24ff9640 1 -- 172.21.15.27:0/2357188480 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d18035070 con 0x7f6d281017b0 2024-04-25T00:13:12.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.966+0000 7f6d2da72640 1 -- 172.21.15.27:0/2357188480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d2819fe70 con 0x7f6d281017b0 2024-04-25T00:13:12.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.967+0000 7f6d2da72640 1 -- 172.21.15.27:0/2357188480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d28108af0 con 0x7f6d281017b0 2024-04-25T00:13:12.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.968+0000 7f6d24ff9640 1 -- 172.21.15.27:0/2357188480 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6d18042030 con 0x7f6d281017b0 2024-04-25T00:13:12.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.968+0000 7f6d24ff9640 1 --2- 172.21.15.27:0/2357188480 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6d100a2b90 0x7f6d100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:12.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.969+0000 7f6d26ffd640 1 --2- 172.21.15.27:0/2357188480 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6d100a2b90 0x7f6d100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:12.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.969+0000 7f6d24ff9640 1 -- 172.21.15.27:0/2357188480 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6d180ef140 con 0x7f6d281017b0 2024-04-25T00:13:12.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.969+0000 7f6d26ffd640 1 --2- 172.21.15.27:0/2357188480 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6d100a2b90 0x7f6d100a5050 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6d28198fe0 tx=0x7f6d1c005ec0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:12.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:12.973+0000 7f6d24ff9640 1 -- 172.21.15.27:0/2357188480 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6d180bc540 con 0x7f6d281017b0 2024-04-25T00:13:13.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:12 smithi139 ceph-mon[32833]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:13.003 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:13:12 smithi139 systemd[1]: Starting Ceph mds.b for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:13:13.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:12 smithi153 ceph-mon[32896]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:13.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:12 smithi027 ceph-mon[29208]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:13.197+0000 7f6d2da72640 1 -- 172.21.15.27:0/2357188480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d28062340 con 0x7f6d281017b0 2024-04-25T00:13:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:13.198+0000 7f6d24ff9640 1 -- 172.21.15.27:0/2357188480 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9662 (secure 0 0 0) 0x7f6d180bbe40 con 0x7f6d281017b0 2024-04-25T00:13:13.198 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:13.198 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24514,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T00:13:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 12 2024-04-25T00:13:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:13.199+0000 7f6d2da72640 1 -- 172.21.15.27:0/2357188480 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6d100a2b90 msgr2=0x7f6d100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:13.199+0000 7f6d2da72640 1 --2- 172.21.15.27:0/2357188480 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6d100a2b90 0x7f6d100a5050 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6d28198fe0 tx=0x7f6d1c005ec0 comp rx=0 tx=0).stop 2024-04-25T00:13:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:13.199+0000 7f6d2da72640 1 -- 172.21.15.27:0/2357188480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d281017b0 msgr2=0x7f6d28197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:13.199+0000 7f6d2da72640 1 --2- 172.21.15.27:0/2357188480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d281017b0 0x7f6d28197ad0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6d18031610 tx=0x7f6d180043b0 comp rx=0 tx=0).stop 2024-04-25T00:13:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:13.199+0000 7f6d2da72640 1 -- 172.21.15.27:0/2357188480 shutdown_connections 2024-04-25T00:13:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:13.199+0000 7f6d2da72640 1 --2- 172.21.15.27:0/2357188480 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6d100a2b90 0x7f6d100a5050 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:13.199+0000 7f6d2da72640 1 --2- 172.21.15.27:0/2357188480 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d28103360 0x7f6d2819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:13.199+0000 7f6d2da72640 1 --2- 172.21.15.27:0/2357188480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d281029a0 0x7f6d28198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:13.199+0000 7f6d2da72640 1 --2- 172.21.15.27:0/2357188480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d281017b0 0x7f6d28197ad0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:13.199+0000 7f6d2da72640 1 -- 172.21.15.27:0/2357188480 >> 172.21.15.27:0/2357188480 conn(0x7f6d280fcd40 msgr2=0x7f6d28105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:13.200+0000 7f6d2da72640 1 -- 172.21.15.27:0/2357188480 shutdown_connections 2024-04-25T00:13:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:13.200+0000 7f6d2da72640 1 -- 172.21.15.27:0/2357188480 wait complete. 2024-04-25T00:13:13.347 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:13:13 smithi139 podman[62974]: 2024-04-25 00:13:12.991304397 +0000 UTC m=+0.018219774 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:13:13.348 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:13:13 smithi139 podman[62974]: 2024-04-25 00:13:13.103443866 +0000 UTC m=+0.130359284 container create 3e317c4309d5710c6b99a6a1534a388826ca6d458be6e262f9ee2f0ca02d710a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T00:13:13.348 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:13:13 smithi139 podman[62974]: 2024-04-25 00:13:13.347917471 +0000 UTC m=+0.374832847 container init 3e317c4309d5710c6b99a6a1534a388826ca6d458be6e262f9ee2f0ca02d710a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T00:13:13.732 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:13:13 smithi139 podman[62974]: 2024-04-25 00:13:13.350652025 +0000 UTC m=+0.377567414 container start 3e317c4309d5710c6b99a6a1534a388826ca6d458be6e262f9ee2f0ca02d710a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T00:13:13.732 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:13:13 smithi139 ceph-mds[62993]: -- 172.21.15.139:0/1091225057 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d801745a40 con 0x55d801718c00 2024-04-25T00:13:13.733 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:13:13 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-b[62989]: starting mds.b at 2024-04-25T00:13:13.733 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:13:13 smithi139 bash[62974]: 3e317c4309d5710c6b99a6a1534a388826ca6d458be6e262f9ee2f0ca02d710a 2024-04-25T00:13:13.733 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:13:13 smithi139 systemd[1]: Started Ceph mds.b for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:13:14.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:13 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2357188480' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:14.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:13 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:14.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:13 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:14.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:13 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:14.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:13 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:14.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:13 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:14.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:13 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:13:14.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:13 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2357188480' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:14.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:13 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:14.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:13 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:14.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:13 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:14.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:13 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:14.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:13 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:14.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:13 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:13:14.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:13 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2357188480' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:14.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:13 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:14.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:13 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:14.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:13 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:14.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:13 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:14.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:13 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:14.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:13 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:13:14.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:14 smithi139 ceph-mon[32833]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:14.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:14 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:boot 2024-04-25T00:13:14.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:14 smithi139 ceph-mon[32833]: fsmap 12 up:standby 2024-04-25T00:13:14.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:14 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T00:13:14.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:14 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:14.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:14 smithi153 ceph-mon[32896]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:14.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:14 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:boot 2024-04-25T00:13:14.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:14 smithi153 ceph-mon[32896]: fsmap 12 up:standby 2024-04-25T00:13:14.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:14 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T00:13:14.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:14 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:14.801 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:14 smithi027 ceph-mon[29208]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:14.801 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:14 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:boot 2024-04-25T00:13:14.801 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:14 smithi027 ceph-mon[29208]: fsmap 12 up:standby 2024-04-25T00:13:14.801 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:14 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T00:13:14.801 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:14 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:15.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:15.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.338+0000 7f24bbffb640 1 -- 172.21.15.27:0/2692066754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b4100680 msgr2=0x7f24b4100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:15.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.338+0000 7f24bbffb640 1 --2- 172.21.15.27:0/2692066754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b4100680 0x7f24b4100aa0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f24b0009900 tx=0x7f24b002a410 comp rx=0 tx=0).stop 2024-04-25T00:13:15.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.338+0000 7f24bbffb640 1 -- 172.21.15.27:0/2692066754 shutdown_connections 2024-04-25T00:13:15.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.338+0000 7f24bbffb640 1 --2- 172.21.15.27:0/2692066754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24b4102230 0x7f24b4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.338+0000 7f24bbffb640 1 --2- 172.21.15.27:0/2692066754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24b4101870 0x7f24b4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.338+0000 7f24bbffb640 1 --2- 172.21.15.27:0/2692066754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b4100680 0x7f24b4100aa0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.338+0000 7f24bbffb640 1 -- 172.21.15.27:0/2692066754 >> 172.21.15.27:0/2692066754 conn(0x7f24b40fbbf0 msgr2=0x7f24b40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:15.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.339+0000 7f24bbffb640 1 -- 172.21.15.27:0/2692066754 shutdown_connections 2024-04-25T00:13:15.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.339+0000 7f24bbffb640 1 -- 172.21.15.27:0/2692066754 wait complete. 2024-04-25T00:13:15.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.339+0000 7f24bbffb640 1 Processor -- start 2024-04-25T00:13:15.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.339+0000 7f24bbffb640 1 -- start start 2024-04-25T00:13:15.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.339+0000 7f24bbffb640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24b4100680 0x7f24b40694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:15.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.339+0000 7f24bbffb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24b4101870 0x7f24b419c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:15.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.339+0000 7f24bbffb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b4102230 0x7f24b419ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:15.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.339+0000 7f24bbffb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f24b410c300 con 0x7f24b4102230 2024-04-25T00:13:15.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24bbffb640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f24b410c180 con 0x7f24b4100680 2024-04-25T00:13:15.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24bbffb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f24b410c480 con 0x7f24b4101870 2024-04-25T00:13:15.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24ba598640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b4102230 0x7f24b419ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:15.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24b9d97640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24b4100680 0x7f24b40694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:15.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24b9596640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24b4101870 0x7f24b419c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:15.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24b9d97640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24b4100680 0x7f24b40694a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:52150/0 (socket says 172.21.15.27:52150) 2024-04-25T00:13:15.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24ba598640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b4102230 0x7f24b419ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59758/0 (socket says 172.21.15.27:59758) 2024-04-25T00:13:15.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24b9d97640 1 -- 172.21.15.27:0/3206354853 learned_addr learned my addr 172.21.15.27:0/3206354853 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:15.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24b9d97640 1 -- 172.21.15.27:0/3206354853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24b4101870 msgr2=0x7f24b419c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:15.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24b9d97640 1 --2- 172.21.15.27:0/3206354853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24b4101870 0x7f24b419c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24b9d97640 1 -- 172.21.15.27:0/3206354853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b4102230 msgr2=0x7f24b419ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:15.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24b9d97640 1 --2- 172.21.15.27:0/3206354853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b4102230 0x7f24b419ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24b9d97640 1 -- 172.21.15.27:0/3206354853 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b419f720 con 0x7f24b4100680 2024-04-25T00:13:15.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24b9596640 1 --2- 172.21.15.27:0/3206354853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24b4101870 0x7f24b419c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:13:15.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24ba598640 1 --2- 172.21.15.27:0/3206354853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b4102230 0x7f24b419ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:13:15.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24b9d97640 1 --2- 172.21.15.27:0/3206354853 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24b4100680 0x7f24b40694a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f24b0031610 tx=0x7f24b0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:15.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24aaffd640 1 -- 172.21.15.27:0/3206354853 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b003f030 con 0x7f24b4100680 2024-04-25T00:13:15.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.340+0000 7f24bbffb640 1 -- 172.21.15.27:0/3206354853 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b419f9b0 con 0x7f24b4100680 2024-04-25T00:13:15.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.341+0000 7f24bbffb640 1 -- 172.21.15.27:0/3206354853 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b419fef0 con 0x7f24b4100680 2024-04-25T00:13:15.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.341+0000 7f24aaffd640 1 -- 172.21.15.27:0/3206354853 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24b0003ad0 con 0x7f24b4100680 2024-04-25T00:13:15.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.341+0000 7f24aaffd640 1 -- 172.21.15.27:0/3206354853 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b0049060 con 0x7f24b4100680 2024-04-25T00:13:15.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.342+0000 7f24bbffb640 1 -- 172.21.15.27:0/3206354853 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24980052c0 con 0x7f24b4100680 2024-04-25T00:13:15.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.343+0000 7f24aaffd640 1 -- 172.21.15.27:0/3206354853 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f24b0042030 con 0x7f24b4100680 2024-04-25T00:13:15.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.343+0000 7f24aaffd640 1 --2- 172.21.15.27:0/3206354853 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f24a00a2b90 0x7f24a00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:15.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.343+0000 7f24b9596640 1 --2- 172.21.15.27:0/3206354853 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f24a00a2b90 0x7f24a00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:15.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.343+0000 7f24aaffd640 1 -- 172.21.15.27:0/3206354853 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f24b00eee40 con 0x7f24b4100680 2024-04-25T00:13:15.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.344+0000 7f24b9596640 1 --2- 172.21.15.27:0/3206354853 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f24a00a2b90 0x7f24a00a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f24b4069210 tx=0x7f24a4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:15.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.348+0000 7f24aaffd640 1 -- 172.21.15.27:0/3206354853 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f24b00bbb50 con 0x7f24b4100680 2024-04-25T00:13:15.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.600+0000 7f24bbffb640 1 -- 172.21.15.27:0/3206354853 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24980055b0 con 0x7f24b4100680 2024-04-25T00:13:15.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.601+0000 7f24aaffd640 1 -- 172.21.15.27:0/3206354853 <== mon.2 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10491 (secure 0 0 0) 0x7f24b00bfe50 con 0x7f24b4100680 2024-04-25T00:13:15.602 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:15.602 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24514,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24521,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-25T00:13:15.602 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 13 2024-04-25T00:13:15.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.603+0000 7f24bbffb640 1 -- 172.21.15.27:0/3206354853 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f24a00a2b90 msgr2=0x7f24a00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:15.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.603+0000 7f24bbffb640 1 --2- 172.21.15.27:0/3206354853 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f24a00a2b90 0x7f24a00a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f24b4069210 tx=0x7f24a4005f90 comp rx=0 tx=0).stop 2024-04-25T00:13:15.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.603+0000 7f24bbffb640 1 -- 172.21.15.27:0/3206354853 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24b4100680 msgr2=0x7f24b40694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:15.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.603+0000 7f24bbffb640 1 --2- 172.21.15.27:0/3206354853 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24b4100680 0x7f24b40694a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f24b0031610 tx=0x7f24b0004270 comp rx=0 tx=0).stop 2024-04-25T00:13:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.603+0000 7f24bbffb640 1 -- 172.21.15.27:0/3206354853 shutdown_connections 2024-04-25T00:13:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.603+0000 7f24bbffb640 1 --2- 172.21.15.27:0/3206354853 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f24a00a2b90 0x7f24a00a5050 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.603+0000 7f24bbffb640 1 --2- 172.21.15.27:0/3206354853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b4102230 0x7f24b419ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.603+0000 7f24bbffb640 1 --2- 172.21.15.27:0/3206354853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24b4101870 0x7f24b419c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.603+0000 7f24bbffb640 1 --2- 172.21.15.27:0/3206354853 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24b4100680 0x7f24b40694a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.603+0000 7f24bbffb640 1 -- 172.21.15.27:0/3206354853 >> 172.21.15.27:0/3206354853 conn(0x7f24b40fbbf0 msgr2=0x7f24b4104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.603+0000 7f24bbffb640 1 -- 172.21.15.27:0/3206354853 shutdown_connections 2024-04-25T00:13:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.603+0000 7f24bbffb640 1 -- 172.21.15.27:0/3206354853 wait complete. 2024-04-25T00:13:15.616 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:13:15.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.721+0000 7ff9b5777640 1 -- 172.21.15.27:0/60813874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b01017a0 msgr2=0x7ff9b0101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:15.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.721+0000 7ff9b5777640 1 --2- 172.21.15.27:0/60813874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b01017a0 0x7ff9b0101bc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0009900 tx=0x7ff9a002a410 comp rx=0 tx=0).stop 2024-04-25T00:13:15.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.721+0000 7ff9b5777640 1 -- 172.21.15.27:0/60813874 shutdown_connections 2024-04-25T00:13:15.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.721+0000 7ff9b5777640 1 --2- 172.21.15.27:0/60813874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9b0103350 0x7ff9b0109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.721+0000 7ff9b5777640 1 --2- 172.21.15.27:0/60813874 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9b0102990 0x7ff9b0102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.721+0000 7ff9b5777640 1 --2- 172.21.15.27:0/60813874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b01017a0 0x7ff9b0101bc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.721+0000 7ff9b5777640 1 -- 172.21.15.27:0/60813874 >> 172.21.15.27:0/60813874 conn(0x7ff9b00fccf0 msgr2=0x7ff9b00ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:15.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.721+0000 7ff9b5777640 1 -- 172.21.15.27:0/60813874 shutdown_connections 2024-04-25T00:13:15.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.721+0000 7ff9b5777640 1 -- 172.21.15.27:0/60813874 wait complete. 2024-04-25T00:13:15.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.721+0000 7ff9b5777640 1 Processor -- start 2024-04-25T00:13:15.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.721+0000 7ff9b5777640 1 -- start start 2024-04-25T00:13:15.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9b5777640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9b01017a0 0x7ff9b0197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:15.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9b5777640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9b0102990 0x7ff9b0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:15.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9b5777640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b0103350 0x7ff9b019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:15.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9b5777640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b010d440 con 0x7ff9b0103350 2024-04-25T00:13:15.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9b5777640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b010d2c0 con 0x7ff9b0102990 2024-04-25T00:13:15.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9b5777640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b010d5c0 con 0x7ff9b01017a0 2024-04-25T00:13:15.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9af7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b0103350 0x7ff9b019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:15.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9af7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b0103350 0x7ff9b019f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59770/0 (socket says 172.21.15.27:59770) 2024-04-25T00:13:15.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9af7fe640 1 -- 172.21.15.27:0/744076267 learned_addr learned my addr 172.21.15.27:0/744076267 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:15.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9ae7fc640 1 --2- 172.21.15.27:0/744076267 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9b0102990 0x7ff9b0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9aeffd640 1 --2- 172.21.15.27:0/744076267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9b01017a0 0x7ff9b0197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9af7fe640 1 -- 172.21.15.27:0/744076267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9b01017a0 msgr2=0x7ff9b0197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9af7fe640 1 --2- 172.21.15.27:0/744076267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9b01017a0 0x7ff9b0197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9af7fe640 1 -- 172.21.15.27:0/744076267 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9b0102990 msgr2=0x7ff9b0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9af7fe640 1 --2- 172.21.15.27:0/744076267 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9b0102990 0x7ff9b0197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9af7fe640 1 -- 172.21.15.27:0/744076267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b019f6d0 con 0x7ff9b0103350 2024-04-25T00:13:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9aeffd640 1 --2- 172.21.15.27:0/744076267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9b01017a0 0x7ff9b0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:13:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.722+0000 7ff9af7fe640 1 --2- 172.21.15.27:0/744076267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b0103350 0x7ff9b019f060 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff9a800eb10 tx=0x7ff9a800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.723+0000 7ff997fff640 1 -- 172.21.15.27:0/744076267 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8004b20 con 0x7ff9b0103350 2024-04-25T00:13:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.723+0000 7ff997fff640 1 -- 172.21.15.27:0/744076267 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9a8004cc0 con 0x7ff9b0103350 2024-04-25T00:13:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.723+0000 7ff9b5777640 1 -- 172.21.15.27:0/744076267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9b019f9c0 con 0x7ff9b0103350 2024-04-25T00:13:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.723+0000 7ff997fff640 1 -- 172.21.15.27:0/744076267 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8010760 con 0x7ff9b0103350 2024-04-25T00:13:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.723+0000 7ff9b5777640 1 -- 172.21.15.27:0/744076267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9b019fec0 con 0x7ff9b0103350 2024-04-25T00:13:15.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.724+0000 7ff9b5777640 1 -- 172.21.15.27:0/744076267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9b0108ac0 con 0x7ff9b0103350 2024-04-25T00:13:15.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.725+0000 7ff997fff640 1 -- 172.21.15.27:0/744076267 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff9a8005020 con 0x7ff9b0103350 2024-04-25T00:13:15.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.726+0000 7ff997fff640 1 --2- 172.21.15.27:0/744076267 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff9980a2b90 0x7ff9980a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:15.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.726+0000 7ff9aeffd640 1 --2- 172.21.15.27:0/744076267 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff9980a2b90 0x7ff9980a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:15.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.726+0000 7ff997fff640 1 -- 172.21.15.27:0/744076267 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff9a80cf150 con 0x7ff9b0103350 2024-04-25T00:13:15.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.726+0000 7ff9aeffd640 1 --2- 172.21.15.27:0/744076267 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff9980a2b90 0x7ff9980a5050 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff9a002aa00 tx=0x7ff9a00315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:15.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.730+0000 7ff997fff640 1 -- 172.21.15.27:0/744076267 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff9a809c550 con 0x7ff9b0103350 2024-04-25T00:13:15.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.919+0000 7ff9b5777640 1 -- 172.21.15.27:0/744076267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff9b002d3c0 con 0x7ff9b0103350 2024-04-25T00:13:15.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.920+0000 7ff997fff640 1 -- 172.21.15.27:0/744076267 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18445 (secure 0 0 0) 0x7ff9a809be50 con 0x7ff9b0103350 2024-04-25T00:13:15.921 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:15.921 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":81,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:12:26.417275+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:13:15.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.922+0000 7ff9b5777640 1 -- 172.21.15.27:0/744076267 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff9980a2b90 msgr2=0x7ff9980a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:15.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.922+0000 7ff9b5777640 1 --2- 172.21.15.27:0/744076267 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff9980a2b90 0x7ff9980a5050 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff9a002aa00 tx=0x7ff9a00315b0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.922+0000 7ff9b5777640 1 -- 172.21.15.27:0/744076267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b0103350 msgr2=0x7ff9b019f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:15.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.922+0000 7ff9b5777640 1 --2- 172.21.15.27:0/744076267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b0103350 0x7ff9b019f060 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff9a800eb10 tx=0x7ff9a800c2d0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.922+0000 7ff9b5777640 1 -- 172.21.15.27:0/744076267 shutdown_connections 2024-04-25T00:13:15.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.922+0000 7ff9b5777640 1 --2- 172.21.15.27:0/744076267 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff9980a2b90 0x7ff9980a5050 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.922+0000 7ff9b5777640 1 --2- 172.21.15.27:0/744076267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b0103350 0x7ff9b019f060 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.922+0000 7ff9b5777640 1 --2- 172.21.15.27:0/744076267 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9b0102990 0x7ff9b0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.922+0000 7ff9b5777640 1 --2- 172.21.15.27:0/744076267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9b01017a0 0x7ff9b0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:15.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.922+0000 7ff9b5777640 1 -- 172.21.15.27:0/744076267 >> 172.21.15.27:0/744076267 conn(0x7ff9b00fccf0 msgr2=0x7ff9b0105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:15.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.922+0000 7ff9b5777640 1 -- 172.21.15.27:0/744076267 shutdown_connections 2024-04-25T00:13:15.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:15.922+0000 7ff9b5777640 1 -- 172.21.15.27:0/744076267 wait complete. 2024-04-25T00:13:15.935 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:08:34.006973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T00:13:15.935 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:13:16.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.040+0000 7f4a446f5640 1 -- 172.21.15.27:0/1599925777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a3c1022a0 msgr2=0x7f4a3c104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.040+0000 7f4a446f5640 1 --2- 172.21.15.27:0/1599925777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a3c1022a0 0x7f4a3c104690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4a38009900 tx=0x7f4a3802a410 comp rx=0 tx=0).stop 2024-04-25T00:13:16.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.040+0000 7f4a446f5640 1 -- 172.21.15.27:0/1599925777 shutdown_connections 2024-04-25T00:13:16.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.040+0000 7f4a446f5640 1 --2- 172.21.15.27:0/1599925777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a3c107500 0x7f4a3c1098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.040+0000 7f4a446f5640 1 --2- 172.21.15.27:0/1599925777 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a3c104bd0 0x7f4a3c106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.040+0000 7f4a446f5640 1 --2- 172.21.15.27:0/1599925777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a3c1022a0 0x7f4a3c104690 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.040+0000 7f4a446f5640 1 -- 172.21.15.27:0/1599925777 >> 172.21.15.27:0/1599925777 conn(0x7f4a3c0fbbf0 msgr2=0x7f4a3c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:16.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.040+0000 7f4a446f5640 1 -- 172.21.15.27:0/1599925777 shutdown_connections 2024-04-25T00:13:16.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.040+0000 7f4a446f5640 1 -- 172.21.15.27:0/1599925777 wait complete. 2024-04-25T00:13:16.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a446f5640 1 Processor -- start 2024-04-25T00:13:16.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a446f5640 1 -- start start 2024-04-25T00:13:16.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a446f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a3c1022a0 0x7f4a3c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:16.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a446f5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a3c104bd0 0x7f4a3c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:16.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a446f5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a3c107500 0x7f4a3c19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:16.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a446f5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c10d410 con 0x7f4a3c107500 2024-04-25T00:13:16.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a446f5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c10d290 con 0x7f4a3c104bd0 2024-04-25T00:13:16.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a446f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c10d590 con 0x7f4a3c1022a0 2024-04-25T00:13:16.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a42c92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a3c107500 0x7f4a3c19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:16.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a42c92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a3c107500 0x7f4a3c19f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59800/0 (socket says 172.21.15.27:59800) 2024-04-25T00:13:16.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a42c92640 1 -- 172.21.15.27:0/3617633669 learned_addr learned my addr 172.21.15.27:0/3617633669 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:16.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a41c90640 1 --2- 172.21.15.27:0/3617633669 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a3c104bd0 0x7f4a3c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:16.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a42491640 1 --2- 172.21.15.27:0/3617633669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a3c1022a0 0x7f4a3c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:16.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a42c92640 1 -- 172.21.15.27:0/3617633669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a3c1022a0 msgr2=0x7f4a3c197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a42c92640 1 --2- 172.21.15.27:0/3617633669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a3c1022a0 0x7f4a3c197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a42c92640 1 -- 172.21.15.27:0/3617633669 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a3c104bd0 msgr2=0x7f4a3c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a42c92640 1 --2- 172.21.15.27:0/3617633669 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a3c104bd0 0x7f4a3c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a42c92640 1 -- 172.21.15.27:0/3617633669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a3c19f700 con 0x7f4a3c107500 2024-04-25T00:13:16.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a42c92640 1 --2- 172.21.15.27:0/3617633669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a3c107500 0x7f4a3c19f090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400eb10 tx=0x7f4a3400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.041+0000 7f4a337fe640 1 -- 172.21.15.27:0/3617633669 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a34004b20 con 0x7f4a3c107500 2024-04-25T00:13:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.042+0000 7f4a337fe640 1 -- 172.21.15.27:0/3617633669 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a34004cc0 con 0x7f4a3c107500 2024-04-25T00:13:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.042+0000 7f4a337fe640 1 -- 172.21.15.27:0/3617633669 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a34010760 con 0x7f4a3c107500 2024-04-25T00:13:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.042+0000 7f4a446f5640 1 -- 172.21.15.27:0/3617633669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a3c19f9f0 con 0x7f4a3c107500 2024-04-25T00:13:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.042+0000 7f4a446f5640 1 -- 172.21.15.27:0/3617633669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a3c19fef0 con 0x7f4a3c107500 2024-04-25T00:13:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.043+0000 7f4a446f5640 1 -- 172.21.15.27:0/3617633669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a3c0fd6e0 con 0x7f4a3c107500 2024-04-25T00:13:16.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.044+0000 7f4a337fe640 1 -- 172.21.15.27:0/3617633669 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4a34005020 con 0x7f4a3c107500 2024-04-25T00:13:16.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.044+0000 7f4a337fe640 1 --2- 172.21.15.27:0/3617633669 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a280a2c10 0x7f4a280a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:16.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.045+0000 7f4a42491640 1 --2- 172.21.15.27:0/3617633669 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a280a2c10 0x7f4a280a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:16.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.045+0000 7f4a337fe640 1 -- 172.21.15.27:0/3617633669 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4a340cf1c0 con 0x7f4a3c107500 2024-04-25T00:13:16.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.045+0000 7f4a42491640 1 --2- 172.21.15.27:0/3617633669 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a280a2c10 0x7f4a280a50d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4a380314e0 tx=0x7f4a38000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:16.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.049+0000 7f4a337fe640 1 -- 172.21.15.27:0/3617633669 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4a3409c5c0 con 0x7f4a3c107500 2024-04-25T00:13:16.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.244+0000 7f4a446f5640 1 -- 172.21.15.27:0/3617633669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4a3c02d3c0 con 0x7f4a3c107500 2024-04-25T00:13:16.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.245+0000 7f4a337fe640 1 -- 172.21.15.27:0/3617633669 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f4a3409bec0 con 0x7f4a3c107500 2024-04-25T00:13:16.246 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T00:13:16.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.247+0000 7f4a446f5640 1 -- 172.21.15.27:0/3617633669 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a280a2c10 msgr2=0x7f4a280a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.247+0000 7f4a446f5640 1 --2- 172.21.15.27:0/3617633669 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a280a2c10 0x7f4a280a50d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4a380314e0 tx=0x7f4a38000bc0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.247+0000 7f4a446f5640 1 -- 172.21.15.27:0/3617633669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a3c107500 msgr2=0x7f4a3c19f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.247+0000 7f4a446f5640 1 --2- 172.21.15.27:0/3617633669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a3c107500 0x7f4a3c19f090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400eb10 tx=0x7f4a3400c2d0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.247+0000 7f4a446f5640 1 -- 172.21.15.27:0/3617633669 shutdown_connections 2024-04-25T00:13:16.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.247+0000 7f4a446f5640 1 --2- 172.21.15.27:0/3617633669 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4a280a2c10 0x7f4a280a50d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.247+0000 7f4a446f5640 1 --2- 172.21.15.27:0/3617633669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a3c107500 0x7f4a3c19f090 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.247+0000 7f4a446f5640 1 --2- 172.21.15.27:0/3617633669 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a3c104bd0 0x7f4a3c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.247+0000 7f4a446f5640 1 --2- 172.21.15.27:0/3617633669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a3c1022a0 0x7f4a3c197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.247+0000 7f4a446f5640 1 -- 172.21.15.27:0/3617633669 >> 172.21.15.27:0/3617633669 conn(0x7f4a3c0fbbf0 msgr2=0x7f4a3c102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:16.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.247+0000 7f4a446f5640 1 -- 172.21.15.27:0/3617633669 shutdown_connections 2024-04-25T00:13:16.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.247+0000 7f4a446f5640 1 -- 172.21.15.27:0/3617633669 wait complete. 2024-04-25T00:13:16.260 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T00:13:16.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.363+0000 7fe148bdc640 1 -- 172.21.15.27:0/2029728681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144104b30 msgr2=0x7fe144106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.363+0000 7fe148bdc640 1 --2- 172.21.15.27:0/2029728681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144104b30 0x7fe144106f20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009900 tx=0x7fe12c02a410 comp rx=0 tx=0).stop 2024-04-25T00:13:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.363+0000 7fe148bdc640 1 -- 172.21.15.27:0/2029728681 shutdown_connections 2024-04-25T00:13:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.363+0000 7fe148bdc640 1 --2- 172.21.15.27:0/2029728681 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe144107460 0x7fe1441098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.363+0000 7fe148bdc640 1 --2- 172.21.15.27:0/2029728681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144104b30 0x7fe144106f20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.363+0000 7fe148bdc640 1 --2- 172.21.15.27:0/2029728681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe144102200 0x7fe1441045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.363+0000 7fe148bdc640 1 -- 172.21.15.27:0/2029728681 >> 172.21.15.27:0/2029728681 conn(0x7fe1440fbbf0 msgr2=0x7fe1440fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.363+0000 7fe148bdc640 1 -- 172.21.15.27:0/2029728681 shutdown_connections 2024-04-25T00:13:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.363+0000 7fe148bdc640 1 -- 172.21.15.27:0/2029728681 wait complete. 2024-04-25T00:13:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.364+0000 7fe148bdc640 1 Processor -- start 2024-04-25T00:13:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.364+0000 7fe148bdc640 1 -- start start 2024-04-25T00:13:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.364+0000 7fe148bdc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144102200 0x7fe144101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.364+0000 7fe148bdc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe144104b30 0x7fe1441001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.364+0000 7fe148bdc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe144107460 0x7fe1441006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.364+0000 7fe148bdc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe14410d410 con 0x7fe144102200 2024-04-25T00:13:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.364+0000 7fe148bdc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe14410d290 con 0x7fe144107460 2024-04-25T00:13:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.364+0000 7fe148bdc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe14410d590 con 0x7fe144104b30 2024-04-25T00:13:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.364+0000 7fe14259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144102200 0x7fe144101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.365+0000 7fe14259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144102200 0x7fe144101b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59806/0 (socket says 172.21.15.27:59806) 2024-04-25T00:13:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.365+0000 7fe14259c640 1 -- 172.21.15.27:0/997064149 learned_addr learned my addr 172.21.15.27:0/997064149 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.365+0000 7fe142d9d640 1 --2- 172.21.15.27:0/997064149 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe144107460 0x7fe1441006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.365+0000 7fe139d9b640 1 --2- 172.21.15.27:0/997064149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe144104b30 0x7fe1441001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.365+0000 7fe14259c640 1 -- 172.21.15.27:0/997064149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe144104b30 msgr2=0x7fe1441001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.365+0000 7fe14259c640 1 --2- 172.21.15.27:0/997064149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe144104b30 0x7fe1441001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.365+0000 7fe14259c640 1 -- 172.21.15.27:0/997064149 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe144107460 msgr2=0x7fe1441006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.365+0000 7fe14259c640 1 --2- 172.21.15.27:0/997064149 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe144107460 0x7fe1441006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.365+0000 7fe14259c640 1 -- 172.21.15.27:0/997064149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe144100f80 con 0x7fe144102200 2024-04-25T00:13:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.365+0000 7fe14259c640 1 --2- 172.21.15.27:0/997064149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144102200 0x7fe144101b70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe13400d770 tx=0x7fe13400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.365+0000 7fe13bfff640 1 -- 172.21.15.27:0/997064149 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe134017070 con 0x7fe144102200 2024-04-25T00:13:16.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.365+0000 7fe148bdc640 1 -- 172.21.15.27:0/997064149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1441a1c10 con 0x7fe144102200 2024-04-25T00:13:16.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.365+0000 7fe13bfff640 1 -- 172.21.15.27:0/997064149 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe134005500 con 0x7fe144102200 2024-04-25T00:13:16.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.365+0000 7fe13bfff640 1 -- 172.21.15.27:0/997064149 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe134004e00 con 0x7fe144102200 2024-04-25T00:13:16.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.365+0000 7fe148bdc640 1 -- 172.21.15.27:0/997064149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1441a2130 con 0x7fe144102200 2024-04-25T00:13:16.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.366+0000 7fe148bdc640 1 -- 172.21.15.27:0/997064149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1440fd6e0 con 0x7fe144102200 2024-04-25T00:13:16.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.367+0000 7fe13bfff640 1 -- 172.21.15.27:0/997064149 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe134004fa0 con 0x7fe144102200 2024-04-25T00:13:16.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.368+0000 7fe13bfff640 1 --2- 172.21.15.27:0/997064149 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1300a2b90 0x7fe1300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:16.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.368+0000 7fe139d9b640 1 --2- 172.21.15.27:0/997064149 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1300a2b90 0x7fe1300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:16.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.368+0000 7fe13bfff640 1 -- 172.21.15.27:0/997064149 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe1340ce980 con 0x7fe144102200 2024-04-25T00:13:16.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.368+0000 7fe139d9b640 1 --2- 172.21.15.27:0/997064149 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1300a2b90 0x7fe1300a5050 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe12c006010 tx=0x7fe12c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.371+0000 7fe13bfff640 1 -- 172.21.15.27:0/997064149 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe13409bd80 con 0x7fe144102200 2024-04-25T00:13:16.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:16.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:13:16.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:16 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3206354853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:16.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:16 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:13:16.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:16 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/744076267' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:16.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:16 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3617633669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:13:16.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.565+0000 7fe148bdc640 1 -- 172.21.15.27:0/997064149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fe14402d3c0 con 0x7fe144102200 2024-04-25T00:13:16.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.566+0000 7fe13bfff640 1 -- 172.21.15.27:0/997064149 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7fe13409b680 con 0x7fe144102200 2024-04-25T00:13:16.567 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:16.567 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-04-25T00:13:16.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:13:16.567 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-04-25T00:13:16.567 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-04-25T00:13:16.567 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:13:16.567 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-04-25T00:13:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.568+0000 7fe148bdc640 1 -- 172.21.15.27:0/997064149 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1300a2b90 msgr2=0x7fe1300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.568+0000 7fe148bdc640 1 --2- 172.21.15.27:0/997064149 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1300a2b90 0x7fe1300a5050 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe12c006010 tx=0x7fe12c000bc0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.568+0000 7fe148bdc640 1 -- 172.21.15.27:0/997064149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144102200 msgr2=0x7fe144101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.568+0000 7fe148bdc640 1 --2- 172.21.15.27:0/997064149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144102200 0x7fe144101b70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe13400d770 tx=0x7fe13400dc30 comp rx=0 tx=0).stop 2024-04-25T00:13:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.568+0000 7fe148bdc640 1 -- 172.21.15.27:0/997064149 shutdown_connections 2024-04-25T00:13:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.568+0000 7fe148bdc640 1 --2- 172.21.15.27:0/997064149 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1300a2b90 0x7fe1300a5050 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.568+0000 7fe148bdc640 1 --2- 172.21.15.27:0/997064149 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe144107460 0x7fe1441006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.568+0000 7fe148bdc640 1 --2- 172.21.15.27:0/997064149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe144104b30 0x7fe1441001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.568+0000 7fe148bdc640 1 --2- 172.21.15.27:0/997064149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144102200 0x7fe144101b70 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.568+0000 7fe148bdc640 1 -- 172.21.15.27:0/997064149 >> 172.21.15.27:0/997064149 conn(0x7fe1440fbbf0 msgr2=0x7fe144102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.568+0000 7fe148bdc640 1 -- 172.21.15.27:0/997064149 shutdown_connections 2024-04-25T00:13:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.568+0000 7fe148bdc640 1 -- 172.21.15.27:0/997064149 wait complete. 2024-04-25T00:13:16.580 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T00:13:16.580 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T00:13:16.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.693+0000 7fcc528ba640 1 -- 172.21.15.27:0/2362144104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc4c1074c0 msgr2=0x7fcc4c1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.693+0000 7fcc528ba640 1 --2- 172.21.15.27:0/2362144104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc4c1074c0 0x7fcc4c1098b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcc44009ca0 tx=0x7fcc4402a8c0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.693+0000 7fcc528ba640 1 -- 172.21.15.27:0/2362144104 shutdown_connections 2024-04-25T00:13:16.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.693+0000 7fcc528ba640 1 --2- 172.21.15.27:0/2362144104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc4c1074c0 0x7fcc4c1098b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.693+0000 7fcc528ba640 1 --2- 172.21.15.27:0/2362144104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc4c104b90 0x7fcc4c106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.693+0000 7fcc528ba640 1 --2- 172.21.15.27:0/2362144104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc4c102260 0x7fcc4c104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.693+0000 7fcc528ba640 1 -- 172.21.15.27:0/2362144104 >> 172.21.15.27:0/2362144104 conn(0x7fcc4c0fbbf0 msgr2=0x7fcc4c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:16.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.693+0000 7fcc528ba640 1 -- 172.21.15.27:0/2362144104 shutdown_connections 2024-04-25T00:13:16.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.693+0000 7fcc528ba640 1 -- 172.21.15.27:0/2362144104 wait complete. 2024-04-25T00:13:16.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.693+0000 7fcc528ba640 1 Processor -- start 2024-04-25T00:13:16.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc528ba640 1 -- start start 2024-04-25T00:13:16.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc528ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc4c102260 0x7fcc4c101bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:16.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc528ba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc4c104b90 0x7fcc4c100210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:16.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc528ba640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc4c1074c0 0x7fcc4c100750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:16.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc528ba640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c10d420 con 0x7fcc4c102260 2024-04-25T00:13:16.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc528ba640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c10d2a0 con 0x7fcc4c1074c0 2024-04-25T00:13:16.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc528ba640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c10d5a0 con 0x7fcc4c104b90 2024-04-25T00:13:16.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc4bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc4c102260 0x7fcc4c101bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:16.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc4bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc4c102260 0x7fcc4c101bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57282/0 (socket says 172.21.15.27:57282) 2024-04-25T00:13:16.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc4bfff640 1 -- 172.21.15.27:0/3577740372 learned_addr learned my addr 172.21.15.27:0/3577740372 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:16.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc4b7fe640 1 --2- 172.21.15.27:0/3577740372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc4c104b90 0x7fcc4c100210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:16.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc50e57640 1 --2- 172.21.15.27:0/3577740372 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc4c1074c0 0x7fcc4c100750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:16.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc4bfff640 1 -- 172.21.15.27:0/3577740372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc4c104b90 msgr2=0x7fcc4c100210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc4bfff640 1 --2- 172.21.15.27:0/3577740372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc4c104b90 0x7fcc4c100210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc4bfff640 1 -- 172.21.15.27:0/3577740372 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc4c1074c0 msgr2=0x7fcc4c100750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc4bfff640 1 --2- 172.21.15.27:0/3577740372 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc4c1074c0 0x7fcc4c100750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc4bfff640 1 -- 172.21.15.27:0/3577740372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc4c100fe0 con 0x7fcc4c102260 2024-04-25T00:13:16.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.694+0000 7fcc4bfff640 1 --2- 172.21.15.27:0/3577740372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc4c102260 0x7fcc4c101bd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c00c920 tx=0x7fcc3c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:16.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.695+0000 7fcc497fa640 1 -- 172.21.15.27:0/3577740372 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c007c70 con 0x7fcc4c102260 2024-04-25T00:13:16.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.695+0000 7fcc528ba640 1 -- 172.21.15.27:0/3577740372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc4c1a1b60 con 0x7fcc4c102260 2024-04-25T00:13:16.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.695+0000 7fcc497fa640 1 -- 172.21.15.27:0/3577740372 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc3c007e10 con 0x7fcc4c102260 2024-04-25T00:13:16.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.695+0000 7fcc497fa640 1 -- 172.21.15.27:0/3577740372 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c018890 con 0x7fcc4c102260 2024-04-25T00:13:16.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.695+0000 7fcc528ba640 1 -- 172.21.15.27:0/3577740372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc4c1a2100 con 0x7fcc4c102260 2024-04-25T00:13:16.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.696+0000 7fcc528ba640 1 -- 172.21.15.27:0/3577740372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc4c0fd6e0 con 0x7fcc4c102260 2024-04-25T00:13:16.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.697+0000 7fcc497fa640 1 -- 172.21.15.27:0/3577740372 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcc3c018a70 con 0x7fcc4c102260 2024-04-25T00:13:16.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.697+0000 7fcc497fa640 1 --2- 172.21.15.27:0/3577740372 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fcc340a2b90 0x7fcc340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:16.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.698+0000 7fcc4b7fe640 1 --2- 172.21.15.27:0/3577740372 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fcc340a2b90 0x7fcc340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:16.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.698+0000 7fcc497fa640 1 -- 172.21.15.27:0/3577740372 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcc3c0cecc0 con 0x7fcc4c102260 2024-04-25T00:13:16.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.698+0000 7fcc4b7fe640 1 --2- 172.21.15.27:0/3577740372 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fcc340a2b90 0x7fcc340a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c101940 tx=0x7fcc4000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:16.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.702+0000 7fcc497fa640 1 -- 172.21.15.27:0/3577740372 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcc3c014030 con 0x7fcc4c102260 2024-04-25T00:13:16.712 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.712 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:16.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:13:16.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:16 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3206354853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:16.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:16 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:13:16.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:16 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/744076267' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:16.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:16 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3617633669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:13:16.714 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.714 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.714 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.714 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.714 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.714 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.714 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:16.714 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:13:16.714 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:16.714 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:16 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3206354853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:16.715 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:16 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:13:16.715 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:16 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/744076267' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:16.715 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:16 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3617633669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:13:16.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:16.887+0000 7fcc528ba640 1 -- 172.21.15.27:0/3577740372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fcc4c02d370 con 0x7fcc4c102260 2024-04-25T00:13:17.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.498+0000 7fcc497fa640 1 -- 172.21.15.27:0/3577740372 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7fcc3c09c0c0 con 0x7fcc4c102260 2024-04-25T00:13:17.499 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-04-25T00:13:17.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.500+0000 7fcc528ba640 1 -- 172.21.15.27:0/3577740372 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fcc340a2b90 msgr2=0x7fcc340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:17.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.500+0000 7fcc528ba640 1 --2- 172.21.15.27:0/3577740372 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fcc340a2b90 0x7fcc340a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c101940 tx=0x7fcc4000b390 comp rx=0 tx=0).stop 2024-04-25T00:13:17.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.500+0000 7fcc528ba640 1 -- 172.21.15.27:0/3577740372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc4c102260 msgr2=0x7fcc4c101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:17.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.500+0000 7fcc528ba640 1 --2- 172.21.15.27:0/3577740372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc4c102260 0x7fcc4c101bd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c00c920 tx=0x7fcc3c00cde0 comp rx=0 tx=0).stop 2024-04-25T00:13:17.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.500+0000 7fcc528ba640 1 -- 172.21.15.27:0/3577740372 shutdown_connections 2024-04-25T00:13:17.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.500+0000 7fcc528ba640 1 --2- 172.21.15.27:0/3577740372 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fcc340a2b90 0x7fcc340a5050 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:17.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.500+0000 7fcc528ba640 1 --2- 172.21.15.27:0/3577740372 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc4c1074c0 0x7fcc4c100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:17.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.500+0000 7fcc528ba640 1 --2- 172.21.15.27:0/3577740372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc4c104b90 0x7fcc4c100210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:17.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.500+0000 7fcc528ba640 1 --2- 172.21.15.27:0/3577740372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc4c102260 0x7fcc4c101bd0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:17.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.500+0000 7fcc528ba640 1 -- 172.21.15.27:0/3577740372 >> 172.21.15.27:0/3577740372 conn(0x7fcc4c0fbbf0 msgr2=0x7fcc4c102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:17.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.500+0000 7fcc528ba640 1 -- 172.21.15.27:0/3577740372 shutdown_connections 2024-04-25T00:13:17.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.500+0000 7fcc528ba640 1 -- 172.21.15.27:0/3577740372 wait complete. 2024-04-25T00:13:17.514 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-25T00:13:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.626+0000 7fe52ec79640 1 -- 172.21.15.27:0/3189965259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5281017a0 msgr2=0x7fe528101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.626+0000 7fe52ec79640 1 --2- 172.21.15.27:0/3189965259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5281017a0 0x7fe528101bc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009900 tx=0x7fe51c02a410 comp rx=0 tx=0).stop 2024-04-25T00:13:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.626+0000 7fe52ec79640 1 -- 172.21.15.27:0/3189965259 shutdown_connections 2024-04-25T00:13:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.626+0000 7fe52ec79640 1 --2- 172.21.15.27:0/3189965259 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe528103350 0x7fe528109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.626+0000 7fe52ec79640 1 --2- 172.21.15.27:0/3189965259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe528102990 0x7fe528102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.626+0000 7fe52ec79640 1 --2- 172.21.15.27:0/3189965259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5281017a0 0x7fe528101bc0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.626+0000 7fe52ec79640 1 -- 172.21.15.27:0/3189965259 >> 172.21.15.27:0/3189965259 conn(0x7fe5280fcd10 msgr2=0x7fe5280ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.626+0000 7fe52ec79640 1 -- 172.21.15.27:0/3189965259 shutdown_connections 2024-04-25T00:13:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.626+0000 7fe52ec79640 1 -- 172.21.15.27:0/3189965259 wait complete. 2024-04-25T00:13:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.626+0000 7fe52ec79640 1 Processor -- start 2024-04-25T00:13:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.626+0000 7fe52ec79640 1 -- start start 2024-04-25T00:13:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe52ec79640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5281017a0 0x7fe528197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe52ec79640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe528102990 0x7fe528198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe52ec79640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe528103350 0x7fe52819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe52ec79640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe52810d410 con 0x7fe528103350 2024-04-25T00:13:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe52ec79640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe52810d290 con 0x7fe5281017a0 2024-04-25T00:13:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe52ec79640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe52810d590 con 0x7fe528102990 2024-04-25T00:13:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe52d216640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe528103350 0x7fe52819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe52d216640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe528103350 0x7fe52819f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57314/0 (socket says 172.21.15.27:57314) 2024-04-25T00:13:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe52d216640 1 -- 172.21.15.27:0/4196073706 learned_addr learned my addr 172.21.15.27:0/4196073706 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe52ca15640 1 --2- 172.21.15.27:0/4196073706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5281017a0 0x7fe528197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe527fff640 1 --2- 172.21.15.27:0/4196073706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe528102990 0x7fe528198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe52d216640 1 -- 172.21.15.27:0/4196073706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe528102990 msgr2=0x7fe528198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:17.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe52d216640 1 --2- 172.21.15.27:0/4196073706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe528102990 0x7fe528198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:17.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe52d216640 1 -- 172.21.15.27:0/4196073706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5281017a0 msgr2=0x7fe528197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:17.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe52d216640 1 --2- 172.21.15.27:0/4196073706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5281017a0 0x7fe528197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:17.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe52d216640 1 -- 172.21.15.27:0/4196073706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe52819f700 con 0x7fe528103350 2024-04-25T00:13:17.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe52d216640 1 --2- 172.21.15.27:0/4196073706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe528103350 0x7fe52819f090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe520009fd0 tx=0x7fe52000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:17.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe525ffb640 1 -- 172.21.15.27:0/4196073706 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe520017070 con 0x7fe528103350 2024-04-25T00:13:17.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe525ffb640 1 -- 172.21.15.27:0/4196073706 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe520003e50 con 0x7fe528103350 2024-04-25T00:13:17.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe52ec79640 1 -- 172.21.15.27:0/4196073706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe52819f9f0 con 0x7fe528103350 2024-04-25T00:13:17.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.627+0000 7fe525ffb640 1 -- 172.21.15.27:0/4196073706 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5200106c0 con 0x7fe528103350 2024-04-25T00:13:17.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.628+0000 7fe52ec79640 1 -- 172.21.15.27:0/4196073706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe52819fef0 con 0x7fe528103350 2024-04-25T00:13:17.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.629+0000 7fe52ec79640 1 -- 172.21.15.27:0/4196073706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe528108b60 con 0x7fe528103350 2024-04-25T00:13:17.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.630+0000 7fe525ffb640 1 -- 172.21.15.27:0/4196073706 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe520003970 con 0x7fe528103350 2024-04-25T00:13:17.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.630+0000 7fe525ffb640 1 --2- 172.21.15.27:0/4196073706 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe5180a2b90 0x7fe5180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:17.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.630+0000 7fe52ca15640 1 --2- 172.21.15.27:0/4196073706 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe5180a2b90 0x7fe5180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:17.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.630+0000 7fe525ffb640 1 -- 172.21.15.27:0/4196073706 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7fe5200cea20 con 0x7fe528103350 2024-04-25T00:13:17.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.631+0000 7fe52ca15640 1 --2- 172.21.15.27:0/4196073706 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe5180a2b90 0x7fe5180a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe51c005ab0 tx=0x7fe51c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.635+0000 7fe525ffb640 1 -- 172.21.15.27:0/4196073706 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe52009b660 con 0x7fe528103350 2024-04-25T00:13:17.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:17 smithi027 ceph-mon[29208]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:17.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:17 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/997064149' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T00:13:17.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:17 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3577740372' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T00:13:17.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:17 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:17.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:17 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:17.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:17 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:17.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:17 smithi139 ceph-mon[32833]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:17.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:17 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/997064149' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T00:13:17.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:17 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3577740372' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T00:13:17.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:17 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:17.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:17 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:17.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:17 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:17.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:17 smithi153 ceph-mon[32896]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:17.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:17 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/997064149' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T00:13:17.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:17 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3577740372' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T00:13:17.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:17 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:17.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:17 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:17.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:17 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:17.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:17.820+0000 7fe52ec79640 1 -- 172.21.15.27:0/4196073706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fe52802d170 con 0x7fe528103350 2024-04-25T00:13:18.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.503+0000 7fe525ffb640 1 -- 172.21.15.27:0/4196073706 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7fe52009f960 con 0x7fe528103350 2024-04-25T00:13:18.504 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-04-25T00:13:18.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.506+0000 7fe52ec79640 1 -- 172.21.15.27:0/4196073706 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe5180a2b90 msgr2=0x7fe5180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:18.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.506+0000 7fe52ec79640 1 --2- 172.21.15.27:0/4196073706 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe5180a2b90 0x7fe5180a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe51c005ab0 tx=0x7fe51c000bc0 comp rx=0 tx=0).stop 2024-04-25T00:13:18.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.506+0000 7fe52ec79640 1 -- 172.21.15.27:0/4196073706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe528103350 msgr2=0x7fe52819f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:18.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.506+0000 7fe52ec79640 1 --2- 172.21.15.27:0/4196073706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe528103350 0x7fe52819f090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe520009fd0 tx=0x7fe52000b260 comp rx=0 tx=0).stop 2024-04-25T00:13:18.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.506+0000 7fe52ec79640 1 -- 172.21.15.27:0/4196073706 shutdown_connections 2024-04-25T00:13:18.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.506+0000 7fe52ec79640 1 --2- 172.21.15.27:0/4196073706 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe5180a2b90 0x7fe5180a5050 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:18.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.506+0000 7fe52ec79640 1 --2- 172.21.15.27:0/4196073706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe528103350 0x7fe52819f090 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:18.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.506+0000 7fe52ec79640 1 --2- 172.21.15.27:0/4196073706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe528102990 0x7fe528198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:18.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.506+0000 7fe52ec79640 1 --2- 172.21.15.27:0/4196073706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5281017a0 0x7fe528197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:18.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.506+0000 7fe52ec79640 1 -- 172.21.15.27:0/4196073706 >> 172.21.15.27:0/4196073706 conn(0x7fe5280fcd10 msgr2=0x7fe528105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:18.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.506+0000 7fe52ec79640 1 -- 172.21.15.27:0/4196073706 shutdown_connections 2024-04-25T00:13:18.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.506+0000 7fe52ec79640 1 -- 172.21.15.27:0/4196073706 wait complete. 2024-04-25T00:13:18.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T00:13:18.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.636+0000 7f33bdd9b640 1 -- 172.21.15.27:0/76996076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33b8107460 msgr2=0x7f33b81098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:18.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.636+0000 7f33bdd9b640 1 --2- 172.21.15.27:0/76996076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33b8107460 0x7f33b81098b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f33b0009900 tx=0x7f33b002a3f0 comp rx=0 tx=0).stop 2024-04-25T00:13:18.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.636+0000 7f33bdd9b640 1 -- 172.21.15.27:0/76996076 shutdown_connections 2024-04-25T00:13:18.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.636+0000 7f33bdd9b640 1 --2- 172.21.15.27:0/76996076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33b8107460 0x7f33b81098b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:18.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.636+0000 7f33bdd9b640 1 --2- 172.21.15.27:0/76996076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f33b8104b30 0x7f33b8106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:18.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.636+0000 7f33bdd9b640 1 --2- 172.21.15.27:0/76996076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b8102200 0x7f33b81045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:18.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.636+0000 7f33bdd9b640 1 -- 172.21.15.27:0/76996076 >> 172.21.15.27:0/76996076 conn(0x7f33b80fbbf0 msgr2=0x7f33b80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:18.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.637+0000 7f33bdd9b640 1 -- 172.21.15.27:0/76996076 shutdown_connections 2024-04-25T00:13:18.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.637+0000 7f33bdd9b640 1 -- 172.21.15.27:0/76996076 wait complete. 2024-04-25T00:13:18.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.637+0000 7f33bdd9b640 1 Processor -- start 2024-04-25T00:13:18.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.637+0000 7f33bdd9b640 1 -- start start 2024-04-25T00:13:18.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.637+0000 7f33bdd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33b8102200 0x7f33b8197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:18.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.637+0000 7f33bdd9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f33b8104b30 0x7f33b8197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:18.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.637+0000 7f33bdd9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b8107460 0x7f33b819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:18.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.637+0000 7f33bdd9b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f33b810d400 con 0x7f33b8102200 2024-04-25T00:13:18.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.637+0000 7f33bdd9b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f33b810d280 con 0x7f33b8104b30 2024-04-25T00:13:18.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.637+0000 7f33bdd9b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f33b810d580 con 0x7f33b8107460 2024-04-25T00:13:18.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.638+0000 7f33b77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33b8102200 0x7f33b8197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:18.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.638+0000 7f33b77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33b8102200 0x7f33b8197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57322/0 (socket says 172.21.15.27:57322) 2024-04-25T00:13:18.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.638+0000 7f33b77fe640 1 -- 172.21.15.27:0/4275348055 learned_addr learned my addr 172.21.15.27:0/4275348055 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:18.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.638+0000 7f33b6ffd640 1 --2- 172.21.15.27:0/4275348055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f33b8104b30 0x7f33b8197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:18.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.638+0000 7f33b7fff640 1 --2- 172.21.15.27:0/4275348055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b8107460 0x7f33b819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:18.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.638+0000 7f33b77fe640 1 -- 172.21.15.27:0/4275348055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b8107460 msgr2=0x7f33b819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:18.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.638+0000 7f33b77fe640 1 --2- 172.21.15.27:0/4275348055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b8107460 0x7f33b819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:18.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.638+0000 7f33b77fe640 1 -- 172.21.15.27:0/4275348055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f33b8104b30 msgr2=0x7f33b8197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:18.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.638+0000 7f33b77fe640 1 --2- 172.21.15.27:0/4275348055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f33b8104b30 0x7f33b8197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:18.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.638+0000 7f33b77fe640 1 -- 172.21.15.27:0/4275348055 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b819f660 con 0x7f33b8102200 2024-04-25T00:13:18.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.638+0000 7f33b7fff640 1 --2- 172.21.15.27:0/4275348055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b8107460 0x7f33b819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:18.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.638+0000 7f33b77fe640 1 --2- 172.21.15.27:0/4275348055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33b8102200 0x7f33b8197a40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f33a800db80 tx=0x7f33a800df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:18.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.638+0000 7f33b4ff9640 1 -- 172.21.15.27:0/4275348055 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a8017070 con 0x7f33b8102200 2024-04-25T00:13:18.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.638+0000 7f33b4ff9640 1 -- 172.21.15.27:0/4275348055 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33a8003e50 con 0x7f33b8102200 2024-04-25T00:13:18.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.638+0000 7f33bdd9b640 1 -- 172.21.15.27:0/4275348055 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33b819f950 con 0x7f33b8102200 2024-04-25T00:13:18.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.638+0000 7f33b4ff9640 1 -- 172.21.15.27:0/4275348055 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a80104c0 con 0x7f33b8102200 2024-04-25T00:13:18.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.638+0000 7f33bdd9b640 1 -- 172.21.15.27:0/4275348055 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33b819fea0 con 0x7f33b8102200 2024-04-25T00:13:18.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.639+0000 7f33bdd9b640 1 -- 172.21.15.27:0/4275348055 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33b80fd6e0 con 0x7f33b8102200 2024-04-25T00:13:18.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.641+0000 7f33b4ff9640 1 -- 172.21.15.27:0/4275348055 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f33a8003970 con 0x7f33b8102200 2024-04-25T00:13:18.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.641+0000 7f33b4ff9640 1 --2- 172.21.15.27:0/4275348055 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f33a40a2b90 0x7f33a40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:18.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.641+0000 7f33b6ffd640 1 --2- 172.21.15.27:0/4275348055 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f33a40a2b90 0x7f33a40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:18.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.641+0000 7f33b4ff9640 1 -- 172.21.15.27:0/4275348055 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f33a80cef70 con 0x7f33b8102200 2024-04-25T00:13:18.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.642+0000 7f33b6ffd640 1 --2- 172.21.15.27:0/4275348055 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f33a40a2b90 0x7f33a40a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f33b8198f50 tx=0x7f33a000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:18.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.646+0000 7f33b4ff9640 1 -- 172.21.15.27:0/4275348055 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f33a809c080 con 0x7f33b8102200 2024-04-25T00:13:18.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:18 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:18.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:18 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:18.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:18 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:18.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:18 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:18.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:18 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:13:18.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:18 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:18.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:18 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3577740372' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T00:13:18.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:18 smithi027 ceph-mon[29208]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T00:13:18.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:18 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4196073706' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-25T00:13:18.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:18 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:18.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:18 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:18.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:18 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:18.755 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:18 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:18.755 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:18 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:13:18.755 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:18 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:18.755 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:18 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3577740372' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T00:13:18.755 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:18 smithi139 ceph-mon[32833]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T00:13:18.755 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:18 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4196073706' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-25T00:13:18.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:18 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:18.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:18 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:18.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:18 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:18.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:18 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:13:18.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:18 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:13:18.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:18 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:13:18.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:18 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3577740372' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T00:13:18.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:18 smithi153 ceph-mon[32896]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T00:13:18.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:18 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4196073706' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-25T00:13:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:18.870+0000 7f33bdd9b640 1 -- 172.21.15.27:0/4275348055 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f33b80623b0 con 0x7f33b8102200 2024-04-25T00:13:19.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.504+0000 7f33b4ff9640 1 -- 172.21.15.27:0/4275348055 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f33a809b980 con 0x7f33b8102200 2024-04-25T00:13:19.505 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T00:13:19.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.507+0000 7f33bdd9b640 1 -- 172.21.15.27:0/4275348055 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f33a40a2b90 msgr2=0x7f33a40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:19.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.507+0000 7f33bdd9b640 1 --2- 172.21.15.27:0/4275348055 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f33a40a2b90 0x7f33a40a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f33b8198f50 tx=0x7f33a000a2d0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.507+0000 7f33bdd9b640 1 -- 172.21.15.27:0/4275348055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33b8102200 msgr2=0x7f33b8197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:19.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.507+0000 7f33bdd9b640 1 --2- 172.21.15.27:0/4275348055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33b8102200 0x7f33b8197a40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f33a800db80 tx=0x7f33a800df80 comp rx=0 tx=0).stop 2024-04-25T00:13:19.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.507+0000 7f33bdd9b640 1 -- 172.21.15.27:0/4275348055 shutdown_connections 2024-04-25T00:13:19.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.507+0000 7f33bdd9b640 1 --2- 172.21.15.27:0/4275348055 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f33a40a2b90 0x7f33a40a5050 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.507+0000 7f33bdd9b640 1 --2- 172.21.15.27:0/4275348055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b8107460 0x7f33b819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.507+0000 7f33bdd9b640 1 --2- 172.21.15.27:0/4275348055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f33b8104b30 0x7f33b8197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.507+0000 7f33bdd9b640 1 --2- 172.21.15.27:0/4275348055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33b8102200 0x7f33b8197a40 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.507+0000 7f33bdd9b640 1 -- 172.21.15.27:0/4275348055 >> 172.21.15.27:0/4275348055 conn(0x7f33b80fbbf0 msgr2=0x7f33b8102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:19.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.507+0000 7f33bdd9b640 1 -- 172.21.15.27:0/4275348055 shutdown_connections 2024-04-25T00:13:19.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.507+0000 7f33bdd9b640 1 -- 172.21.15.27:0/4275348055 wait complete. 2024-04-25T00:13:19.522 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-25T00:13:19.522 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-25T00:13:19.661 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:19 smithi027 ceph-mon[29208]: pgmap v286: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:19.661 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:19 smithi027 ceph-mon[29208]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T00:13:19.661 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:19 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4196073706' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-25T00:13:19.661 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:19 smithi027 ceph-mon[29208]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T00:13:19.662 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:19 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4275348055' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T00:13:19.662 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:19 smithi027 ceph-mon[29208]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T00:13:19.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.674+0000 7f1cf293b640 1 -- 172.21.15.27:0/1186445382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cec101890 msgr2=0x7f1cec101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.674+0000 7f1cf293b640 1 --2- 172.21.15.27:0/1186445382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cec101890 0x7f1cec101d10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0009900 tx=0x7f1ce002a410 comp rx=0 tx=0).stop 2024-04-25T00:13:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.674+0000 7f1cf293b640 1 -- 172.21.15.27:0/1186445382 shutdown_connections 2024-04-25T00:13:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.674+0000 7f1cf293b640 1 --2- 172.21.15.27:0/1186445382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cec102250 0x7f1cec108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.674+0000 7f1cf293b640 1 --2- 172.21.15.27:0/1186445382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cec101890 0x7f1cec101d10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.674+0000 7f1cf293b640 1 --2- 172.21.15.27:0/1186445382 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1cec1006a0 0x7f1cec100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.674+0000 7f1cf293b640 1 -- 172.21.15.27:0/1186445382 >> 172.21.15.27:0/1186445382 conn(0x7f1cec0fbc30 msgr2=0x7f1cec0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.674+0000 7f1cf293b640 1 -- 172.21.15.27:0/1186445382 shutdown_connections 2024-04-25T00:13:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.674+0000 7f1cf293b640 1 -- 172.21.15.27:0/1186445382 wait complete. 2024-04-25T00:13:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.675+0000 7f1cf293b640 1 Processor -- start 2024-04-25T00:13:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.675+0000 7f1cf293b640 1 -- start start 2024-04-25T00:13:19.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.675+0000 7f1cf293b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cec1006a0 0x7f1cec104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:19.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.675+0000 7f1cf293b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cec101890 0x7f1cec102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:19.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.675+0000 7f1cf293b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1cec102250 0x7f1cec1031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:19.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.675+0000 7f1cf293b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec10c320 con 0x7f1cec101890 2024-04-25T00:13:19.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.675+0000 7f1cf293b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec10c1a0 con 0x7f1cec102250 2024-04-25T00:13:19.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.675+0000 7f1cf293b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec10c4a0 con 0x7f1cec1006a0 2024-04-25T00:13:19.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.675+0000 7f1ceb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cec101890 0x7f1cec102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:19.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.675+0000 7f1ceb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cec101890 0x7f1cec102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57344/0 (socket says 172.21.15.27:57344) 2024-04-25T00:13:19.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.675+0000 7f1ceb7fe640 1 -- 172.21.15.27:0/1286959783 learned_addr learned my addr 172.21.15.27:0/1286959783 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:19.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.675+0000 7f1cf0ed8640 1 --2- 172.21.15.27:0/1286959783 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1cec102250 0x7f1cec1031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:19.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.675+0000 7f1cebfff640 1 --2- 172.21.15.27:0/1286959783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cec1006a0 0x7f1cec104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:19.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.676+0000 7f1ceb7fe640 1 -- 172.21.15.27:0/1286959783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cec1006a0 msgr2=0x7f1cec104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:19.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.676+0000 7f1ceb7fe640 1 --2- 172.21.15.27:0/1286959783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cec1006a0 0x7f1cec104630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.676+0000 7f1ceb7fe640 1 -- 172.21.15.27:0/1286959783 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1cec102250 msgr2=0x7f1cec1031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:19.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.676+0000 7f1ceb7fe640 1 --2- 172.21.15.27:0/1286959783 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1cec102250 0x7f1cec1031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.676+0000 7f1ceb7fe640 1 -- 172.21.15.27:0/1286959783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cec103a40 con 0x7f1cec101890 2024-04-25T00:13:19.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.676+0000 7f1ceb7fe640 1 --2- 172.21.15.27:0/1286959783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cec101890 0x7f1cec102c70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0005940 tx=0x7f1ce0004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:19.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.676+0000 7f1ce97fa640 1 -- 172.21.15.27:0/1286959783 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce003f030 con 0x7f1cec101890 2024-04-25T00:13:19.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.676+0000 7f1cf293b640 1 -- 172.21.15.27:0/1286959783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cec1a1bd0 con 0x7f1cec101890 2024-04-25T00:13:19.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.676+0000 7f1ce97fa640 1 -- 172.21.15.27:0/1286959783 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ce0003ce0 con 0x7f1cec101890 2024-04-25T00:13:19.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.676+0000 7f1ce97fa640 1 -- 172.21.15.27:0/1286959783 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0035070 con 0x7f1cec101890 2024-04-25T00:13:19.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.676+0000 7f1cf293b640 1 -- 172.21.15.27:0/1286959783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cec1a2140 con 0x7f1cec101890 2024-04-25T00:13:19.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.677+0000 7f1cf293b640 1 -- 172.21.15.27:0/1286959783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cec1a2370 con 0x7f1cec101890 2024-04-25T00:13:19.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.678+0000 7f1ce97fa640 1 -- 172.21.15.27:0/1286959783 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1ce0042030 con 0x7f1cec101890 2024-04-25T00:13:19.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.679+0000 7f1ce97fa640 1 --2- 172.21.15.27:0/1286959783 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1cd40a2b90 0x7f1cd40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:19.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.680+0000 7f1cebfff640 1 --2- 172.21.15.27:0/1286959783 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1cd40a2b90 0x7f1cd40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:19.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.680+0000 7f1ce97fa640 1 -- 172.21.15.27:0/1286959783 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f1ce00ef9f0 con 0x7f1cec101890 2024-04-25T00:13:19.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.680+0000 7f1cebfff640 1 --2- 172.21.15.27:0/1286959783 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1cd40a2b90 0x7f1cd40a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1cec1016f0 tx=0x7f1cdc00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:19.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.684+0000 7f1ce97fa640 1 -- 172.21.15.27:0/1286959783 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1ce00bc430 con 0x7f1cec101890 2024-04-25T00:13:19.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:19 smithi139 ceph-mon[32833]: pgmap v286: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:19.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:19 smithi139 ceph-mon[32833]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T00:13:19.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:19 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4196073706' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-25T00:13:19.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:19 smithi139 ceph-mon[32833]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T00:13:19.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:19 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4275348055' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T00:13:19.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:19 smithi139 ceph-mon[32833]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T00:13:19.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:19 smithi153 ceph-mon[32896]: pgmap v286: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:19.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:19 smithi153 ceph-mon[32896]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T00:13:19.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:19 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4196073706' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-25T00:13:19.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:19 smithi153 ceph-mon[32896]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T00:13:19.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:19 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4275348055' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T00:13:19.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:19 smithi153 ceph-mon[32896]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T00:13:19.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:19.879+0000 7f1cf293b640 1 -- 172.21.15.27:0/1286959783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f1cec1a2540 con 0x7f1cec101890 2024-04-25T00:13:20.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.404+0000 7f1ce97fa640 1 -- 172.21.15.27:0/1286959783 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7f1ce00c0730 con 0x7f1cec101890 2024-04-25T00:13:20.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.406+0000 7f1cf293b640 1 -- 172.21.15.27:0/1286959783 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1cd40a2b90 msgr2=0x7f1cd40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.406+0000 7f1cf293b640 1 --2- 172.21.15.27:0/1286959783 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1cd40a2b90 0x7f1cd40a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1cec1016f0 tx=0x7f1cdc00b390 comp rx=0 tx=0).stop 2024-04-25T00:13:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.406+0000 7f1cf293b640 1 -- 172.21.15.27:0/1286959783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cec101890 msgr2=0x7f1cec102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.406+0000 7f1cf293b640 1 --2- 172.21.15.27:0/1286959783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cec101890 0x7f1cec102c70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0005940 tx=0x7f1ce0004360 comp rx=0 tx=0).stop 2024-04-25T00:13:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.406+0000 7f1cf293b640 1 -- 172.21.15.27:0/1286959783 shutdown_connections 2024-04-25T00:13:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.406+0000 7f1cf293b640 1 --2- 172.21.15.27:0/1286959783 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1cd40a2b90 0x7f1cd40a5050 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.406+0000 7f1cf293b640 1 --2- 172.21.15.27:0/1286959783 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1cec102250 0x7f1cec1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.406+0000 7f1cf293b640 1 --2- 172.21.15.27:0/1286959783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cec101890 0x7f1cec102c70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.406+0000 7f1cf293b640 1 --2- 172.21.15.27:0/1286959783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cec1006a0 0x7f1cec104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.406+0000 7f1cf293b640 1 -- 172.21.15.27:0/1286959783 >> 172.21.15.27:0/1286959783 conn(0x7f1cec0fbc30 msgr2=0x7f1cec0fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.407+0000 7f1cf293b640 1 -- 172.21.15.27:0/1286959783 shutdown_connections 2024-04-25T00:13:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.407+0000 7f1cf293b640 1 -- 172.21.15.27:0/1286959783 wait complete. 2024-04-25T00:13:20.420 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T00:13:20.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.533+0000 7fe33b76a640 1 -- 172.21.15.27:0/4149667449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334102230 msgr2=0x7fe334108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:20.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.533+0000 7fe33b76a640 1 --2- 172.21.15.27:0/4149667449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334102230 0x7fe334108ac0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe32c009ca0 tx=0x7fe32c02a8c0 comp rx=0 tx=0).stop 2024-04-25T00:13:20.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.534+0000 7fe33b76a640 1 -- 172.21.15.27:0/4149667449 shutdown_connections 2024-04-25T00:13:20.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.534+0000 7fe33b76a640 1 --2- 172.21.15.27:0/4149667449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334102230 0x7fe334108ac0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:20.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.534+0000 7fe33b76a640 1 --2- 172.21.15.27:0/4149667449 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe334101870 0x7fe334101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:20.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.534+0000 7fe33b76a640 1 --2- 172.21.15.27:0/4149667449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe334100680 0x7fe334100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:20.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.534+0000 7fe33b76a640 1 -- 172.21.15.27:0/4149667449 >> 172.21.15.27:0/4149667449 conn(0x7fe3340fbbf0 msgr2=0x7fe3340fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:20.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.534+0000 7fe33b76a640 1 -- 172.21.15.27:0/4149667449 shutdown_connections 2024-04-25T00:13:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.534+0000 7fe33b76a640 1 -- 172.21.15.27:0/4149667449 wait complete. 2024-04-25T00:13:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.534+0000 7fe33b76a640 1 Processor -- start 2024-04-25T00:13:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.534+0000 7fe33b76a640 1 -- start start 2024-04-25T00:13:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.534+0000 7fe33b76a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe334100680 0x7fe3340694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.534+0000 7fe33b76a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe334101870 0x7fe33419c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.534+0000 7fe33b76a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334102230 0x7fe33419ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.534+0000 7fe33b76a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe33410c360 con 0x7fe334102230 2024-04-25T00:13:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.534+0000 7fe33b76a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe33410c1e0 con 0x7fe334101870 2024-04-25T00:13:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.534+0000 7fe33b76a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe33410c4e0 con 0x7fe334100680 2024-04-25T00:13:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.534+0000 7fe339d07640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334102230 0x7fe33419ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.535+0000 7fe339d07640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334102230 0x7fe33419ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57356/0 (socket says 172.21.15.27:57356) 2024-04-25T00:13:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.535+0000 7fe339d07640 1 -- 172.21.15.27:0/3681219608 learned_addr learned my addr 172.21.15.27:0/3681219608 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.535+0000 7fe339506640 1 --2- 172.21.15.27:0/3681219608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe334100680 0x7fe3340694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.535+0000 7fe338d05640 1 --2- 172.21.15.27:0/3681219608 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe334101870 0x7fe33419c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.535+0000 7fe339d07640 1 -- 172.21.15.27:0/3681219608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe334100680 msgr2=0x7fe3340694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.535+0000 7fe339d07640 1 --2- 172.21.15.27:0/3681219608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe334100680 0x7fe3340694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.535+0000 7fe339d07640 1 -- 172.21.15.27:0/3681219608 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe334101870 msgr2=0x7fe33419c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.535+0000 7fe339d07640 1 --2- 172.21.15.27:0/3681219608 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe334101870 0x7fe33419c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.535+0000 7fe339d07640 1 -- 172.21.15.27:0/3681219608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe33419f610 con 0x7fe334102230 2024-04-25T00:13:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.535+0000 7fe339d07640 1 --2- 172.21.15.27:0/3681219608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334102230 0x7fe33419ed50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0098d0 tx=0x7fe32c02f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.535+0000 7fe32a7fc640 1 -- 172.21.15.27:0/3681219608 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c02fc30 con 0x7fe334102230 2024-04-25T00:13:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.535+0000 7fe32a7fc640 1 -- 172.21.15.27:0/3681219608 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe32c003e50 con 0x7fe334102230 2024-04-25T00:13:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.535+0000 7fe33b76a640 1 -- 172.21.15.27:0/3681219608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe33419f8a0 con 0x7fe334102230 2024-04-25T00:13:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.535+0000 7fe32a7fc640 1 -- 172.21.15.27:0/3681219608 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c0305c0 con 0x7fe334102230 2024-04-25T00:13:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.535+0000 7fe33b76a640 1 -- 172.21.15.27:0/3681219608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe33419fec0 con 0x7fe334102230 2024-04-25T00:13:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.536+0000 7fe33b76a640 1 -- 172.21.15.27:0/3681219608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3341078d0 con 0x7fe334102230 2024-04-25T00:13:20.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.537+0000 7fe32a7fc640 1 -- 172.21.15.27:0/3681219608 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe32c003970 con 0x7fe334102230 2024-04-25T00:13:20.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.538+0000 7fe32a7fc640 1 --2- 172.21.15.27:0/3681219608 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe3200a2c10 0x7fe3200a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:20.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.538+0000 7fe339506640 1 --2- 172.21.15.27:0/3681219608 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe3200a2c10 0x7fe3200a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:20.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.538+0000 7fe32a7fc640 1 -- 172.21.15.27:0/3681219608 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7fe32c0f0440 con 0x7fe334102230 2024-04-25T00:13:20.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.539+0000 7fe339506640 1 --2- 172.21.15.27:0/3681219608 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe3200a2c10 0x7fe3200a50d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe3300058c0 tx=0x7fe330005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:20.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.543+0000 7fe32a7fc640 1 -- 172.21.15.27:0/3681219608 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe32c0bd490 con 0x7fe334102230 2024-04-25T00:13:20.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:20 smithi027 ceph-mon[29208]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T00:13:20.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:20 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4275348055' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T00:13:20.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:20 smithi027 ceph-mon[29208]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T00:13:20.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:20 smithi027 ceph-mon[29208]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T00:13:20.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:20 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1286959783' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-25T00:13:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:20.731+0000 7fe33b76a640 1 -- 172.21.15.27:0/3681219608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fe334062340 con 0x7fe334102230 2024-04-25T00:13:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:20 smithi139 ceph-mon[32833]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T00:13:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:20 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4275348055' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T00:13:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:20 smithi139 ceph-mon[32833]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T00:13:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:20 smithi139 ceph-mon[32833]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T00:13:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:20 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1286959783' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-25T00:13:20.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:20 smithi153 ceph-mon[32896]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T00:13:20.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:20 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4275348055' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T00:13:20.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:20 smithi153 ceph-mon[32896]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T00:13:20.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:20 smithi153 ceph-mon[32896]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T00:13:20.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:20 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1286959783' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-25T00:13:21.678 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:21 smithi153 ceph-mon[32896]: pgmap v289: 97 pgs: 31 creating+peering, 15 active+clean, 51 unknown; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:21.678 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:21 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1286959783' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-25T00:13:21.678 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:21 smithi153 ceph-mon[32896]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T00:13:21.678 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:21 smithi153 ceph-mon[32896]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T00:13:21.678 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:21 smithi153 ceph-mon[32896]: Cluster is now healthy 2024-04-25T00:13:21.678 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:21 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:13:21.679 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:21 smithi153 ceph-mon[32896]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T00:13:21.679 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:21 smithi153 ceph-mon[32896]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T00:13:21.679 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:21 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3681219608' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T00:13:21.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:21 smithi027 ceph-mon[29208]: pgmap v289: 97 pgs: 31 creating+peering, 15 active+clean, 51 unknown; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:21.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:21 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1286959783' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-25T00:13:21.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:21 smithi027 ceph-mon[29208]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T00:13:21.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:21 smithi027 ceph-mon[29208]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T00:13:21.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:21 smithi027 ceph-mon[29208]: Cluster is now healthy 2024-04-25T00:13:21.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:21 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:13:21.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:21 smithi027 ceph-mon[29208]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T00:13:21.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:21 smithi027 ceph-mon[29208]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T00:13:21.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:21 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3681219608' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T00:13:21.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:21 smithi139 ceph-mon[32833]: pgmap v289: 97 pgs: 31 creating+peering, 15 active+clean, 51 unknown; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:13:21.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:21 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1286959783' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-25T00:13:21.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:21 smithi139 ceph-mon[32833]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T00:13:21.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:21 smithi139 ceph-mon[32833]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T00:13:21.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:21 smithi139 ceph-mon[32833]: Cluster is now healthy 2024-04-25T00:13:21.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:21 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:13:21.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:21 smithi139 ceph-mon[32833]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T00:13:21.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:21 smithi139 ceph-mon[32833]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T00:13:21.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:21 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3681219608' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T00:13:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.557+0000 7fe32a7fc640 1 -- 172.21.15.27:0/3681219608 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7fe32c0bcd90 con 0x7fe334102230 2024-04-25T00:13:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data_ec' created 2024-04-25T00:13:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.561+0000 7fe33b76a640 1 -- 172.21.15.27:0/3681219608 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe3200a2c10 msgr2=0x7fe3200a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.561+0000 7fe33b76a640 1 --2- 172.21.15.27:0/3681219608 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe3200a2c10 0x7fe3200a50d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe3300058c0 tx=0x7fe330005850 comp rx=0 tx=0).stop 2024-04-25T00:13:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.561+0000 7fe33b76a640 1 -- 172.21.15.27:0/3681219608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334102230 msgr2=0x7fe33419ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.561+0000 7fe33b76a640 1 --2- 172.21.15.27:0/3681219608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334102230 0x7fe33419ed50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0098d0 tx=0x7fe32c02f840 comp rx=0 tx=0).stop 2024-04-25T00:13:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.561+0000 7fe33b76a640 1 -- 172.21.15.27:0/3681219608 shutdown_connections 2024-04-25T00:13:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.561+0000 7fe33b76a640 1 --2- 172.21.15.27:0/3681219608 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe3200a2c10 0x7fe3200a50d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.561+0000 7fe33b76a640 1 --2- 172.21.15.27:0/3681219608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334102230 0x7fe33419ed50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.561+0000 7fe33b76a640 1 --2- 172.21.15.27:0/3681219608 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe334101870 0x7fe33419c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.561+0000 7fe33b76a640 1 --2- 172.21.15.27:0/3681219608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe334100680 0x7fe3340694a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.561+0000 7fe33b76a640 1 -- 172.21.15.27:0/3681219608 >> 172.21.15.27:0/3681219608 conn(0x7fe3340fbbf0 msgr2=0x7fe334104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.562+0000 7fe33b76a640 1 -- 172.21.15.27:0/3681219608 shutdown_connections 2024-04-25T00:13:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.562+0000 7fe33b76a640 1 -- 172.21.15.27:0/3681219608 wait complete. 2024-04-25T00:13:22.580 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-25T00:13:22.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.689+0000 7fbc124cd640 1 -- 172.21.15.27:0/261577185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc0c102990 msgr2=0x7fbc0c102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:22.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.689+0000 7fbc124cd640 1 --2- 172.21.15.27:0/261577185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc0c102990 0x7fbc0c102e10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbc00009900 tx=0x7fbc0002a410 comp rx=0 tx=0).stop 2024-04-25T00:13:22.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.689+0000 7fbc124cd640 1 -- 172.21.15.27:0/261577185 shutdown_connections 2024-04-25T00:13:22.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.689+0000 7fbc124cd640 1 --2- 172.21.15.27:0/261577185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc0c103350 0x7fbc0c109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:22.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.689+0000 7fbc124cd640 1 --2- 172.21.15.27:0/261577185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc0c102990 0x7fbc0c102e10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:22.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.689+0000 7fbc124cd640 1 --2- 172.21.15.27:0/261577185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc0c1017a0 0x7fbc0c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:22.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.689+0000 7fbc124cd640 1 -- 172.21.15.27:0/261577185 >> 172.21.15.27:0/261577185 conn(0x7fbc0c0fcd10 msgr2=0x7fbc0c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:22.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.689+0000 7fbc124cd640 1 -- 172.21.15.27:0/261577185 shutdown_connections 2024-04-25T00:13:22.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.689+0000 7fbc124cd640 1 -- 172.21.15.27:0/261577185 wait complete. 2024-04-25T00:13:22.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc124cd640 1 Processor -- start 2024-04-25T00:13:22.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc124cd640 1 -- start start 2024-04-25T00:13:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc124cd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc0c1017a0 0x7fbc0c0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc124cd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc0c102990 0x7fbc0c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc124cd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc0c103350 0x7fbc0c074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc124cd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c10d420 con 0x7fbc0c1017a0 2024-04-25T00:13:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc124cd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c10d2a0 con 0x7fbc0c103350 2024-04-25T00:13:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc124cd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c10d5a0 con 0x7fbc0c102990 2024-04-25T00:13:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc0bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc0c1017a0 0x7fbc0c0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc0bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc0c1017a0 0x7fbc0c0754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57372/0 (socket says 172.21.15.27:57372) 2024-04-25T00:13:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc0bfff640 1 -- 172.21.15.27:0/1373496556 learned_addr learned my addr 172.21.15.27:0/1373496556 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc0b7fe640 1 --2- 172.21.15.27:0/1373496556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc0c102990 0x7fbc0c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc10a6a640 1 --2- 172.21.15.27:0/1373496556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc0c103350 0x7fbc0c074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc0bfff640 1 -- 172.21.15.27:0/1373496556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc0c102990 msgr2=0x7fbc0c073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc0bfff640 1 --2- 172.21.15.27:0/1373496556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc0c102990 0x7fbc0c073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc0bfff640 1 -- 172.21.15.27:0/1373496556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc0c103350 msgr2=0x7fbc0c074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc0bfff640 1 --2- 172.21.15.27:0/1373496556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc0c103350 0x7fbc0c074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.690+0000 7fbc0bfff640 1 -- 172.21.15.27:0/1373496556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0c074900 con 0x7fbc0c1017a0 2024-04-25T00:13:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.691+0000 7fbc0bfff640 1 --2- 172.21.15.27:0/1373496556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc0c1017a0 0x7fbc0c0754f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc00e9c0 tx=0x7fbbfc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.691+0000 7fbc097fa640 1 -- 172.21.15.27:0/1373496556 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc00cd80 con 0x7fbc0c1017a0 2024-04-25T00:13:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.691+0000 7fbc097fa640 1 -- 172.21.15.27:0/1373496556 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbfc0035f0 con 0x7fbc0c1017a0 2024-04-25T00:13:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.691+0000 7fbc124cd640 1 -- 172.21.15.27:0/1373496556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc0c1a1bc0 con 0x7fbc0c1017a0 2024-04-25T00:13:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.691+0000 7fbc097fa640 1 -- 172.21.15.27:0/1373496556 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc010690 con 0x7fbc0c1017a0 2024-04-25T00:13:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.691+0000 7fbc124cd640 1 -- 172.21.15.27:0/1373496556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc0c1a20e0 con 0x7fbc0c1017a0 2024-04-25T00:13:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.692+0000 7fbc124cd640 1 -- 172.21.15.27:0/1373496556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc0c0689b0 con 0x7fbc0c1017a0 2024-04-25T00:13:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.693+0000 7fbc097fa640 1 -- 172.21.15.27:0/1373496556 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbbfc0037a0 con 0x7fbc0c1017a0 2024-04-25T00:13:22.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.694+0000 7fbc097fa640 1 --2- 172.21.15.27:0/1373496556 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fbbf40a2b90 0x7fbbf40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:22.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.694+0000 7fbc0b7fe640 1 --2- 172.21.15.27:0/1373496556 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fbbf40a2b90 0x7fbbf40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:22.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.694+0000 7fbc097fa640 1 -- 172.21.15.27:0/1373496556 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7fbbfc0cf700 con 0x7fbc0c1017a0 2024-04-25T00:13:22.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.694+0000 7fbc0b7fe640 1 --2- 172.21.15.27:0/1373496556 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fbbf40a2b90 0x7fbbf40a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbc000314e0 tx=0x7fbc00000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.697+0000 7fbc097fa640 1 -- 172.21.15.27:0/1373496556 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbbfc09c4d0 con 0x7fbc0c1017a0 2024-04-25T00:13:22.868 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:22 smithi153 ceph-mon[32896]: pgmap v292: 97 pgs: 31 creating+peering, 60 active+clean, 6 unknown; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-04-25T00:13:22.869 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:22 smithi153 ceph-mon[32896]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T00:13:22.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:22.882+0000 7fbc124cd640 1 -- 172.21.15.27:0/1373496556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fbc0c108ae0 con 0x7fbc0c1017a0 2024-04-25T00:13:22.924 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:22 smithi139 ceph-mon[32833]: pgmap v292: 97 pgs: 31 creating+peering, 60 active+clean, 6 unknown; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-04-25T00:13:22.924 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:22 smithi139 ceph-mon[32833]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T00:13:22.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:22 smithi027 ceph-mon[29208]: pgmap v292: 97 pgs: 31 creating+peering, 60 active+clean, 6 unknown; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-04-25T00:13:22.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:22 smithi027 ceph-mon[29208]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T00:13:23.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.559+0000 7fbc097fa640 1 -- 172.21.15.27:0/1373496556 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7fbbfc09bdd0 con 0x7fbc0c1017a0 2024-04-25T00:13:23.560 INFO:teuthology.orchestra.run.smithi027.stderr:set pool 4 allow_ec_overwrites to true 2024-04-25T00:13:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.565+0000 7fbc124cd640 1 -- 172.21.15.27:0/1373496556 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fbbf40a2b90 msgr2=0x7fbbf40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.565+0000 7fbc124cd640 1 --2- 172.21.15.27:0/1373496556 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fbbf40a2b90 0x7fbbf40a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbc000314e0 tx=0x7fbc00000bc0 comp rx=0 tx=0).stop 2024-04-25T00:13:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.565+0000 7fbc124cd640 1 -- 172.21.15.27:0/1373496556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc0c1017a0 msgr2=0x7fbc0c0754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.565+0000 7fbc124cd640 1 --2- 172.21.15.27:0/1373496556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc0c1017a0 0x7fbc0c0754f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc00e9c0 tx=0x7fbbfc00ee80 comp rx=0 tx=0).stop 2024-04-25T00:13:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.565+0000 7fbc124cd640 1 -- 172.21.15.27:0/1373496556 shutdown_connections 2024-04-25T00:13:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.565+0000 7fbc124cd640 1 --2- 172.21.15.27:0/1373496556 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fbbf40a2b90 0x7fbbf40a5050 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.565+0000 7fbc124cd640 1 --2- 172.21.15.27:0/1373496556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc0c103350 0x7fbc0c074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.565+0000 7fbc124cd640 1 --2- 172.21.15.27:0/1373496556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc0c102990 0x7fbc0c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.565+0000 7fbc124cd640 1 --2- 172.21.15.27:0/1373496556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc0c1017a0 0x7fbc0c0754f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.565+0000 7fbc124cd640 1 -- 172.21.15.27:0/1373496556 >> 172.21.15.27:0/1373496556 conn(0x7fbc0c0fcd10 msgr2=0x7fbc0c105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:23.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.565+0000 7fbc124cd640 1 -- 172.21.15.27:0/1373496556 shutdown_connections 2024-04-25T00:13:23.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.565+0000 7fbc124cd640 1 -- 172.21.15.27:0/1373496556 wait complete. 2024-04-25T00:13:23.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-25T00:13:23.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.698+0000 7f55e3a31640 1 -- 172.21.15.27:0/1166528058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55dc101890 msgr2=0x7f55dc101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.698+0000 7f55e3a31640 1 --2- 172.21.15.27:0/1166528058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55dc101890 0x7f55dc101d10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f55d8009900 tx=0x7f55d802a410 comp rx=0 tx=0).stop 2024-04-25T00:13:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.698+0000 7f55e3a31640 1 -- 172.21.15.27:0/1166528058 shutdown_connections 2024-04-25T00:13:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.698+0000 7f55e3a31640 1 --2- 172.21.15.27:0/1166528058 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55dc102250 0x7f55dc108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.698+0000 7f55e3a31640 1 --2- 172.21.15.27:0/1166528058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55dc101890 0x7f55dc101d10 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.698+0000 7f55e3a31640 1 --2- 172.21.15.27:0/1166528058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55dc1006a0 0x7f55dc100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.698+0000 7f55e3a31640 1 -- 172.21.15.27:0/1166528058 >> 172.21.15.27:0/1166528058 conn(0x7f55dc0fbc30 msgr2=0x7f55dc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.698+0000 7f55e3a31640 1 -- 172.21.15.27:0/1166528058 shutdown_connections 2024-04-25T00:13:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.698+0000 7f55e3a31640 1 -- 172.21.15.27:0/1166528058 wait complete. 2024-04-25T00:13:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.698+0000 7f55e3a31640 1 Processor -- start 2024-04-25T00:13:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.698+0000 7f55e3a31640 1 -- start start 2024-04-25T00:13:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.698+0000 7f55e3a31640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55dc1006a0 0x7f55dc06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.698+0000 7f55e3a31640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55dc101890 0x7f55dc19c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.698+0000 7f55e3a31640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55dc102250 0x7f55dc19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55e3a31640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f55dc10c300 con 0x7f55dc102250 2024-04-25T00:13:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55e3a31640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f55dc10c180 con 0x7f55dc1006a0 2024-04-25T00:13:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55e3a31640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55dc10c480 con 0x7f55dc101890 2024-04-25T00:13:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55e1fce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55dc102250 0x7f55dc19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55e1fce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55dc102250 0x7f55dc19edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57392/0 (socket says 172.21.15.27:57392) 2024-04-25T00:13:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55e17cd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55dc1006a0 0x7f55dc06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55e1fce640 1 -- 172.21.15.27:0/3871865493 learned_addr learned my addr 172.21.15.27:0/3871865493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55e0fcc640 1 --2- 172.21.15.27:0/3871865493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55dc101890 0x7f55dc19c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55e1fce640 1 -- 172.21.15.27:0/3871865493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55dc101890 msgr2=0x7f55dc19c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55e1fce640 1 --2- 172.21.15.27:0/3871865493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55dc101890 0x7f55dc19c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55e1fce640 1 -- 172.21.15.27:0/3871865493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55dc1006a0 msgr2=0x7f55dc06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55e1fce640 1 --2- 172.21.15.27:0/3871865493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55dc1006a0 0x7f55dc06a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55e1fce640 1 -- 172.21.15.27:0/3871865493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55dc19f670 con 0x7f55dc102250 2024-04-25T00:13:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55e1fce640 1 --2- 172.21.15.27:0/3871865493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55dc102250 0x7f55dc19edb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f55d400eb10 tx=0x7f55d400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55d27fc640 1 -- 172.21.15.27:0/3871865493 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d4004b20 con 0x7f55dc102250 2024-04-25T00:13:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55e3a31640 1 -- 172.21.15.27:0/3871865493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55dc19f960 con 0x7f55dc102250 2024-04-25T00:13:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55d27fc640 1 -- 172.21.15.27:0/3871865493 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55d4004cc0 con 0x7f55dc102250 2024-04-25T00:13:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55d27fc640 1 -- 172.21.15.27:0/3871865493 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d4010760 con 0x7f55dc102250 2024-04-25T00:13:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.699+0000 7f55e3a31640 1 -- 172.21.15.27:0/3871865493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55dc19fe30 con 0x7f55dc102250 2024-04-25T00:13:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.700+0000 7f55e3a31640 1 -- 172.21.15.27:0/3871865493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55dc107a10 con 0x7f55dc102250 2024-04-25T00:13:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.702+0000 7f55d27fc640 1 -- 172.21.15.27:0/3871865493 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f55d4005020 con 0x7f55dc102250 2024-04-25T00:13:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.702+0000 7f55d27fc640 1 --2- 172.21.15.27:0/3871865493 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f55c80a2b90 0x7f55c80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.703+0000 7f55e17cd640 1 --2- 172.21.15.27:0/3871865493 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f55c80a2b90 0x7f55c80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.703+0000 7f55d27fc640 1 -- 172.21.15.27:0/3871865493 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f55d40cf810 con 0x7f55dc102250 2024-04-25T00:13:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.703+0000 7f55e17cd640 1 --2- 172.21.15.27:0/3871865493 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f55c80a2b90 0x7f55c80a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f55dc1016f0 tx=0x7f55cc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.707+0000 7f55d27fc640 1 -- 172.21.15.27:0/3871865493 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f55d409c550 con 0x7f55dc102250 2024-04-25T00:13:23.875 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:13:23 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-b[62989]: 2024-04-25T00:13:23.734+0000 7f853bf52700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:13:23.876 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:23 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3681219608' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T00:13:23.876 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:23 smithi139 ceph-mon[32833]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T00:13:23.876 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:23 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1373496556' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-25T00:13:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:23.933+0000 7f55e3a31640 1 -- 172.21.15.27:0/3871865493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f55dc02d270 con 0x7f55dc102250 2024-04-25T00:13:23.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:23 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3681219608' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T00:13:23.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:23 smithi027 ceph-mon[29208]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T00:13:23.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:23 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1373496556' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-25T00:13:24.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:23 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3681219608' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T00:13:24.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:23 smithi153 ceph-mon[32896]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T00:13:24.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:23 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1373496556' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-25T00:13:24.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.581+0000 7f55d27fc640 1 -- 172.21.15.27:0/3871865493 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f55d409be50 con 0x7f55dc102250 2024-04-25T00:13:24.583 INFO:teuthology.orchestra.run.smithi027.stderr:added data pool 4 to fsmap 2024-04-25T00:13:24.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.584+0000 7f55e3a31640 1 -- 172.21.15.27:0/3871865493 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f55c80a2b90 msgr2=0x7f55c80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:24.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.584+0000 7f55e3a31640 1 --2- 172.21.15.27:0/3871865493 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f55c80a2b90 0x7f55c80a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f55dc1016f0 tx=0x7f55cc005f90 comp rx=0 tx=0).stop 2024-04-25T00:13:24.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.584+0000 7f55e3a31640 1 -- 172.21.15.27:0/3871865493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55dc102250 msgr2=0x7f55dc19edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:24.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.584+0000 7f55e3a31640 1 --2- 172.21.15.27:0/3871865493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55dc102250 0x7f55dc19edb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f55d400eb10 tx=0x7f55d400c2d0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.584+0000 7f55e3a31640 1 -- 172.21.15.27:0/3871865493 shutdown_connections 2024-04-25T00:13:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.584+0000 7f55e3a31640 1 --2- 172.21.15.27:0/3871865493 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f55c80a2b90 0x7f55c80a5050 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.584+0000 7f55e3a31640 1 --2- 172.21.15.27:0/3871865493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55dc102250 0x7f55dc19edb0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.584+0000 7f55e3a31640 1 --2- 172.21.15.27:0/3871865493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55dc101890 0x7f55dc19c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.584+0000 7f55e3a31640 1 --2- 172.21.15.27:0/3871865493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55dc1006a0 0x7f55dc06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.584+0000 7f55e3a31640 1 -- 172.21.15.27:0/3871865493 >> 172.21.15.27:0/3871865493 conn(0x7f55dc0fbc30 msgr2=0x7f55dc104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.584+0000 7f55e3a31640 1 -- 172.21.15.27:0/3871865493 shutdown_connections 2024-04-25T00:13:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.584+0000 7f55e3a31640 1 -- 172.21.15.27:0/3871865493 wait complete. 2024-04-25T00:13:24.599 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:24.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.706+0000 7f76aefb7640 1 -- 172.21.15.27:0/280781010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76a8102980 msgr2=0x7f76a8102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:24.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.706+0000 7f76aefb7640 1 --2- 172.21.15.27:0/280781010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76a8102980 0x7f76a8102e00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7694009900 tx=0x7f769402a410 comp rx=0 tx=0).stop 2024-04-25T00:13:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.706+0000 7f76aefb7640 1 -- 172.21.15.27:0/280781010 shutdown_connections 2024-04-25T00:13:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.706+0000 7f76aefb7640 1 --2- 172.21.15.27:0/280781010 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76a8103340 0x7f76a8109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.706+0000 7f76aefb7640 1 --2- 172.21.15.27:0/280781010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76a8102980 0x7f76a8102e00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.706+0000 7f76aefb7640 1 --2- 172.21.15.27:0/280781010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76a8101790 0x7f76a8101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.706+0000 7f76aefb7640 1 -- 172.21.15.27:0/280781010 >> 172.21.15.27:0/280781010 conn(0x7f76a80fcd40 msgr2=0x7f76a80ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.706+0000 7f76aefb7640 1 -- 172.21.15.27:0/280781010 shutdown_connections 2024-04-25T00:13:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.706+0000 7f76aefb7640 1 -- 172.21.15.27:0/280781010 wait complete. 2024-04-25T00:13:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76aefb7640 1 Processor -- start 2024-04-25T00:13:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76aefb7640 1 -- start start 2024-04-25T00:13:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76aefb7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76a8101790 0x7f76a8197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76aefb7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76a8102980 0x7f76a8197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76aefb7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76a8103340 0x7f76a819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76aefb7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f76a810d410 con 0x7f76a8101790 2024-04-25T00:13:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76aefb7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f76a810d290 con 0x7f76a8102980 2024-04-25T00:13:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76aefb7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f76a810d590 con 0x7f76a8103340 2024-04-25T00:13:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76acd53640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76a8101790 0x7f76a8197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76acd53640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76a8101790 0x7f76a8197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57404/0 (socket says 172.21.15.27:57404) 2024-04-25T00:13:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76acd53640 1 -- 172.21.15.27:0/463079743 learned_addr learned my addr 172.21.15.27:0/463079743 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76ad554640 1 --2- 172.21.15.27:0/463079743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76a8103340 0x7f76a819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76a7fff640 1 --2- 172.21.15.27:0/463079743 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76a8102980 0x7f76a8197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76acd53640 1 -- 172.21.15.27:0/463079743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76a8103340 msgr2=0x7f76a819f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76acd53640 1 --2- 172.21.15.27:0/463079743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76a8103340 0x7f76a819f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76acd53640 1 -- 172.21.15.27:0/463079743 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76a8102980 msgr2=0x7f76a8197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76acd53640 1 --2- 172.21.15.27:0/463079743 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76a8102980 0x7f76a8197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76acd53640 1 -- 172.21.15.27:0/463079743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a819f670 con 0x7f76a8101790 2024-04-25T00:13:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76a7fff640 1 --2- 172.21.15.27:0/463079743 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76a8102980 0x7f76a8197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.707+0000 7f76acd53640 1 --2- 172.21.15.27:0/463079743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76a8101790 0x7f76a8197a50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f769c00d950 tx=0x7f769c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.708+0000 7f76a5ffb640 1 -- 172.21.15.27:0/463079743 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769c017070 con 0x7f76a8101790 2024-04-25T00:13:24.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.708+0000 7f76a5ffb640 1 -- 172.21.15.27:0/463079743 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f769c009c30 con 0x7f76a8101790 2024-04-25T00:13:24.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.708+0000 7f76aefb7640 1 -- 172.21.15.27:0/463079743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76a819f960 con 0x7f76a8101790 2024-04-25T00:13:24.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.708+0000 7f76a5ffb640 1 -- 172.21.15.27:0/463079743 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769c00f600 con 0x7f76a8101790 2024-04-25T00:13:24.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.708+0000 7f76aefb7640 1 -- 172.21.15.27:0/463079743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76a819feb0 con 0x7f76a8101790 2024-04-25T00:13:24.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.709+0000 7f76aefb7640 1 -- 172.21.15.27:0/463079743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76a8108ab0 con 0x7f76a8101790 2024-04-25T00:13:24.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.710+0000 7f76a5ffb640 1 -- 172.21.15.27:0/463079743 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f769c01a070 con 0x7f76a8101790 2024-04-25T00:13:24.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.711+0000 7f76a5ffb640 1 --2- 172.21.15.27:0/463079743 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f76980a2c10 0x7f76980a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:24.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.711+0000 7f76a7fff640 1 --2- 172.21.15.27:0/463079743 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f76980a2c10 0x7f76980a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:24.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.711+0000 7f76a5ffb640 1 -- 172.21.15.27:0/463079743 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f769c0ced50 con 0x7f76a8101790 2024-04-25T00:13:24.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.711+0000 7f76a7fff640 1 --2- 172.21.15.27:0/463079743 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f76980a2c10 0x7f76980a50d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7694000c00 tx=0x7f7694031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:24.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.716+0000 7f76a5ffb640 1 -- 172.21.15.27:0/463079743 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f769c0d2030 con 0x7f76a8101790 2024-04-25T00:13:24.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.938+0000 7f76aefb7640 1 -- 172.21.15.27:0/463079743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76a802d170 con 0x7f76a8101790 2024-04-25T00:13:24.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.939+0000 7f76a5ffb640 1 -- 172.21.15.27:0/463079743 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11809 (secure 0 0 0) 0x7f769c009de0 con 0x7f76a8101790 2024-04-25T00:13:24.940 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:24.940 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24514,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:13:23.935300+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24521,"qdb_cluster":[24521]},"id":1}]} 2024-04-25T00:13:24.940 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-04-25T00:13:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.941+0000 7f76aefb7640 1 -- 172.21.15.27:0/463079743 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f76980a2c10 msgr2=0x7f76980a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.941+0000 7f76aefb7640 1 --2- 172.21.15.27:0/463079743 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f76980a2c10 0x7f76980a50d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7694000c00 tx=0x7f7694031400 comp rx=0 tx=0).stop 2024-04-25T00:13:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.941+0000 7f76aefb7640 1 -- 172.21.15.27:0/463079743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76a8101790 msgr2=0x7f76a8197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.941+0000 7f76aefb7640 1 --2- 172.21.15.27:0/463079743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76a8101790 0x7f76a8197a50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f769c00d950 tx=0x7f769c00de10 comp rx=0 tx=0).stop 2024-04-25T00:13:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.941+0000 7f76aefb7640 1 -- 172.21.15.27:0/463079743 shutdown_connections 2024-04-25T00:13:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.941+0000 7f76aefb7640 1 --2- 172.21.15.27:0/463079743 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f76980a2c10 0x7f76980a50d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.941+0000 7f76aefb7640 1 --2- 172.21.15.27:0/463079743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76a8103340 0x7f76a819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.941+0000 7f76aefb7640 1 --2- 172.21.15.27:0/463079743 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76a8102980 0x7f76a8197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.941+0000 7f76aefb7640 1 --2- 172.21.15.27:0/463079743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76a8101790 0x7f76a8197a50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.941+0000 7f76aefb7640 1 -- 172.21.15.27:0/463079743 >> 172.21.15.27:0/463079743 conn(0x7f76a80fcd40 msgr2=0x7f76a8105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.941+0000 7f76aefb7640 1 -- 172.21.15.27:0/463079743 shutdown_connections 2024-04-25T00:13:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:24.941+0000 7f76aefb7640 1 -- 172.21.15.27:0/463079743 wait complete. 2024-04-25T00:13:24.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:24 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1373496556' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-25T00:13:24.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:24 smithi027 ceph-mon[29208]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T00:13:24.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:24 smithi027 ceph-mon[29208]: pgmap v295: 129 pgs: 29 creating+peering, 72 active+clean, 28 unknown; 579 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2024-04-25T00:13:24.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:24 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3871865493' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-25T00:13:24.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:13:25.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:24 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1373496556' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-25T00:13:25.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:24 smithi139 ceph-mon[32833]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T00:13:25.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:24 smithi139 ceph-mon[32833]: pgmap v295: 129 pgs: 29 creating+peering, 72 active+clean, 28 unknown; 579 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2024-04-25T00:13:25.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:24 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3871865493' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-25T00:13:25.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:24 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1373496556' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-25T00:13:25.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:24 smithi153 ceph-mon[32896]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T00:13:25.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:24 smithi153 ceph-mon[32896]: pgmap v295: 129 pgs: 29 creating+peering, 72 active+clean, 28 unknown; 579 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2024-04-25T00:13:25.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:24 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3871865493' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-25T00:13:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.053+0000 7f56c5864640 1 -- 172.21.15.27:0/3743666221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c0102980 msgr2=0x7f56c0102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.053+0000 7f56c5864640 1 --2- 172.21.15.27:0/3743666221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c0102980 0x7f56c0102e00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009900 tx=0x7f56b002a410 comp rx=0 tx=0).stop 2024-04-25T00:13:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.054+0000 7f56c5864640 1 -- 172.21.15.27:0/3743666221 shutdown_connections 2024-04-25T00:13:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.054+0000 7f56c5864640 1 --2- 172.21.15.27:0/3743666221 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56c0103340 0x7f56c0109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.054+0000 7f56c5864640 1 --2- 172.21.15.27:0/3743666221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c0102980 0x7f56c0102e00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.054+0000 7f56c5864640 1 --2- 172.21.15.27:0/3743666221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c0101790 0x7f56c0101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.054+0000 7f56c5864640 1 -- 172.21.15.27:0/3743666221 >> 172.21.15.27:0/3743666221 conn(0x7f56c00fcd20 msgr2=0x7f56c00ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.054+0000 7f56c5864640 1 -- 172.21.15.27:0/3743666221 shutdown_connections 2024-04-25T00:13:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.054+0000 7f56c5864640 1 -- 172.21.15.27:0/3743666221 wait complete. 2024-04-25T00:13:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.054+0000 7f56c5864640 1 Processor -- start 2024-04-25T00:13:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.054+0000 7f56c5864640 1 -- start start 2024-04-25T00:13:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.054+0000 7f56c5864640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56c0101790 0x7f56c0197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.054+0000 7f56c5864640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c0102980 0x7f56c0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.054+0000 7f56c5864640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c0103340 0x7f56c019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.054+0000 7f56c5864640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f56c010d3f0 con 0x7f56c0103340 2024-04-25T00:13:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.054+0000 7f56c5864640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f56c010d270 con 0x7f56c0101790 2024-04-25T00:13:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.054+0000 7f56c5864640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f56c010d570 con 0x7f56c0102980 2024-04-25T00:13:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.055+0000 7f56bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c0103340 0x7f56c019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.055+0000 7f56bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c0103340 0x7f56c019f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57420/0 (socket says 172.21.15.27:57420) 2024-04-25T00:13:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.055+0000 7f56bf7fe640 1 -- 172.21.15.27:0/1761216224 learned_addr learned my addr 172.21.15.27:0/1761216224 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.055+0000 7f56beffd640 1 --2- 172.21.15.27:0/1761216224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56c0101790 0x7f56c0197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.055+0000 7f56be7fc640 1 --2- 172.21.15.27:0/1761216224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c0102980 0x7f56c0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.055+0000 7f56bf7fe640 1 -- 172.21.15.27:0/1761216224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c0102980 msgr2=0x7f56c0197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.055+0000 7f56bf7fe640 1 --2- 172.21.15.27:0/1761216224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c0102980 0x7f56c0197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.055+0000 7f56bf7fe640 1 -- 172.21.15.27:0/1761216224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56c0101790 msgr2=0x7f56c0197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.055+0000 7f56bf7fe640 1 --2- 172.21.15.27:0/1761216224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56c0101790 0x7f56c0197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.055+0000 7f56bf7fe640 1 -- 172.21.15.27:0/1761216224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c019f6c0 con 0x7f56c0103340 2024-04-25T00:13:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.055+0000 7f56be7fc640 1 --2- 172.21.15.27:0/1761216224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c0102980 0x7f56c0197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:13:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.055+0000 7f56bf7fe640 1 --2- 172.21.15.27:0/1761216224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c0103340 0x7f56c019f050 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009fd0 tx=0x7f56b800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.055+0000 7f56c4862640 1 -- 172.21.15.27:0/1761216224 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8017070 con 0x7f56c0103340 2024-04-25T00:13:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.055+0000 7f56c4862640 1 -- 172.21.15.27:0/1761216224 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56b8003e50 con 0x7f56c0103340 2024-04-25T00:13:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.055+0000 7f56c4862640 1 -- 172.21.15.27:0/1761216224 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b80106c0 con 0x7f56c0103340 2024-04-25T00:13:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.055+0000 7f56c5864640 1 -- 172.21.15.27:0/1761216224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c019f9b0 con 0x7f56c0103340 2024-04-25T00:13:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.055+0000 7f56c5864640 1 -- 172.21.15.27:0/1761216224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c019fe80 con 0x7f56c0103340 2024-04-25T00:13:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.056+0000 7f56c5864640 1 -- 172.21.15.27:0/1761216224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56c0108ab0 con 0x7f56c0103340 2024-04-25T00:13:25.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.057+0000 7f56c4862640 1 -- 172.21.15.27:0/1761216224 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f56b8003970 con 0x7f56c0103340 2024-04-25T00:13:25.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.058+0000 7f56c4862640 1 --2- 172.21.15.27:0/1761216224 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f56a80a2b90 0x7f56a80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:25.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.058+0000 7f56beffd640 1 --2- 172.21.15.27:0/1761216224 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f56a80a2b90 0x7f56a80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:25.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.058+0000 7f56c4862640 1 -- 172.21.15.27:0/1761216224 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f56b80ceff0 con 0x7f56c0103340 2024-04-25T00:13:25.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.058+0000 7f56beffd640 1 --2- 172.21.15.27:0/1761216224 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f56a80a2b90 0x7f56a80a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f56b4005fd0 tx=0x7f56b400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:25.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.062+0000 7f56c4862640 1 -- 172.21.15.27:0/1761216224 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f56b809bd10 con 0x7f56c0103340 2024-04-25T00:13:25.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.247+0000 7f56c5864640 1 -- 172.21.15.27:0/1761216224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f56c0062340 con 0x7f56c0103340 2024-04-25T00:13:25.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.249+0000 7f56c4862640 1 -- 172.21.15.27:0/1761216224 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24117 (secure 0 0 0) 0x7f56b809b610 con 0x7f56c0103340 2024-04-25T00:13:25.250 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:25.250 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:13:23.935081+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:13:25.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.250+0000 7f56c5864640 1 -- 172.21.15.27:0/1761216224 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f56a80a2b90 msgr2=0x7f56a80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:25.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.251+0000 7f56c5864640 1 --2- 172.21.15.27:0/1761216224 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f56a80a2b90 0x7f56a80a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f56b4005fd0 tx=0x7f56b400b410 comp rx=0 tx=0).stop 2024-04-25T00:13:25.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.251+0000 7f56c5864640 1 -- 172.21.15.27:0/1761216224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c0103340 msgr2=0x7f56c019f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:25.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.251+0000 7f56c5864640 1 --2- 172.21.15.27:0/1761216224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c0103340 0x7f56c019f050 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009fd0 tx=0x7f56b800b260 comp rx=0 tx=0).stop 2024-04-25T00:13:25.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.251+0000 7f56c5864640 1 -- 172.21.15.27:0/1761216224 shutdown_connections 2024-04-25T00:13:25.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.251+0000 7f56c5864640 1 --2- 172.21.15.27:0/1761216224 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f56a80a2b90 0x7f56a80a5050 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.251+0000 7f56c5864640 1 --2- 172.21.15.27:0/1761216224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c0103340 0x7f56c019f050 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.251+0000 7f56c5864640 1 --2- 172.21.15.27:0/1761216224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56c0102980 0x7f56c0197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.251+0000 7f56c5864640 1 --2- 172.21.15.27:0/1761216224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56c0101790 0x7f56c0197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.251+0000 7f56c5864640 1 -- 172.21.15.27:0/1761216224 >> 172.21.15.27:0/1761216224 conn(0x7f56c00fcd20 msgr2=0x7f56c0105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:25.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.251+0000 7f56c5864640 1 -- 172.21.15.27:0/1761216224 shutdown_connections 2024-04-25T00:13:25.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.251+0000 7f56c5864640 1 -- 172.21.15.27:0/1761216224 wait complete. 2024-04-25T00:13:25.264 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:13:25.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.365+0000 7f2356312640 1 -- 172.21.15.27:0/3736502930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2350100680 msgr2=0x7f2350100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:25.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.365+0000 7f2356312640 1 --2- 172.21.15.27:0/3736502930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2350100680 0x7f2350100aa0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2340009900 tx=0x7f234002a410 comp rx=0 tx=0).stop 2024-04-25T00:13:25.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.365+0000 7f2356312640 1 -- 172.21.15.27:0/3736502930 shutdown_connections 2024-04-25T00:13:25.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.365+0000 7f2356312640 1 --2- 172.21.15.27:0/3736502930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2350102230 0x7f2350108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.365+0000 7f2356312640 1 --2- 172.21.15.27:0/3736502930 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2350101870 0x7f2350101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.365+0000 7f2356312640 1 --2- 172.21.15.27:0/3736502930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2350100680 0x7f2350100aa0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.365+0000 7f2356312640 1 -- 172.21.15.27:0/3736502930 >> 172.21.15.27:0/3736502930 conn(0x7f23500fbbf0 msgr2=0x7f23500fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:25.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.365+0000 7f2356312640 1 -- 172.21.15.27:0/3736502930 shutdown_connections 2024-04-25T00:13:25.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.365+0000 7f2356312640 1 -- 172.21.15.27:0/3736502930 wait complete. 2024-04-25T00:13:25.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.366+0000 7f2356312640 1 Processor -- start 2024-04-25T00:13:25.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.366+0000 7f2356312640 1 -- start start 2024-04-25T00:13:25.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.366+0000 7f2356312640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2350100680 0x7f2350197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:25.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.366+0000 7f2356312640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2350101870 0x7f2350197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:25.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.366+0000 7f2356312640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2350102230 0x7f235019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:25.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.366+0000 7f2356312640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f235010c320 con 0x7f2350100680 2024-04-25T00:13:25.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.366+0000 7f2356312640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f235010c1a0 con 0x7f2350102230 2024-04-25T00:13:25.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.366+0000 7f2356312640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f235010c4a0 con 0x7f2350101870 2024-04-25T00:13:25.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.366+0000 7f234ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2350100680 0x7f2350197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:25.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.366+0000 7f234ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2350100680 0x7f2350197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57440/0 (socket says 172.21.15.27:57440) 2024-04-25T00:13:25.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.366+0000 7f234ffff640 1 -- 172.21.15.27:0/82720479 learned_addr learned my addr 172.21.15.27:0/82720479 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:25.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.367+0000 7f234f7fe640 1 --2- 172.21.15.27:0/82720479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2350101870 0x7f2350197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:25.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.367+0000 7f23548af640 1 --2- 172.21.15.27:0/82720479 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2350102230 0x7f235019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:25.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.367+0000 7f234ffff640 1 -- 172.21.15.27:0/82720479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2350101870 msgr2=0x7f2350197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:25.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.367+0000 7f234ffff640 1 --2- 172.21.15.27:0/82720479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2350101870 0x7f2350197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.367+0000 7f234ffff640 1 -- 172.21.15.27:0/82720479 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2350102230 msgr2=0x7f235019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:25.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.367+0000 7f234ffff640 1 --2- 172.21.15.27:0/82720479 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2350102230 0x7f235019f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.367+0000 7f234ffff640 1 -- 172.21.15.27:0/82720479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f235019f6d0 con 0x7f2350100680 2024-04-25T00:13:25.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.367+0000 7f234f7fe640 1 --2- 172.21.15.27:0/82720479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2350101870 0x7f2350197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:25.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.367+0000 7f234ffff640 1 --2- 172.21.15.27:0/82720479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2350100680 0x7f2350197ab0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2340031440 tx=0x7f2340032e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:25.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.367+0000 7f234d7fa640 1 -- 172.21.15.27:0/82720479 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234003f030 con 0x7f2350100680 2024-04-25T00:13:25.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.367+0000 7f234d7fa640 1 -- 172.21.15.27:0/82720479 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2340003ce0 con 0x7f2350100680 2024-04-25T00:13:25.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.367+0000 7f2356312640 1 -- 172.21.15.27:0/82720479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f235019f960 con 0x7f2350100680 2024-04-25T00:13:25.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.367+0000 7f234d7fa640 1 -- 172.21.15.27:0/82720479 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2340035070 con 0x7f2350100680 2024-04-25T00:13:25.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.367+0000 7f2356312640 1 -- 172.21.15.27:0/82720479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f235019fea0 con 0x7f2350100680 2024-04-25T00:13:25.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.368+0000 7f2356312640 1 -- 172.21.15.27:0/82720479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23501a0070 con 0x7f2350100680 2024-04-25T00:13:25.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.369+0000 7f234d7fa640 1 -- 172.21.15.27:0/82720479 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2340042030 con 0x7f2350100680 2024-04-25T00:13:25.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.370+0000 7f234d7fa640 1 --2- 172.21.15.27:0/82720479 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f23380a2c10 0x7f23380a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:25.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.370+0000 7f234f7fe640 1 --2- 172.21.15.27:0/82720479 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f23380a2c10 0x7f23380a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:25.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.370+0000 7f234d7fa640 1 -- 172.21.15.27:0/82720479 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f23400efd30 con 0x7f2350100680 2024-04-25T00:13:25.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.370+0000 7f234f7fe640 1 --2- 172.21.15.27:0/82720479 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f23380a2c10 0x7f23380a50d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2350198fc0 tx=0x7f2344005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:25.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.373+0000 7f234d7fa640 1 -- 172.21.15.27:0/82720479 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f23400bca50 con 0x7f2350100680 2024-04-25T00:13:25.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.561+0000 7f2356312640 1 -- 172.21.15.27:0/82720479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f23501a0360 con 0x7f2350100680 2024-04-25T00:13:25.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.562+0000 7f234d7fa640 1 -- 172.21.15.27:0/82720479 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24117 (secure 0 0 0) 0x7f23400bc350 con 0x7f2350100680 2024-04-25T00:13:25.564 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:25.564 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:13:23.935081+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:13:25.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.564+0000 7f2356312640 1 -- 172.21.15.27:0/82720479 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f23380a2c10 msgr2=0x7f23380a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:25.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.564+0000 7f2356312640 1 --2- 172.21.15.27:0/82720479 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f23380a2c10 0x7f23380a50d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2350198fc0 tx=0x7f2344005e50 comp rx=0 tx=0).stop 2024-04-25T00:13:25.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.564+0000 7f2356312640 1 -- 172.21.15.27:0/82720479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2350100680 msgr2=0x7f2350197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:25.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.564+0000 7f2356312640 1 --2- 172.21.15.27:0/82720479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2350100680 0x7f2350197ab0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2340031440 tx=0x7f2340032e20 comp rx=0 tx=0).stop 2024-04-25T00:13:25.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.565+0000 7f2356312640 1 -- 172.21.15.27:0/82720479 shutdown_connections 2024-04-25T00:13:25.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.565+0000 7f2356312640 1 --2- 172.21.15.27:0/82720479 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f23380a2c10 0x7f23380a50d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.565+0000 7f2356312640 1 --2- 172.21.15.27:0/82720479 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2350102230 0x7f235019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.565+0000 7f2356312640 1 --2- 172.21.15.27:0/82720479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2350101870 0x7f2350197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.565+0000 7f2356312640 1 --2- 172.21.15.27:0/82720479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2350100680 0x7f2350197ab0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.565+0000 7f2356312640 1 -- 172.21.15.27:0/82720479 >> 172.21.15.27:0/82720479 conn(0x7f23500fbbf0 msgr2=0x7f23500fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:25.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.565+0000 7f2356312640 1 -- 172.21.15.27:0/82720479 shutdown_connections 2024-04-25T00:13:25.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.565+0000 7f2356312640 1 -- 172.21.15.27:0/82720479 wait complete. 2024-04-25T00:13:25.571 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:25 smithi027 ceph-mon[29208]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T00:13:25.572 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:25 smithi027 ceph-mon[29208]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T00:13:25.572 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:25 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3871865493' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-25T00:13:25.572 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:25 smithi027 ceph-mon[29208]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T00:13:25.572 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:25 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/463079743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:25.572 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:25 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1761216224' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:25.572 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:25 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/82720479' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:25.593 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:25.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.694+0000 7f1c349f4640 1 -- 172.21.15.27:0/2182428874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c30103360 msgr2=0x7f1c30109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:25.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.694+0000 7f1c349f4640 1 --2- 172.21.15.27:0/2182428874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c30103360 0x7f1c30109bf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009ca0 tx=0x7f1c2802a8c0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.694+0000 7f1c349f4640 1 -- 172.21.15.27:0/2182428874 shutdown_connections 2024-04-25T00:13:25.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.694+0000 7f1c349f4640 1 --2- 172.21.15.27:0/2182428874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c30103360 0x7f1c30109bf0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.694+0000 7f1c349f4640 1 --2- 172.21.15.27:0/2182428874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c301029a0 0x7f1c30102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.694+0000 7f1c349f4640 1 --2- 172.21.15.27:0/2182428874 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c301017b0 0x7f1c30101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.694+0000 7f1c349f4640 1 -- 172.21.15.27:0/2182428874 >> 172.21.15.27:0/2182428874 conn(0x7f1c300fcd40 msgr2=0x7f1c300ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:25.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.694+0000 7f1c349f4640 1 -- 172.21.15.27:0/2182428874 shutdown_connections 2024-04-25T00:13:25.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.694+0000 7f1c349f4640 1 -- 172.21.15.27:0/2182428874 wait complete. 2024-04-25T00:13:25.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.695+0000 7f1c349f4640 1 Processor -- start 2024-04-25T00:13:25.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.695+0000 7f1c349f4640 1 -- start start 2024-04-25T00:13:25.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.695+0000 7f1c349f4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c301017b0 0x7f1c301979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:25.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.695+0000 7f1c349f4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c301029a0 0x7f1c30197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:25.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.695+0000 7f1c349f4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c30103360 0x7f1c3019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:25.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.695+0000 7f1c349f4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3010d430 con 0x7f1c30103360 2024-04-25T00:13:25.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.695+0000 7f1c349f4640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3010d2b0 con 0x7f1c301029a0 2024-04-25T00:13:25.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.695+0000 7f1c349f4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3010d5b0 con 0x7f1c301017b0 2024-04-25T00:13:25.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.695+0000 7f1c2ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c30103360 0x7f1c3019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:25.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.695+0000 7f1c2ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c30103360 0x7f1c3019ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57456/0 (socket says 172.21.15.27:57456) 2024-04-25T00:13:25.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.695+0000 7f1c2ed9d640 1 -- 172.21.15.27:0/4140432367 learned_addr learned my addr 172.21.15.27:0/4140432367 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:25.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.695+0000 7f1c2e59c640 1 --2- 172.21.15.27:0/4140432367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c301017b0 0x7f1c301979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:25.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.696+0000 7f1c27fff640 1 --2- 172.21.15.27:0/4140432367 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c301029a0 0x7f1c30197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:25.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.696+0000 7f1c2ed9d640 1 -- 172.21.15.27:0/4140432367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c301017b0 msgr2=0x7f1c301979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:25.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.696+0000 7f1c2ed9d640 1 --2- 172.21.15.27:0/4140432367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c301017b0 0x7f1c301979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.696+0000 7f1c2ed9d640 1 -- 172.21.15.27:0/4140432367 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c301029a0 msgr2=0x7f1c30197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:25.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.696+0000 7f1c2ed9d640 1 --2- 172.21.15.27:0/4140432367 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c301029a0 0x7f1c30197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.696+0000 7f1c2ed9d640 1 -- 172.21.15.27:0/4140432367 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c3019f5d0 con 0x7f1c30103360 2024-04-25T00:13:25.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.696+0000 7f1c2ed9d640 1 --2- 172.21.15.27:0/4140432367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c30103360 0x7f1c3019ef60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1c28031ae0 tx=0x7f1c28031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:25.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.696+0000 7f1c277fe640 1 -- 172.21.15.27:0/4140432367 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c28003c30 con 0x7f1c30103360 2024-04-25T00:13:25.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.696+0000 7f1c349f4640 1 -- 172.21.15.27:0/4140432367 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c3019f860 con 0x7f1c30103360 2024-04-25T00:13:25.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.696+0000 7f1c277fe640 1 -- 172.21.15.27:0/4140432367 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c28003dd0 con 0x7f1c30103360 2024-04-25T00:13:25.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.696+0000 7f1c277fe640 1 -- 172.21.15.27:0/4140432367 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2802fcb0 con 0x7f1c30103360 2024-04-25T00:13:25.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.696+0000 7f1c349f4640 1 -- 172.21.15.27:0/4140432367 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c3019fdb0 con 0x7f1c30103360 2024-04-25T00:13:25.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.697+0000 7f1c349f4640 1 -- 172.21.15.27:0/4140432367 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c30108a00 con 0x7f1c30103360 2024-04-25T00:13:25.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.698+0000 7f1c277fe640 1 -- 172.21.15.27:0/4140432367 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1c28004050 con 0x7f1c30103360 2024-04-25T00:13:25.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.699+0000 7f1c277fe640 1 --2- 172.21.15.27:0/4140432367 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1c1c0a2b90 0x7f1c1c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:25.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.699+0000 7f1c2e59c640 1 --2- 172.21.15.27:0/4140432367 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1c1c0a2b90 0x7f1c1c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:25.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.699+0000 7f1c277fe640 1 -- 172.21.15.27:0/4140432367 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1c280f07f0 con 0x7f1c30103360 2024-04-25T00:13:25.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.699+0000 7f1c2e59c640 1 --2- 172.21.15.27:0/4140432367 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1c1c0a2b90 0x7f1c1c0a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1c30102800 tx=0x7f1c20005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:25.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.702+0000 7f1c277fe640 1 -- 172.21.15.27:0/4140432367 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1c280bd510 con 0x7f1c30103360 2024-04-25T00:13:25.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.926+0000 7f1c349f4640 1 -- 172.21.15.27:0/4140432367 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c3002d330 con 0x7f1c30103360 2024-04-25T00:13:25.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.927+0000 7f1c277fe640 1 -- 172.21.15.27:0/4140432367 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11809 (secure 0 0 0) 0x7f1c3002d330 con 0x7f1c30103360 2024-04-25T00:13:25.928 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:25.928 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24514,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:13:23.935300+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24521,"qdb_cluster":[24521]},"id":1}]} 2024-04-25T00:13:25.928 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-04-25T00:13:25.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.928+0000 7f1c349f4640 1 -- 172.21.15.27:0/4140432367 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1c1c0a2b90 msgr2=0x7f1c1c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:25.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.928+0000 7f1c349f4640 1 --2- 172.21.15.27:0/4140432367 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1c1c0a2b90 0x7f1c1c0a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1c30102800 tx=0x7f1c20005f90 comp rx=0 tx=0).stop 2024-04-25T00:13:25.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.928+0000 7f1c349f4640 1 -- 172.21.15.27:0/4140432367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c30103360 msgr2=0x7f1c3019ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:25.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.929+0000 7f1c349f4640 1 --2- 172.21.15.27:0/4140432367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c30103360 0x7f1c3019ef60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1c28031ae0 tx=0x7f1c28031ee0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.929+0000 7f1c349f4640 1 -- 172.21.15.27:0/4140432367 shutdown_connections 2024-04-25T00:13:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.929+0000 7f1c349f4640 1 --2- 172.21.15.27:0/4140432367 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1c1c0a2b90 0x7f1c1c0a5050 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.929+0000 7f1c349f4640 1 --2- 172.21.15.27:0/4140432367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c30103360 0x7f1c3019ef60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.929+0000 7f1c349f4640 1 --2- 172.21.15.27:0/4140432367 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c301029a0 0x7f1c30197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.929+0000 7f1c349f4640 1 --2- 172.21.15.27:0/4140432367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c301017b0 0x7f1c301979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.929+0000 7f1c349f4640 1 -- 172.21.15.27:0/4140432367 >> 172.21.15.27:0/4140432367 conn(0x7f1c300fcd40 msgr2=0x7f1c30105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.929+0000 7f1c349f4640 1 -- 172.21.15.27:0/4140432367 shutdown_connections 2024-04-25T00:13:25.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:25.929+0000 7f1c349f4640 1 -- 172.21.15.27:0/4140432367 wait complete. 2024-04-25T00:13:25.941 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:13:26.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:25 smithi139 ceph-mon[32833]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T00:13:26.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:25 smithi139 ceph-mon[32833]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T00:13:26.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:25 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3871865493' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-25T00:13:26.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:25 smithi139 ceph-mon[32833]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T00:13:26.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:25 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/463079743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:26.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:25 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1761216224' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:26.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:25 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/82720479' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:26.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:25 smithi153 ceph-mon[32896]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T00:13:26.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:25 smithi153 ceph-mon[32896]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T00:13:26.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:25 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3871865493' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-25T00:13:26.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:25 smithi153 ceph-mon[32896]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T00:13:26.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:25 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/463079743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:26.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:25 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1761216224' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:26.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:25 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/82720479' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:26.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.041+0000 7f8d89b70640 1 -- 172.21.15.27:0/248840646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d84102230 msgr2=0x7f8d84108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.041+0000 7f8d89b70640 1 --2- 172.21.15.27:0/248840646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d84102230 0x7f8d84108ac0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c009ca0 tx=0x7f8d7c02a8c0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.041+0000 7f8d89b70640 1 -- 172.21.15.27:0/248840646 shutdown_connections 2024-04-25T00:13:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.041+0000 7f8d89b70640 1 --2- 172.21.15.27:0/248840646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d84102230 0x7f8d84108ac0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.041+0000 7f8d89b70640 1 --2- 172.21.15.27:0/248840646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d84101870 0x7f8d84101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.041+0000 7f8d89b70640 1 --2- 172.21.15.27:0/248840646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d84100680 0x7f8d84100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.041+0000 7f8d89b70640 1 -- 172.21.15.27:0/248840646 >> 172.21.15.27:0/248840646 conn(0x7f8d840fbbf0 msgr2=0x7f8d840fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.042+0000 7f8d89b70640 1 -- 172.21.15.27:0/248840646 shutdown_connections 2024-04-25T00:13:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.042+0000 7f8d89b70640 1 -- 172.21.15.27:0/248840646 wait complete. 2024-04-25T00:13:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.042+0000 7f8d89b70640 1 Processor -- start 2024-04-25T00:13:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.042+0000 7f8d89b70640 1 -- start start 2024-04-25T00:13:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.042+0000 7f8d89b70640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d84100680 0x7f8d841045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.042+0000 7f8d89b70640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d84101870 0x7f8d84102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.042+0000 7f8d89b70640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d84102230 0x7f8d84103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.042+0000 7f8d89b70640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8410c350 con 0x7f8d84100680 2024-04-25T00:13:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.042+0000 7f8d89b70640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8410c1d0 con 0x7f8d84101870 2024-04-25T00:13:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.042+0000 7f8d89b70640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8410c4d0 con 0x7f8d84102230 2024-04-25T00:13:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.043+0000 7f8d837fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d84100680 0x7f8d841045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.043+0000 7f8d837fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d84100680 0x7f8d841045c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57488/0 (socket says 172.21.15.27:57488) 2024-04-25T00:13:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.043+0000 7f8d837fe640 1 -- 172.21.15.27:0/4214261166 learned_addr learned my addr 172.21.15.27:0/4214261166 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.043+0000 7f8d82ffd640 1 --2- 172.21.15.27:0/4214261166 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d84101870 0x7f8d84102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.043+0000 7f8d83fff640 1 --2- 172.21.15.27:0/4214261166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d84102230 0x7f8d84103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.043+0000 7f8d837fe640 1 -- 172.21.15.27:0/4214261166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d84102230 msgr2=0x7f8d84103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.043+0000 7f8d837fe640 1 --2- 172.21.15.27:0/4214261166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d84102230 0x7f8d84103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.043+0000 7f8d837fe640 1 -- 172.21.15.27:0/4214261166 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d84101870 msgr2=0x7f8d84102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.043+0000 7f8d837fe640 1 --2- 172.21.15.27:0/4214261166 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d84101870 0x7f8d84102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.043+0000 7f8d837fe640 1 -- 172.21.15.27:0/4214261166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d84103a50 con 0x7f8d84100680 2024-04-25T00:13:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.043+0000 7f8d82ffd640 1 --2- 172.21.15.27:0/4214261166 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d84101870 0x7f8d84102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:13:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.043+0000 7f8d837fe640 1 --2- 172.21.15.27:0/4214261166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d84100680 0x7f8d841045c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8d7400e9c0 tx=0x7f8d7400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.043+0000 7f8d80ff9640 1 -- 172.21.15.27:0/4214261166 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7400cd80 con 0x7f8d84100680 2024-04-25T00:13:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.043+0000 7f8d80ff9640 1 -- 172.21.15.27:0/4214261166 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d740035f0 con 0x7f8d84100680 2024-04-25T00:13:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.043+0000 7f8d89b70640 1 -- 172.21.15.27:0/4214261166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d841a1bf0 con 0x7f8d84100680 2024-04-25T00:13:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.043+0000 7f8d80ff9640 1 -- 172.21.15.27:0/4214261166 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d74010690 con 0x7f8d84100680 2024-04-25T00:13:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.043+0000 7f8d89b70640 1 -- 172.21.15.27:0/4214261166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d841a20c0 con 0x7f8d84100680 2024-04-25T00:13:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.044+0000 7f8d89b70640 1 -- 172.21.15.27:0/4214261166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d841a22f0 con 0x7f8d84100680 2024-04-25T00:13:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.045+0000 7f8d80ff9640 1 -- 172.21.15.27:0/4214261166 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8d740037a0 con 0x7f8d84100680 2024-04-25T00:13:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.046+0000 7f8d80ff9640 1 --2- 172.21.15.27:0/4214261166 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d6c0a2b90 0x7f8d6c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.046+0000 7f8d82ffd640 1 --2- 172.21.15.27:0/4214261166 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d6c0a2b90 0x7f8d6c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.046+0000 7f8d80ff9640 1 -- 172.21.15.27:0/4214261166 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8d740cf730 con 0x7f8d84100680 2024-04-25T00:13:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.047+0000 7f8d82ffd640 1 --2- 172.21.15.27:0/4214261166 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d6c0a2b90 0x7f8d6c0a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8d780058c0 tx=0x7f8d78005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:26.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.051+0000 7f8d80ff9640 1 -- 172.21.15.27:0/4214261166 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8d7409c450 con 0x7f8d84100680 2024-04-25T00:13:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.237+0000 7f8d89b70640 1 -- 172.21.15.27:0/4214261166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8d841a25e0 con 0x7f8d84100680 2024-04-25T00:13:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.238+0000 7f8d80ff9640 1 -- 172.21.15.27:0/4214261166 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24117 (secure 0 0 0) 0x7f8d7409bd50 con 0x7f8d84100680 2024-04-25T00:13:26.239 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:26.240 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:13:23.935081+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:13:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.240+0000 7f8d89b70640 1 -- 172.21.15.27:0/4214261166 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d6c0a2b90 msgr2=0x7f8d6c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.240+0000 7f8d89b70640 1 --2- 172.21.15.27:0/4214261166 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d6c0a2b90 0x7f8d6c0a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8d780058c0 tx=0x7f8d78005850 comp rx=0 tx=0).stop 2024-04-25T00:13:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.240+0000 7f8d89b70640 1 -- 172.21.15.27:0/4214261166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d84100680 msgr2=0x7f8d841045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.240+0000 7f8d89b70640 1 --2- 172.21.15.27:0/4214261166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d84100680 0x7f8d841045c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8d7400e9c0 tx=0x7f8d7400ee80 comp rx=0 tx=0).stop 2024-04-25T00:13:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.240+0000 7f8d89b70640 1 -- 172.21.15.27:0/4214261166 shutdown_connections 2024-04-25T00:13:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.240+0000 7f8d89b70640 1 --2- 172.21.15.27:0/4214261166 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d6c0a2b90 0x7f8d6c0a5050 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.240+0000 7f8d89b70640 1 --2- 172.21.15.27:0/4214261166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d84102230 0x7f8d84103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.240+0000 7f8d89b70640 1 --2- 172.21.15.27:0/4214261166 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d84101870 0x7f8d84102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.240+0000 7f8d89b70640 1 --2- 172.21.15.27:0/4214261166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d84100680 0x7f8d841045c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.240+0000 7f8d89b70640 1 -- 172.21.15.27:0/4214261166 >> 172.21.15.27:0/4214261166 conn(0x7f8d840fbbf0 msgr2=0x7f8d840fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.240+0000 7f8d89b70640 1 -- 172.21.15.27:0/4214261166 shutdown_connections 2024-04-25T00:13:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.240+0000 7f8d89b70640 1 -- 172.21.15.27:0/4214261166 wait complete. 2024-04-25T00:13:26.255 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T00:13:18.872210+0000', 'modified': '2024-04-25T00:13:23.935300+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.139:6839/2728523692', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6838', 'nonce': 2728523692}, {'type': 'v1', 'addr': '172.21.15.139:6839', 'nonce': 2728523692}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24521, 'qdb_cluster': [24521]} 2024-04-25T00:13:26.255 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T00:13:26.255 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T00:13:26.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.358+0000 7f33792ee640 1 -- 172.21.15.27:0/2121370295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374101870 msgr2=0x7f3374101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.358+0000 7f33792ee640 1 --2- 172.21.15.27:0/2121370295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374101870 0x7f3374101cf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f335c009900 tx=0x7f335c02a410 comp rx=0 tx=0).stop 2024-04-25T00:13:26.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.358+0000 7f33792ee640 1 -- 172.21.15.27:0/2121370295 shutdown_connections 2024-04-25T00:13:26.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.358+0000 7f33792ee640 1 --2- 172.21.15.27:0/2121370295 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3374102230 0x7f3374108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.358+0000 7f33792ee640 1 --2- 172.21.15.27:0/2121370295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374101870 0x7f3374101cf0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.358+0000 7f33792ee640 1 --2- 172.21.15.27:0/2121370295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3374100680 0x7f3374100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.358+0000 7f33792ee640 1 -- 172.21.15.27:0/2121370295 >> 172.21.15.27:0/2121370295 conn(0x7f33740fbbf0 msgr2=0x7f33740fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.358+0000 7f33792ee640 1 -- 172.21.15.27:0/2121370295 shutdown_connections 2024-04-25T00:13:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.358+0000 7f33792ee640 1 -- 172.21.15.27:0/2121370295 wait complete. 2024-04-25T00:13:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.358+0000 7f33792ee640 1 Processor -- start 2024-04-25T00:13:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.358+0000 7f33792ee640 1 -- start start 2024-04-25T00:13:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.358+0000 7f33792ee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3374100680 0x7f3374069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.358+0000 7f33792ee640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3374101870 0x7f337419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.358+0000 7f33792ee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374102230 0x7f337419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.358+0000 7f33792ee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f337410c300 con 0x7f3374102230 2024-04-25T00:13:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.358+0000 7f33792ee640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f337410c180 con 0x7f3374101870 2024-04-25T00:13:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.358+0000 7f33792ee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f337410c480 con 0x7f3374100680 2024-04-25T00:13:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.359+0000 7f33737fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374102230 0x7f337419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.359+0000 7f33737fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374102230 0x7f337419ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57512/0 (socket says 172.21.15.27:57512) 2024-04-25T00:13:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.359+0000 7f33737fe640 1 -- 172.21.15.27:0/4255521721 learned_addr learned my addr 172.21.15.27:0/4255521721 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.359+0000 7f3372ffd640 1 --2- 172.21.15.27:0/4255521721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3374100680 0x7f3374069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.359+0000 7f336bfff640 1 --2- 172.21.15.27:0/4255521721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3374101870 0x7f337419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.359+0000 7f33737fe640 1 -- 172.21.15.27:0/4255521721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3374100680 msgr2=0x7f3374069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.359+0000 7f33737fe640 1 --2- 172.21.15.27:0/4255521721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3374100680 0x7f3374069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.359+0000 7f33737fe640 1 -- 172.21.15.27:0/4255521721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3374101870 msgr2=0x7f337419c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.359+0000 7f33737fe640 1 --2- 172.21.15.27:0/4255521721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3374101870 0x7f337419c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.359+0000 7f33737fe640 1 -- 172.21.15.27:0/4255521721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337419f740 con 0x7f3374102230 2024-04-25T00:13:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.359+0000 7f33737fe640 1 --2- 172.21.15.27:0/4255521721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374102230 0x7f337419ee30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f336c00eaa0 tx=0x7f336c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.359+0000 7f3370ff9640 1 -- 172.21.15.27:0/4255521721 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c00cdb0 con 0x7f3374102230 2024-04-25T00:13:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.359+0000 7f3370ff9640 1 -- 172.21.15.27:0/4255521721 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f336c005020 con 0x7f3374102230 2024-04-25T00:13:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.359+0000 7f33792ee640 1 -- 172.21.15.27:0/4255521721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f337419f9d0 con 0x7f3374102230 2024-04-25T00:13:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.359+0000 7f3370ff9640 1 -- 172.21.15.27:0/4255521721 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c0116b0 con 0x7f3374102230 2024-04-25T00:13:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.359+0000 7f33792ee640 1 -- 172.21.15.27:0/4255521721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f337419fea0 con 0x7f3374102230 2024-04-25T00:13:26.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.362+0000 7f3370ff9640 1 -- 172.21.15.27:0/4255521721 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f336c005560 con 0x7f3374102230 2024-04-25T00:13:26.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.362+0000 7f337bffa740 1 Processor -- start 2024-04-25T00:13:26.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.362+0000 7f337bffa740 1 -- start start 2024-04-25T00:13:26.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.362+0000 7f337bffa740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9150440 0x55e0a9150860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.362+0000 7f337bffa740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a9155930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.362+0000 7f337bffa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9155fa0 0x55e0a9158430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.362+0000 7f337bffa740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e0a90b3f00 con 0x55e0a9150e30 2024-04-25T00:13:26.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.362+0000 7f337bffa740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e0a905c750 con 0x55e0a9150440 2024-04-25T00:13:26.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.362+0000 7f337bffa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e0a90afec0 con 0x55e0a9155fa0 2024-04-25T00:13:26.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.362+0000 7f3370ff9640 1 --2- 172.21.15.27:0/4255521721 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f33600a2b90 0x7f33600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.362+0000 7f3372ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a9155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.363+0000 7f3372ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a9155930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57526/0 (socket says 172.21.15.27:57526) 2024-04-25T00:13:26.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.363+0000 7f3372ffd640 1 -- 172.21.15.27:0/4118871721 learned_addr learned my addr 172.21.15.27:0/4118871721 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:26.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.363+0000 7f336bfff640 1 --2- 172.21.15.27:0/4118871721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9155fa0 0x55e0a9158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.363+0000 7f33737fe640 1 --2- 172.21.15.27:0/4255521721 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f33600a2b90 0x7f33600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.363+0000 7f336bfff640 1 --2- 172.21.15.27:0/4118871721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9150440 0x55e0a9150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.363+0000 7f3370ff9640 1 -- 172.21.15.27:0/4255521721 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f336c0cf420 con 0x7f3374102230 2024-04-25T00:13:26.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.363+0000 7f3372ffd640 1 -- 172.21.15.27:0/4118871721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9155fa0 msgr2=0x55e0a9158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:26.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.363+0000 7f3372ffd640 1 --2- 172.21.15.27:0/4118871721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9155fa0 0x55e0a9158430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.363+0000 7f3372ffd640 1 -- 172.21.15.27:0/4118871721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9150440 msgr2=0x55e0a9150860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.363+0000 7f3372ffd640 1 --2- 172.21.15.27:0/4118871721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9150440 0x55e0a9150860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.363+0000 7f3372ffd640 1 -- 172.21.15.27:0/4118871721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0a8dc82d0 con 0x55e0a9150e30 2024-04-25T00:13:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.363+0000 7f33737fe640 1 --2- 172.21.15.27:0/4255521721 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f33600a2b90 0x7f33600a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f336c09ba80 tx=0x7f336c053040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.363+0000 7f3372ffd640 1 --2- 172.21.15.27:0/4118871721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a9155930 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f336400abf0 tx=0x7f336400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3a2d0bd9e09336 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.363+0000 7f33697fa640 1 -- 172.21.15.27:0/4118871721 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3364005220 con 0x55e0a9150e30 2024-04-25T00:13:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.363+0000 7f33697fa640 1 -- 172.21.15.27:0/4118871721 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3364004c80 con 0x55e0a9150e30 2024-04-25T00:13:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.363+0000 7f33697fa640 1 -- 172.21.15.27:0/4118871721 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33640187c0 con 0x55e0a9150e30 2024-04-25T00:13:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.364+0000 7f337bffa740 1 -- 172.21.15.27:0/4118871721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 msgr2=0x55e0a9155930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.364+0000 7f337bffa740 1 --2- 172.21.15.27:0/4118871721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a9155930 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f336400abf0 tx=0x7f336400f790 comp rx=0 tx=0).stop 2024-04-25T00:13:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.364+0000 7f337bffa740 1 -- 172.21.15.27:0/4118871721 shutdown_connections 2024-04-25T00:13:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.364+0000 7f337bffa740 1 --2- 172.21.15.27:0/4118871721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9155fa0 0x55e0a9158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.364+0000 7f337bffa740 1 --2- 172.21.15.27:0/4118871721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a9155930 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.364+0000 7f337bffa740 1 --2- 172.21.15.27:0/4118871721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9150440 0x55e0a9150860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.364+0000 7f337bffa740 1 -- 172.21.15.27:0/4118871721 >> 172.21.15.27:0/4118871721 conn(0x55e0a914bab0 msgr2=0x55e0a914beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.364+0000 7f337bffa740 1 -- 172.21.15.27:0/4118871721 shutdown_connections 2024-04-25T00:13:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.364+0000 7f337bffa740 1 -- 172.21.15.27:0/4118871721 wait complete. 2024-04-25T00:13:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.364+0000 7f337bffa740 1 -- start start 2024-04-25T00:13:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.364+0000 7f337bffa740 1 Processor -- start 2024-04-25T00:13:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f337bffa740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9150440 0x55e0a915a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f337bffa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9150e30 0x55e0a915ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f337bffa740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9155fa0 0x55e0a9161d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f337bffa740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e0a9158f50 con 0x55e0a9155fa0 2024-04-25T00:13:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f337bffa740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e0a906fcf0 con 0x55e0a9150440 2024-04-25T00:13:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f337bffa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e0a91590d0 con 0x55e0a9150e30 2024-04-25T00:13:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f337bffa740 10 client.0 _ll_register_callbacks cb 0x55e0a906f6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:13:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f336bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9155fa0 0x55e0a9161d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f336bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9155fa0 0x55e0a9161d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57532/0 (socket says 172.21.15.27:57532) 2024-04-25T00:13:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f336bfff640 1 -- 172.21.15.27:0/3873619669 learned_addr learned my addr 172.21.15.27:0/3873619669 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f3372ffd640 1 --2- 172.21.15.27:0/3873619669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9150e30 0x55e0a915ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f336bfff640 1 --2- 172.21.15.27:0/3873619669 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9150440 0x55e0a915a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f336bfff640 1 -- 172.21.15.27:0/3873619669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9150e30 msgr2=0x55e0a915ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f336bfff640 1 --2- 172.21.15.27:0/3873619669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9150e30 0x55e0a915ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f336bfff640 1 -- 172.21.15.27:0/3873619669 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9150440 msgr2=0x55e0a915a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f336bfff640 1 --2- 172.21.15.27:0/3873619669 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9150440 0x55e0a915a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f336bfff640 1 -- 172.21.15.27:0/3873619669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0a8e926b0 con 0x55e0a9155fa0 2024-04-25T00:13:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f336bfff640 1 --2- 172.21.15.27:0/3873619669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9155fa0 0x55e0a9161d60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f335c009900 tx=0x7f335c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bde20b858f48f21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f335a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:13:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f335a7fc640 1 -- 172.21.15.27:0/3873619669 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c035040 con 0x55e0a9155fa0 2024-04-25T00:13:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f335a7fc640 1 -- 172.21.15.27:0/3873619669 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f335c03d890 con 0x55e0a9155fa0 2024-04-25T00:13:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f337bffa740 1 -- 172.21.15.27:0/3873619669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3364004c80 con 0x55e0a9155fa0 2024-04-25T00:13:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f335a7fc640 1 -- 172.21.15.27:0/3873619669 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f335c051050 con 0x55e0a9155fa0 2024-04-25T00:13:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f337bffa740 10 client.14787 fetch_fsmap learned FSMap version 17 2024-04-25T00:13:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.365+0000 7f337bffa740 1 -- 172.21.15.27:0/3873619669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55e0a9164db0 con 0x55e0a9155fa0 2024-04-25T00:13:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.366+0000 7f335a7fc640 1 -- 172.21.15.27:0/3873619669 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3364004c80 con 0x55e0a9155fa0 2024-04-25T00:13:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.366+0000 7f335a7fc640 1 -- 172.21.15.27:0/3873619669 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f335c004780 con 0x55e0a9155fa0 2024-04-25T00:13:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.366+0000 7f337bffa740 10 client.14787 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T00:13:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.366+0000 7f337bffa740 10 client.14787 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T00:13:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.366+0000 7f337bffa740 20 client.14787 populate_metadata read hostname 'smithi027' 2024-04-25T00:13:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.366+0000 7f337bffa740 1 --2- 172.21.15.27:0/3873619669 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55e0a9167460 0x55e0a91688d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.366+0000 7f337bffa740 4 client.14787 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:13:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.366+0000 7f337bffa740 1 -- 172.21.15.27:0/3873619669 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0a90c1440 con 0x55e0a9167460 2024-04-25T00:13:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.366+0000 7f3372ffd640 1 --2- 172.21.15.27:0/3873619669 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55e0a9167460 0x55e0a91688d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.367+0000 7f335a7fc640 10 client.14787 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:13:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.367+0000 7f3372ffd640 1 --2- 172.21.15.27:0/3873619669 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55e0a9167460 0x55e0a91688d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c340d5da9e40a4a server_cookie=2f2e42c14f890421 in_seq=0 out_seq=0 2024-04-25T00:13:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.368+0000 7f335a7fc640 1 -- 172.21.15.27:0/3873619669 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e0a90c1440 con 0x55e0a9167460 2024-04-25T00:13:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.368+0000 7f335a7fc640 10 client.14787 handle_command_reply: tid=0 2024-04-25T00:13:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.368+0000 7f337bffa740 1 client.14787 shutdown 2024-04-25T00:13:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.368+0000 7f337bffa740 1 -- 172.21.15.27:0/3873619669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9155fa0 msgr2=0x55e0a9161d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.368+0000 7f337bffa740 1 --2- 172.21.15.27:0/3873619669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9155fa0 0x55e0a9161d60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f335c009900 tx=0x7f335c009260 comp rx=0 tx=0).stop 2024-04-25T00:13:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.368+0000 7f337bffa740 1 -- 172.21.15.27:0/3873619669 shutdown_connections 2024-04-25T00:13:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.368+0000 7f337bffa740 1 --2- 172.21.15.27:0/3873619669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9155fa0 0x55e0a9161d60 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.368+0000 7f337bffa740 1 --2- 172.21.15.27:0/3873619669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9150e30 0x55e0a915ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.368+0000 7f337bffa740 1 --2- 172.21.15.27:0/3873619669 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55e0a9167460 0x55e0a91688d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.368+0000 7f337bffa740 1 --2- 172.21.15.27:0/3873619669 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9150440 0x55e0a915a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.368+0000 7f335a7fc640 0 client.14787 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:13:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.368+0000 7f337bffa740 1 -- 172.21.15.27:0/3873619669 >> 172.21.15.27:0/3873619669 conn(0x55e0a914bab0 msgr2=0x55e0a914aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.368+0000 7f337bffa740 1 -- 172.21.15.27:0/3873619669 shutdown_connections 2024-04-25T00:13:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.368+0000 7f337bffa740 1 -- 172.21.15.27:0/3873619669 wait complete. 2024-04-25T00:13:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.368+0000 7f337bffa740 20 client.14787 trim_cache size 0 max 16384 2024-04-25T00:13:26.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.386+0000 7f337bffa740 1 Processor -- start 2024-04-25T00:13:26.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.386+0000 7f337bffa740 1 -- start start 2024-04-25T00:13:26.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f337bffa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9150440 0x55e0a915a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f337bffa740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a9161d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f337bffa740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9155fa0 0x55e0a915a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f337bffa740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e0a9166c10 con 0x55e0a9150e30 2024-04-25T00:13:26.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f337bffa740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e0a91672e0 con 0x55e0a9155fa0 2024-04-25T00:13:26.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f337bffa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e0a91679b0 con 0x55e0a9150440 2024-04-25T00:13:26.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f3372ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a9161d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f3372ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a9161d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57548/0 (socket says 172.21.15.27:57548) 2024-04-25T00:13:26.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f3372ffd640 1 -- 172.21.15.27:0/4248005569 learned_addr learned my addr 172.21.15.27:0/4248005569 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:26.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f3372ffd640 1 -- 172.21.15.27:0/4248005569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9150440 msgr2=0x55e0a915a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:26.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f3372ffd640 1 --2- 172.21.15.27:0/4248005569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9150440 0x55e0a915a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f3372ffd640 1 -- 172.21.15.27:0/4248005569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9155fa0 msgr2=0x55e0a915a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:26.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f3372ffd640 1 --2- 172.21.15.27:0/4248005569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9155fa0 0x55e0a915a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f3372ffd640 1 -- 172.21.15.27:0/4248005569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0a914f9d0 con 0x55e0a9150e30 2024-04-25T00:13:26.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f3372ffd640 1 --2- 172.21.15.27:0/4248005569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a9161d20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f336401a040 tx=0x7f3364005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef46744937e2bfac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:26.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f3368ff9640 1 -- 172.21.15.27:0/4248005569 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3364005640 con 0x55e0a9150e30 2024-04-25T00:13:26.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f3368ff9640 1 -- 172.21.15.27:0/4248005569 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f336400a590 con 0x55e0a9150e30 2024-04-25T00:13:26.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.387+0000 7f3368ff9640 1 -- 172.21.15.27:0/4248005569 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3364004be0 con 0x55e0a9150e30 2024-04-25T00:13:26.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.388+0000 7f337bffa740 1 -- 172.21.15.27:0/4248005569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 msgr2=0x55e0a9161d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.388+0000 7f337bffa740 1 --2- 172.21.15.27:0/4248005569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a9161d20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f336401a040 tx=0x7f3364005de0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.388+0000 7f337bffa740 1 -- 172.21.15.27:0/4248005569 shutdown_connections 2024-04-25T00:13:26.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.388+0000 7f337bffa740 1 --2- 172.21.15.27:0/4248005569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9155fa0 0x55e0a915a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.388+0000 7f337bffa740 1 --2- 172.21.15.27:0/4248005569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a9161d20 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.388+0000 7f337bffa740 1 --2- 172.21.15.27:0/4248005569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9150440 0x55e0a915a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.388+0000 7f337bffa740 1 -- 172.21.15.27:0/4248005569 >> 172.21.15.27:0/4248005569 conn(0x55e0a914bab0 msgr2=0x55e0a91521f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:26.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.388+0000 7f337bffa740 1 -- 172.21.15.27:0/4248005569 shutdown_connections 2024-04-25T00:13:26.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.388+0000 7f337bffa740 1 -- 172.21.15.27:0/4248005569 wait complete. 2024-04-25T00:13:26.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.388+0000 7f337bffa740 1 -- start start 2024-04-25T00:13:26.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.388+0000 7f337bffa740 1 Processor -- start 2024-04-25T00:13:26.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f337bffa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9150440 0x55e0a915deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f337bffa740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a915e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f337bffa740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9155fa0 0x55e0a91610a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f337bffa740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e0a9168750 con 0x55e0a9150e30 2024-04-25T00:13:26.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f337bffa740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e0a9168080 con 0x55e0a9155fa0 2024-04-25T00:13:26.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f337bffa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e0a9168e20 con 0x55e0a9150440 2024-04-25T00:13:26.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f337bffa740 10 client.0 _ll_register_callbacks cb 0x7f335c031950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:13:26.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f3372ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a915e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f3372ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a915e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57556/0 (socket says 172.21.15.27:57556) 2024-04-25T00:13:26.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f3372ffd640 1 -- 172.21.15.27:0/824042069 learned_addr learned my addr 172.21.15.27:0/824042069 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:26.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f3372ffd640 1 -- 172.21.15.27:0/824042069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9150440 msgr2=0x55e0a915deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:26.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f3372ffd640 1 --2- 172.21.15.27:0/824042069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9150440 0x55e0a915deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f336bfff640 1 --2- 172.21.15.27:0/824042069 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9155fa0 0x55e0a91610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f3372ffd640 1 -- 172.21.15.27:0/824042069 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9155fa0 msgr2=0x55e0a91610a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:26.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f3372ffd640 1 --2- 172.21.15.27:0/824042069 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9155fa0 0x55e0a91610a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f3372ffd640 1 -- 172.21.15.27:0/824042069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0a91616e0 con 0x55e0a9150e30 2024-04-25T00:13:26.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f336bfff640 1 --2- 172.21.15.27:0/824042069 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9155fa0 0x55e0a91610a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:26.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f3372ffd640 1 --2- 172.21.15.27:0/824042069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a915e3f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f336400ab70 tx=0x7f336400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ada6253db717fb96 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:26.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f335a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:13:26.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f335a7fc640 1 -- 172.21.15.27:0/824042069 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3364037060 con 0x55e0a9150e30 2024-04-25T00:13:26.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f335a7fc640 1 -- 172.21.15.27:0/824042069 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3364004b60 con 0x55e0a9150e30 2024-04-25T00:13:26.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f337bffa740 1 -- 172.21.15.27:0/824042069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f336400a590 con 0x55e0a9150e30 2024-04-25T00:13:26.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f335a7fc640 1 -- 172.21.15.27:0/824042069 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f336403e050 con 0x55e0a9150e30 2024-04-25T00:13:26.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f337bffa740 10 client.14793 fetch_fsmap learned FSMap version 17 2024-04-25T00:13:26.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.389+0000 7f337bffa740 1 -- 172.21.15.27:0/824042069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55e0a91cdb30 con 0x55e0a9150e30 2024-04-25T00:13:26.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.390+0000 7f335a7fc640 1 -- 172.21.15.27:0/824042069 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f336400a590 con 0x55e0a9150e30 2024-04-25T00:13:26.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.390+0000 7f335a7fc640 1 -- 172.21.15.27:0/824042069 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f3364029360 con 0x55e0a9150e30 2024-04-25T00:13:26.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.390+0000 7f337bffa740 10 client.14793 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T00:13:26.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.390+0000 7f337bffa740 10 client.14793 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T00:13:26.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.390+0000 7f337bffa740 20 client.14793 populate_metadata read hostname 'smithi027' 2024-04-25T00:13:26.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.390+0000 7f337bffa740 1 --2- 172.21.15.27:0/824042069 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55e0a91d2110 0x55e0a91d0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.390+0000 7f337bffa740 4 client.14793 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T00:13:26.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.390+0000 7f337bffa740 1 -- 172.21.15.27:0/824042069 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "status"}) -- 0x55e0a90bb790 con 0x55e0a91d2110 2024-04-25T00:13:26.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.390+0000 7f336bfff640 1 --2- 172.21.15.27:0/824042069 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55e0a91d2110 0x55e0a91d0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.390+0000 7f336bfff640 1 --2- 172.21.15.27:0/824042069 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55e0a91d2110 0x55e0a91d0e20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43066a962866dcb5 server_cookie=9fc12ac506bbdd01 in_seq=0 out_seq=0 2024-04-25T00:13:26.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.390+0000 7f335a7fc640 10 client.14793 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:13:26.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f335a7fc640 1 -- 172.21.15.27:0/824042069 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55e0a90bb790 con 0x55e0a91d2110 2024-04-25T00:13:26.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f335a7fc640 10 client.14793 handle_command_reply: tid=0 2024-04-25T00:13:26.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f337bffa740 1 client.14793 shutdown 2024-04-25T00:13:26.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f337bffa740 1 -- 172.21.15.27:0/824042069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 msgr2=0x55e0a915e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f337bffa740 1 --2- 172.21.15.27:0/824042069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a915e3f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f336400ab70 tx=0x7f336400f790 comp rx=0 tx=0).stop 2024-04-25T00:13:26.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f337bffa740 1 -- 172.21.15.27:0/824042069 shutdown_connections 2024-04-25T00:13:26.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f337bffa740 1 --2- 172.21.15.27:0/824042069 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55e0a91d2110 0x55e0a91d0e20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f337bffa740 1 --2- 172.21.15.27:0/824042069 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0a9155fa0 0x55e0a91610a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f335a7fc640 0 client.14793 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:13:26.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f337bffa740 1 --2- 172.21.15.27:0/824042069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0a9150e30 0x55e0a915e3f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f337bffa740 1 --2- 172.21.15.27:0/824042069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0a9150440 0x55e0a915deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f337bffa740 1 -- 172.21.15.27:0/824042069 >> 172.21.15.27:0/824042069 conn(0x55e0a914bab0 msgr2=0x55e0a915a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:26.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f337bffa740 1 -- 172.21.15.27:0/824042069 shutdown_connections 2024-04-25T00:13:26.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f337bffa740 1 -- 172.21.15.27:0/824042069 wait complete. 2024-04-25T00:13:26.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f337bffa740 20 client.14793 trim_cache size 0 max 16384 2024-04-25T00:13:26.399 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:13:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "518857f8-0297-11ef-bc93-c7b262605968", 2024-04-25T00:13:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-04-25T00:13:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24521, 2024-04-25T00:13:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T00:13:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T00:13:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T00:13:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.8849745349999996, 2024-04-25T00:13:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 17, 2024-04-25T00:13:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 89, 2024-04-25T00:13:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T00:13:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 13.016609128000001 2024-04-25T00:13:26.400 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:13:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f33792ee640 1 -- 172.21.15.27:0/4255521721 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f33600a2b90 msgr2=0x7f33600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f33792ee640 1 --2- 172.21.15.27:0/4255521721 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f33600a2b90 0x7f33600a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f336c09ba80 tx=0x7f336c053040 comp rx=0 tx=0).stop 2024-04-25T00:13:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f33792ee640 1 -- 172.21.15.27:0/4255521721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374102230 msgr2=0x7f337419ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f33792ee640 1 --2- 172.21.15.27:0/4255521721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374102230 0x7f337419ee30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f336c00eaa0 tx=0x7f336c00ef60 comp rx=0 tx=0).stop 2024-04-25T00:13:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f33792ee640 1 -- 172.21.15.27:0/4255521721 shutdown_connections 2024-04-25T00:13:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f33792ee640 1 --2- 172.21.15.27:0/4255521721 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f33600a2b90 0x7f33600a5050 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f33792ee640 1 --2- 172.21.15.27:0/4255521721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3374102230 0x7f337419ee30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f33792ee640 1 --2- 172.21.15.27:0/4255521721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3374101870 0x7f337419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f33792ee640 1 --2- 172.21.15.27:0/4255521721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3374100680 0x7f3374069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.391+0000 7f33792ee640 1 -- 172.21.15.27:0/4255521721 >> 172.21.15.27:0/4255521721 conn(0x7f33740fbbf0 msgr2=0x7f3374104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:26.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.392+0000 7f33792ee640 1 -- 172.21.15.27:0/4255521721 shutdown_connections 2024-04-25T00:13:26.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:26.392+0000 7f33792ee640 1 -- 172.21.15.27:0/4255521721 wait complete. 2024-04-25T00:13:26.403 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T00:13:26.403 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T00:13:26.404 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-25T00:13:26.647 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:13:26.898 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:26 smithi027 ceph-mon[29208]: pgmap v297: 129 pgs: 4 creating+activating, 4 creating+peering, 118 active+clean, 3 unknown; 579 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2024-04-25T00:13:26.898 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:26 smithi027 ceph-mon[29208]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T00:13:26.898 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:26 smithi027 ceph-mon[29208]: Cluster is now healthy 2024-04-25T00:13:26.898 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:26 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4140432367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:26.898 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:26 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4214261166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:27.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:26 smithi139 ceph-mon[32833]: pgmap v297: 129 pgs: 4 creating+activating, 4 creating+peering, 118 active+clean, 3 unknown; 579 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2024-04-25T00:13:27.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:26 smithi139 ceph-mon[32833]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T00:13:27.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:26 smithi139 ceph-mon[32833]: Cluster is now healthy 2024-04-25T00:13:27.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:26 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4140432367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:27.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:26 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4214261166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:27.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:26 smithi153 ceph-mon[32896]: pgmap v297: 129 pgs: 4 creating+activating, 4 creating+peering, 118 active+clean, 3 unknown; 579 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2024-04-25T00:13:27.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:26 smithi153 ceph-mon[32896]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T00:13:27.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:26 smithi153 ceph-mon[32896]: Cluster is now healthy 2024-04-25T00:13:27.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:26 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4140432367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:27.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:26 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4214261166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:28.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.286+0000 7fe109a10700 1 -- 172.21.15.27:0/3701258622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe104104cd0 msgr2=0x7fe104105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.286+0000 7fe109a10700 1 --2- 172.21.15.27:0/3701258622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe104104cd0 0x7fe104105150 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc00b0a0 tx=0x7fe0fc00b3b0 comp rx=0 tx=0).stop 2024-04-25T00:13:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.286+0000 7fe109a10700 1 -- 172.21.15.27:0/3701258622 shutdown_connections 2024-04-25T00:13:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.286+0000 7fe109a10700 1 --2- 172.21.15.27:0/3701258622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe104104cd0 0x7fe104105150 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.286+0000 7fe109a10700 1 --2- 172.21.15.27:0/3701258622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe104107260 0x7fe10406a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.286+0000 7fe109a10700 1 --2- 172.21.15.27:0/3701258622 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe10406aea0 0x7fe10406b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.286+0000 7fe109a10700 1 -- 172.21.15.27:0/3701258622 >> 172.21.15.27:0/3701258622 conn(0x7fe104076fa0 msgr2=0x7fe1040773b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.286+0000 7fe109a10700 1 -- 172.21.15.27:0/3701258622 shutdown_connections 2024-04-25T00:13:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.286+0000 7fe109a10700 1 -- 172.21.15.27:0/3701258622 wait complete. 2024-04-25T00:13:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.286+0000 7fe109a10700 1 Processor -- start 2024-04-25T00:13:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe109a10700 1 -- start start 2024-04-25T00:13:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe109a10700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe10406aea0 0x7fe10419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe109a10700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe104104cd0 0x7fe10419f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe109a10700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe104107260 0x7fe1041a37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe109a10700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe104118ba0 con 0x7fe104107260 2024-04-25T00:13:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe109a10700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe104118a20 con 0x7fe104104cd0 2024-04-25T00:13:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe109a10700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe104118d20 con 0x7fe10406aea0 2024-04-25T00:13:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe1037fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe104107260 0x7fe1041a37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe1037fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe104107260 0x7fe1041a37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44110/0 (socket says 172.21.15.27:44110) 2024-04-25T00:13:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe1037fe700 1 -- 172.21.15.27:0/722926171 learned_addr learned my addr 172.21.15.27:0/722926171 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe102ffd700 1 --2- 172.21.15.27:0/722926171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe10406aea0 0x7fe10419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe1027fc700 1 --2- 172.21.15.27:0/722926171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe104104cd0 0x7fe10419f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe1037fe700 1 -- 172.21.15.27:0/722926171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe10406aea0 msgr2=0x7fe10419eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe1037fe700 1 --2- 172.21.15.27:0/722926171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe10406aea0 0x7fe10419eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe1037fe700 1 -- 172.21.15.27:0/722926171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe104104cd0 msgr2=0x7fe10419f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe1037fe700 1 --2- 172.21.15.27:0/722926171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe104104cd0 0x7fe10419f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe1037fe700 1 -- 172.21.15.27:0/722926171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1041a3f00 con 0x7fe104107260 2024-04-25T00:13:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe1027fc700 1 --2- 172.21.15.27:0/722926171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe104104cd0 0x7fe10419f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:13:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.287+0000 7fe1037fe700 1 --2- 172.21.15.27:0/722926171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe104107260 0x7fe1041a37c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc00b070 tx=0x7fe0fc004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.288+0000 7fe108a0e700 1 -- 172.21.15.27:0/722926171 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc021070 con 0x7fe104107260 2024-04-25T00:13:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.288+0000 7fe109a10700 1 -- 172.21.15.27:0/722926171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1041a4190 con 0x7fe104107260 2024-04-25T00:13:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.288+0000 7fe108a0e700 1 -- 172.21.15.27:0/722926171 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0fc003c00 con 0x7fe104107260 2024-04-25T00:13:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.288+0000 7fe108a0e700 1 -- 172.21.15.27:0/722926171 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc0047f0 con 0x7fe104107260 2024-04-25T00:13:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.288+0000 7fe109a10700 1 -- 172.21.15.27:0/722926171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1041aba70 con 0x7fe104107260 2024-04-25T00:13:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.289+0000 7fe109a10700 1 -- 172.21.15.27:0/722926171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1041017b0 con 0x7fe104107260 2024-04-25T00:13:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.290+0000 7fe108a0e700 1 -- 172.21.15.27:0/722926171 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe0fc007980 con 0x7fe104107260 2024-04-25T00:13:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.290+0000 7fe108a0e700 1 --2- 172.21.15.27:0/722926171 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe0ec0a2c20 0x7fe0ec0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.290+0000 7fe102ffd700 1 --2- 172.21.15.27:0/722926171 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe0ec0a2c20 0x7fe0ec0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.290+0000 7fe108a0e700 1 -- 172.21.15.27:0/722926171 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe0fc00f020 con 0x7fe104107260 2024-04-25T00:13:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.291+0000 7fe102ffd700 1 --2- 172.21.15.27:0/722926171 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe0ec0a2c20 0x7fe0ec0a50e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4009c60 tx=0x7fe0f4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.294+0000 7fe108a0e700 1 -- 172.21.15.27:0/722926171 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe0fc02a020 con 0x7fe104107260 2024-04-25T00:13:28.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.546+0000 7fe109a10700 1 -- 172.21.15.27:0/722926171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe1041abc40 con 0x7fe104107260 2024-04-25T00:13:28.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.546+0000 7fe108a0e700 1 -- 172.21.15.27:0/722926171 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7fe0fc0074e0 con 0x7fe104107260 2024-04-25T00:13:28.547 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T00:13:28.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.548+0000 7fe109a10700 1 -- 172.21.15.27:0/722926171 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe0ec0a2c20 msgr2=0x7fe0ec0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:28.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.548+0000 7fe109a10700 1 --2- 172.21.15.27:0/722926171 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe0ec0a2c20 0x7fe0ec0a50e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4009c60 tx=0x7fe0f4009260 comp rx=0 tx=0).stop 2024-04-25T00:13:28.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.548+0000 7fe109a10700 1 -- 172.21.15.27:0/722926171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe104107260 msgr2=0x7fe1041a37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:28.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.548+0000 7fe109a10700 1 --2- 172.21.15.27:0/722926171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe104107260 0x7fe1041a37c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc00b070 tx=0x7fe0fc004c80 comp rx=0 tx=0).stop 2024-04-25T00:13:28.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.548+0000 7fe109a10700 1 -- 172.21.15.27:0/722926171 shutdown_connections 2024-04-25T00:13:28.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.548+0000 7fe109a10700 1 --2- 172.21.15.27:0/722926171 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe0ec0a2c20 0x7fe0ec0a50e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:28.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.548+0000 7fe109a10700 1 --2- 172.21.15.27:0/722926171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe104107260 0x7fe1041a37c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:28.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.548+0000 7fe109a10700 1 --2- 172.21.15.27:0/722926171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe10406aea0 0x7fe10419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:28.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.548+0000 7fe109a10700 1 --2- 172.21.15.27:0/722926171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe104104cd0 0x7fe10419f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:28.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.548+0000 7fe109a10700 1 -- 172.21.15.27:0/722926171 >> 172.21.15.27:0/722926171 conn(0x7fe104076fa0 msgr2=0x7fe104103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:28.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.549+0000 7fe109a10700 1 -- 172.21.15.27:0/722926171 shutdown_connections 2024-04-25T00:13:28.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:28.549+0000 7fe109a10700 1 -- 172.21.15.27:0/722926171 wait complete. 2024-04-25T00:13:28.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:28 smithi027 ceph-mon[29208]: pgmap v298: 129 pgs: 4 creating+activating, 4 creating+peering, 121 active+clean; 579 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-04-25T00:13:28.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:28 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/722926171' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T00:13:29.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:28 smithi139 ceph-mon[32833]: pgmap v298: 129 pgs: 4 creating+activating, 4 creating+peering, 121 active+clean; 579 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-04-25T00:13:29.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:28 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/722926171' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T00:13:29.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:28 smithi153 ceph-mon[32896]: pgmap v298: 129 pgs: 4 creating+activating, 4 creating+peering, 121 active+clean; 579 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-04-25T00:13:29.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:28 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/722926171' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T00:13:29.205 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T00:13:29.205 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T00:13:29.205 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-04-25T00:13:29.205 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-25T00:13:29.205 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T00:13:29.205 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T00:13:29.206 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T00:13:29.206 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T00:13:29.206 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T00:13:29.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:13:29.206 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-04-25T00:13:29.224 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T00:13:29.224 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T00:13:29.224 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T00:13:29.224 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-25T00:13:29.224 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-04-25T00:13:29.224 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-25T00:13:29.224 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T00:13:29.225 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-04-25T00:13:29.225 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-04-25T00:13:29.225 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T00:13:29.225 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-04-25T00:13:29.225 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-04-25T00:13:29.225 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T00:13:29.225 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-04-25T00:13:29.225 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-04-25T00:13:29.225 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T00:13:29.225 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42176sec preferred_lft 42176sec 2024-04-25T00:13:29.225 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-04-25T00:13:29.225 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-25T00:13:29.225 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T00:13:29.225 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-04-25T00:13:29.226 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-04-25T00:13:29.226 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T00:13:29.226 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:13:29.226 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T00:13:29.227 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add name ceph-brx type bridge 2024-04-25T00:13:29.227 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr flush dev ceph-brx 2024-04-25T00:13:29.227 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set ceph-brx up 2024-04-25T00:13:29.227 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T00:13:29.227 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T00:13:29.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:13:29.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:13:29.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:13:29.415 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T00:13:29.440 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-04-25T00:13:29.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:13:29.442 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-04-25T00:13:29.577 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-04-25T00:13:29.577 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T00:13:29.577 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T00:13:29.577 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T00:13:29.577 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T00:13:29.577 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:13:29.578 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T00:13:29.578 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T00:13:29.578 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T00:13:29.578 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T00:13:29.578 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T00:13:29.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:13:29.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:13:29.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:13:29.825 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-04-25T00:13:29.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:13:29.841 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2024-04-25T00:13:29.896 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:13:29.896 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T00:13:29.896 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T00:13:29.896 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-25T00:13:29.896 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T00:13:29.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:13:30.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:13:30.014 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-25T00:13:30.014 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:13:30.014 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T00:13:30.014 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-25T00:13:30.015 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T00:13:30.015 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-25T00:13:30.015 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-25T00:13:30.015 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-25T00:13:30.015 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T00:13:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:13:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:13:30.146 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:13:30.146 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T00:13:30.146 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-04-25T00:13:30.146 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T00:13:30.147 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T00:13:30.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:13:30.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:13:30.223 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-25T00:13:30.223 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T00:13:30.224 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-25T00:13:30.238 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T00:13:30.238 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T00:13:30.239 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-25T00:13:30.294 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-04-25T00:13:30.359 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T00:13:30.415 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-04-25T00:13:30.416 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-04-25T00:13:30.416 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-04-25T00:13:30.416 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-04-25T00:13:30.416 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-04-25T00:13:30.416 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-04-25T00:13:30.416 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-04-25T00:13:30.416 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-04-25T00:13:30.416 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-04-25T00:13:30.416 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-04-25T00:13:30.416 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-04-25T00:13:30.416 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-04-25T00:13:30.416 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T00:13:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-04-25T00:13:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-04-25T00:13:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-04-25T00:13:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-04-25T00:13:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-04-25T00:13:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T00:13:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-04-25T00:13:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T00:13:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T00:13:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T00:13:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T00:13:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-04-25T00:13:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-04-25T00:13:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay 2024-04-25T00:13:30.418 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/4ccb253ceef18af6f4c180b89d6bb8eaf03c23ccc398a56e13eb45c7e6616dd0/merged 2024-04-25T00:13:30.418 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/dfb50772ac34cfca7f4108e49bf6558e91733f742e50bf59ec8a6075c072ff27/merged 2024-04-25T00:13:30.418 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/0 2024-04-25T00:13:30.418 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T00:13:30.418 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/01178c172bb5c4052ebee4f64cc7ae8395689fe6d2edc6de3994cf99e8ad045d/merged 2024-04-25T00:13:30.418 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/f2aa4fde2fa8f7b8de8d648a981d3828472b3c73e8548dee9e54da3a69a73e2f/merged 2024-04-25T00:13:30.418 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/e91efaf87f97290066d857b7c1775fc26a4ab218a8f43d6587f9f6bf0aaf435d/merged 2024-04-25T00:13:30.418 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/48ea4aeefd996c70b3ad6b69b9fc4104d1ec63e62f784e411c8707e36a347bb6/merged 2024-04-25T00:13:30.418 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/427d34534e4b6fbbda4d8014dfd93a152797912c669fb5e76071da9c07020e8d/merged 2024-04-25T00:13:30.418 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/47b5490968ac5b6fc6e17e5be371c6719a6b2d13b765f07b14a90113a12f4bc9/merged 2024-04-25T00:13:30.418 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/395924e71e23f48e281e10f831be46d6fa83159d1ffb05280a06f17b54873257/merged 2024-04-25T00:13:30.418 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/4669533da0784d63661aa9d7efaba26a1e686bc396f09923c3cfa43f71ff8399/merged 2024-04-25T00:13:30.418 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-04-25T00:13:30.418 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T00:13:30.419 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T00:13:30.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:13:30.419 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-04-25T00:13:30.470 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T00:13:30.471 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-25T00:13:30.513 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-04-25T00:13:30.537 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T00:13:30.592 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-04-25T00:13:30.593 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-04-25T00:13:30.593 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-04-25T00:13:30.593 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-04-25T00:13:30.593 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-04-25T00:13:30.593 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-04-25T00:13:30.593 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-04-25T00:13:30.593 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-04-25T00:13:30.593 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-04-25T00:13:30.593 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-04-25T00:13:30.593 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-04-25T00:13:30.593 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-04-25T00:13:30.594 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T00:13:30.594 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-04-25T00:13:30.594 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-04-25T00:13:30.594 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-04-25T00:13:30.594 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-04-25T00:13:30.594 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-04-25T00:13:30.594 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T00:13:30.594 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-04-25T00:13:30.594 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T00:13:30.594 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T00:13:30.594 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T00:13:30.594 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T00:13:30.595 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-04-25T00:13:30.595 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-04-25T00:13:30.595 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay 2024-04-25T00:13:30.595 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/4ccb253ceef18af6f4c180b89d6bb8eaf03c23ccc398a56e13eb45c7e6616dd0/merged 2024-04-25T00:13:30.595 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/dfb50772ac34cfca7f4108e49bf6558e91733f742e50bf59ec8a6075c072ff27/merged 2024-04-25T00:13:30.595 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/0 2024-04-25T00:13:30.595 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T00:13:30.595 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/01178c172bb5c4052ebee4f64cc7ae8395689fe6d2edc6de3994cf99e8ad045d/merged 2024-04-25T00:13:30.595 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/f2aa4fde2fa8f7b8de8d648a981d3828472b3c73e8548dee9e54da3a69a73e2f/merged 2024-04-25T00:13:30.595 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/e91efaf87f97290066d857b7c1775fc26a4ab218a8f43d6587f9f6bf0aaf435d/merged 2024-04-25T00:13:30.595 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/48ea4aeefd996c70b3ad6b69b9fc4104d1ec63e62f784e411c8707e36a347bb6/merged 2024-04-25T00:13:30.595 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/427d34534e4b6fbbda4d8014dfd93a152797912c669fb5e76071da9c07020e8d/merged 2024-04-25T00:13:30.595 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/47b5490968ac5b6fc6e17e5be371c6719a6b2d13b765f07b14a90113a12f4bc9/merged 2024-04-25T00:13:30.595 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/395924e71e23f48e281e10f831be46d6fa83159d1ffb05280a06f17b54873257/merged 2024-04-25T00:13:30.596 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/4669533da0784d63661aa9d7efaba26a1e686bc396f09923c3cfa43f71ff8399/merged 2024-04-25T00:13:30.596 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-04-25T00:13:30.596 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T00:13:30.596 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T00:13:30.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:13:30.596 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-04-25T00:13:30.629 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi027.stderr:2024-04-25T00:13:30.628+0000 7f46577b8480 -1 init, newargv = 0x55cdce021150 newargc=15 2024-04-25T00:13:30.629 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi027.stderr:ceph-fuse[70033]: starting ceph client 2024-04-25T00:13:30.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi027.stderr:ceph-fuse[70033]: starting fuse 2024-04-25T00:13:30.644 INFO:teuthology.orchestra.run.smithi027.stdout:115 2024-04-25T00:13:30.644 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-25T00:13:30.644 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-04-25T00:13:30.644 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-04-25T00:13:30.644 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-04-25T00:13:30.644 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-04-25T00:13:30.644 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-04-25T00:13:30.645 DEBUG:teuthology.orchestra.run.smithi027:> 2024-04-25T00:13:30.645 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-04-25T00:13:30.645 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-25T00:13:30.645 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-04-25T00:13:30.645 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-25T00:13:30.645 DEBUG:teuthology.orchestra.run.smithi027:> 2024-04-25T00:13:30.645 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-04-25T00:13:30.645 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-04-25T00:13:30.645 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-04-25T00:13:30.645 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-04-25T00:13:30.645 DEBUG:teuthology.orchestra.run.smithi027:> 2024-04-25T00:13:30.645 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-04-25T00:13:30.645 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T00:13:30.645 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T00:13:30.646 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T00:13:30.646 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-04-25T00:13:30.646 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-04-25T00:13:30.646 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-04-25T00:13:30.646 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T00:13:30.646 DEBUG:teuthology.orchestra.run.smithi027:> 2024-04-25T00:13:30.646 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.admin")) 2024-04-25T00:13:30.646 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-04-25T00:13:30.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:30 smithi027 ceph-mon[29208]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 3 op/s 2024-04-25T00:13:30.764 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.admin.70033.asok 2024-04-25T00:13:30.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:13:30.775 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.70033.asok 2024-04-25T00:13:30.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:13:30.775 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.70033.asok status 2024-04-25T00:13:30.913 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:13:30.913 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-04-25T00:13:30.913 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "d2c45d977142e024a59b68a8f1dce8f0b9840551", 2024-04-25T00:13:30.913 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T00:13:30.913 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "admin", 2024-04-25T00:13:30.913 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-04-25T00:13:30.913 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-25T00:13:30.913 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "70033", 2024-04-25T00:13:30.913 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-04-25T00:13:30.913 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:13:30.913 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-04-25T00:13:30.913 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-04-25T00:13:30.914 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 14805, 2024-04-25T00:13:30.914 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-04-25T00:13:30.914 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-04-25T00:13:30.914 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-04-25T00:13:30.914 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 14805 2024-04-25T00:13:30.914 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:13:30.914 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-04-25T00:13:30.914 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-04-25T00:13:30.914 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-04-25T00:13:30.914 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3081614259 2024-04-25T00:13:30.914 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:13:30.914 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:13:30.914 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-04-25T00:13:30.915 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-04-25T00:13:30.915 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-04-25T00:13:30.915 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3081614259 2024-04-25T00:13:30.915 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:13:30.915 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.14805 192.168.0.1:0/3081614259", 2024-04-25T00:13:30.915 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/3081614259", 2024-04-25T00:13:30.915 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-04-25T00:13:30.915 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 17, 2024-04-25T00:13:30.915 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 89, 2024-04-25T00:13:30.915 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-04-25T00:13:30.915 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-04-25T00:13:30.915 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-04-25T00:13:30.915 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:13:30.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:13:30.921 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-04-25T00:13:30.921 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-25T00:13:30.937 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-04-25T00:13:30.937 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-25T00:13:30.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:13:30.937 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-25T00:13:31.003 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-25T00:13:31.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:30 smithi139 ceph-mon[32833]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 3 op/s 2024-04-25T00:13:31.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:30 smithi153 ceph-mon[32896]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 3 op/s 2024-04-25T00:13:31.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T00:13:31.118 INFO:teuthology.orchestra.run.smithi027.stdout:# file: . 2024-04-25T00:13:31.118 INFO:teuthology.orchestra.run.smithi027.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-25T00:13:31.118 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:31.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:13:31.124 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:13:31.125 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T00:13:31.142 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:13:31.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.283+0000 7feebaf5a640 1 -- 172.21.15.27:0/3624033450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb4102250 msgr2=0x7feeb4108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.283+0000 7feebaf5a640 1 --2- 172.21.15.27:0/3624033450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb4102250 0x7feeb4108ae0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7feeac009ca0 tx=0x7feeac02a8c0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.283+0000 7feebaf5a640 1 -- 172.21.15.27:0/3624033450 shutdown_connections 2024-04-25T00:13:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.283+0000 7feebaf5a640 1 --2- 172.21.15.27:0/3624033450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb4102250 0x7feeb4108ae0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.283+0000 7feebaf5a640 1 --2- 172.21.15.27:0/3624033450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feeb4101890 0x7feeb4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.283+0000 7feebaf5a640 1 --2- 172.21.15.27:0/3624033450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feeb41006a0 0x7feeb4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.283+0000 7feebaf5a640 1 -- 172.21.15.27:0/3624033450 >> 172.21.15.27:0/3624033450 conn(0x7feeb40fbc30 msgr2=0x7feeb40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.283+0000 7feebaf5a640 1 -- 172.21.15.27:0/3624033450 shutdown_connections 2024-04-25T00:13:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.283+0000 7feebaf5a640 1 -- 172.21.15.27:0/3624033450 wait complete. 2024-04-25T00:13:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feebaf5a640 1 Processor -- start 2024-04-25T00:13:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feebaf5a640 1 -- start start 2024-04-25T00:13:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feebaf5a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feeb41006a0 0x7feeb4197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feebaf5a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb4101890 0x7feeb41981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feebaf5a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feeb4102250 0x7feeb419f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feebaf5a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feeb410c410 con 0x7feeb4101890 2024-04-25T00:13:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feebaf5a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feeb410c290 con 0x7feeb41006a0 2024-04-25T00:13:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feebaf5a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feeb410c590 con 0x7feeb4102250 2024-04-25T00:13:31.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feeb3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb4101890 0x7feeb41981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feeb3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb4101890 0x7feeb41981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44124/0 (socket says 172.21.15.27:44124) 2024-04-25T00:13:31.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feeb3fff640 1 -- 172.21.15.27:0/1166390038 learned_addr learned my addr 172.21.15.27:0/1166390038 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:31.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feeb8cf6640 1 --2- 172.21.15.27:0/1166390038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feeb41006a0 0x7feeb4197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feeb94f7640 1 --2- 172.21.15.27:0/1166390038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feeb4102250 0x7feeb419f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feeb3fff640 1 -- 172.21.15.27:0/1166390038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feeb4102250 msgr2=0x7feeb419f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feeb3fff640 1 --2- 172.21.15.27:0/1166390038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feeb4102250 0x7feeb419f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feeb3fff640 1 -- 172.21.15.27:0/1166390038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feeb41006a0 msgr2=0x7feeb4197cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feeb3fff640 1 --2- 172.21.15.27:0/1166390038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feeb41006a0 0x7feeb4197cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feeb3fff640 1 -- 172.21.15.27:0/1166390038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb419f850 con 0x7feeb4101890 2024-04-25T00:13:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feeb8cf6640 1 --2- 172.21.15.27:0/1166390038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feeb41006a0 0x7feeb4197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:13:31.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.284+0000 7feeb3fff640 1 --2- 172.21.15.27:0/1166390038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb4101890 0x7feeb41981f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7feea800b4a0 tx=0x7feea800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:31.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.285+0000 7feeb1ffb640 1 -- 172.21.15.27:0/1166390038 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea8014070 con 0x7feeb4101890 2024-04-25T00:13:31.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.285+0000 7feebaf5a640 1 -- 172.21.15.27:0/1166390038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeb419fb40 con 0x7feeb4101890 2024-04-25T00:13:31.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.285+0000 7feeb1ffb640 1 -- 172.21.15.27:0/1166390038 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feea8010af0 con 0x7feeb4101890 2024-04-25T00:13:31.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.285+0000 7feeb1ffb640 1 -- 172.21.15.27:0/1166390038 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea8017d10 con 0x7feeb4101890 2024-04-25T00:13:31.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.285+0000 7feebaf5a640 1 -- 172.21.15.27:0/1166390038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeb41a0090 con 0x7feeb4101890 2024-04-25T00:13:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.286+0000 7feebaf5a640 1 -- 172.21.15.27:0/1166390038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeb41a02c0 con 0x7feeb4101890 2024-04-25T00:13:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.287+0000 7feeb1ffb640 1 -- 172.21.15.27:0/1166390038 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7feea8017070 con 0x7feeb4101890 2024-04-25T00:13:31.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.288+0000 7feeb1ffb640 1 --2- 172.21.15.27:0/1166390038 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fee9c0a2b90 0x7fee9c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.288+0000 7feeb8cf6640 1 --2- 172.21.15.27:0/1166390038 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fee9c0a2b90 0x7fee9c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.288+0000 7feeb1ffb640 1 -- 172.21.15.27:0/1166390038 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7feea80ceb20 con 0x7feeb4101890 2024-04-25T00:13:31.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.288+0000 7feeb8cf6640 1 --2- 172.21.15.27:0/1166390038 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fee9c0a2b90 0x7fee9c0a5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7feeb4199110 tx=0x7feea4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:31.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.291+0000 7feeb1ffb640 1 -- 172.21.15.27:0/1166390038 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7feea80cee20 con 0x7feeb4101890 2024-04-25T00:13:31.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.480+0000 7feebaf5a640 1 -- 172.21.15.27:0/1166390038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feeb41a0530 con 0x7feeb4101890 2024-04-25T00:13:31.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.481+0000 7feeb1ffb640 1 -- 172.21.15.27:0/1166390038 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24117 (secure 0 0 0) 0x7feea809b840 con 0x7feeb4101890 2024-04-25T00:13:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:31.483 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:13:23.935081+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:13:31.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.484+0000 7feebaf5a640 1 -- 172.21.15.27:0/1166390038 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fee9c0a2b90 msgr2=0x7fee9c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.484+0000 7feebaf5a640 1 --2- 172.21.15.27:0/1166390038 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fee9c0a2b90 0x7fee9c0a5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7feeb4199110 tx=0x7feea4008040 comp rx=0 tx=0).stop 2024-04-25T00:13:31.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.484+0000 7feebaf5a640 1 -- 172.21.15.27:0/1166390038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb4101890 msgr2=0x7feeb41981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.484+0000 7feebaf5a640 1 --2- 172.21.15.27:0/1166390038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb4101890 0x7feeb41981f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7feea800b4a0 tx=0x7feea800b960 comp rx=0 tx=0).stop 2024-04-25T00:13:31.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.484+0000 7feebaf5a640 1 -- 172.21.15.27:0/1166390038 shutdown_connections 2024-04-25T00:13:31.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.484+0000 7feebaf5a640 1 --2- 172.21.15.27:0/1166390038 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fee9c0a2b90 0x7fee9c0a5050 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.484+0000 7feebaf5a640 1 --2- 172.21.15.27:0/1166390038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feeb4102250 0x7feeb419f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.484+0000 7feebaf5a640 1 --2- 172.21.15.27:0/1166390038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb4101890 0x7feeb41981f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.484+0000 7feebaf5a640 1 --2- 172.21.15.27:0/1166390038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feeb41006a0 0x7feeb4197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.484+0000 7feebaf5a640 1 -- 172.21.15.27:0/1166390038 >> 172.21.15.27:0/1166390038 conn(0x7feeb40fbc30 msgr2=0x7feeb40fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:31.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.484+0000 7feebaf5a640 1 -- 172.21.15.27:0/1166390038 shutdown_connections 2024-04-25T00:13:31.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.484+0000 7feebaf5a640 1 -- 172.21.15.27:0/1166390038 wait complete. 2024-04-25T00:13:31.496 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:08:34.006973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T00:13:17.337307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T00:13:18.253713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T00:13:21.995987+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T00:13:31.497 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:13:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.601+0000 7feb4a672640 1 -- 172.21.15.27:0/2164407382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb44101870 msgr2=0x7feb44101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.601+0000 7feb4a672640 1 --2- 172.21.15.27:0/2164407382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb44101870 0x7feb44101cf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7feb38009900 tx=0x7feb3802a410 comp rx=0 tx=0).stop 2024-04-25T00:13:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.601+0000 7feb4a672640 1 -- 172.21.15.27:0/2164407382 shutdown_connections 2024-04-25T00:13:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.601+0000 7feb4a672640 1 --2- 172.21.15.27:0/2164407382 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb44102230 0x7feb44108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.601+0000 7feb4a672640 1 --2- 172.21.15.27:0/2164407382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb44101870 0x7feb44101cf0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.601+0000 7feb4a672640 1 --2- 172.21.15.27:0/2164407382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb44100680 0x7feb44100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.601+0000 7feb4a672640 1 -- 172.21.15.27:0/2164407382 >> 172.21.15.27:0/2164407382 conn(0x7feb440fbbf0 msgr2=0x7feb440fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.601+0000 7feb4a672640 1 -- 172.21.15.27:0/2164407382 shutdown_connections 2024-04-25T00:13:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.601+0000 7feb4a672640 1 -- 172.21.15.27:0/2164407382 wait complete. 2024-04-25T00:13:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb4a672640 1 Processor -- start 2024-04-25T00:13:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb4a672640 1 -- start start 2024-04-25T00:13:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb4a672640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb44100680 0x7feb44196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb4a672640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb44101870 0x7feb44196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb4a672640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb44102230 0x7feb4419dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb4a672640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feb4410c350 con 0x7feb44101870 2024-04-25T00:13:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb4a672640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feb4410c1d0 con 0x7feb44100680 2024-04-25T00:13:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb4a672640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feb4410c4d0 con 0x7feb44102230 2024-04-25T00:13:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb437fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb44101870 0x7feb44196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb437fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb44101870 0x7feb44196f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44140/0 (socket says 172.21.15.27:44140) 2024-04-25T00:13:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb437fe640 1 -- 172.21.15.27:0/1415071148 learned_addr learned my addr 172.21.15.27:0/1415071148 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb43fff640 1 --2- 172.21.15.27:0/1415071148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb44100680 0x7feb44196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb48c0f640 1 --2- 172.21.15.27:0/1415071148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb44102230 0x7feb4419dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb437fe640 1 -- 172.21.15.27:0/1415071148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb44102230 msgr2=0x7feb4419dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb437fe640 1 --2- 172.21.15.27:0/1415071148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb44102230 0x7feb4419dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb437fe640 1 -- 172.21.15.27:0/1415071148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb44100680 msgr2=0x7feb44196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb437fe640 1 --2- 172.21.15.27:0/1415071148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb44100680 0x7feb44196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb437fe640 1 -- 172.21.15.27:0/1415071148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4419e630 con 0x7feb44101870 2024-04-25T00:13:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.602+0000 7feb43fff640 1 --2- 172.21.15.27:0/1415071148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb44100680 0x7feb44196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:13:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.603+0000 7feb437fe640 1 --2- 172.21.15.27:0/1415071148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb44101870 0x7feb44196f50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7feb38031440 tx=0x7feb38032e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.603+0000 7feb417fa640 1 -- 172.21.15.27:0/1415071148 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3803f030 con 0x7feb44101870 2024-04-25T00:13:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.603+0000 7feb417fa640 1 -- 172.21.15.27:0/1415071148 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb38003ce0 con 0x7feb44101870 2024-04-25T00:13:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.603+0000 7feb4a672640 1 -- 172.21.15.27:0/1415071148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb4419e8c0 con 0x7feb44101870 2024-04-25T00:13:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.603+0000 7feb417fa640 1 -- 172.21.15.27:0/1415071148 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb38035070 con 0x7feb44101870 2024-04-25T00:13:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.603+0000 7feb4a672640 1 -- 172.21.15.27:0/1415071148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb4419edb0 con 0x7feb44101870 2024-04-25T00:13:31.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.604+0000 7feb4a672640 1 -- 172.21.15.27:0/1415071148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb441079a0 con 0x7feb44101870 2024-04-25T00:13:31.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.605+0000 7feb417fa640 1 -- 172.21.15.27:0/1415071148 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7feb38042030 con 0x7feb44101870 2024-04-25T00:13:31.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.606+0000 7feb417fa640 1 --2- 172.21.15.27:0/1415071148 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7feb2c0a2b90 0x7feb2c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.606+0000 7feb43fff640 1 --2- 172.21.15.27:0/1415071148 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7feb2c0a2b90 0x7feb2c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.606+0000 7feb417fa640 1 -- 172.21.15.27:0/1415071148 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7feb380ef9e0 con 0x7feb44101870 2024-04-25T00:13:31.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.606+0000 7feb43fff640 1 --2- 172.21.15.27:0/1415071148 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7feb2c0a2b90 0x7feb2c0a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7feb340058c0 tx=0x7feb34005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:31.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.611+0000 7feb417fa640 1 -- 172.21.15.27:0/1415071148 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7feb380bc700 con 0x7feb44101870 2024-04-25T00:13:31.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.792+0000 7feb4a672640 1 -- 172.21.15.27:0/1415071148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7feb44062340 con 0x7feb44101870 2024-04-25T00:13:31.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.793+0000 7feb417fa640 1 -- 172.21.15.27:0/1415071148 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7feb380dc0c0 con 0x7feb44101870 2024-04-25T00:13:31.794 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T00:13:31.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.795+0000 7feb4a672640 1 -- 172.21.15.27:0/1415071148 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7feb2c0a2b90 msgr2=0x7feb2c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.795+0000 7feb4a672640 1 --2- 172.21.15.27:0/1415071148 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7feb2c0a2b90 0x7feb2c0a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7feb340058c0 tx=0x7feb34005850 comp rx=0 tx=0).stop 2024-04-25T00:13:31.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.795+0000 7feb4a672640 1 -- 172.21.15.27:0/1415071148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb44101870 msgr2=0x7feb44196f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.795+0000 7feb4a672640 1 --2- 172.21.15.27:0/1415071148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb44101870 0x7feb44196f50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7feb38031440 tx=0x7feb38032e20 comp rx=0 tx=0).stop 2024-04-25T00:13:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.795+0000 7feb4a672640 1 -- 172.21.15.27:0/1415071148 shutdown_connections 2024-04-25T00:13:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.795+0000 7feb4a672640 1 --2- 172.21.15.27:0/1415071148 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7feb2c0a2b90 0x7feb2c0a5050 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.795+0000 7feb4a672640 1 --2- 172.21.15.27:0/1415071148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb44102230 0x7feb4419dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.795+0000 7feb4a672640 1 --2- 172.21.15.27:0/1415071148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb44101870 0x7feb44196f50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.795+0000 7feb4a672640 1 --2- 172.21.15.27:0/1415071148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb44100680 0x7feb44196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.795+0000 7feb4a672640 1 -- 172.21.15.27:0/1415071148 >> 172.21.15.27:0/1415071148 conn(0x7feb440fbbf0 msgr2=0x7feb44104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.795+0000 7feb4a672640 1 -- 172.21.15.27:0/1415071148 shutdown_connections 2024-04-25T00:13:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.795+0000 7feb4a672640 1 -- 172.21.15.27:0/1415071148 wait complete. 2024-04-25T00:13:31.806 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:31 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1166390038' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:31.807 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T00:13:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.910+0000 7f9b85cd6640 1 -- 172.21.15.27:0/2602917280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b80102990 msgr2=0x7f9b80102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.910+0000 7f9b85cd6640 1 --2- 172.21.15.27:0/2602917280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b80102990 0x7f9b80102e10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9b74009900 tx=0x7f9b7402a410 comp rx=0 tx=0).stop 2024-04-25T00:13:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.911+0000 7f9b85cd6640 1 -- 172.21.15.27:0/2602917280 shutdown_connections 2024-04-25T00:13:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.911+0000 7f9b85cd6640 1 --2- 172.21.15.27:0/2602917280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b80103350 0x7f9b80109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.911+0000 7f9b85cd6640 1 --2- 172.21.15.27:0/2602917280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b80102990 0x7f9b80102e10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.911+0000 7f9b85cd6640 1 --2- 172.21.15.27:0/2602917280 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b801017a0 0x7f9b80101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.911+0000 7f9b85cd6640 1 -- 172.21.15.27:0/2602917280 >> 172.21.15.27:0/2602917280 conn(0x7f9b800fcd10 msgr2=0x7f9b800ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.911+0000 7f9b85cd6640 1 -- 172.21.15.27:0/2602917280 shutdown_connections 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.911+0000 7f9b85cd6640 1 -- 172.21.15.27:0/2602917280 wait complete. 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.911+0000 7f9b85cd6640 1 Processor -- start 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.911+0000 7f9b85cd6640 1 -- start start 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.911+0000 7f9b85cd6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b801017a0 0x7f9b800754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.911+0000 7f9b85cd6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b80102990 0x7f9b80073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.911+0000 7f9b85cd6640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b80103350 0x7f9b80074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.911+0000 7f9b85cd6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8010d420 con 0x7f9b801017a0 2024-04-25T00:13:31.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.911+0000 7f9b85cd6640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8010d2a0 con 0x7f9b80103350 2024-04-25T00:13:31.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.911+0000 7f9b85cd6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8010d5a0 con 0x7f9b80102990 2024-04-25T00:13:31.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.912+0000 7f9b7f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b801017a0 0x7f9b800754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.912+0000 7f9b7f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b801017a0 0x7f9b800754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44168/0 (socket says 172.21.15.27:44168) 2024-04-25T00:13:31.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.912+0000 7f9b7f7fe640 1 -- 172.21.15.27:0/3103900491 learned_addr learned my addr 172.21.15.27:0/3103900491 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:31.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.912+0000 7f9b7effd640 1 --2- 172.21.15.27:0/3103900491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b80102990 0x7f9b80073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.912+0000 7f9b7ffff640 1 --2- 172.21.15.27:0/3103900491 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b80103350 0x7f9b80074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.912+0000 7f9b7f7fe640 1 -- 172.21.15.27:0/3103900491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b80102990 msgr2=0x7f9b80073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.912+0000 7f9b7f7fe640 1 --2- 172.21.15.27:0/3103900491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b80102990 0x7f9b80073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.912+0000 7f9b7f7fe640 1 -- 172.21.15.27:0/3103900491 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b80103350 msgr2=0x7f9b80074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.912+0000 7f9b7f7fe640 1 --2- 172.21.15.27:0/3103900491 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b80103350 0x7f9b80074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.912+0000 7f9b7f7fe640 1 -- 172.21.15.27:0/3103900491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b80074900 con 0x7f9b801017a0 2024-04-25T00:13:31.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.912+0000 7f9b7effd640 1 --2- 172.21.15.27:0/3103900491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b80102990 0x7f9b80073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:13:31.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.912+0000 7f9b7f7fe640 1 --2- 172.21.15.27:0/3103900491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b801017a0 0x7f9b800754f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9b70009fd0 tx=0x7f9b7000c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:31.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.912+0000 7f9b7cff9640 1 -- 172.21.15.27:0/3103900491 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b70011470 con 0x7f9b801017a0 2024-04-25T00:13:31.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.912+0000 7f9b85cd6640 1 -- 172.21.15.27:0/3103900491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b801a1bc0 con 0x7f9b801017a0 2024-04-25T00:13:31.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.912+0000 7f9b7cff9640 1 -- 172.21.15.27:0/3103900491 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b70011af0 con 0x7f9b801017a0 2024-04-25T00:13:31.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.912+0000 7f9b7cff9640 1 -- 172.21.15.27:0/3103900491 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7001ab90 con 0x7f9b801017a0 2024-04-25T00:13:31.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.912+0000 7f9b85cd6640 1 -- 172.21.15.27:0/3103900491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b801a2130 con 0x7f9b801017a0 2024-04-25T00:13:31.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.913+0000 7f9b85cd6640 1 -- 172.21.15.27:0/3103900491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b80108ae0 con 0x7f9b801017a0 2024-04-25T00:13:31.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.915+0000 7f9b7cff9640 1 -- 172.21.15.27:0/3103900491 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9b70011610 con 0x7f9b801017a0 2024-04-25T00:13:31.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.916+0000 7f9b7cff9640 1 --2- 172.21.15.27:0/3103900491 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9b680a2b90 0x7f9b680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.916+0000 7f9b7effd640 1 --2- 172.21.15.27:0/3103900491 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9b680a2b90 0x7f9b680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.916+0000 7f9b7cff9640 1 -- 172.21.15.27:0/3103900491 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9b700cfc70 con 0x7f9b801017a0 2024-04-25T00:13:31.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.916+0000 7f9b7effd640 1 --2- 172.21.15.27:0/3103900491 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9b680a2b90 0x7f9b680a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9b74000c00 tx=0x7f9b740095f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:31.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:31.920+0000 7f9b7cff9640 1 -- 172.21.15.27:0/3103900491 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9b70016030 con 0x7f9b801017a0 2024-04-25T00:13:32.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:31 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1166390038' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:32.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:31 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1166390038' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:32.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.102+0000 7f9b85cd6640 1 -- 172.21.15.27:0/3103900491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9b8002d170 con 0x7f9b801017a0 2024-04-25T00:13:32.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.103+0000 7f9b7cff9640 1 -- 172.21.15.27:0/3103900491 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f9b7009ca20 con 0x7f9b801017a0 2024-04-25T00:13:32.104 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:13:32.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.104+0000 7f9b85cd6640 1 -- 172.21.15.27:0/3103900491 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9b680a2b90 msgr2=0x7f9b680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:32.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.104+0000 7f9b85cd6640 1 --2- 172.21.15.27:0/3103900491 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9b680a2b90 0x7f9b680a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9b74000c00 tx=0x7f9b740095f0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.104+0000 7f9b85cd6640 1 -- 172.21.15.27:0/3103900491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b801017a0 msgr2=0x7f9b800754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:32.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.104+0000 7f9b85cd6640 1 --2- 172.21.15.27:0/3103900491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b801017a0 0x7f9b800754f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9b70009fd0 tx=0x7f9b7000c270 comp rx=0 tx=0).stop 2024-04-25T00:13:32.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.105+0000 7f9b85cd6640 1 -- 172.21.15.27:0/3103900491 shutdown_connections 2024-04-25T00:13:32.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.105+0000 7f9b85cd6640 1 --2- 172.21.15.27:0/3103900491 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9b680a2b90 0x7f9b680a5050 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.105+0000 7f9b85cd6640 1 --2- 172.21.15.27:0/3103900491 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b80103350 0x7f9b80074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.105+0000 7f9b85cd6640 1 --2- 172.21.15.27:0/3103900491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b80102990 0x7f9b80073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.105+0000 7f9b85cd6640 1 --2- 172.21.15.27:0/3103900491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b801017a0 0x7f9b800754f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.105+0000 7f9b85cd6640 1 -- 172.21.15.27:0/3103900491 >> 172.21.15.27:0/3103900491 conn(0x7f9b800fcd10 msgr2=0x7f9b80105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:32.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.105+0000 7f9b85cd6640 1 -- 172.21.15.27:0/3103900491 shutdown_connections 2024-04-25T00:13:32.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.105+0000 7f9b85cd6640 1 -- 172.21.15.27:0/3103900491 wait complete. 2024-04-25T00:13:32.117 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T00:13:32.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.221+0000 7fd87a6bc640 1 -- 172.21.15.27:0/1052210255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8741006a0 msgr2=0x7fd874100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:32.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.221+0000 7fd87a6bc640 1 --2- 172.21.15.27:0/1052210255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8741006a0 0x7fd874100ac0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd864009900 tx=0x7fd86402a410 comp rx=0 tx=0).stop 2024-04-25T00:13:32.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.221+0000 7fd87a6bc640 1 -- 172.21.15.27:0/1052210255 shutdown_connections 2024-04-25T00:13:32.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.221+0000 7fd87a6bc640 1 --2- 172.21.15.27:0/1052210255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd874102250 0x7fd874108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.221+0000 7fd87a6bc640 1 --2- 172.21.15.27:0/1052210255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd874101890 0x7fd874101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.221+0000 7fd87a6bc640 1 --2- 172.21.15.27:0/1052210255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8741006a0 0x7fd874100ac0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.221+0000 7fd87a6bc640 1 -- 172.21.15.27:0/1052210255 >> 172.21.15.27:0/1052210255 conn(0x7fd8740fbc30 msgr2=0x7fd8740fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:32.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.221+0000 7fd87a6bc640 1 -- 172.21.15.27:0/1052210255 shutdown_connections 2024-04-25T00:13:32.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.221+0000 7fd87a6bc640 1 -- 172.21.15.27:0/1052210255 wait complete. 2024-04-25T00:13:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.222+0000 7fd87a6bc640 1 Processor -- start 2024-04-25T00:13:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.222+0000 7fd87a6bc640 1 -- start start 2024-04-25T00:13:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.222+0000 7fd87a6bc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8741006a0 0x7fd87406a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.222+0000 7fd87a6bc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd874101890 0x7fd87419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.222+0000 7fd87a6bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd874102250 0x7fd87419ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.222+0000 7fd87a6bc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd87410c320 con 0x7fd874102250 2024-04-25T00:13:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.222+0000 7fd87a6bc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd87410c1a0 con 0x7fd8741006a0 2024-04-25T00:13:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.222+0000 7fd87a6bc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd87410c4a0 con 0x7fd874101890 2024-04-25T00:13:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.222+0000 7fd878c59640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd874102250 0x7fd87419ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.222+0000 7fd878c59640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd874102250 0x7fd87419ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44172/0 (socket says 172.21.15.27:44172) 2024-04-25T00:13:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.222+0000 7fd878c59640 1 -- 172.21.15.27:0/2507656525 learned_addr learned my addr 172.21.15.27:0/2507656525 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.222+0000 7fd873fff640 1 --2- 172.21.15.27:0/2507656525 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8741006a0 0x7fd87406a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.222+0000 7fd8737fe640 1 --2- 172.21.15.27:0/2507656525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd874101890 0x7fd87419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.223+0000 7fd878c59640 1 -- 172.21.15.27:0/2507656525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd874101890 msgr2=0x7fd87419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.223+0000 7fd878c59640 1 --2- 172.21.15.27:0/2507656525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd874101890 0x7fd87419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.223+0000 7fd878c59640 1 -- 172.21.15.27:0/2507656525 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8741006a0 msgr2=0x7fd87406a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.223+0000 7fd878c59640 1 --2- 172.21.15.27:0/2507656525 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8741006a0 0x7fd87406a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.223+0000 7fd878c59640 1 -- 172.21.15.27:0/2507656525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87419f7a0 con 0x7fd874102250 2024-04-25T00:13:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.223+0000 7fd8737fe640 1 --2- 172.21.15.27:0/2507656525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd874101890 0x7fd87419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:13:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.223+0000 7fd878c59640 1 --2- 172.21.15.27:0/2507656525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd874102250 0x7fd87419ee90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00eb10 tx=0x7fd86c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.223+0000 7fd8717fa640 1 -- 172.21.15.27:0/2507656525 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c004b20 con 0x7fd874102250 2024-04-25T00:13:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.223+0000 7fd87a6bc640 1 -- 172.21.15.27:0/2507656525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd87419fa30 con 0x7fd874102250 2024-04-25T00:13:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.223+0000 7fd8717fa640 1 -- 172.21.15.27:0/2507656525 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd86c004cc0 con 0x7fd874102250 2024-04-25T00:13:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.223+0000 7fd8717fa640 1 -- 172.21.15.27:0/2507656525 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c010760 con 0x7fd874102250 2024-04-25T00:13:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.223+0000 7fd87a6bc640 1 -- 172.21.15.27:0/2507656525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd87419ff80 con 0x7fd874102250 2024-04-25T00:13:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.224+0000 7fd87a6bc640 1 -- 172.21.15.27:0/2507656525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd874107ae0 con 0x7fd874102250 2024-04-25T00:13:32.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.225+0000 7fd8717fa640 1 -- 172.21.15.27:0/2507656525 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd86c010900 con 0x7fd874102250 2024-04-25T00:13:32.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.226+0000 7fd8717fa640 1 --2- 172.21.15.27:0/2507656525 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd85c0a2b90 0x7fd85c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:32.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.226+0000 7fd873fff640 1 --2- 172.21.15.27:0/2507656525 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd85c0a2b90 0x7fd85c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:32.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.226+0000 7fd8717fa640 1 -- 172.21.15.27:0/2507656525 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd86c0cfab0 con 0x7fd874102250 2024-04-25T00:13:32.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.226+0000 7fd873fff640 1 --2- 172.21.15.27:0/2507656525 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd85c0a2b90 0x7fd85c0a5050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd864005ab0 tx=0x7fd864000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:32.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.229+0000 7fd8717fa640 1 -- 172.21.15.27:0/2507656525 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd86c053740 con 0x7fd874102250 2024-04-25T00:13:32.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.417+0000 7fd87a6bc640 1 -- 172.21.15.27:0/2507656525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd87402d370 con 0x7fd874102250 2024-04-25T00:13:32.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.417+0000 7fd8717fa640 1 -- 172.21.15.27:0/2507656525 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fd86c09c7d0 con 0x7fd874102250 2024-04-25T00:13:32.419 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T00:13:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.420+0000 7fd87a6bc640 1 -- 172.21.15.27:0/2507656525 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd85c0a2b90 msgr2=0x7fd85c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.420+0000 7fd87a6bc640 1 --2- 172.21.15.27:0/2507656525 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd85c0a2b90 0x7fd85c0a5050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd864005ab0 tx=0x7fd864000bc0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.420+0000 7fd87a6bc640 1 -- 172.21.15.27:0/2507656525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd874102250 msgr2=0x7fd87419ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.420+0000 7fd87a6bc640 1 --2- 172.21.15.27:0/2507656525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd874102250 0x7fd87419ee90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00eb10 tx=0x7fd86c00c2d0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.420+0000 7fd87a6bc640 1 -- 172.21.15.27:0/2507656525 shutdown_connections 2024-04-25T00:13:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.420+0000 7fd87a6bc640 1 --2- 172.21.15.27:0/2507656525 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd85c0a2b90 0x7fd85c0a5050 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.420+0000 7fd87a6bc640 1 --2- 172.21.15.27:0/2507656525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd874102250 0x7fd87419ee90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.420+0000 7fd87a6bc640 1 --2- 172.21.15.27:0/2507656525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd874101890 0x7fd87419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.420+0000 7fd87a6bc640 1 --2- 172.21.15.27:0/2507656525 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8741006a0 0x7fd87406a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.420+0000 7fd87a6bc640 1 -- 172.21.15.27:0/2507656525 >> 172.21.15.27:0/2507656525 conn(0x7fd8740fbc30 msgr2=0x7fd874104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.420+0000 7fd87a6bc640 1 -- 172.21.15.27:0/2507656525 shutdown_connections 2024-04-25T00:13:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.420+0000 7fd87a6bc640 1 -- 172.21.15.27:0/2507656525 wait complete. 2024-04-25T00:13:32.432 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T00:13:32.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.535+0000 7fc2acb29640 1 -- 172.21.15.27:0/1903411307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a8102250 msgr2=0x7fc2a8108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:32.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.535+0000 7fc2acb29640 1 --2- 172.21.15.27:0/1903411307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a8102250 0x7fc2a8108ae0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc298009ca0 tx=0x7fc29802a8c0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.535+0000 7fc2acb29640 1 -- 172.21.15.27:0/1903411307 shutdown_connections 2024-04-25T00:13:32.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.535+0000 7fc2acb29640 1 --2- 172.21.15.27:0/1903411307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a8102250 0x7fc2a8108ae0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.535+0000 7fc2acb29640 1 --2- 172.21.15.27:0/1903411307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2a8101890 0x7fc2a8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.535+0000 7fc2acb29640 1 --2- 172.21.15.27:0/1903411307 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2a81006a0 0x7fc2a8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.535+0000 7fc2acb29640 1 -- 172.21.15.27:0/1903411307 >> 172.21.15.27:0/1903411307 conn(0x7fc2a80fbc30 msgr2=0x7fc2a80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:32.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.535+0000 7fc2acb29640 1 -- 172.21.15.27:0/1903411307 shutdown_connections 2024-04-25T00:13:32.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.535+0000 7fc2acb29640 1 -- 172.21.15.27:0/1903411307 wait complete. 2024-04-25T00:13:32.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.536+0000 7fc2acb29640 1 Processor -- start 2024-04-25T00:13:32.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.536+0000 7fc2acb29640 1 -- start start 2024-04-25T00:13:32.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.536+0000 7fc2acb29640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2a81006a0 0x7fc2a8197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:32.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.536+0000 7fc2acb29640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2a8101890 0x7fc2a8197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:32.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.536+0000 7fc2acb29640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a8102250 0x7fc2a819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:32.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.536+0000 7fc2acb29640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a810d3f0 con 0x7fc2a8102250 2024-04-25T00:13:32.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.536+0000 7fc2acb29640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a810d270 con 0x7fc2a8101890 2024-04-25T00:13:32.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.536+0000 7fc2acb29640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a810d570 con 0x7fc2a81006a0 2024-04-25T00:13:32.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.536+0000 7fc2a6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a8102250 0x7fc2a819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.537+0000 7fc2a6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a8102250 0x7fc2a819efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44186/0 (socket says 172.21.15.27:44186) 2024-04-25T00:13:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.537+0000 7fc2a6d9d640 1 -- 172.21.15.27:0/728750913 learned_addr learned my addr 172.21.15.27:0/728750913 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.537+0000 7fc2a659c640 1 --2- 172.21.15.27:0/728750913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2a81006a0 0x7fc2a8197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.537+0000 7fc2a5d9b640 1 --2- 172.21.15.27:0/728750913 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2a8101890 0x7fc2a8197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.537+0000 7fc2a6d9d640 1 -- 172.21.15.27:0/728750913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2a81006a0 msgr2=0x7fc2a8197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.537+0000 7fc2a6d9d640 1 --2- 172.21.15.27:0/728750913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2a81006a0 0x7fc2a8197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.537+0000 7fc2a6d9d640 1 -- 172.21.15.27:0/728750913 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2a8101890 msgr2=0x7fc2a8197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.537+0000 7fc2a6d9d640 1 --2- 172.21.15.27:0/728750913 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2a8101890 0x7fc2a8197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.537+0000 7fc2a6d9d640 1 -- 172.21.15.27:0/728750913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a819f630 con 0x7fc2a8102250 2024-04-25T00:13:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.537+0000 7fc2a6d9d640 1 --2- 172.21.15.27:0/728750913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a8102250 0x7fc2a819efc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc298005e30 tx=0x7fc298031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.537+0000 7fc2977fe640 1 -- 172.21.15.27:0/728750913 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc298003c30 con 0x7fc2a8102250 2024-04-25T00:13:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.537+0000 7fc2acb29640 1 -- 172.21.15.27:0/728750913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2a819f8c0 con 0x7fc2a8102250 2024-04-25T00:13:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.537+0000 7fc2977fe640 1 -- 172.21.15.27:0/728750913 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc298003dd0 con 0x7fc2a8102250 2024-04-25T00:13:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.537+0000 7fc2977fe640 1 -- 172.21.15.27:0/728750913 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29802fde0 con 0x7fc2a8102250 2024-04-25T00:13:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.537+0000 7fc2acb29640 1 -- 172.21.15.27:0/728750913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2a819fe60 con 0x7fc2a8102250 2024-04-25T00:13:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.538+0000 7fc2acb29640 1 -- 172.21.15.27:0/728750913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2a81079c0 con 0x7fc2a8102250 2024-04-25T00:13:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.539+0000 7fc2977fe640 1 -- 172.21.15.27:0/728750913 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc298004050 con 0x7fc2a8102250 2024-04-25T00:13:32.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.540+0000 7fc2977fe640 1 --2- 172.21.15.27:0/728750913 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc2900a2b90 0x7fc2900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:32.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.540+0000 7fc2a659c640 1 --2- 172.21.15.27:0/728750913 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc2900a2b90 0x7fc2900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:32.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.540+0000 7fc2977fe640 1 -- 172.21.15.27:0/728750913 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc2980f0430 con 0x7fc2a8102250 2024-04-25T00:13:32.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.541+0000 7fc2a659c640 1 --2- 172.21.15.27:0/728750913 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc2900a2b90 0x7fc2900a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0005fd0 tx=0x7fc2a000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:32.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.544+0000 7fc2977fe640 1 -- 172.21.15.27:0/728750913 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc2980bd150 con 0x7fc2a8102250 2024-04-25T00:13:32.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:32 smithi027 ceph-mon[29208]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-04-25T00:13:32.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:32 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1415071148' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:13:32.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:32 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3103900491' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:13:32.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:32 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2507656525' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:13:32.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.728+0000 7fc2acb29640 1 -- 172.21.15.27:0/728750913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc2a8062340 con 0x7fc2a8102250 2024-04-25T00:13:32.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.728+0000 7fc2977fe640 1 -- 172.21.15.27:0/728750913 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7fc2980bca50 con 0x7fc2a8102250 2024-04-25T00:13:32.729 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:13:32.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.730+0000 7fc2acb29640 1 -- 172.21.15.27:0/728750913 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc2900a2b90 msgr2=0x7fc2900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:32.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.730+0000 7fc2acb29640 1 --2- 172.21.15.27:0/728750913 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc2900a2b90 0x7fc2900a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0005fd0 tx=0x7fc2a000b410 comp rx=0 tx=0).stop 2024-04-25T00:13:32.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.730+0000 7fc2acb29640 1 -- 172.21.15.27:0/728750913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a8102250 msgr2=0x7fc2a819efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:32.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.730+0000 7fc2acb29640 1 --2- 172.21.15.27:0/728750913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a8102250 0x7fc2a819efc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc298005e30 tx=0x7fc298031ee0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.730+0000 7fc2acb29640 1 -- 172.21.15.27:0/728750913 shutdown_connections 2024-04-25T00:13:32.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.730+0000 7fc2acb29640 1 --2- 172.21.15.27:0/728750913 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc2900a2b90 0x7fc2900a5050 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.730+0000 7fc2acb29640 1 --2- 172.21.15.27:0/728750913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a8102250 0x7fc2a819efc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.730+0000 7fc2acb29640 1 --2- 172.21.15.27:0/728750913 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2a8101890 0x7fc2a8197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.730+0000 7fc2acb29640 1 --2- 172.21.15.27:0/728750913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2a81006a0 0x7fc2a8197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.730+0000 7fc2acb29640 1 -- 172.21.15.27:0/728750913 >> 172.21.15.27:0/728750913 conn(0x7fc2a80fbc30 msgr2=0x7fc2a8104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:32.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.730+0000 7fc2acb29640 1 -- 172.21.15.27:0/728750913 shutdown_connections 2024-04-25T00:13:32.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.730+0000 7fc2acb29640 1 -- 172.21.15.27:0/728750913 wait complete. 2024-04-25T00:13:32.743 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T00:13:32.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:32 smithi153 ceph-mon[32896]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-04-25T00:13:32.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:32 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1415071148' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:13:32.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:32 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3103900491' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:13:32.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:32 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2507656525' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:13:32.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.845+0000 7f3ff61b7640 1 -- 172.21.15.27:0/4187374271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ff01017a0 msgr2=0x7f3ff0101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:32.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.845+0000 7f3ff61b7640 1 --2- 172.21.15.27:0/4187374271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ff01017a0 0x7f3ff0101bc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8009900 tx=0x7f3fd802a410 comp rx=0 tx=0).stop 2024-04-25T00:13:32.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.845+0000 7f3ff61b7640 1 -- 172.21.15.27:0/4187374271 shutdown_connections 2024-04-25T00:13:32.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.845+0000 7f3ff61b7640 1 --2- 172.21.15.27:0/4187374271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ff0103350 0x7f3ff0109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.845+0000 7f3ff61b7640 1 --2- 172.21.15.27:0/4187374271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ff0102990 0x7f3ff0102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.845+0000 7f3ff61b7640 1 --2- 172.21.15.27:0/4187374271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ff01017a0 0x7f3ff0101bc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.845+0000 7f3ff61b7640 1 -- 172.21.15.27:0/4187374271 >> 172.21.15.27:0/4187374271 conn(0x7f3ff00fcd10 msgr2=0x7f3ff00ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:32.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.845+0000 7f3ff61b7640 1 -- 172.21.15.27:0/4187374271 shutdown_connections 2024-04-25T00:13:32.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.845+0000 7f3ff61b7640 1 -- 172.21.15.27:0/4187374271 wait complete. 2024-04-25T00:13:32.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.846+0000 7f3ff61b7640 1 Processor -- start 2024-04-25T00:13:32.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.846+0000 7f3ff61b7640 1 -- start start 2024-04-25T00:13:32.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.846+0000 7f3ff61b7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ff01017a0 0x7f3ff0197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:32.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.846+0000 7f3ff61b7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ff0102990 0x7f3ff0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:32.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.846+0000 7f3ff61b7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ff0103350 0x7f3ff019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.846+0000 7f3ff61b7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff010d3f0 con 0x7f3ff01017a0 2024-04-25T00:13:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.846+0000 7f3ff61b7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff010d270 con 0x7f3ff0103350 2024-04-25T00:13:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.846+0000 7f3ff61b7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff010d570 con 0x7f3ff0102990 2024-04-25T00:13:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.846+0000 7f3fef7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ff01017a0 0x7f3ff0197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.846+0000 7f3fef7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ff01017a0 0x7f3ff0197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44204/0 (socket says 172.21.15.27:44204) 2024-04-25T00:13:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.846+0000 7f3fef7fe640 1 -- 172.21.15.27:0/2411373989 learned_addr learned my addr 172.21.15.27:0/2411373989 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.846+0000 7f3feffff640 1 --2- 172.21.15.27:0/2411373989 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ff0103350 0x7f3ff019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.847+0000 7f3feeffd640 1 --2- 172.21.15.27:0/2411373989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ff0102990 0x7f3ff0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.847+0000 7f3fef7fe640 1 -- 172.21.15.27:0/2411373989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ff0102990 msgr2=0x7f3ff0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.847+0000 7f3fef7fe640 1 --2- 172.21.15.27:0/2411373989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ff0102990 0x7f3ff0197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.847+0000 7f3fef7fe640 1 -- 172.21.15.27:0/2411373989 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ff0103350 msgr2=0x7f3ff019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.847+0000 7f3fef7fe640 1 --2- 172.21.15.27:0/2411373989 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ff0103350 0x7f3ff019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.847+0000 7f3fef7fe640 1 -- 172.21.15.27:0/2411373989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ff019f6d0 con 0x7f3ff01017a0 2024-04-25T00:13:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.847+0000 7f3fef7fe640 1 --2- 172.21.15.27:0/2411373989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ff01017a0 0x7f3ff0197ab0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8031610 tx=0x7f3fd80043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.847+0000 7f3fecff9640 1 -- 172.21.15.27:0/2411373989 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd803f030 con 0x7f3ff01017a0 2024-04-25T00:13:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.847+0000 7f3ff61b7640 1 -- 172.21.15.27:0/2411373989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ff019f960 con 0x7f3ff01017a0 2024-04-25T00:13:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.847+0000 7f3fecff9640 1 -- 172.21.15.27:0/2411373989 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fd8003ce0 con 0x7f3ff01017a0 2024-04-25T00:13:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.847+0000 7f3fecff9640 1 -- 172.21.15.27:0/2411373989 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8035070 con 0x7f3ff01017a0 2024-04-25T00:13:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.847+0000 7f3ff61b7640 1 -- 172.21.15.27:0/2411373989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ff019fea0 con 0x7f3ff01017a0 2024-04-25T00:13:32.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.848+0000 7f3ff61b7640 1 -- 172.21.15.27:0/2411373989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ff0108b90 con 0x7f3ff01017a0 2024-04-25T00:13:32.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.849+0000 7f3fecff9640 1 -- 172.21.15.27:0/2411373989 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3fd8042030 con 0x7f3ff01017a0 2024-04-25T00:13:32.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.850+0000 7f3fecff9640 1 --2- 172.21.15.27:0/2411373989 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3fdc0a2b90 0x7f3fdc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:32.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.850+0000 7f3feeffd640 1 --2- 172.21.15.27:0/2411373989 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3fdc0a2b90 0x7f3fdc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:32.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.850+0000 7f3fecff9640 1 -- 172.21.15.27:0/2411373989 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3fd80ef820 con 0x7f3ff01017a0 2024-04-25T00:13:32.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.850+0000 7f3feeffd640 1 --2- 172.21.15.27:0/2411373989 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3fdc0a2b90 0x7f3fdc0a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0005fd0 tx=0x7f3fe0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:32.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:32.854+0000 7f3fecff9640 1 -- 172.21.15.27:0/2411373989 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3fd80bc540 con 0x7f3ff01017a0 2024-04-25T00:13:33.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:32 smithi139 ceph-mon[32833]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-04-25T00:13:33.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:32 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1415071148' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:13:33.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:32 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3103900491' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:13:33.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:32 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2507656525' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:13:33.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.051+0000 7f3ff61b7640 1 -- 172.21.15.27:0/2411373989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3ff002d170 con 0x7f3ff01017a0 2024-04-25T00:13:33.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.052+0000 7f3fecff9640 1 -- 172.21.15.27:0/2411373989 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+591 (secure 0 0 0) 0x7f3fd80bbe40 con 0x7f3ff01017a0 2024-04-25T00:13:33.053 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3721121684 2024-04-26T00:06:09.484973+0000 2024-04-25T00:13:33.053 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/1186428290 2024-04-26T00:06:09.484973+0000 2024-04-25T00:13:33.053 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2012441698 2024-04-26T00:06:09.484973+0000 2024-04-25T00:13:33.053 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/3082151239 2024-04-26T00:05:42.617192+0000 2024-04-25T00:13:33.053 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/3082151239 2024-04-26T00:05:42.617192+0000 2024-04-25T00:13:33.053 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/1186428290 2024-04-26T00:06:09.484973+0000 2024-04-25T00:13:33.053 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2172615181 2024-04-26T00:05:42.617192+0000 2024-04-25T00:13:33.053 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/316040314 2024-04-26T00:05:42.617192+0000 2024-04-25T00:13:33.053 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1671770035 2024-04-26T00:05:42.617192+0000 2024-04-25T00:13:33.053 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1035759944 2024-04-26T00:06:09.484973+0000 2024-04-25T00:13:33.054 INFO:teuthology.orchestra.run.smithi027.stderr:listed 10 entries 2024-04-25T00:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.053+0000 7f3ff61b7640 1 -- 172.21.15.27:0/2411373989 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3fdc0a2b90 msgr2=0x7f3fdc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.053+0000 7f3ff61b7640 1 --2- 172.21.15.27:0/2411373989 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3fdc0a2b90 0x7f3fdc0a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0005fd0 tx=0x7f3fe0005850 comp rx=0 tx=0).stop 2024-04-25T00:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.053+0000 7f3ff61b7640 1 -- 172.21.15.27:0/2411373989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ff01017a0 msgr2=0x7f3ff0197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.053+0000 7f3ff61b7640 1 --2- 172.21.15.27:0/2411373989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ff01017a0 0x7f3ff0197ab0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8031610 tx=0x7f3fd80043b0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.054+0000 7f3ff61b7640 1 -- 172.21.15.27:0/2411373989 shutdown_connections 2024-04-25T00:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.054+0000 7f3ff61b7640 1 --2- 172.21.15.27:0/2411373989 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3fdc0a2b90 0x7f3fdc0a5050 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.054+0000 7f3ff61b7640 1 --2- 172.21.15.27:0/2411373989 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ff0103350 0x7f3ff019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.054+0000 7f3ff61b7640 1 --2- 172.21.15.27:0/2411373989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ff0102990 0x7f3ff0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.054+0000 7f3ff61b7640 1 --2- 172.21.15.27:0/2411373989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ff01017a0 0x7f3ff0197ab0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.054+0000 7f3ff61b7640 1 -- 172.21.15.27:0/2411373989 >> 172.21.15.27:0/2411373989 conn(0x7f3ff00fcd10 msgr2=0x7f3ff0105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.054+0000 7f3ff61b7640 1 -- 172.21.15.27:0/2411373989 shutdown_connections 2024-04-25T00:13:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.054+0000 7f3ff61b7640 1 -- 172.21.15.27:0/2411373989 wait complete. 2024-04-25T00:13:33.065 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:13:33.065 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T00:13:33.081 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:13:33.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.222+0000 7f2bd291d640 1 -- 172.21.15.27:0/2662846798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc102230 msgr2=0x7f2bcc108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:33.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.222+0000 7f2bd291d640 1 --2- 172.21.15.27:0/2662846798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc102230 0x7f2bcc108ac0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4009ca0 tx=0x7f2bc402a8c0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.222+0000 7f2bd291d640 1 -- 172.21.15.27:0/2662846798 shutdown_connections 2024-04-25T00:13:33.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.222+0000 7f2bd291d640 1 --2- 172.21.15.27:0/2662846798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc102230 0x7f2bcc108ac0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.222+0000 7f2bd291d640 1 --2- 172.21.15.27:0/2662846798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bcc101870 0x7f2bcc101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.222+0000 7f2bd291d640 1 --2- 172.21.15.27:0/2662846798 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2bcc100680 0x7f2bcc100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.222+0000 7f2bd291d640 1 -- 172.21.15.27:0/2662846798 >> 172.21.15.27:0/2662846798 conn(0x7f2bcc0fbbf0 msgr2=0x7f2bcc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:33.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.222+0000 7f2bd291d640 1 -- 172.21.15.27:0/2662846798 shutdown_connections 2024-04-25T00:13:33.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.222+0000 7f2bd291d640 1 -- 172.21.15.27:0/2662846798 wait complete. 2024-04-25T00:13:33.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.222+0000 7f2bd291d640 1 Processor -- start 2024-04-25T00:13:33.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.222+0000 7f2bd291d640 1 -- start start 2024-04-25T00:13:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bd291d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2bcc100680 0x7f2bcc197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bd291d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bcc101870 0x7f2bcc197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bd291d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc102230 0x7f2bcc19eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bd291d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc10c2e0 con 0x7f2bcc102230 2024-04-25T00:13:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bd291d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc10c160 con 0x7f2bcc100680 2024-04-25T00:13:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bd291d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc10c460 con 0x7f2bcc101870 2024-04-25T00:13:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bd0eba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc102230 0x7f2bcc19eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bd0eba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc102230 0x7f2bcc19eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44216/0 (socket says 172.21.15.27:44216) 2024-04-25T00:13:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bd0eba640 1 -- 172.21.15.27:0/387251547 learned_addr learned my addr 172.21.15.27:0/387251547 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bcbfff640 1 --2- 172.21.15.27:0/387251547 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2bcc100680 0x7f2bcc197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bd0eba640 1 -- 172.21.15.27:0/387251547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bcc101870 msgr2=0x7f2bcc197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bd0eba640 1 --2- 172.21.15.27:0/387251547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bcc101870 0x7f2bcc197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bd0eba640 1 -- 172.21.15.27:0/387251547 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2bcc100680 msgr2=0x7f2bcc197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bd0eba640 1 --2- 172.21.15.27:0/387251547 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2bcc100680 0x7f2bcc197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bd0eba640 1 -- 172.21.15.27:0/387251547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bcc19f550 con 0x7f2bcc102230 2024-04-25T00:13:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bcbfff640 1 --2- 172.21.15.27:0/387251547 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2bcc100680 0x7f2bcc197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bd0eba640 1 --2- 172.21.15.27:0/387251547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc102230 0x7f2bcc19eee0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2bc40098d0 tx=0x7f2bc402f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bc97fa640 1 -- 172.21.15.27:0/387251547 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc402fc30 con 0x7f2bcc102230 2024-04-25T00:13:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bd291d640 1 -- 172.21.15.27:0/387251547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bcc19f7e0 con 0x7f2bcc102230 2024-04-25T00:13:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.223+0000 7f2bc97fa640 1 -- 172.21.15.27:0/387251547 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bc4003e50 con 0x7f2bcc102230 2024-04-25T00:13:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.224+0000 7f2bc97fa640 1 -- 172.21.15.27:0/387251547 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc40305c0 con 0x7f2bcc102230 2024-04-25T00:13:33.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.224+0000 7f2bd291d640 1 -- 172.21.15.27:0/387251547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bcc19fd80 con 0x7f2bcc102230 2024-04-25T00:13:33.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.225+0000 7f2bd291d640 1 -- 172.21.15.27:0/387251547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bcc1a0040 con 0x7f2bcc102230 2024-04-25T00:13:33.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.226+0000 7f2bc97fa640 1 -- 172.21.15.27:0/387251547 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2bc4003970 con 0x7f2bcc102230 2024-04-25T00:13:33.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.226+0000 7f2bc97fa640 1 --2- 172.21.15.27:0/387251547 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2bb40a2b90 0x7f2bb40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:33.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.226+0000 7f2bcbfff640 1 --2- 172.21.15.27:0/387251547 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2bb40a2b90 0x7f2bb40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:33.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.226+0000 7f2bc97fa640 1 -- 172.21.15.27:0/387251547 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2bc40f0760 con 0x7f2bcc102230 2024-04-25T00:13:33.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.227+0000 7f2bcbfff640 1 --2- 172.21.15.27:0/387251547 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2bb40a2b90 0x7f2bb40a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc1016d0 tx=0x7f2bbc005db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:33.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.231+0000 7f2bc97fa640 1 -- 172.21.15.27:0/387251547 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2bc40bd480 con 0x7f2bcc102230 2024-04-25T00:13:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.418+0000 7f2bd291d640 1 -- 172.21.15.27:0/387251547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2bcc1a0700 con 0x7f2bcc102230 2024-04-25T00:13:33.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.419+0000 7f2bc97fa640 1 -- 172.21.15.27:0/387251547 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24117 (secure 0 0 0) 0x7f2bc40bcd80 con 0x7f2bcc102230 2024-04-25T00:13:33.421 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:33.421 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:13:23.935081+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:13:33.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.422+0000 7f2bd291d640 1 -- 172.21.15.27:0/387251547 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2bb40a2b90 msgr2=0x7f2bb40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:33.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.422+0000 7f2bd291d640 1 --2- 172.21.15.27:0/387251547 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2bb40a2b90 0x7f2bb40a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc1016d0 tx=0x7f2bbc005db0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.422+0000 7f2bd291d640 1 -- 172.21.15.27:0/387251547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc102230 msgr2=0x7f2bcc19eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:33.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.422+0000 7f2bd291d640 1 --2- 172.21.15.27:0/387251547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc102230 0x7f2bcc19eee0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2bc40098d0 tx=0x7f2bc402f840 comp rx=0 tx=0).stop 2024-04-25T00:13:33.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.422+0000 7f2bd291d640 1 -- 172.21.15.27:0/387251547 shutdown_connections 2024-04-25T00:13:33.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.422+0000 7f2bd291d640 1 --2- 172.21.15.27:0/387251547 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2bb40a2b90 0x7f2bb40a5050 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.422+0000 7f2bd291d640 1 --2- 172.21.15.27:0/387251547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc102230 0x7f2bcc19eee0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.422+0000 7f2bd291d640 1 --2- 172.21.15.27:0/387251547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bcc101870 0x7f2bcc197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.422+0000 7f2bd291d640 1 --2- 172.21.15.27:0/387251547 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2bcc100680 0x7f2bcc197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.422+0000 7f2bd291d640 1 -- 172.21.15.27:0/387251547 >> 172.21.15.27:0/387251547 conn(0x7f2bcc0fbbf0 msgr2=0x7f2bcc0fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:33.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.422+0000 7f2bd291d640 1 -- 172.21.15.27:0/387251547 shutdown_connections 2024-04-25T00:13:33.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.422+0000 7f2bd291d640 1 -- 172.21.15.27:0/387251547 wait complete. 2024-04-25T00:13:33.434 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:08:34.006973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T00:13:17.337307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T00:13:18.253713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T00:13:21.995987+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T00:13:33.435 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:13:33.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.538+0000 7f72afa9a640 1 -- 172.21.15.27:0/3795139024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72a81017a0 msgr2=0x7f72a8101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:33.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.538+0000 7f72afa9a640 1 --2- 172.21.15.27:0/3795139024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72a81017a0 0x7f72a8101bc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009900 tx=0x7f72a402a410 comp rx=0 tx=0).stop 2024-04-25T00:13:33.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.539+0000 7f72afa9a640 1 -- 172.21.15.27:0/3795139024 shutdown_connections 2024-04-25T00:13:33.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.539+0000 7f72afa9a640 1 --2- 172.21.15.27:0/3795139024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a8103350 0x7f72a8109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.539+0000 7f72afa9a640 1 --2- 172.21.15.27:0/3795139024 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72a8102990 0x7f72a8102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.539+0000 7f72afa9a640 1 --2- 172.21.15.27:0/3795139024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72a81017a0 0x7f72a8101bc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.539+0000 7f72afa9a640 1 -- 172.21.15.27:0/3795139024 >> 172.21.15.27:0/3795139024 conn(0x7f72a80fcd10 msgr2=0x7f72a80ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:33.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.539+0000 7f72afa9a640 1 -- 172.21.15.27:0/3795139024 shutdown_connections 2024-04-25T00:13:33.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.539+0000 7f72afa9a640 1 -- 172.21.15.27:0/3795139024 wait complete. 2024-04-25T00:13:33.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.539+0000 7f72afa9a640 1 Processor -- start 2024-04-25T00:13:33.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.539+0000 7f72afa9a640 1 -- start start 2024-04-25T00:13:33.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.539+0000 7f72afa9a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a81017a0 0x7f72a8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:33.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.539+0000 7f72afa9a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72a8102990 0x7f72a8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:33.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.539+0000 7f72afa9a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72a8103350 0x7f72a819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:33.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.539+0000 7f72afa9a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f72a810d420 con 0x7f72a8102990 2024-04-25T00:13:33.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.539+0000 7f72afa9a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f72a810d2a0 con 0x7f72a8103350 2024-04-25T00:13:33.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.539+0000 7f72afa9a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f72a810d5a0 con 0x7f72a81017a0 2024-04-25T00:13:33.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f72ad035640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72a8102990 0x7f72a8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:33.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f72ad035640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72a8102990 0x7f72a8198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44228/0 (socket says 172.21.15.27:44228) 2024-04-25T00:13:33.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f72ad035640 1 -- 172.21.15.27:0/3731570622 learned_addr learned my addr 172.21.15.27:0/3731570622 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f72ad836640 1 --2- 172.21.15.27:0/3731570622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a81017a0 0x7f72a8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f72ae037640 1 --2- 172.21.15.27:0/3731570622 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72a8103350 0x7f72a819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f72ad035640 1 -- 172.21.15.27:0/3731570622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a81017a0 msgr2=0x7f72a8197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f72ad035640 1 --2- 172.21.15.27:0/3731570622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a81017a0 0x7f72a8197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f72ad035640 1 -- 172.21.15.27:0/3731570622 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72a8103350 msgr2=0x7f72a819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f72ad035640 1 --2- 172.21.15.27:0/3731570622 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72a8103350 0x7f72a819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f72ad035640 1 -- 172.21.15.27:0/3731570622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a819f6e0 con 0x7f72a8102990 2024-04-25T00:13:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f72ae037640 1 --2- 172.21.15.27:0/3731570622 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72a8103350 0x7f72a819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f72ad836640 1 --2- 172.21.15.27:0/3731570622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a81017a0 0x7f72a8197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:13:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f72ad035640 1 --2- 172.21.15.27:0/3731570622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72a8102990 0x7f72a8198000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f729800e9f0 tx=0x7f729800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f729effd640 1 -- 172.21.15.27:0/3731570622 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729800cd80 con 0x7f72a8102990 2024-04-25T00:13:33.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f72afa9a640 1 -- 172.21.15.27:0/3731570622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72a819f9d0 con 0x7f72a8102990 2024-04-25T00:13:33.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f729effd640 1 -- 172.21.15.27:0/3731570622 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7298010910 con 0x7f72a8102990 2024-04-25T00:13:33.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f729effd640 1 -- 172.21.15.27:0/3731570622 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72980188f0 con 0x7f72a8102990 2024-04-25T00:13:33.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.540+0000 7f72afa9a640 1 -- 172.21.15.27:0/3731570622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72a819ff70 con 0x7f72a8102990 2024-04-25T00:13:33.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.541+0000 7f72afa9a640 1 -- 172.21.15.27:0/3731570622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72a8108ac0 con 0x7f72a8102990 2024-04-25T00:13:33.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.542+0000 7f729effd640 1 -- 172.21.15.27:0/3731570622 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7298010ac0 con 0x7f72a8102990 2024-04-25T00:13:33.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.543+0000 7f729effd640 1 --2- 172.21.15.27:0/3731570622 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f72940a2b90 0x7f72940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:33.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.543+0000 7f72ad836640 1 --2- 172.21.15.27:0/3731570622 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f72940a2b90 0x7f72940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:33.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.543+0000 7f729effd640 1 -- 172.21.15.27:0/3731570622 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f72980cf770 con 0x7f72a8102990 2024-04-25T00:13:33.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.544+0000 7f72ad836640 1 --2- 172.21.15.27:0/3731570622 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f72940a2b90 0x7f72940a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f72a40314e0 tx=0x7f72a4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:33.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.548+0000 7f729effd640 1 -- 172.21.15.27:0/3731570622 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f729809c490 con 0x7f72a8102990 2024-04-25T00:13:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:33 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/728750913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:13:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:33 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2411373989' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T00:13:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:33 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/387251547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:33.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.733+0000 7f72afa9a640 1 -- 172.21.15.27:0/3731570622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f72a8062340 con 0x7f72a8102990 2024-04-25T00:13:33.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.733+0000 7f729effd640 1 -- 172.21.15.27:0/3731570622 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f729809bd90 con 0x7f72a8102990 2024-04-25T00:13:33.734 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T00:13:33.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.735+0000 7f72afa9a640 1 -- 172.21.15.27:0/3731570622 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f72940a2b90 msgr2=0x7f72940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:33.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.735+0000 7f72afa9a640 1 --2- 172.21.15.27:0/3731570622 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f72940a2b90 0x7f72940a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f72a40314e0 tx=0x7f72a4005fd0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.735+0000 7f72afa9a640 1 -- 172.21.15.27:0/3731570622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72a8102990 msgr2=0x7f72a8198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:33.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.735+0000 7f72afa9a640 1 --2- 172.21.15.27:0/3731570622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72a8102990 0x7f72a8198000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f729800e9f0 tx=0x7f729800eeb0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.735+0000 7f72afa9a640 1 -- 172.21.15.27:0/3731570622 shutdown_connections 2024-04-25T00:13:33.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.735+0000 7f72afa9a640 1 --2- 172.21.15.27:0/3731570622 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f72940a2b90 0x7f72940a5050 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.735+0000 7f72afa9a640 1 --2- 172.21.15.27:0/3731570622 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72a8103350 0x7f72a819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.735+0000 7f72afa9a640 1 --2- 172.21.15.27:0/3731570622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72a8102990 0x7f72a8198000 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.735+0000 7f72afa9a640 1 --2- 172.21.15.27:0/3731570622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a81017a0 0x7f72a8197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.735+0000 7f72afa9a640 1 -- 172.21.15.27:0/3731570622 >> 172.21.15.27:0/3731570622 conn(0x7f72a80fcd10 msgr2=0x7f72a8105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:33.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.736+0000 7f72afa9a640 1 -- 172.21.15.27:0/3731570622 shutdown_connections 2024-04-25T00:13:33.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.736+0000 7f72afa9a640 1 -- 172.21.15.27:0/3731570622 wait complete. 2024-04-25T00:13:33.747 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T00:13:33.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:33 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/728750913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:13:33.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:33 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2411373989' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T00:13:33.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:33 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/387251547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.851+0000 7f7071590640 1 -- 172.21.15.27:0/1933528693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f706c101890 msgr2=0x7f706c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.851+0000 7f7071590640 1 --2- 172.21.15.27:0/1933528693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f706c101890 0x7f706c101d10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7060009900 tx=0x7f706002a410 comp rx=0 tx=0).stop 2024-04-25T00:13:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.851+0000 7f7071590640 1 -- 172.21.15.27:0/1933528693 shutdown_connections 2024-04-25T00:13:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.851+0000 7f7071590640 1 --2- 172.21.15.27:0/1933528693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f706c102250 0x7f706c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.851+0000 7f7071590640 1 --2- 172.21.15.27:0/1933528693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f706c101890 0x7f706c101d10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.851+0000 7f7071590640 1 --2- 172.21.15.27:0/1933528693 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f706c1006a0 0x7f706c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.851+0000 7f7071590640 1 -- 172.21.15.27:0/1933528693 >> 172.21.15.27:0/1933528693 conn(0x7f706c0fbc10 msgr2=0x7f706c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.851+0000 7f7071590640 1 -- 172.21.15.27:0/1933528693 shutdown_connections 2024-04-25T00:13:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.851+0000 7f7071590640 1 -- 172.21.15.27:0/1933528693 wait complete. 2024-04-25T00:13:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.852+0000 7f7071590640 1 Processor -- start 2024-04-25T00:13:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.852+0000 7f7071590640 1 -- start start 2024-04-25T00:13:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.852+0000 7f7071590640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f706c1006a0 0x7f706c06a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.852+0000 7f7071590640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f706c101890 0x7f706c19ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.852+0000 7f7071590640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f706c102250 0x7f706c1a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.852+0000 7f7071590640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f706c10c380 con 0x7f706c102250 2024-04-25T00:13:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.852+0000 7f7071590640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f706c10c200 con 0x7f706c1006a0 2024-04-25T00:13:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.852+0000 7f7071590640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f706c10c500 con 0x7f706c101890 2024-04-25T00:13:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.852+0000 7f706b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f706c102250 0x7f706c1a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.852+0000 7f706b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f706c102250 0x7f706c1a11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44236/0 (socket says 172.21.15.27:44236) 2024-04-25T00:13:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.852+0000 7f706b7fe640 1 -- 172.21.15.27:0/3299384958 learned_addr learned my addr 172.21.15.27:0/3299384958 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.852+0000 7f706a7fc640 1 --2- 172.21.15.27:0/3299384958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f706c101890 0x7f706c19ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.853+0000 7f706affd640 1 --2- 172.21.15.27:0/3299384958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f706c1006a0 0x7f706c06a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.853+0000 7f706b7fe640 1 -- 172.21.15.27:0/3299384958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f706c101890 msgr2=0x7f706c19ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.853+0000 7f706b7fe640 1 --2- 172.21.15.27:0/3299384958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f706c101890 0x7f706c19ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.853+0000 7f706b7fe640 1 -- 172.21.15.27:0/3299384958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f706c1006a0 msgr2=0x7f706c06a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.853+0000 7f706b7fe640 1 --2- 172.21.15.27:0/3299384958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f706c1006a0 0x7f706c06a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.853+0000 7f706b7fe640 1 -- 172.21.15.27:0/3299384958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f706c1a19c0 con 0x7f706c102250 2024-04-25T00:13:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.853+0000 7f706b7fe640 1 --2- 172.21.15.27:0/3299384958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f706c102250 0x7f706c1a11d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f705c00b970 tx=0x7f705c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.853+0000 7f7053fff640 1 -- 172.21.15.27:0/3299384958 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705c019070 con 0x7f706c102250 2024-04-25T00:13:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.853+0000 7f7071590640 1 -- 172.21.15.27:0/3299384958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f706c1a1c50 con 0x7f706c102250 2024-04-25T00:13:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.853+0000 7f7053fff640 1 -- 172.21.15.27:0/3299384958 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f705c00cb40 con 0x7f706c102250 2024-04-25T00:13:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.853+0000 7f7053fff640 1 -- 172.21.15.27:0/3299384958 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705c007650 con 0x7f706c102250 2024-04-25T00:13:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.853+0000 7f7071590640 1 -- 172.21.15.27:0/3299384958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f706c1a21f0 con 0x7f706c102250 2024-04-25T00:13:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.854+0000 7f7071590640 1 -- 172.21.15.27:0/3299384958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f706c1079c0 con 0x7f706c102250 2024-04-25T00:13:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.855+0000 7f7053fff640 1 -- 172.21.15.27:0/3299384958 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f705c01c070 con 0x7f706c102250 2024-04-25T00:13:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.856+0000 7f7053fff640 1 --2- 172.21.15.27:0/3299384958 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f70540a2bc0 0x7f70540a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.856+0000 7f706affd640 1 --2- 172.21.15.27:0/3299384958 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f70540a2bc0 0x7f70540a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.856+0000 7f7053fff640 1 -- 172.21.15.27:0/3299384958 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f705c0ce7d0 con 0x7f706c102250 2024-04-25T00:13:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.856+0000 7f706affd640 1 --2- 172.21.15.27:0/3299384958 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f70540a2bc0 0x7f70540a5080 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f706c1016f0 tx=0x7f706400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:33.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:33.860+0000 7f7053fff640 1 -- 172.21.15.27:0/3299384958 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f705c09aea0 con 0x7f706c102250 2024-04-25T00:13:34.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:33 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/728750913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:13:34.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:33 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2411373989' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T00:13:34.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:33 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/387251547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:34.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.043+0000 7f7071590640 1 -- 172.21.15.27:0/3299384958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f706c062340 con 0x7f706c102250 2024-04-25T00:13:34.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.044+0000 7f7053fff640 1 -- 172.21.15.27:0/3299384958 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f705c09f1a0 con 0x7f706c102250 2024-04-25T00:13:34.052 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:13:34.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.046+0000 7f7071590640 1 -- 172.21.15.27:0/3299384958 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f70540a2bc0 msgr2=0x7f70540a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.046+0000 7f7071590640 1 --2- 172.21.15.27:0/3299384958 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f70540a2bc0 0x7f70540a5080 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f706c1016f0 tx=0x7f706400b390 comp rx=0 tx=0).stop 2024-04-25T00:13:34.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.046+0000 7f7071590640 1 -- 172.21.15.27:0/3299384958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f706c102250 msgr2=0x7f706c1a11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.046+0000 7f7071590640 1 --2- 172.21.15.27:0/3299384958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f706c102250 0x7f706c1a11d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f705c00b970 tx=0x7f705c00be30 comp rx=0 tx=0).stop 2024-04-25T00:13:34.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.046+0000 7f7071590640 1 -- 172.21.15.27:0/3299384958 shutdown_connections 2024-04-25T00:13:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.046+0000 7f7071590640 1 --2- 172.21.15.27:0/3299384958 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f70540a2bc0 0x7f70540a5080 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.046+0000 7f7071590640 1 --2- 172.21.15.27:0/3299384958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f706c102250 0x7f706c1a11d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.046+0000 7f7071590640 1 --2- 172.21.15.27:0/3299384958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f706c101890 0x7f706c19ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.046+0000 7f7071590640 1 --2- 172.21.15.27:0/3299384958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f706c1006a0 0x7f706c06a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.046+0000 7f7071590640 1 -- 172.21.15.27:0/3299384958 >> 172.21.15.27:0/3299384958 conn(0x7f706c0fbc10 msgr2=0x7f706c104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.046+0000 7f7071590640 1 -- 172.21.15.27:0/3299384958 shutdown_connections 2024-04-25T00:13:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.046+0000 7f7071590640 1 -- 172.21.15.27:0/3299384958 wait complete. 2024-04-25T00:13:34.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T00:13:34.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.174+0000 7fe5671f9640 1 -- 172.21.15.27:0/280847249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe560102250 msgr2=0x7fe560108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.174+0000 7fe5671f9640 1 --2- 172.21.15.27:0/280847249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe560102250 0x7fe560108ae0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe558009ca0 tx=0x7fe55802a8c0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.174+0000 7fe5671f9640 1 -- 172.21.15.27:0/280847249 shutdown_connections 2024-04-25T00:13:34.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.174+0000 7fe5671f9640 1 --2- 172.21.15.27:0/280847249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe560102250 0x7fe560108ae0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.174+0000 7fe5671f9640 1 --2- 172.21.15.27:0/280847249 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe560101890 0x7fe560101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.174+0000 7fe5671f9640 1 --2- 172.21.15.27:0/280847249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5601006a0 0x7fe560100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.174+0000 7fe5671f9640 1 -- 172.21.15.27:0/280847249 >> 172.21.15.27:0/280847249 conn(0x7fe5600fbc30 msgr2=0x7fe5600fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.174+0000 7fe5671f9640 1 -- 172.21.15.27:0/280847249 shutdown_connections 2024-04-25T00:13:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.174+0000 7fe5671f9640 1 -- 172.21.15.27:0/280847249 wait complete. 2024-04-25T00:13:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe5671f9640 1 Processor -- start 2024-04-25T00:13:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe5671f9640 1 -- start start 2024-04-25T00:13:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe5671f9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5601006a0 0x7fe560196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe5671f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe560101890 0x7fe560196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe5671f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe560102250 0x7fe56019df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe5671f9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe56010c320 con 0x7fe560102250 2024-04-25T00:13:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe5671f9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe56010c1a0 con 0x7fe5601006a0 2024-04-25T00:13:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe5671f9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe56010c4a0 con 0x7fe560101890 2024-04-25T00:13:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe564f95640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5601006a0 0x7fe560196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe564f95640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5601006a0 0x7fe560196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:35536/0 (socket says 172.21.15.27:35536) 2024-04-25T00:13:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe564f95640 1 -- 172.21.15.27:0/671364389 learned_addr learned my addr 172.21.15.27:0/671364389 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe565796640 1 --2- 172.21.15.27:0/671364389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe560102250 0x7fe56019df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe55ffff640 1 --2- 172.21.15.27:0/671364389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe560101890 0x7fe560196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe565796640 1 -- 172.21.15.27:0/671364389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe560101890 msgr2=0x7fe560196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe565796640 1 --2- 172.21.15.27:0/671364389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe560101890 0x7fe560196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe565796640 1 -- 172.21.15.27:0/671364389 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5601006a0 msgr2=0x7fe560196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe565796640 1 --2- 172.21.15.27:0/671364389 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5601006a0 0x7fe560196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.175+0000 7fe565796640 1 -- 172.21.15.27:0/671364389 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe56019e5b0 con 0x7fe560102250 2024-04-25T00:13:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.176+0000 7fe565796640 1 --2- 172.21.15.27:0/671364389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe560102250 0x7fe56019df40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe558031ae0 tx=0x7fe558031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.176+0000 7fe55dffb640 1 -- 172.21.15.27:0/671364389 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe558049070 con 0x7fe560102250 2024-04-25T00:13:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.176+0000 7fe55dffb640 1 -- 172.21.15.27:0/671364389 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe558003c40 con 0x7fe560102250 2024-04-25T00:13:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.176+0000 7fe5671f9640 1 -- 172.21.15.27:0/671364389 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe560069d40 con 0x7fe560102250 2024-04-25T00:13:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.176+0000 7fe55dffb640 1 -- 172.21.15.27:0/671364389 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55802fae0 con 0x7fe560102250 2024-04-25T00:13:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.176+0000 7fe5671f9640 1 -- 172.21.15.27:0/671364389 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe56006a260 con 0x7fe560102250 2024-04-25T00:13:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.177+0000 7fe5671f9640 1 -- 172.21.15.27:0/671364389 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5600689b0 con 0x7fe560102250 2024-04-25T00:13:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.178+0000 7fe55dffb640 1 -- 172.21.15.27:0/671364389 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe55804c070 con 0x7fe560102250 2024-04-25T00:13:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.179+0000 7fe55dffb640 1 --2- 172.21.15.27:0/671364389 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe5480a2b90 0x7fe5480a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.179+0000 7fe55dffb640 1 -- 172.21.15.27:0/671364389 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe5580f1a50 con 0x7fe560102250 2024-04-25T00:13:34.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.179+0000 7fe564f95640 1 --2- 172.21.15.27:0/671364389 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe5480a2b90 0x7fe5480a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.179+0000 7fe564f95640 1 --2- 172.21.15.27:0/671364389 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe5480a2b90 0x7fe5480a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe550005fd0 tx=0x7fe55000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:34.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.182+0000 7fe55dffb640 1 -- 172.21.15.27:0/671364389 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe558080740 con 0x7fe560102250 2024-04-25T00:13:34.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.367+0000 7fe5671f9640 1 -- 172.21.15.27:0/671364389 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe560107640 con 0x7fe560102250 2024-04-25T00:13:34.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.368+0000 7fe55dffb640 1 -- 172.21.15.27:0/671364389 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fe5580be120 con 0x7fe560102250 2024-04-25T00:13:34.369 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T00:13:34.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.370+0000 7fe5671f9640 1 -- 172.21.15.27:0/671364389 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe5480a2b90 msgr2=0x7fe5480a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.370+0000 7fe5671f9640 1 --2- 172.21.15.27:0/671364389 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe5480a2b90 0x7fe5480a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe550005fd0 tx=0x7fe55000b410 comp rx=0 tx=0).stop 2024-04-25T00:13:34.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.370+0000 7fe5671f9640 1 -- 172.21.15.27:0/671364389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe560102250 msgr2=0x7fe56019df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.370+0000 7fe5671f9640 1 --2- 172.21.15.27:0/671364389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe560102250 0x7fe56019df40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe558031ae0 tx=0x7fe558031ee0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.370+0000 7fe5671f9640 1 -- 172.21.15.27:0/671364389 shutdown_connections 2024-04-25T00:13:34.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.370+0000 7fe5671f9640 1 --2- 172.21.15.27:0/671364389 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe5480a2b90 0x7fe5480a5050 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.370+0000 7fe5671f9640 1 --2- 172.21.15.27:0/671364389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe560102250 0x7fe56019df40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.370+0000 7fe5671f9640 1 --2- 172.21.15.27:0/671364389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe560101890 0x7fe560196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.370+0000 7fe5671f9640 1 --2- 172.21.15.27:0/671364389 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5601006a0 0x7fe560196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.370+0000 7fe5671f9640 1 -- 172.21.15.27:0/671364389 >> 172.21.15.27:0/671364389 conn(0x7fe5600fbc30 msgr2=0x7fe560104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:34.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.370+0000 7fe5671f9640 1 -- 172.21.15.27:0/671364389 shutdown_connections 2024-04-25T00:13:34.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.370+0000 7fe5671f9640 1 -- 172.21.15.27:0/671364389 wait complete. 2024-04-25T00:13:34.382 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T00:13:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.487+0000 7f6a4af29640 1 -- 172.21.15.27:0/3333673066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a44102da0 msgr2=0x7f6a441031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.487+0000 7f6a4af29640 1 --2- 172.21.15.27:0/3333673066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a44102da0 0x7f6a441031c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009900 tx=0x7f6a3c02a410 comp rx=0 tx=0).stop 2024-04-25T00:13:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.487+0000 7f6a4af29640 1 -- 172.21.15.27:0/3333673066 shutdown_connections 2024-04-25T00:13:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.487+0000 7f6a4af29640 1 --2- 172.21.15.27:0/3333673066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a441001d0 0x7f6a44109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.487+0000 7f6a4af29640 1 --2- 172.21.15.27:0/3333673066 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a440ff810 0x7f6a440ffc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.487+0000 7f6a4af29640 1 --2- 172.21.15.27:0/3333673066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a44102da0 0x7f6a441031c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.487+0000 7f6a4af29640 1 -- 172.21.15.27:0/3333673066 >> 172.21.15.27:0/3333673066 conn(0x7f6a440fb370 msgr2=0x7f6a440fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.487+0000 7f6a4af29640 1 -- 172.21.15.27:0/3333673066 shutdown_connections 2024-04-25T00:13:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.487+0000 7f6a4af29640 1 -- 172.21.15.27:0/3333673066 wait complete. 2024-04-25T00:13:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a4af29640 1 Processor -- start 2024-04-25T00:13:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a4af29640 1 -- start start 2024-04-25T00:13:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a4af29640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a440ff810 0x7f6a44197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a4af29640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a441001d0 0x7f6a44197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a4af29640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a44102da0 0x7f6a4419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a4af29640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4410d420 con 0x7f6a441001d0 2024-04-25T00:13:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a4af29640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4410d2a0 con 0x7f6a440ff810 2024-04-25T00:13:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a4af29640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4410d5a0 con 0x7f6a44102da0 2024-04-25T00:13:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a43fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a441001d0 0x7f6a44197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a43fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a441001d0 0x7f6a44197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44280/0 (socket says 172.21.15.27:44280) 2024-04-25T00:13:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a43fff640 1 -- 172.21.15.27:0/3375374105 learned_addr learned my addr 172.21.15.27:0/3375374105 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a494c6640 1 --2- 172.21.15.27:0/3375374105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a44102da0 0x7f6a4419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a48cc5640 1 --2- 172.21.15.27:0/3375374105 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a440ff810 0x7f6a44197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a43fff640 1 -- 172.21.15.27:0/3375374105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a44102da0 msgr2=0x7f6a4419f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a43fff640 1 --2- 172.21.15.27:0/3375374105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a44102da0 0x7f6a4419f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a43fff640 1 -- 172.21.15.27:0/3375374105 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a440ff810 msgr2=0x7f6a44197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a43fff640 1 --2- 172.21.15.27:0/3375374105 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a440ff810 0x7f6a44197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.488+0000 7f6a43fff640 1 -- 172.21.15.27:0/3375374105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4419f6b0 con 0x7f6a441001d0 2024-04-25T00:13:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.489+0000 7f6a43fff640 1 --2- 172.21.15.27:0/3375374105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a441001d0 0x7f6a44197fd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400ebc0 tx=0x7f6a3400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.489+0000 7f6a41ffb640 1 -- 172.21.15.27:0/3375374105 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34010630 con 0x7f6a441001d0 2024-04-25T00:13:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.489+0000 7f6a41ffb640 1 -- 172.21.15.27:0/3375374105 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a34010cb0 con 0x7f6a441001d0 2024-04-25T00:13:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.489+0000 7f6a4af29640 1 -- 172.21.15.27:0/3375374105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a4419f9a0 con 0x7f6a441001d0 2024-04-25T00:13:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.489+0000 7f6a41ffb640 1 -- 172.21.15.27:0/3375374105 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34018bd0 con 0x7f6a441001d0 2024-04-25T00:13:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.489+0000 7f6a4af29640 1 -- 172.21.15.27:0/3375374105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a4419ff40 con 0x7f6a441001d0 2024-04-25T00:13:34.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.490+0000 7f6a4af29640 1 -- 172.21.15.27:0/3375374105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a44108c80 con 0x7f6a441001d0 2024-04-25T00:13:34.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.491+0000 7f6a41ffb640 1 -- 172.21.15.27:0/3375374105 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6a340107d0 con 0x7f6a441001d0 2024-04-25T00:13:34.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.492+0000 7f6a41ffb640 1 --2- 172.21.15.27:0/3375374105 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6a300a2c10 0x7f6a300a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.492+0000 7f6a48cc5640 1 --2- 172.21.15.27:0/3375374105 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6a300a2c10 0x7f6a300a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.492+0000 7f6a41ffb640 1 -- 172.21.15.27:0/3375374105 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6a340cf8d0 con 0x7f6a441001d0 2024-04-25T00:13:34.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.492+0000 7f6a48cc5640 1 --2- 172.21.15.27:0/3375374105 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6a300a2c10 0x7f6a300a50d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c006010 tx=0x7f6a3c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.496+0000 7f6a41ffb640 1 -- 172.21.15.27:0/3375374105 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6a3409c5f0 con 0x7f6a441001d0 2024-04-25T00:13:34.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.681+0000 7f6a4af29640 1 -- 172.21.15.27:0/3375374105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6a44062340 con 0x7f6a441001d0 2024-04-25T00:13:34.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.681+0000 7f6a41ffb640 1 -- 172.21.15.27:0/3375374105 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f6a3409bef0 con 0x7f6a441001d0 2024-04-25T00:13:34.683 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:13:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.683+0000 7f6a4af29640 1 -- 172.21.15.27:0/3375374105 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6a300a2c10 msgr2=0x7f6a300a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.683+0000 7f6a4af29640 1 --2- 172.21.15.27:0/3375374105 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6a300a2c10 0x7f6a300a50d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c006010 tx=0x7f6a3c000bc0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.683+0000 7f6a4af29640 1 -- 172.21.15.27:0/3375374105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a441001d0 msgr2=0x7f6a44197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.683+0000 7f6a4af29640 1 --2- 172.21.15.27:0/3375374105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a441001d0 0x7f6a44197fd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400ebc0 tx=0x7f6a3400c6a0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.683+0000 7f6a4af29640 1 -- 172.21.15.27:0/3375374105 shutdown_connections 2024-04-25T00:13:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.683+0000 7f6a4af29640 1 --2- 172.21.15.27:0/3375374105 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6a300a2c10 0x7f6a300a50d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.683+0000 7f6a4af29640 1 --2- 172.21.15.27:0/3375374105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a44102da0 0x7f6a4419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.683+0000 7f6a4af29640 1 --2- 172.21.15.27:0/3375374105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a441001d0 0x7f6a44197fd0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.683+0000 7f6a4af29640 1 --2- 172.21.15.27:0/3375374105 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a440ff810 0x7f6a44197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.683+0000 7f6a4af29640 1 -- 172.21.15.27:0/3375374105 >> 172.21.15.27:0/3375374105 conn(0x7f6a440fb370 msgr2=0x7f6a44105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.683+0000 7f6a4af29640 1 -- 172.21.15.27:0/3375374105 shutdown_connections 2024-04-25T00:13:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.684+0000 7f6a4af29640 1 -- 172.21.15.27:0/3375374105 wait complete. 2024-04-25T00:13:34.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:34 smithi027 ceph-mon[29208]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-25T00:13:34.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:34 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3731570622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:13:34.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:34 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3299384958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:13:34.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:34 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/671364389' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:13:34.695 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T00:13:34.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.800+0000 7f537789a640 1 -- 172.21.15.27:0/3322022543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53701017a0 msgr2=0x7f5370101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.800+0000 7f537789a640 1 --2- 172.21.15.27:0/3322022543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53701017a0 0x7f5370101bc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f536c009900 tx=0x7f536c02a410 comp rx=0 tx=0).stop 2024-04-25T00:13:34.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.800+0000 7f537789a640 1 -- 172.21.15.27:0/3322022543 shutdown_connections 2024-04-25T00:13:34.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.800+0000 7f537789a640 1 --2- 172.21.15.27:0/3322022543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5370103350 0x7f5370109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.800+0000 7f537789a640 1 --2- 172.21.15.27:0/3322022543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5370102990 0x7f5370102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.800+0000 7f537789a640 1 --2- 172.21.15.27:0/3322022543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53701017a0 0x7f5370101bc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.800+0000 7f537789a640 1 -- 172.21.15.27:0/3322022543 >> 172.21.15.27:0/3322022543 conn(0x7f53700fcd10 msgr2=0x7f53700ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:34.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.800+0000 7f537789a640 1 -- 172.21.15.27:0/3322022543 shutdown_connections 2024-04-25T00:13:34.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.800+0000 7f537789a640 1 -- 172.21.15.27:0/3322022543 wait complete. 2024-04-25T00:13:34.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.801+0000 7f537789a640 1 Processor -- start 2024-04-25T00:13:34.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.801+0000 7f537789a640 1 -- start start 2024-04-25T00:13:34.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.801+0000 7f537789a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53701017a0 0x7f5370199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.801+0000 7f537789a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5370102990 0x7f537019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.801+0000 7f537789a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5370103350 0x7f53701a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.801+0000 7f537789a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f537010d3f0 con 0x7f5370103350 2024-04-25T00:13:34.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.801+0000 7f537789a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f537010d270 con 0x7f53701017a0 2024-04-25T00:13:34.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.801+0000 7f537789a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f537010d570 con 0x7f5370102990 2024-04-25T00:13:34.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.801+0000 7f5375e37640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5370103350 0x7f53701a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.801+0000 7f5375e37640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5370103350 0x7f53701a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44302/0 (socket says 172.21.15.27:44302) 2024-04-25T00:13:34.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.801+0000 7f5375e37640 1 -- 172.21.15.27:0/2857835460 learned_addr learned my addr 172.21.15.27:0/2857835460 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:34.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.801+0000 7f5375636640 1 --2- 172.21.15.27:0/2857835460 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53701017a0 0x7f5370199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.801+0000 7f5374e35640 1 --2- 172.21.15.27:0/2857835460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5370102990 0x7f537019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.802+0000 7f5375e37640 1 -- 172.21.15.27:0/2857835460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5370102990 msgr2=0x7f537019a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.802+0000 7f5375e37640 1 --2- 172.21.15.27:0/2857835460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5370102990 0x7f537019a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.802+0000 7f5375e37640 1 -- 172.21.15.27:0/2857835460 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53701017a0 msgr2=0x7f5370199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.802+0000 7f5375e37640 1 --2- 172.21.15.27:0/2857835460 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53701017a0 0x7f5370199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.802+0000 7f5375e37640 1 -- 172.21.15.27:0/2857835460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53701a18b0 con 0x7f5370103350 2024-04-25T00:13:34.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.802+0000 7f5375e37640 1 --2- 172.21.15.27:0/2857835460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5370103350 0x7f53701a1210 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f536800cce0 tx=0x7f536800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:34.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.802+0000 7f53627fc640 1 -- 172.21.15.27:0/2857835460 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536800f450 con 0x7f5370103350 2024-04-25T00:13:34.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.802+0000 7f53627fc640 1 -- 172.21.15.27:0/2857835460 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f536800fad0 con 0x7f5370103350 2024-04-25T00:13:34.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.802+0000 7f537789a640 1 -- 172.21.15.27:0/2857835460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53701a1ba0 con 0x7f5370103350 2024-04-25T00:13:34.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.802+0000 7f53627fc640 1 -- 172.21.15.27:0/2857835460 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5368018c10 con 0x7f5370103350 2024-04-25T00:13:34.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.802+0000 7f537789a640 1 -- 172.21.15.27:0/2857835460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53701a20c0 con 0x7f5370103350 2024-04-25T00:13:34.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.803+0000 7f537789a640 1 -- 172.21.15.27:0/2857835460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53701a22f0 con 0x7f5370103350 2024-04-25T00:13:34.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.805+0000 7f53627fc640 1 -- 172.21.15.27:0/2857835460 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f536800f5f0 con 0x7f5370103350 2024-04-25T00:13:34.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.805+0000 7f53627fc640 1 --2- 172.21.15.27:0/2857835460 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f535c0a2c10 0x7f535c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.805+0000 7f5375636640 1 --2- 172.21.15.27:0/2857835460 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f535c0a2c10 0x7f535c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.805+0000 7f53627fc640 1 -- 172.21.15.27:0/2857835460 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f53680cf8e0 con 0x7f5370103350 2024-04-25T00:13:34.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.806+0000 7f5375636640 1 --2- 172.21.15.27:0/2857835460 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f535c0a2c10 0x7f535c0a50d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f536c005ab0 tx=0x7f536c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:34.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:34.809+0000 7f53627fc640 1 -- 172.21.15.27:0/2857835460 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5368014030 con 0x7f5370103350 2024-04-25T00:13:35.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:34 smithi139 ceph-mon[32833]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-25T00:13:35.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:34 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3731570622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:13:35.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:34 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3299384958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:13:35.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:34 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/671364389' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:13:35.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:35.007+0000 7f537789a640 1 -- 172.21.15.27:0/2857835460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f53701a24c0 con 0x7f5370103350 2024-04-25T00:13:35.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:35.007+0000 7f53627fc640 1 -- 172.21.15.27:0/2857835460 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+591 (secure 0 0 0) 0x7f536809c710 con 0x7f5370103350 2024-04-25T00:13:35.009 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3721121684 2024-04-26T00:06:09.484973+0000 2024-04-25T00:13:35.009 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/1186428290 2024-04-26T00:06:09.484973+0000 2024-04-25T00:13:35.009 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2012441698 2024-04-26T00:06:09.484973+0000 2024-04-25T00:13:35.009 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/3082151239 2024-04-26T00:05:42.617192+0000 2024-04-25T00:13:35.009 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/3082151239 2024-04-26T00:05:42.617192+0000 2024-04-25T00:13:35.009 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/1186428290 2024-04-26T00:06:09.484973+0000 2024-04-25T00:13:35.009 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2172615181 2024-04-26T00:05:42.617192+0000 2024-04-25T00:13:35.009 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/316040314 2024-04-26T00:05:42.617192+0000 2024-04-25T00:13:35.009 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1671770035 2024-04-26T00:05:42.617192+0000 2024-04-25T00:13:35.009 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1035759944 2024-04-26T00:06:09.484973+0000 2024-04-25T00:13:35.009 INFO:teuthology.orchestra.run.smithi027.stderr:listed 10 entries 2024-04-25T00:13:35.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:35.009+0000 7f537789a640 1 -- 172.21.15.27:0/2857835460 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f535c0a2c10 msgr2=0x7f535c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:35.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:35.009+0000 7f537789a640 1 --2- 172.21.15.27:0/2857835460 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f535c0a2c10 0x7f535c0a50d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f536c005ab0 tx=0x7f536c000bc0 comp rx=0 tx=0).stop 2024-04-25T00:13:35.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:35.009+0000 7f537789a640 1 -- 172.21.15.27:0/2857835460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5370103350 msgr2=0x7f53701a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:35.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:35.009+0000 7f537789a640 1 --2- 172.21.15.27:0/2857835460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5370103350 0x7f53701a1210 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f536800cce0 tx=0x7f536800a620 comp rx=0 tx=0).stop 2024-04-25T00:13:35.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:35.009+0000 7f537789a640 1 -- 172.21.15.27:0/2857835460 shutdown_connections 2024-04-25T00:13:35.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:35.009+0000 7f537789a640 1 --2- 172.21.15.27:0/2857835460 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f535c0a2c10 0x7f535c0a50d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:35.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:35.009+0000 7f537789a640 1 --2- 172.21.15.27:0/2857835460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5370103350 0x7f53701a1210 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:35.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:35.009+0000 7f537789a640 1 --2- 172.21.15.27:0/2857835460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5370102990 0x7f537019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:35.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:35.009+0000 7f537789a640 1 --2- 172.21.15.27:0/2857835460 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53701017a0 0x7f5370199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:35.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:35.009+0000 7f537789a640 1 -- 172.21.15.27:0/2857835460 >> 172.21.15.27:0/2857835460 conn(0x7f53700fcd10 msgr2=0x7f5370073970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:35.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:35.009+0000 7f537789a640 1 -- 172.21.15.27:0/2857835460 shutdown_connections 2024-04-25T00:13:35.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:35.009+0000 7f537789a640 1 -- 172.21.15.27:0/2857835460 wait complete. 2024-04-25T00:13:35.022 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-04-25T00:13:35.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:13:35.022 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-25T00:13:35.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:34 smithi153 ceph-mon[32896]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-25T00:13:35.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:34 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3731570622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:13:35.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:34 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3299384958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:13:35.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:34 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/671364389' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:13:35.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi027.stderr:ceph-fuse[70033]: fuse finished with error 0 and tester_r 0 2024-04-25T00:13:35.049 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:13:35.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:35 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3375374105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:13:35.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:35 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2857835460' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T00:13:36.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:35 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3375374105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:13:36.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:35 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2857835460' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T00:13:36.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:35 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3375374105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:13:36.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:35 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2857835460' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T00:13:36.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:36 smithi027 ceph-mon[29208]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 279 B/s wr, 0 op/s 2024-04-25T00:13:37.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:36 smithi139 ceph-mon[32833]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 279 B/s wr, 0 op/s 2024-04-25T00:13:37.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:36 smithi153 ceph-mon[32896]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 279 B/s wr, 0 op/s 2024-04-25T00:13:38.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:38 smithi027 ceph-mon[29208]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-25T00:13:39.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:38 smithi139 ceph-mon[32833]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-25T00:13:39.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:38 smithi153 ceph-mon[32896]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-25T00:13:40.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:40 smithi027 ceph-mon[29208]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T00:13:41.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:40 smithi139 ceph-mon[32833]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T00:13:41.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:40 smithi153 ceph-mon[32896]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T00:13:41.153 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T00:13:41.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:13:41.154 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-25T00:13:41.187 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:13:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.343+0000 7fe82293b640 1 -- 172.21.15.27:0/2417549326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c1017a0 msgr2=0x7fe81c101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.343+0000 7fe82293b640 1 --2- 172.21.15.27:0/2417549326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c1017a0 0x7fe81c101bc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009900 tx=0x7fe80c02a410 comp rx=0 tx=0).stop 2024-04-25T00:13:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.343+0000 7fe82293b640 1 -- 172.21.15.27:0/2417549326 shutdown_connections 2024-04-25T00:13:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.343+0000 7fe82293b640 1 --2- 172.21.15.27:0/2417549326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe81c103350 0x7fe81c109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.343+0000 7fe82293b640 1 --2- 172.21.15.27:0/2417549326 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe81c102990 0x7fe81c102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.343+0000 7fe82293b640 1 --2- 172.21.15.27:0/2417549326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c1017a0 0x7fe81c101bc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.343+0000 7fe82293b640 1 -- 172.21.15.27:0/2417549326 >> 172.21.15.27:0/2417549326 conn(0x7fe81c0fccf0 msgr2=0x7fe81c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.343+0000 7fe82293b640 1 -- 172.21.15.27:0/2417549326 shutdown_connections 2024-04-25T00:13:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.343+0000 7fe82293b640 1 -- 172.21.15.27:0/2417549326 wait complete. 2024-04-25T00:13:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.344+0000 7fe82293b640 1 Processor -- start 2024-04-25T00:13:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.344+0000 7fe82293b640 1 -- start start 2024-04-25T00:13:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.344+0000 7fe82293b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe81c1017a0 0x7fe81c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.344+0000 7fe82293b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe81c102990 0x7fe81c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe82293b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c103350 0x7fe81c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe82293b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c10d3f0 con 0x7fe81c103350 2024-04-25T00:13:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe82293b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c10d270 con 0x7fe81c102990 2024-04-25T00:13:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe82293b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c10d570 con 0x7fe81c1017a0 2024-04-25T00:13:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe820ed8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c103350 0x7fe81c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe820ed8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c103350 0x7fe81c19eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39680/0 (socket says 172.21.15.27:39680) 2024-04-25T00:13:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe820ed8640 1 -- 172.21.15.27:0/2421930884 learned_addr learned my addr 172.21.15.27:0/2421930884 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe81b7fe640 1 --2- 172.21.15.27:0/2421930884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe81c102990 0x7fe81c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe81bfff640 1 --2- 172.21.15.27:0/2421930884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe81c1017a0 0x7fe81c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe820ed8640 1 -- 172.21.15.27:0/2421930884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe81c1017a0 msgr2=0x7fe81c197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe820ed8640 1 --2- 172.21.15.27:0/2421930884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe81c1017a0 0x7fe81c197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe820ed8640 1 -- 172.21.15.27:0/2421930884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe81c102990 msgr2=0x7fe81c197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe820ed8640 1 --2- 172.21.15.27:0/2421930884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe81c102990 0x7fe81c197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe820ed8640 1 -- 172.21.15.27:0/2421930884 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81c19f660 con 0x7fe81c103350 2024-04-25T00:13:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe820ed8640 1 --2- 172.21.15.27:0/2421930884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c103350 0x7fe81c19eff0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe81400cce0 tx=0x7fe81400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe8197fa640 1 -- 172.21.15.27:0/2421930884 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81400f450 con 0x7fe81c103350 2024-04-25T00:13:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe82293b640 1 -- 172.21.15.27:0/2421930884 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe81c19f950 con 0x7fe81c103350 2024-04-25T00:13:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe8197fa640 1 -- 172.21.15.27:0/2421930884 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe81400fad0 con 0x7fe81c103350 2024-04-25T00:13:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe82293b640 1 -- 172.21.15.27:0/2421930884 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe81c19fea0 con 0x7fe81c103350 2024-04-25T00:13:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.345+0000 7fe8197fa640 1 -- 172.21.15.27:0/2421930884 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814018c10 con 0x7fe81c103350 2024-04-25T00:13:41.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.347+0000 7fe82293b640 1 -- 172.21.15.27:0/2421930884 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe81c108ac0 con 0x7fe81c103350 2024-04-25T00:13:41.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.348+0000 7fe8197fa640 1 -- 172.21.15.27:0/2421930884 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe81400f5f0 con 0x7fe81c103350 2024-04-25T00:13:41.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.349+0000 7fe8197fa640 1 --2- 172.21.15.27:0/2421930884 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe8040a2b90 0x7fe8040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:41.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.349+0000 7fe81bfff640 1 --2- 172.21.15.27:0/2421930884 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe8040a2b90 0x7fe8040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:41.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.349+0000 7fe8197fa640 1 -- 172.21.15.27:0/2421930884 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe8140cf870 con 0x7fe81c103350 2024-04-25T00:13:41.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.349+0000 7fe81bfff640 1 --2- 172.21.15.27:0/2421930884 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe8040a2b90 0x7fe8040a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe80c005ab0 tx=0x7fe80c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:41.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.353+0000 7fe8197fa640 1 -- 172.21.15.27:0/2421930884 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe814014030 con 0x7fe81c103350 2024-04-25T00:13:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.537+0000 7fe82293b640 1 -- 172.21.15.27:0/2421930884 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe81c062340 con 0x7fe81c103350 2024-04-25T00:13:41.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.539+0000 7fe8197fa640 1 -- 172.21.15.27:0/2421930884 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24117 (secure 0 0 0) 0x7fe81409c620 con 0x7fe81c103350 2024-04-25T00:13:41.539 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:41.539 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:13:23.935081+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:13:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.541+0000 7fe82293b640 1 -- 172.21.15.27:0/2421930884 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe8040a2b90 msgr2=0x7fe8040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.541+0000 7fe82293b640 1 --2- 172.21.15.27:0/2421930884 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe8040a2b90 0x7fe8040a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe80c005ab0 tx=0x7fe80c000bc0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.541+0000 7fe82293b640 1 -- 172.21.15.27:0/2421930884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c103350 msgr2=0x7fe81c19eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.541+0000 7fe82293b640 1 --2- 172.21.15.27:0/2421930884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c103350 0x7fe81c19eff0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe81400cce0 tx=0x7fe81400a620 comp rx=0 tx=0).stop 2024-04-25T00:13:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.541+0000 7fe82293b640 1 -- 172.21.15.27:0/2421930884 shutdown_connections 2024-04-25T00:13:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.541+0000 7fe82293b640 1 --2- 172.21.15.27:0/2421930884 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe8040a2b90 0x7fe8040a5050 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.541+0000 7fe82293b640 1 --2- 172.21.15.27:0/2421930884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c103350 0x7fe81c19eff0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.541+0000 7fe82293b640 1 --2- 172.21.15.27:0/2421930884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe81c102990 0x7fe81c197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.541+0000 7fe82293b640 1 --2- 172.21.15.27:0/2421930884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe81c1017a0 0x7fe81c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.541+0000 7fe82293b640 1 -- 172.21.15.27:0/2421930884 >> 172.21.15.27:0/2421930884 conn(0x7fe81c0fccf0 msgr2=0x7fe81c105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.541+0000 7fe82293b640 1 -- 172.21.15.27:0/2421930884 shutdown_connections 2024-04-25T00:13:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.541+0000 7fe82293b640 1 -- 172.21.15.27:0/2421930884 wait complete. 2024-04-25T00:13:41.552 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:13:41.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.655+0000 7f5923570640 1 -- 172.21.15.27:0/2369220889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f591c101890 msgr2=0x7f591c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:41.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.655+0000 7f5923570640 1 --2- 172.21.15.27:0/2369220889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f591c101890 0x7f591c101d10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5918009900 tx=0x7f591802a410 comp rx=0 tx=0).stop 2024-04-25T00:13:41.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.655+0000 7f5923570640 1 -- 172.21.15.27:0/2369220889 shutdown_connections 2024-04-25T00:13:41.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.655+0000 7f5923570640 1 --2- 172.21.15.27:0/2369220889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f591c102250 0x7f591c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.655+0000 7f5923570640 1 --2- 172.21.15.27:0/2369220889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f591c101890 0x7f591c101d10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.655+0000 7f5923570640 1 --2- 172.21.15.27:0/2369220889 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f591c1006a0 0x7f591c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.655+0000 7f5923570640 1 -- 172.21.15.27:0/2369220889 >> 172.21.15.27:0/2369220889 conn(0x7f591c0fbc30 msgr2=0x7f591c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:41.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.655+0000 7f5923570640 1 -- 172.21.15.27:0/2369220889 shutdown_connections 2024-04-25T00:13:41.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.655+0000 7f5923570640 1 -- 172.21.15.27:0/2369220889 wait complete. 2024-04-25T00:13:41.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.655+0000 7f5923570640 1 Processor -- start 2024-04-25T00:13:41.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f5923570640 1 -- start start 2024-04-25T00:13:41.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f5923570640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f591c1006a0 0x7f591c197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:41.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f5923570640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f591c101890 0x7f591c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:41.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f5923570640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f591c102250 0x7f591c19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:41.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f5923570640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f591c10c320 con 0x7f591c101890 2024-04-25T00:13:41.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f5923570640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f591c10c1a0 con 0x7f591c1006a0 2024-04-25T00:13:41.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f5923570640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f591c10c4a0 con 0x7f591c102250 2024-04-25T00:13:41.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f5920b0b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f591c101890 0x7f591c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:41.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f5920b0b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f591c101890 0x7f591c197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39690/0 (socket says 172.21.15.27:39690) 2024-04-25T00:13:41.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f5920b0b640 1 -- 172.21.15.27:0/4020925868 learned_addr learned my addr 172.21.15.27:0/4020925868 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:41.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f5921b0d640 1 --2- 172.21.15.27:0/4020925868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f591c102250 0x7f591c19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:41.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f592130c640 1 --2- 172.21.15.27:0/4020925868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f591c1006a0 0x7f591c197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:41.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f5920b0b640 1 -- 172.21.15.27:0/4020925868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f591c102250 msgr2=0x7f591c19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:41.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f5920b0b640 1 --2- 172.21.15.27:0/4020925868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f591c102250 0x7f591c19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f5920b0b640 1 -- 172.21.15.27:0/4020925868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f591c1006a0 msgr2=0x7f591c197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:41.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f5920b0b640 1 --2- 172.21.15.27:0/4020925868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f591c1006a0 0x7f591c197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f5920b0b640 1 -- 172.21.15.27:0/4020925868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f591c19f6d0 con 0x7f591c101890 2024-04-25T00:13:41.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f5920b0b640 1 --2- 172.21.15.27:0/4020925868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f591c101890 0x7f591c197ff0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5918009230 tx=0x7f5918004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:41.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.656+0000 7f590e7fc640 1 -- 172.21.15.27:0/4020925868 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591803f030 con 0x7f591c101890 2024-04-25T00:13:41.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.657+0000 7f5923570640 1 -- 172.21.15.27:0/4020925868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f591c19f960 con 0x7f591c101890 2024-04-25T00:13:41.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.657+0000 7f590e7fc640 1 -- 172.21.15.27:0/4020925868 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5918003ce0 con 0x7f591c101890 2024-04-25T00:13:41.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.657+0000 7f590e7fc640 1 -- 172.21.15.27:0/4020925868 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5918035070 con 0x7f591c101890 2024-04-25T00:13:41.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.657+0000 7f5923570640 1 -- 172.21.15.27:0/4020925868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f591c19fe50 con 0x7f591c101890 2024-04-25T00:13:41.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.658+0000 7f5923570640 1 -- 172.21.15.27:0/4020925868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f591c1a0020 con 0x7f591c101890 2024-04-25T00:13:41.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.659+0000 7f590e7fc640 1 -- 172.21.15.27:0/4020925868 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5918042030 con 0x7f591c101890 2024-04-25T00:13:41.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.659+0000 7f590e7fc640 1 --2- 172.21.15.27:0/4020925868 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f59080a2b90 0x7f59080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:41.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.659+0000 7f592130c640 1 --2- 172.21.15.27:0/4020925868 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f59080a2b90 0x7f59080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:41.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.659+0000 7f590e7fc640 1 -- 172.21.15.27:0/4020925868 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f59180efc70 con 0x7f591c101890 2024-04-25T00:13:41.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.660+0000 7f592130c640 1 --2- 172.21.15.27:0/4020925868 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f59080a2b90 0x7f59080a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5910005fd0 tx=0x7f5910005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:41.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.664+0000 7f590e7fc640 1 -- 172.21.15.27:0/4020925868 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f59180bc990 con 0x7f591c101890 2024-04-25T00:13:41.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.848+0000 7f5923570640 1 -- 172.21.15.27:0/4020925868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f591c1a03b0 con 0x7f591c101890 2024-04-25T00:13:41.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.850+0000 7f590e7fc640 1 -- 172.21.15.27:0/4020925868 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24117 (secure 0 0 0) 0x7f59180bc290 con 0x7f591c101890 2024-04-25T00:13:41.850 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:41.850 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:13:23.935081+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:13:41.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.852+0000 7f5923570640 1 -- 172.21.15.27:0/4020925868 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f59080a2b90 msgr2=0x7f59080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:41.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.852+0000 7f5923570640 1 --2- 172.21.15.27:0/4020925868 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f59080a2b90 0x7f59080a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5910005fd0 tx=0x7f5910005850 comp rx=0 tx=0).stop 2024-04-25T00:13:41.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.852+0000 7f5923570640 1 -- 172.21.15.27:0/4020925868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f591c101890 msgr2=0x7f591c197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:41.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.852+0000 7f5923570640 1 --2- 172.21.15.27:0/4020925868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f591c101890 0x7f591c197ff0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5918009230 tx=0x7f5918004360 comp rx=0 tx=0).stop 2024-04-25T00:13:41.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.852+0000 7f5923570640 1 -- 172.21.15.27:0/4020925868 shutdown_connections 2024-04-25T00:13:41.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.852+0000 7f5923570640 1 --2- 172.21.15.27:0/4020925868 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f59080a2b90 0x7f59080a5050 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.852+0000 7f5923570640 1 --2- 172.21.15.27:0/4020925868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f591c102250 0x7f591c19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.852+0000 7f5923570640 1 --2- 172.21.15.27:0/4020925868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f591c101890 0x7f591c197ff0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.852+0000 7f5923570640 1 --2- 172.21.15.27:0/4020925868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f591c1006a0 0x7f591c197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.852+0000 7f5923570640 1 -- 172.21.15.27:0/4020925868 >> 172.21.15.27:0/4020925868 conn(0x7f591c0fbc30 msgr2=0x7f591c0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.852+0000 7f5923570640 1 -- 172.21.15.27:0/4020925868 shutdown_connections 2024-04-25T00:13:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.852+0000 7f5923570640 1 -- 172.21.15.27:0/4020925868 wait complete. 2024-04-25T00:13:41.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:41 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2421930884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:41.863 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T00:13:41.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.968+0000 7f619a8a5640 1 -- 172.21.15.27:0/2031332378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61941006a0 msgr2=0x7f6194100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:41.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.968+0000 7f619a8a5640 1 --2- 172.21.15.27:0/2031332378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61941006a0 0x7f6194100ac0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6188009900 tx=0x7f618802a410 comp rx=0 tx=0).stop 2024-04-25T00:13:41.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.968+0000 7f619a8a5640 1 -- 172.21.15.27:0/2031332378 shutdown_connections 2024-04-25T00:13:41.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.968+0000 7f619a8a5640 1 --2- 172.21.15.27:0/2031332378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6194102250 0x7f6194108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.968+0000 7f619a8a5640 1 --2- 172.21.15.27:0/2031332378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6194101890 0x7f6194101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.968+0000 7f619a8a5640 1 --2- 172.21.15.27:0/2031332378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61941006a0 0x7f6194100ac0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.968+0000 7f619a8a5640 1 -- 172.21.15.27:0/2031332378 >> 172.21.15.27:0/2031332378 conn(0x7f61940fbc30 msgr2=0x7f61940fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:41.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.968+0000 7f619a8a5640 1 -- 172.21.15.27:0/2031332378 shutdown_connections 2024-04-25T00:13:41.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.968+0000 7f619a8a5640 1 -- 172.21.15.27:0/2031332378 wait complete. 2024-04-25T00:13:41.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.969+0000 7f619a8a5640 1 Processor -- start 2024-04-25T00:13:41.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.969+0000 7f619a8a5640 1 -- start start 2024-04-25T00:13:41.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.969+0000 7f619a8a5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61941006a0 0x7f619410b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:41.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.969+0000 7f619a8a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6194101890 0x7f6194109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:41.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.969+0000 7f619a8a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6194102250 0x7f6194109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:41.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.969+0000 7f619a8a5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f619410d4a0 con 0x7f6194101890 2024-04-25T00:13:41.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.969+0000 7f619a8a5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f619410d320 con 0x7f61941006a0 2024-04-25T00:13:41.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.969+0000 7f619a8a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f619410d620 con 0x7f6194102250 2024-04-25T00:13:41.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.969+0000 7f61937fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6194101890 0x7f6194109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:41.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.969+0000 7f61937fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6194101890 0x7f6194109660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39716/0 (socket says 172.21.15.27:39716) 2024-04-25T00:13:41.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.970+0000 7f61937fe640 1 -- 172.21.15.27:0/3071462309 learned_addr learned my addr 172.21.15.27:0/3071462309 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:41.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.970+0000 7f6198e42640 1 --2- 172.21.15.27:0/3071462309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6194102250 0x7f6194109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:41.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.970+0000 7f6193fff640 1 --2- 172.21.15.27:0/3071462309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61941006a0 0x7f619410b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:41.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.970+0000 7f61937fe640 1 -- 172.21.15.27:0/3071462309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6194102250 msgr2=0x7f6194109ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:41.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.970+0000 7f61937fe640 1 --2- 172.21.15.27:0/3071462309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6194102250 0x7f6194109ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.970+0000 7f61937fe640 1 -- 172.21.15.27:0/3071462309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61941006a0 msgr2=0x7f619410b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:41.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.970+0000 7f61937fe640 1 --2- 172.21.15.27:0/3071462309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61941006a0 0x7f619410b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:41.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.970+0000 7f61937fe640 1 -- 172.21.15.27:0/3071462309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f619410a430 con 0x7f6194101890 2024-04-25T00:13:41.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.970+0000 7f61937fe640 1 --2- 172.21.15.27:0/3071462309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6194101890 0x7f6194109660 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f618400d950 tx=0x7f618400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.970+0000 7f61917fa640 1 -- 172.21.15.27:0/3071462309 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6184017070 con 0x7f6194101890 2024-04-25T00:13:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.970+0000 7f619a8a5640 1 -- 172.21.15.27:0/3071462309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61941a1bd0 con 0x7f6194101890 2024-04-25T00:13:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.970+0000 7f61917fa640 1 -- 172.21.15.27:0/3071462309 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6184009c30 con 0x7f6194101890 2024-04-25T00:13:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.970+0000 7f61917fa640 1 -- 172.21.15.27:0/3071462309 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618400f6b0 con 0x7f6194101890 2024-04-25T00:13:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.970+0000 7f619a8a5640 1 -- 172.21.15.27:0/3071462309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61941a2120 con 0x7f6194101890 2024-04-25T00:13:41.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.971+0000 7f619a8a5640 1 -- 172.21.15.27:0/3071462309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6194107a40 con 0x7f6194101890 2024-04-25T00:13:41.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.972+0000 7f61917fa640 1 -- 172.21.15.27:0/3071462309 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f61840109d0 con 0x7f6194101890 2024-04-25T00:13:41.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.973+0000 7f61917fa640 1 --2- 172.21.15.27:0/3071462309 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f617c0a2b90 0x7f617c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:41.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.973+0000 7f6193fff640 1 --2- 172.21.15.27:0/3071462309 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f617c0a2b90 0x7f617c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:41.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.973+0000 7f61917fa640 1 -- 172.21.15.27:0/3071462309 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f61840cede0 con 0x7f6194101890 2024-04-25T00:13:41.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.974+0000 7f6193fff640 1 --2- 172.21.15.27:0/3071462309 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f617c0a2b90 0x7f617c0a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f618802aa00 tx=0x7f61880315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:41.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:41.978+0000 7f61917fa640 1 -- 172.21.15.27:0/3071462309 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6184009de0 con 0x7f6194101890 2024-04-25T00:13:42.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:41 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2421930884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:42.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:41 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2421930884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:42.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.202+0000 7f619a8a5640 1 -- 172.21.15.27:0/3071462309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6194062370 con 0x7f6194101890 2024-04-25T00:13:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.625+0000 7f61917fa640 1 -- 172.21.15.27:0/3071462309 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f618409b4b0 con 0x7f6194101890 2024-04-25T00:13:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.629+0000 7f619a8a5640 1 -- 172.21.15.27:0/3071462309 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f617c0a2b90 msgr2=0x7f617c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.629+0000 7f619a8a5640 1 --2- 172.21.15.27:0/3071462309 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f617c0a2b90 0x7f617c0a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f618802aa00 tx=0x7f61880315b0 comp rx=0 tx=0).stop 2024-04-25T00:13:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.629+0000 7f619a8a5640 1 -- 172.21.15.27:0/3071462309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6194101890 msgr2=0x7f6194109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.629+0000 7f619a8a5640 1 --2- 172.21.15.27:0/3071462309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6194101890 0x7f6194109660 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f618400d950 tx=0x7f618400de10 comp rx=0 tx=0).stop 2024-04-25T00:13:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.629+0000 7f619a8a5640 1 -- 172.21.15.27:0/3071462309 shutdown_connections 2024-04-25T00:13:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.629+0000 7f619a8a5640 1 --2- 172.21.15.27:0/3071462309 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f617c0a2b90 0x7f617c0a5050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.629+0000 7f619a8a5640 1 --2- 172.21.15.27:0/3071462309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6194102250 0x7f6194109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.629+0000 7f619a8a5640 1 --2- 172.21.15.27:0/3071462309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6194101890 0x7f6194109660 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.629+0000 7f619a8a5640 1 --2- 172.21.15.27:0/3071462309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61941006a0 0x7f619410b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.629+0000 7f619a8a5640 1 -- 172.21.15.27:0/3071462309 >> 172.21.15.27:0/3071462309 conn(0x7f61940fbc30 msgr2=0x7f6194104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.629+0000 7f619a8a5640 1 -- 172.21.15.27:0/3071462309 shutdown_connections 2024-04-25T00:13:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.629+0000 7f619a8a5640 1 -- 172.21.15.27:0/3071462309 wait complete. 2024-04-25T00:13:42.640 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-25T00:13:42.640 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-25T00:13:42.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.746+0000 7f9a9e69e640 1 -- 172.21.15.27:0/865129843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a98101870 msgr2=0x7f9a98101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.746+0000 7f9a9e69e640 1 --2- 172.21.15.27:0/865129843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a98101870 0x7f9a98101cf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009900 tx=0x7f9a8c02a410 comp rx=0 tx=0).stop 2024-04-25T00:13:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.746+0000 7f9a9e69e640 1 -- 172.21.15.27:0/865129843 shutdown_connections 2024-04-25T00:13:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.746+0000 7f9a9e69e640 1 --2- 172.21.15.27:0/865129843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a98102230 0x7f9a98108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.746+0000 7f9a9e69e640 1 --2- 172.21.15.27:0/865129843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a98101870 0x7f9a98101cf0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.746+0000 7f9a9e69e640 1 --2- 172.21.15.27:0/865129843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a98100680 0x7f9a98100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.746+0000 7f9a9e69e640 1 -- 172.21.15.27:0/865129843 >> 172.21.15.27:0/865129843 conn(0x7f9a980fbbf0 msgr2=0x7f9a980fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.746+0000 7f9a9e69e640 1 -- 172.21.15.27:0/865129843 shutdown_connections 2024-04-25T00:13:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.747+0000 7f9a9e69e640 1 -- 172.21.15.27:0/865129843 wait complete. 2024-04-25T00:13:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.747+0000 7f9a9e69e640 1 Processor -- start 2024-04-25T00:13:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.747+0000 7f9a9e69e640 1 -- start start 2024-04-25T00:13:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.747+0000 7f9a9e69e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a98100680 0x7f9a98069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.747+0000 7f9a9e69e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a98101870 0x7f9a9819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.747+0000 7f9a9e69e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a98102230 0x7f9a9819ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.747+0000 7f9a9e69e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9810c350 con 0x7f9a98100680 2024-04-25T00:13:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.747+0000 7f9a9e69e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9810c1d0 con 0x7f9a98101870 2024-04-25T00:13:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.747+0000 7f9a9e69e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9810c4d0 con 0x7f9a98102230 2024-04-25T00:13:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.748+0000 7f9a97fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a98100680 0x7f9a98069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.748+0000 7f9a97fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a98100680 0x7f9a98069440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39726/0 (socket says 172.21.15.27:39726) 2024-04-25T00:13:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.748+0000 7f9a97fff640 1 -- 172.21.15.27:0/1576176542 learned_addr learned my addr 172.21.15.27:0/1576176542 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.748+0000 7f9a977fe640 1 --2- 172.21.15.27:0/1576176542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a98101870 0x7f9a9819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.748+0000 7f9a9cc3b640 1 --2- 172.21.15.27:0/1576176542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a98102230 0x7f9a9819ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.748+0000 7f9a97fff640 1 -- 172.21.15.27:0/1576176542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a98102230 msgr2=0x7f9a9819ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.748+0000 7f9a97fff640 1 --2- 172.21.15.27:0/1576176542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a98102230 0x7f9a9819ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.748+0000 7f9a97fff640 1 -- 172.21.15.27:0/1576176542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a98101870 msgr2=0x7f9a9819c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.748+0000 7f9a97fff640 1 --2- 172.21.15.27:0/1576176542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a98101870 0x7f9a9819c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.748+0000 7f9a97fff640 1 -- 172.21.15.27:0/1576176542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9819f770 con 0x7f9a98100680 2024-04-25T00:13:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.748+0000 7f9a977fe640 1 --2- 172.21.15.27:0/1576176542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a98101870 0x7f9a9819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:13:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.748+0000 7f9a97fff640 1 --2- 172.21.15.27:0/1576176542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a98100680 0x7f9a98069440 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800d950 tx=0x7f9a8800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.748+0000 7f9a957fa640 1 -- 172.21.15.27:0/1576176542 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88017070 con 0x7f9a98100680 2024-04-25T00:13:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.748+0000 7f9a9e69e640 1 -- 172.21.15.27:0/1576176542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a9819fa00 con 0x7f9a98100680 2024-04-25T00:13:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.748+0000 7f9a957fa640 1 -- 172.21.15.27:0/1576176542 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a88009c30 con 0x7f9a98100680 2024-04-25T00:13:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.748+0000 7f9a957fa640 1 -- 172.21.15.27:0/1576176542 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8800f600 con 0x7f9a98100680 2024-04-25T00:13:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.748+0000 7f9a9e69e640 1 -- 172.21.15.27:0/1576176542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a9819fed0 con 0x7f9a98100680 2024-04-25T00:13:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.749+0000 7f9a9e69e640 1 -- 172.21.15.27:0/1576176542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a98107a90 con 0x7f9a98100680 2024-04-25T00:13:42.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.750+0000 7f9a957fa640 1 -- 172.21.15.27:0/1576176542 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9a8801a070 con 0x7f9a98100680 2024-04-25T00:13:42.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.751+0000 7f9a957fa640 1 --2- 172.21.15.27:0/1576176542 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9a800a2b90 0x7f9a800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:42.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.751+0000 7f9a977fe640 1 --2- 172.21.15.27:0/1576176542 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9a800a2b90 0x7f9a800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:42.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.751+0000 7f9a957fa640 1 -- 172.21.15.27:0/1576176542 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9a880cedc0 con 0x7f9a98100680 2024-04-25T00:13:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.752+0000 7f9a977fe640 1 --2- 172.21.15.27:0/1576176542 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9a800a2b90 0x7f9a800a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c006010 tx=0x7f9a8c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.756+0000 7f9a957fa640 1 -- 172.21.15.27:0/1576176542 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9a88009de0 con 0x7f9a98100680 2024-04-25T00:13:42.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:42 smithi027 ceph-mon[29208]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T00:13:42.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:42 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4020925868' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:42.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:42 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3071462309' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T00:13:42.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:42.986+0000 7f9a9e69e640 1 -- 172.21.15.27:0/1576176542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f9a980623e0 con 0x7f9a98100680 2024-04-25T00:13:43.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:42 smithi139 ceph-mon[32833]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T00:13:43.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:42 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4020925868' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:43.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:42 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3071462309' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T00:13:43.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:42 smithi153 ceph-mon[32896]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T00:13:43.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:42 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4020925868' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:43.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:42 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3071462309' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T00:13:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.633+0000 7f9a957fa640 1 -- 172.21.15.27:0/1576176542 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f9a8809bae0 con 0x7f9a98100680 2024-04-25T00:13:43.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.635+0000 7f9a9e69e640 1 -- 172.21.15.27:0/1576176542 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9a800a2b90 msgr2=0x7f9a800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:43.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.635+0000 7f9a9e69e640 1 --2- 172.21.15.27:0/1576176542 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9a800a2b90 0x7f9a800a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c006010 tx=0x7f9a8c000bc0 comp rx=0 tx=0).stop 2024-04-25T00:13:43.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.635+0000 7f9a9e69e640 1 -- 172.21.15.27:0/1576176542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a98100680 msgr2=0x7f9a98069440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:43.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.635+0000 7f9a9e69e640 1 --2- 172.21.15.27:0/1576176542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a98100680 0x7f9a98069440 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800d950 tx=0x7f9a8800de10 comp rx=0 tx=0).stop 2024-04-25T00:13:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.635+0000 7f9a9e69e640 1 -- 172.21.15.27:0/1576176542 shutdown_connections 2024-04-25T00:13:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.635+0000 7f9a9e69e640 1 --2- 172.21.15.27:0/1576176542 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9a800a2b90 0x7f9a800a5050 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.635+0000 7f9a9e69e640 1 --2- 172.21.15.27:0/1576176542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a98102230 0x7f9a9819ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.635+0000 7f9a9e69e640 1 --2- 172.21.15.27:0/1576176542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a98101870 0x7f9a9819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.635+0000 7f9a9e69e640 1 --2- 172.21.15.27:0/1576176542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a98100680 0x7f9a98069440 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.635+0000 7f9a9e69e640 1 -- 172.21.15.27:0/1576176542 >> 172.21.15.27:0/1576176542 conn(0x7f9a980fbbf0 msgr2=0x7f9a98104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.636+0000 7f9a9e69e640 1 -- 172.21.15.27:0/1576176542 shutdown_connections 2024-04-25T00:13:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.636+0000 7f9a9e69e640 1 -- 172.21.15.27:0/1576176542 wait complete. 2024-04-25T00:13:43.651 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-25T00:13:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.760+0000 7fb664b96640 1 -- 172.21.15.27:0/2500238031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb660102250 msgr2=0x7fb660108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.760+0000 7fb664b96640 1 --2- 172.21.15.27:0/2500238031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb660102250 0x7fb660108ae0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb650009ca0 tx=0x7fb65002a8c0 comp rx=0 tx=0).stop 2024-04-25T00:13:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.761+0000 7fb664b96640 1 -- 172.21.15.27:0/2500238031 shutdown_connections 2024-04-25T00:13:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.761+0000 7fb664b96640 1 --2- 172.21.15.27:0/2500238031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb660102250 0x7fb660108ae0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.761+0000 7fb664b96640 1 --2- 172.21.15.27:0/2500238031 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb660101890 0x7fb660101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.761+0000 7fb664b96640 1 --2- 172.21.15.27:0/2500238031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6601006a0 0x7fb660100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.761+0000 7fb664b96640 1 -- 172.21.15.27:0/2500238031 >> 172.21.15.27:0/2500238031 conn(0x7fb6600fbc30 msgr2=0x7fb6600fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.761+0000 7fb664b96640 1 -- 172.21.15.27:0/2500238031 shutdown_connections 2024-04-25T00:13:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.761+0000 7fb664b96640 1 -- 172.21.15.27:0/2500238031 wait complete. 2024-04-25T00:13:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.761+0000 7fb664b96640 1 Processor -- start 2024-04-25T00:13:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.761+0000 7fb664b96640 1 -- start start 2024-04-25T00:13:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.761+0000 7fb664b96640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6601006a0 0x7fb660197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.761+0000 7fb664b96640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb660101890 0x7fb660197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.761+0000 7fb664b96640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb660102250 0x7fb66019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.761+0000 7fb664b96640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb66010d400 con 0x7fb6601006a0 2024-04-25T00:13:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.761+0000 7fb664b96640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb66010d280 con 0x7fb660101890 2024-04-25T00:13:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.761+0000 7fb664b96640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb66010d580 con 0x7fb660102250 2024-04-25T00:13:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.762+0000 7fb65e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6601006a0 0x7fb660197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.762+0000 7fb65e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6601006a0 0x7fb660197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39750/0 (socket says 172.21.15.27:39750) 2024-04-25T00:13:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.762+0000 7fb65e59c640 1 -- 172.21.15.27:0/1550585566 learned_addr learned my addr 172.21.15.27:0/1550585566 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.762+0000 7fb65ed9d640 1 --2- 172.21.15.27:0/1550585566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb660102250 0x7fb66019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.762+0000 7fb65dd9b640 1 --2- 172.21.15.27:0/1550585566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb660101890 0x7fb660197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:43.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.762+0000 7fb65e59c640 1 -- 172.21.15.27:0/1550585566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb660102250 msgr2=0x7fb66019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:43.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.762+0000 7fb65e59c640 1 --2- 172.21.15.27:0/1550585566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb660102250 0x7fb66019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:43.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.762+0000 7fb65e59c640 1 -- 172.21.15.27:0/1550585566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb660101890 msgr2=0x7fb660197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:43.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.762+0000 7fb65e59c640 1 --2- 172.21.15.27:0/1550585566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb660101890 0x7fb660197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:43.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.762+0000 7fb65e59c640 1 -- 172.21.15.27:0/1550585566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb66019f660 con 0x7fb6601006a0 2024-04-25T00:13:43.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.762+0000 7fb65dd9b640 1 --2- 172.21.15.27:0/1550585566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb660101890 0x7fb660197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:43.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.762+0000 7fb65e59c640 1 --2- 172.21.15.27:0/1550585566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6601006a0 0x7fb660197a40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb65800d950 tx=0x7fb65800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:43.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.762+0000 7fb64f7fe640 1 -- 172.21.15.27:0/1550585566 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb658017020 con 0x7fb6601006a0 2024-04-25T00:13:43.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.762+0000 7fb664b96640 1 -- 172.21.15.27:0/1550585566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb66019f950 con 0x7fb6601006a0 2024-04-25T00:13:43.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.762+0000 7fb64f7fe640 1 -- 172.21.15.27:0/1550585566 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb658009c30 con 0x7fb6601006a0 2024-04-25T00:13:43.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.762+0000 7fb64f7fe640 1 -- 172.21.15.27:0/1550585566 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65800f6b0 con 0x7fb6601006a0 2024-04-25T00:13:43.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.762+0000 7fb664b96640 1 -- 172.21.15.27:0/1550585566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb66019fef0 con 0x7fb6601006a0 2024-04-25T00:13:43.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.763+0000 7fb664b96640 1 -- 172.21.15.27:0/1550585566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6600689b0 con 0x7fb6601006a0 2024-04-25T00:13:43.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.764+0000 7fb64f7fe640 1 -- 172.21.15.27:0/1550585566 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb65801a020 con 0x7fb6601006a0 2024-04-25T00:13:43.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.765+0000 7fb64f7fe640 1 --2- 172.21.15.27:0/1550585566 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb6480a2b90 0x7fb6480a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:43.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.765+0000 7fb65dd9b640 1 --2- 172.21.15.27:0/1550585566 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb6480a2b90 0x7fb6480a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:43.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.765+0000 7fb64f7fe640 1 -- 172.21.15.27:0/1550585566 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb6580cf3a0 con 0x7fb6601006a0 2024-04-25T00:13:43.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.766+0000 7fb65dd9b640 1 --2- 172.21.15.27:0/1550585566 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb6480a2b90 0x7fb6480a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb654005fd0 tx=0x7fb6540094d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:43.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.769+0000 7fb64f7fe640 1 -- 172.21.15.27:0/1550585566 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb658021020 con 0x7fb6601006a0 2024-04-25T00:13:43.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:43 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3071462309' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T00:13:43.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:43 smithi027 ceph-mon[29208]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T00:13:43.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:43 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1576176542' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T00:13:43.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:43.993+0000 7fb664b96640 1 -- 172.21.15.27:0/1550585566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fb6601079c0 con 0x7fb6601006a0 2024-04-25T00:13:44.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:43 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3071462309' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T00:13:44.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:43 smithi139 ceph-mon[32833]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T00:13:44.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:43 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1576176542' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T00:13:44.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:43 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3071462309' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T00:13:44.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:43 smithi153 ceph-mon[32896]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T00:13:44.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:43 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1576176542' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T00:13:44.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.641+0000 7fb64f7fe640 1 -- 172.21.15.27:0/1550585566 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7fb65809c0c0 con 0x7fb6601006a0 2024-04-25T00:13:44.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.644+0000 7fb664b96640 1 -- 172.21.15.27:0/1550585566 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb6480a2b90 msgr2=0x7fb6480a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:44.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.644+0000 7fb664b96640 1 --2- 172.21.15.27:0/1550585566 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb6480a2b90 0x7fb6480a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb654005fd0 tx=0x7fb6540094d0 comp rx=0 tx=0).stop 2024-04-25T00:13:44.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.644+0000 7fb664b96640 1 -- 172.21.15.27:0/1550585566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6601006a0 msgr2=0x7fb660197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:44.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.644+0000 7fb664b96640 1 --2- 172.21.15.27:0/1550585566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6601006a0 0x7fb660197a40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb65800d950 tx=0x7fb65800de10 comp rx=0 tx=0).stop 2024-04-25T00:13:44.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.644+0000 7fb664b96640 1 -- 172.21.15.27:0/1550585566 shutdown_connections 2024-04-25T00:13:44.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.644+0000 7fb664b96640 1 --2- 172.21.15.27:0/1550585566 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb6480a2b90 0x7fb6480a5050 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:44.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.644+0000 7fb664b96640 1 --2- 172.21.15.27:0/1550585566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb660102250 0x7fb66019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:44.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.644+0000 7fb664b96640 1 --2- 172.21.15.27:0/1550585566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb660101890 0x7fb660197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:44.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.644+0000 7fb664b96640 1 --2- 172.21.15.27:0/1550585566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6601006a0 0x7fb660197a40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:44.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.644+0000 7fb664b96640 1 -- 172.21.15.27:0/1550585566 >> 172.21.15.27:0/1550585566 conn(0x7fb6600fbc30 msgr2=0x7fb660104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:44.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.645+0000 7fb664b96640 1 -- 172.21.15.27:0/1550585566 shutdown_connections 2024-04-25T00:13:44.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.645+0000 7fb664b96640 1 -- 172.21.15.27:0/1550585566 wait complete. 2024-04-25T00:13:44.660 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-25T00:13:44.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.768+0000 7f7ce0cb3640 1 -- 172.21.15.27:0/2353753686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cdc102250 msgr2=0x7f7cdc108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:44.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.768+0000 7f7ce0cb3640 1 --2- 172.21.15.27:0/2353753686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cdc102250 0x7f7cdc108ae0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009ca0 tx=0x7f7cd402a8c0 comp rx=0 tx=0).stop 2024-04-25T00:13:44.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.768+0000 7f7ce0cb3640 1 -- 172.21.15.27:0/2353753686 shutdown_connections 2024-04-25T00:13:44.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.768+0000 7f7ce0cb3640 1 --2- 172.21.15.27:0/2353753686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cdc102250 0x7f7cdc108ae0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:44.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.768+0000 7f7ce0cb3640 1 --2- 172.21.15.27:0/2353753686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7cdc101890 0x7f7cdc101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:44.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.768+0000 7f7ce0cb3640 1 --2- 172.21.15.27:0/2353753686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cdc1006a0 0x7f7cdc100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:44.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.768+0000 7f7ce0cb3640 1 -- 172.21.15.27:0/2353753686 >> 172.21.15.27:0/2353753686 conn(0x7f7cdc0fbc30 msgr2=0x7f7cdc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:44.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.768+0000 7f7ce0cb3640 1 -- 172.21.15.27:0/2353753686 shutdown_connections 2024-04-25T00:13:44.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.768+0000 7f7ce0cb3640 1 -- 172.21.15.27:0/2353753686 wait complete. 2024-04-25T00:13:44.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.769+0000 7f7ce0cb3640 1 Processor -- start 2024-04-25T00:13:44.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.769+0000 7f7ce0cb3640 1 -- start start 2024-04-25T00:13:44.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.769+0000 7f7ce0cb3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cdc1006a0 0x7f7cdc06a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:44.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.769+0000 7f7ce0cb3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7cdc101890 0x7f7cdc19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:44.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.769+0000 7f7ce0cb3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cdc102250 0x7f7cdc1a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:44.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.769+0000 7f7ce0cb3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc10c320 con 0x7f7cdc102250 2024-04-25T00:13:44.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.769+0000 7f7ce0cb3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc10c1a0 con 0x7f7cdc101890 2024-04-25T00:13:44.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.769+0000 7f7ce0cb3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc10c4a0 con 0x7f7cdc1006a0 2024-04-25T00:13:44.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.769+0000 7f7cdad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cdc102250 0x7f7cdc1a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:44.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.769+0000 7f7cdad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cdc102250 0x7f7cdc1a1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39762/0 (socket says 172.21.15.27:39762) 2024-04-25T00:13:44.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.769+0000 7f7cdad9d640 1 -- 172.21.15.27:0/3649224471 learned_addr learned my addr 172.21.15.27:0/3649224471 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:44.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.769+0000 7f7cd3fff640 1 --2- 172.21.15.27:0/3649224471 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7cdc101890 0x7f7cdc19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:44.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.769+0000 7f7cda59c640 1 --2- 172.21.15.27:0/3649224471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cdc1006a0 0x7f7cdc06a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:44.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.770+0000 7f7cd3fff640 1 -- 172.21.15.27:0/3649224471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cdc1006a0 msgr2=0x7f7cdc06a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:44.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.770+0000 7f7cd3fff640 1 --2- 172.21.15.27:0/3649224471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cdc1006a0 0x7f7cdc06a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:44.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.770+0000 7f7cd3fff640 1 -- 172.21.15.27:0/3649224471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cdc102250 msgr2=0x7f7cdc1a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:44.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.770+0000 7f7cd3fff640 1 --2- 172.21.15.27:0/3649224471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cdc102250 0x7f7cdc1a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:44.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.770+0000 7f7cd3fff640 1 -- 172.21.15.27:0/3649224471 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cdc1a1920 con 0x7f7cdc101890 2024-04-25T00:13:44.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.770+0000 7f7cdad9d640 1 --2- 172.21.15.27:0/3649224471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cdc102250 0x7f7cdc1a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:13:44.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.770+0000 7f7cd3fff640 1 --2- 172.21.15.27:0/3649224471 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7cdc101890 0x7f7cdc19ebe0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400e9f0 tx=0x7f7cc400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:44.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.770+0000 7f7cd37fe640 1 -- 172.21.15.27:0/3649224471 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc400ccc0 con 0x7f7cdc101890 2024-04-25T00:13:44.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.770+0000 7f7cd37fe640 1 -- 172.21.15.27:0/3649224471 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cc4010910 con 0x7f7cdc101890 2024-04-25T00:13:44.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.770+0000 7f7ce0cb3640 1 -- 172.21.15.27:0/3649224471 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cdc1a1bb0 con 0x7f7cdc101890 2024-04-25T00:13:44.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.770+0000 7f7cd37fe640 1 -- 172.21.15.27:0/3649224471 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4018960 con 0x7f7cdc101890 2024-04-25T00:13:44.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.770+0000 7f7ce0cb3640 1 -- 172.21.15.27:0/3649224471 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cdc1a2100 con 0x7f7cdc101890 2024-04-25T00:13:44.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.771+0000 7f7ce0cb3640 1 -- 172.21.15.27:0/3649224471 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cdc1079c0 con 0x7f7cdc101890 2024-04-25T00:13:44.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.772+0000 7f7cd37fe640 1 -- 172.21.15.27:0/3649224471 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7cc4010ac0 con 0x7f7cdc101890 2024-04-25T00:13:44.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.773+0000 7f7cd37fe640 1 --2- 172.21.15.27:0/3649224471 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7cc80a2bc0 0x7f7cc80a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:44.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.773+0000 7f7cd37fe640 1 -- 172.21.15.27:0/3649224471 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7cc40cfa70 con 0x7f7cdc101890 2024-04-25T00:13:44.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.773+0000 7f7cda59c640 1 --2- 172.21.15.27:0/3649224471 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7cc80a2bc0 0x7f7cc80a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:44.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.773+0000 7f7cda59c640 1 --2- 172.21.15.27:0/3649224471 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7cc80a2bc0 0x7f7cc80a5080 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc005fd0 tx=0x7f7ccc005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:44.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:44.776+0000 7f7cd37fe640 1 -- 172.21.15.27:0/3649224471 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7cc409c730 con 0x7f7cdc101890 2024-04-25T00:13:44.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:44 smithi027 ceph-mon[29208]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T00:13:44.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:44 smithi027 ceph-mon[29208]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T00:13:44.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:44 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1576176542' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T00:13:44.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:44 smithi027 ceph-mon[29208]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T00:13:44.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:44 smithi027 ceph-mon[29208]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T00:13:44.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:44 smithi027 ceph-mon[29208]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T00:13:44.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:44 smithi027 ceph-mon[29208]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T00:13:44.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:44 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1550585566' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T00:13:45.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.002+0000 7f7ce0cb3640 1 -- 172.21.15.27:0/3649224471 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f7cdc02d210 con 0x7f7cdc101890 2024-04-25T00:13:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:44 smithi139 ceph-mon[32833]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T00:13:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:44 smithi139 ceph-mon[32833]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T00:13:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:44 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1576176542' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T00:13:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:44 smithi139 ceph-mon[32833]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T00:13:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:44 smithi139 ceph-mon[32833]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T00:13:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:44 smithi139 ceph-mon[32833]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T00:13:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:44 smithi139 ceph-mon[32833]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T00:13:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:44 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1550585566' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T00:13:45.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:44 smithi153 ceph-mon[32896]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T00:13:45.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:44 smithi153 ceph-mon[32896]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T00:13:45.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:44 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1576176542' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T00:13:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:44 smithi153 ceph-mon[32896]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T00:13:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:44 smithi153 ceph-mon[32896]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T00:13:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:44 smithi153 ceph-mon[32896]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T00:13:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:44 smithi153 ceph-mon[32896]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T00:13:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:44 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1550585566' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T00:13:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.559+0000 7f7cd37fe640 1 -- 172.21.15.27:0/3649224471 <== mon.2 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f7cc409c030 con 0x7f7cdc101890 2024-04-25T00:13:45.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.562+0000 7f7ce0cb3640 1 -- 172.21.15.27:0/3649224471 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7cc80a2bc0 msgr2=0x7f7cc80a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:45.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.562+0000 7f7ce0cb3640 1 --2- 172.21.15.27:0/3649224471 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7cc80a2bc0 0x7f7cc80a5080 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc005fd0 tx=0x7f7ccc005850 comp rx=0 tx=0).stop 2024-04-25T00:13:45.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.562+0000 7f7ce0cb3640 1 -- 172.21.15.27:0/3649224471 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7cdc101890 msgr2=0x7f7cdc19ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:45.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.562+0000 7f7ce0cb3640 1 --2- 172.21.15.27:0/3649224471 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7cdc101890 0x7f7cdc19ebe0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400e9f0 tx=0x7f7cc400eeb0 comp rx=0 tx=0).stop 2024-04-25T00:13:45.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.562+0000 7f7ce0cb3640 1 -- 172.21.15.27:0/3649224471 shutdown_connections 2024-04-25T00:13:45.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.562+0000 7f7ce0cb3640 1 --2- 172.21.15.27:0/3649224471 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7cc80a2bc0 0x7f7cc80a5080 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:45.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.562+0000 7f7ce0cb3640 1 --2- 172.21.15.27:0/3649224471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cdc102250 0x7f7cdc1a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:45.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.562+0000 7f7ce0cb3640 1 --2- 172.21.15.27:0/3649224471 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7cdc101890 0x7f7cdc19ebe0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:45.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.562+0000 7f7ce0cb3640 1 --2- 172.21.15.27:0/3649224471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cdc1006a0 0x7f7cdc06a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:45.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.562+0000 7f7ce0cb3640 1 -- 172.21.15.27:0/3649224471 >> 172.21.15.27:0/3649224471 conn(0x7f7cdc0fbc30 msgr2=0x7f7cdc104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:45.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.563+0000 7f7ce0cb3640 1 -- 172.21.15.27:0/3649224471 shutdown_connections 2024-04-25T00:13:45.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.563+0000 7f7ce0cb3640 1 -- 172.21.15.27:0/3649224471 wait complete. 2024-04-25T00:13:45.580 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-25T00:13:45.580 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-25T00:13:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.692+0000 7f60ad2a5640 1 -- 172.21.15.27:0/1468288710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60a8101870 msgr2=0x7f60a8101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.692+0000 7f60ad2a5640 1 --2- 172.21.15.27:0/1468288710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60a8101870 0x7f60a8101cf0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6090009900 tx=0x7f609002a410 comp rx=0 tx=0).stop 2024-04-25T00:13:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.692+0000 7f60ad2a5640 1 -- 172.21.15.27:0/1468288710 shutdown_connections 2024-04-25T00:13:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.692+0000 7f60ad2a5640 1 --2- 172.21.15.27:0/1468288710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60a8102230 0x7f60a8108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.692+0000 7f60ad2a5640 1 --2- 172.21.15.27:0/1468288710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60a8101870 0x7f60a8101cf0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.692+0000 7f60ad2a5640 1 --2- 172.21.15.27:0/1468288710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60a8100680 0x7f60a8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.692+0000 7f60ad2a5640 1 -- 172.21.15.27:0/1468288710 >> 172.21.15.27:0/1468288710 conn(0x7f60a80fbbf0 msgr2=0x7f60a80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.693+0000 7f60ad2a5640 1 -- 172.21.15.27:0/1468288710 shutdown_connections 2024-04-25T00:13:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.693+0000 7f60ad2a5640 1 -- 172.21.15.27:0/1468288710 wait complete. 2024-04-25T00:13:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.693+0000 7f60ad2a5640 1 Processor -- start 2024-04-25T00:13:45.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.693+0000 7f60ad2a5640 1 -- start start 2024-04-25T00:13:45.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.693+0000 7f60ad2a5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60a8100680 0x7f60a8197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.693+0000 7f60ad2a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60a8101870 0x7f60a8198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.693+0000 7f60ad2a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60a8102230 0x7f60a819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f60ad2a5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f60a810c300 con 0x7f60a8101870 2024-04-25T00:13:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f60ad2a5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f60a810c180 con 0x7f60a8100680 2024-04-25T00:13:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f60ad2a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f60a810c480 con 0x7f60a8102230 2024-04-25T00:13:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f60a6ffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60a8100680 0x7f60a8197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f60a67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60a8101870 0x7f60a8198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f60a6ffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60a8100680 0x7f60a8197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:55744/0 (socket says 172.21.15.27:55744) 2024-04-25T00:13:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f60a6ffd640 1 -- 172.21.15.27:0/2135526048 learned_addr learned my addr 172.21.15.27:0/2135526048 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f60a77fe640 1 --2- 172.21.15.27:0/2135526048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60a8102230 0x7f60a819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f60a6ffd640 1 -- 172.21.15.27:0/2135526048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60a8102230 msgr2=0x7f60a819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f60a6ffd640 1 --2- 172.21.15.27:0/2135526048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60a8102230 0x7f60a819f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f60a6ffd640 1 -- 172.21.15.27:0/2135526048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60a8101870 msgr2=0x7f60a8198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f60a6ffd640 1 --2- 172.21.15.27:0/2135526048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60a8101870 0x7f60a8198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f60a6ffd640 1 -- 172.21.15.27:0/2135526048 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60a819f6f0 con 0x7f60a8100680 2024-04-25T00:13:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f60a67fc640 1 --2- 172.21.15.27:0/2135526048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60a8101870 0x7f60a8198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:13:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f60a6ffd640 1 --2- 172.21.15.27:0/2135526048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60a8100680 0x7f60a8197ad0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f60a000e9c0 tx=0x7f60a000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f609ffff640 1 -- 172.21.15.27:0/2135526048 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a000cde0 con 0x7f60a8100680 2024-04-25T00:13:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f60ad2a5640 1 -- 172.21.15.27:0/2135526048 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60a819f9e0 con 0x7f60a8100680 2024-04-25T00:13:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f609ffff640 1 -- 172.21.15.27:0/2135526048 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60a0010910 con 0x7f60a8100680 2024-04-25T00:13:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f609ffff640 1 -- 172.21.15.27:0/2135526048 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a0018a10 con 0x7f60a8100680 2024-04-25T00:13:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.694+0000 7f60ad2a5640 1 -- 172.21.15.27:0/2135526048 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60a819ff50 con 0x7f60a8100680 2024-04-25T00:13:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.695+0000 7f60ad2a5640 1 -- 172.21.15.27:0/2135526048 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60a81a0180 con 0x7f60a8100680 2024-04-25T00:13:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.696+0000 7f609ffff640 1 -- 172.21.15.27:0/2135526048 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f60a001f020 con 0x7f60a8100680 2024-04-25T00:13:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.697+0000 7f609ffff640 1 --2- 172.21.15.27:0/2135526048 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f60940a2b90 0x7f60940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.697+0000 7f60a67fc640 1 --2- 172.21.15.27:0/2135526048 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f60940a2b90 0x7f60940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.697+0000 7f609ffff640 1 -- 172.21.15.27:0/2135526048 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f60a00cf490 con 0x7f60a8100680 2024-04-25T00:13:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.697+0000 7f60a67fc640 1 --2- 172.21.15.27:0/2135526048 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f60940a2b90 0x7f60940a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f60900314e0 tx=0x7f6090000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:45.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.702+0000 7f609ffff640 1 -- 172.21.15.27:0/2135526048 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f60a009c1b0 con 0x7f60a8100680 2024-04-25T00:13:45.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.915+0000 7f60ad2a5640 1 -- 172.21.15.27:0/2135526048 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f60a80fde20 con 0x7f60940a2b90 2024-04-25T00:13:45.921 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1550585566' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T00:13:45.921 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] up:active 2024-04-25T00:13:45.921 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-mon[29208]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T00:13:45.921 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-mon[29208]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T00:13:45.921 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-mon[29208]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-25T00:13:45.921 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-mon[29208]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-25T00:13:45.921 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3649224471' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T00:13:45.922 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-mon[29208]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T00:13:45.922 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-mon[29208]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T00:13:45.922 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] up:active 2024-04-25T00:13:45.922 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-mon[29208]: daemon mds.l assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T00:13:45.922 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-mon[29208]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T00:13:45.922 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-mon[29208]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=l=up:creating} 3 up:standby-replay 5 up:standby 2024-04-25T00:13:45.922 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-mon[29208]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-04-25T00:13:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.941+0000 7f609ffff640 1 -- 172.21.15.27:0/2135526048 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f60a80fde20 con 0x7f60940a2b90 2024-04-25T00:13:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.943+0000 7f60ad2a5640 1 -- 172.21.15.27:0/2135526048 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f60940a2b90 msgr2=0x7f60940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.943+0000 7f60ad2a5640 1 --2- 172.21.15.27:0/2135526048 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f60940a2b90 0x7f60940a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f60900314e0 tx=0x7f6090000bc0 comp rx=0 tx=0).stop 2024-04-25T00:13:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.943+0000 7f60ad2a5640 1 -- 172.21.15.27:0/2135526048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60a8100680 msgr2=0x7f60a8197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.943+0000 7f60ad2a5640 1 --2- 172.21.15.27:0/2135526048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60a8100680 0x7f60a8197ad0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f60a000e9c0 tx=0x7f60a000ee80 comp rx=0 tx=0).stop 2024-04-25T00:13:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.943+0000 7f60ad2a5640 1 -- 172.21.15.27:0/2135526048 shutdown_connections 2024-04-25T00:13:45.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.943+0000 7f60ad2a5640 1 --2- 172.21.15.27:0/2135526048 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f60940a2b90 0x7f60940a5050 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:45.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.943+0000 7f60ad2a5640 1 --2- 172.21.15.27:0/2135526048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60a8102230 0x7f60a819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:45.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.943+0000 7f60ad2a5640 1 --2- 172.21.15.27:0/2135526048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60a8101870 0x7f60a8198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:45.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.943+0000 7f60ad2a5640 1 --2- 172.21.15.27:0/2135526048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60a8100680 0x7f60a8197ad0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:45.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.943+0000 7f60ad2a5640 1 -- 172.21.15.27:0/2135526048 >> 172.21.15.27:0/2135526048 conn(0x7f60a80fbbf0 msgr2=0x7f60a80fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:45.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.943+0000 7f60ad2a5640 1 -- 172.21.15.27:0/2135526048 shutdown_connections 2024-04-25T00:13:45.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:45.943+0000 7f60ad2a5640 1 -- 172.21.15.27:0/2135526048 wait complete. 2024-04-25T00:13:45.954 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-25T00:13:46.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:45 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1550585566' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T00:13:46.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:45 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] up:active 2024-04-25T00:13:46.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:45 smithi139 ceph-mon[32833]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T00:13:46.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:45 smithi139 ceph-mon[32833]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T00:13:46.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:45 smithi139 ceph-mon[32833]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-25T00:13:46.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:45 smithi139 ceph-mon[32833]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-25T00:13:46.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:45 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3649224471' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T00:13:46.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:45 smithi139 ceph-mon[32833]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T00:13:46.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:45 smithi139 ceph-mon[32833]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T00:13:46.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:45 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] up:active 2024-04-25T00:13:46.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:45 smithi139 ceph-mon[32833]: daemon mds.l assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T00:13:46.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:45 smithi139 ceph-mon[32833]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T00:13:46.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:45 smithi139 ceph-mon[32833]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=l=up:creating} 3 up:standby-replay 5 up:standby 2024-04-25T00:13:46.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:45 smithi139 ceph-mon[32833]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-04-25T00:13:46.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:45 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1550585566' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T00:13:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:45 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] up:active 2024-04-25T00:13:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:45 smithi153 ceph-mon[32896]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T00:13:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:45 smithi153 ceph-mon[32896]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T00:13:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:45 smithi153 ceph-mon[32896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-25T00:13:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:45 smithi153 ceph-mon[32896]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-25T00:13:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:45 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3649224471' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T00:13:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:45 smithi153 ceph-mon[32896]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T00:13:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:45 smithi153 ceph-mon[32896]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T00:13:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:45 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] up:active 2024-04-25T00:13:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:45 smithi153 ceph-mon[32896]: daemon mds.l assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T00:13:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:45 smithi153 ceph-mon[32896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T00:13:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:45 smithi153 ceph-mon[32896]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=l=up:creating} 3 up:standby-replay 5 up:standby 2024-04-25T00:13:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:45 smithi153 ceph-mon[32896]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-04-25T00:13:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.061+0000 7f94ddcba640 1 -- 172.21.15.27:0/4186764773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d8102280 msgr2=0x7f94d8104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.061+0000 7f94ddcba640 1 --2- 172.21.15.27:0/4186764773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d8102280 0x7f94d8104670 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f94cc009900 tx=0x7f94cc02a410 comp rx=0 tx=0).stop 2024-04-25T00:13:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.061+0000 7f94ddcba640 1 -- 172.21.15.27:0/4186764773 shutdown_connections 2024-04-25T00:13:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.061+0000 7f94ddcba640 1 --2- 172.21.15.27:0/4186764773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94d81074e0 0x7f94d81098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.061+0000 7f94ddcba640 1 --2- 172.21.15.27:0/4186764773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d8104bb0 0x7f94d8106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.061+0000 7f94ddcba640 1 --2- 172.21.15.27:0/4186764773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d8102280 0x7f94d8104670 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.061+0000 7f94ddcba640 1 -- 172.21.15.27:0/4186764773 >> 172.21.15.27:0/4186764773 conn(0x7f94d80fbc30 msgr2=0x7f94d80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.061+0000 7f94ddcba640 1 -- 172.21.15.27:0/4186764773 shutdown_connections 2024-04-25T00:13:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.061+0000 7f94ddcba640 1 -- 172.21.15.27:0/4186764773 wait complete. 2024-04-25T00:13:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.062+0000 7f94ddcba640 1 Processor -- start 2024-04-25T00:13:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.062+0000 7f94ddcba640 1 -- start start 2024-04-25T00:13:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.062+0000 7f94ddcba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d8102280 0x7f94d8197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.062+0000 7f94ddcba640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d8104bb0 0x7f94d8197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.062+0000 7f94ddcba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94d81074e0 0x7f94d819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.062+0000 7f94ddcba640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f94d810d440 con 0x7f94d8102280 2024-04-25T00:13:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.062+0000 7f94ddcba640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f94d810d2c0 con 0x7f94d8104bb0 2024-04-25T00:13:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.062+0000 7f94ddcba640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f94d810d5c0 con 0x7f94d81074e0 2024-04-25T00:13:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.062+0000 7f94d77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d8102280 0x7f94d8197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.062+0000 7f94d77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d8102280 0x7f94d8197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39830/0 (socket says 172.21.15.27:39830) 2024-04-25T00:13:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.062+0000 7f94d77fe640 1 -- 172.21.15.27:0/456635725 learned_addr learned my addr 172.21.15.27:0/456635725 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.062+0000 7f94d6ffd640 1 --2- 172.21.15.27:0/456635725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d8104bb0 0x7f94d8197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.062+0000 7f94d77fe640 1 -- 172.21.15.27:0/456635725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94d81074e0 msgr2=0x7f94d819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.062+0000 7f94d77fe640 1 --2- 172.21.15.27:0/456635725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94d81074e0 0x7f94d819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.062+0000 7f94d77fe640 1 -- 172.21.15.27:0/456635725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d8104bb0 msgr2=0x7f94d8197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.062+0000 7f94d77fe640 1 --2- 172.21.15.27:0/456635725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d8104bb0 0x7f94d8197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.062+0000 7f94d77fe640 1 -- 172.21.15.27:0/456635725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d819f6d0 con 0x7f94d8102280 2024-04-25T00:13:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.063+0000 7f94d77fe640 1 --2- 172.21.15.27:0/456635725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d8102280 0x7f94d8197ab0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f94cc009230 tx=0x7f94cc031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.063+0000 7f94d4ff9640 1 -- 172.21.15.27:0/456635725 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94cc03f030 con 0x7f94d8102280 2024-04-25T00:13:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.063+0000 7f94d4ff9640 1 -- 172.21.15.27:0/456635725 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94cc003ad0 con 0x7f94d8102280 2024-04-25T00:13:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.063+0000 7f94ddcba640 1 -- 172.21.15.27:0/456635725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94d819f960 con 0x7f94d8102280 2024-04-25T00:13:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.063+0000 7f94d4ff9640 1 -- 172.21.15.27:0/456635725 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94cc035070 con 0x7f94d8102280 2024-04-25T00:13:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.063+0000 7f94ddcba640 1 -- 172.21.15.27:0/456635725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94d819fe50 con 0x7f94d8102280 2024-04-25T00:13:46.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.064+0000 7f94ddcba640 1 -- 172.21.15.27:0/456635725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94d80fd720 con 0x7f94d8102280 2024-04-25T00:13:46.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.065+0000 7f94d4ff9640 1 -- 172.21.15.27:0/456635725 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f94cc042030 con 0x7f94d8102280 2024-04-25T00:13:46.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.065+0000 7f94d4ff9640 1 --2- 172.21.15.27:0/456635725 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f94c00a2b90 0x7f94c00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.066+0000 7f94d6ffd640 1 --2- 172.21.15.27:0/456635725 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f94c00a2b90 0x7f94c00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.066+0000 7f94d4ff9640 1 -- 172.21.15.27:0/456635725 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f94cc0ef870 con 0x7f94d8102280 2024-04-25T00:13:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.066+0000 7f94d6ffd640 1 --2- 172.21.15.27:0/456635725 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f94c00a2b90 0x7f94c00a5050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f94d8198fc0 tx=0x7f94c8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.070+0000 7f94d4ff9640 1 -- 172.21.15.27:0/456635725 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f94cc0bc530 con 0x7f94d8102280 2024-04-25T00:13:46.192 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:13:45.920+0000 7f6967f45700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:13:46.192 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:13:45.920+0000 7f6967f45700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:13:46.192 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:13:45.920+0000 7f6967f45700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:13:46.192 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:13:45.920+0000 7f6967f45700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:13:46.192 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:13:45 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:13:45.920+0000 7f6967f45700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:13:46.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.285+0000 7f94ddcba640 1 -- 172.21.15.27:0/456635725 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f94d8061c10 con 0x7f94c00a2b90 2024-04-25T00:13:46.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.312+0000 7f94d4ff9640 1 -- 172.21.15.27:0/456635725 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f94d8061c10 con 0x7f94c00a2b90 2024-04-25T00:13:46.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.314+0000 7f94ddcba640 1 -- 172.21.15.27:0/456635725 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f94c00a2b90 msgr2=0x7f94c00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:46.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.314+0000 7f94ddcba640 1 --2- 172.21.15.27:0/456635725 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f94c00a2b90 0x7f94c00a5050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f94d8198fc0 tx=0x7f94c8005f90 comp rx=0 tx=0).stop 2024-04-25T00:13:46.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.314+0000 7f94ddcba640 1 -- 172.21.15.27:0/456635725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d8102280 msgr2=0x7f94d8197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:46.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.314+0000 7f94ddcba640 1 --2- 172.21.15.27:0/456635725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d8102280 0x7f94d8197ab0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f94cc009230 tx=0x7f94cc031b60 comp rx=0 tx=0).stop 2024-04-25T00:13:46.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.314+0000 7f94ddcba640 1 -- 172.21.15.27:0/456635725 shutdown_connections 2024-04-25T00:13:46.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.314+0000 7f94ddcba640 1 --2- 172.21.15.27:0/456635725 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f94c00a2b90 0x7f94c00a5050 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.314+0000 7f94ddcba640 1 --2- 172.21.15.27:0/456635725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94d81074e0 0x7f94d819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.314+0000 7f94ddcba640 1 --2- 172.21.15.27:0/456635725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d8104bb0 0x7f94d8197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.314+0000 7f94ddcba640 1 --2- 172.21.15.27:0/456635725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d8102280 0x7f94d8197ab0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.314+0000 7f94ddcba640 1 -- 172.21.15.27:0/456635725 >> 172.21.15.27:0/456635725 conn(0x7f94d80fbc30 msgr2=0x7f94d8102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:46.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.315+0000 7f94ddcba640 1 -- 172.21.15.27:0/456635725 shutdown_connections 2024-04-25T00:13:46.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.315+0000 7f94ddcba640 1 -- 172.21.15.27:0/456635725 wait complete. 2024-04-25T00:13:46.327 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-25T00:13:46.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.434+0000 7fd6f3370640 1 -- 172.21.15.27:0/698924314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6ec101860 msgr2=0x7fd6ec101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:46.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.434+0000 7fd6f3370640 1 --2- 172.21.15.27:0/698924314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6ec101860 0x7fd6ec101ce0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009900 tx=0x7fd6e002a410 comp rx=0 tx=0).stop 2024-04-25T00:13:46.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.434+0000 7fd6f3370640 1 -- 172.21.15.27:0/698924314 shutdown_connections 2024-04-25T00:13:46.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.434+0000 7fd6f3370640 1 --2- 172.21.15.27:0/698924314 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6ec102220 0x7fd6ec108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.434+0000 7fd6f3370640 1 --2- 172.21.15.27:0/698924314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6ec101860 0x7fd6ec101ce0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.434+0000 7fd6f3370640 1 --2- 172.21.15.27:0/698924314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6ec100670 0x7fd6ec100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.434+0000 7fd6f3370640 1 -- 172.21.15.27:0/698924314 >> 172.21.15.27:0/698924314 conn(0x7fd6ec0fbbe0 msgr2=0x7fd6ec0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:46.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.434+0000 7fd6f3370640 1 -- 172.21.15.27:0/698924314 shutdown_connections 2024-04-25T00:13:46.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.434+0000 7fd6f3370640 1 -- 172.21.15.27:0/698924314 wait complete. 2024-04-25T00:13:46.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.434+0000 7fd6f3370640 1 Processor -- start 2024-04-25T00:13:46.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.434+0000 7fd6f3370640 1 -- start start 2024-04-25T00:13:46.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f3370640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6ec100670 0x7fd6ec197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f3370640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6ec101860 0x7fd6ec198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f3370640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6ec102220 0x7fd6ec19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f3370640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ec10c300 con 0x7fd6ec101860 2024-04-25T00:13:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f3370640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ec10c180 con 0x7fd6ec100670 2024-04-25T00:13:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f3370640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ec10c480 con 0x7fd6ec102220 2024-04-25T00:13:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f090b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6ec101860 0x7fd6ec198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f090b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6ec101860 0x7fd6ec198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39848/0 (socket says 172.21.15.27:39848) 2024-04-25T00:13:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f090b640 1 -- 172.21.15.27:0/3633297449 learned_addr learned my addr 172.21.15.27:0/3633297449 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f110c640 1 --2- 172.21.15.27:0/3633297449 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6ec100670 0x7fd6ec197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f190d640 1 --2- 172.21.15.27:0/3633297449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6ec102220 0x7fd6ec19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f090b640 1 -- 172.21.15.27:0/3633297449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6ec102220 msgr2=0x7fd6ec19f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f090b640 1 --2- 172.21.15.27:0/3633297449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6ec102220 0x7fd6ec19f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f090b640 1 -- 172.21.15.27:0/3633297449 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6ec100670 msgr2=0x7fd6ec197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:46.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f090b640 1 --2- 172.21.15.27:0/3633297449 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6ec100670 0x7fd6ec197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f090b640 1 -- 172.21.15.27:0/3633297449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6ec19f710 con 0x7fd6ec101860 2024-04-25T00:13:46.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f090b640 1 --2- 172.21.15.27:0/3633297449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6ec101860 0x7fd6ec198030 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009230 tx=0x7fd6e0031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:46.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6de7fc640 1 -- 172.21.15.27:0/3633297449 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e003f030 con 0x7fd6ec101860 2024-04-25T00:13:46.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f3370640 1 -- 172.21.15.27:0/3633297449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6ec19f9a0 con 0x7fd6ec101860 2024-04-25T00:13:46.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6de7fc640 1 -- 172.21.15.27:0/3633297449 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6e0003ad0 con 0x7fd6ec101860 2024-04-25T00:13:46.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6de7fc640 1 -- 172.21.15.27:0/3633297449 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0035070 con 0x7fd6ec101860 2024-04-25T00:13:46.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.435+0000 7fd6f3370640 1 -- 172.21.15.27:0/3633297449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6ec19fee0 con 0x7fd6ec101860 2024-04-25T00:13:46.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.436+0000 7fd6f3370640 1 -- 172.21.15.27:0/3633297449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6ec1a0110 con 0x7fd6ec101860 2024-04-25T00:13:46.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.438+0000 7fd6de7fc640 1 -- 172.21.15.27:0/3633297449 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd6e0042030 con 0x7fd6ec101860 2024-04-25T00:13:46.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.438+0000 7fd6de7fc640 1 --2- 172.21.15.27:0/3633297449 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd6d80a2b90 0x7fd6d80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:46.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.439+0000 7fd6f110c640 1 --2- 172.21.15.27:0/3633297449 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd6d80a2b90 0x7fd6d80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:46.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.439+0000 7fd6de7fc640 1 -- 172.21.15.27:0/3633297449 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd6e00ef790 con 0x7fd6ec101860 2024-04-25T00:13:46.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.439+0000 7fd6f110c640 1 --2- 172.21.15.27:0/3633297449 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd6d80a2b90 0x7fd6d80a5050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80058c0 tx=0x7fd6e800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:46.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.443+0000 7fd6de7fc640 1 -- 172.21.15.27:0/3633297449 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd6e00bbe60 con 0x7fd6ec101860 2024-04-25T00:13:46.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.668+0000 7fd6f3370640 1 -- 172.21.15.27:0/3633297449 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fd6ec0fdde0 con 0x7fd6d80a2b90 2024-04-25T00:13:46.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.673+0000 7fd6de7fc640 1 -- 172.21.15.27:0/3633297449 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fd6ec0fdde0 con 0x7fd6d80a2b90 2024-04-25T00:13:46.673 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/qa/sv_0/b1ee4ef1-3103-4f0b-9ffb-7f9a2869af95 2024-04-25T00:13:46.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.675+0000 7fd6f3370640 1 -- 172.21.15.27:0/3633297449 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd6d80a2b90 msgr2=0x7fd6d80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:46.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.675+0000 7fd6f3370640 1 --2- 172.21.15.27:0/3633297449 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd6d80a2b90 0x7fd6d80a5050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80058c0 tx=0x7fd6e800a2d0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.675+0000 7fd6f3370640 1 -- 172.21.15.27:0/3633297449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6ec101860 msgr2=0x7fd6ec198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:46.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.675+0000 7fd6f3370640 1 --2- 172.21.15.27:0/3633297449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6ec101860 0x7fd6ec198030 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009230 tx=0x7fd6e0031b60 comp rx=0 tx=0).stop 2024-04-25T00:13:46.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.675+0000 7fd6f3370640 1 -- 172.21.15.27:0/3633297449 shutdown_connections 2024-04-25T00:13:46.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.675+0000 7fd6f3370640 1 --2- 172.21.15.27:0/3633297449 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd6d80a2b90 0x7fd6d80a5050 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.675+0000 7fd6f3370640 1 --2- 172.21.15.27:0/3633297449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6ec102220 0x7fd6ec19f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.675+0000 7fd6f3370640 1 --2- 172.21.15.27:0/3633297449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6ec101860 0x7fd6ec198030 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.675+0000 7fd6f3370640 1 --2- 172.21.15.27:0/3633297449 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6ec100670 0x7fd6ec197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.675+0000 7fd6f3370640 1 -- 172.21.15.27:0/3633297449 >> 172.21.15.27:0/3633297449 conn(0x7fd6ec0fbbe0 msgr2=0x7fd6ec0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:46.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.675+0000 7fd6f3370640 1 -- 172.21.15.27:0/3633297449 shutdown_connections 2024-04-25T00:13:46.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.676+0000 7fd6f3370640 1 -- 172.21.15.27:0/3633297449 wait complete. 2024-04-25T00:13:46.686 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-25T00:13:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.793+0000 7f23f97f9640 1 -- 172.21.15.27:0/2977249441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f4101870 msgr2=0x7f23f4101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.793+0000 7f23f97f9640 1 --2- 172.21.15.27:0/2977249441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f4101870 0x7f23f4101cf0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009900 tx=0x7f23e802a410 comp rx=0 tx=0).stop 2024-04-25T00:13:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.793+0000 7f23f97f9640 1 -- 172.21.15.27:0/2977249441 shutdown_connections 2024-04-25T00:13:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.793+0000 7f23f97f9640 1 --2- 172.21.15.27:0/2977249441 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23f4102230 0x7f23f4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.793+0000 7f23f97f9640 1 --2- 172.21.15.27:0/2977249441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f4101870 0x7f23f4101cf0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.793+0000 7f23f97f9640 1 --2- 172.21.15.27:0/2977249441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23f4100680 0x7f23f4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.793+0000 7f23f97f9640 1 -- 172.21.15.27:0/2977249441 >> 172.21.15.27:0/2977249441 conn(0x7f23f40fbbf0 msgr2=0x7f23f40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.793+0000 7f23f97f9640 1 -- 172.21.15.27:0/2977249441 shutdown_connections 2024-04-25T00:13:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.793+0000 7f23f97f9640 1 -- 172.21.15.27:0/2977249441 wait complete. 2024-04-25T00:13:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.793+0000 7f23f97f9640 1 Processor -- start 2024-04-25T00:13:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f97f9640 1 -- start start 2024-04-25T00:13:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f97f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23f4100680 0x7f23f410b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f97f9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23f4101870 0x7f23f4109640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f97f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f4102230 0x7f23f4109b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f97f9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f23f410d420 con 0x7f23f4102230 2024-04-25T00:13:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f97f9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f23f410d2a0 con 0x7f23f4101870 2024-04-25T00:13:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f97f9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f23f410d5a0 con 0x7f23f4100680 2024-04-25T00:13:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f4102230 0x7f23f4109b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f4102230 0x7f23f4109b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51862/0 (socket says 172.21.15.27:51862) 2024-04-25T00:13:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f37fe640 1 -- 172.21.15.27:0/3887222493 learned_addr learned my addr 172.21.15.27:0/3887222493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f27fc640 1 --2- 172.21.15.27:0/3887222493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23f4101870 0x7f23f4109640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f2ffd640 1 --2- 172.21.15.27:0/3887222493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23f4100680 0x7f23f410b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f37fe640 1 -- 172.21.15.27:0/3887222493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23f4100680 msgr2=0x7f23f410b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f37fe640 1 --2- 172.21.15.27:0/3887222493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23f4100680 0x7f23f410b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f37fe640 1 -- 172.21.15.27:0/3887222493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23f4101870 msgr2=0x7f23f4109640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f37fe640 1 --2- 172.21.15.27:0/3887222493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23f4101870 0x7f23f4109640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f37fe640 1 -- 172.21.15.27:0/3887222493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f410a410 con 0x7f23f4102230 2024-04-25T00:13:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.794+0000 7f23f2ffd640 1 --2- 172.21.15.27:0/3887222493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23f4100680 0x7f23f410b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.795+0000 7f23f37fe640 1 --2- 172.21.15.27:0/3887222493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f4102230 0x7f23f4109b80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00eb10 tx=0x7f23ec00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.795+0000 7f23dbfff640 1 -- 172.21.15.27:0/3887222493 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ec004b20 con 0x7f23f4102230 2024-04-25T00:13:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.795+0000 7f23dbfff640 1 -- 172.21.15.27:0/3887222493 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23ec004cc0 con 0x7f23f4102230 2024-04-25T00:13:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.795+0000 7f23f97f9640 1 -- 172.21.15.27:0/3887222493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23f41a1bf0 con 0x7f23f4102230 2024-04-25T00:13:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.795+0000 7f23dbfff640 1 -- 172.21.15.27:0/3887222493 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ec010760 con 0x7f23f4102230 2024-04-25T00:13:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.795+0000 7f23f97f9640 1 -- 172.21.15.27:0/3887222493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23f41a20c0 con 0x7f23f4102230 2024-04-25T00:13:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.796+0000 7f23f97f9640 1 -- 172.21.15.27:0/3887222493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23f4068560 con 0x7f23f4102230 2024-04-25T00:13:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.797+0000 7f23dbfff640 1 -- 172.21.15.27:0/3887222493 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f23ec005020 con 0x7f23f4102230 2024-04-25T00:13:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.797+0000 7f23dbfff640 1 --2- 172.21.15.27:0/3887222493 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f23dc0a2b90 0x7f23dc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.798+0000 7f23f2ffd640 1 --2- 172.21.15.27:0/3887222493 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f23dc0a2b90 0x7f23dc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.798+0000 7f23dbfff640 1 -- 172.21.15.27:0/3887222493 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f23ec0cf920 con 0x7f23f4102230 2024-04-25T00:13:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.798+0000 7f23f2ffd640 1 --2- 172.21.15.27:0/3887222493 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f23dc0a2b90 0x7f23dc0a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f23e40058c0 tx=0x7f23e4005d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:46.802+0000 7f23dbfff640 1 -- 172.21.15.27:0/3887222493 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f23ec09c640 con 0x7f23f4102230 2024-04-25T00:13:46.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:46 smithi027 ceph-mon[29208]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 3 op/s 2024-04-25T00:13:46.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:46 smithi027 ceph-mon[29208]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:13:46.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:46 smithi027 ceph-mon[29208]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:13:46.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:46 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] up:active 2024-04-25T00:13:46.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:46 smithi027 ceph-mon[29208]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T00:13:46.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:46 smithi027 ceph-mon[29208]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T00:13:46.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:46 smithi027 ceph-mon[29208]: Cluster is now healthy 2024-04-25T00:13:46.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:46 smithi027 ceph-mon[29208]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=l=up:active} 3 up:standby-replay 5 up:standby 2024-04-25T00:13:46.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:46 smithi027 ceph-mon[29208]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-04-25T00:13:46.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:46 smithi027 ceph-mon[29208]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-04-25T00:13:47.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:46 smithi139 ceph-mon[32833]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 3 op/s 2024-04-25T00:13:47.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:46 smithi139 ceph-mon[32833]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:13:47.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:46 smithi139 ceph-mon[32833]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:13:47.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:46 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] up:active 2024-04-25T00:13:47.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:46 smithi139 ceph-mon[32833]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T00:13:47.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:46 smithi139 ceph-mon[32833]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T00:13:47.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:46 smithi139 ceph-mon[32833]: Cluster is now healthy 2024-04-25T00:13:47.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:46 smithi139 ceph-mon[32833]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=l=up:active} 3 up:standby-replay 5 up:standby 2024-04-25T00:13:47.005 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:46 smithi139 ceph-mon[32833]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-04-25T00:13:47.005 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:46 smithi139 ceph-mon[32833]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-04-25T00:13:47.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.013+0000 7f23f97f9640 1 -- 172.21.15.27:0/3887222493 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f23f4061c10 con 0x7f23dc0a2b90 2024-04-25T00:13:47.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:46 smithi153 ceph-mon[32896]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 3 op/s 2024-04-25T00:13:47.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:46 smithi153 ceph-mon[32896]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:13:47.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:46 smithi153 ceph-mon[32896]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:13:47.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:46 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] up:active 2024-04-25T00:13:47.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:46 smithi153 ceph-mon[32896]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T00:13:47.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:46 smithi153 ceph-mon[32896]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T00:13:47.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:46 smithi153 ceph-mon[32896]: Cluster is now healthy 2024-04-25T00:13:47.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:46 smithi153 ceph-mon[32896]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=l=up:active} 3 up:standby-replay 5 up:standby 2024-04-25T00:13:47.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:46 smithi153 ceph-mon[32896]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-04-25T00:13:47.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:46 smithi153 ceph-mon[32896]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-04-25T00:13:47.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.036+0000 7f23dbfff640 1 -- 172.21.15.27:0/3887222493 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f23f4061c10 con 0x7f23dc0a2b90 2024-04-25T00:13:47.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.038+0000 7f23f97f9640 1 -- 172.21.15.27:0/3887222493 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f23dc0a2b90 msgr2=0x7f23dc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:47.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.038+0000 7f23f97f9640 1 --2- 172.21.15.27:0/3887222493 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f23dc0a2b90 0x7f23dc0a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f23e40058c0 tx=0x7f23e4005d50 comp rx=0 tx=0).stop 2024-04-25T00:13:47.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.038+0000 7f23f97f9640 1 -- 172.21.15.27:0/3887222493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f4102230 msgr2=0x7f23f4109b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:47.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.038+0000 7f23f97f9640 1 --2- 172.21.15.27:0/3887222493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f4102230 0x7f23f4109b80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00eb10 tx=0x7f23ec00c2d0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.038+0000 7f23f97f9640 1 -- 172.21.15.27:0/3887222493 shutdown_connections 2024-04-25T00:13:47.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.038+0000 7f23f97f9640 1 --2- 172.21.15.27:0/3887222493 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f23dc0a2b90 0x7f23dc0a5050 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.038+0000 7f23f97f9640 1 --2- 172.21.15.27:0/3887222493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f4102230 0x7f23f4109b80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.038+0000 7f23f97f9640 1 --2- 172.21.15.27:0/3887222493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23f4101870 0x7f23f4109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.038+0000 7f23f97f9640 1 --2- 172.21.15.27:0/3887222493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23f4100680 0x7f23f410b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.038+0000 7f23f97f9640 1 -- 172.21.15.27:0/3887222493 >> 172.21.15.27:0/3887222493 conn(0x7f23f40fbbf0 msgr2=0x7f23f4104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:47.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.038+0000 7f23f97f9640 1 -- 172.21.15.27:0/3887222493 shutdown_connections 2024-04-25T00:13:47.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.038+0000 7f23f97f9640 1 -- 172.21.15.27:0/3887222493 wait complete. 2024-04-25T00:13:47.049 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-25T00:13:47.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.156+0000 7f72e868c640 1 -- 172.21.15.27:0/1891357931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72e0107460 msgr2=0x7f72e01098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.156+0000 7f72e868c640 1 --2- 172.21.15.27:0/1891357931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72e0107460 0x7f72e01098b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009ca0 tx=0x7f72d002a8c0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.156+0000 7f72e868c640 1 -- 172.21.15.27:0/1891357931 shutdown_connections 2024-04-25T00:13:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.156+0000 7f72e868c640 1 --2- 172.21.15.27:0/1891357931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72e0107460 0x7f72e01098b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.156+0000 7f72e868c640 1 --2- 172.21.15.27:0/1891357931 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72e0104b30 0x7f72e0106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.156+0000 7f72e868c640 1 --2- 172.21.15.27:0/1891357931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72e0102200 0x7f72e01045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.156+0000 7f72e868c640 1 -- 172.21.15.27:0/1891357931 >> 172.21.15.27:0/1891357931 conn(0x7f72e00fbbf0 msgr2=0x7f72e00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.156+0000 7f72e868c640 1 -- 172.21.15.27:0/1891357931 shutdown_connections 2024-04-25T00:13:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.156+0000 7f72e868c640 1 -- 172.21.15.27:0/1891357931 wait complete. 2024-04-25T00:13:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e868c640 1 Processor -- start 2024-04-25T00:13:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e868c640 1 -- start start 2024-04-25T00:13:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e868c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72e0102200 0x7f72e0197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e868c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72e0104b30 0x7f72e0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:47.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e868c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72e0107460 0x7f72e019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:47.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e868c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f72e010d420 con 0x7f72e0104b30 2024-04-25T00:13:47.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e868c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f72e010d2a0 con 0x7f72e0107460 2024-04-25T00:13:47.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e868c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f72e010d5a0 con 0x7f72e0102200 2024-04-25T00:13:47.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e5c27640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72e0104b30 0x7f72e0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:47.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e5c27640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72e0104b30 0x7f72e0197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51866/0 (socket says 172.21.15.27:51866) 2024-04-25T00:13:47.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e5c27640 1 -- 172.21.15.27:0/608453425 learned_addr learned my addr 172.21.15.27:0/608453425 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:47.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e6c29640 1 --2- 172.21.15.27:0/608453425 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72e0107460 0x7f72e019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:47.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e6428640 1 --2- 172.21.15.27:0/608453425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72e0102200 0x7f72e0197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:47.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e5c27640 1 -- 172.21.15.27:0/608453425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72e0102200 msgr2=0x7f72e0197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:47.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e5c27640 1 --2- 172.21.15.27:0/608453425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72e0102200 0x7f72e0197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e5c27640 1 -- 172.21.15.27:0/608453425 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72e0107460 msgr2=0x7f72e019f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:47.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e5c27640 1 --2- 172.21.15.27:0/608453425 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72e0107460 0x7f72e019f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e5c27640 1 -- 172.21.15.27:0/608453425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e019f6b0 con 0x7f72e0104b30 2024-04-25T00:13:47.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e6428640 1 --2- 172.21.15.27:0/608453425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72e0102200 0x7f72e0197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:47.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72e5c27640 1 --2- 172.21.15.27:0/608453425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72e0104b30 0x7f72e0197fd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f72d400e9c0 tx=0x7f72d400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:47.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.157+0000 7f72df7fe640 1 -- 172.21.15.27:0/608453425 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d400cd80 con 0x7f72e0104b30 2024-04-25T00:13:47.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.158+0000 7f72df7fe640 1 -- 172.21.15.27:0/608453425 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72d4005500 con 0x7f72e0104b30 2024-04-25T00:13:47.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.158+0000 7f72e868c640 1 -- 172.21.15.27:0/608453425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e019f9a0 con 0x7f72e0104b30 2024-04-25T00:13:47.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.158+0000 7f72df7fe640 1 -- 172.21.15.27:0/608453425 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d4010690 con 0x7f72e0104b30 2024-04-25T00:13:47.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.158+0000 7f72e868c640 1 -- 172.21.15.27:0/608453425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e019ff40 con 0x7f72e0104b30 2024-04-25T00:13:47.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.158+0000 7f72e868c640 1 -- 172.21.15.27:0/608453425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72e00fd6e0 con 0x7f72e0104b30 2024-04-25T00:13:47.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.160+0000 7f72df7fe640 1 -- 172.21.15.27:0/608453425 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f72d40046c0 con 0x7f72e0104b30 2024-04-25T00:13:47.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.161+0000 7f72df7fe640 1 --2- 172.21.15.27:0/608453425 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f72c80a2c10 0x7f72c80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:47.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.161+0000 7f72e6428640 1 --2- 172.21.15.27:0/608453425 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f72c80a2c10 0x7f72c80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:47.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.161+0000 7f72df7fe640 1 -- 172.21.15.27:0/608453425 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f72d40cf820 con 0x7f72e0104b30 2024-04-25T00:13:47.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.161+0000 7f72e6428640 1 --2- 172.21.15.27:0/608453425 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f72c80a2c10 0x7f72c80a50d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009b40 tx=0x7f72d8005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:47.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.164+0000 7f72df7fe640 1 -- 172.21.15.27:0/608453425 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f72d4053740 con 0x7f72e0104b30 2024-04-25T00:13:47.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.381+0000 7f72e868c640 1 -- 172.21.15.27:0/608453425 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f72e0061b80 con 0x7f72c80a2c10 2024-04-25T00:13:47.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.386+0000 7f72df7fe640 1 -- 172.21.15.27:0/608453425 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f72e0061b80 con 0x7f72c80a2c10 2024-04-25T00:13:47.386 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/qa/sv_1/c29414da-1300-48ce-9002-67b724424f7f 2024-04-25T00:13:47.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.388+0000 7f72e868c640 1 -- 172.21.15.27:0/608453425 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f72c80a2c10 msgr2=0x7f72c80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:47.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.388+0000 7f72e868c640 1 --2- 172.21.15.27:0/608453425 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f72c80a2c10 0x7f72c80a50d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009b40 tx=0x7f72d8005e50 comp rx=0 tx=0).stop 2024-04-25T00:13:47.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.388+0000 7f72e868c640 1 -- 172.21.15.27:0/608453425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72e0104b30 msgr2=0x7f72e0197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:47.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.388+0000 7f72e868c640 1 --2- 172.21.15.27:0/608453425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72e0104b30 0x7f72e0197fd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f72d400e9c0 tx=0x7f72d400ee80 comp rx=0 tx=0).stop 2024-04-25T00:13:47.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.388+0000 7f72e868c640 1 -- 172.21.15.27:0/608453425 shutdown_connections 2024-04-25T00:13:47.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.388+0000 7f72e868c640 1 --2- 172.21.15.27:0/608453425 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f72c80a2c10 0x7f72c80a50d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.388+0000 7f72e868c640 1 --2- 172.21.15.27:0/608453425 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72e0107460 0x7f72e019f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.388+0000 7f72e868c640 1 --2- 172.21.15.27:0/608453425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72e0104b30 0x7f72e0197fd0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.388+0000 7f72e868c640 1 --2- 172.21.15.27:0/608453425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72e0102200 0x7f72e0197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.388+0000 7f72e868c640 1 -- 172.21.15.27:0/608453425 >> 172.21.15.27:0/608453425 conn(0x7f72e00fbbf0 msgr2=0x7f72e0102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:47.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.388+0000 7f72e868c640 1 -- 172.21.15.27:0/608453425 shutdown_connections 2024-04-25T00:13:47.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.388+0000 7f72e868c640 1 -- 172.21.15.27:0/608453425 wait complete. 2024-04-25T00:13:47.399 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:47.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.501+0000 7f5930231640 1 -- 172.21.15.27:0/1919124847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5928102220 msgr2=0x7f5928104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:47.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.501+0000 7f5930231640 1 --2- 172.21.15.27:0/1919124847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5928102220 0x7f5928104610 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5924009900 tx=0x7f592402a410 comp rx=0 tx=0).stop 2024-04-25T00:13:47.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.501+0000 7f5930231640 1 -- 172.21.15.27:0/1919124847 shutdown_connections 2024-04-25T00:13:47.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.501+0000 7f5930231640 1 --2- 172.21.15.27:0/1919124847 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5928107480 0x7f59281098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.501+0000 7f5930231640 1 --2- 172.21.15.27:0/1919124847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5928104b50 0x7f5928106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.501+0000 7f5930231640 1 --2- 172.21.15.27:0/1919124847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5928102220 0x7f5928104610 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.501+0000 7f5930231640 1 -- 172.21.15.27:0/1919124847 >> 172.21.15.27:0/1919124847 conn(0x7f59280fbc30 msgr2=0x7f59280fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:47.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.501+0000 7f5930231640 1 -- 172.21.15.27:0/1919124847 shutdown_connections 2024-04-25T00:13:47.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.501+0000 7f5930231640 1 -- 172.21.15.27:0/1919124847 wait complete. 2024-04-25T00:13:47.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f5930231640 1 Processor -- start 2024-04-25T00:13:47.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f5930231640 1 -- start start 2024-04-25T00:13:47.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f5930231640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5928102220 0x7f5928197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:47.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f5930231640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5928104b50 0x7f5928198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:47.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f5930231640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5928107480 0x7f592819f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:47.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f5930231640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f592810d4f0 con 0x7f5928102220 2024-04-25T00:13:47.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f5930231640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f592810d370 con 0x7f5928107480 2024-04-25T00:13:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f5930231640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f592810d670 con 0x7f5928104b50 2024-04-25T00:13:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f592dfcd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5928102220 0x7f5928197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f592dfcd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5928102220 0x7f5928197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51876/0 (socket says 172.21.15.27:51876) 2024-04-25T00:13:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f592dfcd640 1 -- 172.21.15.27:0/34035507 learned_addr learned my addr 172.21.15.27:0/34035507 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f592e7ce640 1 --2- 172.21.15.27:0/34035507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5928107480 0x7f592819f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f592d7cc640 1 --2- 172.21.15.27:0/34035507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5928104b50 0x7f5928198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f592dfcd640 1 -- 172.21.15.27:0/34035507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5928104b50 msgr2=0x7f5928198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f592dfcd640 1 --2- 172.21.15.27:0/34035507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5928104b50 0x7f5928198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f592dfcd640 1 -- 172.21.15.27:0/34035507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5928107480 msgr2=0x7f592819f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:47.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f592dfcd640 1 --2- 172.21.15.27:0/34035507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5928107480 0x7f592819f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f592dfcd640 1 -- 172.21.15.27:0/34035507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f592819f750 con 0x7f5928102220 2024-04-25T00:13:47.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.502+0000 7f592dfcd640 1 --2- 172.21.15.27:0/34035507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5928102220 0x7f5928197b30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5924031440 tx=0x7f5924032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:47.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.503+0000 7f591effd640 1 -- 172.21.15.27:0/34035507 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f592403f030 con 0x7f5928102220 2024-04-25T00:13:47.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.503+0000 7f5930231640 1 -- 172.21.15.27:0/34035507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f592819f9e0 con 0x7f5928102220 2024-04-25T00:13:47.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.503+0000 7f591effd640 1 -- 172.21.15.27:0/34035507 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5924003ce0 con 0x7f5928102220 2024-04-25T00:13:47.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.503+0000 7f591effd640 1 -- 172.21.15.27:0/34035507 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5924035070 con 0x7f5928102220 2024-04-25T00:13:47.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.503+0000 7f5930231640 1 -- 172.21.15.27:0/34035507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f592819ff50 con 0x7f5928102220 2024-04-25T00:13:47.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.504+0000 7f5930231640 1 -- 172.21.15.27:0/34035507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59280fd720 con 0x7f5928102220 2024-04-25T00:13:47.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.505+0000 7f591effd640 1 -- 172.21.15.27:0/34035507 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5924042030 con 0x7f5928102220 2024-04-25T00:13:47.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.505+0000 7f591effd640 1 --2- 172.21.15.27:0/34035507 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f59140a2b90 0x7f59140a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:47.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.506+0000 7f592d7cc640 1 --2- 172.21.15.27:0/34035507 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f59140a2b90 0x7f59140a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:47.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.506+0000 7f591effd640 1 -- 172.21.15.27:0/34035507 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f59240ef840 con 0x7f5928102220 2024-04-25T00:13:47.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.506+0000 7f592d7cc640 1 --2- 172.21.15.27:0/34035507 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f59140a2b90 0x7f59140a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5928199040 tx=0x7f5918008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:47.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.510+0000 7f591effd640 1 -- 172.21.15.27:0/34035507 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f59240bc560 con 0x7f5928102220 2024-04-25T00:13:47.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:47 smithi027 ceph-mon[29208]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:13:47.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:47 smithi027 ceph-mon[29208]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:13:47.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:47 smithi027 ceph-mon[29208]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:13:47.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:47 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] up:active 2024-04-25T00:13:47.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:47 smithi027 ceph-mon[29208]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-25T00:13:47.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:47 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:13:47.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.742+0000 7f5930231640 1 -- 172.21.15.27:0/34035507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5928062340 con 0x7f5928102220 2024-04-25T00:13:47.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.743+0000 7f591effd640 1 -- 172.21.15.27:0/34035507 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11944 (secure 0 0 0) 0x7f59240bbe60 con 0x7f5928102220 2024-04-25T00:13:47.743 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:47.744 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:13:47.575422+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24428,24496,24521,24430,24514]},"id":1}]} 2024-04-25T00:13:47.744 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 28 2024-04-25T00:13:47.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.745+0000 7f5930231640 1 -- 172.21.15.27:0/34035507 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f59140a2b90 msgr2=0x7f59140a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:47.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.745+0000 7f5930231640 1 --2- 172.21.15.27:0/34035507 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f59140a2b90 0x7f59140a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5928199040 tx=0x7f5918008040 comp rx=0 tx=0).stop 2024-04-25T00:13:47.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.745+0000 7f5930231640 1 -- 172.21.15.27:0/34035507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5928102220 msgr2=0x7f5928197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:47.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.745+0000 7f5930231640 1 --2- 172.21.15.27:0/34035507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5928102220 0x7f5928197b30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5924031440 tx=0x7f5924032ce0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.745+0000 7f5930231640 1 -- 172.21.15.27:0/34035507 shutdown_connections 2024-04-25T00:13:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.745+0000 7f5930231640 1 --2- 172.21.15.27:0/34035507 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f59140a2b90 0x7f59140a5050 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.745+0000 7f5930231640 1 --2- 172.21.15.27:0/34035507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5928107480 0x7f592819f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.745+0000 7f5930231640 1 --2- 172.21.15.27:0/34035507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5928104b50 0x7f5928198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.745+0000 7f5930231640 1 --2- 172.21.15.27:0/34035507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5928102220 0x7f5928197b30 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.745+0000 7f5930231640 1 -- 172.21.15.27:0/34035507 >> 172.21.15.27:0/34035507 conn(0x7f59280fbc30 msgr2=0x7f5928102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.745+0000 7f5930231640 1 -- 172.21.15.27:0/34035507 shutdown_connections 2024-04-25T00:13:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.745+0000 7f5930231640 1 -- 172.21.15.27:0/34035507 wait complete. 2024-04-25T00:13:47.756 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:13:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.858+0000 7f183904c640 1 -- 172.21.15.27:0/1165142414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1834101870 msgr2=0x7f1834101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.858+0000 7f183904c640 1 --2- 172.21.15.27:0/1165142414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1834101870 0x7f1834101cf0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1824009900 tx=0x7f182402a410 comp rx=0 tx=0).stop 2024-04-25T00:13:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.858+0000 7f183904c640 1 -- 172.21.15.27:0/1165142414 shutdown_connections 2024-04-25T00:13:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.858+0000 7f183904c640 1 --2- 172.21.15.27:0/1165142414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1834102230 0x7f1834108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.858+0000 7f183904c640 1 --2- 172.21.15.27:0/1165142414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1834101870 0x7f1834101cf0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.858+0000 7f183904c640 1 --2- 172.21.15.27:0/1165142414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1834100680 0x7f1834100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.858+0000 7f183904c640 1 -- 172.21.15.27:0/1165142414 >> 172.21.15.27:0/1165142414 conn(0x7f18340fbbf0 msgr2=0x7f18340fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.858+0000 7f183904c640 1 -- 172.21.15.27:0/1165142414 shutdown_connections 2024-04-25T00:13:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.858+0000 7f183904c640 1 -- 172.21.15.27:0/1165142414 wait complete. 2024-04-25T00:13:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.859+0000 7f183904c640 1 Processor -- start 2024-04-25T00:13:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.859+0000 7f183904c640 1 -- start start 2024-04-25T00:13:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.859+0000 7f183904c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1834100680 0x7f18340694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.859+0000 7f183904c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1834101870 0x7f183419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.859+0000 7f183904c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1834102230 0x7f183419ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f183904c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f183410c3c0 con 0x7f1834102230 2024-04-25T00:13:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f183904c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f183410c240 con 0x7f1834100680 2024-04-25T00:13:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f183904c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f183410c540 con 0x7f1834101870 2024-04-25T00:13:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f183359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1834102230 0x7f183419ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f1832d9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1834100680 0x7f18340694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f183359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1834102230 0x7f183419ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51888/0 (socket says 172.21.15.27:51888) 2024-04-25T00:13:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f183359e640 1 -- 172.21.15.27:0/1813188172 learned_addr learned my addr 172.21.15.27:0/1813188172 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f183259c640 1 --2- 172.21.15.27:0/1813188172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1834101870 0x7f183419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f1832d9d640 1 -- 172.21.15.27:0/1813188172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1834101870 msgr2=0x7f183419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f1832d9d640 1 --2- 172.21.15.27:0/1813188172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1834101870 0x7f183419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f1832d9d640 1 -- 172.21.15.27:0/1813188172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1834102230 msgr2=0x7f183419ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f1832d9d640 1 --2- 172.21.15.27:0/1813188172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1834102230 0x7f183419ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f1832d9d640 1 -- 172.21.15.27:0/1813188172 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f183419f750 con 0x7f1834100680 2024-04-25T00:13:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f183359e640 1 --2- 172.21.15.27:0/1813188172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1834102230 0x7f183419ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:13:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f1832d9d640 1 --2- 172.21.15.27:0/1813188172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1834100680 0x7f18340694a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f181c00b3c0 tx=0x7f181c00b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f182bfff640 1 -- 172.21.15.27:0/1813188172 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c011830 con 0x7f1834100680 2024-04-25T00:13:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f183904c640 1 -- 172.21.15.27:0/1813188172 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f183419fa40 con 0x7f1834100680 2024-04-25T00:13:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f182bfff640 1 -- 172.21.15.27:0/1813188172 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f181c016070 con 0x7f1834100680 2024-04-25T00:13:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.860+0000 7f182bfff640 1 -- 172.21.15.27:0/1813188172 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c00f2e0 con 0x7f1834100680 2024-04-25T00:13:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.861+0000 7f183904c640 1 -- 172.21.15.27:0/1813188172 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f183419ffb0 con 0x7f1834100680 2024-04-25T00:13:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.862+0000 7f183904c640 1 -- 172.21.15.27:0/1813188172 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18341079a0 con 0x7f1834100680 2024-04-25T00:13:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.863+0000 7f182bfff640 1 -- 172.21.15.27:0/1813188172 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f181c0119d0 con 0x7f1834100680 2024-04-25T00:13:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.863+0000 7f182bfff640 1 --2- 172.21.15.27:0/1813188172 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f18200a2b90 0x7f18200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.864+0000 7f183259c640 1 --2- 172.21.15.27:0/1813188172 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f18200a2b90 0x7f18200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.864+0000 7f182bfff640 1 -- 172.21.15.27:0/1813188172 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f181c0ceee0 con 0x7f1834100680 2024-04-25T00:13:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.864+0000 7f183259c640 1 --2- 172.21.15.27:0/1813188172 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f18200a2b90 0x7f18200a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1824031440 tx=0x7f18240315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:47.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:47.871+0000 7f182bfff640 1 -- 172.21.15.27:0/1813188172 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f181c09bba0 con 0x7f1834100680 2024-04-25T00:13:48.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:47 smithi139 ceph-mon[32833]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:13:48.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:47 smithi139 ceph-mon[32833]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:13:48.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:47 smithi139 ceph-mon[32833]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:13:48.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:47 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] up:active 2024-04-25T00:13:48.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:47 smithi139 ceph-mon[32833]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-25T00:13:48.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:47 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:13:48.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:47 smithi153 ceph-mon[32896]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:13:48.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:47 smithi153 ceph-mon[32896]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:13:48.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:47 smithi153 ceph-mon[32896]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:13:48.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:47 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] up:active 2024-04-25T00:13:48.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:47 smithi153 ceph-mon[32896]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-25T00:13:48.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:47 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:13:48.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.056+0000 7f183904c640 1 -- 172.21.15.27:0/1813188172 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f183402d390 con 0x7f1834100680 2024-04-25T00:13:48.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.058+0000 7f182bfff640 1 -- 172.21.15.27:0/1813188172 <== mon.2 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24117 (secure 0 0 0) 0x7f181c09b4a0 con 0x7f1834100680 2024-04-25T00:13:48.058 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:48.059 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:13:23.935081+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:13:48.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.060+0000 7f183904c640 1 -- 172.21.15.27:0/1813188172 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f18200a2b90 msgr2=0x7f18200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.060+0000 7f183904c640 1 --2- 172.21.15.27:0/1813188172 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f18200a2b90 0x7f18200a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1824031440 tx=0x7f18240315b0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.060+0000 7f183904c640 1 -- 172.21.15.27:0/1813188172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1834100680 msgr2=0x7f18340694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.060+0000 7f183904c640 1 --2- 172.21.15.27:0/1813188172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1834100680 0x7f18340694a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f181c00b3c0 tx=0x7f181c00b880 comp rx=0 tx=0).stop 2024-04-25T00:13:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.060+0000 7f183904c640 1 -- 172.21.15.27:0/1813188172 shutdown_connections 2024-04-25T00:13:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.060+0000 7f183904c640 1 --2- 172.21.15.27:0/1813188172 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f18200a2b90 0x7f18200a5050 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.060+0000 7f183904c640 1 --2- 172.21.15.27:0/1813188172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1834102230 0x7f183419ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.060+0000 7f183904c640 1 --2- 172.21.15.27:0/1813188172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1834101870 0x7f183419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.060+0000 7f183904c640 1 --2- 172.21.15.27:0/1813188172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1834100680 0x7f18340694a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.060+0000 7f183904c640 1 -- 172.21.15.27:0/1813188172 >> 172.21.15.27:0/1813188172 conn(0x7f18340fbbf0 msgr2=0x7f1834104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.060+0000 7f183904c640 1 -- 172.21.15.27:0/1813188172 shutdown_connections 2024-04-25T00:13:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.060+0000 7f183904c640 1 -- 172.21.15.27:0/1813188172 wait complete. 2024-04-25T00:13:48.072 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T00:13:48.072 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:13:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.174+0000 7fcbb586b640 1 -- 172.21.15.27:0/2834258160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb0101870 msgr2=0x7fcbb0101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.174+0000 7fcbb586b640 1 --2- 172.21.15.27:0/2834258160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb0101870 0x7fcbb0101cf0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009900 tx=0x7fcba402a410 comp rx=0 tx=0).stop 2024-04-25T00:13:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.174+0000 7fcbb586b640 1 -- 172.21.15.27:0/2834258160 shutdown_connections 2024-04-25T00:13:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.174+0000 7fcbb586b640 1 --2- 172.21.15.27:0/2834258160 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcbb0102230 0x7fcbb0108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.174+0000 7fcbb586b640 1 --2- 172.21.15.27:0/2834258160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb0101870 0x7fcbb0101cf0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.174+0000 7fcbb586b640 1 --2- 172.21.15.27:0/2834258160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb0100680 0x7fcbb0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.174+0000 7fcbb586b640 1 -- 172.21.15.27:0/2834258160 >> 172.21.15.27:0/2834258160 conn(0x7fcbb00fbbf0 msgr2=0x7fcbb00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.174+0000 7fcbb586b640 1 -- 172.21.15.27:0/2834258160 shutdown_connections 2024-04-25T00:13:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.174+0000 7fcbb586b640 1 -- 172.21.15.27:0/2834258160 wait complete. 2024-04-25T00:13:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.175+0000 7fcbb586b640 1 Processor -- start 2024-04-25T00:13:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.175+0000 7fcbb586b640 1 -- start start 2024-04-25T00:13:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.175+0000 7fcbb586b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb0100680 0x7fcbb0197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.175+0000 7fcbb586b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb0101870 0x7fcbb0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.175+0000 7fcbb586b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcbb0102230 0x7fcbb019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.175+0000 7fcbb586b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb010d420 con 0x7fcbb0101870 2024-04-25T00:13:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.175+0000 7fcbb586b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb010d2a0 con 0x7fcbb0102230 2024-04-25T00:13:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.175+0000 7fcbb586b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb010d5a0 con 0x7fcbb0100680 2024-04-25T00:13:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.175+0000 7fcbae7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb0101870 0x7fcbb0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.175+0000 7fcbae7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb0101870 0x7fcbb0197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51916/0 (socket says 172.21.15.27:51916) 2024-04-25T00:13:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.175+0000 7fcbae7fc640 1 -- 172.21.15.27:0/1941080558 learned_addr learned my addr 172.21.15.27:0/1941080558 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:48.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.176+0000 7fcbaf7fe640 1 --2- 172.21.15.27:0/1941080558 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcbb0102230 0x7fcbb019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:48.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.176+0000 7fcbaeffd640 1 --2- 172.21.15.27:0/1941080558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb0100680 0x7fcbb0197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:48.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.176+0000 7fcbae7fc640 1 -- 172.21.15.27:0/1941080558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb0100680 msgr2=0x7fcbb0197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:48.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.176+0000 7fcbae7fc640 1 --2- 172.21.15.27:0/1941080558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb0100680 0x7fcbb0197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.176+0000 7fcbae7fc640 1 -- 172.21.15.27:0/1941080558 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcbb0102230 msgr2=0x7fcbb019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:48.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.176+0000 7fcbae7fc640 1 --2- 172.21.15.27:0/1941080558 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcbb0102230 0x7fcbb019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.176+0000 7fcbae7fc640 1 -- 172.21.15.27:0/1941080558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb019f6c0 con 0x7fcbb0101870 2024-04-25T00:13:48.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.176+0000 7fcbae7fc640 1 --2- 172.21.15.27:0/1941080558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb0101870 0x7fcbb0197fe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009230 tx=0x7fcba4031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:48.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.176+0000 7fcbb4869640 1 -- 172.21.15.27:0/1941080558 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba403f030 con 0x7fcbb0101870 2024-04-25T00:13:48.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.176+0000 7fcbb4869640 1 -- 172.21.15.27:0/1941080558 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcba4003ad0 con 0x7fcbb0101870 2024-04-25T00:13:48.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.176+0000 7fcbb586b640 1 -- 172.21.15.27:0/1941080558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbb019f950 con 0x7fcbb0101870 2024-04-25T00:13:48.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.176+0000 7fcbb4869640 1 -- 172.21.15.27:0/1941080558 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba4035070 con 0x7fcbb0101870 2024-04-25T00:13:48.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.176+0000 7fcbb586b640 1 -- 172.21.15.27:0/1941080558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbb019fe40 con 0x7fcbb0101870 2024-04-25T00:13:48.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.177+0000 7fcbb586b640 1 -- 172.21.15.27:0/1941080558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbb01079a0 con 0x7fcbb0101870 2024-04-25T00:13:48.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.178+0000 7fcbb4869640 1 -- 172.21.15.27:0/1941080558 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcba4042030 con 0x7fcbb0101870 2024-04-25T00:13:48.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.179+0000 7fcbb4869640 1 --2- 172.21.15.27:0/1941080558 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fcb980a2b90 0x7fcb980a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:48.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.179+0000 7fcbaeffd640 1 --2- 172.21.15.27:0/1941080558 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fcb980a2b90 0x7fcb980a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:48.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.179+0000 7fcbb4869640 1 -- 172.21.15.27:0/1941080558 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fcba40ef840 con 0x7fcbb0101870 2024-04-25T00:13:48.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.180+0000 7fcbaeffd640 1 --2- 172.21.15.27:0/1941080558 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fcb980a2b90 0x7fcb980a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcbb01016d0 tx=0x7fcba0009870 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:48.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.184+0000 7fcbb4869640 1 -- 172.21.15.27:0/1941080558 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcba40bc560 con 0x7fcbb0101870 2024-04-25T00:13:48.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.367+0000 7fcbb586b640 1 -- 172.21.15.27:0/1941080558 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcbb0061b80 con 0x7fcb980a2b90 2024-04-25T00:13:48.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.378+0000 7fcbb4869640 1 -- 172.21.15.27:0/1941080558 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340525 (secure 0 0 0) 0x7fcbb0061b80 con 0x7fcb980a2b90 2024-04-25T00:13:48.383 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:48.385 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T00:13:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.381+0000 7fcbb586b640 1 -- 172.21.15.27:0/1941080558 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fcb980a2b90 msgr2=0x7fcb980a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.381+0000 7fcbb586b640 1 --2- 172.21.15.27:0/1941080558 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fcb980a2b90 0x7fcb980a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcbb01016d0 tx=0x7fcba0009870 comp rx=0 tx=0).stop 2024-04-25T00:13:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.381+0000 7fcbb586b640 1 -- 172.21.15.27:0/1941080558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb0101870 msgr2=0x7fcbb0197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.381+0000 7fcbb586b640 1 --2- 172.21.15.27:0/1941080558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb0101870 0x7fcbb0197fe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009230 tx=0x7fcba4031b60 comp rx=0 tx=0).stop 2024-04-25T00:13:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.381+0000 7fcbb586b640 1 -- 172.21.15.27:0/1941080558 shutdown_connections 2024-04-25T00:13:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.381+0000 7fcbb586b640 1 --2- 172.21.15.27:0/1941080558 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fcb980a2b90 0x7fcb980a5050 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.381+0000 7fcbb586b640 1 --2- 172.21.15.27:0/1941080558 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcbb0102230 0x7fcbb019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.381+0000 7fcbb586b640 1 --2- 172.21.15.27:0/1941080558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb0101870 0x7fcbb0197fe0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.381+0000 7fcbb586b640 1 --2- 172.21.15.27:0/1941080558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb0100680 0x7fcbb0197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.381+0000 7fcbb586b640 1 -- 172.21.15.27:0/1941080558 >> 172.21.15.27:0/1941080558 conn(0x7fcbb00fbbf0 msgr2=0x7fcbb0104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.382+0000 7fcbb586b640 1 -- 172.21.15.27:0/1941080558 shutdown_connections 2024-04-25T00:13:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.382+0000 7fcbb586b640 1 -- 172.21.15.27:0/1941080558 wait complete. 2024-04-25T00:13:48.393 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-04-25T00:13:47.576732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595322,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":270,"num_write_kb":2974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":213,"ondisk_log_size":213,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":348232,"kb_used_data":11544,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124593592,"statfs":{"total":1151940427776,"available":1151583838208,"internally_reserved":0,"allocated":11821056,"data_stored":7513434,"data_compressed":283200,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":344634681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":455,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003941"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447749+0000","last_change":"2024-04-25T00:13:19.429658+0000","last_active":"2024-04-25T00:13:27.447749+0000","last_peered":"2024-04-25T00:13:27.447749+0000","last_clean":"2024-04-25T00:13:27.447749+0000","last_became_active":"2024-04-25T00:13:19.428426+0000","last_became_peered":"2024-04-25T00:13:19.428426+0000","last_unstale":"2024-04-25T00:13:27.447749+0000","last_undegraded":"2024-04-25T00:13:27.447749+0000","last_fullsized":"2024-04-25T00:13:27.447749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:10.814361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446796+0000","last_change":"2024-04-25T00:13:19.425504+0000","last_active":"2024-04-25T00:13:27.446796+0000","last_peered":"2024-04-25T00:13:27.446796+0000","last_clean":"2024-04-25T00:13:27.446796+0000","last_became_active":"2024-04-25T00:13:19.425170+0000","last_became_peered":"2024-04-25T00:13:19.425170+0000","last_unstale":"2024-04-25T00:13:27.446796+0000","last_undegraded":"2024-04-25T00:13:27.446796+0000","last_fullsized":"2024-04-25T00:13:27.446796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:31:13.826981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446369+0000","last_change":"2024-04-25T00:13:19.420842+0000","last_active":"2024-04-25T00:13:27.446369+0000","last_peered":"2024-04-25T00:13:27.446369+0000","last_clean":"2024-04-25T00:13:27.446369+0000","last_became_active":"2024-04-25T00:13:19.420241+0000","last_became_peered":"2024-04-25T00:13:19.420241+0000","last_unstale":"2024-04-25T00:13:27.446369+0000","last_undegraded":"2024-04-25T00:13:27.446369+0000","last_fullsized":"2024-04-25T00:13:27.446369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:52:39.500432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771920+0000","last_change":"2024-04-25T00:13:19.429800+0000","last_active":"2024-04-25T00:13:26.771920+0000","last_peered":"2024-04-25T00:13:26.771920+0000","last_clean":"2024-04-25T00:13:26.771920+0000","last_became_active":"2024-04-25T00:13:19.429388+0000","last_became_peered":"2024-04-25T00:13:19.429388+0000","last_unstale":"2024-04-25T00:13:26.771920+0000","last_undegraded":"2024-04-25T00:13:26.771920+0000","last_fullsized":"2024-04-25T00:13:26.771920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:45.344870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.859856+0000","last_change":"2024-04-25T00:13:19.416519+0000","last_active":"2024-04-25T00:13:25.859856+0000","last_peered":"2024-04-25T00:13:25.859856+0000","last_clean":"2024-04-25T00:13:25.859856+0000","last_became_active":"2024-04-25T00:13:19.416080+0000","last_became_peered":"2024-04-25T00:13:19.416080+0000","last_unstale":"2024-04-25T00:13:25.859856+0000","last_undegraded":"2024-04-25T00:13:25.859856+0000","last_fullsized":"2024-04-25T00:13:25.859856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:04.450160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.574047+0000","last_change":"2024-04-25T00:13:19.424378+0000","last_active":"2024-04-25T00:13:24.574047+0000","last_peered":"2024-04-25T00:13:24.574047+0000","last_clean":"2024-04-25T00:13:24.574047+0000","last_became_active":"2024-04-25T00:13:19.423918+0000","last_became_peered":"2024-04-25T00:13:19.423918+0000","last_unstale":"2024-04-25T00:13:24.574047+0000","last_undegraded":"2024-04-25T00:13:24.574047+0000","last_fullsized":"2024-04-25T00:13:24.574047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:01:11.397328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.589752+0000","last_change":"2024-04-25T00:13:19.424729+0000","last_active":"2024-04-25T00:13:25.589752+0000","last_peered":"2024-04-25T00:13:25.589752+0000","last_clean":"2024-04-25T00:13:25.589752+0000","last_became_active":"2024-04-25T00:13:19.422874+0000","last_became_peered":"2024-04-25T00:13:19.422874+0000","last_unstale":"2024-04-25T00:13:25.589752+0000","last_undegraded":"2024-04-25T00:13:25.589752+0000","last_fullsized":"2024-04-25T00:13:25.589752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:52:39.483589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.208841+0000","last_change":"2024-04-25T00:13:19.418043+0000","last_active":"2024-04-25T00:13:27.208841+0000","last_peered":"2024-04-25T00:13:27.208841+0000","last_clean":"2024-04-25T00:13:27.208841+0000","last_became_active":"2024-04-25T00:13:19.417443+0000","last_became_peered":"2024-04-25T00:13:19.417443+0000","last_unstale":"2024-04-25T00:13:27.208841+0000","last_undegraded":"2024-04-25T00:13:27.208841+0000","last_fullsized":"2024-04-25T00:13:27.208841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:47:53.141744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.397956+0000","last_change":"2024-04-25T00:13:19.420989+0000","last_active":"2024-04-25T00:13:27.397956+0000","last_peered":"2024-04-25T00:13:27.397956+0000","last_clean":"2024-04-25T00:13:27.397956+0000","last_became_active":"2024-04-25T00:13:19.420669+0000","last_became_peered":"2024-04-25T00:13:19.420669+0000","last_unstale":"2024-04-25T00:13:27.397956+0000","last_undegraded":"2024-04-25T00:13:27.397956+0000","last_fullsized":"2024-04-25T00:13:27.397956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:18.236132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.574336+0000","last_change":"2024-04-25T00:13:19.424176+0000","last_active":"2024-04-25T00:13:24.574336+0000","last_peered":"2024-04-25T00:13:24.574336+0000","last_clean":"2024-04-25T00:13:24.574336+0000","last_became_active":"2024-04-25T00:13:19.423656+0000","last_became_peered":"2024-04-25T00:13:19.423656+0000","last_unstale":"2024-04-25T00:13:24.574336+0000","last_undegraded":"2024-04-25T00:13:24.574336+0000","last_fullsized":"2024-04-25T00:13:24.574336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:24:41.697077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.590388+0000","last_change":"2024-04-25T00:13:19.419624+0000","last_active":"2024-04-25T00:13:25.590388+0000","last_peered":"2024-04-25T00:13:25.590388+0000","last_clean":"2024-04-25T00:13:25.590388+0000","last_became_active":"2024-04-25T00:13:19.419263+0000","last_became_peered":"2024-04-25T00:13:19.419263+0000","last_unstale":"2024-04-25T00:13:25.590388+0000","last_undegraded":"2024-04-25T00:13:25.590388+0000","last_fullsized":"2024-04-25T00:13:25.590388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:29.130733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771798+0000","last_change":"2024-04-25T00:13:19.427865+0000","last_active":"2024-04-25T00:13:26.771798+0000","last_peered":"2024-04-25T00:13:26.771798+0000","last_clean":"2024-04-25T00:13:26.771798+0000","last_became_active":"2024-04-25T00:13:19.427538+0000","last_became_peered":"2024-04-25T00:13:19.427538+0000","last_unstale":"2024-04-25T00:13:26.771798+0000","last_undegraded":"2024-04-25T00:13:26.771798+0000","last_fullsized":"2024-04-25T00:13:26.771798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:05.136948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.666147+0000","last_change":"2024-04-25T00:13:19.419261+0000","last_active":"2024-04-25T00:13:27.666147+0000","last_peered":"2024-04-25T00:13:27.666147+0000","last_clean":"2024-04-25T00:13:27.666147+0000","last_became_active":"2024-04-25T00:13:19.418938+0000","last_became_peered":"2024-04-25T00:13:19.418938+0000","last_unstale":"2024-04-25T00:13:27.666147+0000","last_undegraded":"2024-04-25T00:13:27.666147+0000","last_fullsized":"2024-04-25T00:13:27.666147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:44.971640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446336+0000","last_change":"2024-04-25T00:13:19.417654+0000","last_active":"2024-04-25T00:13:27.446336+0000","last_peered":"2024-04-25T00:13:27.446336+0000","last_clean":"2024-04-25T00:13:27.446336+0000","last_became_active":"2024-04-25T00:13:19.417319+0000","last_became_peered":"2024-04-25T00:13:19.417319+0000","last_unstale":"2024-04-25T00:13:27.446336+0000","last_undegraded":"2024-04-25T00:13:27.446336+0000","last_fullsized":"2024-04-25T00:13:27.446336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:43:54.447937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446578+0000","last_change":"2024-04-25T00:13:19.420761+0000","last_active":"2024-04-25T00:13:27.446578+0000","last_peered":"2024-04-25T00:13:27.446578+0000","last_clean":"2024-04-25T00:13:27.446578+0000","last_became_active":"2024-04-25T00:13:19.420334+0000","last_became_peered":"2024-04-25T00:13:19.420334+0000","last_unstale":"2024-04-25T00:13:27.446578+0000","last_undegraded":"2024-04-25T00:13:27.446578+0000","last_fullsized":"2024-04-25T00:13:27.446578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:20.136470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.397368+0000","last_change":"2024-04-25T00:13:19.420132+0000","last_active":"2024-04-25T00:13:27.397368+0000","last_peered":"2024-04-25T00:13:27.397368+0000","last_clean":"2024-04-25T00:13:27.397368+0000","last_became_active":"2024-04-25T00:13:19.418910+0000","last_became_peered":"2024-04-25T00:13:19.418910+0000","last_unstale":"2024-04-25T00:13:27.397368+0000","last_undegraded":"2024-04-25T00:13:27.397368+0000","last_fullsized":"2024-04-25T00:13:27.397368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:17.895412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447561+0000","last_change":"2024-04-25T00:13:19.430242+0000","last_active":"2024-04-25T00:13:27.447561+0000","last_peered":"2024-04-25T00:13:27.447561+0000","last_clean":"2024-04-25T00:13:27.447561+0000","last_became_active":"2024-04-25T00:13:19.429077+0000","last_became_peered":"2024-04-25T00:13:19.429077+0000","last_unstale":"2024-04-25T00:13:27.447561+0000","last_undegraded":"2024-04-25T00:13:27.447561+0000","last_fullsized":"2024-04-25T00:13:27.447561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:34:14.351157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.577522+0000","last_change":"2024-04-25T00:13:19.414375+0000","last_active":"2024-04-25T00:13:24.577522+0000","last_peered":"2024-04-25T00:13:24.577522+0000","last_clean":"2024-04-25T00:13:24.577522+0000","last_became_active":"2024-04-25T00:13:19.414003+0000","last_became_peered":"2024-04-25T00:13:19.414003+0000","last_unstale":"2024-04-25T00:13:24.577522+0000","last_undegraded":"2024-04-25T00:13:24.577522+0000","last_fullsized":"2024-04-25T00:13:24.577522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:43:50.612743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573887+0000","last_change":"2024-04-25T00:13:19.415322+0000","last_active":"2024-04-25T00:13:24.573887+0000","last_peered":"2024-04-25T00:13:24.573887+0000","last_clean":"2024-04-25T00:13:24.573887+0000","last_became_active":"2024-04-25T00:13:19.414925+0000","last_became_peered":"2024-04-25T00:13:19.414925+0000","last_unstale":"2024-04-25T00:13:24.573887+0000","last_undegraded":"2024-04-25T00:13:24.573887+0000","last_fullsized":"2024-04-25T00:13:24.573887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:58.140640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573296+0000","last_change":"2024-04-25T00:13:23.584763+0000","last_active":"2024-04-25T00:13:24.573296+0000","last_peered":"2024-04-25T00:13:24.573296+0000","last_clean":"2024-04-25T00:13:24.573296+0000","last_became_active":"2024-04-25T00:13:23.584303+0000","last_became_peered":"2024-04-25T00:13:23.584303+0000","last_unstale":"2024-04-25T00:13:24.573296+0000","last_undegraded":"2024-04-25T00:13:24.573296+0000","last_fullsized":"2024-04-25T00:13:24.573296+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:54:11.447867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.577673+0000","last_change":"2024-04-25T00:13:18.519576+0000","last_active":"2024-04-25T00:13:24.577673+0000","last_peered":"2024-04-25T00:13:24.577673+0000","last_clean":"2024-04-25T00:13:24.577673+0000","last_became_active":"2024-04-25T00:13:18.519201+0000","last_became_peered":"2024-04-25T00:13:18.519201+0000","last_unstale":"2024-04-25T00:13:24.577673+0000","last_undegraded":"2024-04-25T00:13:24.577673+0000","last_fullsized":"2024-04-25T00:13:24.577673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:19.681096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.209228+0000","last_change":"2024-04-25T00:13:19.419098+0000","last_active":"2024-04-25T00:13:27.209228+0000","last_peered":"2024-04-25T00:13:27.209228+0000","last_clean":"2024-04-25T00:13:27.209228+0000","last_became_active":"2024-04-25T00:13:19.418649+0000","last_became_peered":"2024-04-25T00:13:19.418649+0000","last_unstale":"2024-04-25T00:13:27.209228+0000","last_undegraded":"2024-04-25T00:13:27.209228+0000","last_fullsized":"2024-04-25T00:13:27.209228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:04.561470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771249+0000","last_change":"2024-04-25T00:13:23.580354+0000","last_active":"2024-04-25T00:13:26.771249+0000","last_peered":"2024-04-25T00:13:26.771249+0000","last_clean":"2024-04-25T00:13:26.771249+0000","last_became_active":"2024-04-25T00:13:23.579944+0000","last_became_peered":"2024-04-25T00:13:23.579944+0000","last_unstale":"2024-04-25T00:13:26.771249+0000","last_undegraded":"2024-04-25T00:13:26.771249+0000","last_fullsized":"2024-04-25T00:13:26.771249+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:48:57.788974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.209268+0000","last_change":"2024-04-25T00:13:18.527471+0000","last_active":"2024-04-25T00:13:27.209268+0000","last_peered":"2024-04-25T00:13:27.209268+0000","last_clean":"2024-04-25T00:13:27.209268+0000","last_became_active":"2024-04-25T00:13:18.527138+0000","last_became_peered":"2024-04-25T00:13:18.527138+0000","last_unstale":"2024-04-25T00:13:27.209268+0000","last_undegraded":"2024-04-25T00:13:27.209268+0000","last_fullsized":"2024-04-25T00:13:27.209268+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:38.003719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446684+0000","last_change":"2024-04-25T00:13:19.420563+0000","last_active":"2024-04-25T00:13:27.446684+0000","last_peered":"2024-04-25T00:13:27.446684+0000","last_clean":"2024-04-25T00:13:27.446684+0000","last_became_active":"2024-04-25T00:13:19.420142+0000","last_became_peered":"2024-04-25T00:13:19.420142+0000","last_unstale":"2024-04-25T00:13:27.446684+0000","last_undegraded":"2024-04-25T00:13:27.446684+0000","last_fullsized":"2024-04-25T00:13:27.446684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:23:08.423795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771374+0000","last_change":"2024-04-25T00:13:23.580657+0000","last_active":"2024-04-25T00:13:26.771374+0000","last_peered":"2024-04-25T00:13:26.771374+0000","last_clean":"2024-04-25T00:13:26.771374+0000","last_became_active":"2024-04-25T00:13:23.580253+0000","last_became_peered":"2024-04-25T00:13:23.580253+0000","last_unstale":"2024-04-25T00:13:26.771374+0000","last_undegraded":"2024-04-25T00:13:26.771374+0000","last_fullsized":"2024-04-25T00:13:26.771374+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:54:02.842805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:44.657231+0000","last_change":"2024-04-25T00:13:18.520509+0000","last_active":"2024-04-25T00:13:44.657231+0000","last_peered":"2024-04-25T00:13:44.657231+0000","last_clean":"2024-04-25T00:13:44.657231+0000","last_became_active":"2024-04-25T00:13:18.520139+0000","last_became_peered":"2024-04-25T00:13:18.520139+0000","last_unstale":"2024-04-25T00:13:44.657231+0000","last_undegraded":"2024-04-25T00:13:44.657231+0000","last_fullsized":"2024-04-25T00:13:44.657231+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:08.797670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.397687+0000","last_change":"2024-04-25T00:13:19.419115+0000","last_active":"2024-04-25T00:13:27.397687+0000","last_peered":"2024-04-25T00:13:27.397687+0000","last_clean":"2024-04-25T00:13:27.397687+0000","last_became_active":"2024-04-25T00:13:19.417865+0000","last_became_peered":"2024-04-25T00:13:19.417865+0000","last_unstale":"2024-04-25T00:13:27.397687+0000","last_undegraded":"2024-04-25T00:13:27.397687+0000","last_fullsized":"2024-04-25T00:13:27.397687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:20:11.173803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771365+0000","last_change":"2024-04-25T00:13:23.585052+0000","last_active":"2024-04-25T00:13:26.771365+0000","last_peered":"2024-04-25T00:13:26.771365+0000","last_clean":"2024-04-25T00:13:26.771365+0000","last_became_active":"2024-04-25T00:13:23.584696+0000","last_became_peered":"2024-04-25T00:13:23.584696+0000","last_unstale":"2024-04-25T00:13:26.771365+0000","last_undegraded":"2024-04-25T00:13:26.771365+0000","last_fullsized":"2024-04-25T00:13:26.771365+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:26.894183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.860271+0000","last_change":"2024-04-25T00:13:18.522793+0000","last_active":"2024-04-25T00:13:25.860271+0000","last_peered":"2024-04-25T00:13:25.860271+0000","last_clean":"2024-04-25T00:13:25.860271+0000","last_became_active":"2024-04-25T00:13:18.522453+0000","last_became_peered":"2024-04-25T00:13:18.522453+0000","last_unstale":"2024-04-25T00:13:25.860271+0000","last_undegraded":"2024-04-25T00:13:25.860271+0000","last_fullsized":"2024-04-25T00:13:25.860271+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:08.368214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.448211+0000","last_change":"2024-04-25T00:13:19.418573+0000","last_active":"2024-04-25T00:13:27.448211+0000","last_peered":"2024-04-25T00:13:27.448211+0000","last_clean":"2024-04-25T00:13:27.448211+0000","last_became_active":"2024-04-25T00:13:19.418219+0000","last_became_peered":"2024-04-25T00:13:19.418219+0000","last_unstale":"2024-04-25T00:13:27.448211+0000","last_undegraded":"2024-04-25T00:13:27.448211+0000","last_fullsized":"2024-04-25T00:13:27.448211+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:08:44.453017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573796+0000","last_change":"2024-04-25T00:13:23.578716+0000","last_active":"2024-04-25T00:13:24.573796+0000","last_peered":"2024-04-25T00:13:24.573796+0000","last_clean":"2024-04-25T00:13:24.573796+0000","last_became_active":"2024-04-25T00:13:23.578325+0000","last_became_peered":"2024-04-25T00:13:23.578325+0000","last_unstale":"2024-04-25T00:13:24.573796+0000","last_undegraded":"2024-04-25T00:13:24.573796+0000","last_fullsized":"2024-04-25T00:13:24.573796+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:39:57.145656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447998+0000","last_change":"2024-04-25T00:13:18.527369+0000","last_active":"2024-04-25T00:13:27.447998+0000","last_peered":"2024-04-25T00:13:27.447998+0000","last_clean":"2024-04-25T00:13:27.447998+0000","last_became_active":"2024-04-25T00:13:18.526032+0000","last_became_peered":"2024-04-25T00:13:18.526032+0000","last_unstale":"2024-04-25T00:13:27.447998+0000","last_undegraded":"2024-04-25T00:13:27.447998+0000","last_fullsized":"2024-04-25T00:13:27.447998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:52.635184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.859887+0000","last_change":"2024-04-25T00:13:19.413305+0000","last_active":"2024-04-25T00:13:25.859887+0000","last_peered":"2024-04-25T00:13:25.859887+0000","last_clean":"2024-04-25T00:13:25.859887+0000","last_became_active":"2024-04-25T00:13:19.412956+0000","last_became_peered":"2024-04-25T00:13:19.412956+0000","last_unstale":"2024-04-25T00:13:25.859887+0000","last_undegraded":"2024-04-25T00:13:25.859887+0000","last_fullsized":"2024-04-25T00:13:25.859887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:54.436470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573607+0000","last_change":"2024-04-25T00:13:23.585624+0000","last_active":"2024-04-25T00:13:24.573607+0000","last_peered":"2024-04-25T00:13:24.573607+0000","last_clean":"2024-04-25T00:13:24.573607+0000","last_became_active":"2024-04-25T00:13:23.585215+0000","last_became_peered":"2024-04-25T00:13:23.585215+0000","last_unstale":"2024-04-25T00:13:24.573607+0000","last_undegraded":"2024-04-25T00:13:24.573607+0000","last_fullsized":"2024-04-25T00:13:24.573607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:01.427226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573909+0000","last_change":"2024-04-25T00:13:18.517507+0000","last_active":"2024-04-25T00:13:24.573909+0000","last_peered":"2024-04-25T00:13:24.573909+0000","last_clean":"2024-04-25T00:13:24.573909+0000","last_became_active":"2024-04-25T00:13:18.517099+0000","last_became_peered":"2024-04-25T00:13:18.517099+0000","last_unstale":"2024-04-25T00:13:24.573909+0000","last_undegraded":"2024-04-25T00:13:24.573909+0000","last_fullsized":"2024-04-25T00:13:24.573909+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:42:17.357610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.448111+0000","last_change":"2024-04-25T00:13:19.422134+0000","last_active":"2024-04-25T00:13:27.448111+0000","last_peered":"2024-04-25T00:13:27.448111+0000","last_clean":"2024-04-25T00:13:27.448111+0000","last_became_active":"2024-04-25T00:13:19.421167+0000","last_became_peered":"2024-04-25T00:13:19.421167+0000","last_unstale":"2024-04-25T00:13:27.448111+0000","last_undegraded":"2024-04-25T00:13:27.448111+0000","last_fullsized":"2024-04-25T00:13:27.448111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:03:25.451560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447625+0000","last_change":"2024-04-25T00:13:23.582526+0000","last_active":"2024-04-25T00:13:27.447625+0000","last_peered":"2024-04-25T00:13:27.447625+0000","last_clean":"2024-04-25T00:13:27.447625+0000","last_became_active":"2024-04-25T00:13:23.581221+0000","last_became_peered":"2024-04-25T00:13:23.581221+0000","last_unstale":"2024-04-25T00:13:27.447625+0000","last_undegraded":"2024-04-25T00:13:27.447625+0000","last_fullsized":"2024-04-25T00:13:27.447625+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:46.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:44.658568+0000","last_change":"2024-04-25T00:13:18.522245+0000","last_active":"2024-04-25T00:13:44.658568+0000","last_peered":"2024-04-25T00:13:44.658568+0000","last_clean":"2024-04-25T00:13:44.658568+0000","last_became_active":"2024-04-25T00:13:18.521440+0000","last_became_peered":"2024-04-25T00:13:18.521440+0000","last_unstale":"2024-04-25T00:13:44.658568+0000","last_undegraded":"2024-04-25T00:13:44.658568+0000","last_fullsized":"2024-04-25T00:13:44.658568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:21:37.847964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.208709+0000","last_change":"2024-04-25T00:13:19.422771+0000","last_active":"2024-04-25T00:13:27.208709+0000","last_peered":"2024-04-25T00:13:27.208709+0000","last_clean":"2024-04-25T00:13:27.208709+0000","last_became_active":"2024-04-25T00:13:19.422387+0000","last_became_peered":"2024-04-25T00:13:19.422387+0000","last_unstale":"2024-04-25T00:13:27.208709+0000","last_undegraded":"2024-04-25T00:13:27.208709+0000","last_fullsized":"2024-04-25T00:13:27.208709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:15.647186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446740+0000","last_change":"2024-04-25T00:13:23.586235+0000","last_active":"2024-04-25T00:13:27.446740+0000","last_peered":"2024-04-25T00:13:27.446740+0000","last_clean":"2024-04-25T00:13:27.446740+0000","last_became_active":"2024-04-25T00:13:23.585803+0000","last_became_peered":"2024-04-25T00:13:23.585803+0000","last_unstale":"2024-04-25T00:13:27.446740+0000","last_undegraded":"2024-04-25T00:13:27.446740+0000","last_fullsized":"2024-04-25T00:13:27.446740+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:46.235435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447814+0000","last_change":"2024-04-25T00:13:18.525283+0000","last_active":"2024-04-25T00:13:27.447814+0000","last_peered":"2024-04-25T00:13:27.447814+0000","last_clean":"2024-04-25T00:13:27.447814+0000","last_became_active":"2024-04-25T00:13:18.524939+0000","last_became_peered":"2024-04-25T00:13:18.524939+0000","last_unstale":"2024-04-25T00:13:27.447814+0000","last_undegraded":"2024-04-25T00:13:27.447814+0000","last_fullsized":"2024-04-25T00:13:27.447814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:09.258498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.576780+0000","last_change":"2024-04-25T00:13:19.415746+0000","last_active":"2024-04-25T00:13:24.576780+0000","last_peered":"2024-04-25T00:13:24.576780+0000","last_clean":"2024-04-25T00:13:24.576780+0000","last_became_active":"2024-04-25T00:13:19.415325+0000","last_became_peered":"2024-04-25T00:13:19.415325+0000","last_unstale":"2024-04-25T00:13:24.576780+0000","last_undegraded":"2024-04-25T00:13:24.576780+0000","last_fullsized":"2024-04-25T00:13:24.576780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:17:48.268845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.208805+0000","last_change":"2024-04-25T00:13:23.583153+0000","last_active":"2024-04-25T00:13:27.208805+0000","last_peered":"2024-04-25T00:13:27.208805+0000","last_clean":"2024-04-25T00:13:27.208805+0000","last_became_active":"2024-04-25T00:13:23.580257+0000","last_became_peered":"2024-04-25T00:13:23.580257+0000","last_unstale":"2024-04-25T00:13:27.208805+0000","last_undegraded":"2024-04-25T00:13:27.208805+0000","last_fullsized":"2024-04-25T00:13:27.208805+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:09:32.990253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.574122+0000","last_change":"2024-04-25T00:13:18.520365+0000","last_active":"2024-04-25T00:13:24.574122+0000","last_peered":"2024-04-25T00:13:24.574122+0000","last_clean":"2024-04-25T00:13:24.574122+0000","last_became_active":"2024-04-25T00:13:18.519918+0000","last_became_peered":"2024-04-25T00:13:18.519918+0000","last_unstale":"2024-04-25T00:13:24.574122+0000","last_undegraded":"2024-04-25T00:13:24.574122+0000","last_fullsized":"2024-04-25T00:13:24.574122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:46.415197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.397970+0000","last_change":"2024-04-25T00:13:19.414341+0000","last_active":"2024-04-25T00:13:27.397970+0000","last_peered":"2024-04-25T00:13:27.397970+0000","last_clean":"2024-04-25T00:13:27.397970+0000","last_became_active":"2024-04-25T00:13:19.413793+0000","last_became_peered":"2024-04-25T00:13:19.413793+0000","last_unstale":"2024-04-25T00:13:27.397970+0000","last_undegraded":"2024-04-25T00:13:27.397970+0000","last_fullsized":"2024-04-25T00:13:27.397970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:55.927082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.576729+0000","last_change":"2024-04-25T00:13:23.590631+0000","last_active":"2024-04-25T00:13:24.576729+0000","last_peered":"2024-04-25T00:13:24.576729+0000","last_clean":"2024-04-25T00:13:24.576729+0000","last_became_active":"2024-04-25T00:13:23.589980+0000","last_became_peered":"2024-04-25T00:13:23.589980+0000","last_unstale":"2024-04-25T00:13:24.576729+0000","last_undegraded":"2024-04-25T00:13:24.576729+0000","last_fullsized":"2024-04-25T00:13:24.576729+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:18.234048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:43.652199+0000","last_change":"2024-04-25T00:13:18.526353+0000","last_active":"2024-04-25T00:13:43.652199+0000","last_peered":"2024-04-25T00:13:43.652199+0000","last_clean":"2024-04-25T00:13:43.652199+0000","last_became_active":"2024-04-25T00:13:18.525951+0000","last_became_peered":"2024-04-25T00:13:18.525951+0000","last_unstale":"2024-04-25T00:13:43.652199+0000","last_undegraded":"2024-04-25T00:13:43.652199+0000","last_fullsized":"2024-04-25T00:13:43.652199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:10:29.597797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446700+0000","last_change":"2024-04-25T00:13:19.425486+0000","last_active":"2024-04-25T00:13:27.446700+0000","last_peered":"2024-04-25T00:13:27.446700+0000","last_clean":"2024-04-25T00:13:27.446700+0000","last_became_active":"2024-04-25T00:13:19.425087+0000","last_became_peered":"2024-04-25T00:13:19.425087+0000","last_unstale":"2024-04-25T00:13:27.446700+0000","last_undegraded":"2024-04-25T00:13:27.446700+0000","last_fullsized":"2024-04-25T00:13:27.446700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:00.216683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771549+0000","last_change":"2024-04-25T00:13:23.579747+0000","last_active":"2024-04-25T00:13:26.771549+0000","last_peered":"2024-04-25T00:13:26.771549+0000","last_clean":"2024-04-25T00:13:26.771549+0000","last_became_active":"2024-04-25T00:13:23.579346+0000","last_became_peered":"2024-04-25T00:13:23.579346+0000","last_unstale":"2024-04-25T00:13:26.771549+0000","last_undegraded":"2024-04-25T00:13:26.771549+0000","last_fullsized":"2024-04-25T00:13:26.771549+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:10:18.969942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.448126+0000","last_change":"2024-04-25T00:13:18.524707+0000","last_active":"2024-04-25T00:13:27.448126+0000","last_peered":"2024-04-25T00:13:27.448126+0000","last_clean":"2024-04-25T00:13:27.448126+0000","last_became_active":"2024-04-25T00:13:18.524352+0000","last_became_peered":"2024-04-25T00:13:18.524352+0000","last_unstale":"2024-04-25T00:13:27.448126+0000","last_undegraded":"2024-04-25T00:13:27.448126+0000","last_fullsized":"2024-04-25T00:13:27.448126+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:29.992843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.397816+0000","last_change":"2024-04-25T00:13:19.421845+0000","last_active":"2024-04-25T00:13:27.397816+0000","last_peered":"2024-04-25T00:13:27.397816+0000","last_clean":"2024-04-25T00:13:27.397816+0000","last_became_active":"2024-04-25T00:13:19.421458+0000","last_became_peered":"2024-04-25T00:13:19.421458+0000","last_unstale":"2024-04-25T00:13:27.397816+0000","last_undegraded":"2024-04-25T00:13:27.397816+0000","last_fullsized":"2024-04-25T00:13:27.397816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:09.286604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.576864+0000","last_change":"2024-04-25T00:13:23.583721+0000","last_active":"2024-04-25T00:13:24.576864+0000","last_peered":"2024-04-25T00:13:24.576864+0000","last_clean":"2024-04-25T00:13:24.576864+0000","last_became_active":"2024-04-25T00:13:23.583340+0000","last_became_peered":"2024-04-25T00:13:23.583340+0000","last_unstale":"2024-04-25T00:13:24.576864+0000","last_undegraded":"2024-04-25T00:13:24.576864+0000","last_fullsized":"2024-04-25T00:13:24.576864+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:26.446468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'10","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:35.050566+0000","last_change":"2024-04-25T00:13:18.520653+0000","last_active":"2024-04-25T00:13:35.050566+0000","last_peered":"2024-04-25T00:13:35.050566+0000","last_clean":"2024-04-25T00:13:35.050566+0000","last_became_active":"2024-04-25T00:13:18.520278+0000","last_became_peered":"2024-04-25T00:13:18.520278+0000","last_unstale":"2024-04-25T00:13:35.050566+0000","last_undegraded":"2024-04-25T00:13:35.050566+0000","last_fullsized":"2024-04-25T00:13:35.050566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:58.311179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.860126+0000","last_change":"2024-04-25T00:13:19.421599+0000","last_active":"2024-04-25T00:13:25.860126+0000","last_peered":"2024-04-25T00:13:25.860126+0000","last_clean":"2024-04-25T00:13:25.860126+0000","last_became_active":"2024-04-25T00:13:19.421107+0000","last_became_peered":"2024-04-25T00:13:19.421107+0000","last_unstale":"2024-04-25T00:13:25.860126+0000","last_undegraded":"2024-04-25T00:13:25.860126+0000","last_fullsized":"2024-04-25T00:13:25.860126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:59.577987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.448527+0000","last_change":"2024-04-25T00:13:23.580538+0000","last_active":"2024-04-25T00:13:27.448527+0000","last_peered":"2024-04-25T00:13:27.448527+0000","last_clean":"2024-04-25T00:13:27.448527+0000","last_became_active":"2024-04-25T00:13:23.580095+0000","last_became_peered":"2024-04-25T00:13:23.580095+0000","last_unstale":"2024-04-25T00:13:27.448527+0000","last_undegraded":"2024-04-25T00:13:27.448527+0000","last_fullsized":"2024-04-25T00:13:27.448527+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:04.189530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573667+0000","last_change":"2024-04-25T00:13:18.522501+0000","last_active":"2024-04-25T00:13:24.573667+0000","last_peered":"2024-04-25T00:13:24.573667+0000","last_clean":"2024-04-25T00:13:24.573667+0000","last_became_active":"2024-04-25T00:13:18.522160+0000","last_became_peered":"2024-04-25T00:13:18.522160+0000","last_unstale":"2024-04-25T00:13:24.573667+0000","last_undegraded":"2024-04-25T00:13:24.573667+0000","last_fullsized":"2024-04-25T00:13:24.573667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:31.876731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.665997+0000","last_change":"2024-04-25T00:13:19.417957+0000","last_active":"2024-04-25T00:13:27.665997+0000","last_peered":"2024-04-25T00:13:27.665997+0000","last_clean":"2024-04-25T00:13:27.665997+0000","last_became_active":"2024-04-25T00:13:19.417600+0000","last_became_peered":"2024-04-25T00:13:19.417600+0000","last_unstale":"2024-04-25T00:13:27.665997+0000","last_undegraded":"2024-04-25T00:13:27.665997+0000","last_fullsized":"2024-04-25T00:13:27.665997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:24.600965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447214+0000","last_change":"2024-04-25T00:13:23.582279+0000","last_active":"2024-04-25T00:13:27.447214+0000","last_peered":"2024-04-25T00:13:27.447214+0000","last_clean":"2024-04-25T00:13:27.447214+0000","last_became_active":"2024-04-25T00:13:23.581766+0000","last_became_peered":"2024-04-25T00:13:23.581766+0000","last_unstale":"2024-04-25T00:13:27.447214+0000","last_undegraded":"2024-04-25T00:13:27.447214+0000","last_fullsized":"2024-04-25T00:13:27.447214+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:36.115870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447847+0000","last_change":"2024-04-25T00:13:18.527443+0000","last_active":"2024-04-25T00:13:27.447847+0000","last_peered":"2024-04-25T00:13:27.447847+0000","last_clean":"2024-04-25T00:13:27.447847+0000","last_became_active":"2024-04-25T00:13:18.526919+0000","last_became_peered":"2024-04-25T00:13:18.526919+0000","last_unstale":"2024-04-25T00:13:27.447847+0000","last_undegraded":"2024-04-25T00:13:27.447847+0000","last_fullsized":"2024-04-25T00:13:27.447847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:32.737580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.590010+0000","last_change":"2024-04-25T00:13:19.425947+0000","last_active":"2024-04-25T00:13:25.590010+0000","last_peered":"2024-04-25T00:13:25.590010+0000","last_clean":"2024-04-25T00:13:25.590010+0000","last_became_active":"2024-04-25T00:13:19.425613+0000","last_became_peered":"2024-04-25T00:13:19.425613+0000","last_unstale":"2024-04-25T00:13:25.590010+0000","last_undegraded":"2024-04-25T00:13:25.590010+0000","last_fullsized":"2024-04-25T00:13:25.590010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:28:34.729375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.577535+0000","last_change":"2024-04-25T00:13:23.587794+0000","last_active":"2024-04-25T00:13:24.577535+0000","last_peered":"2024-04-25T00:13:24.577535+0000","last_clean":"2024-04-25T00:13:24.577535+0000","last_became_active":"2024-04-25T00:13:23.587334+0000","last_became_peered":"2024-04-25T00:13:23.587334+0000","last_unstale":"2024-04-25T00:13:24.577535+0000","last_undegraded":"2024-04-25T00:13:24.577535+0000","last_fullsized":"2024-04-25T00:13:24.577535+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:58.460439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771365+0000","last_change":"2024-04-25T00:13:18.521701+0000","last_active":"2024-04-25T00:13:26.771365+0000","last_peered":"2024-04-25T00:13:26.771365+0000","last_clean":"2024-04-25T00:13:26.771365+0000","last_became_active":"2024-04-25T00:13:18.521360+0000","last_became_peered":"2024-04-25T00:13:18.521360+0000","last_unstale":"2024-04-25T00:13:26.771365+0000","last_undegraded":"2024-04-25T00:13:26.771365+0000","last_fullsized":"2024-04-25T00:13:26.771365+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:18:11.567426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.589791+0000","last_change":"2024-04-25T00:13:19.424162+0000","last_active":"2024-04-25T00:13:25.589791+0000","last_peered":"2024-04-25T00:13:25.589791+0000","last_clean":"2024-04-25T00:13:25.589791+0000","last_became_active":"2024-04-25T00:13:19.423845+0000","last_became_peered":"2024-04-25T00:13:19.423845+0000","last_unstale":"2024-04-25T00:13:25.589791+0000","last_undegraded":"2024-04-25T00:13:25.589791+0000","last_fullsized":"2024-04-25T00:13:25.589791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:33.887577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.666404+0000","last_change":"2024-04-25T00:13:23.582102+0000","last_active":"2024-04-25T00:13:27.666404+0000","last_peered":"2024-04-25T00:13:27.666404+0000","last_clean":"2024-04-25T00:13:27.666404+0000","last_became_active":"2024-04-25T00:13:23.581679+0000","last_became_peered":"2024-04-25T00:13:23.581679+0000","last_unstale":"2024-04-25T00:13:27.666404+0000","last_undegraded":"2024-04-25T00:13:27.666404+0000","last_fullsized":"2024-04-25T00:13:27.666404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:05:39.518634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:44.660651+0000","last_change":"2024-04-25T00:13:18.518285+0000","last_active":"2024-04-25T00:13:44.660651+0000","last_peered":"2024-04-25T00:13:44.660651+0000","last_clean":"2024-04-25T00:13:44.660651+0000","last_became_active":"2024-04-25T00:13:18.517826+0000","last_became_peered":"2024-04-25T00:13:18.517826+0000","last_unstale":"2024-04-25T00:13:44.660651+0000","last_undegraded":"2024-04-25T00:13:44.660651+0000","last_fullsized":"2024-04-25T00:13:44.660651+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:12.901225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447582+0000","last_change":"2024-04-25T00:13:19.419691+0000","last_active":"2024-04-25T00:13:27.447582+0000","last_peered":"2024-04-25T00:13:27.447582+0000","last_clean":"2024-04-25T00:13:27.447582+0000","last_became_active":"2024-04-25T00:13:19.419278+0000","last_became_peered":"2024-04-25T00:13:19.419278+0000","last_unstale":"2024-04-25T00:13:27.447582+0000","last_undegraded":"2024-04-25T00:13:27.447582+0000","last_fullsized":"2024-04-25T00:13:27.447582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:19.204170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447604+0000","last_change":"2024-04-25T00:13:23.578445+0000","last_active":"2024-04-25T00:13:27.447604+0000","last_peered":"2024-04-25T00:13:27.447604+0000","last_clean":"2024-04-25T00:13:27.447604+0000","last_became_active":"2024-04-25T00:13:23.578048+0000","last_became_peered":"2024-04-25T00:13:23.578048+0000","last_unstale":"2024-04-25T00:13:27.447604+0000","last_undegraded":"2024-04-25T00:13:27.447604+0000","last_fullsized":"2024-04-25T00:13:27.447604+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:55.072460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.860137+0000","last_change":"2024-04-25T00:13:18.515939+0000","last_active":"2024-04-25T00:13:25.860137+0000","last_peered":"2024-04-25T00:13:25.860137+0000","last_clean":"2024-04-25T00:13:25.860137+0000","last_became_active":"2024-04-25T00:13:18.515597+0000","last_became_peered":"2024-04-25T00:13:18.515597+0000","last_unstale":"2024-04-25T00:13:25.860137+0000","last_undegraded":"2024-04-25T00:13:25.860137+0000","last_fullsized":"2024-04-25T00:13:25.860137+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:03:30.266404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447429+0000","last_change":"2024-04-25T00:13:19.428048+0000","last_active":"2024-04-25T00:13:27.447429+0000","last_peered":"2024-04-25T00:13:27.447429+0000","last_clean":"2024-04-25T00:13:27.447429+0000","last_became_active":"2024-04-25T00:13:19.427622+0000","last_became_peered":"2024-04-25T00:13:19.427622+0000","last_unstale":"2024-04-25T00:13:27.447429+0000","last_undegraded":"2024-04-25T00:13:27.447429+0000","last_fullsized":"2024-04-25T00:13:27.447429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:29:25.039299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.859946+0000","last_change":"2024-04-25T00:13:23.862576+0000","last_active":"2024-04-25T00:13:25.859946+0000","last_peered":"2024-04-25T00:13:25.859946+0000","last_clean":"2024-04-25T00:13:25.859946+0000","last_became_active":"2024-04-25T00:13:23.862024+0000","last_became_peered":"2024-04-25T00:13:23.862024+0000","last_unstale":"2024-04-25T00:13:25.859946+0000","last_undegraded":"2024-04-25T00:13:25.859946+0000","last_fullsized":"2024-04-25T00:13:25.859946+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:17.249680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.577148+0000","last_change":"2024-04-25T00:13:18.522503+0000","last_active":"2024-04-25T00:13:24.577148+0000","last_peered":"2024-04-25T00:13:24.577148+0000","last_clean":"2024-04-25T00:13:24.577148+0000","last_became_active":"2024-04-25T00:13:18.522126+0000","last_became_peered":"2024-04-25T00:13:18.522126+0000","last_unstale":"2024-04-25T00:13:24.577148+0000","last_undegraded":"2024-04-25T00:13:24.577148+0000","last_fullsized":"2024-04-25T00:13:24.577148+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:47:57.530098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.574107+0000","last_change":"2024-04-25T00:13:19.416862+0000","last_active":"2024-04-25T00:13:24.574107+0000","last_peered":"2024-04-25T00:13:24.574107+0000","last_clean":"2024-04-25T00:13:24.574107+0000","last_became_active":"2024-04-25T00:13:19.416375+0000","last_became_peered":"2024-04-25T00:13:19.416375+0000","last_unstale":"2024-04-25T00:13:24.574107+0000","last_undegraded":"2024-04-25T00:13:24.574107+0000","last_fullsized":"2024-04-25T00:13:24.574107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:43:32.031623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.666186+0000","last_change":"2024-04-25T00:13:23.582387+0000","last_active":"2024-04-25T00:13:27.666186+0000","last_peered":"2024-04-25T00:13:27.666186+0000","last_clean":"2024-04-25T00:13:27.666186+0000","last_became_active":"2024-04-25T00:13:23.581984+0000","last_became_peered":"2024-04-25T00:13:23.581984+0000","last_unstale":"2024-04-25T00:13:27.666186+0000","last_undegraded":"2024-04-25T00:13:27.666186+0000","last_fullsized":"2024-04-25T00:13:27.666186+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:07:16.388293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.590004+0000","last_change":"2024-04-25T00:13:18.524121+0000","last_active":"2024-04-25T00:13:25.590004+0000","last_peered":"2024-04-25T00:13:25.590004+0000","last_clean":"2024-04-25T00:13:25.590004+0000","last_became_active":"2024-04-25T00:13:18.523731+0000","last_became_peered":"2024-04-25T00:13:18.523731+0000","last_unstale":"2024-04-25T00:13:25.590004+0000","last_undegraded":"2024-04-25T00:13:25.590004+0000","last_fullsized":"2024-04-25T00:13:25.590004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:22.105952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573942+0000","last_change":"2024-04-25T00:13:19.424753+0000","last_active":"2024-04-25T00:13:24.573942+0000","last_peered":"2024-04-25T00:13:24.573942+0000","last_clean":"2024-04-25T00:13:24.573942+0000","last_became_active":"2024-04-25T00:13:19.423214+0000","last_became_peered":"2024-04-25T00:13:19.423214+0000","last_unstale":"2024-04-25T00:13:24.573942+0000","last_undegraded":"2024-04-25T00:13:24.573942+0000","last_fullsized":"2024-04-25T00:13:24.573942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:41.627442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.666263+0000","last_change":"2024-04-25T00:13:23.862424+0000","last_active":"2024-04-25T00:13:27.666263+0000","last_peered":"2024-04-25T00:13:27.666263+0000","last_clean":"2024-04-25T00:13:27.666263+0000","last_became_active":"2024-04-25T00:13:23.862006+0000","last_became_peered":"2024-04-25T00:13:23.862006+0000","last_unstale":"2024-04-25T00:13:27.666263+0000","last_undegraded":"2024-04-25T00:13:27.666263+0000","last_fullsized":"2024-04-25T00:13:27.666263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:20:59.959545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447446+0000","last_change":"2024-04-25T00:13:19.420617+0000","last_active":"2024-04-25T00:13:27.447446+0000","last_peered":"2024-04-25T00:13:27.447446+0000","last_clean":"2024-04-25T00:13:27.447446+0000","last_became_active":"2024-04-25T00:13:19.420274+0000","last_became_peered":"2024-04-25T00:13:19.420274+0000","last_unstale":"2024-04-25T00:13:27.447446+0000","last_undegraded":"2024-04-25T00:13:27.447446+0000","last_fullsized":"2024-04-25T00:13:27.447446+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:07.515196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:44.660508+0000","last_change":"2024-04-25T00:13:18.529753+0000","last_active":"2024-04-25T00:13:44.660508+0000","last_peered":"2024-04-25T00:13:44.660508+0000","last_clean":"2024-04-25T00:13:44.660508+0000","last_became_active":"2024-04-25T00:13:18.525521+0000","last_became_peered":"2024-04-25T00:13:18.525521+0000","last_unstale":"2024-04-25T00:13:44.660508+0000","last_undegraded":"2024-04-25T00:13:44.660508+0000","last_fullsized":"2024-04-25T00:13:44.660508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:17.044367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.590252+0000","last_change":"2024-04-25T00:13:23.589613+0000","last_active":"2024-04-25T00:13:25.590252+0000","last_peered":"2024-04-25T00:13:25.590252+0000","last_clean":"2024-04-25T00:13:25.590252+0000","last_became_active":"2024-04-25T00:13:23.589323+0000","last_became_peered":"2024-04-25T00:13:23.589323+0000","last_unstale":"2024-04-25T00:13:25.590252+0000","last_undegraded":"2024-04-25T00:13:25.590252+0000","last_fullsized":"2024-04-25T00:13:25.590252+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:34:42.272706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.208757+0000","last_change":"2024-04-25T00:13:19.417775+0000","last_active":"2024-04-25T00:13:27.208757+0000","last_peered":"2024-04-25T00:13:27.208757+0000","last_clean":"2024-04-25T00:13:27.208757+0000","last_became_active":"2024-04-25T00:13:19.415096+0000","last_became_peered":"2024-04-25T00:13:19.415096+0000","last_unstale":"2024-04-25T00:13:27.208757+0000","last_undegraded":"2024-04-25T00:13:27.208757+0000","last_fullsized":"2024-04-25T00:13:27.208757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:54.975923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:43.651456+0000","last_change":"2024-04-25T00:13:18.514560+0000","last_active":"2024-04-25T00:13:43.651456+0000","last_peered":"2024-04-25T00:13:43.651456+0000","last_clean":"2024-04-25T00:13:43.651456+0000","last_became_active":"2024-04-25T00:13:18.514034+0000","last_became_peered":"2024-04-25T00:13:18.514034+0000","last_unstale":"2024-04-25T00:13:43.651456+0000","last_undegraded":"2024-04-25T00:13:43.651456+0000","last_fullsized":"2024-04-25T00:13:43.651456+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:35:18.110206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.589663+0000","last_change":"2024-04-25T00:12:01.540807+0000","last_active":"2024-04-25T00:13:25.589663+0000","last_peered":"2024-04-25T00:13:25.589663+0000","last_clean":"2024-04-25T00:13:25.589663+0000","last_became_active":"2024-04-25T00:12:01.540272+0000","last_became_peered":"2024-04-25T00:12:01.540272+0000","last_unstale":"2024-04-25T00:13:25.589663+0000","last_undegraded":"2024-04-25T00:13:25.589663+0000","last_fullsized":"2024-04-25T00:13:25.589663+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:08:34.007429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:08:34.007429+0000","last_clean_scrub_stamp":"2024-04-25T00:08:34.007429+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:50:14.642604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.448376+0000","last_change":"2024-04-25T00:13:23.862493+0000","last_active":"2024-04-25T00:13:27.448376+0000","last_peered":"2024-04-25T00:13:27.448376+0000","last_clean":"2024-04-25T00:13:27.448376+0000","last_became_active":"2024-04-25T00:13:23.861972+0000","last_became_peered":"2024-04-25T00:13:23.861972+0000","last_unstale":"2024-04-25T00:13:27.448376+0000","last_undegraded":"2024-04-25T00:13:27.448376+0000","last_fullsized":"2024-04-25T00:13:27.448376+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:49:13.507357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.665746+0000","last_change":"2024-04-25T00:13:19.421920+0000","last_active":"2024-04-25T00:13:27.665746+0000","last_peered":"2024-04-25T00:13:27.665746+0000","last_clean":"2024-04-25T00:13:27.665746+0000","last_became_active":"2024-04-25T00:13:19.421585+0000","last_became_peered":"2024-04-25T00:13:19.421585+0000","last_unstale":"2024-04-25T00:13:27.665746+0000","last_undegraded":"2024-04-25T00:13:27.665746+0000","last_fullsized":"2024-04-25T00:13:27.665746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:53:24.937520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:44.659765+0000","last_change":"2024-04-25T00:13:18.527062+0000","last_active":"2024-04-25T00:13:44.659765+0000","last_peered":"2024-04-25T00:13:44.659765+0000","last_clean":"2024-04-25T00:13:44.659765+0000","last_became_active":"2024-04-25T00:13:18.526338+0000","last_became_peered":"2024-04-25T00:13:18.526338+0000","last_unstale":"2024-04-25T00:13:44.659765+0000","last_undegraded":"2024-04-25T00:13:44.659765+0000","last_fullsized":"2024-04-25T00:13:44.659765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:21.756728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447387+0000","last_change":"2024-04-25T00:13:23.575162+0000","last_active":"2024-04-25T00:13:27.447387+0000","last_peered":"2024-04-25T00:13:27.447387+0000","last_clean":"2024-04-25T00:13:27.447387+0000","last_became_active":"2024-04-25T00:13:23.574722+0000","last_became_peered":"2024-04-25T00:13:23.574722+0000","last_unstale":"2024-04-25T00:13:27.447387+0000","last_undegraded":"2024-04-25T00:13:27.447387+0000","last_fullsized":"2024-04-25T00:13:27.447387+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:52.958539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.860339+0000","last_change":"2024-04-25T00:13:18.523019+0000","last_active":"2024-04-25T00:13:25.860339+0000","last_peered":"2024-04-25T00:13:25.860339+0000","last_clean":"2024-04-25T00:13:25.860339+0000","last_became_active":"2024-04-25T00:13:18.522677+0000","last_became_peered":"2024-04-25T00:13:18.522677+0000","last_unstale":"2024-04-25T00:13:25.860339+0000","last_undegraded":"2024-04-25T00:13:25.860339+0000","last_fullsized":"2024-04-25T00:13:25.860339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:14.470154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.666228+0000","last_change":"2024-04-25T00:13:19.417672+0000","last_active":"2024-04-25T00:13:27.666228+0000","last_peered":"2024-04-25T00:13:27.666228+0000","last_clean":"2024-04-25T00:13:27.666228+0000","last_became_active":"2024-04-25T00:13:19.417323+0000","last_became_peered":"2024-04-25T00:13:19.417323+0000","last_unstale":"2024-04-25T00:13:27.666228+0000","last_undegraded":"2024-04-25T00:13:27.666228+0000","last_fullsized":"2024-04-25T00:13:27.666228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:53.734784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.208782+0000","last_change":"2024-04-25T00:13:23.862320+0000","last_active":"2024-04-25T00:13:27.208782+0000","last_peered":"2024-04-25T00:13:27.208782+0000","last_clean":"2024-04-25T00:13:27.208782+0000","last_became_active":"2024-04-25T00:13:23.861895+0000","last_became_peered":"2024-04-25T00:13:23.861895+0000","last_unstale":"2024-04-25T00:13:27.208782+0000","last_undegraded":"2024-04-25T00:13:27.208782+0000","last_fullsized":"2024-04-25T00:13:27.208782+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:16:21.324727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.577535+0000","last_change":"2024-04-25T00:13:18.520989+0000","last_active":"2024-04-25T00:13:24.577535+0000","last_peered":"2024-04-25T00:13:24.577535+0000","last_clean":"2024-04-25T00:13:24.577535+0000","last_became_active":"2024-04-25T00:13:18.520566+0000","last_became_peered":"2024-04-25T00:13:18.520566+0000","last_unstale":"2024-04-25T00:13:24.577535+0000","last_undegraded":"2024-04-25T00:13:24.577535+0000","last_fullsized":"2024-04-25T00:13:24.577535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:22:32.532352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.208988+0000","last_change":"2024-04-25T00:13:19.422741+0000","last_active":"2024-04-25T00:13:27.208988+0000","last_peered":"2024-04-25T00:13:27.208988+0000","last_clean":"2024-04-25T00:13:27.208988+0000","last_became_active":"2024-04-25T00:13:19.422379+0000","last_became_peered":"2024-04-25T00:13:19.422379+0000","last_unstale":"2024-04-25T00:13:27.208988+0000","last_undegraded":"2024-04-25T00:13:27.208988+0000","last_fullsized":"2024-04-25T00:13:27.208988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:41.337641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573283+0000","last_change":"2024-04-25T00:13:23.577888+0000","last_active":"2024-04-25T00:13:24.573283+0000","last_peered":"2024-04-25T00:13:24.573283+0000","last_clean":"2024-04-25T00:13:24.573283+0000","last_became_active":"2024-04-25T00:13:23.577440+0000","last_became_peered":"2024-04-25T00:13:23.577440+0000","last_unstale":"2024-04-25T00:13:24.573283+0000","last_undegraded":"2024-04-25T00:13:24.573283+0000","last_fullsized":"2024-04-25T00:13:24.573283+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:03.760277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:43.650947+0000","last_change":"2024-04-25T00:13:18.527996+0000","last_active":"2024-04-25T00:13:43.650947+0000","last_peered":"2024-04-25T00:13:43.650947+0000","last_clean":"2024-04-25T00:13:43.650947+0000","last_became_active":"2024-04-25T00:13:18.527636+0000","last_became_peered":"2024-04-25T00:13:18.527636+0000","last_unstale":"2024-04-25T00:13:43.650947+0000","last_undegraded":"2024-04-25T00:13:43.650947+0000","last_fullsized":"2024-04-25T00:13:43.650947+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:52.137108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446673+0000","last_change":"2024-04-25T00:13:19.425538+0000","last_active":"2024-04-25T00:13:27.446673+0000","last_peered":"2024-04-25T00:13:27.446673+0000","last_clean":"2024-04-25T00:13:27.446673+0000","last_became_active":"2024-04-25T00:13:19.420790+0000","last_became_peered":"2024-04-25T00:13:19.420790+0000","last_unstale":"2024-04-25T00:13:27.446673+0000","last_undegraded":"2024-04-25T00:13:27.446673+0000","last_fullsized":"2024-04-25T00:13:27.446673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:04.319759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771854+0000","last_change":"2024-04-25T00:13:23.863187+0000","last_active":"2024-04-25T00:13:26.771854+0000","last_peered":"2024-04-25T00:13:26.771854+0000","last_clean":"2024-04-25T00:13:26.771854+0000","last_became_active":"2024-04-25T00:13:23.862206+0000","last_became_peered":"2024-04-25T00:13:23.862206+0000","last_unstale":"2024-04-25T00:13:26.771854+0000","last_undegraded":"2024-04-25T00:13:26.771854+0000","last_fullsized":"2024-04-25T00:13:26.771854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:57:17.395007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:43.650679+0000","last_change":"2024-04-25T00:13:18.529224+0000","last_active":"2024-04-25T00:13:43.650679+0000","last_peered":"2024-04-25T00:13:43.650679+0000","last_clean":"2024-04-25T00:13:43.650679+0000","last_became_active":"2024-04-25T00:13:18.528908+0000","last_became_peered":"2024-04-25T00:13:18.528908+0000","last_unstale":"2024-04-25T00:13:43.650679+0000","last_undegraded":"2024-04-25T00:13:43.650679+0000","last_fullsized":"2024-04-25T00:13:43.650679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:27.035726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.859754+0000","last_change":"2024-04-25T00:13:19.422223+0000","last_active":"2024-04-25T00:13:25.859754+0000","last_peered":"2024-04-25T00:13:25.859754+0000","last_clean":"2024-04-25T00:13:25.859754+0000","last_became_active":"2024-04-25T00:13:19.421524+0000","last_became_peered":"2024-04-25T00:13:19.421524+0000","last_unstale":"2024-04-25T00:13:25.859754+0000","last_undegraded":"2024-04-25T00:13:25.859754+0000","last_fullsized":"2024-04-25T00:13:25.859754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:17:01.849503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.577624+0000","last_change":"2024-04-25T00:13:23.583015+0000","last_active":"2024-04-25T00:13:24.577624+0000","last_peered":"2024-04-25T00:13:24.577624+0000","last_clean":"2024-04-25T00:13:24.577624+0000","last_became_active":"2024-04-25T00:13:23.582612+0000","last_became_peered":"2024-04-25T00:13:23.582612+0000","last_unstale":"2024-04-25T00:13:24.577624+0000","last_undegraded":"2024-04-25T00:13:24.577624+0000","last_fullsized":"2024-04-25T00:13:24.577624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:36.872281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:43.651932+0000","last_change":"2024-04-25T00:13:18.530027+0000","last_active":"2024-04-25T00:13:43.651932+0000","last_peered":"2024-04-25T00:13:43.651932+0000","last_clean":"2024-04-25T00:13:43.651932+0000","last_became_active":"2024-04-25T00:13:18.529717+0000","last_became_peered":"2024-04-25T00:13:18.529717+0000","last_unstale":"2024-04-25T00:13:43.651932+0000","last_undegraded":"2024-04-25T00:13:43.651932+0000","last_fullsized":"2024-04-25T00:13:43.651932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:23:31.020637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446886+0000","last_change":"2024-04-25T00:13:19.428650+0000","last_active":"2024-04-25T00:13:27.446886+0000","last_peered":"2024-04-25T00:13:27.446886+0000","last_clean":"2024-04-25T00:13:27.446886+0000","last_became_active":"2024-04-25T00:13:19.428156+0000","last_became_peered":"2024-04-25T00:13:19.428156+0000","last_unstale":"2024-04-25T00:13:27.446886+0000","last_undegraded":"2024-04-25T00:13:27.446886+0000","last_fullsized":"2024-04-25T00:13:27.446886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:43.094529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771776+0000","last_change":"2024-04-25T00:13:23.579125+0000","last_active":"2024-04-25T00:13:26.771776+0000","last_peered":"2024-04-25T00:13:26.771776+0000","last_clean":"2024-04-25T00:13:26.771776+0000","last_became_active":"2024-04-25T00:13:23.578728+0000","last_became_peered":"2024-04-25T00:13:23.578728+0000","last_unstale":"2024-04-25T00:13:26.771776+0000","last_undegraded":"2024-04-25T00:13:26.771776+0000","last_fullsized":"2024-04-25T00:13:26.771776+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:16:16.834399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573393+0000","last_change":"2024-04-25T00:13:18.528564+0000","last_active":"2024-04-25T00:13:24.573393+0000","last_peered":"2024-04-25T00:13:24.573393+0000","last_clean":"2024-04-25T00:13:24.573393+0000","last_became_active":"2024-04-25T00:13:18.528203+0000","last_became_peered":"2024-04-25T00:13:18.528203+0000","last_unstale":"2024-04-25T00:13:24.573393+0000","last_undegraded":"2024-04-25T00:13:24.573393+0000","last_fullsized":"2024-04-25T00:13:24.573393+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:47.087562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.209152+0000","last_change":"2024-04-25T00:13:19.414848+0000","last_active":"2024-04-25T00:13:27.209152+0000","last_peered":"2024-04-25T00:13:27.209152+0000","last_clean":"2024-04-25T00:13:27.209152+0000","last_became_active":"2024-04-25T00:13:19.414370+0000","last_became_peered":"2024-04-25T00:13:19.414370+0000","last_unstale":"2024-04-25T00:13:27.209152+0000","last_undegraded":"2024-04-25T00:13:27.209152+0000","last_fullsized":"2024-04-25T00:13:27.209152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:41:08.779171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771847+0000","last_change":"2024-04-25T00:13:23.862577+0000","last_active":"2024-04-25T00:13:26.771847+0000","last_peered":"2024-04-25T00:13:26.771847+0000","last_clean":"2024-04-25T00:13:26.771847+0000","last_became_active":"2024-04-25T00:13:23.862098+0000","last_became_peered":"2024-04-25T00:13:23.862098+0000","last_unstale":"2024-04-25T00:13:26.771847+0000","last_undegraded":"2024-04-25T00:13:26.771847+0000","last_fullsized":"2024-04-25T00:13:26.771847+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:08:00.344041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:43.651052+0000","last_change":"2024-04-25T00:13:18.520638+0000","last_active":"2024-04-25T00:13:43.651052+0000","last_peered":"2024-04-25T00:13:43.651052+0000","last_clean":"2024-04-25T00:13:43.651052+0000","last_became_active":"2024-04-25T00:13:18.520268+0000","last_became_peered":"2024-04-25T00:13:18.520268+0000","last_unstale":"2024-04-25T00:13:43.651052+0000","last_undegraded":"2024-04-25T00:13:43.651052+0000","last_fullsized":"2024-04-25T00:13:43.651052+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:31:00.094521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.590016+0000","last_change":"2024-04-25T00:13:19.427992+0000","last_active":"2024-04-25T00:13:25.590016+0000","last_peered":"2024-04-25T00:13:25.590016+0000","last_clean":"2024-04-25T00:13:25.590016+0000","last_became_active":"2024-04-25T00:13:19.427665+0000","last_became_peered":"2024-04-25T00:13:19.427665+0000","last_unstale":"2024-04-25T00:13:25.590016+0000","last_undegraded":"2024-04-25T00:13:25.590016+0000","last_fullsized":"2024-04-25T00:13:25.590016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:12:31.582185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.397517+0000","last_change":"2024-04-25T00:13:23.582430+0000","last_active":"2024-04-25T00:13:27.397517+0000","last_peered":"2024-04-25T00:13:27.397517+0000","last_clean":"2024-04-25T00:13:27.397517+0000","last_became_active":"2024-04-25T00:13:23.582008+0000","last_became_peered":"2024-04-25T00:13:23.582008+0000","last_unstale":"2024-04-25T00:13:27.397517+0000","last_undegraded":"2024-04-25T00:13:27.397517+0000","last_fullsized":"2024-04-25T00:13:27.397517+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:11.068720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573822+0000","last_change":"2024-04-25T00:13:18.528030+0000","last_active":"2024-04-25T00:13:24.573822+0000","last_peered":"2024-04-25T00:13:24.573822+0000","last_clean":"2024-04-25T00:13:24.573822+0000","last_became_active":"2024-04-25T00:13:18.527636+0000","last_became_peered":"2024-04-25T00:13:18.527636+0000","last_unstale":"2024-04-25T00:13:24.573822+0000","last_undegraded":"2024-04-25T00:13:24.573822+0000","last_fullsized":"2024-04-25T00:13:24.573822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:36.311810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.590460+0000","last_change":"2024-04-25T00:13:19.428181+0000","last_active":"2024-04-25T00:13:25.590460+0000","last_peered":"2024-04-25T00:13:25.590460+0000","last_clean":"2024-04-25T00:13:25.590460+0000","last_became_active":"2024-04-25T00:13:19.427785+0000","last_became_peered":"2024-04-25T00:13:19.427785+0000","last_unstale":"2024-04-25T00:13:25.590460+0000","last_undegraded":"2024-04-25T00:13:25.590460+0000","last_fullsized":"2024-04-25T00:13:25.590460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:31.745679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446376+0000","last_change":"2024-04-25T00:13:23.590162+0000","last_active":"2024-04-25T00:13:27.446376+0000","last_peered":"2024-04-25T00:13:27.446376+0000","last_clean":"2024-04-25T00:13:27.446376+0000","last_became_active":"2024-04-25T00:13:23.589708+0000","last_became_peered":"2024-04-25T00:13:23.589708+0000","last_unstale":"2024-04-25T00:13:27.446376+0000","last_undegraded":"2024-04-25T00:13:27.446376+0000","last_fullsized":"2024-04-25T00:13:27.446376+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:39:15.606032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.448275+0000","last_change":"2024-04-25T00:13:18.529622+0000","last_active":"2024-04-25T00:13:27.448275+0000","last_peered":"2024-04-25T00:13:27.448275+0000","last_clean":"2024-04-25T00:13:27.448275+0000","last_became_active":"2024-04-25T00:13:18.529305+0000","last_became_peered":"2024-04-25T00:13:18.529305+0000","last_unstale":"2024-04-25T00:13:27.448275+0000","last_undegraded":"2024-04-25T00:13:27.448275+0000","last_fullsized":"2024-04-25T00:13:27.448275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:10:43.334712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.590139+0000","last_change":"2024-04-25T00:13:19.421783+0000","last_active":"2024-04-25T00:13:25.590139+0000","last_peered":"2024-04-25T00:13:25.590139+0000","last_clean":"2024-04-25T00:13:25.590139+0000","last_became_active":"2024-04-25T00:13:19.421447+0000","last_became_peered":"2024-04-25T00:13:19.421447+0000","last_unstale":"2024-04-25T00:13:25.590139+0000","last_undegraded":"2024-04-25T00:13:25.590139+0000","last_fullsized":"2024-04-25T00:13:25.590139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:16.107220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447155+0000","last_change":"2024-04-25T00:13:23.578252+0000","last_active":"2024-04-25T00:13:27.447155+0000","last_peered":"2024-04-25T00:13:27.447155+0000","last_clean":"2024-04-25T00:13:27.447155+0000","last_became_active":"2024-04-25T00:13:23.577779+0000","last_became_peered":"2024-04-25T00:13:23.577779+0000","last_unstale":"2024-04-25T00:13:27.447155+0000","last_undegraded":"2024-04-25T00:13:27.447155+0000","last_fullsized":"2024-04-25T00:13:27.447155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:43.822447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.666353+0000","last_change":"2024-04-25T00:13:18.523583+0000","last_active":"2024-04-25T00:13:27.666353+0000","last_peered":"2024-04-25T00:13:27.666353+0000","last_clean":"2024-04-25T00:13:27.666353+0000","last_became_active":"2024-04-25T00:13:18.523238+0000","last_became_peered":"2024-04-25T00:13:18.523238+0000","last_unstale":"2024-04-25T00:13:27.666353+0000","last_undegraded":"2024-04-25T00:13:27.666353+0000","last_fullsized":"2024-04-25T00:13:27.666353+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:03.869276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.590390+0000","last_change":"2024-04-25T00:13:19.426021+0000","last_active":"2024-04-25T00:13:25.590390+0000","last_peered":"2024-04-25T00:13:25.590390+0000","last_clean":"2024-04-25T00:13:25.590390+0000","last_became_active":"2024-04-25T00:13:19.425676+0000","last_became_peered":"2024-04-25T00:13:19.425676+0000","last_unstale":"2024-04-25T00:13:25.590390+0000","last_undegraded":"2024-04-25T00:13:25.590390+0000","last_fullsized":"2024-04-25T00:13:25.590390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:10.044202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446979+0000","last_change":"2024-04-25T00:13:19.422402+0000","last_active":"2024-04-25T00:13:27.446979+0000","last_peered":"2024-04-25T00:13:27.446979+0000","last_clean":"2024-04-25T00:13:27.446979+0000","last_became_active":"2024-04-25T00:13:19.422057+0000","last_became_peered":"2024-04-25T00:13:19.422057+0000","last_unstale":"2024-04-25T00:13:27.446979+0000","last_undegraded":"2024-04-25T00:13:27.446979+0000","last_fullsized":"2024-04-25T00:13:27.446979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:31:43.703030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573658+0000","last_change":"2024-04-25T00:13:19.425434+0000","last_active":"2024-04-25T00:13:24.573658+0000","last_peered":"2024-04-25T00:13:24.573658+0000","last_clean":"2024-04-25T00:13:24.573658+0000","last_became_active":"2024-04-25T00:13:19.425051+0000","last_became_peered":"2024-04-25T00:13:19.425051+0000","last_unstale":"2024-04-25T00:13:24.573658+0000","last_undegraded":"2024-04-25T00:13:24.573658+0000","last_fullsized":"2024-04-25T00:13:24.573658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:17.568415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573288+0000","last_change":"2024-04-25T00:13:19.413835+0000","last_active":"2024-04-25T00:13:24.573288+0000","last_peered":"2024-04-25T00:13:24.573288+0000","last_clean":"2024-04-25T00:13:24.573288+0000","last_became_active":"2024-04-25T00:13:19.413406+0000","last_became_peered":"2024-04-25T00:13:19.413406+0000","last_unstale":"2024-04-25T00:13:24.573288+0000","last_undegraded":"2024-04-25T00:13:24.573288+0000","last_fullsized":"2024-04-25T00:13:24.573288+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:33.852391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446547+0000","last_change":"2024-04-25T00:13:19.426875+0000","last_active":"2024-04-25T00:13:27.446547+0000","last_peered":"2024-04-25T00:13:27.446547+0000","last_clean":"2024-04-25T00:13:27.446547+0000","last_became_active":"2024-04-25T00:13:19.425828+0000","last_became_peered":"2024-04-25T00:13:19.425828+0000","last_unstale":"2024-04-25T00:13:27.446547+0000","last_undegraded":"2024-04-25T00:13:27.446547+0000","last_fullsized":"2024-04-25T00:13:27.446547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:21.119551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.666359+0000","last_change":"2024-04-25T00:13:19.423637+0000","last_active":"2024-04-25T00:13:27.666359+0000","last_peered":"2024-04-25T00:13:27.666359+0000","last_clean":"2024-04-25T00:13:27.666359+0000","last_became_active":"2024-04-25T00:13:19.423272+0000","last_became_peered":"2024-04-25T00:13:19.423272+0000","last_unstale":"2024-04-25T00:13:27.666359+0000","last_undegraded":"2024-04-25T00:13:27.666359+0000","last_fullsized":"2024-04-25T00:13:27.666359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:32:38.794966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447519+0000","last_change":"2024-04-25T00:13:19.423538+0000","last_active":"2024-04-25T00:13:27.447519+0000","last_peered":"2024-04-25T00:13:27.447519+0000","last_clean":"2024-04-25T00:13:27.447519+0000","last_became_active":"2024-04-25T00:13:19.423028+0000","last_became_peered":"2024-04-25T00:13:19.423028+0000","last_unstale":"2024-04-25T00:13:27.447519+0000","last_undegraded":"2024-04-25T00:13:27.447519+0000","last_fullsized":"2024-04-25T00:13:27.447519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:29:41.975044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.574048+0000","last_change":"2024-04-25T00:13:19.416899+0000","last_active":"2024-04-25T00:13:24.574048+0000","last_peered":"2024-04-25T00:13:24.574048+0000","last_clean":"2024-04-25T00:13:24.574048+0000","last_became_active":"2024-04-25T00:13:19.416505+0000","last_became_peered":"2024-04-25T00:13:19.416505+0000","last_unstale":"2024-04-25T00:13:24.574048+0000","last_undegraded":"2024-04-25T00:13:24.574048+0000","last_fullsized":"2024-04-25T00:13:24.574048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:37:40.204119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.209117+0000","last_change":"2024-04-25T00:13:19.426215+0000","last_active":"2024-04-25T00:13:27.209117+0000","last_peered":"2024-04-25T00:13:27.209117+0000","last_clean":"2024-04-25T00:13:27.209117+0000","last_became_active":"2024-04-25T00:13:19.422804+0000","last_became_peered":"2024-04-25T00:13:19.422804+0000","last_unstale":"2024-04-25T00:13:27.209117+0000","last_undegraded":"2024-04-25T00:13:27.209117+0000","last_fullsized":"2024-04-25T00:13:27.209117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:24:44.630891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447319+0000","last_change":"2024-04-25T00:13:19.413564+0000","last_active":"2024-04-25T00:13:27.447319+0000","last_peered":"2024-04-25T00:13:27.447319+0000","last_clean":"2024-04-25T00:13:27.447319+0000","last_became_active":"2024-04-25T00:13:19.413223+0000","last_became_peered":"2024-04-25T00:13:19.413223+0000","last_unstale":"2024-04-25T00:13:27.447319+0000","last_undegraded":"2024-04-25T00:13:27.447319+0000","last_fullsized":"2024-04-25T00:13:27.447319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:45.629018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.208768+0000","last_change":"2024-04-25T00:13:19.425945+0000","last_active":"2024-04-25T00:13:27.208768+0000","last_peered":"2024-04-25T00:13:27.208768+0000","last_clean":"2024-04-25T00:13:27.208768+0000","last_became_active":"2024-04-25T00:13:19.422499+0000","last_became_peered":"2024-04-25T00:13:19.422499+0000","last_unstale":"2024-04-25T00:13:27.208768+0000","last_undegraded":"2024-04-25T00:13:27.208768+0000","last_fullsized":"2024-04-25T00:13:27.208768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:45.299300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.397742+0000","last_change":"2024-04-25T00:13:19.418857+0000","last_active":"2024-04-25T00:13:27.397742+0000","last_peered":"2024-04-25T00:13:27.397742+0000","last_clean":"2024-04-25T00:13:27.397742+0000","last_became_active":"2024-04-25T00:13:19.418535+0000","last_became_peered":"2024-04-25T00:13:19.418535+0000","last_unstale":"2024-04-25T00:13:27.397742+0000","last_undegraded":"2024-04-25T00:13:27.397742+0000","last_fullsized":"2024-04-25T00:13:27.397742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:44.288129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.665972+0000","last_change":"2024-04-25T00:13:19.420550+0000","last_active":"2024-04-25T00:13:27.665972+0000","last_peered":"2024-04-25T00:13:27.665972+0000","last_clean":"2024-04-25T00:13:27.665972+0000","last_became_active":"2024-04-25T00:13:19.420235+0000","last_became_peered":"2024-04-25T00:13:19.420235+0000","last_unstale":"2024-04-25T00:13:27.665972+0000","last_undegraded":"2024-04-25T00:13:27.665972+0000","last_fullsized":"2024-04-25T00:13:27.665972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:59:36.543776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447420+0000","last_change":"2024-04-25T00:13:19.429995+0000","last_active":"2024-04-25T00:13:27.447420+0000","last_peered":"2024-04-25T00:13:27.447420+0000","last_clean":"2024-04-25T00:13:27.447420+0000","last_became_active":"2024-04-25T00:13:19.429185+0000","last_became_peered":"2024-04-25T00:13:19.429185+0000","last_unstale":"2024-04-25T00:13:27.447420+0000","last_undegraded":"2024-04-25T00:13:27.447420+0000","last_fullsized":"2024-04-25T00:13:27.447420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:29:27.974479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":213064,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4954,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":18024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383699,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645334,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1071619,"data_compressed":50438,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":477841,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":1068787,"data_compressed":50438,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495660,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738443,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":477909,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":481091,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":970752,"data_stored":477319,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":50438,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:13:48.400 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:13:48.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.535+0000 7fc84b1f9640 1 -- 172.21.15.27:0/79839990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc844102250 msgr2=0x7fc844108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:48.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.535+0000 7fc84b1f9640 1 --2- 172.21.15.27:0/79839990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc844102250 0x7fc844108ae0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc83c009ca0 tx=0x7fc83c02a8c0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.535+0000 7fc84b1f9640 1 -- 172.21.15.27:0/79839990 shutdown_connections 2024-04-25T00:13:48.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.535+0000 7fc84b1f9640 1 --2- 172.21.15.27:0/79839990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc844102250 0x7fc844108ae0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.535+0000 7fc84b1f9640 1 --2- 172.21.15.27:0/79839990 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc844101890 0x7fc844101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.535+0000 7fc84b1f9640 1 --2- 172.21.15.27:0/79839990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8441006a0 0x7fc844100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.535+0000 7fc84b1f9640 1 -- 172.21.15.27:0/79839990 >> 172.21.15.27:0/79839990 conn(0x7fc8440fbc30 msgr2=0x7fc8440fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:48.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.535+0000 7fc84b1f9640 1 -- 172.21.15.27:0/79839990 shutdown_connections 2024-04-25T00:13:48.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.535+0000 7fc84b1f9640 1 -- 172.21.15.27:0/79839990 wait complete. 2024-04-25T00:13:48.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.536+0000 7fc84b1f9640 1 Processor -- start 2024-04-25T00:13:48.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.536+0000 7fc84b1f9640 1 -- start start 2024-04-25T00:13:48.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.536+0000 7fc84b1f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8441006a0 0x7fc844197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:48.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.536+0000 7fc84b1f9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc844101890 0x7fc844198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:48.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.536+0000 7fc84b1f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc844102250 0x7fc84419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:48.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.536+0000 7fc84b1f9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc84410c300 con 0x7fc844102250 2024-04-25T00:13:48.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.536+0000 7fc84b1f9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc84410c180 con 0x7fc844101890 2024-04-25T00:13:48.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.536+0000 7fc84b1f9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc84410c480 con 0x7fc8441006a0 2024-04-25T00:13:48.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.536+0000 7fc849796640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc844102250 0x7fc84419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:48.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.536+0000 7fc83bfff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc844101890 0x7fc844198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:48.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.536+0000 7fc83bfff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc844101890 0x7fc844198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:55950/0 (socket says 172.21.15.27:55950) 2024-04-25T00:13:48.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.536+0000 7fc83bfff640 1 -- 172.21.15.27:0/3987420994 learned_addr learned my addr 172.21.15.27:0/3987420994 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:48.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.536+0000 7fc848f95640 1 --2- 172.21.15.27:0/3987420994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8441006a0 0x7fc844197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:48.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.537+0000 7fc849796640 1 -- 172.21.15.27:0/3987420994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8441006a0 msgr2=0x7fc844197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:48.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.537+0000 7fc849796640 1 --2- 172.21.15.27:0/3987420994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8441006a0 0x7fc844197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.537+0000 7fc849796640 1 -- 172.21.15.27:0/3987420994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc844101890 msgr2=0x7fc844198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:48.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.537+0000 7fc849796640 1 --2- 172.21.15.27:0/3987420994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc844101890 0x7fc844198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.537+0000 7fc849796640 1 -- 172.21.15.27:0/3987420994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84419f6f0 con 0x7fc844102250 2024-04-25T00:13:48.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.537+0000 7fc83bfff640 1 --2- 172.21.15.27:0/3987420994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc844101890 0x7fc844198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:13:48.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.537+0000 7fc849796640 1 --2- 172.21.15.27:0/3987420994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc844102250 0x7fc84419f080 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc83c009bf0 tx=0x7fc83c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:48.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.537+0000 7fc839ffb640 1 -- 172.21.15.27:0/3987420994 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c003b70 con 0x7fc844102250 2024-04-25T00:13:48.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.537+0000 7fc839ffb640 1 -- 172.21.15.27:0/3987420994 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc83c003d10 con 0x7fc844102250 2024-04-25T00:13:48.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.537+0000 7fc84b1f9640 1 -- 172.21.15.27:0/3987420994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc84419f980 con 0x7fc844102250 2024-04-25T00:13:48.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.537+0000 7fc839ffb640 1 -- 172.21.15.27:0/3987420994 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c02fc10 con 0x7fc844102250 2024-04-25T00:13:48.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.537+0000 7fc84b1f9640 1 -- 172.21.15.27:0/3987420994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc84419fed0 con 0x7fc844102250 2024-04-25T00:13:48.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.538+0000 7fc84b1f9640 1 -- 172.21.15.27:0/3987420994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8441a0130 con 0x7fc844102250 2024-04-25T00:13:48.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.539+0000 7fc839ffb640 1 -- 172.21.15.27:0/3987420994 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc83c004050 con 0x7fc844102250 2024-04-25T00:13:48.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.540+0000 7fc839ffb640 1 --2- 172.21.15.27:0/3987420994 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc82c0a2b90 0x7fc82c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:48.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.540+0000 7fc839ffb640 1 -- 172.21.15.27:0/3987420994 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc83c0f06a0 con 0x7fc844102250 2024-04-25T00:13:48.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.540+0000 7fc848f95640 1 --2- 172.21.15.27:0/3987420994 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc82c0a2b90 0x7fc82c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:48.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.541+0000 7fc848f95640 1 --2- 172.21.15.27:0/3987420994 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc82c0a2b90 0x7fc82c0a5050 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc8441016f0 tx=0x7fc84000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:48.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.544+0000 7fc839ffb640 1 -- 172.21.15.27:0/3987420994 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc83c0bd4e0 con 0x7fc844102250 2024-04-25T00:13:48.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.588+0000 7fc839ffb640 1 -- 172.21.15.27:0/3987420994 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc83c0b9450 con 0x7fc844102250 2024-04-25T00:13:48.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.733+0000 7fc84b1f9640 1 -- 172.21.15.27:0/3987420994 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc8441a0300 con 0x7fc82c0a2b90 2024-04-25T00:13:48.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.743+0000 7fc839ffb640 1 -- 172.21.15.27:0/3987420994 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340525 (secure 0 0 0) 0x7fc8441a0300 con 0x7fc82c0a2b90 2024-04-25T00:13:48.747 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T00:13:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.747+0000 7fc84b1f9640 1 -- 172.21.15.27:0/3987420994 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc82c0a2b90 msgr2=0x7fc82c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.747+0000 7fc84b1f9640 1 --2- 172.21.15.27:0/3987420994 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc82c0a2b90 0x7fc82c0a5050 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc8441016f0 tx=0x7fc84000b390 comp rx=0 tx=0).stop 2024-04-25T00:13:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.747+0000 7fc84b1f9640 1 -- 172.21.15.27:0/3987420994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc844102250 msgr2=0x7fc84419f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.747+0000 7fc84b1f9640 1 --2- 172.21.15.27:0/3987420994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc844102250 0x7fc84419f080 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc83c009bf0 tx=0x7fc83c004ab0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.747+0000 7fc84b1f9640 1 -- 172.21.15.27:0/3987420994 shutdown_connections 2024-04-25T00:13:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.747+0000 7fc84b1f9640 1 --2- 172.21.15.27:0/3987420994 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc82c0a2b90 0x7fc82c0a5050 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.747+0000 7fc84b1f9640 1 --2- 172.21.15.27:0/3987420994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc844102250 0x7fc84419f080 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.747+0000 7fc84b1f9640 1 --2- 172.21.15.27:0/3987420994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc844101890 0x7fc844198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.747+0000 7fc84b1f9640 1 --2- 172.21.15.27:0/3987420994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8441006a0 0x7fc844197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.747+0000 7fc84b1f9640 1 -- 172.21.15.27:0/3987420994 >> 172.21.15.27:0/3987420994 conn(0x7fc8440fbc30 msgr2=0x7fc8440fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.747+0000 7fc84b1f9640 1 -- 172.21.15.27:0/3987420994 shutdown_connections 2024-04-25T00:13:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:48.747+0000 7fc84b1f9640 1 -- 172.21.15.27:0/3987420994 wait complete. 2024-04-25T00:13:48.758 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-04-25T00:13:47.576732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595322,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":270,"num_write_kb":2974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":213,"ondisk_log_size":213,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":348232,"kb_used_data":11544,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124593592,"statfs":{"total":1151940427776,"available":1151583838208,"internally_reserved":0,"allocated":11821056,"data_stored":7513434,"data_compressed":283200,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":344634681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":455,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003941"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447749+0000","last_change":"2024-04-25T00:13:19.429658+0000","last_active":"2024-04-25T00:13:27.447749+0000","last_peered":"2024-04-25T00:13:27.447749+0000","last_clean":"2024-04-25T00:13:27.447749+0000","last_became_active":"2024-04-25T00:13:19.428426+0000","last_became_peered":"2024-04-25T00:13:19.428426+0000","last_unstale":"2024-04-25T00:13:27.447749+0000","last_undegraded":"2024-04-25T00:13:27.447749+0000","last_fullsized":"2024-04-25T00:13:27.447749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:10.814361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446796+0000","last_change":"2024-04-25T00:13:19.425504+0000","last_active":"2024-04-25T00:13:27.446796+0000","last_peered":"2024-04-25T00:13:27.446796+0000","last_clean":"2024-04-25T00:13:27.446796+0000","last_became_active":"2024-04-25T00:13:19.425170+0000","last_became_peered":"2024-04-25T00:13:19.425170+0000","last_unstale":"2024-04-25T00:13:27.446796+0000","last_undegraded":"2024-04-25T00:13:27.446796+0000","last_fullsized":"2024-04-25T00:13:27.446796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:31:13.826981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446369+0000","last_change":"2024-04-25T00:13:19.420842+0000","last_active":"2024-04-25T00:13:27.446369+0000","last_peered":"2024-04-25T00:13:27.446369+0000","last_clean":"2024-04-25T00:13:27.446369+0000","last_became_active":"2024-04-25T00:13:19.420241+0000","last_became_peered":"2024-04-25T00:13:19.420241+0000","last_unstale":"2024-04-25T00:13:27.446369+0000","last_undegraded":"2024-04-25T00:13:27.446369+0000","last_fullsized":"2024-04-25T00:13:27.446369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:52:39.500432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771920+0000","last_change":"2024-04-25T00:13:19.429800+0000","last_active":"2024-04-25T00:13:26.771920+0000","last_peered":"2024-04-25T00:13:26.771920+0000","last_clean":"2024-04-25T00:13:26.771920+0000","last_became_active":"2024-04-25T00:13:19.429388+0000","last_became_peered":"2024-04-25T00:13:19.429388+0000","last_unstale":"2024-04-25T00:13:26.771920+0000","last_undegraded":"2024-04-25T00:13:26.771920+0000","last_fullsized":"2024-04-25T00:13:26.771920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:45.344870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.859856+0000","last_change":"2024-04-25T00:13:19.416519+0000","last_active":"2024-04-25T00:13:25.859856+0000","last_peered":"2024-04-25T00:13:25.859856+0000","last_clean":"2024-04-25T00:13:25.859856+0000","last_became_active":"2024-04-25T00:13:19.416080+0000","last_became_peered":"2024-04-25T00:13:19.416080+0000","last_unstale":"2024-04-25T00:13:25.859856+0000","last_undegraded":"2024-04-25T00:13:25.859856+0000","last_fullsized":"2024-04-25T00:13:25.859856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:04.450160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.574047+0000","last_change":"2024-04-25T00:13:19.424378+0000","last_active":"2024-04-25T00:13:24.574047+0000","last_peered":"2024-04-25T00:13:24.574047+0000","last_clean":"2024-04-25T00:13:24.574047+0000","last_became_active":"2024-04-25T00:13:19.423918+0000","last_became_peered":"2024-04-25T00:13:19.423918+0000","last_unstale":"2024-04-25T00:13:24.574047+0000","last_undegraded":"2024-04-25T00:13:24.574047+0000","last_fullsized":"2024-04-25T00:13:24.574047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:01:11.397328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.589752+0000","last_change":"2024-04-25T00:13:19.424729+0000","last_active":"2024-04-25T00:13:25.589752+0000","last_peered":"2024-04-25T00:13:25.589752+0000","last_clean":"2024-04-25T00:13:25.589752+0000","last_became_active":"2024-04-25T00:13:19.422874+0000","last_became_peered":"2024-04-25T00:13:19.422874+0000","last_unstale":"2024-04-25T00:13:25.589752+0000","last_undegraded":"2024-04-25T00:13:25.589752+0000","last_fullsized":"2024-04-25T00:13:25.589752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:52:39.483589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.208841+0000","last_change":"2024-04-25T00:13:19.418043+0000","last_active":"2024-04-25T00:13:27.208841+0000","last_peered":"2024-04-25T00:13:27.208841+0000","last_clean":"2024-04-25T00:13:27.208841+0000","last_became_active":"2024-04-25T00:13:19.417443+0000","last_became_peered":"2024-04-25T00:13:19.417443+0000","last_unstale":"2024-04-25T00:13:27.208841+0000","last_undegraded":"2024-04-25T00:13:27.208841+0000","last_fullsized":"2024-04-25T00:13:27.208841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:47:53.141744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.397956+0000","last_change":"2024-04-25T00:13:19.420989+0000","last_active":"2024-04-25T00:13:27.397956+0000","last_peered":"2024-04-25T00:13:27.397956+0000","last_clean":"2024-04-25T00:13:27.397956+0000","last_became_active":"2024-04-25T00:13:19.420669+0000","last_became_peered":"2024-04-25T00:13:19.420669+0000","last_unstale":"2024-04-25T00:13:27.397956+0000","last_undegraded":"2024-04-25T00:13:27.397956+0000","last_fullsized":"2024-04-25T00:13:27.397956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:18.236132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.574336+0000","last_change":"2024-04-25T00:13:19.424176+0000","last_active":"2024-04-25T00:13:24.574336+0000","last_peered":"2024-04-25T00:13:24.574336+0000","last_clean":"2024-04-25T00:13:24.574336+0000","last_became_active":"2024-04-25T00:13:19.423656+0000","last_became_peered":"2024-04-25T00:13:19.423656+0000","last_unstale":"2024-04-25T00:13:24.574336+0000","last_undegraded":"2024-04-25T00:13:24.574336+0000","last_fullsized":"2024-04-25T00:13:24.574336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:24:41.697077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.590388+0000","last_change":"2024-04-25T00:13:19.419624+0000","last_active":"2024-04-25T00:13:25.590388+0000","last_peered":"2024-04-25T00:13:25.590388+0000","last_clean":"2024-04-25T00:13:25.590388+0000","last_became_active":"2024-04-25T00:13:19.419263+0000","last_became_peered":"2024-04-25T00:13:19.419263+0000","last_unstale":"2024-04-25T00:13:25.590388+0000","last_undegraded":"2024-04-25T00:13:25.590388+0000","last_fullsized":"2024-04-25T00:13:25.590388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:29.130733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771798+0000","last_change":"2024-04-25T00:13:19.427865+0000","last_active":"2024-04-25T00:13:26.771798+0000","last_peered":"2024-04-25T00:13:26.771798+0000","last_clean":"2024-04-25T00:13:26.771798+0000","last_became_active":"2024-04-25T00:13:19.427538+0000","last_became_peered":"2024-04-25T00:13:19.427538+0000","last_unstale":"2024-04-25T00:13:26.771798+0000","last_undegraded":"2024-04-25T00:13:26.771798+0000","last_fullsized":"2024-04-25T00:13:26.771798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:05.136948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.666147+0000","last_change":"2024-04-25T00:13:19.419261+0000","last_active":"2024-04-25T00:13:27.666147+0000","last_peered":"2024-04-25T00:13:27.666147+0000","last_clean":"2024-04-25T00:13:27.666147+0000","last_became_active":"2024-04-25T00:13:19.418938+0000","last_became_peered":"2024-04-25T00:13:19.418938+0000","last_unstale":"2024-04-25T00:13:27.666147+0000","last_undegraded":"2024-04-25T00:13:27.666147+0000","last_fullsized":"2024-04-25T00:13:27.666147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:44.971640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446336+0000","last_change":"2024-04-25T00:13:19.417654+0000","last_active":"2024-04-25T00:13:27.446336+0000","last_peered":"2024-04-25T00:13:27.446336+0000","last_clean":"2024-04-25T00:13:27.446336+0000","last_became_active":"2024-04-25T00:13:19.417319+0000","last_became_peered":"2024-04-25T00:13:19.417319+0000","last_unstale":"2024-04-25T00:13:27.446336+0000","last_undegraded":"2024-04-25T00:13:27.446336+0000","last_fullsized":"2024-04-25T00:13:27.446336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:43:54.447937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446578+0000","last_change":"2024-04-25T00:13:19.420761+0000","last_active":"2024-04-25T00:13:27.446578+0000","last_peered":"2024-04-25T00:13:27.446578+0000","last_clean":"2024-04-25T00:13:27.446578+0000","last_became_active":"2024-04-25T00:13:19.420334+0000","last_became_peered":"2024-04-25T00:13:19.420334+0000","last_unstale":"2024-04-25T00:13:27.446578+0000","last_undegraded":"2024-04-25T00:13:27.446578+0000","last_fullsized":"2024-04-25T00:13:27.446578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:20.136470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.397368+0000","last_change":"2024-04-25T00:13:19.420132+0000","last_active":"2024-04-25T00:13:27.397368+0000","last_peered":"2024-04-25T00:13:27.397368+0000","last_clean":"2024-04-25T00:13:27.397368+0000","last_became_active":"2024-04-25T00:13:19.418910+0000","last_became_peered":"2024-04-25T00:13:19.418910+0000","last_unstale":"2024-04-25T00:13:27.397368+0000","last_undegraded":"2024-04-25T00:13:27.397368+0000","last_fullsized":"2024-04-25T00:13:27.397368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:17.895412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447561+0000","last_change":"2024-04-25T00:13:19.430242+0000","last_active":"2024-04-25T00:13:27.447561+0000","last_peered":"2024-04-25T00:13:27.447561+0000","last_clean":"2024-04-25T00:13:27.447561+0000","last_became_active":"2024-04-25T00:13:19.429077+0000","last_became_peered":"2024-04-25T00:13:19.429077+0000","last_unstale":"2024-04-25T00:13:27.447561+0000","last_undegraded":"2024-04-25T00:13:27.447561+0000","last_fullsized":"2024-04-25T00:13:27.447561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:34:14.351157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.577522+0000","last_change":"2024-04-25T00:13:19.414375+0000","last_active":"2024-04-25T00:13:24.577522+0000","last_peered":"2024-04-25T00:13:24.577522+0000","last_clean":"2024-04-25T00:13:24.577522+0000","last_became_active":"2024-04-25T00:13:19.414003+0000","last_became_peered":"2024-04-25T00:13:19.414003+0000","last_unstale":"2024-04-25T00:13:24.577522+0000","last_undegraded":"2024-04-25T00:13:24.577522+0000","last_fullsized":"2024-04-25T00:13:24.577522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:43:50.612743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573887+0000","last_change":"2024-04-25T00:13:19.415322+0000","last_active":"2024-04-25T00:13:24.573887+0000","last_peered":"2024-04-25T00:13:24.573887+0000","last_clean":"2024-04-25T00:13:24.573887+0000","last_became_active":"2024-04-25T00:13:19.414925+0000","last_became_peered":"2024-04-25T00:13:19.414925+0000","last_unstale":"2024-04-25T00:13:24.573887+0000","last_undegraded":"2024-04-25T00:13:24.573887+0000","last_fullsized":"2024-04-25T00:13:24.573887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:58.140640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573296+0000","last_change":"2024-04-25T00:13:23.584763+0000","last_active":"2024-04-25T00:13:24.573296+0000","last_peered":"2024-04-25T00:13:24.573296+0000","last_clean":"2024-04-25T00:13:24.573296+0000","last_became_active":"2024-04-25T00:13:23.584303+0000","last_became_peered":"2024-04-25T00:13:23.584303+0000","last_unstale":"2024-04-25T00:13:24.573296+0000","last_undegraded":"2024-04-25T00:13:24.573296+0000","last_fullsized":"2024-04-25T00:13:24.573296+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:54:11.447867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.577673+0000","last_change":"2024-04-25T00:13:18.519576+0000","last_active":"2024-04-25T00:13:24.577673+0000","last_peered":"2024-04-25T00:13:24.577673+0000","last_clean":"2024-04-25T00:13:24.577673+0000","last_became_active":"2024-04-25T00:13:18.519201+0000","last_became_peered":"2024-04-25T00:13:18.519201+0000","last_unstale":"2024-04-25T00:13:24.577673+0000","last_undegraded":"2024-04-25T00:13:24.577673+0000","last_fullsized":"2024-04-25T00:13:24.577673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:19.681096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.209228+0000","last_change":"2024-04-25T00:13:19.419098+0000","last_active":"2024-04-25T00:13:27.209228+0000","last_peered":"2024-04-25T00:13:27.209228+0000","last_clean":"2024-04-25T00:13:27.209228+0000","last_became_active":"2024-04-25T00:13:19.418649+0000","last_became_peered":"2024-04-25T00:13:19.418649+0000","last_unstale":"2024-04-25T00:13:27.209228+0000","last_undegraded":"2024-04-25T00:13:27.209228+0000","last_fullsized":"2024-04-25T00:13:27.209228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:04.561470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771249+0000","last_change":"2024-04-25T00:13:23.580354+0000","last_active":"2024-04-25T00:13:26.771249+0000","last_peered":"2024-04-25T00:13:26.771249+0000","last_clean":"2024-04-25T00:13:26.771249+0000","last_became_active":"2024-04-25T00:13:23.579944+0000","last_became_peered":"2024-04-25T00:13:23.579944+0000","last_unstale":"2024-04-25T00:13:26.771249+0000","last_undegraded":"2024-04-25T00:13:26.771249+0000","last_fullsized":"2024-04-25T00:13:26.771249+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:48:57.788974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.209268+0000","last_change":"2024-04-25T00:13:18.527471+0000","last_active":"2024-04-25T00:13:27.209268+0000","last_peered":"2024-04-25T00:13:27.209268+0000","last_clean":"2024-04-25T00:13:27.209268+0000","last_became_active":"2024-04-25T00:13:18.527138+0000","last_became_peered":"2024-04-25T00:13:18.527138+0000","last_unstale":"2024-04-25T00:13:27.209268+0000","last_undegraded":"2024-04-25T00:13:27.209268+0000","last_fullsized":"2024-04-25T00:13:27.209268+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:38.003719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446684+0000","last_change":"2024-04-25T00:13:19.420563+0000","last_active":"2024-04-25T00:13:27.446684+0000","last_peered":"2024-04-25T00:13:27.446684+0000","last_clean":"2024-04-25T00:13:27.446684+0000","last_became_active":"2024-04-25T00:13:19.420142+0000","last_became_peered":"2024-04-25T00:13:19.420142+0000","last_unstale":"2024-04-25T00:13:27.446684+0000","last_undegraded":"2024-04-25T00:13:27.446684+0000","last_fullsized":"2024-04-25T00:13:27.446684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:23:08.423795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771374+0000","last_change":"2024-04-25T00:13:23.580657+0000","last_active":"2024-04-25T00:13:26.771374+0000","last_peered":"2024-04-25T00:13:26.771374+0000","last_clean":"2024-04-25T00:13:26.771374+0000","last_became_active":"2024-04-25T00:13:23.580253+0000","last_became_peered":"2024-04-25T00:13:23.580253+0000","last_unstale":"2024-04-25T00:13:26.771374+0000","last_undegraded":"2024-04-25T00:13:26.771374+0000","last_fullsized":"2024-04-25T00:13:26.771374+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:54:02.842805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:44.657231+0000","last_change":"2024-04-25T00:13:18.520509+0000","last_active":"2024-04-25T00:13:44.657231+0000","last_peered":"2024-04-25T00:13:44.657231+0000","last_clean":"2024-04-25T00:13:44.657231+0000","last_became_active":"2024-04-25T00:13:18.520139+0000","last_became_peered":"2024-04-25T00:13:18.520139+0000","last_unstale":"2024-04-25T00:13:44.657231+0000","last_undegraded":"2024-04-25T00:13:44.657231+0000","last_fullsized":"2024-04-25T00:13:44.657231+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:08.797670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.397687+0000","last_change":"2024-04-25T00:13:19.419115+0000","last_active":"2024-04-25T00:13:27.397687+0000","last_peered":"2024-04-25T00:13:27.397687+0000","last_clean":"2024-04-25T00:13:27.397687+0000","last_became_active":"2024-04-25T00:13:19.417865+0000","last_became_peered":"2024-04-25T00:13:19.417865+0000","last_unstale":"2024-04-25T00:13:27.397687+0000","last_undegraded":"2024-04-25T00:13:27.397687+0000","last_fullsized":"2024-04-25T00:13:27.397687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:20:11.173803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771365+0000","last_change":"2024-04-25T00:13:23.585052+0000","last_active":"2024-04-25T00:13:26.771365+0000","last_peered":"2024-04-25T00:13:26.771365+0000","last_clean":"2024-04-25T00:13:26.771365+0000","last_became_active":"2024-04-25T00:13:23.584696+0000","last_became_peered":"2024-04-25T00:13:23.584696+0000","last_unstale":"2024-04-25T00:13:26.771365+0000","last_undegraded":"2024-04-25T00:13:26.771365+0000","last_fullsized":"2024-04-25T00:13:26.771365+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:26.894183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.860271+0000","last_change":"2024-04-25T00:13:18.522793+0000","last_active":"2024-04-25T00:13:25.860271+0000","last_peered":"2024-04-25T00:13:25.860271+0000","last_clean":"2024-04-25T00:13:25.860271+0000","last_became_active":"2024-04-25T00:13:18.522453+0000","last_became_peered":"2024-04-25T00:13:18.522453+0000","last_unstale":"2024-04-25T00:13:25.860271+0000","last_undegraded":"2024-04-25T00:13:25.860271+0000","last_fullsized":"2024-04-25T00:13:25.860271+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:08.368214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.448211+0000","last_change":"2024-04-25T00:13:19.418573+0000","last_active":"2024-04-25T00:13:27.448211+0000","last_peered":"2024-04-25T00:13:27.448211+0000","last_clean":"2024-04-25T00:13:27.448211+0000","last_became_active":"2024-04-25T00:13:19.418219+0000","last_became_peered":"2024-04-25T00:13:19.418219+0000","last_unstale":"2024-04-25T00:13:27.448211+0000","last_undegraded":"2024-04-25T00:13:27.448211+0000","last_fullsized":"2024-04-25T00:13:27.448211+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:08:44.453017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573796+0000","last_change":"2024-04-25T00:13:23.578716+0000","last_active":"2024-04-25T00:13:24.573796+0000","last_peered":"2024-04-25T00:13:24.573796+0000","last_clean":"2024-04-25T00:13:24.573796+0000","last_became_active":"2024-04-25T00:13:23.578325+0000","last_became_peered":"2024-04-25T00:13:23.578325+0000","last_unstale":"2024-04-25T00:13:24.573796+0000","last_undegraded":"2024-04-25T00:13:24.573796+0000","last_fullsized":"2024-04-25T00:13:24.573796+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:39:57.145656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447998+0000","last_change":"2024-04-25T00:13:18.527369+0000","last_active":"2024-04-25T00:13:27.447998+0000","last_peered":"2024-04-25T00:13:27.447998+0000","last_clean":"2024-04-25T00:13:27.447998+0000","last_became_active":"2024-04-25T00:13:18.526032+0000","last_became_peered":"2024-04-25T00:13:18.526032+0000","last_unstale":"2024-04-25T00:13:27.447998+0000","last_undegraded":"2024-04-25T00:13:27.447998+0000","last_fullsized":"2024-04-25T00:13:27.447998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:52.635184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.859887+0000","last_change":"2024-04-25T00:13:19.413305+0000","last_active":"2024-04-25T00:13:25.859887+0000","last_peered":"2024-04-25T00:13:25.859887+0000","last_clean":"2024-04-25T00:13:25.859887+0000","last_became_active":"2024-04-25T00:13:19.412956+0000","last_became_peered":"2024-04-25T00:13:19.412956+0000","last_unstale":"2024-04-25T00:13:25.859887+0000","last_undegraded":"2024-04-25T00:13:25.859887+0000","last_fullsized":"2024-04-25T00:13:25.859887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:54.436470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573607+0000","last_change":"2024-04-25T00:13:23.585624+0000","last_active":"2024-04-25T00:13:24.573607+0000","last_peered":"2024-04-25T00:13:24.573607+0000","last_clean":"2024-04-25T00:13:24.573607+0000","last_became_active":"2024-04-25T00:13:23.585215+0000","last_became_peered":"2024-04-25T00:13:23.585215+0000","last_unstale":"2024-04-25T00:13:24.573607+0000","last_undegraded":"2024-04-25T00:13:24.573607+0000","last_fullsized":"2024-04-25T00:13:24.573607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:01.427226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573909+0000","last_change":"2024-04-25T00:13:18.517507+0000","last_active":"2024-04-25T00:13:24.573909+0000","last_peered":"2024-04-25T00:13:24.573909+0000","last_clean":"2024-04-25T00:13:24.573909+0000","last_became_active":"2024-04-25T00:13:18.517099+0000","last_became_peered":"2024-04-25T00:13:18.517099+0000","last_unstale":"2024-04-25T00:13:24.573909+0000","last_undegraded":"2024-04-25T00:13:24.573909+0000","last_fullsized":"2024-04-25T00:13:24.573909+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:42:17.357610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.448111+0000","last_change":"2024-04-25T00:13:19.422134+0000","last_active":"2024-04-25T00:13:27.448111+0000","last_peered":"2024-04-25T00:13:27.448111+0000","last_clean":"2024-04-25T00:13:27.448111+0000","last_became_active":"2024-04-25T00:13:19.421167+0000","last_became_peered":"2024-04-25T00:13:19.421167+0000","last_unstale":"2024-04-25T00:13:27.448111+0000","last_undegraded":"2024-04-25T00:13:27.448111+0000","last_fullsized":"2024-04-25T00:13:27.448111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:03:25.451560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447625+0000","last_change":"2024-04-25T00:13:23.582526+0000","last_active":"2024-04-25T00:13:27.447625+0000","last_peered":"2024-04-25T00:13:27.447625+0000","last_clean":"2024-04-25T00:13:27.447625+0000","last_became_active":"2024-04-25T00:13:23.581221+0000","last_became_peered":"2024-04-25T00:13:23.581221+0000","last_unstale":"2024-04-25T00:13:27.447625+0000","last_undegraded":"2024-04-25T00:13:27.447625+0000","last_fullsized":"2024-04-25T00:13:27.447625+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:46.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:44.658568+0000","last_change":"2024-04-25T00:13:18.522245+0000","last_active":"2024-04-25T00:13:44.658568+0000","last_peered":"2024-04-25T00:13:44.658568+0000","last_clean":"2024-04-25T00:13:44.658568+0000","last_became_active":"2024-04-25T00:13:18.521440+0000","last_became_peered":"2024-04-25T00:13:18.521440+0000","last_unstale":"2024-04-25T00:13:44.658568+0000","last_undegraded":"2024-04-25T00:13:44.658568+0000","last_fullsized":"2024-04-25T00:13:44.658568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:21:37.847964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.208709+0000","last_change":"2024-04-25T00:13:19.422771+0000","last_active":"2024-04-25T00:13:27.208709+0000","last_peered":"2024-04-25T00:13:27.208709+0000","last_clean":"2024-04-25T00:13:27.208709+0000","last_became_active":"2024-04-25T00:13:19.422387+0000","last_became_peered":"2024-04-25T00:13:19.422387+0000","last_unstale":"2024-04-25T00:13:27.208709+0000","last_undegraded":"2024-04-25T00:13:27.208709+0000","last_fullsized":"2024-04-25T00:13:27.208709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:15.647186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446740+0000","last_change":"2024-04-25T00:13:23.586235+0000","last_active":"2024-04-25T00:13:27.446740+0000","last_peered":"2024-04-25T00:13:27.446740+0000","last_clean":"2024-04-25T00:13:27.446740+0000","last_became_active":"2024-04-25T00:13:23.585803+0000","last_became_peered":"2024-04-25T00:13:23.585803+0000","last_unstale":"2024-04-25T00:13:27.446740+0000","last_undegraded":"2024-04-25T00:13:27.446740+0000","last_fullsized":"2024-04-25T00:13:27.446740+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:46.235435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447814+0000","last_change":"2024-04-25T00:13:18.525283+0000","last_active":"2024-04-25T00:13:27.447814+0000","last_peered":"2024-04-25T00:13:27.447814+0000","last_clean":"2024-04-25T00:13:27.447814+0000","last_became_active":"2024-04-25T00:13:18.524939+0000","last_became_peered":"2024-04-25T00:13:18.524939+0000","last_unstale":"2024-04-25T00:13:27.447814+0000","last_undegraded":"2024-04-25T00:13:27.447814+0000","last_fullsized":"2024-04-25T00:13:27.447814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:09.258498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.576780+0000","last_change":"2024-04-25T00:13:19.415746+0000","last_active":"2024-04-25T00:13:24.576780+0000","last_peered":"2024-04-25T00:13:24.576780+0000","last_clean":"2024-04-25T00:13:24.576780+0000","last_became_active":"2024-04-25T00:13:19.415325+0000","last_became_peered":"2024-04-25T00:13:19.415325+0000","last_unstale":"2024-04-25T00:13:24.576780+0000","last_undegraded":"2024-04-25T00:13:24.576780+0000","last_fullsized":"2024-04-25T00:13:24.576780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:17:48.268845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.208805+0000","last_change":"2024-04-25T00:13:23.583153+0000","last_active":"2024-04-25T00:13:27.208805+0000","last_peered":"2024-04-25T00:13:27.208805+0000","last_clean":"2024-04-25T00:13:27.208805+0000","last_became_active":"2024-04-25T00:13:23.580257+0000","last_became_peered":"2024-04-25T00:13:23.580257+0000","last_unstale":"2024-04-25T00:13:27.208805+0000","last_undegraded":"2024-04-25T00:13:27.208805+0000","last_fullsized":"2024-04-25T00:13:27.208805+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:09:32.990253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.574122+0000","last_change":"2024-04-25T00:13:18.520365+0000","last_active":"2024-04-25T00:13:24.574122+0000","last_peered":"2024-04-25T00:13:24.574122+0000","last_clean":"2024-04-25T00:13:24.574122+0000","last_became_active":"2024-04-25T00:13:18.519918+0000","last_became_peered":"2024-04-25T00:13:18.519918+0000","last_unstale":"2024-04-25T00:13:24.574122+0000","last_undegraded":"2024-04-25T00:13:24.574122+0000","last_fullsized":"2024-04-25T00:13:24.574122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:46.415197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.397970+0000","last_change":"2024-04-25T00:13:19.414341+0000","last_active":"2024-04-25T00:13:27.397970+0000","last_peered":"2024-04-25T00:13:27.397970+0000","last_clean":"2024-04-25T00:13:27.397970+0000","last_became_active":"2024-04-25T00:13:19.413793+0000","last_became_peered":"2024-04-25T00:13:19.413793+0000","last_unstale":"2024-04-25T00:13:27.397970+0000","last_undegraded":"2024-04-25T00:13:27.397970+0000","last_fullsized":"2024-04-25T00:13:27.397970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:55.927082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.576729+0000","last_change":"2024-04-25T00:13:23.590631+0000","last_active":"2024-04-25T00:13:24.576729+0000","last_peered":"2024-04-25T00:13:24.576729+0000","last_clean":"2024-04-25T00:13:24.576729+0000","last_became_active":"2024-04-25T00:13:23.589980+0000","last_became_peered":"2024-04-25T00:13:23.589980+0000","last_unstale":"2024-04-25T00:13:24.576729+0000","last_undegraded":"2024-04-25T00:13:24.576729+0000","last_fullsized":"2024-04-25T00:13:24.576729+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:18.234048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:43.652199+0000","last_change":"2024-04-25T00:13:18.526353+0000","last_active":"2024-04-25T00:13:43.652199+0000","last_peered":"2024-04-25T00:13:43.652199+0000","last_clean":"2024-04-25T00:13:43.652199+0000","last_became_active":"2024-04-25T00:13:18.525951+0000","last_became_peered":"2024-04-25T00:13:18.525951+0000","last_unstale":"2024-04-25T00:13:43.652199+0000","last_undegraded":"2024-04-25T00:13:43.652199+0000","last_fullsized":"2024-04-25T00:13:43.652199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:10:29.597797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446700+0000","last_change":"2024-04-25T00:13:19.425486+0000","last_active":"2024-04-25T00:13:27.446700+0000","last_peered":"2024-04-25T00:13:27.446700+0000","last_clean":"2024-04-25T00:13:27.446700+0000","last_became_active":"2024-04-25T00:13:19.425087+0000","last_became_peered":"2024-04-25T00:13:19.425087+0000","last_unstale":"2024-04-25T00:13:27.446700+0000","last_undegraded":"2024-04-25T00:13:27.446700+0000","last_fullsized":"2024-04-25T00:13:27.446700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:00.216683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771549+0000","last_change":"2024-04-25T00:13:23.579747+0000","last_active":"2024-04-25T00:13:26.771549+0000","last_peered":"2024-04-25T00:13:26.771549+0000","last_clean":"2024-04-25T00:13:26.771549+0000","last_became_active":"2024-04-25T00:13:23.579346+0000","last_became_peered":"2024-04-25T00:13:23.579346+0000","last_unstale":"2024-04-25T00:13:26.771549+0000","last_undegraded":"2024-04-25T00:13:26.771549+0000","last_fullsized":"2024-04-25T00:13:26.771549+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:10:18.969942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.448126+0000","last_change":"2024-04-25T00:13:18.524707+0000","last_active":"2024-04-25T00:13:27.448126+0000","last_peered":"2024-04-25T00:13:27.448126+0000","last_clean":"2024-04-25T00:13:27.448126+0000","last_became_active":"2024-04-25T00:13:18.524352+0000","last_became_peered":"2024-04-25T00:13:18.524352+0000","last_unstale":"2024-04-25T00:13:27.448126+0000","last_undegraded":"2024-04-25T00:13:27.448126+0000","last_fullsized":"2024-04-25T00:13:27.448126+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:29.992843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.397816+0000","last_change":"2024-04-25T00:13:19.421845+0000","last_active":"2024-04-25T00:13:27.397816+0000","last_peered":"2024-04-25T00:13:27.397816+0000","last_clean":"2024-04-25T00:13:27.397816+0000","last_became_active":"2024-04-25T00:13:19.421458+0000","last_became_peered":"2024-04-25T00:13:19.421458+0000","last_unstale":"2024-04-25T00:13:27.397816+0000","last_undegraded":"2024-04-25T00:13:27.397816+0000","last_fullsized":"2024-04-25T00:13:27.397816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:09.286604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.576864+0000","last_change":"2024-04-25T00:13:23.583721+0000","last_active":"2024-04-25T00:13:24.576864+0000","last_peered":"2024-04-25T00:13:24.576864+0000","last_clean":"2024-04-25T00:13:24.576864+0000","last_became_active":"2024-04-25T00:13:23.583340+0000","last_became_peered":"2024-04-25T00:13:23.583340+0000","last_unstale":"2024-04-25T00:13:24.576864+0000","last_undegraded":"2024-04-25T00:13:24.576864+0000","last_fullsized":"2024-04-25T00:13:24.576864+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:26.446468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'10","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:35.050566+0000","last_change":"2024-04-25T00:13:18.520653+0000","last_active":"2024-04-25T00:13:35.050566+0000","last_peered":"2024-04-25T00:13:35.050566+0000","last_clean":"2024-04-25T00:13:35.050566+0000","last_became_active":"2024-04-25T00:13:18.520278+0000","last_became_peered":"2024-04-25T00:13:18.520278+0000","last_unstale":"2024-04-25T00:13:35.050566+0000","last_undegraded":"2024-04-25T00:13:35.050566+0000","last_fullsized":"2024-04-25T00:13:35.050566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:58.311179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.860126+0000","last_change":"2024-04-25T00:13:19.421599+0000","last_active":"2024-04-25T00:13:25.860126+0000","last_peered":"2024-04-25T00:13:25.860126+0000","last_clean":"2024-04-25T00:13:25.860126+0000","last_became_active":"2024-04-25T00:13:19.421107+0000","last_became_peered":"2024-04-25T00:13:19.421107+0000","last_unstale":"2024-04-25T00:13:25.860126+0000","last_undegraded":"2024-04-25T00:13:25.860126+0000","last_fullsized":"2024-04-25T00:13:25.860126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:59.577987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.448527+0000","last_change":"2024-04-25T00:13:23.580538+0000","last_active":"2024-04-25T00:13:27.448527+0000","last_peered":"2024-04-25T00:13:27.448527+0000","last_clean":"2024-04-25T00:13:27.448527+0000","last_became_active":"2024-04-25T00:13:23.580095+0000","last_became_peered":"2024-04-25T00:13:23.580095+0000","last_unstale":"2024-04-25T00:13:27.448527+0000","last_undegraded":"2024-04-25T00:13:27.448527+0000","last_fullsized":"2024-04-25T00:13:27.448527+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:04.189530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573667+0000","last_change":"2024-04-25T00:13:18.522501+0000","last_active":"2024-04-25T00:13:24.573667+0000","last_peered":"2024-04-25T00:13:24.573667+0000","last_clean":"2024-04-25T00:13:24.573667+0000","last_became_active":"2024-04-25T00:13:18.522160+0000","last_became_peered":"2024-04-25T00:13:18.522160+0000","last_unstale":"2024-04-25T00:13:24.573667+0000","last_undegraded":"2024-04-25T00:13:24.573667+0000","last_fullsized":"2024-04-25T00:13:24.573667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:31.876731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.665997+0000","last_change":"2024-04-25T00:13:19.417957+0000","last_active":"2024-04-25T00:13:27.665997+0000","last_peered":"2024-04-25T00:13:27.665997+0000","last_clean":"2024-04-25T00:13:27.665997+0000","last_became_active":"2024-04-25T00:13:19.417600+0000","last_became_peered":"2024-04-25T00:13:19.417600+0000","last_unstale":"2024-04-25T00:13:27.665997+0000","last_undegraded":"2024-04-25T00:13:27.665997+0000","last_fullsized":"2024-04-25T00:13:27.665997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:24.600965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447214+0000","last_change":"2024-04-25T00:13:23.582279+0000","last_active":"2024-04-25T00:13:27.447214+0000","last_peered":"2024-04-25T00:13:27.447214+0000","last_clean":"2024-04-25T00:13:27.447214+0000","last_became_active":"2024-04-25T00:13:23.581766+0000","last_became_peered":"2024-04-25T00:13:23.581766+0000","last_unstale":"2024-04-25T00:13:27.447214+0000","last_undegraded":"2024-04-25T00:13:27.447214+0000","last_fullsized":"2024-04-25T00:13:27.447214+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:36.115870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447847+0000","last_change":"2024-04-25T00:13:18.527443+0000","last_active":"2024-04-25T00:13:27.447847+0000","last_peered":"2024-04-25T00:13:27.447847+0000","last_clean":"2024-04-25T00:13:27.447847+0000","last_became_active":"2024-04-25T00:13:18.526919+0000","last_became_peered":"2024-04-25T00:13:18.526919+0000","last_unstale":"2024-04-25T00:13:27.447847+0000","last_undegraded":"2024-04-25T00:13:27.447847+0000","last_fullsized":"2024-04-25T00:13:27.447847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:32.737580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.590010+0000","last_change":"2024-04-25T00:13:19.425947+0000","last_active":"2024-04-25T00:13:25.590010+0000","last_peered":"2024-04-25T00:13:25.590010+0000","last_clean":"2024-04-25T00:13:25.590010+0000","last_became_active":"2024-04-25T00:13:19.425613+0000","last_became_peered":"2024-04-25T00:13:19.425613+0000","last_unstale":"2024-04-25T00:13:25.590010+0000","last_undegraded":"2024-04-25T00:13:25.590010+0000","last_fullsized":"2024-04-25T00:13:25.590010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:28:34.729375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.577535+0000","last_change":"2024-04-25T00:13:23.587794+0000","last_active":"2024-04-25T00:13:24.577535+0000","last_peered":"2024-04-25T00:13:24.577535+0000","last_clean":"2024-04-25T00:13:24.577535+0000","last_became_active":"2024-04-25T00:13:23.587334+0000","last_became_peered":"2024-04-25T00:13:23.587334+0000","last_unstale":"2024-04-25T00:13:24.577535+0000","last_undegraded":"2024-04-25T00:13:24.577535+0000","last_fullsized":"2024-04-25T00:13:24.577535+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:58.460439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771365+0000","last_change":"2024-04-25T00:13:18.521701+0000","last_active":"2024-04-25T00:13:26.771365+0000","last_peered":"2024-04-25T00:13:26.771365+0000","last_clean":"2024-04-25T00:13:26.771365+0000","last_became_active":"2024-04-25T00:13:18.521360+0000","last_became_peered":"2024-04-25T00:13:18.521360+0000","last_unstale":"2024-04-25T00:13:26.771365+0000","last_undegraded":"2024-04-25T00:13:26.771365+0000","last_fullsized":"2024-04-25T00:13:26.771365+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:18:11.567426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.589791+0000","last_change":"2024-04-25T00:13:19.424162+0000","last_active":"2024-04-25T00:13:25.589791+0000","last_peered":"2024-04-25T00:13:25.589791+0000","last_clean":"2024-04-25T00:13:25.589791+0000","last_became_active":"2024-04-25T00:13:19.423845+0000","last_became_peered":"2024-04-25T00:13:19.423845+0000","last_unstale":"2024-04-25T00:13:25.589791+0000","last_undegraded":"2024-04-25T00:13:25.589791+0000","last_fullsized":"2024-04-25T00:13:25.589791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:33.887577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.666404+0000","last_change":"2024-04-25T00:13:23.582102+0000","last_active":"2024-04-25T00:13:27.666404+0000","last_peered":"2024-04-25T00:13:27.666404+0000","last_clean":"2024-04-25T00:13:27.666404+0000","last_became_active":"2024-04-25T00:13:23.581679+0000","last_became_peered":"2024-04-25T00:13:23.581679+0000","last_unstale":"2024-04-25T00:13:27.666404+0000","last_undegraded":"2024-04-25T00:13:27.666404+0000","last_fullsized":"2024-04-25T00:13:27.666404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:05:39.518634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:44.660651+0000","last_change":"2024-04-25T00:13:18.518285+0000","last_active":"2024-04-25T00:13:44.660651+0000","last_peered":"2024-04-25T00:13:44.660651+0000","last_clean":"2024-04-25T00:13:44.660651+0000","last_became_active":"2024-04-25T00:13:18.517826+0000","last_became_peered":"2024-04-25T00:13:18.517826+0000","last_unstale":"2024-04-25T00:13:44.660651+0000","last_undegraded":"2024-04-25T00:13:44.660651+0000","last_fullsized":"2024-04-25T00:13:44.660651+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:12.901225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447582+0000","last_change":"2024-04-25T00:13:19.419691+0000","last_active":"2024-04-25T00:13:27.447582+0000","last_peered":"2024-04-25T00:13:27.447582+0000","last_clean":"2024-04-25T00:13:27.447582+0000","last_became_active":"2024-04-25T00:13:19.419278+0000","last_became_peered":"2024-04-25T00:13:19.419278+0000","last_unstale":"2024-04-25T00:13:27.447582+0000","last_undegraded":"2024-04-25T00:13:27.447582+0000","last_fullsized":"2024-04-25T00:13:27.447582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:19.204170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447604+0000","last_change":"2024-04-25T00:13:23.578445+0000","last_active":"2024-04-25T00:13:27.447604+0000","last_peered":"2024-04-25T00:13:27.447604+0000","last_clean":"2024-04-25T00:13:27.447604+0000","last_became_active":"2024-04-25T00:13:23.578048+0000","last_became_peered":"2024-04-25T00:13:23.578048+0000","last_unstale":"2024-04-25T00:13:27.447604+0000","last_undegraded":"2024-04-25T00:13:27.447604+0000","last_fullsized":"2024-04-25T00:13:27.447604+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:55.072460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.860137+0000","last_change":"2024-04-25T00:13:18.515939+0000","last_active":"2024-04-25T00:13:25.860137+0000","last_peered":"2024-04-25T00:13:25.860137+0000","last_clean":"2024-04-25T00:13:25.860137+0000","last_became_active":"2024-04-25T00:13:18.515597+0000","last_became_peered":"2024-04-25T00:13:18.515597+0000","last_unstale":"2024-04-25T00:13:25.860137+0000","last_undegraded":"2024-04-25T00:13:25.860137+0000","last_fullsized":"2024-04-25T00:13:25.860137+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:03:30.266404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447429+0000","last_change":"2024-04-25T00:13:19.428048+0000","last_active":"2024-04-25T00:13:27.447429+0000","last_peered":"2024-04-25T00:13:27.447429+0000","last_clean":"2024-04-25T00:13:27.447429+0000","last_became_active":"2024-04-25T00:13:19.427622+0000","last_became_peered":"2024-04-25T00:13:19.427622+0000","last_unstale":"2024-04-25T00:13:27.447429+0000","last_undegraded":"2024-04-25T00:13:27.447429+0000","last_fullsized":"2024-04-25T00:13:27.447429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:29:25.039299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.859946+0000","last_change":"2024-04-25T00:13:23.862576+0000","last_active":"2024-04-25T00:13:25.859946+0000","last_peered":"2024-04-25T00:13:25.859946+0000","last_clean":"2024-04-25T00:13:25.859946+0000","last_became_active":"2024-04-25T00:13:23.862024+0000","last_became_peered":"2024-04-25T00:13:23.862024+0000","last_unstale":"2024-04-25T00:13:25.859946+0000","last_undegraded":"2024-04-25T00:13:25.859946+0000","last_fullsized":"2024-04-25T00:13:25.859946+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:17.249680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.577148+0000","last_change":"2024-04-25T00:13:18.522503+0000","last_active":"2024-04-25T00:13:24.577148+0000","last_peered":"2024-04-25T00:13:24.577148+0000","last_clean":"2024-04-25T00:13:24.577148+0000","last_became_active":"2024-04-25T00:13:18.522126+0000","last_became_peered":"2024-04-25T00:13:18.522126+0000","last_unstale":"2024-04-25T00:13:24.577148+0000","last_undegraded":"2024-04-25T00:13:24.577148+0000","last_fullsized":"2024-04-25T00:13:24.577148+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:47:57.530098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.574107+0000","last_change":"2024-04-25T00:13:19.416862+0000","last_active":"2024-04-25T00:13:24.574107+0000","last_peered":"2024-04-25T00:13:24.574107+0000","last_clean":"2024-04-25T00:13:24.574107+0000","last_became_active":"2024-04-25T00:13:19.416375+0000","last_became_peered":"2024-04-25T00:13:19.416375+0000","last_unstale":"2024-04-25T00:13:24.574107+0000","last_undegraded":"2024-04-25T00:13:24.574107+0000","last_fullsized":"2024-04-25T00:13:24.574107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:43:32.031623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.666186+0000","last_change":"2024-04-25T00:13:23.582387+0000","last_active":"2024-04-25T00:13:27.666186+0000","last_peered":"2024-04-25T00:13:27.666186+0000","last_clean":"2024-04-25T00:13:27.666186+0000","last_became_active":"2024-04-25T00:13:23.581984+0000","last_became_peered":"2024-04-25T00:13:23.581984+0000","last_unstale":"2024-04-25T00:13:27.666186+0000","last_undegraded":"2024-04-25T00:13:27.666186+0000","last_fullsized":"2024-04-25T00:13:27.666186+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:07:16.388293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.590004+0000","last_change":"2024-04-25T00:13:18.524121+0000","last_active":"2024-04-25T00:13:25.590004+0000","last_peered":"2024-04-25T00:13:25.590004+0000","last_clean":"2024-04-25T00:13:25.590004+0000","last_became_active":"2024-04-25T00:13:18.523731+0000","last_became_peered":"2024-04-25T00:13:18.523731+0000","last_unstale":"2024-04-25T00:13:25.590004+0000","last_undegraded":"2024-04-25T00:13:25.590004+0000","last_fullsized":"2024-04-25T00:13:25.590004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:22.105952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573942+0000","last_change":"2024-04-25T00:13:19.424753+0000","last_active":"2024-04-25T00:13:24.573942+0000","last_peered":"2024-04-25T00:13:24.573942+0000","last_clean":"2024-04-25T00:13:24.573942+0000","last_became_active":"2024-04-25T00:13:19.423214+0000","last_became_peered":"2024-04-25T00:13:19.423214+0000","last_unstale":"2024-04-25T00:13:24.573942+0000","last_undegraded":"2024-04-25T00:13:24.573942+0000","last_fullsized":"2024-04-25T00:13:24.573942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:41.627442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.666263+0000","last_change":"2024-04-25T00:13:23.862424+0000","last_active":"2024-04-25T00:13:27.666263+0000","last_peered":"2024-04-25T00:13:27.666263+0000","last_clean":"2024-04-25T00:13:27.666263+0000","last_became_active":"2024-04-25T00:13:23.862006+0000","last_became_peered":"2024-04-25T00:13:23.862006+0000","last_unstale":"2024-04-25T00:13:27.666263+0000","last_undegraded":"2024-04-25T00:13:27.666263+0000","last_fullsized":"2024-04-25T00:13:27.666263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:20:59.959545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447446+0000","last_change":"2024-04-25T00:13:19.420617+0000","last_active":"2024-04-25T00:13:27.447446+0000","last_peered":"2024-04-25T00:13:27.447446+0000","last_clean":"2024-04-25T00:13:27.447446+0000","last_became_active":"2024-04-25T00:13:19.420274+0000","last_became_peered":"2024-04-25T00:13:19.420274+0000","last_unstale":"2024-04-25T00:13:27.447446+0000","last_undegraded":"2024-04-25T00:13:27.447446+0000","last_fullsized":"2024-04-25T00:13:27.447446+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:07.515196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:44.660508+0000","last_change":"2024-04-25T00:13:18.529753+0000","last_active":"2024-04-25T00:13:44.660508+0000","last_peered":"2024-04-25T00:13:44.660508+0000","last_clean":"2024-04-25T00:13:44.660508+0000","last_became_active":"2024-04-25T00:13:18.525521+0000","last_became_peered":"2024-04-25T00:13:18.525521+0000","last_unstale":"2024-04-25T00:13:44.660508+0000","last_undegraded":"2024-04-25T00:13:44.660508+0000","last_fullsized":"2024-04-25T00:13:44.660508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:17.044367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.590252+0000","last_change":"2024-04-25T00:13:23.589613+0000","last_active":"2024-04-25T00:13:25.590252+0000","last_peered":"2024-04-25T00:13:25.590252+0000","last_clean":"2024-04-25T00:13:25.590252+0000","last_became_active":"2024-04-25T00:13:23.589323+0000","last_became_peered":"2024-04-25T00:13:23.589323+0000","last_unstale":"2024-04-25T00:13:25.590252+0000","last_undegraded":"2024-04-25T00:13:25.590252+0000","last_fullsized":"2024-04-25T00:13:25.590252+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:34:42.272706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.208757+0000","last_change":"2024-04-25T00:13:19.417775+0000","last_active":"2024-04-25T00:13:27.208757+0000","last_peered":"2024-04-25T00:13:27.208757+0000","last_clean":"2024-04-25T00:13:27.208757+0000","last_became_active":"2024-04-25T00:13:19.415096+0000","last_became_peered":"2024-04-25T00:13:19.415096+0000","last_unstale":"2024-04-25T00:13:27.208757+0000","last_undegraded":"2024-04-25T00:13:27.208757+0000","last_fullsized":"2024-04-25T00:13:27.208757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:54.975923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:43.651456+0000","last_change":"2024-04-25T00:13:18.514560+0000","last_active":"2024-04-25T00:13:43.651456+0000","last_peered":"2024-04-25T00:13:43.651456+0000","last_clean":"2024-04-25T00:13:43.651456+0000","last_became_active":"2024-04-25T00:13:18.514034+0000","last_became_peered":"2024-04-25T00:13:18.514034+0000","last_unstale":"2024-04-25T00:13:43.651456+0000","last_undegraded":"2024-04-25T00:13:43.651456+0000","last_fullsized":"2024-04-25T00:13:43.651456+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:35:18.110206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.589663+0000","last_change":"2024-04-25T00:12:01.540807+0000","last_active":"2024-04-25T00:13:25.589663+0000","last_peered":"2024-04-25T00:13:25.589663+0000","last_clean":"2024-04-25T00:13:25.589663+0000","last_became_active":"2024-04-25T00:12:01.540272+0000","last_became_peered":"2024-04-25T00:12:01.540272+0000","last_unstale":"2024-04-25T00:13:25.589663+0000","last_undegraded":"2024-04-25T00:13:25.589663+0000","last_fullsized":"2024-04-25T00:13:25.589663+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:08:34.007429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:08:34.007429+0000","last_clean_scrub_stamp":"2024-04-25T00:08:34.007429+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:50:14.642604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.448376+0000","last_change":"2024-04-25T00:13:23.862493+0000","last_active":"2024-04-25T00:13:27.448376+0000","last_peered":"2024-04-25T00:13:27.448376+0000","last_clean":"2024-04-25T00:13:27.448376+0000","last_became_active":"2024-04-25T00:13:23.861972+0000","last_became_peered":"2024-04-25T00:13:23.861972+0000","last_unstale":"2024-04-25T00:13:27.448376+0000","last_undegraded":"2024-04-25T00:13:27.448376+0000","last_fullsized":"2024-04-25T00:13:27.448376+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:49:13.507357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.665746+0000","last_change":"2024-04-25T00:13:19.421920+0000","last_active":"2024-04-25T00:13:27.665746+0000","last_peered":"2024-04-25T00:13:27.665746+0000","last_clean":"2024-04-25T00:13:27.665746+0000","last_became_active":"2024-04-25T00:13:19.421585+0000","last_became_peered":"2024-04-25T00:13:19.421585+0000","last_unstale":"2024-04-25T00:13:27.665746+0000","last_undegraded":"2024-04-25T00:13:27.665746+0000","last_fullsized":"2024-04-25T00:13:27.665746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:53:24.937520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:44.659765+0000","last_change":"2024-04-25T00:13:18.527062+0000","last_active":"2024-04-25T00:13:44.659765+0000","last_peered":"2024-04-25T00:13:44.659765+0000","last_clean":"2024-04-25T00:13:44.659765+0000","last_became_active":"2024-04-25T00:13:18.526338+0000","last_became_peered":"2024-04-25T00:13:18.526338+0000","last_unstale":"2024-04-25T00:13:44.659765+0000","last_undegraded":"2024-04-25T00:13:44.659765+0000","last_fullsized":"2024-04-25T00:13:44.659765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:21.756728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447387+0000","last_change":"2024-04-25T00:13:23.575162+0000","last_active":"2024-04-25T00:13:27.447387+0000","last_peered":"2024-04-25T00:13:27.447387+0000","last_clean":"2024-04-25T00:13:27.447387+0000","last_became_active":"2024-04-25T00:13:23.574722+0000","last_became_peered":"2024-04-25T00:13:23.574722+0000","last_unstale":"2024-04-25T00:13:27.447387+0000","last_undegraded":"2024-04-25T00:13:27.447387+0000","last_fullsized":"2024-04-25T00:13:27.447387+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:52.958539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.860339+0000","last_change":"2024-04-25T00:13:18.523019+0000","last_active":"2024-04-25T00:13:25.860339+0000","last_peered":"2024-04-25T00:13:25.860339+0000","last_clean":"2024-04-25T00:13:25.860339+0000","last_became_active":"2024-04-25T00:13:18.522677+0000","last_became_peered":"2024-04-25T00:13:18.522677+0000","last_unstale":"2024-04-25T00:13:25.860339+0000","last_undegraded":"2024-04-25T00:13:25.860339+0000","last_fullsized":"2024-04-25T00:13:25.860339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:14.470154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.666228+0000","last_change":"2024-04-25T00:13:19.417672+0000","last_active":"2024-04-25T00:13:27.666228+0000","last_peered":"2024-04-25T00:13:27.666228+0000","last_clean":"2024-04-25T00:13:27.666228+0000","last_became_active":"2024-04-25T00:13:19.417323+0000","last_became_peered":"2024-04-25T00:13:19.417323+0000","last_unstale":"2024-04-25T00:13:27.666228+0000","last_undegraded":"2024-04-25T00:13:27.666228+0000","last_fullsized":"2024-04-25T00:13:27.666228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:53.734784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.208782+0000","last_change":"2024-04-25T00:13:23.862320+0000","last_active":"2024-04-25T00:13:27.208782+0000","last_peered":"2024-04-25T00:13:27.208782+0000","last_clean":"2024-04-25T00:13:27.208782+0000","last_became_active":"2024-04-25T00:13:23.861895+0000","last_became_peered":"2024-04-25T00:13:23.861895+0000","last_unstale":"2024-04-25T00:13:27.208782+0000","last_undegraded":"2024-04-25T00:13:27.208782+0000","last_fullsized":"2024-04-25T00:13:27.208782+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:16:21.324727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.577535+0000","last_change":"2024-04-25T00:13:18.520989+0000","last_active":"2024-04-25T00:13:24.577535+0000","last_peered":"2024-04-25T00:13:24.577535+0000","last_clean":"2024-04-25T00:13:24.577535+0000","last_became_active":"2024-04-25T00:13:18.520566+0000","last_became_peered":"2024-04-25T00:13:18.520566+0000","last_unstale":"2024-04-25T00:13:24.577535+0000","last_undegraded":"2024-04-25T00:13:24.577535+0000","last_fullsized":"2024-04-25T00:13:24.577535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:22:32.532352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.208988+0000","last_change":"2024-04-25T00:13:19.422741+0000","last_active":"2024-04-25T00:13:27.208988+0000","last_peered":"2024-04-25T00:13:27.208988+0000","last_clean":"2024-04-25T00:13:27.208988+0000","last_became_active":"2024-04-25T00:13:19.422379+0000","last_became_peered":"2024-04-25T00:13:19.422379+0000","last_unstale":"2024-04-25T00:13:27.208988+0000","last_undegraded":"2024-04-25T00:13:27.208988+0000","last_fullsized":"2024-04-25T00:13:27.208988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:41.337641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573283+0000","last_change":"2024-04-25T00:13:23.577888+0000","last_active":"2024-04-25T00:13:24.573283+0000","last_peered":"2024-04-25T00:13:24.573283+0000","last_clean":"2024-04-25T00:13:24.573283+0000","last_became_active":"2024-04-25T00:13:23.577440+0000","last_became_peered":"2024-04-25T00:13:23.577440+0000","last_unstale":"2024-04-25T00:13:24.573283+0000","last_undegraded":"2024-04-25T00:13:24.573283+0000","last_fullsized":"2024-04-25T00:13:24.573283+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:03.760277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:43.650947+0000","last_change":"2024-04-25T00:13:18.527996+0000","last_active":"2024-04-25T00:13:43.650947+0000","last_peered":"2024-04-25T00:13:43.650947+0000","last_clean":"2024-04-25T00:13:43.650947+0000","last_became_active":"2024-04-25T00:13:18.527636+0000","last_became_peered":"2024-04-25T00:13:18.527636+0000","last_unstale":"2024-04-25T00:13:43.650947+0000","last_undegraded":"2024-04-25T00:13:43.650947+0000","last_fullsized":"2024-04-25T00:13:43.650947+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:52.137108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446673+0000","last_change":"2024-04-25T00:13:19.425538+0000","last_active":"2024-04-25T00:13:27.446673+0000","last_peered":"2024-04-25T00:13:27.446673+0000","last_clean":"2024-04-25T00:13:27.446673+0000","last_became_active":"2024-04-25T00:13:19.420790+0000","last_became_peered":"2024-04-25T00:13:19.420790+0000","last_unstale":"2024-04-25T00:13:27.446673+0000","last_undegraded":"2024-04-25T00:13:27.446673+0000","last_fullsized":"2024-04-25T00:13:27.446673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:04.319759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771854+0000","last_change":"2024-04-25T00:13:23.863187+0000","last_active":"2024-04-25T00:13:26.771854+0000","last_peered":"2024-04-25T00:13:26.771854+0000","last_clean":"2024-04-25T00:13:26.771854+0000","last_became_active":"2024-04-25T00:13:23.862206+0000","last_became_peered":"2024-04-25T00:13:23.862206+0000","last_unstale":"2024-04-25T00:13:26.771854+0000","last_undegraded":"2024-04-25T00:13:26.771854+0000","last_fullsized":"2024-04-25T00:13:26.771854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:57:17.395007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:43.650679+0000","last_change":"2024-04-25T00:13:18.529224+0000","last_active":"2024-04-25T00:13:43.650679+0000","last_peered":"2024-04-25T00:13:43.650679+0000","last_clean":"2024-04-25T00:13:43.650679+0000","last_became_active":"2024-04-25T00:13:18.528908+0000","last_became_peered":"2024-04-25T00:13:18.528908+0000","last_unstale":"2024-04-25T00:13:43.650679+0000","last_undegraded":"2024-04-25T00:13:43.650679+0000","last_fullsized":"2024-04-25T00:13:43.650679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:27.035726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.859754+0000","last_change":"2024-04-25T00:13:19.422223+0000","last_active":"2024-04-25T00:13:25.859754+0000","last_peered":"2024-04-25T00:13:25.859754+0000","last_clean":"2024-04-25T00:13:25.859754+0000","last_became_active":"2024-04-25T00:13:19.421524+0000","last_became_peered":"2024-04-25T00:13:19.421524+0000","last_unstale":"2024-04-25T00:13:25.859754+0000","last_undegraded":"2024-04-25T00:13:25.859754+0000","last_fullsized":"2024-04-25T00:13:25.859754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:17:01.849503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.577624+0000","last_change":"2024-04-25T00:13:23.583015+0000","last_active":"2024-04-25T00:13:24.577624+0000","last_peered":"2024-04-25T00:13:24.577624+0000","last_clean":"2024-04-25T00:13:24.577624+0000","last_became_active":"2024-04-25T00:13:23.582612+0000","last_became_peered":"2024-04-25T00:13:23.582612+0000","last_unstale":"2024-04-25T00:13:24.577624+0000","last_undegraded":"2024-04-25T00:13:24.577624+0000","last_fullsized":"2024-04-25T00:13:24.577624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:36.872281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:43.651932+0000","last_change":"2024-04-25T00:13:18.530027+0000","last_active":"2024-04-25T00:13:43.651932+0000","last_peered":"2024-04-25T00:13:43.651932+0000","last_clean":"2024-04-25T00:13:43.651932+0000","last_became_active":"2024-04-25T00:13:18.529717+0000","last_became_peered":"2024-04-25T00:13:18.529717+0000","last_unstale":"2024-04-25T00:13:43.651932+0000","last_undegraded":"2024-04-25T00:13:43.651932+0000","last_fullsized":"2024-04-25T00:13:43.651932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:23:31.020637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446886+0000","last_change":"2024-04-25T00:13:19.428650+0000","last_active":"2024-04-25T00:13:27.446886+0000","last_peered":"2024-04-25T00:13:27.446886+0000","last_clean":"2024-04-25T00:13:27.446886+0000","last_became_active":"2024-04-25T00:13:19.428156+0000","last_became_peered":"2024-04-25T00:13:19.428156+0000","last_unstale":"2024-04-25T00:13:27.446886+0000","last_undegraded":"2024-04-25T00:13:27.446886+0000","last_fullsized":"2024-04-25T00:13:27.446886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:43.094529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771776+0000","last_change":"2024-04-25T00:13:23.579125+0000","last_active":"2024-04-25T00:13:26.771776+0000","last_peered":"2024-04-25T00:13:26.771776+0000","last_clean":"2024-04-25T00:13:26.771776+0000","last_became_active":"2024-04-25T00:13:23.578728+0000","last_became_peered":"2024-04-25T00:13:23.578728+0000","last_unstale":"2024-04-25T00:13:26.771776+0000","last_undegraded":"2024-04-25T00:13:26.771776+0000","last_fullsized":"2024-04-25T00:13:26.771776+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:16:16.834399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573393+0000","last_change":"2024-04-25T00:13:18.528564+0000","last_active":"2024-04-25T00:13:24.573393+0000","last_peered":"2024-04-25T00:13:24.573393+0000","last_clean":"2024-04-25T00:13:24.573393+0000","last_became_active":"2024-04-25T00:13:18.528203+0000","last_became_peered":"2024-04-25T00:13:18.528203+0000","last_unstale":"2024-04-25T00:13:24.573393+0000","last_undegraded":"2024-04-25T00:13:24.573393+0000","last_fullsized":"2024-04-25T00:13:24.573393+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:47.087562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.209152+0000","last_change":"2024-04-25T00:13:19.414848+0000","last_active":"2024-04-25T00:13:27.209152+0000","last_peered":"2024-04-25T00:13:27.209152+0000","last_clean":"2024-04-25T00:13:27.209152+0000","last_became_active":"2024-04-25T00:13:19.414370+0000","last_became_peered":"2024-04-25T00:13:19.414370+0000","last_unstale":"2024-04-25T00:13:27.209152+0000","last_undegraded":"2024-04-25T00:13:27.209152+0000","last_fullsized":"2024-04-25T00:13:27.209152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:41:08.779171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:26.771847+0000","last_change":"2024-04-25T00:13:23.862577+0000","last_active":"2024-04-25T00:13:26.771847+0000","last_peered":"2024-04-25T00:13:26.771847+0000","last_clean":"2024-04-25T00:13:26.771847+0000","last_became_active":"2024-04-25T00:13:23.862098+0000","last_became_peered":"2024-04-25T00:13:23.862098+0000","last_unstale":"2024-04-25T00:13:26.771847+0000","last_undegraded":"2024-04-25T00:13:26.771847+0000","last_fullsized":"2024-04-25T00:13:26.771847+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:08:00.344041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:43.651052+0000","last_change":"2024-04-25T00:13:18.520638+0000","last_active":"2024-04-25T00:13:43.651052+0000","last_peered":"2024-04-25T00:13:43.651052+0000","last_clean":"2024-04-25T00:13:43.651052+0000","last_became_active":"2024-04-25T00:13:18.520268+0000","last_became_peered":"2024-04-25T00:13:18.520268+0000","last_unstale":"2024-04-25T00:13:43.651052+0000","last_undegraded":"2024-04-25T00:13:43.651052+0000","last_fullsized":"2024-04-25T00:13:43.651052+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:31:00.094521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.590016+0000","last_change":"2024-04-25T00:13:19.427992+0000","last_active":"2024-04-25T00:13:25.590016+0000","last_peered":"2024-04-25T00:13:25.590016+0000","last_clean":"2024-04-25T00:13:25.590016+0000","last_became_active":"2024-04-25T00:13:19.427665+0000","last_became_peered":"2024-04-25T00:13:19.427665+0000","last_unstale":"2024-04-25T00:13:25.590016+0000","last_undegraded":"2024-04-25T00:13:25.590016+0000","last_fullsized":"2024-04-25T00:13:25.590016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:12:31.582185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.397517+0000","last_change":"2024-04-25T00:13:23.582430+0000","last_active":"2024-04-25T00:13:27.397517+0000","last_peered":"2024-04-25T00:13:27.397517+0000","last_clean":"2024-04-25T00:13:27.397517+0000","last_became_active":"2024-04-25T00:13:23.582008+0000","last_became_peered":"2024-04-25T00:13:23.582008+0000","last_unstale":"2024-04-25T00:13:27.397517+0000","last_undegraded":"2024-04-25T00:13:27.397517+0000","last_fullsized":"2024-04-25T00:13:27.397517+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:11.068720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573822+0000","last_change":"2024-04-25T00:13:18.528030+0000","last_active":"2024-04-25T00:13:24.573822+0000","last_peered":"2024-04-25T00:13:24.573822+0000","last_clean":"2024-04-25T00:13:24.573822+0000","last_became_active":"2024-04-25T00:13:18.527636+0000","last_became_peered":"2024-04-25T00:13:18.527636+0000","last_unstale":"2024-04-25T00:13:24.573822+0000","last_undegraded":"2024-04-25T00:13:24.573822+0000","last_fullsized":"2024-04-25T00:13:24.573822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:36.311810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.590460+0000","last_change":"2024-04-25T00:13:19.428181+0000","last_active":"2024-04-25T00:13:25.590460+0000","last_peered":"2024-04-25T00:13:25.590460+0000","last_clean":"2024-04-25T00:13:25.590460+0000","last_became_active":"2024-04-25T00:13:19.427785+0000","last_became_peered":"2024-04-25T00:13:19.427785+0000","last_unstale":"2024-04-25T00:13:25.590460+0000","last_undegraded":"2024-04-25T00:13:25.590460+0000","last_fullsized":"2024-04-25T00:13:25.590460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:31.745679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446376+0000","last_change":"2024-04-25T00:13:23.590162+0000","last_active":"2024-04-25T00:13:27.446376+0000","last_peered":"2024-04-25T00:13:27.446376+0000","last_clean":"2024-04-25T00:13:27.446376+0000","last_became_active":"2024-04-25T00:13:23.589708+0000","last_became_peered":"2024-04-25T00:13:23.589708+0000","last_unstale":"2024-04-25T00:13:27.446376+0000","last_undegraded":"2024-04-25T00:13:27.446376+0000","last_fullsized":"2024-04-25T00:13:27.446376+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:39:15.606032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.448275+0000","last_change":"2024-04-25T00:13:18.529622+0000","last_active":"2024-04-25T00:13:27.448275+0000","last_peered":"2024-04-25T00:13:27.448275+0000","last_clean":"2024-04-25T00:13:27.448275+0000","last_became_active":"2024-04-25T00:13:18.529305+0000","last_became_peered":"2024-04-25T00:13:18.529305+0000","last_unstale":"2024-04-25T00:13:27.448275+0000","last_undegraded":"2024-04-25T00:13:27.448275+0000","last_fullsized":"2024-04-25T00:13:27.448275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:10:43.334712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.590139+0000","last_change":"2024-04-25T00:13:19.421783+0000","last_active":"2024-04-25T00:13:25.590139+0000","last_peered":"2024-04-25T00:13:25.590139+0000","last_clean":"2024-04-25T00:13:25.590139+0000","last_became_active":"2024-04-25T00:13:19.421447+0000","last_became_peered":"2024-04-25T00:13:19.421447+0000","last_unstale":"2024-04-25T00:13:25.590139+0000","last_undegraded":"2024-04-25T00:13:25.590139+0000","last_fullsized":"2024-04-25T00:13:25.590139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:16.107220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447155+0000","last_change":"2024-04-25T00:13:23.578252+0000","last_active":"2024-04-25T00:13:27.447155+0000","last_peered":"2024-04-25T00:13:27.447155+0000","last_clean":"2024-04-25T00:13:27.447155+0000","last_became_active":"2024-04-25T00:13:23.577779+0000","last_became_peered":"2024-04-25T00:13:23.577779+0000","last_unstale":"2024-04-25T00:13:27.447155+0000","last_undegraded":"2024-04-25T00:13:27.447155+0000","last_fullsized":"2024-04-25T00:13:27.447155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:43.822447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.666353+0000","last_change":"2024-04-25T00:13:18.523583+0000","last_active":"2024-04-25T00:13:27.666353+0000","last_peered":"2024-04-25T00:13:27.666353+0000","last_clean":"2024-04-25T00:13:27.666353+0000","last_became_active":"2024-04-25T00:13:18.523238+0000","last_became_peered":"2024-04-25T00:13:18.523238+0000","last_unstale":"2024-04-25T00:13:27.666353+0000","last_undegraded":"2024-04-25T00:13:27.666353+0000","last_fullsized":"2024-04-25T00:13:27.666353+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:03.869276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:25.590390+0000","last_change":"2024-04-25T00:13:19.426021+0000","last_active":"2024-04-25T00:13:25.590390+0000","last_peered":"2024-04-25T00:13:25.590390+0000","last_clean":"2024-04-25T00:13:25.590390+0000","last_became_active":"2024-04-25T00:13:19.425676+0000","last_became_peered":"2024-04-25T00:13:19.425676+0000","last_unstale":"2024-04-25T00:13:25.590390+0000","last_undegraded":"2024-04-25T00:13:25.590390+0000","last_fullsized":"2024-04-25T00:13:25.590390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:10.044202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446979+0000","last_change":"2024-04-25T00:13:19.422402+0000","last_active":"2024-04-25T00:13:27.446979+0000","last_peered":"2024-04-25T00:13:27.446979+0000","last_clean":"2024-04-25T00:13:27.446979+0000","last_became_active":"2024-04-25T00:13:19.422057+0000","last_became_peered":"2024-04-25T00:13:19.422057+0000","last_unstale":"2024-04-25T00:13:27.446979+0000","last_undegraded":"2024-04-25T00:13:27.446979+0000","last_fullsized":"2024-04-25T00:13:27.446979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:31:43.703030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573658+0000","last_change":"2024-04-25T00:13:19.425434+0000","last_active":"2024-04-25T00:13:24.573658+0000","last_peered":"2024-04-25T00:13:24.573658+0000","last_clean":"2024-04-25T00:13:24.573658+0000","last_became_active":"2024-04-25T00:13:19.425051+0000","last_became_peered":"2024-04-25T00:13:19.425051+0000","last_unstale":"2024-04-25T00:13:24.573658+0000","last_undegraded":"2024-04-25T00:13:24.573658+0000","last_fullsized":"2024-04-25T00:13:24.573658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:17.568415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.573288+0000","last_change":"2024-04-25T00:13:19.413835+0000","last_active":"2024-04-25T00:13:24.573288+0000","last_peered":"2024-04-25T00:13:24.573288+0000","last_clean":"2024-04-25T00:13:24.573288+0000","last_became_active":"2024-04-25T00:13:19.413406+0000","last_became_peered":"2024-04-25T00:13:19.413406+0000","last_unstale":"2024-04-25T00:13:24.573288+0000","last_undegraded":"2024-04-25T00:13:24.573288+0000","last_fullsized":"2024-04-25T00:13:24.573288+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:33.852391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.446547+0000","last_change":"2024-04-25T00:13:19.426875+0000","last_active":"2024-04-25T00:13:27.446547+0000","last_peered":"2024-04-25T00:13:27.446547+0000","last_clean":"2024-04-25T00:13:27.446547+0000","last_became_active":"2024-04-25T00:13:19.425828+0000","last_became_peered":"2024-04-25T00:13:19.425828+0000","last_unstale":"2024-04-25T00:13:27.446547+0000","last_undegraded":"2024-04-25T00:13:27.446547+0000","last_fullsized":"2024-04-25T00:13:27.446547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:21.119551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.666359+0000","last_change":"2024-04-25T00:13:19.423637+0000","last_active":"2024-04-25T00:13:27.666359+0000","last_peered":"2024-04-25T00:13:27.666359+0000","last_clean":"2024-04-25T00:13:27.666359+0000","last_became_active":"2024-04-25T00:13:19.423272+0000","last_became_peered":"2024-04-25T00:13:19.423272+0000","last_unstale":"2024-04-25T00:13:27.666359+0000","last_undegraded":"2024-04-25T00:13:27.666359+0000","last_fullsized":"2024-04-25T00:13:27.666359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:32:38.794966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447519+0000","last_change":"2024-04-25T00:13:19.423538+0000","last_active":"2024-04-25T00:13:27.447519+0000","last_peered":"2024-04-25T00:13:27.447519+0000","last_clean":"2024-04-25T00:13:27.447519+0000","last_became_active":"2024-04-25T00:13:19.423028+0000","last_became_peered":"2024-04-25T00:13:19.423028+0000","last_unstale":"2024-04-25T00:13:27.447519+0000","last_undegraded":"2024-04-25T00:13:27.447519+0000","last_fullsized":"2024-04-25T00:13:27.447519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:29:41.975044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:24.574048+0000","last_change":"2024-04-25T00:13:19.416899+0000","last_active":"2024-04-25T00:13:24.574048+0000","last_peered":"2024-04-25T00:13:24.574048+0000","last_clean":"2024-04-25T00:13:24.574048+0000","last_became_active":"2024-04-25T00:13:19.416505+0000","last_became_peered":"2024-04-25T00:13:19.416505+0000","last_unstale":"2024-04-25T00:13:24.574048+0000","last_undegraded":"2024-04-25T00:13:24.574048+0000","last_fullsized":"2024-04-25T00:13:24.574048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:37:40.204119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.209117+0000","last_change":"2024-04-25T00:13:19.426215+0000","last_active":"2024-04-25T00:13:27.209117+0000","last_peered":"2024-04-25T00:13:27.209117+0000","last_clean":"2024-04-25T00:13:27.209117+0000","last_became_active":"2024-04-25T00:13:19.422804+0000","last_became_peered":"2024-04-25T00:13:19.422804+0000","last_unstale":"2024-04-25T00:13:27.209117+0000","last_undegraded":"2024-04-25T00:13:27.209117+0000","last_fullsized":"2024-04-25T00:13:27.209117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:24:44.630891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447319+0000","last_change":"2024-04-25T00:13:19.413564+0000","last_active":"2024-04-25T00:13:27.447319+0000","last_peered":"2024-04-25T00:13:27.447319+0000","last_clean":"2024-04-25T00:13:27.447319+0000","last_became_active":"2024-04-25T00:13:19.413223+0000","last_became_peered":"2024-04-25T00:13:19.413223+0000","last_unstale":"2024-04-25T00:13:27.447319+0000","last_undegraded":"2024-04-25T00:13:27.447319+0000","last_fullsized":"2024-04-25T00:13:27.447319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:45.629018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.208768+0000","last_change":"2024-04-25T00:13:19.425945+0000","last_active":"2024-04-25T00:13:27.208768+0000","last_peered":"2024-04-25T00:13:27.208768+0000","last_clean":"2024-04-25T00:13:27.208768+0000","last_became_active":"2024-04-25T00:13:19.422499+0000","last_became_peered":"2024-04-25T00:13:19.422499+0000","last_unstale":"2024-04-25T00:13:27.208768+0000","last_undegraded":"2024-04-25T00:13:27.208768+0000","last_fullsized":"2024-04-25T00:13:27.208768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:45.299300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.397742+0000","last_change":"2024-04-25T00:13:19.418857+0000","last_active":"2024-04-25T00:13:27.397742+0000","last_peered":"2024-04-25T00:13:27.397742+0000","last_clean":"2024-04-25T00:13:27.397742+0000","last_became_active":"2024-04-25T00:13:19.418535+0000","last_became_peered":"2024-04-25T00:13:19.418535+0000","last_unstale":"2024-04-25T00:13:27.397742+0000","last_undegraded":"2024-04-25T00:13:27.397742+0000","last_fullsized":"2024-04-25T00:13:27.397742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:44.288129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.665972+0000","last_change":"2024-04-25T00:13:19.420550+0000","last_active":"2024-04-25T00:13:27.665972+0000","last_peered":"2024-04-25T00:13:27.665972+0000","last_clean":"2024-04-25T00:13:27.665972+0000","last_became_active":"2024-04-25T00:13:19.420235+0000","last_became_peered":"2024-04-25T00:13:19.420235+0000","last_unstale":"2024-04-25T00:13:27.665972+0000","last_undegraded":"2024-04-25T00:13:27.665972+0000","last_fullsized":"2024-04-25T00:13:27.665972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:59:36.543776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T00:13:27.447420+0000","last_change":"2024-04-25T00:13:19.429995+0000","last_active":"2024-04-25T00:13:27.447420+0000","last_peered":"2024-04-25T00:13:27.447420+0000","last_clean":"2024-04-25T00:13:27.447420+0000","last_became_active":"2024-04-25T00:13:19.429185+0000","last_became_peered":"2024-04-25T00:13:19.429185+0000","last_unstale":"2024-04-25T00:13:27.447420+0000","last_undegraded":"2024-04-25T00:13:27.447420+0000","last_fullsized":"2024-04-25T00:13:27.447420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:29:27.974479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":213064,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4954,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":18024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383699,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645334,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1071619,"data_compressed":50438,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":477841,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":1068787,"data_compressed":50438,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495660,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738443,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":477909,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":481091,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":970752,"data_stored":477319,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14654,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":50438,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:13:48.764 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T00:13:48.764 INFO:tasks.cephadm:Setting up client nodes... 2024-04-25T00:13:48.764 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-25T00:13:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:48 smithi027 ceph-mon[29208]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 3 op/s 2024-04-25T00:13:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:48 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/34035507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:48 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1813188172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:48 smithi027 ceph-mon[29208]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:13:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:48 smithi027 ceph-mon[29208]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T00:13:49.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:48 smithi139 ceph-mon[32833]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 3 op/s 2024-04-25T00:13:49.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:48 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/34035507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:49.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:48 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1813188172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:49.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:48 smithi139 ceph-mon[32833]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:13:49.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:48 smithi139 ceph-mon[32833]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T00:13:49.004 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:13:48 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-b[62989]: 2024-04-25T00:13:48.736+0000 7f853bf52700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T00:13:49.004 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:13:48 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-b[62989]: 2024-04-25T00:13:48.736+0000 7f853bf52700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-25T00:13:49.004 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:13:48 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-b[62989]: 2024-04-25T00:13:48.736+0000 7f853bf52700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T00:13:49.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:48 smithi153 ceph-mon[32896]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 3 op/s 2024-04-25T00:13:49.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:48 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/34035507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:13:49.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:48 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1813188172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:49.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:48 smithi153 ceph-mon[32896]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:13:49.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:48 smithi153 ceph-mon[32896]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T00:13:49.046 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:13:49.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:49 smithi027 ceph-mon[29208]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:13:50.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:49 smithi139 ceph-mon[32833]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:13:50.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:49 smithi153 ceph-mon[32896]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:13:50.795 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:50 smithi027 ceph-mon[29208]: pgmap v309: 129 pgs: 129 active+clean; 622 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 7.0 KiB/s wr, 15 op/s 2024-04-25T00:13:51.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:50 smithi139 ceph-mon[32833]: pgmap v309: 129 pgs: 129 active+clean; 622 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 7.0 KiB/s wr, 15 op/s 2024-04-25T00:13:51.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:50 smithi153 ceph-mon[32896]: pgmap v309: 129 pgs: 129 active+clean; 622 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 7.0 KiB/s wr, 15 op/s 2024-04-25T00:13:51.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.909+0000 7f9832b70700 1 -- 172.21.15.27:0/2261729859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f982c101b60 msgr2=0x7f982c101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:51.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.909+0000 7f9832b70700 1 --2- 172.21.15.27:0/2261729859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f982c101b60 0x7f982c101fc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f98200099c0 tx=0x7f9820009cd0 comp rx=0 tx=0).stop 2024-04-25T00:13:51.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.909+0000 7f9832b70700 1 -- 172.21.15.27:0/2261729859 shutdown_connections 2024-04-25T00:13:51.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.909+0000 7f9832b70700 1 --2- 172.21.15.27:0/2261729859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f982c102500 0x7f982c108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:51.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.909+0000 7f9832b70700 1 --2- 172.21.15.27:0/2261729859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c100960 0x7f982c100d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:51.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.909+0000 7f9832b70700 1 --2- 172.21.15.27:0/2261729859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f982c101b60 0x7f982c101fc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:51.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.909+0000 7f9832b70700 1 -- 172.21.15.27:0/2261729859 >> 172.21.15.27:0/2261729859 conn(0x7f982c0fbf00 msgr2=0x7f982c0fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:51.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.909+0000 7f9832b70700 1 -- 172.21.15.27:0/2261729859 shutdown_connections 2024-04-25T00:13:51.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.909+0000 7f9832b70700 1 -- 172.21.15.27:0/2261729859 wait complete. 2024-04-25T00:13:51.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.910+0000 7f9832b70700 1 Processor -- start 2024-04-25T00:13:51.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.910+0000 7f9832b70700 1 -- start start 2024-04-25T00:13:51.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.910+0000 7f9832b70700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f982c100960 0x7f982c197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:51.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.910+0000 7f9832b70700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c101b60 0x7f982c197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:51.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.910+0000 7f9832b70700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f982c102500 0x7f982c19e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:51.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.910+0000 7f9832b70700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f982c10ce00 con 0x7f982c100960 2024-04-25T00:13:51.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.910+0000 7f9832b70700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f982c10cc80 con 0x7f982c102500 2024-04-25T00:13:51.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.910+0000 7f9832b70700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f982c10cf80 con 0x7f982c101b60 2024-04-25T00:13:51.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.910+0000 7f983090c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f982c100960 0x7f982c197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:51.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.910+0000 7f983090c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f982c100960 0x7f982c197320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51958/0 (socket says 172.21.15.27:51958) 2024-04-25T00:13:51.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.910+0000 7f983090c700 1 -- 172.21.15.27:0/3255936387 learned_addr learned my addr 172.21.15.27:0/3255936387 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:51.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.910+0000 7f982bfff700 1 --2- 172.21.15.27:0/3255936387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c101b60 0x7f982c197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.911+0000 7f983110d700 1 --2- 172.21.15.27:0/3255936387 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f982c102500 0x7f982c19e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.911+0000 7f983090c700 1 -- 172.21.15.27:0/3255936387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c101b60 msgr2=0x7f982c197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.911+0000 7f983090c700 1 --2- 172.21.15.27:0/3255936387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c101b60 0x7f982c197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.911+0000 7f983090c700 1 -- 172.21.15.27:0/3255936387 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f982c102500 msgr2=0x7f982c19e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.911+0000 7f983090c700 1 --2- 172.21.15.27:0/3255936387 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f982c102500 0x7f982c19e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.911+0000 7f983090c700 1 -- 172.21.15.27:0/3255936387 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f982c19ef80 con 0x7f982c100960 2024-04-25T00:13:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.911+0000 7f982bfff700 1 --2- 172.21.15.27:0/3255936387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c101b60 0x7f982c197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:13:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.911+0000 7f983110d700 1 --2- 172.21.15.27:0/3255936387 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f982c102500 0x7f982c19e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.911+0000 7f983090c700 1 --2- 172.21.15.27:0/3255936387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f982c100960 0x7f982c197320 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f981c00d9b0 tx=0x7f981c00dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.911+0000 7f9829ffb700 1 -- 172.21.15.27:0/3255936387 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981c017030 con 0x7f982c100960 2024-04-25T00:13:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.911+0000 7f9832b70700 1 -- 172.21.15.27:0/3255936387 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f982c19f270 con 0x7f982c100960 2024-04-25T00:13:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.911+0000 7f9829ffb700 1 -- 172.21.15.27:0/3255936387 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f981c004500 con 0x7f982c100960 2024-04-25T00:13:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.911+0000 7f9829ffb700 1 -- 172.21.15.27:0/3255936387 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981c003aa0 con 0x7f982c100960 2024-04-25T00:13:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.911+0000 7f9832b70700 1 -- 172.21.15.27:0/3255936387 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f982c19f7b0 con 0x7f982c100960 2024-04-25T00:13:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.912+0000 7f9832b70700 1 -- 172.21.15.27:0/3255936387 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f982c0507a0 con 0x7f982c100960 2024-04-25T00:13:51.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.913+0000 7f9829ffb700 1 -- 172.21.15.27:0/3255936387 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f981c01a070 con 0x7f982c100960 2024-04-25T00:13:51.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.914+0000 7f9829ffb700 1 --2- 172.21.15.27:0/3255936387 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f98140a2d60 0x7f98140a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:51.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.914+0000 7f982bfff700 1 --2- 172.21.15.27:0/3255936387 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f98140a2d60 0x7f98140a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:51.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.914+0000 7f9829ffb700 1 -- 172.21.15.27:0/3255936387 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f981c0cedf0 con 0x7f982c100960 2024-04-25T00:13:51.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.914+0000 7f982bfff700 1 --2- 172.21.15.27:0/3255936387 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f98140a2d60 0x7f98140a5220 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9820000c00 tx=0x7f9820005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:51.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:50.919+0000 7f9829ffb700 1 -- 172.21.15.27:0/3255936387 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f981c09b9b0 con 0x7f982c100960 2024-04-25T00:13:51.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:51.210+0000 7f9832b70700 1 -- 172.21.15.27:0/3255936387 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f982c02dc90 con 0x7f982c100960 2024-04-25T00:13:51.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:51.213+0000 7f9829ffb700 1 -- 172.21.15.27:0/3255936387 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f982c02dc90 con 0x7f982c100960 2024-04-25T00:13:51.213 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-04-25T00:13:51.213 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQA/oClmCXKbDBAAc1cK0YjH8aytChN5/sNTTg== 2024-04-25T00:13:51.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:51.215+0000 7f9832b70700 1 -- 172.21.15.27:0/3255936387 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f98140a2d60 msgr2=0x7f98140a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:51.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:51.215+0000 7f9832b70700 1 --2- 172.21.15.27:0/3255936387 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f98140a2d60 0x7f98140a5220 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9820000c00 tx=0x7f9820005fd0 comp rx=0 tx=0).stop 2024-04-25T00:13:51.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:51.215+0000 7f9832b70700 1 -- 172.21.15.27:0/3255936387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f982c100960 msgr2=0x7f982c197320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:51.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:51.215+0000 7f9832b70700 1 --2- 172.21.15.27:0/3255936387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f982c100960 0x7f982c197320 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f981c00d9b0 tx=0x7f981c00dd70 comp rx=0 tx=0).stop 2024-04-25T00:13:51.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:51.215+0000 7f9832b70700 1 -- 172.21.15.27:0/3255936387 shutdown_connections 2024-04-25T00:13:51.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:51.215+0000 7f9832b70700 1 --2- 172.21.15.27:0/3255936387 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f98140a2d60 0x7f98140a5220 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:51.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:51.215+0000 7f9832b70700 1 --2- 172.21.15.27:0/3255936387 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f982c102500 0x7f982c19e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:51.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:51.215+0000 7f9832b70700 1 --2- 172.21.15.27:0/3255936387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f982c100960 0x7f982c197320 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:51.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:51.215+0000 7f9832b70700 1 --2- 172.21.15.27:0/3255936387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c101b60 0x7f982c197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:51.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:51.215+0000 7f9832b70700 1 -- 172.21.15.27:0/3255936387 >> 172.21.15.27:0/3255936387 conn(0x7f982c0fbf00 msgr2=0x7f982c104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:51.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:51.215+0000 7f9832b70700 1 -- 172.21.15.27:0/3255936387 shutdown_connections 2024-04-25T00:13:51.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:51.215+0000 7f9832b70700 1 -- 172.21.15.27:0/3255936387 wait complete. 2024-04-25T00:13:51.791 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:51 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3255936387' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T00:13:51.791 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:51 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3255936387' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T00:13:51.922 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:51 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3255936387' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T00:13:51.922 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:51 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3255936387' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T00:13:51.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:51 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3255936387' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T00:13:51.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:51 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3255936387' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T00:13:52.119 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:13:52.119 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-25T00:13:52.119 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T00:13:52.156 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T00:13:52.156 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-25T00:13:52.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-25T00:13:52.447 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:13:53.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:53 smithi139 ceph-mon[32833]: pgmap v310: 129 pgs: 129 active+clean; 623 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 7.7 KiB/s wr, 20 op/s 2024-04-25T00:13:53.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:53 smithi153 ceph-mon[32896]: pgmap v310: 129 pgs: 129 active+clean; 623 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 7.7 KiB/s wr, 20 op/s 2024-04-25T00:13:53.674 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:53 smithi027 ceph-mon[29208]: pgmap v310: 129 pgs: 129 active+clean; 623 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 7.7 KiB/s wr, 20 op/s 2024-04-25T00:13:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.259+0000 7fc4c9ed0700 1 -- 172.21.15.27:0/1279326280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c4100e30 msgr2=0x7fc4c4101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.259+0000 7fc4c9ed0700 1 --2- 172.21.15.27:0/1279326280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c4100e30 0x7fc4c4101210 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80099c0 tx=0x7fc4b8009cd0 comp rx=0 tx=0).stop 2024-04-25T00:13:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.259+0000 7fc4c9ed0700 1 -- 172.21.15.27:0/1279326280 shutdown_connections 2024-04-25T00:13:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.259+0000 7fc4c9ed0700 1 --2- 172.21.15.27:0/1279326280 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc4c4107650 0x7fc4c410f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.259+0000 7fc4c9ed0700 1 --2- 172.21.15.27:0/1279326280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c4100e30 0x7fc4c4101210 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.259+0000 7fc4c9ed0700 1 --2- 172.21.15.27:0/1279326280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4c41017e0 0x7fc4c4106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.259+0000 7fc4c9ed0700 1 -- 172.21.15.27:0/1279326280 >> 172.21.15.27:0/1279326280 conn(0x7fc4c4076450 msgr2=0x7fc4c4076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.259+0000 7fc4c9ed0700 1 -- 172.21.15.27:0/1279326280 shutdown_connections 2024-04-25T00:13:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.259+0000 7fc4c9ed0700 1 -- 172.21.15.27:0/1279326280 wait complete. 2024-04-25T00:13:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.260+0000 7fc4c9ed0700 1 Processor -- start 2024-04-25T00:13:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.260+0000 7fc4c9ed0700 1 -- start start 2024-04-25T00:13:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.260+0000 7fc4c9ed0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c4100e30 0x7fc4c419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.260+0000 7fc4c9ed0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4c41017e0 0x7fc4c419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.260+0000 7fc4c9ed0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc4c4107650 0x7fc4c41987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.260+0000 7fc4c9ed0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c41125a0 con 0x7fc4c4100e30 2024-04-25T00:13:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.260+0000 7fc4c9ed0700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c4112420 con 0x7fc4c4107650 2024-04-25T00:13:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.260+0000 7fc4c9ed0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c4112720 con 0x7fc4c41017e0 2024-04-25T00:13:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.260+0000 7fc4c37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c4100e30 0x7fc4c419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.260+0000 7fc4c37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c4100e30 0x7fc4c419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51978/0 (socket says 172.21.15.27:51978) 2024-04-25T00:13:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.260+0000 7fc4c37fe700 1 -- 172.21.15.27:0/3131764233 learned_addr learned my addr 172.21.15.27:0/3131764233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.260+0000 7fc4c2ffd700 1 --2- 172.21.15.27:0/3131764233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4c41017e0 0x7fc4c419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.260+0000 7fc4c3fff700 1 --2- 172.21.15.27:0/3131764233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc4c4107650 0x7fc4c41987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.260+0000 7fc4c37fe700 1 -- 172.21.15.27:0/3131764233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4c41017e0 msgr2=0x7fc4c419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.260+0000 7fc4c37fe700 1 --2- 172.21.15.27:0/3131764233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4c41017e0 0x7fc4c419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.260+0000 7fc4c37fe700 1 -- 172.21.15.27:0/3131764233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc4c4107650 msgr2=0x7fc4c41987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.260+0000 7fc4c37fe700 1 --2- 172.21.15.27:0/3131764233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc4c4107650 0x7fc4c41987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.261+0000 7fc4c37fe700 1 -- 172.21.15.27:0/3131764233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4c41990a0 con 0x7fc4c4100e30 2024-04-25T00:13:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.261+0000 7fc4c2ffd700 1 --2- 172.21.15.27:0/3131764233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4c41017e0 0x7fc4c419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:13:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.261+0000 7fc4c37fe700 1 --2- 172.21.15.27:0/3131764233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c4100e30 0x7fc4c419e6a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8006010 tx=0x7fc4b8003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.261+0000 7fc4c0ff9700 1 -- 172.21.15.27:0/3131764233 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8003680 con 0x7fc4c4100e30 2024-04-25T00:13:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.261+0000 7fc4c0ff9700 1 -- 172.21.15.27:0/3131764233 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4b800f460 con 0x7fc4c4100e30 2024-04-25T00:13:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.261+0000 7fc4c0ff9700 1 -- 172.21.15.27:0/3131764233 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8017650 con 0x7fc4c4100e30 2024-04-25T00:13:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.261+0000 7fc4c9ed0700 1 -- 172.21.15.27:0/3131764233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4c4199330 con 0x7fc4c4100e30 2024-04-25T00:13:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.261+0000 7fc4c9ed0700 1 -- 172.21.15.27:0/3131764233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4c41a5450 con 0x7fc4c4100e30 2024-04-25T00:13:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.262+0000 7fc4c9ed0700 1 -- 172.21.15.27:0/3131764233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4c406a480 con 0x7fc4c4100e30 2024-04-25T00:13:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.263+0000 7fc4c0ff9700 1 -- 172.21.15.27:0/3131764233 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc4b800f610 con 0x7fc4c4100e30 2024-04-25T00:13:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.264+0000 7fc4c0ff9700 1 --2- 172.21.15.27:0/3131764233 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc4ac0a2e50 0x7fc4ac0a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.264+0000 7fc4c2ffd700 1 --2- 172.21.15.27:0/3131764233 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc4ac0a2e50 0x7fc4ac0a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:54.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.264+0000 7fc4c0ff9700 1 -- 172.21.15.27:0/3131764233 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc4b80d0370 con 0x7fc4c4100e30 2024-04-25T00:13:54.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.264+0000 7fc4c2ffd700 1 --2- 172.21.15.27:0/3131764233 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc4ac0a2e50 0x7fc4ac0a5310 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4005fd0 tx=0x7fc4b4005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:54.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.268+0000 7fc4c0ff9700 1 -- 172.21.15.27:0/3131764233 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc4b8013070 con 0x7fc4c4100e30 2024-04-25T00:13:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.535+0000 7fc4c9ed0700 1 -- 172.21.15.27:0/3131764233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fc4c402db30 con 0x7fc4c4100e30 2024-04-25T00:13:54.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.539+0000 7fc4c0ff9700 1 -- 172.21.15.27:0/3131764233 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263200 (secure 0 0 0) 0x7fc4b8020370 con 0x7fc4c4100e30 2024-04-25T00:13:54.543 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:54.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.542+0000 7fc4c9ed0700 1 -- 172.21.15.27:0/3131764233 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc4ac0a2e50 msgr2=0x7fc4ac0a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:54.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.542+0000 7fc4c9ed0700 1 --2- 172.21.15.27:0/3131764233 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc4ac0a2e50 0x7fc4ac0a5310 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4005fd0 tx=0x7fc4b4005e30 comp rx=0 tx=0).stop 2024-04-25T00:13:54.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.542+0000 7fc4c9ed0700 1 -- 172.21.15.27:0/3131764233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c4100e30 msgr2=0x7fc4c419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:54.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.542+0000 7fc4c9ed0700 1 --2- 172.21.15.27:0/3131764233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c4100e30 0x7fc4c419e6a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8006010 tx=0x7fc4b8003eb0 comp rx=0 tx=0).stop 2024-04-25T00:13:54.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.542+0000 7fc4c9ed0700 1 -- 172.21.15.27:0/3131764233 shutdown_connections 2024-04-25T00:13:54.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.542+0000 7fc4c9ed0700 1 --2- 172.21.15.27:0/3131764233 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc4ac0a2e50 0x7fc4ac0a5310 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:54.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.542+0000 7fc4c9ed0700 1 --2- 172.21.15.27:0/3131764233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc4c4107650 0x7fc4c41987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:54.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.542+0000 7fc4c9ed0700 1 --2- 172.21.15.27:0/3131764233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c4100e30 0x7fc4c419e6a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:54.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.542+0000 7fc4c9ed0700 1 --2- 172.21.15.27:0/3131764233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4c41017e0 0x7fc4c419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:54.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.542+0000 7fc4c9ed0700 1 -- 172.21.15.27:0/3131764233 >> 172.21.15.27:0/3131764233 conn(0x7fc4c4076450 msgr2=0x7fc4c40feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:54.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.543+0000 7fc4c9ed0700 1 -- 172.21.15.27:0/3131764233 shutdown_connections 2024-04-25T00:13:54.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:54.543+0000 7fc4c9ed0700 1 -- 172.21.15.27:0/3131764233 wait complete. 2024-04-25T00:13:55.393 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":2161554778},{"type":"v1","addr":"172.21.15.27:6801","nonce":2161554778}]},"active_addr":"172.21.15.27:6801/2161554778","active_change":"2024-04-25T00:06:09.485087+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.27:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":4085720546}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":236412562}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":2528869729}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1885111026}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1652011380}]}]} 2024-04-25T00:13:55.398 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-25T00:13:55.398 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-25T00:13:55.398 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T00:13:55.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:55 smithi139 ceph-mon[32833]: pgmap v311: 129 pgs: 129 active+clean; 628 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.3 KiB/s wr, 22 op/s 2024-04-25T00:13:55.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:55 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3131764233' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T00:13:55.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:55 smithi153 ceph-mon[32896]: pgmap v311: 129 pgs: 129 active+clean; 628 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.3 KiB/s wr, 22 op/s 2024-04-25T00:13:55.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:55 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3131764233' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T00:13:55.656 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:13:55.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:55 smithi027 ceph-mon[29208]: pgmap v311: 129 pgs: 129 active+clean; 628 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.3 KiB/s wr, 22 op/s 2024-04-25T00:13:55.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:55 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3131764233' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T00:13:57.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.473+0000 7f098dc56700 1 -- 172.21.15.27:0/1672561009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09881082b0 msgr2=0x7f0988108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:57.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.473+0000 7f098dc56700 1 --2- 172.21.15.27:0/1672561009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09881082b0 0x7f0988108690 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f097800b0a0 tx=0x7f097800b3b0 comp rx=0 tx=0).stop 2024-04-25T00:13:57.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.473+0000 7f098dc56700 1 -- 172.21.15.27:0/1672561009 shutdown_connections 2024-04-25T00:13:57.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.473+0000 7f098dc56700 1 --2- 172.21.15.27:0/1672561009 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f098806a8e0 0x7f098810f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:57.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.473+0000 7f098dc56700 1 --2- 172.21.15.27:0/1672561009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09881082b0 0x7f0988108690 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:57.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.473+0000 7f098dc56700 1 --2- 172.21.15.27:0/1672561009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0988069f40 0x7f098806a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:57.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.473+0000 7f098dc56700 1 -- 172.21.15.27:0/1672561009 >> 172.21.15.27:0/1672561009 conn(0x7f09880780e0 msgr2=0x7f09880784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:57.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.473+0000 7f098dc56700 1 -- 172.21.15.27:0/1672561009 shutdown_connections 2024-04-25T00:13:57.476 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:57 smithi027 ceph-mon[29208]: pgmap v312: 129 pgs: 129 active+clean; 628 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.3 KiB/s wr, 24 op/s 2024-04-25T00:13:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.473+0000 7f098dc56700 1 -- 172.21.15.27:0/1672561009 wait complete. 2024-04-25T00:13:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.474+0000 7f098dc56700 1 Processor -- start 2024-04-25T00:13:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.474+0000 7f098dc56700 1 -- start start 2024-04-25T00:13:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.475+0000 7f098dc56700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0988069f40 0x7f098810e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.475+0000 7f098dc56700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f098806a8e0 0x7f098810e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.475+0000 7f098dc56700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09881082b0 0x7f09881a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.475+0000 7f098dc56700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f09881125c0 con 0x7f09881082b0 2024-04-25T00:13:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.475+0000 7f098dc56700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0988112440 con 0x7f0988069f40 2024-04-25T00:13:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.475+0000 7f098dc56700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0988112740 con 0x7f098806a8e0 2024-04-25T00:13:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.475+0000 7f0987fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09881082b0 0x7f09881a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.475+0000 7f0987fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09881082b0 0x7f09881a3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54726/0 (socket says 172.21.15.27:54726) 2024-04-25T00:13:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.475+0000 7f0987fff700 1 -- 172.21.15.27:0/1083199599 learned_addr learned my addr 172.21.15.27:0/1083199599 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:13:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.475+0000 7f09877fe700 1 --2- 172.21.15.27:0/1083199599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0988069f40 0x7f098810e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.475+0000 7f0986ffd700 1 --2- 172.21.15.27:0/1083199599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f098806a8e0 0x7f098810e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.475+0000 7f0987fff700 1 -- 172.21.15.27:0/1083199599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f098806a8e0 msgr2=0x7f098810e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.475+0000 7f0987fff700 1 --2- 172.21.15.27:0/1083199599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f098806a8e0 0x7f098810e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.475+0000 7f0987fff700 1 -- 172.21.15.27:0/1083199599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0988069f40 msgr2=0x7f098810e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.475+0000 7f0987fff700 1 --2- 172.21.15.27:0/1083199599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0988069f40 0x7f098810e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.475+0000 7f0987fff700 1 -- 172.21.15.27:0/1083199599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09881a4080 con 0x7f09881082b0 2024-04-25T00:13:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.476+0000 7f0987fff700 1 --2- 172.21.15.27:0/1083199599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09881082b0 0x7f09881a3940 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f098000c970 tx=0x7f098000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.476+0000 7f0984ff9700 1 -- 172.21.15.27:0/1083199599 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0980004a30 con 0x7f09881082b0 2024-04-25T00:13:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.476+0000 7f0984ff9700 1 -- 172.21.15.27:0/1083199599 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0980005500 con 0x7f09881082b0 2024-04-25T00:13:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.476+0000 7f098dc56700 1 -- 172.21.15.27:0/1083199599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09881a4370 con 0x7f09881082b0 2024-04-25T00:13:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.476+0000 7f0984ff9700 1 -- 172.21.15.27:0/1083199599 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098000ecf0 con 0x7f09881082b0 2024-04-25T00:13:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.476+0000 7f098dc56700 1 -- 172.21.15.27:0/1083199599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09881abbf0 con 0x7f09881082b0 2024-04-25T00:13:57.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.478+0000 7f098dc56700 1 -- 172.21.15.27:0/1083199599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0988068f20 con 0x7f09881082b0 2024-04-25T00:13:57.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.478+0000 7f0984ff9700 1 -- 172.21.15.27:0/1083199599 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f0980005020 con 0x7f09881082b0 2024-04-25T00:13:57.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.479+0000 7f0984ff9700 1 --2- 172.21.15.27:0/1083199599 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f09740a2cb0 0x7f09740a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:57.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.479+0000 7f09877fe700 1 --2- 172.21.15.27:0/1083199599 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f09740a2cb0 0x7f09740a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:57.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.479+0000 7f0984ff9700 1 -- 172.21.15.27:0/1083199599 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f09800ceef0 con 0x7f09881082b0 2024-04-25T00:13:57.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.479+0000 7f09877fe700 1 --2- 172.21.15.27:0/1083199599 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f09740a2cb0 0x7f09740a5170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f097800b070 tx=0x7f0978006210 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.483+0000 7f0984ff9700 1 -- 172.21.15.27:0/1083199599 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0980014030 con 0x7f09881082b0 2024-04-25T00:13:57.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:57 smithi139 ceph-mon[32833]: pgmap v312: 129 pgs: 129 active+clean; 628 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.3 KiB/s wr, 24 op/s 2024-04-25T00:13:57.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:57 smithi153 ceph-mon[32896]: pgmap v312: 129 pgs: 129 active+clean; 628 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.3 KiB/s wr, 24 op/s 2024-04-25T00:13:57.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.691+0000 7f098dc56700 1 -- 172.21.15.27:0/1083199599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09881abe20 con 0x7f09881082b0 2024-04-25T00:13:57.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.692+0000 7f0984ff9700 1 -- 172.21.15.27:0/1083199599 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24117 (secure 0 0 0) 0x7f098009bab0 con 0x7f09881082b0 2024-04-25T00:13:57.693 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:13:57.694 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:13:23.935081+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:13:57.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.694+0000 7f098dc56700 1 -- 172.21.15.27:0/1083199599 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f09740a2cb0 msgr2=0x7f09740a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:57.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.694+0000 7f098dc56700 1 --2- 172.21.15.27:0/1083199599 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f09740a2cb0 0x7f09740a5170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f097800b070 tx=0x7f0978006210 comp rx=0 tx=0).stop 2024-04-25T00:13:57.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.694+0000 7f098dc56700 1 -- 172.21.15.27:0/1083199599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09881082b0 msgr2=0x7f09881a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:57.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.694+0000 7f098dc56700 1 --2- 172.21.15.27:0/1083199599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09881082b0 0x7f09881a3940 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f098000c970 tx=0x7f098000cc80 comp rx=0 tx=0).stop 2024-04-25T00:13:57.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.694+0000 7f098dc56700 1 -- 172.21.15.27:0/1083199599 shutdown_connections 2024-04-25T00:13:57.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.694+0000 7f098dc56700 1 --2- 172.21.15.27:0/1083199599 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f09740a2cb0 0x7f09740a5170 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:57.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.694+0000 7f098dc56700 1 --2- 172.21.15.27:0/1083199599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09881082b0 0x7f09881a3940 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:57.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.694+0000 7f098dc56700 1 --2- 172.21.15.27:0/1083199599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0988069f40 0x7f098810e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:57.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.694+0000 7f098dc56700 1 --2- 172.21.15.27:0/1083199599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f098806a8e0 0x7f098810e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:57.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.694+0000 7f098dc56700 1 -- 172.21.15.27:0/1083199599 >> 172.21.15.27:0/1083199599 conn(0x7f09880780e0 msgr2=0x7f09880ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:57.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.695+0000 7f098dc56700 1 -- 172.21.15.27:0/1083199599 shutdown_connections 2024-04-25T00:13:57.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:13:57.695+0000 7f098dc56700 1 -- 172.21.15.27:0/1083199599 wait complete. 2024-04-25T00:13:58.465 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-25T00:13:58.465 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T00:13:58.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:58 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1083199599' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:58.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:58 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1083199599' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:58.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:58 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1083199599' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:13:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:13:59.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:13:59 smithi027 ceph-mon[29208]: pgmap v313: 129 pgs: 129 active+clean; 628 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.9 KiB/s wr, 21 op/s 2024-04-25T00:13:59.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:13:59 smithi139 ceph-mon[32833]: pgmap v313: 129 pgs: 129 active+clean; 628 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.9 KiB/s wr, 21 op/s 2024-04-25T00:13:59.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:13:59 smithi153 ceph-mon[32896]: pgmap v313: 129 pgs: 129 active+clean; 628 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.9 KiB/s wr, 21 op/s 2024-04-25T00:14:00.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.439+0000 7f5643605700 1 -- 172.21.15.27:0/2567556211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f563c100e50 msgr2=0x7f563c101230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:00.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.439+0000 7f5643605700 1 --2- 172.21.15.27:0/2567556211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f563c100e50 0x7f563c101230 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f56380099c0 tx=0x7f5638009cd0 comp rx=0 tx=0).stop 2024-04-25T00:14:00.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.439+0000 7f5643605700 1 -- 172.21.15.27:0/2567556211 shutdown_connections 2024-04-25T00:14:00.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.439+0000 7f5643605700 1 --2- 172.21.15.27:0/2567556211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f563c107670 0x7f563c10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:00.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.439+0000 7f5643605700 1 --2- 172.21.15.27:0/2567556211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f563c100e50 0x7f563c101230 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:00.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.439+0000 7f5643605700 1 --2- 172.21.15.27:0/2567556211 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f563c101800 0x7f563c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:00.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.439+0000 7f5643605700 1 -- 172.21.15.27:0/2567556211 >> 172.21.15.27:0/2567556211 conn(0x7f563c076450 msgr2=0x7f563c076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:00.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.439+0000 7f5643605700 1 -- 172.21.15.27:0/2567556211 shutdown_connections 2024-04-25T00:14:00.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.439+0000 7f5643605700 1 -- 172.21.15.27:0/2567556211 wait complete. 2024-04-25T00:14:00.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.440+0000 7f5643605700 1 Processor -- start 2024-04-25T00:14:00.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.440+0000 7f5643605700 1 -- start start 2024-04-25T00:14:00.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.440+0000 7f5643605700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f563c100e50 0x7f563c1a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:00.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.440+0000 7f5643605700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f563c101800 0x7f563c1a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:00.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.440+0000 7f5643605700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f563c107670 0x7f563c19aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:00.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.440+0000 7f5643605700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f563c112610 con 0x7f563c107670 2024-04-25T00:14:00.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.440+0000 7f5643605700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f563c112490 con 0x7f563c100e50 2024-04-25T00:14:00.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.440+0000 7f5643605700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f563c112790 con 0x7f563c101800 2024-04-25T00:14:00.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.440+0000 7f5641ba2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f563c107670 0x7f563c19aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:00.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.440+0000 7f5641ba2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f563c107670 0x7f563c19aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54744/0 (socket says 172.21.15.27:54744) 2024-04-25T00:14:00.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.440+0000 7f5641ba2700 1 -- 172.21.15.27:0/1192504278 learned_addr learned my addr 172.21.15.27:0/1192504278 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:00.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.440+0000 7f5640ba0700 1 --2- 172.21.15.27:0/1192504278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f563c101800 0x7f563c1a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:00.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.441+0000 7f56413a1700 1 --2- 172.21.15.27:0/1192504278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f563c100e50 0x7f563c1a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:00.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.441+0000 7f5641ba2700 1 -- 172.21.15.27:0/1192504278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f563c101800 msgr2=0x7f563c1a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:00.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.441+0000 7f5641ba2700 1 --2- 172.21.15.27:0/1192504278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f563c101800 0x7f563c1a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:00.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.441+0000 7f5641ba2700 1 -- 172.21.15.27:0/1192504278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f563c100e50 msgr2=0x7f563c1a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:00.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.441+0000 7f5641ba2700 1 --2- 172.21.15.27:0/1192504278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f563c100e50 0x7f563c1a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:00.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.441+0000 7f5641ba2700 1 -- 172.21.15.27:0/1192504278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f563c19b180 con 0x7f563c107670 2024-04-25T00:14:00.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.441+0000 7f5641ba2700 1 --2- 172.21.15.27:0/1192504278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f563c107670 0x7f563c19aa20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f563400b970 tx=0x7f563400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:00.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.441+0000 7f56327fc700 1 -- 172.21.15.27:0/1192504278 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5634019070 con 0x7f563c107670 2024-04-25T00:14:00.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.441+0000 7f5643605700 1 -- 172.21.15.27:0/1192504278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f563c19b470 con 0x7f563c107670 2024-04-25T00:14:00.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.441+0000 7f56327fc700 1 -- 172.21.15.27:0/1192504278 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f563400ce10 con 0x7f563c107670 2024-04-25T00:14:00.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.441+0000 7f56327fc700 1 -- 172.21.15.27:0/1192504278 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5634007370 con 0x7f563c107670 2024-04-25T00:14:00.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.441+0000 7f5643605700 1 -- 172.21.15.27:0/1192504278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f563c1a77b0 con 0x7f563c107670 2024-04-25T00:14:00.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.442+0000 7f5643605700 1 -- 172.21.15.27:0/1192504278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f563c068f20 con 0x7f563c107670 2024-04-25T00:14:00.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.443+0000 7f56327fc700 1 -- 172.21.15.27:0/1192504278 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f563401c070 con 0x7f563c107670 2024-04-25T00:14:00.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.444+0000 7f56327fc700 1 --2- 172.21.15.27:0/1192504278 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f56280a2cb0 0x7f56280a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:00.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.444+0000 7f56413a1700 1 --2- 172.21.15.27:0/1192504278 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f56280a2cb0 0x7f56280a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:00.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.444+0000 7f56327fc700 1 -- 172.21.15.27:0/1192504278 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f56340ce680 con 0x7f563c107670 2024-04-25T00:14:00.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.445+0000 7f56413a1700 1 --2- 172.21.15.27:0/1192504278 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f56280a2cb0 0x7f56280a5170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5638000c00 tx=0x7f5638005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:00.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.450+0000 7f56327fc700 1 -- 172.21.15.27:0/1192504278 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f563409b350 con 0x7f563c107670 2024-04-25T00:14:00.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.654+0000 7f5643605700 1 -- 172.21.15.27:0/1192504278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f563c19bf50 con 0x7f563c107670 2024-04-25T00:14:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.656+0000 7f56327fc700 1 -- 172.21.15.27:0/1192504278 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24117 (secure 0 0 0) 0x7f563409ac50 con 0x7f563c107670 2024-04-25T00:14:00.658 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:14:00.658 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:13:23.935081+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:14:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.659+0000 7f5643605700 1 -- 172.21.15.27:0/1192504278 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f56280a2cb0 msgr2=0x7f56280a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.659+0000 7f5643605700 1 --2- 172.21.15.27:0/1192504278 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f56280a2cb0 0x7f56280a5170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5638000c00 tx=0x7f5638005fd0 comp rx=0 tx=0).stop 2024-04-25T00:14:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.659+0000 7f5643605700 1 -- 172.21.15.27:0/1192504278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f563c107670 msgr2=0x7f563c19aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.659+0000 7f5643605700 1 --2- 172.21.15.27:0/1192504278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f563c107670 0x7f563c19aa20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f563400b970 tx=0x7f563400bc80 comp rx=0 tx=0).stop 2024-04-25T00:14:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.659+0000 7f5643605700 1 -- 172.21.15.27:0/1192504278 shutdown_connections 2024-04-25T00:14:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.659+0000 7f5643605700 1 --2- 172.21.15.27:0/1192504278 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f56280a2cb0 0x7f56280a5170 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.659+0000 7f5643605700 1 --2- 172.21.15.27:0/1192504278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f563c107670 0x7f563c19aa20 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.659+0000 7f5643605700 1 --2- 172.21.15.27:0/1192504278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f563c100e50 0x7f563c1a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.659+0000 7f5643605700 1 --2- 172.21.15.27:0/1192504278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f563c101800 0x7f563c1a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.659+0000 7f5643605700 1 -- 172.21.15.27:0/1192504278 >> 172.21.15.27:0/1192504278 conn(0x7f563c076450 msgr2=0x7f563c0fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.659+0000 7f5643605700 1 -- 172.21.15.27:0/1192504278 shutdown_connections 2024-04-25T00:14:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:00.659+0000 7f5643605700 1 -- 172.21.15.27:0/1192504278 wait complete. 2024-04-25T00:14:01.313 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:01 smithi027 ceph-mon[29208]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 25 op/s 2024-04-25T00:14:01.314 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:01 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1192504278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:14:01.343 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-25T00:14:01.343 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-25T00:14:01.344 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-25T00:14:01.344 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-25T00:14:01.344 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-25T00:14:01.345 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-25T00:14:01.345 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-25T00:14:01.345 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-25T00:14:01.346 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-25T00:14:01.346 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-25T00:14:01.347 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-25T00:14:01.347 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-25T00:14:01.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:01 smithi139 ceph-mon[32833]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 25 op/s 2024-04-25T00:14:01.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:01 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1192504278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:14:01.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:01 smithi153 ceph-mon[32896]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 25 op/s 2024-04-25T00:14:01.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:01 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1192504278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:14:01.942 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:02.711 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:02.714 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:02.716 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:03.437 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:03.439 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:03.439 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:03.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:03 smithi027 ceph-mon[29208]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-25T00:14:03.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:03 smithi139 ceph-mon[32833]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-25T00:14:03.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:03 smithi153 ceph-mon[32896]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-25T00:14:03.881 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:04.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.515+0000 7f48516fa700 1 -- 172.21.15.27:0/1520411569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f484c1071a0 msgr2=0x7f484c107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:04.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.515+0000 7f48516fa700 1 --2- 172.21.15.27:0/1520411569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f484c1071a0 0x7f484c107580 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4844009990 tx=0x7f4844009ca0 comp rx=0 tx=0).stop 2024-04-25T00:14:04.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.515+0000 7f48516fa700 1 -- 172.21.15.27:0/1520411569 shutdown_connections 2024-04-25T00:14:04.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.515+0000 7f48516fa700 1 --2- 172.21.15.27:0/1520411569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f484c101af0 0x7f484c10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:04.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.515+0000 7f48516fa700 1 --2- 172.21.15.27:0/1520411569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f484c1071a0 0x7f484c107580 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:04.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.515+0000 7f48516fa700 1 --2- 172.21.15.27:0/1520411569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f484c101130 0x7f484c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:04.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.515+0000 7f48516fa700 1 -- 172.21.15.27:0/1520411569 >> 172.21.15.27:0/1520411569 conn(0x7f484c0780e0 msgr2=0x7f484c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:04.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.515+0000 7f48516fa700 1 -- 172.21.15.27:0/1520411569 shutdown_connections 2024-04-25T00:14:04.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.515+0000 7f48516fa700 1 -- 172.21.15.27:0/1520411569 wait complete. 2024-04-25T00:14:04.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.515+0000 7f48516fa700 1 Processor -- start 2024-04-25T00:14:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.516+0000 7f48516fa700 1 -- start start 2024-04-25T00:14:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.516+0000 7f48516fa700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f484c101130 0x7f484c199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.516+0000 7f48516fa700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f484c101af0 0x7f484c199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.516+0000 7f48516fa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f484c1071a0 0x7f484c1a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.516+0000 7f48516fa700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f484c111440 con 0x7f484c101130 2024-04-25T00:14:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.516+0000 7f48516fa700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f484c1112c0 con 0x7f484c101af0 2024-04-25T00:14:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.516+0000 7f48516fa700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f484c1115c0 con 0x7f484c1071a0 2024-04-25T00:14:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.516+0000 7f484affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f484c101130 0x7f484c199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.516+0000 7f484affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f484c101130 0x7f484c199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54772/0 (socket says 172.21.15.27:54772) 2024-04-25T00:14:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.516+0000 7f484affd700 1 -- 172.21.15.27:0/373799036 learned_addr learned my addr 172.21.15.27:0/373799036 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.516+0000 7f484a7fc700 1 --2- 172.21.15.27:0/373799036 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f484c101af0 0x7f484c199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.516+0000 7f484b7fe700 1 --2- 172.21.15.27:0/373799036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f484c1071a0 0x7f484c1a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.516+0000 7f484affd700 1 -- 172.21.15.27:0/373799036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f484c1071a0 msgr2=0x7f484c1a4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.516+0000 7f484affd700 1 --2- 172.21.15.27:0/373799036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f484c1071a0 0x7f484c1a4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.516+0000 7f484affd700 1 -- 172.21.15.27:0/373799036 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f484c101af0 msgr2=0x7f484c199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.516+0000 7f484affd700 1 --2- 172.21.15.27:0/373799036 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f484c101af0 0x7f484c199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.516+0000 7f484affd700 1 -- 172.21.15.27:0/373799036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f484c1a4dc0 con 0x7f484c101130 2024-04-25T00:14:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.517+0000 7f484affd700 1 --2- 172.21.15.27:0/373799036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f484c101130 0x7f484c199800 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4844000c00 tx=0x7f48440047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.517+0000 7f4843fff700 1 -- 172.21.15.27:0/373799036 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4844027070 con 0x7f484c101130 2024-04-25T00:14:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.517+0000 7f4843fff700 1 -- 172.21.15.27:0/373799036 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4844003e40 con 0x7f484c101130 2024-04-25T00:14:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.517+0000 7f48516fa700 1 -- 172.21.15.27:0/373799036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f484c1a4f90 con 0x7f484c101130 2024-04-25T00:14:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.517+0000 7f4843fff700 1 -- 172.21.15.27:0/373799036 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4844022d10 con 0x7f484c101130 2024-04-25T00:14:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.517+0000 7f48516fa700 1 -- 172.21.15.27:0/373799036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f484c1a5380 con 0x7f484c101130 2024-04-25T00:14:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.518+0000 7f48516fa700 1 -- 172.21.15.27:0/373799036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f484c0628c0 con 0x7f484c101130 2024-04-25T00:14:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.519+0000 7f4843fff700 1 -- 172.21.15.27:0/373799036 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f484402a070 con 0x7f484c101130 2024-04-25T00:14:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.519+0000 7f4843fff700 1 --2- 172.21.15.27:0/373799036 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f48380a2cb0 0x7f48380a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.520+0000 7f484a7fc700 1 --2- 172.21.15.27:0/373799036 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f48380a2cb0 0x7f48380a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.520+0000 7f4843fff700 1 -- 172.21.15.27:0/373799036 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f48440d0bc0 con 0x7f484c101130 2024-04-25T00:14:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.520+0000 7f4843fff700 1 --2- 172.21.15.27:0/373799036 >> [v2:172.21.15.139:6800/1346503461,v1:172.21.15.139:6801/1346503461] conn(0x7f48380ad970 0x7f48380afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.520+0000 7f4843fff700 1 -- 172.21.15.27:0/373799036 --> [v2:172.21.15.139:6800/1346503461,v1:172.21.15.139:6801/1346503461] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4844010ec0 con 0x7f48380ad970 2024-04-25T00:14:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.520+0000 7f4843fff700 1 -- 172.21.15.27:0/373799036 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f48440d1010 con 0x7f484c101130 2024-04-25T00:14:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.520+0000 7f484a7fc700 1 --2- 172.21.15.27:0/373799036 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f48380a2cb0 0x7f48380a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f484c0786c0 tx=0x7f483400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.520+0000 7f484b7fe700 1 --2- 172.21.15.27:0/373799036 >> [v2:172.21.15.139:6800/1346503461,v1:172.21.15.139:6801/1346503461] conn(0x7f48380ad970 0x7f48380afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.521+0000 7f484b7fe700 1 --2- 172.21.15.27:0/373799036 >> [v2:172.21.15.139:6800/1346503461,v1:172.21.15.139:6801/1346503461] conn(0x7f48380ad970 0x7f48380afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.522+0000 7f4843fff700 1 -- 172.21.15.27:0/373799036 <== osd.1 v2:172.21.15.139:6800/1346503461 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4844010ec0 con 0x7f48380ad970 2024-04-25T00:14:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.538+0000 7f48516fa700 1 -- 172.21.15.27:0/373799036 --> [v2:172.21.15.139:6800/1346503461,v1:172.21.15.139:6801/1346503461] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f484c062d40 con 0x7f48380ad970 2024-04-25T00:14:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.538+0000 7f4843fff700 1 -- 172.21.15.27:0/373799036 <== osd.1 v2:172.21.15.139:6800/1346503461 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f484c062d40 con 0x7f48380ad970 2024-04-25T00:14:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.539+0000 7f48516fa700 1 -- 172.21.15.27:0/373799036 >> [v2:172.21.15.139:6800/1346503461,v1:172.21.15.139:6801/1346503461] conn(0x7f48380ad970 msgr2=0x7f48380afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.539+0000 7f48516fa700 1 --2- 172.21.15.27:0/373799036 >> [v2:172.21.15.139:6800/1346503461,v1:172.21.15.139:6801/1346503461] conn(0x7f48380ad970 0x7f48380afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.539+0000 7f48516fa700 1 -- 172.21.15.27:0/373799036 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f48380a2cb0 msgr2=0x7f48380a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.539+0000 7f48516fa700 1 --2- 172.21.15.27:0/373799036 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f48380a2cb0 0x7f48380a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f484c0786c0 tx=0x7f483400b410 comp rx=0 tx=0).stop 2024-04-25T00:14:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.539+0000 7f48516fa700 1 -- 172.21.15.27:0/373799036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f484c101130 msgr2=0x7f484c199800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.539+0000 7f48516fa700 1 --2- 172.21.15.27:0/373799036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f484c101130 0x7f484c199800 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4844000c00 tx=0x7f48440047b0 comp rx=0 tx=0).stop 2024-04-25T00:14:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.539+0000 7f484b7fe700 1 -- 172.21.15.27:0/373799036 reap_dead start 2024-04-25T00:14:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.539+0000 7f48516fa700 1 -- 172.21.15.27:0/373799036 shutdown_connections 2024-04-25T00:14:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.539+0000 7f48516fa700 1 -- 172.21.15.27:0/373799036 >> 172.21.15.27:0/373799036 conn(0x7f484c0780e0 msgr2=0x7f484c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.539+0000 7f48516fa700 1 -- 172.21.15.27:0/373799036 shutdown_connections 2024-04-25T00:14:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:04.539+0000 7f48516fa700 1 -- 172.21.15.27:0/373799036 wait complete. 2024-04-25T00:14:05.324 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:05 smithi027 ceph-mon[29208]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-04-25T00:14:05.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:05 smithi139 ceph-mon[32833]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-04-25T00:14:05.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:05 smithi153 ceph-mon[32896]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-04-25T00:14:07.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:07 smithi139 ceph-mon[32833]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-25T00:14:07.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:07 smithi153 ceph-mon[32896]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-25T00:14:07.626 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:07 smithi027 ceph-mon[29208]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-25T00:14:07.654 INFO:teuthology.orchestra.run.smithi027.stdout:55834574922 2024-04-25T00:14:07.655 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-25T00:14:09.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:09 smithi139 ceph-mon[32833]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-04-25T00:14:09.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:09 smithi153 ceph-mon[32896]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-04-25T00:14:09.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:09 smithi027 ceph-mon[29208]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-04-25T00:14:10.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-25T00:14:10.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-25T00:14:10.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-25T00:14:10.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-04-25T00:14:10.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-04-25T00:14:10.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-25T00:14:10.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2024-04-25T00:14:10.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-04-25T00:14:10.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-04-25T00:14:10.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]: dispatch 2024-04-25T00:14:10.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-25T00:14:10.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-25T00:14:10.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-25T00:14:10.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-04-25T00:14:10.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-04-25T00:14:10.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-25T00:14:10.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2024-04-25T00:14:10.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-04-25T00:14:10.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-04-25T00:14:10.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]: dispatch 2024-04-25T00:14:10.584 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-25T00:14:10.584 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-25T00:14:10.584 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-25T00:14:10.584 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-04-25T00:14:10.584 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-04-25T00:14:10.584 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-25T00:14:10.584 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2024-04-25T00:14:10.584 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-04-25T00:14:10.584 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-04-25T00:14:10.585 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]: dispatch 2024-04-25T00:14:10.719 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:10.719 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:10.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.931+0000 7f9083299700 1 -- 172.21.15.27:0/230527207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c101ae0 msgr2=0x7f907c10dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:10.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.931+0000 7f9083299700 1 --2- 172.21.15.27:0/230527207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c101ae0 0x7f907c10dfd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f907400b0a0 tx=0x7f907400b3b0 comp rx=0 tx=0).stop 2024-04-25T00:14:10.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.931+0000 7f9083299700 1 -- 172.21.15.27:0/230527207 shutdown_connections 2024-04-25T00:14:10.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.931+0000 7f9083299700 1 --2- 172.21.15.27:0/230527207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c101ae0 0x7f907c10dfd0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:10.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.931+0000 7f9083299700 1 --2- 172.21.15.27:0/230527207 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f907c107190 0x7f907c107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:10.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.931+0000 7f9083299700 1 --2- 172.21.15.27:0/230527207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f907c101120 0x7f907c1015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:10.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.931+0000 7f9083299700 1 -- 172.21.15.27:0/230527207 >> 172.21.15.27:0/230527207 conn(0x7f907c0780a0 msgr2=0x7f907c0784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:10.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.931+0000 7f9083299700 1 -- 172.21.15.27:0/230527207 shutdown_connections 2024-04-25T00:14:10.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.931+0000 7f9083299700 1 -- 172.21.15.27:0/230527207 wait complete. 2024-04-25T00:14:10.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9083299700 1 Processor -- start 2024-04-25T00:14:10.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9083299700 1 -- start start 2024-04-25T00:14:10.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9083299700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f907c101120 0x7f907c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:10.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9083299700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c101ae0 0x7f907c19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:10.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9083299700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f907c107190 0x7f907c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:10.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9083299700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f907c112520 con 0x7f907c101ae0 2024-04-25T00:14:10.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9083299700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f907c1123a0 con 0x7f907c101120 2024-04-25T00:14:10.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9083299700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f907c1126a0 con 0x7f907c107190 2024-04-25T00:14:10.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9080834700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c101ae0 0x7f907c19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:10.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9080834700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c101ae0 0x7f907c19ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47064/0 (socket says 172.21.15.27:47064) 2024-04-25T00:14:10.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9080834700 1 -- 172.21.15.27:0/3740765148 learned_addr learned my addr 172.21.15.27:0/3740765148 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:10.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9081035700 1 --2- 172.21.15.27:0/3740765148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f907c101120 0x7f907c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9081836700 1 --2- 172.21.15.27:0/3740765148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f907c107190 0x7f907c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9080834700 1 -- 172.21.15.27:0/3740765148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f907c107190 msgr2=0x7f907c198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9080834700 1 --2- 172.21.15.27:0/3740765148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f907c107190 0x7f907c198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9080834700 1 -- 172.21.15.27:0/3740765148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f907c101120 msgr2=0x7f907c19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9080834700 1 --2- 172.21.15.27:0/3740765148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f907c101120 0x7f907c19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9080834700 1 -- 172.21.15.27:0/3740765148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f907c199040 con 0x7f907c101ae0 2024-04-25T00:14:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.932+0000 7f9081836700 1 --2- 172.21.15.27:0/3740765148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f907c107190 0x7f907c198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:14:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.933+0000 7f9080834700 1 --2- 172.21.15.27:0/3740765148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c101ae0 0x7f907c19ebd0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f906c00c920 tx=0x7f906c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.933+0000 7f90727fc700 1 -- 172.21.15.27:0/3740765148 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906c007b00 con 0x7f907c101ae0 2024-04-25T00:14:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.933+0000 7f90727fc700 1 -- 172.21.15.27:0/3740765148 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f906c004d10 con 0x7f907c101ae0 2024-04-25T00:14:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.933+0000 7f9083299700 1 -- 172.21.15.27:0/3740765148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f907c199330 con 0x7f907c101ae0 2024-04-25T00:14:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.933+0000 7f90727fc700 1 -- 172.21.15.27:0/3740765148 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906c005680 con 0x7f907c101ae0 2024-04-25T00:14:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.933+0000 7f9083299700 1 -- 172.21.15.27:0/3740765148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f907c1a5610 con 0x7f907c101ae0 2024-04-25T00:14:10.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.934+0000 7f9083299700 1 -- 172.21.15.27:0/3740765148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f907c100430 con 0x7f907c101ae0 2024-04-25T00:14:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.935+0000 7f90727fc700 1 -- 172.21.15.27:0/3740765148 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f906c004750 con 0x7f907c101ae0 2024-04-25T00:14:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.935+0000 7f90727fc700 1 --2- 172.21.15.27:0/3740765148 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f90680a2cb0 0x7f90680a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.936+0000 7f9081035700 1 --2- 172.21.15.27:0/3740765148 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f90680a2cb0 0x7f90680a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.936+0000 7f90727fc700 1 -- 172.21.15.27:0/3740765148 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f906c0cf4f0 con 0x7f907c101ae0 2024-04-25T00:14:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.936+0000 7f90727fc700 1 --2- 172.21.15.27:0/3740765148 >> [v2:172.21.15.139:6808/856291222,v1:172.21.15.139:6809/856291222] conn(0x7f90680ade10 0x7f90680b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.936+0000 7f90727fc700 1 -- 172.21.15.27:0/3740765148 --> [v2:172.21.15.139:6808/856291222,v1:172.21.15.139:6809/856291222] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f90680b0980 con 0x7f90680ade10 2024-04-25T00:14:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.936+0000 7f90727fc700 1 -- 172.21.15.27:0/3740765148 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f906c0cf940 con 0x7f907c101ae0 2024-04-25T00:14:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.936+0000 7f9081035700 1 --2- 172.21.15.27:0/3740765148 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f90680a2cb0 0x7f90680a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f907800abd0 tx=0x7f907800a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.936+0000 7f9081836700 1 --2- 172.21.15.27:0/3740765148 >> [v2:172.21.15.139:6808/856291222,v1:172.21.15.139:6809/856291222] conn(0x7f90680ade10 0x7f90680b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.937+0000 7f9081836700 1 --2- 172.21.15.27:0/3740765148 >> [v2:172.21.15.139:6808/856291222,v1:172.21.15.139:6809/856291222] conn(0x7f90680ade10 0x7f90680b02b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:10.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.938+0000 7f90727fc700 1 -- 172.21.15.27:0/3740765148 <== osd.4 v2:172.21.15.139:6808/856291222 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f90680b0980 con 0x7f90680ade10 2024-04-25T00:14:10.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.954+0000 7f9083299700 1 -- 172.21.15.27:0/3740765148 --> [v2:172.21.15.139:6808/856291222,v1:172.21.15.139:6809/856291222] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f907c102840 con 0x7f90680ade10 2024-04-25T00:14:10.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.955+0000 7f90727fc700 1 -- 172.21.15.27:0/3740765148 <== osd.4 v2:172.21.15.139:6808/856291222 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f907c102840 con 0x7f90680ade10 2024-04-25T00:14:10.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.955+0000 7f9083299700 1 -- 172.21.15.27:0/3740765148 >> [v2:172.21.15.139:6808/856291222,v1:172.21.15.139:6809/856291222] conn(0x7f90680ade10 msgr2=0x7f90680b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:10.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.955+0000 7f9083299700 1 --2- 172.21.15.27:0/3740765148 >> [v2:172.21.15.139:6808/856291222,v1:172.21.15.139:6809/856291222] conn(0x7f90680ade10 0x7f90680b02b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:10.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.955+0000 7f9083299700 1 -- 172.21.15.27:0/3740765148 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f90680a2cb0 msgr2=0x7f90680a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:10.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.955+0000 7f9083299700 1 --2- 172.21.15.27:0/3740765148 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f90680a2cb0 0x7f90680a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f907800abd0 tx=0x7f907800a380 comp rx=0 tx=0).stop 2024-04-25T00:14:10.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.955+0000 7f9083299700 1 -- 172.21.15.27:0/3740765148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c101ae0 msgr2=0x7f907c19ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:10.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.955+0000 7f9083299700 1 --2- 172.21.15.27:0/3740765148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c101ae0 0x7f907c19ebd0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f906c00c920 tx=0x7f906c00cc30 comp rx=0 tx=0).stop 2024-04-25T00:14:10.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.955+0000 7f9081836700 1 -- 172.21.15.27:0/3740765148 reap_dead start 2024-04-25T00:14:10.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.955+0000 7f9083299700 1 -- 172.21.15.27:0/3740765148 shutdown_connections 2024-04-25T00:14:10.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.955+0000 7f9083299700 1 -- 172.21.15.27:0/3740765148 >> 172.21.15.27:0/3740765148 conn(0x7f907c0780a0 msgr2=0x7f907c0ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:10.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.955+0000 7f9083299700 1 -- 172.21.15.27:0/3740765148 shutdown_connections 2024-04-25T00:14:10.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:10.955+0000 7f9083299700 1 -- 172.21.15.27:0/3740765148 wait complete. 2024-04-25T00:14:11.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.254+0000 7f9c43be2700 1 -- 172.21.15.27:0/3353092704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c3c100c60 msgr2=0x7f9c3c1010e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:11.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.254+0000 7f9c43be2700 1 --2- 172.21.15.27:0/3353092704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c3c100c60 0x7f9c3c1010e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c009990 tx=0x7f9c2c009ca0 comp rx=0 tx=0).stop 2024-04-25T00:14:11.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.254+0000 7f9c43be2700 1 -- 172.21.15.27:0/3353092704 shutdown_connections 2024-04-25T00:14:11.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.254+0000 7f9c43be2700 1 --2- 172.21.15.27:0/3353092704 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c3c101620 0x7f9c3c10f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:11.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.254+0000 7f9c43be2700 1 --2- 172.21.15.27:0/3353092704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c3c108410 0x7f9c3c1087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:11.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.254+0000 7f9c43be2700 1 --2- 172.21.15.27:0/3353092704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c3c100c60 0x7f9c3c1010e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:11.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.254+0000 7f9c43be2700 1 -- 172.21.15.27:0/3353092704 >> 172.21.15.27:0/3353092704 conn(0x7f9c3c0763a0 msgr2=0x7f9c3c0767b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:11.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.254+0000 7f9c43be2700 1 -- 172.21.15.27:0/3353092704 shutdown_connections 2024-04-25T00:14:11.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.254+0000 7f9c43be2700 1 -- 172.21.15.27:0/3353092704 wait complete. 2024-04-25T00:14:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.254+0000 7f9c43be2700 1 Processor -- start 2024-04-25T00:14:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.254+0000 7f9c43be2700 1 -- start start 2024-04-25T00:14:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.255+0000 7f9c43be2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c3c100c60 0x7f9c3c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.255+0000 7f9c43be2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c3c101620 0x7f9c3c19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.255+0000 7f9c43be2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c3c108410 0x7f9c3c1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.255+0000 7f9c43be2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c112530 con 0x7f9c3c108410 2024-04-25T00:14:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.255+0000 7f9c43be2700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c1123b0 con 0x7f9c3c101620 2024-04-25T00:14:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.255+0000 7f9c43be2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c1126b0 con 0x7f9c3c100c60 2024-04-25T00:14:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.255+0000 7f9c4217f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c3c108410 0x7f9c3c1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.255+0000 7f9c4217f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c3c108410 0x7f9c3c1987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47106/0 (socket says 172.21.15.27:47106) 2024-04-25T00:14:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.255+0000 7f9c4217f700 1 -- 172.21.15.27:0/1907977621 learned_addr learned my addr 172.21.15.27:0/1907977621 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.255+0000 7f9c4197e700 1 --2- 172.21.15.27:0/1907977621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c3c100c60 0x7f9c3c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.255+0000 7f9c4117d700 1 --2- 172.21.15.27:0/1907977621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c3c101620 0x7f9c3c19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.255+0000 7f9c4217f700 1 -- 172.21.15.27:0/1907977621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c3c100c60 msgr2=0x7f9c3c19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:11.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.255+0000 7f9c4217f700 1 --2- 172.21.15.27:0/1907977621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c3c100c60 0x7f9c3c19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:11.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.255+0000 7f9c4217f700 1 -- 172.21.15.27:0/1907977621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c3c101620 msgr2=0x7f9c3c19ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:11.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.255+0000 7f9c4217f700 1 --2- 172.21.15.27:0/1907977621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c3c101620 0x7f9c3c19ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:11.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.255+0000 7f9c4217f700 1 -- 172.21.15.27:0/1907977621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3c199090 con 0x7f9c3c108410 2024-04-25T00:14:11.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.255+0000 7f9c4217f700 1 --2- 172.21.15.27:0/1907977621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c3c108410 0x7f9c3c1987b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400ef30 tx=0x7f9c3400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:11.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.262+0000 7f9c32ffd700 1 -- 172.21.15.27:0/1907977621 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c340092e0 con 0x7f9c3c108410 2024-04-25T00:14:11.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.262+0000 7f9c32ffd700 1 -- 172.21.15.27:0/1907977621 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c34004bb0 con 0x7f9c3c108410 2024-04-25T00:14:11.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.262+0000 7f9c32ffd700 1 -- 172.21.15.27:0/1907977621 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c34004280 con 0x7f9c3c108410 2024-04-25T00:14:11.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.263+0000 7f9c43be2700 1 -- 172.21.15.27:0/1907977621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c3c199320 con 0x7f9c3c108410 2024-04-25T00:14:11.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.263+0000 7f9c43be2700 1 -- 172.21.15.27:0/1907977621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c3c1a5440 con 0x7f9c3c108410 2024-04-25T00:14:11.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.267+0000 7f9c43be2700 1 -- 172.21.15.27:0/1907977621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9c3c0629c0 con 0x7f9c3c108410 2024-04-25T00:14:11.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.267+0000 7f9c32ffd700 1 -- 172.21.15.27:0/1907977621 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9c3401b030 con 0x7f9c3c108410 2024-04-25T00:14:11.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.268+0000 7f9c32ffd700 1 --2- 172.21.15.27:0/1907977621 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9c280a2d30 0x7f9c280a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:11.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.268+0000 7f9c32ffd700 1 -- 172.21.15.27:0/1907977621 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9c340cef90 con 0x7f9c3c108410 2024-04-25T00:14:11.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.269+0000 7f9c32ffd700 1 --2- 172.21.15.27:0/1907977621 >> [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528] conn(0x7f9c280ade90 0x7f9c280b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:11.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.269+0000 7f9c32ffd700 1 -- 172.21.15.27:0/1907977621 --> [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9c280b0a00 con 0x7f9c280ade90 2024-04-25T00:14:11.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.269+0000 7f9c32ffd700 1 -- 172.21.15.27:0/1907977621 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f9c340d15f0 con 0x7f9c3c108410 2024-04-25T00:14:11.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.269+0000 7f9c4117d700 1 --2- 172.21.15.27:0/1907977621 >> [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528] conn(0x7f9c280ade90 0x7f9c280b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:11.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.269+0000 7f9c4197e700 1 --2- 172.21.15.27:0/1907977621 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9c280a2d30 0x7f9c280a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:11.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.270+0000 7f9c4117d700 1 --2- 172.21.15.27:0/1907977621 >> [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528] conn(0x7f9c280ade90 0x7f9c280b0330 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:11.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.271+0000 7f9c4197e700 1 --2- 172.21.15.27:0/1907977621 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9c280a2d30 0x7f9c280a51f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9c38009c60 tx=0x7f9c38009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:11.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.271+0000 7f9c32ffd700 1 -- 172.21.15.27:0/1907977621 <== osd.0 v2:172.21.15.27:6802/238818528 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9c280b0a00 con 0x7f9c280ade90 2024-04-25T00:14:11.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.288+0000 7f9c43be2700 1 -- 172.21.15.27:0/1907977621 --> [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9c3c062e40 con 0x7f9c280ade90 2024-04-25T00:14:11.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.288+0000 7f9c32ffd700 1 -- 172.21.15.27:0/1907977621 <== osd.0 v2:172.21.15.27:6802/238818528 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9c3c062e40 con 0x7f9c280ade90 2024-04-25T00:14:11.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.289+0000 7f9c43be2700 1 -- 172.21.15.27:0/1907977621 >> [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528] conn(0x7f9c280ade90 msgr2=0x7f9c280b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:11.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.289+0000 7f9c43be2700 1 --2- 172.21.15.27:0/1907977621 >> [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528] conn(0x7f9c280ade90 0x7f9c280b0330 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:11.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.289+0000 7f9c43be2700 1 -- 172.21.15.27:0/1907977621 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9c280a2d30 msgr2=0x7f9c280a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:11.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.289+0000 7f9c43be2700 1 --2- 172.21.15.27:0/1907977621 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9c280a2d30 0x7f9c280a51f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9c38009c60 tx=0x7f9c38009260 comp rx=0 tx=0).stop 2024-04-25T00:14:11.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.289+0000 7f9c43be2700 1 -- 172.21.15.27:0/1907977621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c3c108410 msgr2=0x7f9c3c1987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:11.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.289+0000 7f9c43be2700 1 --2- 172.21.15.27:0/1907977621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c3c108410 0x7f9c3c1987b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400ef30 tx=0x7f9c3400c3a0 comp rx=0 tx=0).stop 2024-04-25T00:14:11.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.289+0000 7f9c4217f700 1 -- 172.21.15.27:0/1907977621 reap_dead start 2024-04-25T00:14:11.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.289+0000 7f9c43be2700 1 -- 172.21.15.27:0/1907977621 shutdown_connections 2024-04-25T00:14:11.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.289+0000 7f9c43be2700 1 -- 172.21.15.27:0/1907977621 >> 172.21.15.27:0/1907977621 conn(0x7f9c3c0763a0 msgr2=0x7f9c3c0ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:11.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.289+0000 7f9c43be2700 1 -- 172.21.15.27:0/1907977621 shutdown_connections 2024-04-25T00:14:11.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.289+0000 7f9c43be2700 1 -- 172.21.15.27:0/1907977621 wait complete. 2024-04-25T00:14:11.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.139+0000 7f62184b8700 1 -- 172.21.15.27:0/3478969170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6210101af0 msgr2=0x7f621010dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:11.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.139+0000 7f62184b8700 1 --2- 172.21.15.27:0/3478969170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6210101af0 0x7f621010dfe0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f620800b0a0 tx=0x7f620800b3b0 comp rx=0 tx=0).stop 2024-04-25T00:14:11.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.139+0000 7f62184b8700 1 -- 172.21.15.27:0/3478969170 shutdown_connections 2024-04-25T00:14:11.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.139+0000 7f62184b8700 1 --2- 172.21.15.27:0/3478969170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6210101af0 0x7f621010dfe0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:11.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.139+0000 7f62184b8700 1 --2- 172.21.15.27:0/3478969170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62101071a0 0x7f6210107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:11.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.139+0000 7f62184b8700 1 --2- 172.21.15.27:0/3478969170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6210101130 0x7f62101015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:11.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.139+0000 7f62184b8700 1 -- 172.21.15.27:0/3478969170 >> 172.21.15.27:0/3478969170 conn(0x7f62100780e0 msgr2=0x7f62100784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:11.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.139+0000 7f62184b8700 1 -- 172.21.15.27:0/3478969170 shutdown_connections 2024-04-25T00:14:11.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.139+0000 7f62184b8700 1 -- 172.21.15.27:0/3478969170 wait complete. 2024-04-25T00:14:11.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.140+0000 7f62184b8700 1 Processor -- start 2024-04-25T00:14:11.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.140+0000 7f62184b8700 1 -- start start 2024-04-25T00:14:11.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.140+0000 7f62184b8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6210101130 0x7f621019d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:11.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.140+0000 7f62184b8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6210101af0 0x7f621019db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:11.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.140+0000 7f62184b8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62101071a0 0x7f62101976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:11.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.140+0000 7f62184b8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6210111430 con 0x7f62101071a0 2024-04-25T00:14:11.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.140+0000 7f62184b8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f62101112b0 con 0x7f6210101af0 2024-04-25T00:14:11.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.140+0000 7f62184b8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f62101115b0 con 0x7f6210101130 2024-04-25T00:14:11.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.141+0000 7f6216254700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6210101130 0x7f621019d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:11.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.141+0000 7f6216a55700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62101071a0 0x7f62101976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:11.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.141+0000 7f6216254700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6210101130 0x7f621019d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:57974/0 (socket says 172.21.15.27:57974) 2024-04-25T00:14:11.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.141+0000 7f6216254700 1 -- 172.21.15.27:0/2872520003 learned_addr learned my addr 172.21.15.27:0/2872520003 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:11.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.141+0000 7f6215a53700 1 --2- 172.21.15.27:0/2872520003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6210101af0 0x7f621019db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:11.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.141+0000 7f6216a55700 1 -- 172.21.15.27:0/2872520003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6210101130 msgr2=0x7f621019d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:11.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.141+0000 7f6216a55700 1 --2- 172.21.15.27:0/2872520003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6210101130 0x7f621019d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:11.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.141+0000 7f6216a55700 1 -- 172.21.15.27:0/2872520003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6210101af0 msgr2=0x7f621019db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:11.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.141+0000 7f6216a55700 1 --2- 172.21.15.27:0/2872520003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6210101af0 0x7f621019db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:11.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.141+0000 7f6216a55700 1 -- 172.21.15.27:0/2872520003 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6210197fd0 con 0x7f62101071a0 2024-04-25T00:14:11.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.141+0000 7f6215a53700 1 --2- 172.21.15.27:0/2872520003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6210101af0 0x7f621019db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:14:11.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.141+0000 7f6216a55700 1 --2- 172.21.15.27:0/2872520003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62101071a0 0x7f62101976f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f620800d010 tx=0x7f6208004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:11.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.141+0000 7f62077fe700 1 -- 172.21.15.27:0/2872520003 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6208021070 con 0x7f62101071a0 2024-04-25T00:14:11.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.141+0000 7f62077fe700 1 -- 172.21.15.27:0/2872520003 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6208007c00 con 0x7f62101071a0 2024-04-25T00:14:11.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.141+0000 7f62184b8700 1 -- 172.21.15.27:0/2872520003 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6210198260 con 0x7f62101071a0 2024-04-25T00:14:11.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.141+0000 7f62077fe700 1 -- 172.21.15.27:0/2872520003 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62080052a0 con 0x7f62101071a0 2024-04-25T00:14:11.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.141+0000 7f62184b8700 1 -- 172.21.15.27:0/2872520003 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62101a4550 con 0x7f62101071a0 2024-04-25T00:14:11.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.142+0000 7f62184b8700 1 -- 172.21.15.27:0/2872520003 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6210062980 con 0x7f62101071a0 2024-04-25T00:14:11.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.144+0000 7f62077fe700 1 -- 172.21.15.27:0/2872520003 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6208024070 con 0x7f62101071a0 2024-04-25T00:14:11.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.144+0000 7f62077fe700 1 --2- 172.21.15.27:0/2872520003 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f61fc0a2cb0 0x7f61fc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:11.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.144+0000 7f6216254700 1 --2- 172.21.15.27:0/2872520003 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f61fc0a2cb0 0x7f61fc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:11.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.144+0000 7f62077fe700 1 -- 172.21.15.27:0/2872520003 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f620800f020 con 0x7f62101071a0 2024-04-25T00:14:11.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.144+0000 7f62077fe700 1 --2- 172.21.15.27:0/2872520003 >> [v2:172.21.15.139:6824/3154318186,v1:172.21.15.139:6825/3154318186] conn(0x7f61fc0ade10 0x7f61fc0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:11.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.145+0000 7f62077fe700 1 -- 172.21.15.27:0/2872520003 --> [v2:172.21.15.139:6824/3154318186,v1:172.21.15.139:6825/3154318186] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6208026e50 con 0x7f61fc0ade10 2024-04-25T00:14:11.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.145+0000 7f62077fe700 1 -- 172.21.15.27:0/2872520003 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f62080d1370 con 0x7f62101071a0 2024-04-25T00:14:11.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.145+0000 7f6216254700 1 --2- 172.21.15.27:0/2872520003 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f61fc0a2cb0 0x7f61fc0a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f620c00bcb0 tx=0x7f620c00b480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:11.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.145+0000 7f6215a53700 1 --2- 172.21.15.27:0/2872520003 >> [v2:172.21.15.139:6824/3154318186,v1:172.21.15.139:6825/3154318186] conn(0x7f61fc0ade10 0x7f61fc0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:11.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.145+0000 7f6215a53700 1 --2- 172.21.15.27:0/2872520003 >> [v2:172.21.15.139:6824/3154318186,v1:172.21.15.139:6825/3154318186] conn(0x7f61fc0ade10 0x7f61fc0b02b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:11.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.146+0000 7f62077fe700 1 -- 172.21.15.27:0/2872520003 <== osd.10 v2:172.21.15.139:6824/3154318186 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6208026e50 con 0x7f61fc0ade10 2024-04-25T00:14:11.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.163+0000 7f62184b8700 1 -- 172.21.15.27:0/2872520003 --> [v2:172.21.15.139:6824/3154318186,v1:172.21.15.139:6825/3154318186] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6210062d80 con 0x7f61fc0ade10 2024-04-25T00:14:11.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.163+0000 7f62077fe700 1 -- 172.21.15.27:0/2872520003 <== osd.10 v2:172.21.15.139:6824/3154318186 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6210062d80 con 0x7f61fc0ade10 2024-04-25T00:14:11.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.164+0000 7f62184b8700 1 -- 172.21.15.27:0/2872520003 >> [v2:172.21.15.139:6824/3154318186,v1:172.21.15.139:6825/3154318186] conn(0x7f61fc0ade10 msgr2=0x7f61fc0b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:11.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.164+0000 7f62184b8700 1 --2- 172.21.15.27:0/2872520003 >> [v2:172.21.15.139:6824/3154318186,v1:172.21.15.139:6825/3154318186] conn(0x7f61fc0ade10 0x7f61fc0b02b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:11.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.164+0000 7f62184b8700 1 -- 172.21.15.27:0/2872520003 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f61fc0a2cb0 msgr2=0x7f61fc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:11.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.164+0000 7f62184b8700 1 --2- 172.21.15.27:0/2872520003 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f61fc0a2cb0 0x7f61fc0a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f620c00bcb0 tx=0x7f620c00b480 comp rx=0 tx=0).stop 2024-04-25T00:14:11.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.164+0000 7f62184b8700 1 -- 172.21.15.27:0/2872520003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62101071a0 msgr2=0x7f62101976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:11.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.164+0000 7f62184b8700 1 --2- 172.21.15.27:0/2872520003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62101071a0 0x7f62101976f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f620800d010 tx=0x7f6208004d50 comp rx=0 tx=0).stop 2024-04-25T00:14:11.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.164+0000 7f6216a55700 1 -- 172.21.15.27:0/2872520003 reap_dead start 2024-04-25T00:14:11.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.164+0000 7f62184b8700 1 -- 172.21.15.27:0/2872520003 shutdown_connections 2024-04-25T00:14:11.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.164+0000 7f62184b8700 1 -- 172.21.15.27:0/2872520003 >> 172.21.15.27:0/2872520003 conn(0x7f62100780e0 msgr2=0x7f62100ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:11.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.164+0000 7f62184b8700 1 -- 172.21.15.27:0/2872520003 shutdown_connections 2024-04-25T00:14:11.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:11.164+0000 7f62184b8700 1 -- 172.21.15.27:0/2872520003 wait complete. 2024-04-25T00:14:11.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:11 smithi027 ceph-mon[29208]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-25T00:14:11.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-25T00:14:11.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-25T00:14:11.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-25T00:14:11.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-04-25T00:14:11.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-04-25T00:14:11.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-25T00:14:11.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2024-04-25T00:14:11.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-04-25T00:14:11.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-04-25T00:14:11.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]': finished 2024-04-25T00:14:11.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:11 smithi027 ceph-mon[29208]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T00:14:11.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:11 smithi139 ceph-mon[32833]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-25T00:14:11.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-25T00:14:11.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-25T00:14:11.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-25T00:14:11.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-04-25T00:14:11.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-04-25T00:14:11.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-25T00:14:11.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2024-04-25T00:14:11.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-04-25T00:14:11.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-04-25T00:14:11.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]': finished 2024-04-25T00:14:11.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:11 smithi139 ceph-mon[32833]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T00:14:11.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:11 smithi153 ceph-mon[32896]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-25T00:14:11.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-25T00:14:11.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-25T00:14:11.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-25T00:14:11.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-04-25T00:14:11.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-04-25T00:14:11.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-25T00:14:11.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2024-04-25T00:14:11.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-04-25T00:14:11.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-04-25T00:14:11.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]': finished 2024-04-25T00:14:11.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:11 smithi153 ceph-mon[32896]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T00:14:12.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:12 smithi153 ceph-mon[32896]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T00:14:12.588 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:12 smithi027 ceph-mon[29208]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T00:14:12.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:12 smithi139 ceph-mon[32833]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T00:14:13.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:13 smithi153 ceph-mon[32896]: pgmap v322: 129 pgs: 4 peering, 125 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 383 B/s wr, 10 op/s 2024-04-25T00:14:13.593 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:13 smithi027 ceph-mon[29208]: pgmap v322: 129 pgs: 4 peering, 125 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 383 B/s wr, 10 op/s 2024-04-25T00:14:13.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:13 smithi139 ceph-mon[32833]: pgmap v322: 129 pgs: 4 peering, 125 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 383 B/s wr, 10 op/s 2024-04-25T00:14:14.389 INFO:teuthology.orchestra.run.smithi027.stdout:309237645340 2024-04-25T00:14:14.389 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-25T00:14:15.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:15 smithi027 ceph-mon[29208]: pgmap v323: 129 pgs: 4 peering, 125 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 383 B/s wr, 9 op/s 2024-04-25T00:14:15.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:15 smithi153 ceph-mon[32896]: pgmap v323: 129 pgs: 4 peering, 125 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 383 B/s wr, 9 op/s 2024-04-25T00:14:15.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:15 smithi139 ceph-mon[32833]: pgmap v323: 129 pgs: 4 peering, 125 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 383 B/s wr, 9 op/s 2024-04-25T00:14:16.867 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:17.264 INFO:teuthology.orchestra.run.smithi027.stdout:34359738449 2024-04-25T00:14:17.264 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-25T00:14:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.484+0000 7f626aa2a700 1 -- 172.21.15.27:0/4201465496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f626410ece0 msgr2=0x7f62641110d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.484+0000 7f626aa2a700 1 --2- 172.21.15.27:0/4201465496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f626410ece0 0x7f62641110d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f625c00b0a0 tx=0x7f625c00b3b0 comp rx=0 tx=0).stop 2024-04-25T00:14:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.484+0000 7f626aa2a700 1 -- 172.21.15.27:0/4201465496 shutdown_connections 2024-04-25T00:14:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.484+0000 7f626aa2a700 1 --2- 172.21.15.27:0/4201465496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f626410ece0 0x7f62641110d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.484+0000 7f626aa2a700 1 --2- 172.21.15.27:0/4201465496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6264069f40 0x7f626406a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.484+0000 7f626aa2a700 1 --2- 172.21.15.27:0/4201465496 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f626406a860 0x7f626410e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:17.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.484+0000 7f626aa2a700 1 -- 172.21.15.27:0/4201465496 >> 172.21.15.27:0/4201465496 conn(0x7f62640780e0 msgr2=0x7f62640784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:17.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.484+0000 7f626aa2a700 1 -- 172.21.15.27:0/4201465496 shutdown_connections 2024-04-25T00:14:17.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.484+0000 7f626aa2a700 1 -- 172.21.15.27:0/4201465496 wait complete. 2024-04-25T00:14:17.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.485+0000 7f626aa2a700 1 Processor -- start 2024-04-25T00:14:17.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.485+0000 7f626aa2a700 1 -- start start 2024-04-25T00:14:17.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.485+0000 7f626aa2a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6264069f40 0x7f62641a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:17.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.485+0000 7f626aa2a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f626406a860 0x7f62641a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:17.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.485+0000 7f626aa2a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f626410ece0 0x7f626419a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:17.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.485+0000 7f626aa2a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6264114730 con 0x7f6264069f40 2024-04-25T00:14:17.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.485+0000 7f626aa2a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f62641145b0 con 0x7f626406a860 2024-04-25T00:14:17.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.485+0000 7f626aa2a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f62641148b0 con 0x7f626410ece0 2024-04-25T00:14:17.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.485+0000 7f6263fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6264069f40 0x7f62641a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:17.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.485+0000 7f6263fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6264069f40 0x7f62641a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46016/0 (socket says 172.21.15.27:46016) 2024-04-25T00:14:17.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.485+0000 7f6263fff700 1 -- 172.21.15.27:0/1517648027 learned_addr learned my addr 172.21.15.27:0/1517648027 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:17.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.485+0000 7f6268fc7700 1 --2- 172.21.15.27:0/1517648027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f626410ece0 0x7f626419a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:17.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.486+0000 7f62637fe700 1 --2- 172.21.15.27:0/1517648027 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f626406a860 0x7f62641a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:17.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.486+0000 7f6263fff700 1 -- 172.21.15.27:0/1517648027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f626410ece0 msgr2=0x7f626419a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:17.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.486+0000 7f6263fff700 1 --2- 172.21.15.27:0/1517648027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f626410ece0 0x7f626419a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:17.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.486+0000 7f6263fff700 1 -- 172.21.15.27:0/1517648027 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f626406a860 msgr2=0x7f62641a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:17.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.486+0000 7f6263fff700 1 --2- 172.21.15.27:0/1517648027 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f626406a860 0x7f62641a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:17.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.486+0000 7f6263fff700 1 -- 172.21.15.27:0/1517648027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f626419b0d0 con 0x7f6264069f40 2024-04-25T00:14:17.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.486+0000 7f62637fe700 1 --2- 172.21.15.27:0/1517648027 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f626406a860 0x7f62641a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:14:17.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.486+0000 7f6263fff700 1 --2- 172.21.15.27:0/1517648027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6264069f40 0x7f62641a07b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f625400d950 tx=0x7f625400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:17.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.486+0000 7f62617fa700 1 -- 172.21.15.27:0/1517648027 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6254016070 con 0x7f6264069f40 2024-04-25T00:14:17.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.486+0000 7f626aa2a700 1 -- 172.21.15.27:0/1517648027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f626419b3c0 con 0x7f6264069f40 2024-04-25T00:14:17.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.486+0000 7f62617fa700 1 -- 172.21.15.27:0/1517648027 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6254009e20 con 0x7f6264069f40 2024-04-25T00:14:17.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.486+0000 7f62617fa700 1 -- 172.21.15.27:0/1517648027 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625400f620 con 0x7f6264069f40 2024-04-25T00:14:17.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.486+0000 7f626aa2a700 1 -- 172.21.15.27:0/1517648027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62641a7680 con 0x7f6264069f40 2024-04-25T00:14:17.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:17 smithi027 ceph-mon[29208]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 383 B/s wr, 12 op/s 2024-04-25T00:14:17.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.487+0000 7f626aa2a700 1 -- 172.21.15.27:0/1517648027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f62640629c0 con 0x7f6264069f40 2024-04-25T00:14:17.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.489+0000 7f62617fa700 1 -- 172.21.15.27:0/1517648027 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6254019070 con 0x7f6264069f40 2024-04-25T00:14:17.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.489+0000 7f62617fa700 1 --2- 172.21.15.27:0/1517648027 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f62500a2cb0 0x7f62500a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:17.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.489+0000 7f62617fa700 1 -- 172.21.15.27:0/1517648027 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f62540cf160 con 0x7f6264069f40 2024-04-25T00:14:17.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.489+0000 7f62637fe700 1 --2- 172.21.15.27:0/1517648027 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f62500a2cb0 0x7f62500a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:17.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.490+0000 7f62617fa700 1 --2- 172.21.15.27:0/1517648027 >> [v2:172.21.15.139:6816/4023269486,v1:172.21.15.139:6817/4023269486] conn(0x7f62500ade10 0x7f62500b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:17.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.490+0000 7f62617fa700 1 -- 172.21.15.27:0/1517648027 --> [v2:172.21.15.139:6816/4023269486,v1:172.21.15.139:6817/4023269486] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6254010e50 con 0x7f62500ade10 2024-04-25T00:14:17.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.490+0000 7f62617fa700 1 -- 172.21.15.27:0/1517648027 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f62540d6050 con 0x7f6264069f40 2024-04-25T00:14:17.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.490+0000 7f62637fe700 1 --2- 172.21.15.27:0/1517648027 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f62500a2cb0 0x7f62500a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f624c005e10 tx=0x7f624c005d60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:17.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.490+0000 7f6268fc7700 1 --2- 172.21.15.27:0/1517648027 >> [v2:172.21.15.139:6816/4023269486,v1:172.21.15.139:6817/4023269486] conn(0x7f62500ade10 0x7f62500b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:17.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.490+0000 7f6268fc7700 1 --2- 172.21.15.27:0/1517648027 >> [v2:172.21.15.139:6816/4023269486,v1:172.21.15.139:6817/4023269486] conn(0x7f62500ade10 0x7f62500b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:17.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.491+0000 7f62617fa700 1 -- 172.21.15.27:0/1517648027 <== osd.7 v2:172.21.15.139:6816/4023269486 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6254010e50 con 0x7f62500ade10 2024-04-25T00:14:17.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.511+0000 7f626aa2a700 1 -- 172.21.15.27:0/1517648027 --> [v2:172.21.15.139:6816/4023269486,v1:172.21.15.139:6817/4023269486] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6264062dc0 con 0x7f62500ade10 2024-04-25T00:14:17.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.512+0000 7f62617fa700 1 -- 172.21.15.27:0/1517648027 <== osd.7 v2:172.21.15.139:6816/4023269486 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6264062dc0 con 0x7f62500ade10 2024-04-25T00:14:17.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.513+0000 7f626aa2a700 1 -- 172.21.15.27:0/1517648027 >> [v2:172.21.15.139:6816/4023269486,v1:172.21.15.139:6817/4023269486] conn(0x7f62500ade10 msgr2=0x7f62500b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:17.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.513+0000 7f626aa2a700 1 --2- 172.21.15.27:0/1517648027 >> [v2:172.21.15.139:6816/4023269486,v1:172.21.15.139:6817/4023269486] conn(0x7f62500ade10 0x7f62500b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:17.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.513+0000 7f626aa2a700 1 -- 172.21.15.27:0/1517648027 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f62500a2cb0 msgr2=0x7f62500a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:17.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.513+0000 7f626aa2a700 1 --2- 172.21.15.27:0/1517648027 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f62500a2cb0 0x7f62500a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f624c005e10 tx=0x7f624c005d60 comp rx=0 tx=0).stop 2024-04-25T00:14:17.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.513+0000 7f626aa2a700 1 -- 172.21.15.27:0/1517648027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6264069f40 msgr2=0x7f62641a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:17.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.513+0000 7f626aa2a700 1 --2- 172.21.15.27:0/1517648027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6264069f40 0x7f62641a07b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f625400d950 tx=0x7f625400dc60 comp rx=0 tx=0).stop 2024-04-25T00:14:17.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.513+0000 7f6268fc7700 1 -- 172.21.15.27:0/1517648027 reap_dead start 2024-04-25T00:14:17.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.513+0000 7f626aa2a700 1 -- 172.21.15.27:0/1517648027 shutdown_connections 2024-04-25T00:14:17.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.513+0000 7f626aa2a700 1 -- 172.21.15.27:0/1517648027 >> 172.21.15.27:0/1517648027 conn(0x7f62640780e0 msgr2=0x7f62641100f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:17.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.513+0000 7f626aa2a700 1 -- 172.21.15.27:0/1517648027 shutdown_connections 2024-04-25T00:14:17.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:17.513+0000 7f626aa2a700 1 -- 172.21.15.27:0/1517648027 wait complete. 2024-04-25T00:14:17.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:17 smithi153 ceph-mon[32896]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 383 B/s wr, 12 op/s 2024-04-25T00:14:17.619 INFO:teuthology.orchestra.run.smithi027.stdout:141733920827 2024-04-25T00:14:17.619 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-25T00:14:17.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:17 smithi139 ceph-mon[32833]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 383 B/s wr, 12 op/s 2024-04-25T00:14:18.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.401+0000 7f59f4884700 1 -- 172.21.15.27:0/2597261487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f0107590 msgr2=0x7f59f010f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:18.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.401+0000 7f59f4884700 1 --2- 172.21.15.27:0/2597261487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f0107590 0x7f59f010f160 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f59e800b0a0 tx=0x7f59e800b3b0 comp rx=0 tx=0).stop 2024-04-25T00:14:18.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.401+0000 7f59f4884700 1 -- 172.21.15.27:0/2597261487 shutdown_connections 2024-04-25T00:14:18.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.401+0000 7f59f4884700 1 --2- 172.21.15.27:0/2597261487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f0107590 0x7f59f010f160 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:18.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.401+0000 7f59f4884700 1 --2- 172.21.15.27:0/2597261487 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59f0102510 0x7f59f01028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:18.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.401+0000 7f59f4884700 1 --2- 172.21.15.27:0/2597261487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59f0102ec0 0x7f59f0106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:18.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.401+0000 7f59f4884700 1 -- 172.21.15.27:0/2597261487 >> 172.21.15.27:0/2597261487 conn(0x7f59f00793a0 msgr2=0x7f59f00797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:18.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.401+0000 7f59f4884700 1 -- 172.21.15.27:0/2597261487 shutdown_connections 2024-04-25T00:14:18.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.401+0000 7f59f4884700 1 -- 172.21.15.27:0/2597261487 wait complete. 2024-04-25T00:14:18.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.402+0000 7f59f4884700 1 Processor -- start 2024-04-25T00:14:18.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.402+0000 7f59f4884700 1 -- start start 2024-04-25T00:14:18.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.402+0000 7f59f4884700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f0102510 0x7f59f019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:18.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.402+0000 7f59f4884700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59f0102ec0 0x7f59f019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:18.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.402+0000 7f59f4884700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59f0107590 0x7f59f0198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:18.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.402+0000 7f59f4884700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f59f0112570 con 0x7f59f0102510 2024-04-25T00:14:18.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.402+0000 7f59f4884700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f59f01123f0 con 0x7f59f0107590 2024-04-25T00:14:18.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.402+0000 7f59f4884700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f59f01126f0 con 0x7f59f0102ec0 2024-04-25T00:14:18.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.402+0000 7f59ee59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f0102510 0x7f59f019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:18.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.402+0000 7f59ee59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f0102510 0x7f59f019e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46044/0 (socket says 172.21.15.27:46044) 2024-04-25T00:14:18.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.402+0000 7f59ee59c700 1 -- 172.21.15.27:0/854113443 learned_addr learned my addr 172.21.15.27:0/854113443 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:18.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.402+0000 7f59edd9b700 1 --2- 172.21.15.27:0/854113443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59f0102ec0 0x7f59f019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:18.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.402+0000 7f59eed9d700 1 --2- 172.21.15.27:0/854113443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59f0107590 0x7f59f0198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.403+0000 7f59ee59c700 1 -- 172.21.15.27:0/854113443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59f0102ec0 msgr2=0x7f59f019ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.403+0000 7f59ee59c700 1 --2- 172.21.15.27:0/854113443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59f0102ec0 0x7f59f019ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.403+0000 7f59ee59c700 1 -- 172.21.15.27:0/854113443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59f0107590 msgr2=0x7f59f0198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.403+0000 7f59ee59c700 1 --2- 172.21.15.27:0/854113443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59f0107590 0x7f59f0198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.403+0000 7f59ee59c700 1 -- 172.21.15.27:0/854113443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f0198ff0 con 0x7f59f0102510 2024-04-25T00:14:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.403+0000 7f59ee59c700 1 --2- 172.21.15.27:0/854113443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f0102510 0x7f59f019e690 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f59e000e9a0 tx=0x7f59e000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.403+0000 7f59df7fe700 1 -- 172.21.15.27:0/854113443 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e000ca10 con 0x7f59f0102510 2024-04-25T00:14:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.403+0000 7f59f4884700 1 -- 172.21.15.27:0/854113443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59f01992e0 con 0x7f59f0102510 2024-04-25T00:14:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.403+0000 7f59df7fe700 1 -- 172.21.15.27:0/854113443 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59e000cbb0 con 0x7f59f0102510 2024-04-25T00:14:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.403+0000 7f59df7fe700 1 -- 172.21.15.27:0/854113443 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e0018680 con 0x7f59f0102510 2024-04-25T00:14:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.403+0000 7f59f4884700 1 -- 172.21.15.27:0/854113443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59f01a54f0 con 0x7f59f0102510 2024-04-25T00:14:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.404+0000 7f59f4884700 1 -- 172.21.15.27:0/854113443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f59f0062940 con 0x7f59f0102510 2024-04-25T00:14:18.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.405+0000 7f59df7fe700 1 -- 172.21.15.27:0/854113443 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f59e0010430 con 0x7f59f0102510 2024-04-25T00:14:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.406+0000 7f59df7fe700 1 --2- 172.21.15.27:0/854113443 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f59d80a2cb0 0x7f59d80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.406+0000 7f59edd9b700 1 --2- 172.21.15.27:0/854113443 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f59d80a2cb0 0x7f59d80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.406+0000 7f59df7fe700 1 -- 172.21.15.27:0/854113443 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f59e00cff80 con 0x7f59f0102510 2024-04-25T00:14:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.406+0000 7f59df7fe700 1 --2- 172.21.15.27:0/854113443 >> [v2:172.21.15.153:6800/662156492,v1:172.21.15.153:6801/662156492] conn(0x7f59d80ade10 0x7f59d80b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.406+0000 7f59df7fe700 1 -- 172.21.15.27:0/854113443 --> [v2:172.21.15.153:6800/662156492,v1:172.21.15.153:6801/662156492] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f59e001bec0 con 0x7f59d80ade10 2024-04-25T00:14:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.406+0000 7f59df7fe700 1 -- 172.21.15.27:0/854113443 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f59e00d03d0 con 0x7f59f0102510 2024-04-25T00:14:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.407+0000 7f59edd9b700 1 --2- 172.21.15.27:0/854113443 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f59d80a2cb0 0x7f59d80a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f59f0108a50 tx=0x7f59e40073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.407+0000 7f59eed9d700 1 --2- 172.21.15.27:0/854113443 >> [v2:172.21.15.153:6800/662156492,v1:172.21.15.153:6801/662156492] conn(0x7f59d80ade10 0x7f59d80b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.407+0000 7f59eed9d700 1 --2- 172.21.15.27:0/854113443 >> [v2:172.21.15.153:6800/662156492,v1:172.21.15.153:6801/662156492] conn(0x7f59d80ade10 0x7f59d80b02b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.408+0000 7f59df7fe700 1 -- 172.21.15.27:0/854113443 <== osd.2 v2:172.21.15.153:6800/662156492 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f59e001bec0 con 0x7f59d80ade10 2024-04-25T00:14:18.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.424+0000 7f59f4884700 1 -- 172.21.15.27:0/854113443 --> [v2:172.21.15.153:6800/662156492,v1:172.21.15.153:6801/662156492] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f59f0062d40 con 0x7f59d80ade10 2024-04-25T00:14:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.425+0000 7f59df7fe700 1 -- 172.21.15.27:0/854113443 <== osd.2 v2:172.21.15.153:6800/662156492 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f59f0062d40 con 0x7f59d80ade10 2024-04-25T00:14:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.425+0000 7f59f4884700 1 -- 172.21.15.27:0/854113443 >> [v2:172.21.15.153:6800/662156492,v1:172.21.15.153:6801/662156492] conn(0x7f59d80ade10 msgr2=0x7f59d80b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.425+0000 7f59f4884700 1 --2- 172.21.15.27:0/854113443 >> [v2:172.21.15.153:6800/662156492,v1:172.21.15.153:6801/662156492] conn(0x7f59d80ade10 0x7f59d80b02b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.425+0000 7f59f4884700 1 -- 172.21.15.27:0/854113443 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f59d80a2cb0 msgr2=0x7f59d80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.425+0000 7f59f4884700 1 --2- 172.21.15.27:0/854113443 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f59d80a2cb0 0x7f59d80a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f59f0108a50 tx=0x7f59e40073d0 comp rx=0 tx=0).stop 2024-04-25T00:14:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.425+0000 7f59f4884700 1 -- 172.21.15.27:0/854113443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f0102510 msgr2=0x7f59f019e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:18.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.425+0000 7f59f4884700 1 --2- 172.21.15.27:0/854113443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f0102510 0x7f59f019e690 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f59e000e9a0 tx=0x7f59e000ecb0 comp rx=0 tx=0).stop 2024-04-25T00:14:18.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.425+0000 7f59eed9d700 1 -- 172.21.15.27:0/854113443 reap_dead start 2024-04-25T00:14:18.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.426+0000 7f59f4884700 1 -- 172.21.15.27:0/854113443 shutdown_connections 2024-04-25T00:14:18.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.426+0000 7f59f4884700 1 -- 172.21.15.27:0/854113443 >> 172.21.15.27:0/854113443 conn(0x7f59f00793a0 msgr2=0x7f59f0100400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:18.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.426+0000 7f59f4884700 1 -- 172.21.15.27:0/854113443 shutdown_connections 2024-04-25T00:14:18.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:18.426+0000 7f59f4884700 1 -- 172.21.15.27:0/854113443 wait complete. 2024-04-25T00:14:18.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:18 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:14:18.551 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:18 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:14:18.723 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:18 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:14:19.613 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:19 smithi027 ceph-mon[29208]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:19.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:19 smithi139 ceph-mon[32833]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:19.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:19 smithi153 ceph-mon[32896]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:20.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:20 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:20.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:20 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:20.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:20 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:20.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:20 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:20 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:20 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:20 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:20 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:20.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:20 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:20.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:20 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:20.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:20 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:20.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:20 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.784+0000 7f54b322c700 1 -- 172.21.15.27:0/2558783217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54ac102e90 msgr2=0x7f54ac106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.784+0000 7f54b322c700 1 --2- 172.21.15.27:0/2558783217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54ac102e90 0x7f54ac106e90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5498009990 tx=0x7f5498009ca0 comp rx=0 tx=0).stop 2024-04-25T00:14:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.785+0000 7f54b322c700 1 -- 172.21.15.27:0/2558783217 shutdown_connections 2024-04-25T00:14:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.785+0000 7f54b322c700 1 --2- 172.21.15.27:0/2558783217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54ac107570 0x7f54ac10f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.785+0000 7f54b322c700 1 --2- 172.21.15.27:0/2558783217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54ac1024e0 0x7f54ac1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.785+0000 7f54b322c700 1 --2- 172.21.15.27:0/2558783217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54ac102e90 0x7f54ac106e90 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.785+0000 7f54b322c700 1 -- 172.21.15.27:0/2558783217 >> 172.21.15.27:0/2558783217 conn(0x7f54ac079390 msgr2=0x7f54ac0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.785+0000 7f54b322c700 1 -- 172.21.15.27:0/2558783217 shutdown_connections 2024-04-25T00:14:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.785+0000 7f54b322c700 1 -- 172.21.15.27:0/2558783217 wait complete. 2024-04-25T00:14:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.786+0000 7f54b322c700 1 Processor -- start 2024-04-25T00:14:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.786+0000 7f54b322c700 1 -- start start 2024-04-25T00:14:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.786+0000 7f54b322c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54ac1024e0 0x7f54ac074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.786+0000 7f54b322c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54ac102e90 0x7f54ac074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.786+0000 7f54b322c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54ac107570 0x7f54ac074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.786+0000 7f54b322c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f54ac112590 con 0x7f54ac107570 2024-04-25T00:14:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.786+0000 7f54b322c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f54ac112410 con 0x7f54ac1024e0 2024-04-25T00:14:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.786+0000 7f54b322c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f54ac112710 con 0x7f54ac102e90 2024-04-25T00:14:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.786+0000 7f54b17c9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54ac107570 0x7f54ac074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.786+0000 7f54b17c9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54ac107570 0x7f54ac074fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46054/0 (socket says 172.21.15.27:46054) 2024-04-25T00:14:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.786+0000 7f54b17c9700 1 -- 172.21.15.27:0/2847013440 learned_addr learned my addr 172.21.15.27:0/2847013440 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.786+0000 7f54b0fc8700 1 --2- 172.21.15.27:0/2847013440 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54ac1024e0 0x7f54ac074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.787+0000 7f54abfff700 1 --2- 172.21.15.27:0/2847013440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54ac102e90 0x7f54ac074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.787+0000 7f54b17c9700 1 -- 172.21.15.27:0/2847013440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54ac102e90 msgr2=0x7f54ac074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.787+0000 7f54b17c9700 1 --2- 172.21.15.27:0/2847013440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54ac102e90 0x7f54ac074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.787+0000 7f54b17c9700 1 -- 172.21.15.27:0/2847013440 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54ac1024e0 msgr2=0x7f54ac074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.787+0000 7f54b17c9700 1 --2- 172.21.15.27:0/2847013440 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54ac1024e0 0x7f54ac074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.787+0000 7f54b17c9700 1 -- 172.21.15.27:0/2847013440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54ac1a7030 con 0x7f54ac107570 2024-04-25T00:14:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.787+0000 7f54b17c9700 1 --2- 172.21.15.27:0/2847013440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54ac107570 0x7f54ac074fb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f54a4007ee0 tx=0x7f54a400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.787+0000 7f54a9ffb700 1 -- 172.21.15.27:0/2847013440 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a4004bb0 con 0x7f54ac107570 2024-04-25T00:14:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.787+0000 7f54a9ffb700 1 -- 172.21.15.27:0/2847013440 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54a4004d50 con 0x7f54ac107570 2024-04-25T00:14:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.787+0000 7f54b322c700 1 -- 172.21.15.27:0/2847013440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54ac1a7320 con 0x7f54ac107570 2024-04-25T00:14:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.787+0000 7f54a9ffb700 1 -- 172.21.15.27:0/2847013440 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a4005960 con 0x7f54ac107570 2024-04-25T00:14:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.787+0000 7f54b322c700 1 -- 172.21.15.27:0/2847013440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54ac1a77e0 con 0x7f54ac107570 2024-04-25T00:14:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.788+0000 7f54b322c700 1 -- 172.21.15.27:0/2847013440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f54ac062940 con 0x7f54ac107570 2024-04-25T00:14:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.789+0000 7f54a9ffb700 1 -- 172.21.15.27:0/2847013440 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f54a401a020 con 0x7f54ac107570 2024-04-25T00:14:20.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.790+0000 7f54a9ffb700 1 --2- 172.21.15.27:0/2847013440 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f549c0a2cb0 0x7f549c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:20.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.790+0000 7f54b0fc8700 1 --2- 172.21.15.27:0/2847013440 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f549c0a2cb0 0x7f549c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:20.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.790+0000 7f54a9ffb700 1 -- 172.21.15.27:0/2847013440 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f54a40cf390 con 0x7f54ac107570 2024-04-25T00:14:20.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.790+0000 7f54a9ffb700 1 --2- 172.21.15.27:0/2847013440 >> [v2:172.21.15.153:6808/3798123772,v1:172.21.15.153:6809/3798123772] conn(0x7f549c0ade10 0x7f549c0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:20.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.790+0000 7f54a9ffb700 1 -- 172.21.15.27:0/2847013440 --> [v2:172.21.15.153:6808/3798123772,v1:172.21.15.153:6809/3798123772] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f54a4018ec0 con 0x7f549c0ade10 2024-04-25T00:14:20.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.790+0000 7f54a9ffb700 1 -- 172.21.15.27:0/2847013440 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f54a40cf7e0 con 0x7f54ac107570 2024-04-25T00:14:20.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.791+0000 7f54abfff700 1 --2- 172.21.15.27:0/2847013440 >> [v2:172.21.15.153:6808/3798123772,v1:172.21.15.153:6809/3798123772] conn(0x7f549c0ade10 0x7f549c0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:20.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.791+0000 7f54b0fc8700 1 --2- 172.21.15.27:0/2847013440 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f549c0a2cb0 0x7f549c0a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f54a0009a10 tx=0x7f54a0009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:20.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.791+0000 7f54abfff700 1 --2- 172.21.15.27:0/2847013440 >> [v2:172.21.15.153:6808/3798123772,v1:172.21.15.153:6809/3798123772] conn(0x7f549c0ade10 0x7f549c0b02b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:20.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.792+0000 7f54a9ffb700 1 -- 172.21.15.27:0/2847013440 <== osd.5 v2:172.21.15.153:6808/3798123772 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f54a4018ec0 con 0x7f549c0ade10 2024-04-25T00:14:20.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.808+0000 7f54b322c700 1 -- 172.21.15.27:0/2847013440 --> [v2:172.21.15.153:6808/3798123772,v1:172.21.15.153:6809/3798123772] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f54ac062d40 con 0x7f549c0ade10 2024-04-25T00:14:20.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.809+0000 7f54a9ffb700 1 -- 172.21.15.27:0/2847013440 <== osd.5 v2:172.21.15.153:6808/3798123772 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f54ac062d40 con 0x7f549c0ade10 2024-04-25T00:14:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.809+0000 7f54b322c700 1 -- 172.21.15.27:0/2847013440 >> [v2:172.21.15.153:6808/3798123772,v1:172.21.15.153:6809/3798123772] conn(0x7f549c0ade10 msgr2=0x7f549c0b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.809+0000 7f54b322c700 1 --2- 172.21.15.27:0/2847013440 >> [v2:172.21.15.153:6808/3798123772,v1:172.21.15.153:6809/3798123772] conn(0x7f549c0ade10 0x7f549c0b02b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.809+0000 7f54b322c700 1 -- 172.21.15.27:0/2847013440 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f549c0a2cb0 msgr2=0x7f549c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.809+0000 7f54b322c700 1 --2- 172.21.15.27:0/2847013440 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f549c0a2cb0 0x7f549c0a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f54a0009a10 tx=0x7f54a0009190 comp rx=0 tx=0).stop 2024-04-25T00:14:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.809+0000 7f54b322c700 1 -- 172.21.15.27:0/2847013440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54ac107570 msgr2=0x7f54ac074fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.809+0000 7f54b322c700 1 --2- 172.21.15.27:0/2847013440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54ac107570 0x7f54ac074fb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f54a4007ee0 tx=0x7f54a400d3b0 comp rx=0 tx=0).stop 2024-04-25T00:14:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.809+0000 7f54b17c9700 1 -- 172.21.15.27:0/2847013440 reap_dead start 2024-04-25T00:14:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.809+0000 7f54b322c700 1 -- 172.21.15.27:0/2847013440 shutdown_connections 2024-04-25T00:14:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.809+0000 7f54b322c700 1 -- 172.21.15.27:0/2847013440 >> 172.21.15.27:0/2847013440 conn(0x7f54ac079390 msgr2=0x7f54ac101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.810+0000 7f54b322c700 1 -- 172.21.15.27:0/2847013440 shutdown_connections 2024-04-25T00:14:20.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:20.810+0000 7f54b322c700 1 -- 172.21.15.27:0/2847013440 wait complete. 2024-04-25T00:14:21.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.181+0000 7fb3af1e4700 1 -- 172.21.15.27:0/2820870424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3a81071a0 msgr2=0x7fb3a8107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:21.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.181+0000 7fb3af1e4700 1 --2- 172.21.15.27:0/2820870424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3a81071a0 0x7fb3a8107580 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb398009990 tx=0x7fb398009ca0 comp rx=0 tx=0).stop 2024-04-25T00:14:21.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.181+0000 7fb3af1e4700 1 -- 172.21.15.27:0/2820870424 shutdown_connections 2024-04-25T00:14:21.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.182+0000 7fb3af1e4700 1 --2- 172.21.15.27:0/2820870424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a8101af0 0x7fb3a810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:21.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.182+0000 7fb3af1e4700 1 --2- 172.21.15.27:0/2820870424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3a81071a0 0x7fb3a8107580 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:21.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.182+0000 7fb3af1e4700 1 --2- 172.21.15.27:0/2820870424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3a8101130 0x7fb3a81015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:21.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.182+0000 7fb3af1e4700 1 -- 172.21.15.27:0/2820870424 >> 172.21.15.27:0/2820870424 conn(0x7fb3a80780e0 msgr2=0x7fb3a80784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:21.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.182+0000 7fb3af1e4700 1 -- 172.21.15.27:0/2820870424 shutdown_connections 2024-04-25T00:14:21.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.182+0000 7fb3af1e4700 1 -- 172.21.15.27:0/2820870424 wait complete. 2024-04-25T00:14:21.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.182+0000 7fb3af1e4700 1 Processor -- start 2024-04-25T00:14:21.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.182+0000 7fb3af1e4700 1 -- start start 2024-04-25T00:14:21.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.183+0000 7fb3af1e4700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3a8101130 0x7fb3a8102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:21.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.183+0000 7fb3af1e4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3a8101af0 0x7fb3a8103350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:21.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.183+0000 7fb3af1e4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a81071a0 0x7fb3a8103890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:21.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.183+0000 7fb3af1e4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a8111440 con 0x7fb3a8101af0 2024-04-25T00:14:21.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.183+0000 7fb3af1e4700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a81112c0 con 0x7fb3a8101130 2024-04-25T00:14:21.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.183+0000 7fb3af1e4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a81115c0 con 0x7fb3a81071a0 2024-04-25T00:14:21.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.183+0000 7fb3ad781700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a81071a0 0x7fb3a8103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:21.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.183+0000 7fb3a7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3a8101af0 0x7fb3a8103350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:21.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.183+0000 7fb3ad781700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a81071a0 0x7fb3a8103890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:59568/0 (socket says 172.21.15.27:59568) 2024-04-25T00:14:21.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.183+0000 7fb3ad781700 1 -- 172.21.15.27:0/2856462553 learned_addr learned my addr 172.21.15.27:0/2856462553 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:21.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.183+0000 7fb3acf80700 1 --2- 172.21.15.27:0/2856462553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3a8101130 0x7fb3a8102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:21.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.184+0000 7fb3ad781700 1 -- 172.21.15.27:0/2856462553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3a8101130 msgr2=0x7fb3a8102e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:21.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.184+0000 7fb3ad781700 1 --2- 172.21.15.27:0/2856462553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3a8101130 0x7fb3a8102e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:21.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.184+0000 7fb3ad781700 1 -- 172.21.15.27:0/2856462553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3a8101af0 msgr2=0x7fb3a8103350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:21.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.184+0000 7fb3ad781700 1 --2- 172.21.15.27:0/2856462553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3a8101af0 0x7fb3a8103350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:21.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.184+0000 7fb3ad781700 1 -- 172.21.15.27:0/2856462553 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a81a6ee0 con 0x7fb3a81071a0 2024-04-25T00:14:21.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.184+0000 7fb3acf80700 1 --2- 172.21.15.27:0/2856462553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3a8101130 0x7fb3a8102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:14:21.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.184+0000 7fb3a7fff700 1 --2- 172.21.15.27:0/2856462553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3a8101af0 0x7fb3a8103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:14:21.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.184+0000 7fb3ad781700 1 --2- 172.21.15.27:0/2856462553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a81071a0 0x7fb3a8103890 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb3a000ada0 tx=0x7fb3a000c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:21.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.184+0000 7fb3a5ffb700 1 -- 172.21.15.27:0/2856462553 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0017020 con 0x7fb3a81071a0 2024-04-25T00:14:21.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.184+0000 7fb3af1e4700 1 -- 172.21.15.27:0/2856462553 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3a81a71d0 con 0x7fb3a81071a0 2024-04-25T00:14:21.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.184+0000 7fb3a5ffb700 1 -- 172.21.15.27:0/2856462553 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3a000a0d0 con 0x7fb3a81071a0 2024-04-25T00:14:21.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.184+0000 7fb3a5ffb700 1 -- 172.21.15.27:0/2856462553 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0005480 con 0x7fb3a81071a0 2024-04-25T00:14:21.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.184+0000 7fb3af1e4700 1 -- 172.21.15.27:0/2856462553 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3a81a7710 con 0x7fb3a81071a0 2024-04-25T00:14:21.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.185+0000 7fb3af1e4700 1 -- 172.21.15.27:0/2856462553 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb3a81051e0 con 0x7fb3a81071a0 2024-04-25T00:14:21.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.187+0000 7fb3a5ffb700 1 -- 172.21.15.27:0/2856462553 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb3a0005620 con 0x7fb3a81071a0 2024-04-25T00:14:21.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.187+0000 7fb3a5ffb700 1 --2- 172.21.15.27:0/2856462553 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb3900a2cb0 0x7fb3900a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:21.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.187+0000 7fb3acf80700 1 --2- 172.21.15.27:0/2856462553 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb3900a2cb0 0x7fb3900a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:21.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.187+0000 7fb3a5ffb700 1 -- 172.21.15.27:0/2856462553 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb3a00cf090 con 0x7fb3a81071a0 2024-04-25T00:14:21.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.187+0000 7fb3a5ffb700 1 --2- 172.21.15.27:0/2856462553 >> [v2:172.21.15.27:6818/769569119,v1:172.21.15.27:6819/769569119] conn(0x7fb3900ade10 0x7fb3900b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:21.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.187+0000 7fb3a5ffb700 1 -- 172.21.15.27:0/2856462553 --> [v2:172.21.15.27:6818/769569119,v1:172.21.15.27:6819/769569119] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb3900b0980 con 0x7fb3900ade10 2024-04-25T00:14:21.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.187+0000 7fb3a5ffb700 1 -- 172.21.15.27:0/2856462553 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fb3a00cf4e0 con 0x7fb3a81071a0 2024-04-25T00:14:21.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.188+0000 7fb3a7fff700 1 --2- 172.21.15.27:0/2856462553 >> [v2:172.21.15.27:6818/769569119,v1:172.21.15.27:6819/769569119] conn(0x7fb3900ade10 0x7fb3900b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:21.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.188+0000 7fb3acf80700 1 --2- 172.21.15.27:0/2856462553 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb3900a2cb0 0x7fb3900a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb39800b2e0 tx=0x7fb39801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:21.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.188+0000 7fb3a7fff700 1 --2- 172.21.15.27:0/2856462553 >> [v2:172.21.15.27:6818/769569119,v1:172.21.15.27:6819/769569119] conn(0x7fb3900ade10 0x7fb3900b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:21.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.189+0000 7fb3a5ffb700 1 -- 172.21.15.27:0/2856462553 <== osd.6 v2:172.21.15.27:6818/769569119 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb3900b0980 con 0x7fb3900ade10 2024-04-25T00:14:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.206+0000 7fb3af1e4700 1 -- 172.21.15.27:0/2856462553 --> [v2:172.21.15.27:6818/769569119,v1:172.21.15.27:6819/769569119] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb3a8102850 con 0x7fb3900ade10 2024-04-25T00:14:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.206+0000 7fb3a5ffb700 1 -- 172.21.15.27:0/2856462553 <== osd.6 v2:172.21.15.27:6818/769569119 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb3a8102850 con 0x7fb3900ade10 2024-04-25T00:14:21.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.206+0000 7fb3af1e4700 1 -- 172.21.15.27:0/2856462553 >> [v2:172.21.15.27:6818/769569119,v1:172.21.15.27:6819/769569119] conn(0x7fb3900ade10 msgr2=0x7fb3900b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:21.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.207+0000 7fb3af1e4700 1 --2- 172.21.15.27:0/2856462553 >> [v2:172.21.15.27:6818/769569119,v1:172.21.15.27:6819/769569119] conn(0x7fb3900ade10 0x7fb3900b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:21.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.207+0000 7fb3af1e4700 1 -- 172.21.15.27:0/2856462553 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb3900a2cb0 msgr2=0x7fb3900a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:21.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.207+0000 7fb3af1e4700 1 --2- 172.21.15.27:0/2856462553 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb3900a2cb0 0x7fb3900a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb39800b2e0 tx=0x7fb39801a040 comp rx=0 tx=0).stop 2024-04-25T00:14:21.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.207+0000 7fb3af1e4700 1 -- 172.21.15.27:0/2856462553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a81071a0 msgr2=0x7fb3a8103890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.207+0000 7fb3af1e4700 1 --2- 172.21.15.27:0/2856462553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3a81071a0 0x7fb3a8103890 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb3a000ada0 tx=0x7fb3a000c330 comp rx=0 tx=0).stop 2024-04-25T00:14:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.207+0000 7fb3ad781700 1 -- 172.21.15.27:0/2856462553 reap_dead start 2024-04-25T00:14:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.207+0000 7fb3af1e4700 1 -- 172.21.15.27:0/2856462553 shutdown_connections 2024-04-25T00:14:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.207+0000 7fb3af1e4700 1 -- 172.21.15.27:0/2856462553 >> 172.21.15.27:0/2856462553 conn(0x7fb3a80780e0 msgr2=0x7fb3a8100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.207+0000 7fb3af1e4700 1 -- 172.21.15.27:0/2856462553 shutdown_connections 2024-04-25T00:14:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.207+0000 7fb3af1e4700 1 -- 172.21.15.27:0/2856462553 wait complete. 2024-04-25T00:14:21.397 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:21 smithi027 ceph-mon[29208]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:21.654 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:21 smithi139 ceph-mon[32833]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:21.678 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:21 smithi153 ceph-mon[32896]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:21.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.803+0000 7fe87d7fd700 1 -- 172.21.15.27:0/2747407072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8781024e0 msgr2=0x7fe8781028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:21.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.803+0000 7fe87d7fd700 1 --2- 172.21.15.27:0/2747407072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8781024e0 0x7fe8781028c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe868009990 tx=0x7fe868009ca0 comp rx=0 tx=0).stop 2024-04-25T00:14:21.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.803+0000 7fe87d7fd700 1 -- 172.21.15.27:0/2747407072 shutdown_connections 2024-04-25T00:14:21.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.803+0000 7fe87d7fd700 1 --2- 172.21.15.27:0/2747407072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe878107570 0x7fe87810f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:21.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.803+0000 7fe87d7fd700 1 --2- 172.21.15.27:0/2747407072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8781024e0 0x7fe8781028c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:21.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.803+0000 7fe87d7fd700 1 --2- 172.21.15.27:0/2747407072 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe878102e90 0x7fe878106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:21.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.803+0000 7fe87d7fd700 1 -- 172.21.15.27:0/2747407072 >> 172.21.15.27:0/2747407072 conn(0x7fe878079390 msgr2=0x7fe8780797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:21.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.803+0000 7fe87d7fd700 1 -- 172.21.15.27:0/2747407072 shutdown_connections 2024-04-25T00:14:21.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.803+0000 7fe87d7fd700 1 -- 172.21.15.27:0/2747407072 wait complete. 2024-04-25T00:14:21.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.804+0000 7fe87d7fd700 1 Processor -- start 2024-04-25T00:14:21.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.804+0000 7fe87d7fd700 1 -- start start 2024-04-25T00:14:21.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.804+0000 7fe87d7fd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8781024e0 0x7fe878074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:21.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.804+0000 7fe87d7fd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe878102e90 0x7fe878074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.804+0000 7fe87d7fd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe878107570 0x7fe878074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.804+0000 7fe87d7fd700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe878112560 con 0x7fe878107570 2024-04-25T00:14:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.804+0000 7fe87d7fd700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe8781123e0 con 0x7fe878102e90 2024-04-25T00:14:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.804+0000 7fe87d7fd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe8781126e0 con 0x7fe8781024e0 2024-04-25T00:14:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.804+0000 7fe8777fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe878107570 0x7fe878074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.804+0000 7fe876ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8781024e0 0x7fe878074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.804+0000 7fe8777fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe878107570 0x7fe878074fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46078/0 (socket says 172.21.15.27:46078) 2024-04-25T00:14:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.805+0000 7fe8777fe700 1 -- 172.21.15.27:0/143293071 learned_addr learned my addr 172.21.15.27:0/143293071 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.805+0000 7fe876ffd700 1 --2- 172.21.15.27:0/143293071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8781024e0 0x7fe878074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:59590/0 (socket says 172.21.15.27:59590) 2024-04-25T00:14:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.805+0000 7fe8767fc700 1 --2- 172.21.15.27:0/143293071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe878102e90 0x7fe878074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.805+0000 7fe876ffd700 1 -- 172.21.15.27:0/143293071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe878102e90 msgr2=0x7fe878074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.805+0000 7fe876ffd700 1 --2- 172.21.15.27:0/143293071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe878102e90 0x7fe878074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.805+0000 7fe876ffd700 1 -- 172.21.15.27:0/143293071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe878107570 msgr2=0x7fe878074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:21.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.805+0000 7fe876ffd700 1 --2- 172.21.15.27:0/143293071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe878107570 0x7fe878074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:21.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.805+0000 7fe876ffd700 1 -- 172.21.15.27:0/143293071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8781a6e60 con 0x7fe8781024e0 2024-04-25T00:14:21.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.805+0000 7fe8777fe700 1 --2- 172.21.15.27:0/143293071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe878107570 0x7fe878074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:14:21.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.805+0000 7fe876ffd700 1 --2- 172.21.15.27:0/143293071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8781024e0 0x7fe878074530 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe8680059d0 tx=0x7fe86800bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:21.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.805+0000 7fe85ffff700 1 -- 172.21.15.27:0/143293071 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868027070 con 0x7fe8781024e0 2024-04-25T00:14:21.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.805+0000 7fe87d7fd700 1 -- 172.21.15.27:0/143293071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8781a70f0 con 0x7fe8781024e0 2024-04-25T00:14:21.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.805+0000 7fe85ffff700 1 -- 172.21.15.27:0/143293071 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe868003d70 con 0x7fe8781024e0 2024-04-25T00:14:21.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.805+0000 7fe85ffff700 1 -- 172.21.15.27:0/143293071 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86800f460 con 0x7fe8781024e0 2024-04-25T00:14:21.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.805+0000 7fe87d7fd700 1 -- 172.21.15.27:0/143293071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8781a75a0 con 0x7fe8781024e0 2024-04-25T00:14:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.806+0000 7fe87d7fd700 1 -- 172.21.15.27:0/143293071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe878062900 con 0x7fe8781024e0 2024-04-25T00:14:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.808+0000 7fe85ffff700 1 -- 172.21.15.27:0/143293071 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe86802a070 con 0x7fe8781024e0 2024-04-25T00:14:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.808+0000 7fe85ffff700 1 --2- 172.21.15.27:0/143293071 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe8600a2cb0 0x7fe8600a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.808+0000 7fe8767fc700 1 --2- 172.21.15.27:0/143293071 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe8600a2cb0 0x7fe8600a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.808+0000 7fe85ffff700 1 -- 172.21.15.27:0/143293071 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe8680d0ea0 con 0x7fe8781024e0 2024-04-25T00:14:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.809+0000 7fe85ffff700 1 --2- 172.21.15.27:0/143293071 >> [v2:172.21.15.153:6824/2357170710,v1:172.21.15.153:6825/2357170710] conn(0x7fe8600ade10 0x7fe8600b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.809+0000 7fe85ffff700 1 -- 172.21.15.27:0/143293071 --> [v2:172.21.15.153:6824/2357170710,v1:172.21.15.153:6825/2357170710] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe868010ea0 con 0x7fe8600ade10 2024-04-25T00:14:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.809+0000 7fe85ffff700 1 -- 172.21.15.27:0/143293071 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fe8680d12f0 con 0x7fe8781024e0 2024-04-25T00:14:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.809+0000 7fe8767fc700 1 --2- 172.21.15.27:0/143293071 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe8600a2cb0 0x7fe8600a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe878108870 tx=0x7fe86c00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.809+0000 7fe8777fe700 1 --2- 172.21.15.27:0/143293071 >> [v2:172.21.15.153:6824/2357170710,v1:172.21.15.153:6825/2357170710] conn(0x7fe8600ade10 0x7fe8600b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.809+0000 7fe8777fe700 1 --2- 172.21.15.27:0/143293071 >> [v2:172.21.15.153:6824/2357170710,v1:172.21.15.153:6825/2357170710] conn(0x7fe8600ade10 0x7fe8600b02b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.810+0000 7fe85ffff700 1 -- 172.21.15.27:0/143293071 <== osd.11 v2:172.21.15.153:6824/2357170710 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe868010ea0 con 0x7fe8600ade10 2024-04-25T00:14:21.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.826+0000 7fe87d7fd700 1 -- 172.21.15.27:0/143293071 --> [v2:172.21.15.153:6824/2357170710,v1:172.21.15.153:6825/2357170710] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe878062db0 con 0x7fe8600ade10 2024-04-25T00:14:21.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.827+0000 7fe85ffff700 1 -- 172.21.15.27:0/143293071 <== osd.11 v2:172.21.15.153:6824/2357170710 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe878062db0 con 0x7fe8600ade10 2024-04-25T00:14:21.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.827+0000 7fe87d7fd700 1 -- 172.21.15.27:0/143293071 >> [v2:172.21.15.153:6824/2357170710,v1:172.21.15.153:6825/2357170710] conn(0x7fe8600ade10 msgr2=0x7fe8600b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:21.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.827+0000 7fe87d7fd700 1 --2- 172.21.15.27:0/143293071 >> [v2:172.21.15.153:6824/2357170710,v1:172.21.15.153:6825/2357170710] conn(0x7fe8600ade10 0x7fe8600b02b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.827+0000 7fe87d7fd700 1 -- 172.21.15.27:0/143293071 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe8600a2cb0 msgr2=0x7fe8600a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.827+0000 7fe87d7fd700 1 --2- 172.21.15.27:0/143293071 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe8600a2cb0 0x7fe8600a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe878108870 tx=0x7fe86c00a380 comp rx=0 tx=0).stop 2024-04-25T00:14:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.827+0000 7fe87d7fd700 1 -- 172.21.15.27:0/143293071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8781024e0 msgr2=0x7fe878074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.827+0000 7fe87d7fd700 1 --2- 172.21.15.27:0/143293071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8781024e0 0x7fe878074530 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe8680059d0 tx=0x7fe86800bab0 comp rx=0 tx=0).stop 2024-04-25T00:14:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.827+0000 7fe8777fe700 1 -- 172.21.15.27:0/143293071 reap_dead start 2024-04-25T00:14:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.827+0000 7fe87d7fd700 1 -- 172.21.15.27:0/143293071 shutdown_connections 2024-04-25T00:14:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.827+0000 7fe87d7fd700 1 -- 172.21.15.27:0/143293071 >> 172.21.15.27:0/143293071 conn(0x7fe878079390 msgr2=0x7fe878100290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.828+0000 7fe87d7fd700 1 -- 172.21.15.27:0/143293071 shutdown_connections 2024-04-25T00:14:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:21.828+0000 7fe87d7fd700 1 -- 172.21.15.27:0/143293071 wait complete. 2024-04-25T00:14:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.022+0000 7f0994071700 1 -- 172.21.15.27:0/1033332020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f098c069f40 msgr2=0x7f098c06a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.022+0000 7f0994071700 1 --2- 172.21.15.27:0/1033332020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f098c069f40 0x7f098c06a320 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f098400b0a0 tx=0x7f098400b3b0 comp rx=0 tx=0).stop 2024-04-25T00:14:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.022+0000 7f0994071700 1 -- 172.21.15.27:0/1033332020 shutdown_connections 2024-04-25T00:14:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.022+0000 7f0994071700 1 --2- 172.21.15.27:0/1033332020 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f098c10ece0 0x7f098c1110d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.022+0000 7f0994071700 1 --2- 172.21.15.27:0/1033332020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f098c069f40 0x7f098c06a320 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.022+0000 7f0994071700 1 --2- 172.21.15.27:0/1033332020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f098c06a860 0x7f098c10e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.022+0000 7f0994071700 1 -- 172.21.15.27:0/1033332020 >> 172.21.15.27:0/1033332020 conn(0x7f098c0780e0 msgr2=0x7f098c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.022+0000 7f0994071700 1 -- 172.21.15.27:0/1033332020 shutdown_connections 2024-04-25T00:14:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.022+0000 7f0994071700 1 -- 172.21.15.27:0/1033332020 wait complete. 2024-04-25T00:14:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.023+0000 7f0994071700 1 Processor -- start 2024-04-25T00:14:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.023+0000 7f0994071700 1 -- start start 2024-04-25T00:14:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.023+0000 7f0994071700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f098c069f40 0x7f098c1a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.023+0000 7f0994071700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f098c06a860 0x7f098c1a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.023+0000 7f0994071700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f098c10ece0 0x7f098c19aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.023+0000 7f0994071700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f098c114760 con 0x7f098c06a860 2024-04-25T00:14:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.023+0000 7f0994071700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f098c1145e0 con 0x7f098c069f40 2024-04-25T00:14:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.023+0000 7f0994071700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f098c1148e0 con 0x7f098c10ece0 2024-04-25T00:14:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.024+0000 7f099160c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f098c06a860 0x7f098c1a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.024+0000 7f099260e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f098c10ece0 0x7f098c19aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.024+0000 7f099260e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f098c10ece0 0x7f098c19aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:59600/0 (socket says 172.21.15.27:59600) 2024-04-25T00:14:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.024+0000 7f099160c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f098c06a860 0x7f098c1a0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46092/0 (socket says 172.21.15.27:46092) 2024-04-25T00:14:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.024+0000 7f099260e700 1 -- 172.21.15.27:0/3424991564 learned_addr learned my addr 172.21.15.27:0/3424991564 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.024+0000 7f0991e0d700 1 --2- 172.21.15.27:0/3424991564 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f098c069f40 0x7f098c1a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:22.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.024+0000 7f099260e700 1 -- 172.21.15.27:0/3424991564 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f098c069f40 msgr2=0x7f098c1a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:22.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.024+0000 7f099260e700 1 --2- 172.21.15.27:0/3424991564 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f098c069f40 0x7f098c1a08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:22.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.024+0000 7f099260e700 1 -- 172.21.15.27:0/3424991564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f098c06a860 msgr2=0x7f098c1a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:22.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.024+0000 7f099260e700 1 --2- 172.21.15.27:0/3424991564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f098c06a860 0x7f098c1a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:22.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.024+0000 7f099260e700 1 -- 172.21.15.27:0/3424991564 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f098c19b200 con 0x7f098c10ece0 2024-04-25T00:14:22.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.024+0000 7f099260e700 1 --2- 172.21.15.27:0/3424991564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f098c10ece0 0x7f098c19aaa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f097c00d740 tx=0x7f097c00da50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:22.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.024+0000 7f098affd700 1 -- 172.21.15.27:0/3424991564 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c014030 con 0x7f098c10ece0 2024-04-25T00:14:22.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.024+0000 7f0994071700 1 -- 172.21.15.27:0/3424991564 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f098c19b4f0 con 0x7f098c10ece0 2024-04-25T00:14:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.024+0000 7f098affd700 1 -- 172.21.15.27:0/3424991564 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f097c004500 con 0x7f098c10ece0 2024-04-25T00:14:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.024+0000 7f0994071700 1 -- 172.21.15.27:0/3424991564 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f098c1a77b0 con 0x7f098c10ece0 2024-04-25T00:14:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.024+0000 7f098affd700 1 -- 172.21.15.27:0/3424991564 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c003950 con 0x7f098c10ece0 2024-04-25T00:14:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.025+0000 7f0994071700 1 -- 172.21.15.27:0/3424991564 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0978001030 con 0x7f098c10ece0 2024-04-25T00:14:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.027+0000 7f098affd700 1 -- 172.21.15.27:0/3424991564 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f097c00f690 con 0x7f098c10ece0 2024-04-25T00:14:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.027+0000 7f098affd700 1 --2- 172.21.15.27:0/3424991564 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f09740a2cb0 0x7f09740a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.027+0000 7f0991e0d700 1 --2- 172.21.15.27:0/3424991564 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f09740a2cb0 0x7f09740a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.027+0000 7f098affd700 1 -- 172.21.15.27:0/3424991564 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f097c0cf560 con 0x7f098c10ece0 2024-04-25T00:14:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.028+0000 7f098affd700 1 --2- 172.21.15.27:0/3424991564 >> [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594] conn(0x7f09740ade10 0x7f09740b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.028+0000 7f098affd700 1 -- 172.21.15.27:0/3424991564 --> [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09740b0980 con 0x7f09740ade10 2024-04-25T00:14:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.028+0000 7f098affd700 1 -- 172.21.15.27:0/3424991564 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f097c0cf910 con 0x7f098c10ece0 2024-04-25T00:14:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.028+0000 7f099160c700 1 --2- 172.21.15.27:0/3424991564 >> [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594] conn(0x7f09740ade10 0x7f09740b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.028+0000 7f0991e0d700 1 --2- 172.21.15.27:0/3424991564 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f09740a2cb0 0x7f09740a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0984009e60 tx=0x7f098400b700 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.028+0000 7f099160c700 1 --2- 172.21.15.27:0/3424991564 >> [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594] conn(0x7f09740ade10 0x7f09740b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.029+0000 7f098affd700 1 -- 172.21.15.27:0/3424991564 <== osd.9 v2:172.21.15.27:6826/2816562594 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f09740b0980 con 0x7f09740ade10 2024-04-25T00:14:22.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.046+0000 7f0994071700 1 -- 172.21.15.27:0/3424991564 --> [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0978002e60 con 0x7f09740ade10 2024-04-25T00:14:22.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.046+0000 7f098affd700 1 -- 172.21.15.27:0/3424991564 <== osd.9 v2:172.21.15.27:6826/2816562594 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0978002e60 con 0x7f09740ade10 2024-04-25T00:14:22.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.046+0000 7f0994071700 1 -- 172.21.15.27:0/3424991564 >> [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594] conn(0x7f09740ade10 msgr2=0x7f09740b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:22.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.046+0000 7f0994071700 1 --2- 172.21.15.27:0/3424991564 >> [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594] conn(0x7f09740ade10 0x7f09740b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:22.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.046+0000 7f0994071700 1 -- 172.21.15.27:0/3424991564 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f09740a2cb0 msgr2=0x7f09740a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:22.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.046+0000 7f0994071700 1 --2- 172.21.15.27:0/3424991564 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f09740a2cb0 0x7f09740a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0984009e60 tx=0x7f098400b700 comp rx=0 tx=0).stop 2024-04-25T00:14:22.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.046+0000 7f0994071700 1 -- 172.21.15.27:0/3424991564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f098c10ece0 msgr2=0x7f098c19aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:22.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.046+0000 7f0994071700 1 --2- 172.21.15.27:0/3424991564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f098c10ece0 0x7f098c19aaa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f097c00d740 tx=0x7f097c00da50 comp rx=0 tx=0).stop 2024-04-25T00:14:22.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.046+0000 7f099260e700 1 -- 172.21.15.27:0/3424991564 reap_dead start 2024-04-25T00:14:22.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.046+0000 7f0994071700 1 -- 172.21.15.27:0/3424991564 shutdown_connections 2024-04-25T00:14:22.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.046+0000 7f0994071700 1 -- 172.21.15.27:0/3424991564 >> 172.21.15.27:0/3424991564 conn(0x7f098c0780e0 msgr2=0x7f098c110190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:22.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.047+0000 7f0994071700 1 -- 172.21.15.27:0/3424991564 shutdown_connections 2024-04-25T00:14:22.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:22.047+0000 7f0994071700 1 -- 172.21.15.27:0/3424991564 wait complete. 2024-04-25T00:14:23.047 INFO:teuthology.orchestra.run.smithi027.stdout:77309411399 2024-04-25T00:14:23.048 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-25T00:14:23.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:23 smithi027 ceph-mon[29208]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:23.731 INFO:teuthology.orchestra.run.smithi027.stdout:163208757305 2024-04-25T00:14:23.732 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-25T00:14:23.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:23 smithi139 ceph-mon[32833]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:23.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:23 smithi153 ceph-mon[32896]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.945+0000 7fd9aec46700 1 -- 172.21.15.27:0/7465884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a806b5b0 msgr2=0x7fd9a8106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.945+0000 7fd9aec46700 1 --2- 172.21.15.27:0/7465884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a806b5b0 0x7fd9a8106eb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd9940099c0 tx=0x7fd994009cd0 comp rx=0 tx=0).stop 2024-04-25T00:14:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.945+0000 7fd9aec46700 1 -- 172.21.15.27:0/7465884 shutdown_connections 2024-04-25T00:14:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.945+0000 7fd9aec46700 1 --2- 172.21.15.27:0/7465884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9a81075e0 0x7fd9a810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.945+0000 7fd9aec46700 1 --2- 172.21.15.27:0/7465884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd9a806ac00 0x7fd9a806afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.945+0000 7fd9aec46700 1 --2- 172.21.15.27:0/7465884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a806b5b0 0x7fd9a8106eb0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.945+0000 7fd9aec46700 1 -- 172.21.15.27:0/7465884 >> 172.21.15.27:0/7465884 conn(0x7fd9a80771a0 msgr2=0x7fd9a80775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.945+0000 7fd9aec46700 1 -- 172.21.15.27:0/7465884 shutdown_connections 2024-04-25T00:14:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.945+0000 7fd9aec46700 1 -- 172.21.15.27:0/7465884 wait complete. 2024-04-25T00:14:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9aec46700 1 Processor -- start 2024-04-25T00:14:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9aec46700 1 -- start start 2024-04-25T00:14:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9aec46700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd9a806ac00 0x7fd9a819e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9aec46700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9a806b5b0 0x7fd9a819ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9aec46700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a81075e0 0x7fd9a8198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9aec46700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a8112680 con 0x7fd9a81075e0 2024-04-25T00:14:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9aec46700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a8112500 con 0x7fd9a806ac00 2024-04-25T00:14:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9aec46700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a8112800 con 0x7fd9a806b5b0 2024-04-25T00:14:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9ad1e3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a81075e0 0x7fd9a8198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:23.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9ad1e3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a81075e0 0x7fd9a8198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46106/0 (socket says 172.21.15.27:46106) 2024-04-25T00:14:23.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9ad1e3700 1 -- 172.21.15.27:0/1948334903 learned_addr learned my addr 172.21.15.27:0/1948334903 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:23.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9ac9e2700 1 --2- 172.21.15.27:0/1948334903 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd9a806ac00 0x7fd9a819e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:23.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9a7fff700 1 --2- 172.21.15.27:0/1948334903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9a806b5b0 0x7fd9a819ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:23.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9ad1e3700 1 -- 172.21.15.27:0/1948334903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9a806b5b0 msgr2=0x7fd9a819ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:23.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9ad1e3700 1 --2- 172.21.15.27:0/1948334903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9a806b5b0 0x7fd9a819ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:23.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9ad1e3700 1 -- 172.21.15.27:0/1948334903 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd9a806ac00 msgr2=0x7fd9a819e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:23.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9ad1e3700 1 --2- 172.21.15.27:0/1948334903 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd9a806ac00 0x7fd9a819e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:23.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.946+0000 7fd9ad1e3700 1 -- 172.21.15.27:0/1948334903 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a81991a0 con 0x7fd9a81075e0 2024-04-25T00:14:23.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.947+0000 7fd9ad1e3700 1 --2- 172.21.15.27:0/1948334903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a81075e0 0x7fd9a8198a40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000b970 tx=0x7fd9a000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:23.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.947+0000 7fd9a5ffb700 1 -- 172.21.15.27:0/1948334903 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0019070 con 0x7fd9a81075e0 2024-04-25T00:14:23.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.947+0000 7fd9aec46700 1 -- 172.21.15.27:0/1948334903 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9a8199490 con 0x7fd9a81075e0 2024-04-25T00:14:23.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.947+0000 7fd9a5ffb700 1 -- 172.21.15.27:0/1948334903 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9a000ce10 con 0x7fd9a81075e0 2024-04-25T00:14:23.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.947+0000 7fd9a5ffb700 1 -- 172.21.15.27:0/1948334903 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0007370 con 0x7fd9a81075e0 2024-04-25T00:14:23.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.947+0000 7fd9aec46700 1 -- 172.21.15.27:0/1948334903 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a81a5700 con 0x7fd9a81075e0 2024-04-25T00:14:23.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.948+0000 7fd9aec46700 1 -- 172.21.15.27:0/1948334903 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd9a80629c0 con 0x7fd9a81075e0 2024-04-25T00:14:23.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.949+0000 7fd9a5ffb700 1 -- 172.21.15.27:0/1948334903 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd9a001c070 con 0x7fd9a81075e0 2024-04-25T00:14:23.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.950+0000 7fd9a5ffb700 1 --2- 172.21.15.27:0/1948334903 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd9980a2cb0 0x7fd9980a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:23.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.950+0000 7fd9ac9e2700 1 --2- 172.21.15.27:0/1948334903 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd9980a2cb0 0x7fd9980a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:23.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.950+0000 7fd9a5ffb700 1 -- 172.21.15.27:0/1948334903 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd9a00cefb0 con 0x7fd9a81075e0 2024-04-25T00:14:23.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.950+0000 7fd9a5ffb700 1 --2- 172.21.15.27:0/1948334903 >> [v2:172.21.15.153:6816/2251501067,v1:172.21.15.153:6817/2251501067] conn(0x7fd9980ade10 0x7fd9980b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:23.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.950+0000 7fd9a5ffb700 1 -- 172.21.15.27:0/1948334903 --> [v2:172.21.15.153:6816/2251501067,v1:172.21.15.153:6817/2251501067] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9a001ce50 con 0x7fd9980ade10 2024-04-25T00:14:23.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.950+0000 7fd9a5ffb700 1 -- 172.21.15.27:0/1948334903 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fd9a00cf400 con 0x7fd9a81075e0 2024-04-25T00:14:23.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.950+0000 7fd9ac9e2700 1 --2- 172.21.15.27:0/1948334903 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd9980a2cb0 0x7fd9980a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8107170 tx=0x7fd99c009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:23.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.950+0000 7fd9a7fff700 1 --2- 172.21.15.27:0/1948334903 >> [v2:172.21.15.153:6816/2251501067,v1:172.21.15.153:6817/2251501067] conn(0x7fd9980ade10 0x7fd9980b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:23.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.951+0000 7fd9a7fff700 1 --2- 172.21.15.27:0/1948334903 >> [v2:172.21.15.153:6816/2251501067,v1:172.21.15.153:6817/2251501067] conn(0x7fd9980ade10 0x7fd9980b02b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:23.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.952+0000 7fd9a5ffb700 1 -- 172.21.15.27:0/1948334903 <== osd.8 v2:172.21.15.153:6816/2251501067 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd9a001ce50 con 0x7fd9980ade10 2024-04-25T00:14:23.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.968+0000 7fd9aec46700 1 -- 172.21.15.27:0/1948334903 --> [v2:172.21.15.153:6816/2251501067,v1:172.21.15.153:6817/2251501067] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd9a8062e40 con 0x7fd9980ade10 2024-04-25T00:14:23.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.969+0000 7fd9a5ffb700 1 -- 172.21.15.27:0/1948334903 <== osd.8 v2:172.21.15.153:6816/2251501067 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd9a8062e40 con 0x7fd9980ade10 2024-04-25T00:14:23.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.969+0000 7fd9aec46700 1 -- 172.21.15.27:0/1948334903 >> [v2:172.21.15.153:6816/2251501067,v1:172.21.15.153:6817/2251501067] conn(0x7fd9980ade10 msgr2=0x7fd9980b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:23.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.969+0000 7fd9aec46700 1 --2- 172.21.15.27:0/1948334903 >> [v2:172.21.15.153:6816/2251501067,v1:172.21.15.153:6817/2251501067] conn(0x7fd9980ade10 0x7fd9980b02b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:23.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.969+0000 7fd9aec46700 1 -- 172.21.15.27:0/1948334903 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd9980a2cb0 msgr2=0x7fd9980a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.969+0000 7fd9aec46700 1 --2- 172.21.15.27:0/1948334903 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fd9980a2cb0 0x7fd9980a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8107170 tx=0x7fd99c009190 comp rx=0 tx=0).stop 2024-04-25T00:14:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.969+0000 7fd9aec46700 1 -- 172.21.15.27:0/1948334903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a81075e0 msgr2=0x7fd9a8198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.969+0000 7fd9aec46700 1 --2- 172.21.15.27:0/1948334903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a81075e0 0x7fd9a8198a40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000b970 tx=0x7fd9a000bc80 comp rx=0 tx=0).stop 2024-04-25T00:14:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.969+0000 7fd9ad1e3700 1 -- 172.21.15.27:0/1948334903 reap_dead start 2024-04-25T00:14:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.969+0000 7fd9aec46700 1 -- 172.21.15.27:0/1948334903 shutdown_connections 2024-04-25T00:14:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.969+0000 7fd9aec46700 1 -- 172.21.15.27:0/1948334903 >> 172.21.15.27:0/1948334903 conn(0x7fd9a80771a0 msgr2=0x7fd9a80ff5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.969+0000 7fd9aec46700 1 -- 172.21.15.27:0/1948334903 shutdown_connections 2024-04-25T00:14:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:23.969+0000 7fd9aec46700 1 -- 172.21.15.27:0/1948334903 wait complete. 2024-04-25T00:14:24.391 INFO:teuthology.orchestra.run.smithi027.stdout:219043332142 2024-04-25T00:14:24.391 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-25T00:14:24.822 INFO:teuthology.orchestra.run.smithi027.stdout:197568495668 2024-04-25T00:14:24.823 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-25T00:14:25.029 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:25.405 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:25 smithi027 ceph-mon[29208]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:25.431 INFO:teuthology.orchestra.run.smithi027.stdout:287762808868 2024-04-25T00:14:25.432 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-25T00:14:25.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:25 smithi139 ceph-mon[32833]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:25.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:25 smithi153 ceph-mon[32896]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:25.910 INFO:teuthology.orchestra.run.smithi027.stdout:253403070506 2024-04-25T00:14:25.910 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-25T00:14:26.457 INFO:teuthology.orchestra.run.smithi027.stdout:343597383707 2024-04-25T00:14:26.458 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-25T00:14:26.779 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:26.779 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:27.658 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:27 smithi027 ceph-mon[29208]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:27.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:27 smithi139 ceph-mon[32833]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:27.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:27 smithi153 ceph-mon[32896]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:28.849 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:29.624 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:29 smithi027 ceph-mon[29208]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:29.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:29 smithi139 ceph-mon[32833]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:29.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:29 smithi153 ceph-mon[32896]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T00:14:31.220 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:31.221 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:31.221 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:31.222 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:31.225 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:31.228 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:31.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.247+0000 7ff51759e700 1 -- 172.21.15.27:0/1191815891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff51810a430 msgr2=0x7ff51810a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:31.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.247+0000 7ff51759e700 1 --2- 172.21.15.27:0/1191815891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff51810a430 0x7ff51810a810 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff51000b3e0 tx=0x7ff51000b6f0 comp rx=0 tx=0).stop 2024-04-25T00:14:31.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.247+0000 7ff51759e700 1 -- 172.21.15.27:0/1191815891 shutdown_connections 2024-04-25T00:14:31.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.247+0000 7ff51759e700 1 --2- 172.21.15.27:0/1191815891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff518104d40 0x7ff5181111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:31.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.247+0000 7ff51759e700 1 --2- 172.21.15.27:0/1191815891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff51810a430 0x7ff51810a810 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:31.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.247+0000 7ff51759e700 1 --2- 172.21.15.27:0/1191815891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff518104380 0x7ff518104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:31.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.247+0000 7ff51759e700 1 -- 172.21.15.27:0/1191815891 >> 172.21.15.27:0/1191815891 conn(0x7ff518077f70 msgr2=0x7ff518102880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:31.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.247+0000 7ff51759e700 1 -- 172.21.15.27:0/1191815891 shutdown_connections 2024-04-25T00:14:31.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.247+0000 7ff51759e700 1 -- 172.21.15.27:0/1191815891 wait complete. 2024-04-25T00:14:31.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.249+0000 7ff51759e700 1 Processor -- start 2024-04-25T00:14:31.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.249+0000 7ff51759e700 1 -- start start 2024-04-25T00:14:31.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.249+0000 7ff51759e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff518104380 0x7ff5181a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:31.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.249+0000 7ff51759e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff518104d40 0x7ff5181a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:31.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.249+0000 7ff51759e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff51810a430 0x7ff51819a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:31.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.249+0000 7ff51759e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff518114760 con 0x7ff518104380 2024-04-25T00:14:31.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.249+0000 7ff51759e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff5181145e0 con 0x7ff51810a430 2024-04-25T00:14:31.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.249+0000 7ff51759e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff5181148e0 con 0x7ff518104d40 2024-04-25T00:14:31.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.250+0000 7ff51659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff518104380 0x7ff5181a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:31.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.250+0000 7ff516d9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff51810a430 0x7ff51819a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:31.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.250+0000 7ff516d9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff51810a430 0x7ff51819a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:37356/0 (socket says 172.21.15.27:37356) 2024-04-25T00:14:31.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.250+0000 7ff516d9d700 1 -- 172.21.15.27:0/3828069288 learned_addr learned my addr 172.21.15.27:0/3828069288 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:31.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.251+0000 7ff515d9b700 1 --2- 172.21.15.27:0/3828069288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff518104d40 0x7ff5181a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:31.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.251+0000 7ff51659c700 1 -- 172.21.15.27:0/3828069288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff518104d40 msgr2=0x7ff5181a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:31.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.251+0000 7ff51659c700 1 --2- 172.21.15.27:0/3828069288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff518104d40 0x7ff5181a0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:31.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.251+0000 7ff51659c700 1 -- 172.21.15.27:0/3828069288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff51810a430 msgr2=0x7ff51819a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:31.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.251+0000 7ff51659c700 1 --2- 172.21.15.27:0/3828069288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff51810a430 0x7ff51819a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:31.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.251+0000 7ff51659c700 1 -- 172.21.15.27:0/3828069288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff51819b040 con 0x7ff518104380 2024-04-25T00:14:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.252+0000 7ff51659c700 1 --2- 172.21.15.27:0/3828069288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff518104380 0x7ff5181a0720 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff51000b3e0 tx=0x7ff5100079a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.253+0000 7ff50f7fe700 1 -- 172.21.15.27:0/3828069288 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff510021070 con 0x7ff518104380 2024-04-25T00:14:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.253+0000 7ff50f7fe700 1 -- 172.21.15.27:0/3828069288 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff51001d930 con 0x7ff518104380 2024-04-25T00:14:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.253+0000 7ff50f7fe700 1 -- 172.21.15.27:0/3828069288 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5100053a0 con 0x7ff518104380 2024-04-25T00:14:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.253+0000 7ff51759e700 1 -- 172.21.15.27:0/3828069288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff51819b2d0 con 0x7ff518104380 2024-04-25T00:14:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.253+0000 7ff51759e700 1 -- 172.21.15.27:0/3828069288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff51807a6b0 con 0x7ff518104380 2024-04-25T00:14:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.255+0000 7ff51759e700 1 -- 172.21.15.27:0/3828069288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff4f8001030 con 0x7ff518104380 2024-04-25T00:14:31.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.258+0000 7ff50f7fe700 1 -- 172.21.15.27:0/3828069288 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff510024070 con 0x7ff518104380 2024-04-25T00:14:31.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.259+0000 7ff50f7fe700 1 --2- 172.21.15.27:0/3828069288 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff5040a2df0 0x7ff5040a52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:31.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.259+0000 7ff50f7fe700 1 -- 172.21.15.27:0/3828069288 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff5100d0970 con 0x7ff518104380 2024-04-25T00:14:31.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.259+0000 7ff50f7fe700 1 --2- 172.21.15.27:0/3828069288 >> [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629] conn(0x7ff5040adf50 0x7ff5040b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:31.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.259+0000 7ff50f7fe700 1 -- 172.21.15.27:0/3828069288 --> [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5100020b0 con 0x7ff5040adf50 2024-04-25T00:14:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.259+0000 7ff50f7fe700 1 -- 172.21.15.27:0/3828069288 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7ff5100d0d60 con 0x7ff518104380 2024-04-25T00:14:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.261+0000 7ff516d9d700 1 --2- 172.21.15.27:0/3828069288 >> [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629] conn(0x7ff5040adf50 0x7ff5040b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.261+0000 7ff516d9d700 1 --2- 172.21.15.27:0/3828069288 >> [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629] conn(0x7ff5040adf50 0x7ff5040b03f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.262+0000 7ff515d9b700 1 --2- 172.21.15.27:0/3828069288 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff5040a2df0 0x7ff5040a52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.262+0000 7ff515d9b700 1 --2- 172.21.15.27:0/3828069288 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff5040a2df0 0x7ff5040a52b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff508006fd0 tx=0x7ff508008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.263+0000 7ff50f7fe700 1 -- 172.21.15.27:0/3828069288 <== osd.3 v2:172.21.15.27:6810/2150456629 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5100020b0 con 0x7ff5040adf50 2024-04-25T00:14:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.288+0000 7ff51759e700 1 -- 172.21.15.27:0/3828069288 --> [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff4f8002e60 con 0x7ff5040adf50 2024-04-25T00:14:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.288+0000 7ff50f7fe700 1 -- 172.21.15.27:0/3828069288 <== osd.3 v2:172.21.15.27:6810/2150456629 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff4f8002e60 con 0x7ff5040adf50 2024-04-25T00:14:31.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.289+0000 7ff51759e700 1 -- 172.21.15.27:0/3828069288 >> [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629] conn(0x7ff5040adf50 msgr2=0x7ff5040b03f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:31.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.289+0000 7ff51759e700 1 --2- 172.21.15.27:0/3828069288 >> [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629] conn(0x7ff5040adf50 0x7ff5040b03f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.289+0000 7ff51759e700 1 -- 172.21.15.27:0/3828069288 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff5040a2df0 msgr2=0x7ff5040a52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.289+0000 7ff51759e700 1 --2- 172.21.15.27:0/3828069288 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff5040a2df0 0x7ff5040a52b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff508006fd0 tx=0x7ff508008040 comp rx=0 tx=0).stop 2024-04-25T00:14:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.289+0000 7ff51759e700 1 -- 172.21.15.27:0/3828069288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff518104380 msgr2=0x7ff5181a0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.289+0000 7ff51759e700 1 --2- 172.21.15.27:0/3828069288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff518104380 0x7ff5181a0720 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff51000b3e0 tx=0x7ff5100079a0 comp rx=0 tx=0).stop 2024-04-25T00:14:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.289+0000 7ff516d9d700 1 -- 172.21.15.27:0/3828069288 reap_dead start 2024-04-25T00:14:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.289+0000 7ff51759e700 1 -- 172.21.15.27:0/3828069288 shutdown_connections 2024-04-25T00:14:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.289+0000 7ff51759e700 1 -- 172.21.15.27:0/3828069288 >> 172.21.15.27:0/3828069288 conn(0x7ff518077f70 msgr2=0x7ff518078e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.289+0000 7ff51759e700 1 -- 172.21.15.27:0/3828069288 shutdown_connections 2024-04-25T00:14:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:31.289+0000 7ff51759e700 1 -- 172.21.15.27:0/3828069288 wait complete. 2024-04-25T00:14:31.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:31 smithi027 ceph-mon[29208]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:14:31.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:31 smithi139 ceph-mon[32833]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:14:31.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:31 smithi153 ceph-mon[32896]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:14:32.515 INFO:teuthology.orchestra.run.smithi027.stdout:107374182469 2024-04-25T00:14:32.516 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-25T00:14:32.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.813+0000 7f3989525700 1 -- 172.21.15.27:0/570854644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39841082c0 msgr2=0x7f39841086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:32.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.813+0000 7f3989525700 1 --2- 172.21.15.27:0/570854644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39841082c0 0x7f39841086a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3974009990 tx=0x7f3974009ca0 comp rx=0 tx=0).stop 2024-04-25T00:14:32.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.813+0000 7f3989525700 1 -- 172.21.15.27:0/570854644 shutdown_connections 2024-04-25T00:14:32.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.813+0000 7f3989525700 1 --2- 172.21.15.27:0/570854644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39841011c0 0x7f398410f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:32.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.813+0000 7f3989525700 1 --2- 172.21.15.27:0/570854644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39841082c0 0x7f39841086a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:32.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.813+0000 7f3989525700 1 --2- 172.21.15.27:0/570854644 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3984100820 0x7f3984100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:32.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.813+0000 7f3989525700 1 -- 172.21.15.27:0/570854644 >> 172.21.15.27:0/570854644 conn(0x7f39840777a0 msgr2=0x7f39840feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:32.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.813+0000 7f3989525700 1 -- 172.21.15.27:0/570854644 shutdown_connections 2024-04-25T00:14:32.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.813+0000 7f3989525700 1 -- 172.21.15.27:0/570854644 wait complete. 2024-04-25T00:14:32.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f3989525700 1 Processor -- start 2024-04-25T00:14:32.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f3989525700 1 -- start start 2024-04-25T00:14:32.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f3989525700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3984100820 0x7f3984101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:32.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f3989525700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39841011c0 0x7f3984101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:32.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f3989525700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39841082c0 0x7f3984102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:32.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f3989525700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3984112520 con 0x7f39841082c0 2024-04-25T00:14:32.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f3989525700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f39841123a0 con 0x7f39841011c0 2024-04-25T00:14:32.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f3989525700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39841126a0 con 0x7f3984100820 2024-04-25T00:14:32.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f39837fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39841082c0 0x7f3984102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:32.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f39837fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39841082c0 0x7f3984102480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52456/0 (socket says 172.21.15.27:52456) 2024-04-25T00:14:32.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f39837fe700 1 -- 172.21.15.27:0/3591707331 learned_addr learned my addr 172.21.15.27:0/3591707331 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:32.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f3982ffd700 1 --2- 172.21.15.27:0/3591707331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3984100820 0x7f3984101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:32.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f39827fc700 1 --2- 172.21.15.27:0/3591707331 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39841011c0 0x7f3984101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:32.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f39837fe700 1 -- 172.21.15.27:0/3591707331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3984100820 msgr2=0x7f3984101a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:32.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f39837fe700 1 --2- 172.21.15.27:0/3591707331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3984100820 0x7f3984101a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:32.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f39837fe700 1 -- 172.21.15.27:0/3591707331 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39841011c0 msgr2=0x7f3984101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:32.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f39837fe700 1 --2- 172.21.15.27:0/3591707331 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39841011c0 0x7f3984101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:32.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.814+0000 7f39837fe700 1 -- 172.21.15.27:0/3591707331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39841a6ee0 con 0x7f39841082c0 2024-04-25T00:14:32.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.815+0000 7f39827fc700 1 --2- 172.21.15.27:0/3591707331 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39841011c0 0x7f3984101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:14:32.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.815+0000 7f39837fe700 1 --2- 172.21.15.27:0/3591707331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39841082c0 0x7f3984102480 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f397c007ee0 tx=0x7f397c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:32.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.815+0000 7f396bfff700 1 -- 172.21.15.27:0/3591707331 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397c00dbd0 con 0x7f39841082c0 2024-04-25T00:14:32.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.815+0000 7f396bfff700 1 -- 172.21.15.27:0/3591707331 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f397c00dd70 con 0x7f39841082c0 2024-04-25T00:14:32.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.815+0000 7f396bfff700 1 -- 172.21.15.27:0/3591707331 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397c005a60 con 0x7f39841082c0 2024-04-25T00:14:32.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.815+0000 7f3989525700 1 -- 172.21.15.27:0/3591707331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39841a71d0 con 0x7f39841082c0 2024-04-25T00:14:32.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.815+0000 7f3989525700 1 -- 172.21.15.27:0/3591707331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39841a76c0 con 0x7f39841082c0 2024-04-25T00:14:32.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.816+0000 7f3989525700 1 -- 172.21.15.27:0/3591707331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3984103b40 con 0x7f39841082c0 2024-04-25T00:14:32.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.817+0000 7f396bfff700 1 -- 172.21.15.27:0/3591707331 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f397c01a020 con 0x7f39841082c0 2024-04-25T00:14:32.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.818+0000 7f396bfff700 1 --2- 172.21.15.27:0/3591707331 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f396c0a2cb0 0x7f396c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:32.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.818+0000 7f3982ffd700 1 --2- 172.21.15.27:0/3591707331 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f396c0a2cb0 0x7f396c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:32.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.818+0000 7f396bfff700 1 -- 172.21.15.27:0/3591707331 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f397c0ce620 con 0x7f39841082c0 2024-04-25T00:14:32.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.818+0000 7f3982ffd700 1 --2- 172.21.15.27:0/3591707331 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f396c0a2cb0 0x7f396c0a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f397400b5c0 tx=0x7f3974005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:32.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:32.822+0000 7f396bfff700 1 -- 172.21.15.27:0/3591707331 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f397c010070 con 0x7f39841082c0 2024-04-25T00:14:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:33.036+0000 7f3989525700 1 -- 172.21.15.27:0/3591707331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f398402dcf0 con 0x7f39841082c0 2024-04-25T00:14:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:33.037+0000 7f396bfff700 1 -- 172.21.15.27:0/3591707331 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f397c09b0c0 con 0x7f39841082c0 2024-04-25T00:14:33.037 INFO:teuthology.orchestra.run.smithi027.stdout:55834574927 2024-04-25T00:14:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:33.039+0000 7f3989525700 1 -- 172.21.15.27:0/3591707331 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f396c0a2cb0 msgr2=0x7f396c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:33.039+0000 7f3989525700 1 --2- 172.21.15.27:0/3591707331 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f396c0a2cb0 0x7f396c0a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f397400b5c0 tx=0x7f3974005960 comp rx=0 tx=0).stop 2024-04-25T00:14:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:33.039+0000 7f3989525700 1 -- 172.21.15.27:0/3591707331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39841082c0 msgr2=0x7f3984102480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:33.039+0000 7f3989525700 1 --2- 172.21.15.27:0/3591707331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39841082c0 0x7f3984102480 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f397c007ee0 tx=0x7f397c00d3b0 comp rx=0 tx=0).stop 2024-04-25T00:14:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:33.039+0000 7f3989525700 1 -- 172.21.15.27:0/3591707331 shutdown_connections 2024-04-25T00:14:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:33.039+0000 7f3989525700 1 --2- 172.21.15.27:0/3591707331 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f396c0a2cb0 0x7f396c0a5170 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:33.039+0000 7f3989525700 1 --2- 172.21.15.27:0/3591707331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39841082c0 0x7f3984102480 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:33.039+0000 7f3989525700 1 --2- 172.21.15.27:0/3591707331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3984100820 0x7f3984101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:33.039+0000 7f3989525700 1 --2- 172.21.15.27:0/3591707331 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39841011c0 0x7f3984101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:33.039+0000 7f3989525700 1 -- 172.21.15.27:0/3591707331 >> 172.21.15.27:0/3591707331 conn(0x7f39840777a0 msgr2=0x7f39840feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:33.039+0000 7f3989525700 1 -- 172.21.15.27:0/3591707331 shutdown_connections 2024-04-25T00:14:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:33.039+0000 7f3989525700 1 -- 172.21.15.27:0/3591707331 wait complete. 2024-04-25T00:14:33.561 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:33 smithi027 ceph-mon[29208]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T00:14:33.561 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:33 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3591707331' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T00:14:33.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:33 smithi139 ceph-mon[32833]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T00:14:33.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:33 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3591707331' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T00:14:33.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:33 smithi153 ceph-mon[32896]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T00:14:33.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:33 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3591707331' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T00:14:35.600 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:35 smithi027 ceph-mon[29208]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-25T00:14:35.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:35 smithi139 ceph-mon[32833]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-25T00:14:35.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:35 smithi153 ceph-mon[32896]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-25T00:14:35.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.927+0000 7f8d7bdd6700 1 -- 172.21.15.27:0/3669519324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7406b300 msgr2=0x7f8d7410f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.927+0000 7f8d7bdd6700 1 --2- 172.21.15.27:0/3669519324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7406b300 0x7f8d7410f0f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c00b0a0 tx=0x7f8d6c00b3b0 comp rx=0 tx=0).stop 2024-04-25T00:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.927+0000 7f8d7bdd6700 1 -- 172.21.15.27:0/3669519324 shutdown_connections 2024-04-25T00:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.927+0000 7f8d7bdd6700 1 --2- 172.21.15.27:0/3669519324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7406b300 0x7f8d7410f0f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.927+0000 7f8d7bdd6700 1 --2- 172.21.15.27:0/3669519324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d741082b0 0x7f8d74108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.927+0000 7f8d7bdd6700 1 --2- 172.21.15.27:0/3669519324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7406a960 0x7f8d7406adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.927+0000 7f8d7bdd6700 1 -- 172.21.15.27:0/3669519324 >> 172.21.15.27:0/3669519324 conn(0x7f8d740780e0 msgr2=0x7f8d740784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.928+0000 7f8d7bdd6700 1 -- 172.21.15.27:0/3669519324 shutdown_connections 2024-04-25T00:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.928+0000 7f8d7bdd6700 1 -- 172.21.15.27:0/3669519324 wait complete. 2024-04-25T00:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.928+0000 7f8d7bdd6700 1 Processor -- start 2024-04-25T00:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.928+0000 7f8d7bdd6700 1 -- start start 2024-04-25T00:14:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.928+0000 7f8d7bdd6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7406a960 0x7f8d74101230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.928+0000 7f8d7bdd6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7406b300 0x7f8d74101770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.928+0000 7f8d7bdd6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d741082b0 0x7f8d74101cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.928+0000 7f8d7bdd6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8d741125b0 con 0x7f8d7406b300 2024-04-25T00:14:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.928+0000 7f8d7bdd6700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8d74112430 con 0x7f8d741082b0 2024-04-25T00:14:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.928+0000 7f8d7bdd6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8d74112730 con 0x7f8d7406a960 2024-04-25T00:14:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.929+0000 7f8d79371700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7406b300 0x7f8d74101770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.929+0000 7f8d79371700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7406b300 0x7f8d74101770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52466/0 (socket says 172.21.15.27:52466) 2024-04-25T00:14:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.929+0000 7f8d79371700 1 -- 172.21.15.27:0/3341199603 learned_addr learned my addr 172.21.15.27:0/3341199603 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.929+0000 7f8d79b72700 1 --2- 172.21.15.27:0/3341199603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7406a960 0x7f8d74101230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.929+0000 7f8d7a373700 1 --2- 172.21.15.27:0/3341199603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d741082b0 0x7f8d74101cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.929+0000 7f8d79371700 1 -- 172.21.15.27:0/3341199603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7406a960 msgr2=0x7f8d74101230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.929+0000 7f8d79371700 1 --2- 172.21.15.27:0/3341199603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7406a960 0x7f8d74101230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.929+0000 7f8d79371700 1 -- 172.21.15.27:0/3341199603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d741082b0 msgr2=0x7f8d74101cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.929+0000 7f8d79371700 1 --2- 172.21.15.27:0/3341199603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d741082b0 0x7f8d74101cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.929+0000 7f8d79371700 1 -- 172.21.15.27:0/3341199603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d741a6f30 con 0x7f8d7406b300 2024-04-25T00:14:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.929+0000 7f8d79371700 1 --2- 172.21.15.27:0/3341199603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7406b300 0x7f8d74101770 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8d6400b530 tx=0x7f8d6400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.929+0000 7f8d6affd700 1 -- 172.21.15.27:0/3341199603 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d64015020 con 0x7f8d7406b300 2024-04-25T00:14:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.929+0000 7f8d7bdd6700 1 -- 172.21.15.27:0/3341199603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d741a7220 con 0x7f8d7406b300 2024-04-25T00:14:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.929+0000 7f8d6affd700 1 -- 172.21.15.27:0/3341199603 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d64011d10 con 0x7f8d7406b300 2024-04-25T00:14:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.929+0000 7f8d6affd700 1 -- 172.21.15.27:0/3341199603 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6400f2e0 con 0x7f8d7406b300 2024-04-25T00:14:35.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.929+0000 7f8d7bdd6700 1 -- 172.21.15.27:0/3341199603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d741a7760 con 0x7f8d7406b300 2024-04-25T00:14:35.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.930+0000 7f8d7bdd6700 1 -- 172.21.15.27:0/3341199603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d74068f20 con 0x7f8d7406b300 2024-04-25T00:14:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.932+0000 7f8d6affd700 1 -- 172.21.15.27:0/3341199603 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8d64018020 con 0x7f8d7406b300 2024-04-25T00:14:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.932+0000 7f8d6affd700 1 --2- 172.21.15.27:0/3341199603 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d600a2cb0 0x7f8d600a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.932+0000 7f8d6affd700 1 -- 172.21.15.27:0/3341199603 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8d640ceb90 con 0x7f8d7406b300 2024-04-25T00:14:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.932+0000 7f8d79b72700 1 --2- 172.21.15.27:0/3341199603 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d600a2cb0 0x7f8d600a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.933+0000 7f8d79b72700 1 --2- 172.21.15.27:0/3341199603 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d600a2cb0 0x7f8d600a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8d70005950 tx=0x7f8d7000a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:35.937+0000 7f8d6affd700 1 -- 172.21.15.27:0/3341199603 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8d6401f020 con 0x7f8d7406b300 2024-04-25T00:14:36.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:36.144+0000 7f8d7bdd6700 1 -- 172.21.15.27:0/3341199603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f8d7402db30 con 0x7f8d7406b300 2024-04-25T00:14:36.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:36.145+0000 7f8d6affd700 1 -- 172.21.15.27:0/3341199603 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8d6409b740 con 0x7f8d7406b300 2024-04-25T00:14:36.146 INFO:teuthology.orchestra.run.smithi027.stdout:141733920831 2024-04-25T00:14:36.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:36.147+0000 7f8d7bdd6700 1 -- 172.21.15.27:0/3341199603 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d600a2cb0 msgr2=0x7f8d600a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:36.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:36.147+0000 7f8d7bdd6700 1 --2- 172.21.15.27:0/3341199603 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d600a2cb0 0x7f8d600a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8d70005950 tx=0x7f8d7000a300 comp rx=0 tx=0).stop 2024-04-25T00:14:36.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:36.147+0000 7f8d7bdd6700 1 -- 172.21.15.27:0/3341199603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7406b300 msgr2=0x7f8d74101770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:36.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:36.147+0000 7f8d7bdd6700 1 --2- 172.21.15.27:0/3341199603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7406b300 0x7f8d74101770 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8d6400b530 tx=0x7f8d6400b840 comp rx=0 tx=0).stop 2024-04-25T00:14:36.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:36.147+0000 7f8d7bdd6700 1 -- 172.21.15.27:0/3341199603 shutdown_connections 2024-04-25T00:14:36.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:36.147+0000 7f8d7bdd6700 1 --2- 172.21.15.27:0/3341199603 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8d600a2cb0 0x7f8d600a5170 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:36.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:36.147+0000 7f8d7bdd6700 1 --2- 172.21.15.27:0/3341199603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d741082b0 0x7f8d74101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:36.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:36.147+0000 7f8d7bdd6700 1 --2- 172.21.15.27:0/3341199603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7406a960 0x7f8d74101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:36.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:36.147+0000 7f8d7bdd6700 1 --2- 172.21.15.27:0/3341199603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7406b300 0x7f8d74101770 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:36.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:36.147+0000 7f8d7bdd6700 1 -- 172.21.15.27:0/3341199603 >> 172.21.15.27:0/3341199603 conn(0x7f8d740780e0 msgr2=0x7f8d740ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:36.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:36.147+0000 7f8d7bdd6700 1 -- 172.21.15.27:0/3341199603 shutdown_connections 2024-04-25T00:14:36.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:36.147+0000 7f8d7bdd6700 1 -- 172.21.15.27:0/3341199603 wait complete. 2024-04-25T00:14:36.440 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:36 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3341199603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T00:14:36.594 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574922 got 55834574927 for osd.1 2024-04-25T00:14:36.594 DEBUG:teuthology.parallel:result is None 2024-04-25T00:14:36.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:36 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3341199603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T00:14:36.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:36 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3341199603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T00:14:37.248 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:37.634 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:37 smithi027 ceph-mon[29208]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-04-25T00:14:37.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:37 smithi139 ceph-mon[32833]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-04-25T00:14:37.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:37 smithi153 ceph-mon[32896]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-04-25T00:14:38.960 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920827 got 141733920831 for osd.4 2024-04-25T00:14:38.961 DEBUG:teuthology.parallel:result is None 2024-04-25T00:14:39.586 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:39 smithi027 ceph-mon[29208]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:14:39.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:39 smithi139 ceph-mon[32833]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:14:39.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:39 smithi153 ceph-mon[32896]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:14:41.361 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:41 smithi027 ceph-mon[29208]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:14:41.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:41 smithi139 ceph-mon[32833]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:14:41.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:41 smithi153 ceph-mon[32896]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:14:42.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.086+0000 7f9a8499b700 1 -- 172.21.15.27:0/2499697372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a8006a930 msgr2=0x7f9a8006ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:42.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.086+0000 7f9a8499b700 1 --2- 172.21.15.27:0/2499697372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a8006a930 0x7f9a8006ad90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9a74009990 tx=0x7f9a74009ca0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.086+0000 7f9a8499b700 1 -- 172.21.15.27:0/2499697372 shutdown_connections 2024-04-25T00:14:42.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.086+0000 7f9a8499b700 1 --2- 172.21.15.27:0/2499697372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a8006b2d0 0x7f9a8010f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.086+0000 7f9a8499b700 1 --2- 172.21.15.27:0/2499697372 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a801082e0 0x7f9a801086c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.086+0000 7f9a8499b700 1 --2- 172.21.15.27:0/2499697372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a8006a930 0x7f9a8006ad90 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.086+0000 7f9a8499b700 1 -- 172.21.15.27:0/2499697372 >> 172.21.15.27:0/2499697372 conn(0x7f9a80078170 msgr2=0x7f9a80078580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:42.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.086+0000 7f9a8499b700 1 -- 172.21.15.27:0/2499697372 shutdown_connections 2024-04-25T00:14:42.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.086+0000 7f9a8499b700 1 -- 172.21.15.27:0/2499697372 wait complete. 2024-04-25T00:14:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.087+0000 7f9a8499b700 1 Processor -- start 2024-04-25T00:14:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.087+0000 7f9a8499b700 1 -- start start 2024-04-25T00:14:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.087+0000 7f9a8499b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a8006a930 0x7f9a80101230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.087+0000 7f9a8499b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a8006b2d0 0x7f9a80101770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.087+0000 7f9a8499b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a801082e0 0x7f9a80101cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.087+0000 7f9a8499b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9a80112540 con 0x7f9a801082e0 2024-04-25T00:14:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.087+0000 7f9a8499b700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9a801123c0 con 0x7f9a8006a930 2024-04-25T00:14:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.087+0000 7f9a8499b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9a801126c0 con 0x7f9a8006b2d0 2024-04-25T00:14:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.087+0000 7f9a7ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a801082e0 0x7f9a80101cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.087+0000 7f9a7ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a801082e0 0x7f9a80101cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48878/0 (socket says 172.21.15.27:48878) 2024-04-25T00:14:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.087+0000 7f9a7ed9d700 1 -- 172.21.15.27:0/108009961 learned_addr learned my addr 172.21.15.27:0/108009961 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.087+0000 7f9a7e59c700 1 --2- 172.21.15.27:0/108009961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a8006a930 0x7f9a80101230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.087+0000 7f9a7dd9b700 1 --2- 172.21.15.27:0/108009961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a8006b2d0 0x7f9a80101770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:42.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.088+0000 7f9a7ed9d700 1 -- 172.21.15.27:0/108009961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a8006b2d0 msgr2=0x7f9a80101770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:42.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.088+0000 7f9a7ed9d700 1 --2- 172.21.15.27:0/108009961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a8006b2d0 0x7f9a80101770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.088+0000 7f9a7ed9d700 1 -- 172.21.15.27:0/108009961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a8006a930 msgr2=0x7f9a80101230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:42.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.088+0000 7f9a7ed9d700 1 --2- 172.21.15.27:0/108009961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a8006a930 0x7f9a80101230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.088+0000 7f9a7ed9d700 1 -- 172.21.15.27:0/108009961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a801a6fc0 con 0x7f9a801082e0 2024-04-25T00:14:42.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.088+0000 7f9a7ed9d700 1 --2- 172.21.15.27:0/108009961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a801082e0 0x7f9a80101cb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9a78007ee0 tx=0x7f9a7800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:42.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.089+0000 7f9a6f7fe700 1 -- 172.21.15.27:0/108009961 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7800dd20 con 0x7f9a801082e0 2024-04-25T00:14:42.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.089+0000 7f9a6f7fe700 1 -- 172.21.15.27:0/108009961 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a78004020 con 0x7f9a801082e0 2024-04-25T00:14:42.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.089+0000 7f9a8499b700 1 -- 172.21.15.27:0/108009961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a801a72b0 con 0x7f9a801082e0 2024-04-25T00:14:42.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.089+0000 7f9a6f7fe700 1 -- 172.21.15.27:0/108009961 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a78015660 con 0x7f9a801082e0 2024-04-25T00:14:42.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.090+0000 7f9a8499b700 1 -- 172.21.15.27:0/108009961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a801a77c0 con 0x7f9a801082e0 2024-04-25T00:14:42.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.091+0000 7f9a8499b700 1 -- 172.21.15.27:0/108009961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a80103990 con 0x7f9a801082e0 2024-04-25T00:14:42.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.092+0000 7f9a6f7fe700 1 -- 172.21.15.27:0/108009961 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9a780041d0 con 0x7f9a801082e0 2024-04-25T00:14:42.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.093+0000 7f9a6f7fe700 1 --2- 172.21.15.27:0/108009961 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9a680a2d30 0x7f9a680a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:42.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.093+0000 7f9a7e59c700 1 --2- 172.21.15.27:0/108009961 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9a680a2d30 0x7f9a680a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:42.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.093+0000 7f9a6f7fe700 1 -- 172.21.15.27:0/108009961 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9a780ceef0 con 0x7f9a801082e0 2024-04-25T00:14:42.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.093+0000 7f9a7e59c700 1 --2- 172.21.15.27:0/108009961 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9a680a2d30 0x7f9a680a51f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9a70005950 tx=0x7f9a7000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:42.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.098+0000 7f9a6f7fe700 1 -- 172.21.15.27:0/108009961 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9a78010070 con 0x7f9a801082e0 2024-04-25T00:14:42.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.310+0000 7f9a8499b700 1 -- 172.21.15.27:0/108009961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f9a8002db30 con 0x7f9a801082e0 2024-04-25T00:14:42.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.310+0000 7f9a6f7fe700 1 -- 172.21.15.27:0/108009961 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f9a7809b990 con 0x7f9a801082e0 2024-04-25T00:14:42.311 INFO:teuthology.orchestra.run.smithi027.stdout:163208757309 2024-04-25T00:14:42.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.312+0000 7f9a8499b700 1 -- 172.21.15.27:0/108009961 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9a680a2d30 msgr2=0x7f9a680a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:42.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.312+0000 7f9a8499b700 1 --2- 172.21.15.27:0/108009961 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9a680a2d30 0x7f9a680a51f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9a70005950 tx=0x7f9a7000b410 comp rx=0 tx=0).stop 2024-04-25T00:14:42.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.312+0000 7f9a8499b700 1 -- 172.21.15.27:0/108009961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a801082e0 msgr2=0x7f9a80101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:42.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.312+0000 7f9a8499b700 1 --2- 172.21.15.27:0/108009961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a801082e0 0x7f9a80101cb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9a78007ee0 tx=0x7f9a7800d3b0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.312+0000 7f9a8499b700 1 -- 172.21.15.27:0/108009961 shutdown_connections 2024-04-25T00:14:42.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.312+0000 7f9a8499b700 1 --2- 172.21.15.27:0/108009961 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9a680a2d30 0x7f9a680a51f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.312+0000 7f9a8499b700 1 --2- 172.21.15.27:0/108009961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a801082e0 0x7f9a80101cb0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.312+0000 7f9a8499b700 1 --2- 172.21.15.27:0/108009961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a8006a930 0x7f9a80101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.312+0000 7f9a8499b700 1 --2- 172.21.15.27:0/108009961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a8006b2d0 0x7f9a80101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.312+0000 7f9a8499b700 1 -- 172.21.15.27:0/108009961 >> 172.21.15.27:0/108009961 conn(0x7f9a80078170 msgr2=0x7f9a800fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:42.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.312+0000 7f9a8499b700 1 -- 172.21.15.27:0/108009961 shutdown_connections 2024-04-25T00:14:42.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:42.312+0000 7f9a8499b700 1 -- 172.21.15.27:0/108009961 wait complete. 2024-04-25T00:14:42.603 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:42 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/108009961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T00:14:42.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:42 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/108009961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T00:14:42.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:42 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/108009961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T00:14:43.555 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:43 smithi027 ceph-mon[29208]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:14:43.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:43 smithi139 ceph-mon[32833]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:14:43.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:43 smithi153 ceph-mon[32896]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:14:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.958+0000 7fb174550700 1 -- 172.21.15.27:0/3157597398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb16c101130 msgr2=0x7fb16c1015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.958+0000 7fb174550700 1 --2- 172.21.15.27:0/3157597398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb16c101130 0x7fb16c1015b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009990 tx=0x7fb15c009ca0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.958+0000 7fb174550700 1 -- 172.21.15.27:0/3157597398 shutdown_connections 2024-04-25T00:14:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.958+0000 7fb174550700 1 --2- 172.21.15.27:0/3157597398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb16c101af0 0x7fb16c10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.958+0000 7fb174550700 1 --2- 172.21.15.27:0/3157597398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb16c1071a0 0x7fb16c107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.958+0000 7fb174550700 1 --2- 172.21.15.27:0/3157597398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb16c101130 0x7fb16c1015b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.958+0000 7fb174550700 1 -- 172.21.15.27:0/3157597398 >> 172.21.15.27:0/3157597398 conn(0x7fb16c0780e0 msgr2=0x7fb16c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.958+0000 7fb174550700 1 -- 172.21.15.27:0/3157597398 shutdown_connections 2024-04-25T00:14:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.958+0000 7fb174550700 1 -- 172.21.15.27:0/3157597398 wait complete. 2024-04-25T00:14:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb174550700 1 Processor -- start 2024-04-25T00:14:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb174550700 1 -- start start 2024-04-25T00:14:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb174550700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb16c101130 0x7fb16c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb174550700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb16c101af0 0x7fb16c19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb174550700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb16c1071a0 0x7fb16c198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb174550700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c111480 con 0x7fb16c1071a0 2024-04-25T00:14:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb174550700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c111300 con 0x7fb16c101130 2024-04-25T00:14:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb174550700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c111600 con 0x7fb16c101af0 2024-04-25T00:14:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb172aed700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb16c1071a0 0x7fb16c198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb172aed700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb16c1071a0 0x7fb16c198860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48904/0 (socket says 172.21.15.27:48904) 2024-04-25T00:14:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb172aed700 1 -- 172.21.15.27:0/2171476638 learned_addr learned my addr 172.21.15.27:0/2171476638 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:44.183 INFO:teuthology.orchestra.run.smithi027.stdout:197568495672 2024-04-25T00:14:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb1722ec700 1 --2- 172.21.15.27:0/2171476638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb16c101130 0x7fb16c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb171aeb700 1 --2- 172.21.15.27:0/2171476638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb16c101af0 0x7fb16c19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb172aed700 1 -- 172.21.15.27:0/2171476638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb16c101af0 msgr2=0x7fb16c19ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb172aed700 1 --2- 172.21.15.27:0/2171476638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb16c101af0 0x7fb16c19ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb172aed700 1 -- 172.21.15.27:0/2171476638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb16c101130 msgr2=0x7fb16c19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb172aed700 1 --2- 172.21.15.27:0/2171476638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb16c101130 0x7fb16c19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.959+0000 7fb172aed700 1 -- 172.21.15.27:0/2171476638 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb16c199040 con 0x7fb16c1071a0 2024-04-25T00:14:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.960+0000 7fb172aed700 1 --2- 172.21.15.27:0/2171476638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb16c1071a0 0x7fb16c198860 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb164007ee0 tx=0x7fb16400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.960+0000 7fb1637fe700 1 -- 172.21.15.27:0/2171476638 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16400dbd0 con 0x7fb16c1071a0 2024-04-25T00:14:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.960+0000 7fb1637fe700 1 -- 172.21.15.27:0/2171476638 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb16400dd70 con 0x7fb16c1071a0 2024-04-25T00:14:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.960+0000 7fb1637fe700 1 -- 172.21.15.27:0/2171476638 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb164005a60 con 0x7fb16c1071a0 2024-04-25T00:14:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.960+0000 7fb174550700 1 -- 172.21.15.27:0/2171476638 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb16c199330 con 0x7fb16c1071a0 2024-04-25T00:14:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.960+0000 7fb174550700 1 -- 172.21.15.27:0/2171476638 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb16c1a54f0 con 0x7fb16c1071a0 2024-04-25T00:14:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.961+0000 7fb174550700 1 -- 172.21.15.27:0/2171476638 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb16c102850 con 0x7fb16c1071a0 2024-04-25T00:14:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.962+0000 7fb1637fe700 1 -- 172.21.15.27:0/2171476638 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb16401a020 con 0x7fb16c1071a0 2024-04-25T00:14:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.963+0000 7fb1637fe700 1 --2- 172.21.15.27:0/2171476638 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb1580a2d30 0x7fb1580a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.963+0000 7fb1722ec700 1 --2- 172.21.15.27:0/2171476638 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb1580a2d30 0x7fb1580a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.963+0000 7fb1637fe700 1 -- 172.21.15.27:0/2171476638 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb1640cf270 con 0x7fb16c1071a0 2024-04-25T00:14:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.963+0000 7fb1722ec700 1 --2- 172.21.15.27:0/2171476638 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb1580a2d30 0x7fb1580a51f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb168009be0 tx=0x7fb168009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:43.966+0000 7fb1637fe700 1 -- 172.21.15.27:0/2171476638 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb164010070 con 0x7fb16c1071a0 2024-04-25T00:14:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.173+0000 7fb174550700 1 -- 172.21.15.27:0/2171476638 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fb16c1a5720 con 0x7fb16c1071a0 2024-04-25T00:14:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.173+0000 7fb1637fe700 1 -- 172.21.15.27:0/2171476638 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb16409bd10 con 0x7fb16c1071a0 2024-04-25T00:14:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.175+0000 7fb174550700 1 -- 172.21.15.27:0/2171476638 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb1580a2d30 msgr2=0x7fb1580a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.175+0000 7fb174550700 1 --2- 172.21.15.27:0/2171476638 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb1580a2d30 0x7fb1580a51f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb168009be0 tx=0x7fb168009260 comp rx=0 tx=0).stop 2024-04-25T00:14:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.175+0000 7fb174550700 1 -- 172.21.15.27:0/2171476638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb16c1071a0 msgr2=0x7fb16c198860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.175+0000 7fb174550700 1 --2- 172.21.15.27:0/2171476638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb16c1071a0 0x7fb16c198860 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb164007ee0 tx=0x7fb16400d3b0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.175+0000 7fb174550700 1 -- 172.21.15.27:0/2171476638 shutdown_connections 2024-04-25T00:14:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.175+0000 7fb174550700 1 --2- 172.21.15.27:0/2171476638 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb1580a2d30 0x7fb1580a51f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.175+0000 7fb174550700 1 --2- 172.21.15.27:0/2171476638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb16c1071a0 0x7fb16c198860 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.175+0000 7fb174550700 1 --2- 172.21.15.27:0/2171476638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb16c101130 0x7fb16c19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.176+0000 7fb174550700 1 --2- 172.21.15.27:0/2171476638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb16c101af0 0x7fb16c19ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.176+0000 7fb174550700 1 -- 172.21.15.27:0/2171476638 >> 172.21.15.27:0/2171476638 conn(0x7fb16c0780e0 msgr2=0x7fb16c105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.176+0000 7fb174550700 1 -- 172.21.15.27:0/2171476638 shutdown_connections 2024-04-25T00:14:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.176+0000 7fb174550700 1 -- 172.21.15.27:0/2171476638 wait complete. 2024-04-25T00:14:44.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:44 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2171476638' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T00:14:44.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:44 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2171476638' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T00:14:44.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:44 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2171476638' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T00:14:44.933 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757305 got 163208757309 for osd.5 2024-04-25T00:14:44.933 DEBUG:teuthology.parallel:result is None 2024-04-25T00:14:45.586 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:45 smithi027 ceph-mon[29208]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-25T00:14:45.586 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:45 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4087207059' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T00:14:45.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:45 smithi139 ceph-mon[32833]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-25T00:14:45.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:45 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4087207059' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T00:14:45.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:45 smithi153 ceph-mon[32896]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-25T00:14:45.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:45 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4087207059' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T00:14:45.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.907+0000 7f98d40a2700 1 -- 172.21.15.27:0/3239786321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98cc108360 msgr2=0x7f98cc108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:45.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.907+0000 7f98d40a2700 1 --2- 172.21.15.27:0/3239786321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98cc108360 0x7f98cc108740 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f98c8009990 tx=0x7f98c8009ca0 comp rx=0 tx=0).stop 2024-04-25T00:14:45.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.907+0000 7f98d40a2700 1 -- 172.21.15.27:0/3239786321 shutdown_connections 2024-04-25T00:14:45.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.907+0000 7f98d40a2700 1 --2- 172.21.15.27:0/3239786321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98cc102c80 0x7f98cc10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.907+0000 7f98d40a2700 1 --2- 172.21.15.27:0/3239786321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98cc108360 0x7f98cc108740 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.907+0000 7f98d40a2700 1 --2- 172.21.15.27:0/3239786321 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98cc1022c0 0x7f98cc102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.907+0000 7f98d40a2700 1 -- 172.21.15.27:0/3239786321 >> 172.21.15.27:0/3239786321 conn(0x7f98cc079420 msgr2=0x7f98cc079830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.907+0000 7f98d40a2700 1 -- 172.21.15.27:0/3239786321 shutdown_connections 2024-04-25T00:14:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.907+0000 7f98d40a2700 1 -- 172.21.15.27:0/3239786321 wait complete. 2024-04-25T00:14:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.907+0000 7f98d40a2700 1 Processor -- start 2024-04-25T00:14:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.907+0000 7f98d40a2700 1 -- start start 2024-04-25T00:14:45.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d40a2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98cc1022c0 0x7f98cc074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:45.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d40a2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98cc102c80 0x7f98cc074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:45.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d40a2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98cc108360 0x7f98cc074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:45.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d40a2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f98cc1125a0 con 0x7f98cc102c80 2024-04-25T00:14:45.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d40a2700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f98cc112420 con 0x7f98cc1022c0 2024-04-25T00:14:45.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d40a2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f98cc112720 con 0x7f98cc108360 2024-04-25T00:14:45.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d163d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98cc102c80 0x7f98cc074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:45.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d163d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98cc102c80 0x7f98cc074a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48954/0 (socket says 172.21.15.27:48954) 2024-04-25T00:14:45.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d163d700 1 -- 172.21.15.27:0/3935438146 learned_addr learned my addr 172.21.15.27:0/3935438146 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:45.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d263f700 1 --2- 172.21.15.27:0/3935438146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98cc108360 0x7f98cc074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:45.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d1e3e700 1 --2- 172.21.15.27:0/3935438146 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98cc1022c0 0x7f98cc074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:45.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d163d700 1 -- 172.21.15.27:0/3935438146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98cc108360 msgr2=0x7f98cc074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:45.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d163d700 1 --2- 172.21.15.27:0/3935438146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98cc108360 0x7f98cc074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:45.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d163d700 1 -- 172.21.15.27:0/3935438146 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98cc1022c0 msgr2=0x7f98cc074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:45.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d163d700 1 --2- 172.21.15.27:0/3935438146 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98cc1022c0 0x7f98cc074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:45.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d163d700 1 -- 172.21.15.27:0/3935438146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98cc1a6f40 con 0x7f98cc102c80 2024-04-25T00:14:45.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d263f700 1 --2- 172.21.15.27:0/3935438146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98cc108360 0x7f98cc074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:14:45.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.908+0000 7f98d163d700 1 --2- 172.21.15.27:0/3935438146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98cc102c80 0x7f98cc074a70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00c9b0 tx=0x7f98bc00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:45.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.909+0000 7f98c2ffd700 1 -- 172.21.15.27:0/3935438146 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc007b00 con 0x7f98cc102c80 2024-04-25T00:14:45.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.909+0000 7f98d40a2700 1 -- 172.21.15.27:0/3935438146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98cc1a7230 con 0x7f98cc102c80 2024-04-25T00:14:45.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.909+0000 7f98d40a2700 1 -- 172.21.15.27:0/3935438146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98cc1a77c0 con 0x7f98cc102c80 2024-04-25T00:14:45.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.909+0000 7f98c2ffd700 1 -- 172.21.15.27:0/3935438146 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98bc007ca0 con 0x7f98cc102c80 2024-04-25T00:14:45.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.909+0000 7f98c2ffd700 1 -- 172.21.15.27:0/3935438146 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc018780 con 0x7f98cc102c80 2024-04-25T00:14:45.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.910+0000 7f98d40a2700 1 -- 172.21.15.27:0/3935438146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98b0005330 con 0x7f98cc102c80 2024-04-25T00:14:45.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.911+0000 7f98c2ffd700 1 -- 172.21.15.27:0/3935438146 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f98bc018920 con 0x7f98cc102c80 2024-04-25T00:14:45.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.912+0000 7f98c2ffd700 1 --2- 172.21.15.27:0/3935438146 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f98b80a2d30 0x7f98b80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:45.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.912+0000 7f98d1e3e700 1 --2- 172.21.15.27:0/3935438146 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f98b80a2d30 0x7f98b80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:45.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.912+0000 7f98c2ffd700 1 -- 172.21.15.27:0/3935438146 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f98bc0cf670 con 0x7f98cc102c80 2024-04-25T00:14:45.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.912+0000 7f98d1e3e700 1 --2- 172.21.15.27:0/3935438146 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f98b80a2d30 0x7f98b80a51f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f98c800b5c0 tx=0x7f98c8005a90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:45.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:45.916+0000 7f98c2ffd700 1 -- 172.21.15.27:0/3935438146 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f98bc014030 con 0x7f98cc102c80 2024-04-25T00:14:46.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:46.127+0000 7f98d40a2700 1 -- 172.21.15.27:0/3935438146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f98b00058f0 con 0x7f98cc102c80 2024-04-25T00:14:46.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:46.128+0000 7f98c2ffd700 1 -- 172.21.15.27:0/3935438146 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f98bc09c220 con 0x7f98cc102c80 2024-04-25T00:14:46.129 INFO:teuthology.orchestra.run.smithi027.stdout:253403070510 2024-04-25T00:14:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:46.130+0000 7f98d40a2700 1 -- 172.21.15.27:0/3935438146 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f98b80a2d30 msgr2=0x7f98b80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:46.130+0000 7f98d40a2700 1 --2- 172.21.15.27:0/3935438146 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f98b80a2d30 0x7f98b80a51f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f98c800b5c0 tx=0x7f98c8005a90 comp rx=0 tx=0).stop 2024-04-25T00:14:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:46.130+0000 7f98d40a2700 1 -- 172.21.15.27:0/3935438146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98cc102c80 msgr2=0x7f98cc074a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:46.130+0000 7f98d40a2700 1 --2- 172.21.15.27:0/3935438146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98cc102c80 0x7f98cc074a70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00c9b0 tx=0x7f98bc00cd70 comp rx=0 tx=0).stop 2024-04-25T00:14:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:46.130+0000 7f98d40a2700 1 -- 172.21.15.27:0/3935438146 shutdown_connections 2024-04-25T00:14:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:46.130+0000 7f98d40a2700 1 --2- 172.21.15.27:0/3935438146 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f98b80a2d30 0x7f98b80a51f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:46.130+0000 7f98d40a2700 1 --2- 172.21.15.27:0/3935438146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98cc108360 0x7f98cc074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:46.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:46.130+0000 7f98d40a2700 1 --2- 172.21.15.27:0/3935438146 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98cc1022c0 0x7f98cc074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:46.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:46.130+0000 7f98d40a2700 1 --2- 172.21.15.27:0/3935438146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98cc102c80 0x7f98cc074a70 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:46.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:46.130+0000 7f98d40a2700 1 -- 172.21.15.27:0/3935438146 >> 172.21.15.27:0/3935438146 conn(0x7f98cc079420 msgr2=0x7f98cc100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:46.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:46.130+0000 7f98d40a2700 1 -- 172.21.15.27:0/3935438146 shutdown_connections 2024-04-25T00:14:46.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:46.130+0000 7f98d40a2700 1 -- 172.21.15.27:0/3935438146 wait complete. 2024-04-25T00:14:46.344 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495668 got 197568495672 for osd.6 2024-04-25T00:14:46.347 DEBUG:teuthology.parallel:result is None 2024-04-25T00:14:46.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:46 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3935438146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T00:14:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.621+0000 7fe19c1fa700 1 -- 172.21.15.27:0/765909251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1941082b0 msgr2=0x7fe194108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.621+0000 7fe19c1fa700 1 --2- 172.21.15.27:0/765909251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1941082b0 0x7fe194108690 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe184009990 tx=0x7fe184009ca0 comp rx=0 tx=0).stop 2024-04-25T00:14:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.621+0000 7fe19c1fa700 1 -- 172.21.15.27:0/765909251 shutdown_connections 2024-04-25T00:14:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.621+0000 7fe19c1fa700 1 --2- 172.21.15.27:0/765909251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19406a8e0 0x7fe19410f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.621+0000 7fe19c1fa700 1 --2- 172.21.15.27:0/765909251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1941082b0 0x7fe194108690 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.621+0000 7fe19c1fa700 1 --2- 172.21.15.27:0/765909251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe194069f40 0x7fe19406a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.621+0000 7fe19c1fa700 1 -- 172.21.15.27:0/765909251 >> 172.21.15.27:0/765909251 conn(0x7fe1940780e0 msgr2=0x7fe1940784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.621+0000 7fe19c1fa700 1 -- 172.21.15.27:0/765909251 shutdown_connections 2024-04-25T00:14:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.622+0000 7fe19c1fa700 1 -- 172.21.15.27:0/765909251 wait complete. 2024-04-25T00:14:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.622+0000 7fe19c1fa700 1 Processor -- start 2024-04-25T00:14:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.622+0000 7fe19c1fa700 1 -- start start 2024-04-25T00:14:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.622+0000 7fe19c1fa700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe194069f40 0x7fe19419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.622+0000 7fe19c1fa700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe19406a8e0 0x7fe19419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.622+0000 7fe19c1fa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1941082b0 0x7fe194198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.622+0000 7fe19c1fa700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe194112510 con 0x7fe19406a8e0 2024-04-25T00:14:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.622+0000 7fe19c1fa700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe194112390 con 0x7fe194069f40 2024-04-25T00:14:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.622+0000 7fe19c1fa700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe194112690 con 0x7fe1941082b0 2024-04-25T00:14:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.622+0000 7fe199795700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe19406a8e0 0x7fe19419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.623+0000 7fe199795700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe19406a8e0 0x7fe19419ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48930/0 (socket says 172.21.15.27:48930) 2024-04-25T00:14:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.623+0000 7fe199795700 1 -- 172.21.15.27:0/4087207059 learned_addr learned my addr 172.21.15.27:0/4087207059 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.623+0000 7fe199f96700 1 --2- 172.21.15.27:0/4087207059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe194069f40 0x7fe19419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.623+0000 7fe19a797700 1 --2- 172.21.15.27:0/4087207059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1941082b0 0x7fe194198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:46.744 INFO:teuthology.orchestra.run.smithi027.stdout:309237645347 2024-04-25T00:14:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.623+0000 7fe199795700 1 -- 172.21.15.27:0/4087207059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1941082b0 msgr2=0x7fe194198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.623+0000 7fe199795700 1 --2- 172.21.15.27:0/4087207059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1941082b0 0x7fe194198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.623+0000 7fe199795700 1 -- 172.21.15.27:0/4087207059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe194069f40 msgr2=0x7fe19419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.623+0000 7fe199795700 1 --2- 172.21.15.27:0/4087207059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe194069f40 0x7fe19419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.623+0000 7fe199795700 1 -- 172.21.15.27:0/4087207059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe194198ff0 con 0x7fe19406a8e0 2024-04-25T00:14:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.623+0000 7fe19a797700 1 --2- 172.21.15.27:0/4087207059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1941082b0 0x7fe194198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:14:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.623+0000 7fe199795700 1 --2- 172.21.15.27:0/4087207059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe19406a8e0 0x7fe19419ec10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe19000d950 tx=0x7fe19000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.623+0000 7fe18affd700 1 -- 172.21.15.27:0/4087207059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe190016070 con 0x7fe19406a8e0 2024-04-25T00:14:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.623+0000 7fe18affd700 1 -- 172.21.15.27:0/4087207059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe190009e20 con 0x7fe19406a8e0 2024-04-25T00:14:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.623+0000 7fe19c1fa700 1 -- 172.21.15.27:0/4087207059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1941992e0 con 0x7fe19406a8e0 2024-04-25T00:14:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.623+0000 7fe18affd700 1 -- 172.21.15.27:0/4087207059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19000f620 con 0x7fe19406a8e0 2024-04-25T00:14:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.623+0000 7fe19c1fa700 1 -- 172.21.15.27:0/4087207059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1941a55a0 con 0x7fe19406a8e0 2024-04-25T00:14:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.624+0000 7fe19c1fa700 1 -- 172.21.15.27:0/4087207059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe194068f20 con 0x7fe19406a8e0 2024-04-25T00:14:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.626+0000 7fe18affd700 1 -- 172.21.15.27:0/4087207059 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe190019070 con 0x7fe19406a8e0 2024-04-25T00:14:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.627+0000 7fe18affd700 1 --2- 172.21.15.27:0/4087207059 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1800a2cb0 0x7fe1800a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.627+0000 7fe199f96700 1 --2- 172.21.15.27:0/4087207059 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1800a2cb0 0x7fe1800a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.627+0000 7fe18affd700 1 -- 172.21.15.27:0/4087207059 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe1900cf0f0 con 0x7fe19406a8e0 2024-04-25T00:14:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.627+0000 7fe199f96700 1 --2- 172.21.15.27:0/4087207059 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1800a2cb0 0x7fe1800a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe184006010 tx=0x7fe18400b560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.630+0000 7fe18affd700 1 -- 172.21.15.27:0/4087207059 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe19009bb90 con 0x7fe19406a8e0 2024-04-25T00:14:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.836+0000 7fe19c1fa700 1 -- 172.21.15.27:0/4087207059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fe19402db30 con 0x7fe19406a8e0 2024-04-25T00:14:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.837+0000 7fe18affd700 1 -- 172.21.15.27:0/4087207059 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fe19009b490 con 0x7fe19406a8e0 2024-04-25T00:14:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.839+0000 7fe19c1fa700 1 -- 172.21.15.27:0/4087207059 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1800a2cb0 msgr2=0x7fe1800a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.839+0000 7fe19c1fa700 1 --2- 172.21.15.27:0/4087207059 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1800a2cb0 0x7fe1800a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe184006010 tx=0x7fe18400b560 comp rx=0 tx=0).stop 2024-04-25T00:14:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.839+0000 7fe19c1fa700 1 -- 172.21.15.27:0/4087207059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe19406a8e0 msgr2=0x7fe19419ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.839+0000 7fe19c1fa700 1 --2- 172.21.15.27:0/4087207059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe19406a8e0 0x7fe19419ec10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe19000d950 tx=0x7fe19000dc60 comp rx=0 tx=0).stop 2024-04-25T00:14:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.839+0000 7fe19c1fa700 1 -- 172.21.15.27:0/4087207059 shutdown_connections 2024-04-25T00:14:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.839+0000 7fe19c1fa700 1 --2- 172.21.15.27:0/4087207059 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe1800a2cb0 0x7fe1800a5170 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.839+0000 7fe19c1fa700 1 --2- 172.21.15.27:0/4087207059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1941082b0 0x7fe194198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.839+0000 7fe19c1fa700 1 --2- 172.21.15.27:0/4087207059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe194069f40 0x7fe19419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.839+0000 7fe19c1fa700 1 --2- 172.21.15.27:0/4087207059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe19406a8e0 0x7fe19419ec10 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.839+0000 7fe19c1fa700 1 -- 172.21.15.27:0/4087207059 >> 172.21.15.27:0/4087207059 conn(0x7fe1940780e0 msgr2=0x7fe1940fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.839+0000 7fe19c1fa700 1 -- 172.21.15.27:0/4087207059 shutdown_connections 2024-04-25T00:14:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:44.839+0000 7fe19c1fa700 1 -- 172.21.15.27:0/4087207059 wait complete. 2024-04-25T00:14:46.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:46 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3935438146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T00:14:46.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:46 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3935438146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T00:14:47.486 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645340 got 309237645347 for osd.10 2024-04-25T00:14:47.486 DEBUG:teuthology.parallel:result is None 2024-04-25T00:14:47.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:47 smithi027 ceph-mon[29208]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:14:47.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:47 smithi139 ceph-mon[32833]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:14:47.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:47 smithi153 ceph-mon[32896]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:14:48.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.991+0000 7f1bb125a700 1 -- 172.21.15.27:0/27083685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac101800 msgr2=0x7f1bac106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:48.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.991+0000 7f1bb125a700 1 --2- 172.21.15.27:0/27083685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac101800 0x7f1bac106f40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1ba00099c0 tx=0x7f1ba0009cd0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.991+0000 7f1bb125a700 1 -- 172.21.15.27:0/27083685 shutdown_connections 2024-04-25T00:14:48.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.991+0000 7f1bb125a700 1 --2- 172.21.15.27:0/27083685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bac107670 0x7f1bac10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.991+0000 7f1bb125a700 1 --2- 172.21.15.27:0/27083685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bac100e50 0x7f1bac101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.991+0000 7f1bb125a700 1 --2- 172.21.15.27:0/27083685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac101800 0x7f1bac106f40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.991+0000 7f1bb125a700 1 -- 172.21.15.27:0/27083685 >> 172.21.15.27:0/27083685 conn(0x7f1bac076450 msgr2=0x7f1bac076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:48.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.991+0000 7f1bb125a700 1 -- 172.21.15.27:0/27083685 shutdown_connections 2024-04-25T00:14:48.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.992+0000 7f1bb125a700 1 -- 172.21.15.27:0/27083685 wait complete. 2024-04-25T00:14:48.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.992+0000 7f1bb125a700 1 Processor -- start 2024-04-25T00:14:48.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.992+0000 7f1bb125a700 1 -- start start 2024-04-25T00:14:48.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.993+0000 7f1bb125a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac100e50 0x7f1bac19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:48.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.993+0000 7f1bb125a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bac101800 0x7f1bac19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:48.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.993+0000 7f1bb125a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bac107670 0x7f1bac198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:48.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.993+0000 7f1bb125a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac112580 con 0x7f1bac100e50 2024-04-25T00:14:48.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.993+0000 7f1bb125a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac112400 con 0x7f1bac107670 2024-04-25T00:14:48.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.993+0000 7f1bb125a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac112700 con 0x7f1bac101800 2024-04-25T00:14:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.993+0000 7f1baad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac100e50 0x7f1bac19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.993+0000 7f1baad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac100e50 0x7f1bac19e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45212/0 (socket says 172.21.15.27:45212) 2024-04-25T00:14:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.993+0000 7f1baad9d700 1 -- 172.21.15.27:0/3763738219 learned_addr learned my addr 172.21.15.27:0/3763738219 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.993+0000 7f1baa59c700 1 --2- 172.21.15.27:0/3763738219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bac101800 0x7f1bac19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.993+0000 7f1bab59e700 1 --2- 172.21.15.27:0/3763738219 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bac107670 0x7f1bac198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.993+0000 7f1baad9d700 1 -- 172.21.15.27:0/3763738219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bac101800 msgr2=0x7f1bac19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.993+0000 7f1baad9d700 1 --2- 172.21.15.27:0/3763738219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bac101800 0x7f1bac19ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.993+0000 7f1baad9d700 1 -- 172.21.15.27:0/3763738219 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bac107670 msgr2=0x7f1bac198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.993+0000 7f1baad9d700 1 --2- 172.21.15.27:0/3763738219 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bac107670 0x7f1bac198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.993+0000 7f1baad9d700 1 -- 172.21.15.27:0/3763738219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bac199020 con 0x7f1bac100e50 2024-04-25T00:14:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.994+0000 7f1baad9d700 1 --2- 172.21.15.27:0/3763738219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac100e50 0x7f1bac19e670 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c00d950 tx=0x7f1b9c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.994+0000 7f1b93fff700 1 -- 172.21.15.27:0/3763738219 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c016070 con 0x7f1bac100e50 2024-04-25T00:14:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.994+0000 7f1b93fff700 1 -- 172.21.15.27:0/3763738219 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b9c00fd70 con 0x7f1bac100e50 2024-04-25T00:14:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.994+0000 7f1bb125a700 1 -- 172.21.15.27:0/3763738219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bac199310 con 0x7f1bac100e50 2024-04-25T00:14:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.994+0000 7f1b93fff700 1 -- 172.21.15.27:0/3763738219 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c00e610 con 0x7f1bac100e50 2024-04-25T00:14:48.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.994+0000 7f1bb125a700 1 -- 172.21.15.27:0/3763738219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bac1a55f0 con 0x7f1bac100e50 2024-04-25T00:14:48.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.995+0000 7f1bb125a700 1 -- 172.21.15.27:0/3763738219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bac06a420 con 0x7f1bac100e50 2024-04-25T00:14:48.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.996+0000 7f1b93fff700 1 -- 172.21.15.27:0/3763738219 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1b9c019070 con 0x7f1bac100e50 2024-04-25T00:14:48.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.997+0000 7f1b93fff700 1 --2- 172.21.15.27:0/3763738219 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1b940a2cb0 0x7f1b940a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:48.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.997+0000 7f1b93fff700 1 -- 172.21.15.27:0/3763738219 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1b9c0cefe0 con 0x7f1bac100e50 2024-04-25T00:14:48.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.997+0000 7f1baa59c700 1 --2- 172.21.15.27:0/3763738219 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1b940a2cb0 0x7f1b940a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:48.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:47.998+0000 7f1baa59c700 1 --2- 172.21.15.27:0/3763738219 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1b940a2cb0 0x7f1b940a5170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1bac199f90 tx=0x7f1ba0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:48.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.004+0000 7f1b93fff700 1 -- 172.21.15.27:0/3763738219 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1b9c09ba80 con 0x7f1bac100e50 2024-04-25T00:14:48.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.225+0000 7f1bb125a700 1 -- 172.21.15.27:0/3763738219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f1bac199c70 con 0x7f1bac100e50 2024-04-25T00:14:48.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.225+0000 7f1b93fff700 1 -- 172.21.15.27:0/3763738219 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1b9c09b380 con 0x7f1bac100e50 2024-04-25T00:14:48.226 INFO:teuthology.orchestra.run.smithi027.stdout:77309411405 2024-04-25T00:14:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.227+0000 7f1bb125a700 1 -- 172.21.15.27:0/3763738219 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1b940a2cb0 msgr2=0x7f1b940a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.227+0000 7f1bb125a700 1 --2- 172.21.15.27:0/3763738219 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1b940a2cb0 0x7f1b940a5170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1bac199f90 tx=0x7f1ba0005fb0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.227+0000 7f1bb125a700 1 -- 172.21.15.27:0/3763738219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac100e50 msgr2=0x7f1bac19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.227+0000 7f1bb125a700 1 --2- 172.21.15.27:0/3763738219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac100e50 0x7f1bac19e670 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c00d950 tx=0x7f1b9c00dc60 comp rx=0 tx=0).stop 2024-04-25T00:14:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.227+0000 7f1bb125a700 1 -- 172.21.15.27:0/3763738219 shutdown_connections 2024-04-25T00:14:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.227+0000 7f1bb125a700 1 --2- 172.21.15.27:0/3763738219 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f1b940a2cb0 0x7f1b940a5170 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.227+0000 7f1bb125a700 1 --2- 172.21.15.27:0/3763738219 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bac107670 0x7f1bac198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.227+0000 7f1bb125a700 1 --2- 172.21.15.27:0/3763738219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac100e50 0x7f1bac19e670 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.227+0000 7f1bb125a700 1 --2- 172.21.15.27:0/3763738219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bac101800 0x7f1bac19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.227+0000 7f1bb125a700 1 -- 172.21.15.27:0/3763738219 >> 172.21.15.27:0/3763738219 conn(0x7f1bac076450 msgr2=0x7f1bac0ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:48.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.227+0000 7f1bb125a700 1 -- 172.21.15.27:0/3763738219 shutdown_connections 2024-04-25T00:14:48.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.227+0000 7f1bb125a700 1 -- 172.21.15.27:0/3763738219 wait complete. 2024-04-25T00:14:48.569 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:48 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3763738219' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T00:14:48.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.349+0000 7fa818772700 1 -- 172.21.15.27:0/1639599457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa810101ce0 msgr2=0x7fa810102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:48.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.349+0000 7fa818772700 1 --2- 172.21.15.27:0/1639599457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa810101ce0 0x7fa810102160 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa8000099c0 tx=0x7fa800009cd0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.349+0000 7fa818772700 1 -- 172.21.15.27:0/1639599457 shutdown_connections 2024-04-25T00:14:48.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.349+0000 7fa818772700 1 --2- 172.21.15.27:0/1639599457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa8101026a0 0x7fa81010e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.349+0000 7fa818772700 1 --2- 172.21.15.27:0/1639599457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa810107400 0x7fa8101077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.349+0000 7fa818772700 1 --2- 172.21.15.27:0/1639599457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa810101ce0 0x7fa810102160 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.349+0000 7fa818772700 1 -- 172.21.15.27:0/1639599457 >> 172.21.15.27:0/1639599457 conn(0x7fa810076450 msgr2=0x7fa810076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:48.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.349+0000 7fa818772700 1 -- 172.21.15.27:0/1639599457 shutdown_connections 2024-04-25T00:14:48.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.349+0000 7fa818772700 1 -- 172.21.15.27:0/1639599457 wait complete. 2024-04-25T00:14:48.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.350+0000 7fa818772700 1 Processor -- start 2024-04-25T00:14:48.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.350+0000 7fa818772700 1 -- start start 2024-04-25T00:14:48.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.350+0000 7fa818772700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa810101ce0 0x7fa81019e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:48.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.350+0000 7fa818772700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8101026a0 0x7fa81019ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:48.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.350+0000 7fa818772700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa810107400 0x7fa8101989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:48.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.350+0000 7fa818772700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa8101115f0 con 0x7fa810101ce0 2024-04-25T00:14:48.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.350+0000 7fa818772700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa810111470 con 0x7fa810107400 2024-04-25T00:14:48.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.350+0000 7fa818772700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa810111770 con 0x7fa8101026a0 2024-04-25T00:14:48.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.350+0000 7fa81650e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa810101ce0 0x7fa81019e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:48.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.350+0000 7fa81650e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa810101ce0 0x7fa81019e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45226/0 (socket says 172.21.15.27:45226) 2024-04-25T00:14:48.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.350+0000 7fa81650e700 1 -- 172.21.15.27:0/3576908863 learned_addr learned my addr 172.21.15.27:0/3576908863 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:48.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.350+0000 7fa815d0d700 1 --2- 172.21.15.27:0/3576908863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8101026a0 0x7fa81019ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:48.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.351+0000 7fa816d0f700 1 --2- 172.21.15.27:0/3576908863 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa810107400 0x7fa8101989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:48.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.351+0000 7fa81650e700 1 -- 172.21.15.27:0/3576908863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8101026a0 msgr2=0x7fa81019ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:48.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.351+0000 7fa81650e700 1 --2- 172.21.15.27:0/3576908863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8101026a0 0x7fa81019ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.351+0000 7fa81650e700 1 -- 172.21.15.27:0/3576908863 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa810107400 msgr2=0x7fa8101989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:48.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.351+0000 7fa81650e700 1 --2- 172.21.15.27:0/3576908863 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa810107400 0x7fa8101989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.351+0000 7fa81650e700 1 -- 172.21.15.27:0/3576908863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa810199130 con 0x7fa810101ce0 2024-04-25T00:14:48.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.351+0000 7fa81650e700 1 --2- 172.21.15.27:0/3576908863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa810101ce0 0x7fa81019e7c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00d950 tx=0x7fa80c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:48.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.351+0000 7fa8077fe700 1 -- 172.21.15.27:0/3576908863 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c016070 con 0x7fa810101ce0 2024-04-25T00:14:48.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.351+0000 7fa8077fe700 1 -- 172.21.15.27:0/3576908863 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa80c009e20 con 0x7fa810101ce0 2024-04-25T00:14:48.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.351+0000 7fa8077fe700 1 -- 172.21.15.27:0/3576908863 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c00f620 con 0x7fa810101ce0 2024-04-25T00:14:48.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.351+0000 7fa818772700 1 -- 172.21.15.27:0/3576908863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa810199420 con 0x7fa810101ce0 2024-04-25T00:14:48.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.351+0000 7fa818772700 1 -- 172.21.15.27:0/3576908863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8101a5690 con 0x7fa810101ce0 2024-04-25T00:14:48.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.352+0000 7fa818772700 1 -- 172.21.15.27:0/3576908863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa810068f20 con 0x7fa810101ce0 2024-04-25T00:14:48.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.353+0000 7fa8077fe700 1 -- 172.21.15.27:0/3576908863 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa80c019070 con 0x7fa810101ce0 2024-04-25T00:14:48.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.354+0000 7fa8077fe700 1 --2- 172.21.15.27:0/3576908863 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa7fc0a2cb0 0x7fa7fc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:48.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.354+0000 7fa815d0d700 1 --2- 172.21.15.27:0/3576908863 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa7fc0a2cb0 0x7fa7fc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:48.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.354+0000 7fa8077fe700 1 -- 172.21.15.27:0/3576908863 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa80c0cf0f0 con 0x7fa810101ce0 2024-04-25T00:14:48.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.354+0000 7fa815d0d700 1 --2- 172.21.15.27:0/3576908863 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa7fc0a2cb0 0x7fa7fc0a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa810076ac0 tx=0x7fa800005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:48.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.358+0000 7fa8077fe700 1 -- 172.21.15.27:0/3576908863 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa80c09bb90 con 0x7fa810101ce0 2024-04-25T00:14:48.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.567+0000 7fa818772700 1 -- 172.21.15.27:0/3576908863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fa81002db30 con 0x7fa810101ce0 2024-04-25T00:14:48.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.568+0000 7fa8077fe700 1 -- 172.21.15.27:0/3576908863 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa80c09b490 con 0x7fa810101ce0 2024-04-25T00:14:48.589 INFO:teuthology.orchestra.run.smithi027.stdout:287762808873 2024-04-25T00:14:48.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.570+0000 7fa818772700 1 -- 172.21.15.27:0/3576908863 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa7fc0a2cb0 msgr2=0x7fa7fc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:48.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.570+0000 7fa818772700 1 --2- 172.21.15.27:0/3576908863 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa7fc0a2cb0 0x7fa7fc0a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa810076ac0 tx=0x7fa800005fb0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.570+0000 7fa818772700 1 -- 172.21.15.27:0/3576908863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa810101ce0 msgr2=0x7fa81019e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:48.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.570+0000 7fa818772700 1 --2- 172.21.15.27:0/3576908863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa810101ce0 0x7fa81019e7c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00d950 tx=0x7fa80c00dc60 comp rx=0 tx=0).stop 2024-04-25T00:14:48.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.570+0000 7fa818772700 1 -- 172.21.15.27:0/3576908863 shutdown_connections 2024-04-25T00:14:48.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.570+0000 7fa818772700 1 --2- 172.21.15.27:0/3576908863 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa7fc0a2cb0 0x7fa7fc0a5170 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.570+0000 7fa818772700 1 --2- 172.21.15.27:0/3576908863 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa810107400 0x7fa8101989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.570+0000 7fa818772700 1 --2- 172.21.15.27:0/3576908863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa810101ce0 0x7fa81019e7c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.570+0000 7fa818772700 1 --2- 172.21.15.27:0/3576908863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8101026a0 0x7fa81019ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:48.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.570+0000 7fa818772700 1 -- 172.21.15.27:0/3576908863 >> 172.21.15.27:0/3576908863 conn(0x7fa810076450 msgr2=0x7fa810100cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:48.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.570+0000 7fa818772700 1 -- 172.21.15.27:0/3576908863 shutdown_connections 2024-04-25T00:14:48.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:48.570+0000 7fa818772700 1 -- 172.21.15.27:0/3576908863 wait complete. 2024-04-25T00:14:48.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:48 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3763738219' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T00:14:48.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:48 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3763738219' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T00:14:49.365 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808868 got 287762808873 for osd.9 2024-04-25T00:14:49.365 DEBUG:teuthology.parallel:result is None 2024-04-25T00:14:49.605 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:49 smithi027 ceph-mon[29208]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:14:49.605 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:49 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3576908863' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T00:14:49.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:49 smithi139 ceph-mon[32833]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:14:49.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:49 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3576908863' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T00:14:49.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:49 smithi153 ceph-mon[32896]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:14:49.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:49 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3576908863' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T00:14:49.966 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411399 got 77309411405 for osd.2 2024-04-25T00:14:49.966 DEBUG:teuthology.parallel:result is None 2024-04-25T00:14:50.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.637+0000 7f8761137700 1 -- 172.21.15.27:0/1398263324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f875c069f40 msgr2=0x7f875c06a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:50.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.637+0000 7f8761137700 1 --2- 172.21.15.27:0/1398263324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f875c069f40 0x7f875c06a3a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8750009990 tx=0x7f8750009ca0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.637+0000 7f8761137700 1 -- 172.21.15.27:0/1398263324 shutdown_connections 2024-04-25T00:14:50.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.637+0000 7f8761137700 1 --2- 172.21.15.27:0/1398263324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f875c06a8e0 0x7f875c10f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.637+0000 7f8761137700 1 --2- 172.21.15.27:0/1398263324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f875c1082b0 0x7f875c108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.637+0000 7f8761137700 1 --2- 172.21.15.27:0/1398263324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f875c069f40 0x7f875c06a3a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.637+0000 7f8761137700 1 -- 172.21.15.27:0/1398263324 >> 172.21.15.27:0/1398263324 conn(0x7f875c0780e0 msgr2=0x7f875c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:50.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.637+0000 7f8761137700 1 -- 172.21.15.27:0/1398263324 shutdown_connections 2024-04-25T00:14:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.637+0000 7f8761137700 1 -- 172.21.15.27:0/1398263324 wait complete. 2024-04-25T00:14:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.638+0000 7f8761137700 1 Processor -- start 2024-04-25T00:14:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.638+0000 7f8761137700 1 -- start start 2024-04-25T00:14:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.638+0000 7f8761137700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f875c069f40 0x7f875c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.638+0000 7f8761137700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f875c06a8e0 0x7f875c19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.638+0000 7f8761137700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f875c1082b0 0x7f875c1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.638+0000 7f8761137700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f875c112510 con 0x7f875c069f40 2024-04-25T00:14:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.638+0000 7f8761137700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f875c112390 con 0x7f875c1082b0 2024-04-25T00:14:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.638+0000 7f8761137700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f875c112690 con 0x7f875c06a8e0 2024-04-25T00:14:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.638+0000 7f875ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f875c069f40 0x7f875c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.638+0000 7f875ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f875c069f40 0x7f875c19e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45250/0 (socket says 172.21.15.27:45250) 2024-04-25T00:14:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.638+0000 7f875ad9d700 1 -- 172.21.15.27:0/2004262592 learned_addr learned my addr 172.21.15.27:0/2004262592 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.638+0000 7f875b59e700 1 --2- 172.21.15.27:0/2004262592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f875c1082b0 0x7f875c1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.639+0000 7f875a59c700 1 --2- 172.21.15.27:0/2004262592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f875c06a8e0 0x7f875c19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:50.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.639+0000 7f875ad9d700 1 -- 172.21.15.27:0/2004262592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f875c06a8e0 msgr2=0x7f875c19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:50.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.639+0000 7f875ad9d700 1 --2- 172.21.15.27:0/2004262592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f875c06a8e0 0x7f875c19ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.639+0000 7f875ad9d700 1 -- 172.21.15.27:0/2004262592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f875c1082b0 msgr2=0x7f875c1987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:50.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.639+0000 7f875ad9d700 1 --2- 172.21.15.27:0/2004262592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f875c1082b0 0x7f875c1987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.639+0000 7f875ad9d700 1 -- 172.21.15.27:0/2004262592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f875c198fd0 con 0x7f875c069f40 2024-04-25T00:14:50.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.639+0000 7f875a59c700 1 --2- 172.21.15.27:0/2004262592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f875c06a8e0 0x7f875c19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:14:50.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.639+0000 7f875ad9d700 1 --2- 172.21.15.27:0/2004262592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f875c069f40 0x7f875c19e670 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f874c00d980 tx=0x7f874c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:50.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.639+0000 7f8743fff700 1 -- 172.21.15.27:0/2004262592 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c016070 con 0x7f875c069f40 2024-04-25T00:14:50.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.639+0000 7f8761137700 1 -- 172.21.15.27:0/2004262592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f875c1992c0 con 0x7f875c069f40 2024-04-25T00:14:50.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.639+0000 7f8743fff700 1 -- 172.21.15.27:0/2004262592 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f874c004d10 con 0x7f875c069f40 2024-04-25T00:14:50.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.639+0000 7f8743fff700 1 -- 172.21.15.27:0/2004262592 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c0055f0 con 0x7f875c069f40 2024-04-25T00:14:50.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.639+0000 7f8761137700 1 -- 172.21.15.27:0/2004262592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f875c1a54d0 con 0x7f875c069f40 2024-04-25T00:14:50.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.640+0000 7f8761137700 1 -- 172.21.15.27:0/2004262592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f875c103960 con 0x7f875c069f40 2024-04-25T00:14:50.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.641+0000 7f8743fff700 1 -- 172.21.15.27:0/2004262592 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f874c004750 con 0x7f875c069f40 2024-04-25T00:14:50.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.642+0000 7f8743fff700 1 --2- 172.21.15.27:0/2004262592 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f87440a2cb0 0x7f87440a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:50.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.642+0000 7f8743fff700 1 -- 172.21.15.27:0/2004262592 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f874c0cf860 con 0x7f875c069f40 2024-04-25T00:14:50.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.642+0000 7f875a59c700 1 --2- 172.21.15.27:0/2004262592 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f87440a2cb0 0x7f87440a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:50.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.643+0000 7f875a59c700 1 --2- 172.21.15.27:0/2004262592 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f87440a2cb0 0x7f87440a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f875000b5c0 tx=0x7f8750005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:50.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.646+0000 7f8743fff700 1 -- 172.21.15.27:0/2004262592 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f874c01f020 con 0x7f875c069f40 2024-04-25T00:14:50.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.852+0000 7fba61ec8700 1 -- 172.21.15.27:0/2906556401 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba5c101af0 msgr2=0x7fba5c10dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:50.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.852+0000 7fba61ec8700 1 --2- 172.21.15.27:0/2906556401 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba5c101af0 0x7fba5c10dfe0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fba5400b0a0 tx=0x7fba5400b3b0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.852+0000 7fba61ec8700 1 -- 172.21.15.27:0/2906556401 shutdown_connections 2024-04-25T00:14:50.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.852+0000 7fba61ec8700 1 --2- 172.21.15.27:0/2906556401 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba5c101af0 0x7fba5c10dfe0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.852+0000 7fba61ec8700 1 --2- 172.21.15.27:0/2906556401 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba5c1071a0 0x7fba5c107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.852+0000 7fba61ec8700 1 --2- 172.21.15.27:0/2906556401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba5c101130 0x7fba5c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.852+0000 7fba61ec8700 1 -- 172.21.15.27:0/2906556401 >> 172.21.15.27:0/2906556401 conn(0x7fba5c0780e0 msgr2=0x7fba5c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:50.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.852+0000 7fba61ec8700 1 -- 172.21.15.27:0/2906556401 shutdown_connections 2024-04-25T00:14:50.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.852+0000 7fba61ec8700 1 -- 172.21.15.27:0/2906556401 wait complete. 2024-04-25T00:14:50.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.853+0000 7fba61ec8700 1 Processor -- start 2024-04-25T00:14:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.853+0000 7fba61ec8700 1 -- start start 2024-04-25T00:14:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.853+0000 7fba61ec8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba5c101130 0x7fba5c102fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.853+0000 7fba61ec8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba5c101af0 0x7fba5c103510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.853+0000 7fba61ec8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba5c1071a0 0x7fba5c103a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.853+0000 7fba61ec8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fba5c111440 con 0x7fba5c101af0 2024-04-25T00:14:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.853+0000 7fba61ec8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fba5c1112c0 con 0x7fba5c101130 2024-04-25T00:14:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.853+0000 7fba61ec8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba5c1115c0 con 0x7fba5c1071a0 2024-04-25T00:14:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.854+0000 7fba5affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba5c101af0 0x7fba5c103510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.854+0000 7fba5affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba5c101af0 0x7fba5c103510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45270/0 (socket says 172.21.15.27:45270) 2024-04-25T00:14:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.854+0000 7fba5affd700 1 -- 172.21.15.27:0/1216343216 learned_addr learned my addr 172.21.15.27:0/1216343216 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.854+0000 7fba5b7fe700 1 --2- 172.21.15.27:0/1216343216 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba5c101130 0x7fba5c102fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.854+0000 7fba5bfff700 1 --2- 172.21.15.27:0/1216343216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba5c1071a0 0x7fba5c103a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.854+0000 7fba5b7fe700 1 -- 172.21.15.27:0/1216343216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba5c1071a0 msgr2=0x7fba5c103a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.854+0000 7fba5b7fe700 1 --2- 172.21.15.27:0/1216343216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba5c1071a0 0x7fba5c103a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.854+0000 7fba5b7fe700 1 -- 172.21.15.27:0/1216343216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba5c101af0 msgr2=0x7fba5c103510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.854+0000 7fba5b7fe700 1 --2- 172.21.15.27:0/1216343216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba5c101af0 0x7fba5c103510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.854+0000 7fba5b7fe700 1 -- 172.21.15.27:0/1216343216 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba5c1a6fa0 con 0x7fba5c101130 2024-04-25T00:14:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.854+0000 7fba5b7fe700 1 --2- 172.21.15.27:0/1216343216 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba5c101130 0x7fba5c102fd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00e9f0 tx=0x7fba4c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.855+0000 7fba58ff9700 1 -- 172.21.15.27:0/1216343216 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4c00c9c0 con 0x7fba5c101130 2024-04-25T00:14:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.855+0000 7fba61ec8700 1 -- 172.21.15.27:0/1216343216 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba5c1a7290 con 0x7fba5c101130 2024-04-25T00:14:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.855+0000 7fba58ff9700 1 -- 172.21.15.27:0/1216343216 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba4c00cb60 con 0x7fba5c101130 2024-04-25T00:14:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.855+0000 7fba61ec8700 1 -- 172.21.15.27:0/1216343216 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba5c1a7820 con 0x7fba5c101130 2024-04-25T00:14:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.855+0000 7fba58ff9700 1 -- 172.21.15.27:0/1216343216 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4c018680 con 0x7fba5c101130 2024-04-25T00:14:50.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.856+0000 7fba61ec8700 1 -- 172.21.15.27:0/1216343216 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba5c068f20 con 0x7fba5c101130 2024-04-25T00:14:50.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.857+0000 7fba58ff9700 1 -- 172.21.15.27:0/1216343216 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fba4c010430 con 0x7fba5c101130 2024-04-25T00:14:50.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.858+0000 7fba58ff9700 1 --2- 172.21.15.27:0/1216343216 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fba440a2d30 0x7fba440a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:50.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.858+0000 7fba5affd700 1 --2- 172.21.15.27:0/1216343216 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fba440a2d30 0x7fba440a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:50.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.858+0000 7fba58ff9700 1 -- 172.21.15.27:0/1216343216 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fba4c0cf860 con 0x7fba5c101130 2024-04-25T00:14:50.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.858+0000 7fba5affd700 1 --2- 172.21.15.27:0/1216343216 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fba440a2d30 0x7fba440a51f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fba5c06af80 tx=0x7fba5000b480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:50.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.861+0000 7fba58ff9700 1 -- 172.21.15.27:0/1216343216 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fba4c09c2a0 con 0x7fba5c101130 2024-04-25T00:14:50.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.862+0000 7f8761137700 1 -- 172.21.15.27:0/2004262592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f875c02db30 con 0x7f875c069f40 2024-04-25T00:14:50.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.863+0000 7f8743fff700 1 -- 172.21.15.27:0/2004262592 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f874c09c300 con 0x7f875c069f40 2024-04-25T00:14:50.864 INFO:teuthology.orchestra.run.smithi027.stdout:219043332148 2024-04-25T00:14:50.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.865+0000 7f8761137700 1 -- 172.21.15.27:0/2004262592 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f87440a2cb0 msgr2=0x7f87440a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:50.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.865+0000 7f8761137700 1 --2- 172.21.15.27:0/2004262592 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f87440a2cb0 0x7f87440a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f875000b5c0 tx=0x7f8750005fb0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.865+0000 7f8761137700 1 -- 172.21.15.27:0/2004262592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f875c069f40 msgr2=0x7f875c19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:50.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.865+0000 7f8761137700 1 --2- 172.21.15.27:0/2004262592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f875c069f40 0x7f875c19e670 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f874c00d980 tx=0x7f874c00dc90 comp rx=0 tx=0).stop 2024-04-25T00:14:50.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.865+0000 7f8761137700 1 -- 172.21.15.27:0/2004262592 shutdown_connections 2024-04-25T00:14:50.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.865+0000 7f8761137700 1 --2- 172.21.15.27:0/2004262592 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f87440a2cb0 0x7f87440a5170 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.865+0000 7f8761137700 1 --2- 172.21.15.27:0/2004262592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f875c1082b0 0x7f875c1987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.865+0000 7f8761137700 1 --2- 172.21.15.27:0/2004262592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f875c069f40 0x7f875c19e670 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.865+0000 7f8761137700 1 --2- 172.21.15.27:0/2004262592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f875c06a8e0 0x7f875c19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.865+0000 7f8761137700 1 -- 172.21.15.27:0/2004262592 >> 172.21.15.27:0/2004262592 conn(0x7f875c0780e0 msgr2=0x7f875c0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:50.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.866+0000 7f8761137700 1 -- 172.21.15.27:0/2004262592 shutdown_connections 2024-04-25T00:14:50.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:50.866+0000 7f8761137700 1 -- 172.21.15.27:0/2004262592 wait complete. 2024-04-25T00:14:51.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:51.089+0000 7fba61ec8700 1 -- 172.21.15.27:0/1216343216 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fba5c1a7ad0 con 0x7fba5c101130 2024-04-25T00:14:51.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:51.090+0000 7fba58ff9700 1 -- 172.21.15.27:0/1216343216 <== mon.2 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fba4c09bba0 con 0x7fba5c101130 2024-04-25T00:14:51.092 INFO:teuthology.orchestra.run.smithi027.stdout:343597383712 2024-04-25T00:14:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:51.093+0000 7fba61ec8700 1 -- 172.21.15.27:0/1216343216 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fba440a2d30 msgr2=0x7fba440a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:51.093+0000 7fba61ec8700 1 --2- 172.21.15.27:0/1216343216 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fba440a2d30 0x7fba440a51f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fba5c06af80 tx=0x7fba5000b480 comp rx=0 tx=0).stop 2024-04-25T00:14:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:51.093+0000 7fba61ec8700 1 -- 172.21.15.27:0/1216343216 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba5c101130 msgr2=0x7fba5c102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:51.093+0000 7fba61ec8700 1 --2- 172.21.15.27:0/1216343216 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba5c101130 0x7fba5c102fd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00e9f0 tx=0x7fba4c00ed00 comp rx=0 tx=0).stop 2024-04-25T00:14:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:51.093+0000 7fba61ec8700 1 -- 172.21.15.27:0/1216343216 shutdown_connections 2024-04-25T00:14:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:51.093+0000 7fba61ec8700 1 --2- 172.21.15.27:0/1216343216 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fba440a2d30 0x7fba440a51f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:51.093+0000 7fba61ec8700 1 --2- 172.21.15.27:0/1216343216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba5c1071a0 0x7fba5c103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:51.093+0000 7fba61ec8700 1 --2- 172.21.15.27:0/1216343216 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba5c101130 0x7fba5c102fd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:51.093+0000 7fba61ec8700 1 --2- 172.21.15.27:0/1216343216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba5c101af0 0x7fba5c103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:51.093+0000 7fba61ec8700 1 -- 172.21.15.27:0/1216343216 >> 172.21.15.27:0/1216343216 conn(0x7fba5c0780e0 msgr2=0x7fba5c105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:51.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:51.093+0000 7fba61ec8700 1 -- 172.21.15.27:0/1216343216 shutdown_connections 2024-04-25T00:14:51.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:51.093+0000 7fba61ec8700 1 -- 172.21.15.27:0/1216343216 wait complete. 2024-04-25T00:14:51.352 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:51 smithi027 ceph-mon[29208]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:14:51.352 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:51 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2004262592' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T00:14:51.352 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:51 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:51.352 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:51 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:51.352 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:51 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1216343216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T00:14:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:51 smithi139 ceph-mon[32833]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:14:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:51 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2004262592' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T00:14:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:51 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:51 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:51.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:51 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1216343216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T00:14:51.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:51 smithi153 ceph-mon[32896]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:14:51.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:51 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2004262592' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T00:14:51.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:51 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:51.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:51 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:51.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:51 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1216343216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T00:14:52.175 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070506 got 253403070510 for osd.8 2024-04-25T00:14:52.175 DEBUG:teuthology.parallel:result is None 2024-04-25T00:14:52.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:52 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:14:52.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:52 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:14:52.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:52 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:52.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.371+0000 7f18a6332700 1 -- 172.21.15.27:0/426721117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18a0107590 msgr2=0x7f18a010f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.371+0000 7f18a6332700 1 --2- 172.21.15.27:0/426721117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18a0107590 0x7f18a010f160 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f189800b0a0 tx=0x7f189800b3b0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.371+0000 7f18a6332700 1 -- 172.21.15.27:0/426721117 shutdown_connections 2024-04-25T00:14:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.371+0000 7f18a6332700 1 --2- 172.21.15.27:0/426721117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18a0107590 0x7f18a010f160 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.371+0000 7f18a6332700 1 --2- 172.21.15.27:0/426721117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a0102510 0x7f18a01028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.371+0000 7f18a6332700 1 --2- 172.21.15.27:0/426721117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f18a0102ec0 0x7f18a0106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.371+0000 7f18a6332700 1 -- 172.21.15.27:0/426721117 >> 172.21.15.27:0/426721117 conn(0x7f18a00793a0 msgr2=0x7f18a00797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.371+0000 7f18a6332700 1 -- 172.21.15.27:0/426721117 shutdown_connections 2024-04-25T00:14:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.371+0000 7f18a6332700 1 -- 172.21.15.27:0/426721117 wait complete. 2024-04-25T00:14:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.372+0000 7f18a6332700 1 Processor -- start 2024-04-25T00:14:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.372+0000 7f18a6332700 1 -- start start 2024-04-25T00:14:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.372+0000 7f18a6332700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f18a0102510 0x7f18a0074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.372+0000 7f18a6332700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18a0102ec0 0x7f18a0074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.372+0000 7f18a6332700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a0107590 0x7f18a0074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.372+0000 7f18a6332700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f18a01126a0 con 0x7f18a0102ec0 2024-04-25T00:14:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.372+0000 7f18a6332700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f18a0112520 con 0x7f18a0102510 2024-04-25T00:14:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.372+0000 7f18a6332700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f18a0112820 con 0x7f18a0107590 2024-04-25T00:14:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.373+0000 7f189f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18a0102ec0 0x7f18a0074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.373+0000 7f189f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18a0102ec0 0x7f18a0074a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45294/0 (socket says 172.21.15.27:45294) 2024-04-25T00:14:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.373+0000 7f189f7fe700 1 -- 172.21.15.27:0/2835071112 learned_addr learned my addr 172.21.15.27:0/2835071112 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.373+0000 7f189ffff700 1 --2- 172.21.15.27:0/2835071112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f18a0102510 0x7f18a0074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.373+0000 7f18a48cf700 1 --2- 172.21.15.27:0/2835071112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a0107590 0x7f18a0074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.373+0000 7f189f7fe700 1 -- 172.21.15.27:0/2835071112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a0107590 msgr2=0x7f18a0074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.373+0000 7f189f7fe700 1 --2- 172.21.15.27:0/2835071112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a0107590 0x7f18a0074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.373+0000 7f189f7fe700 1 -- 172.21.15.27:0/2835071112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f18a0102510 msgr2=0x7f18a0074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.373+0000 7f189f7fe700 1 --2- 172.21.15.27:0/2835071112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f18a0102510 0x7f18a0074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.373+0000 7f189f7fe700 1 -- 172.21.15.27:0/2835071112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a01a6fe0 con 0x7f18a0102ec0 2024-04-25T00:14:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.373+0000 7f18a48cf700 1 --2- 172.21.15.27:0/2835071112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a0107590 0x7f18a0074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:14:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.373+0000 7f189f7fe700 1 --2- 172.21.15.27:0/2835071112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18a0102ec0 0x7f18a0074a70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f189400b500 tx=0x7f189400b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.373+0000 7f189d7fa700 1 -- 172.21.15.27:0/2835071112 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1894015020 con 0x7f18a0102ec0 2024-04-25T00:14:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.373+0000 7f18a6332700 1 -- 172.21.15.27:0/2835071112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a01a72d0 con 0x7f18a0102ec0 2024-04-25T00:14:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.373+0000 7f189d7fa700 1 -- 172.21.15.27:0/2835071112 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1894011d10 con 0x7f18a0102ec0 2024-04-25T00:14:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.373+0000 7f189d7fa700 1 -- 172.21.15.27:0/2835071112 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189400f2e0 con 0x7f18a0102ec0 2024-04-25T00:14:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.373+0000 7f18a6332700 1 -- 172.21.15.27:0/2835071112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a01a7860 con 0x7f18a0102ec0 2024-04-25T00:14:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.374+0000 7f18a6332700 1 -- 172.21.15.27:0/2835071112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18a0103990 con 0x7f18a0102ec0 2024-04-25T00:14:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.375+0000 7f189d7fa700 1 -- 172.21.15.27:0/2835071112 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1894018020 con 0x7f18a0102ec0 2024-04-25T00:14:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.376+0000 7f189d7fa700 1 --2- 172.21.15.27:0/2835071112 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f18880a2cb0 0x7f18880a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.376+0000 7f189d7fa700 1 -- 172.21.15.27:0/2835071112 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f18940cebd0 con 0x7f18a0102ec0 2024-04-25T00:14:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.376+0000 7f189ffff700 1 --2- 172.21.15.27:0/2835071112 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f18880a2cb0 0x7f18880a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:52.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.377+0000 7f189ffff700 1 --2- 172.21.15.27:0/2835071112 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f18880a2cb0 0x7f18880a5170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1890009a10 tx=0x7f1890009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:52.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.380+0000 7f189d7fa700 1 -- 172.21.15.27:0/2835071112 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f189401f020 con 0x7f18a0102ec0 2024-04-25T00:14:52.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.594+0000 7f18a6332700 1 -- 172.21.15.27:0/2835071112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f18a0075a60 con 0x7f18a0102ec0 2024-04-25T00:14:52.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.594+0000 7f189d7fa700 1 -- 172.21.15.27:0/2835071112 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f189409b780 con 0x7f18a0102ec0 2024-04-25T00:14:52.596 INFO:teuthology.orchestra.run.smithi027.stdout:34359738457 2024-04-25T00:14:52.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.596+0000 7f18a6332700 1 -- 172.21.15.27:0/2835071112 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f18880a2cb0 msgr2=0x7f18880a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:52.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.596+0000 7f18a6332700 1 --2- 172.21.15.27:0/2835071112 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f18880a2cb0 0x7f18880a5170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1890009a10 tx=0x7f1890009190 comp rx=0 tx=0).stop 2024-04-25T00:14:52.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.596+0000 7f18a6332700 1 -- 172.21.15.27:0/2835071112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18a0102ec0 msgr2=0x7f18a0074a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:52.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.596+0000 7f18a6332700 1 --2- 172.21.15.27:0/2835071112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18a0102ec0 0x7f18a0074a70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f189400b500 tx=0x7f189400b810 comp rx=0 tx=0).stop 2024-04-25T00:14:52.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.597+0000 7f18a6332700 1 -- 172.21.15.27:0/2835071112 shutdown_connections 2024-04-25T00:14:52.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.597+0000 7f18a6332700 1 --2- 172.21.15.27:0/2835071112 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f18880a2cb0 0x7f18880a5170 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.597+0000 7f18a6332700 1 --2- 172.21.15.27:0/2835071112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a0107590 0x7f18a0074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.597+0000 7f18a6332700 1 --2- 172.21.15.27:0/2835071112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f18a0102510 0x7f18a0074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.597+0000 7f18a6332700 1 --2- 172.21.15.27:0/2835071112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18a0102ec0 0x7f18a0074a70 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.597+0000 7f18a6332700 1 -- 172.21.15.27:0/2835071112 >> 172.21.15.27:0/2835071112 conn(0x7f18a00793a0 msgr2=0x7f18a01002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.597+0000 7f18a6332700 1 -- 172.21.15.27:0/2835071112 shutdown_connections 2024-04-25T00:14:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:52.597+0000 7f18a6332700 1 -- 172.21.15.27:0/2835071112 wait complete. 2024-04-25T00:14:52.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:52 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:14:52.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:52 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:14:52.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:52 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:52.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:52 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:14:52.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:52 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:14:52.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:52 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' 2024-04-25T00:14:53.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:53 smithi027 ceph-mon[29208]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:14:53.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:53 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2835071112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T00:14:53.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:53 smithi139 ceph-mon[32833]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:14:53.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:53 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2835071112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T00:14:53.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:53 smithi153 ceph-mon[32896]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:14:53.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:53 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2835071112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T00:14:54.294 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383707 got 343597383712 for osd.11 2024-04-25T00:14:54.294 DEBUG:teuthology.parallel:result is None 2024-04-25T00:14:54.808 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738449 got 34359738457 for osd.0 2024-04-25T00:14:54.808 DEBUG:teuthology.parallel:result is None 2024-04-25T00:14:55.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.256+0000 7f2b71e79700 1 -- 172.21.15.27:0/2786582677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b6c101af0 msgr2=0x7f2b6c10dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:55.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.256+0000 7f2b71e79700 1 --2- 172.21.15.27:0/2786582677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b6c101af0 0x7f2b6c10dfe0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2b64009990 tx=0x7f2b64009ca0 comp rx=0 tx=0).stop 2024-04-25T00:14:55.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.256+0000 7f2b71e79700 1 -- 172.21.15.27:0/2786582677 shutdown_connections 2024-04-25T00:14:55.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.256+0000 7f2b71e79700 1 --2- 172.21.15.27:0/2786582677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b6c101af0 0x7f2b6c10dfe0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:55.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.256+0000 7f2b71e79700 1 --2- 172.21.15.27:0/2786582677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b6c1071a0 0x7f2b6c107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:55.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.256+0000 7f2b71e79700 1 --2- 172.21.15.27:0/2786582677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6c101130 0x7f2b6c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:55.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.256+0000 7f2b71e79700 1 -- 172.21.15.27:0/2786582677 >> 172.21.15.27:0/2786582677 conn(0x7f2b6c0780e0 msgr2=0x7f2b6c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:55.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.256+0000 7f2b71e79700 1 -- 172.21.15.27:0/2786582677 shutdown_connections 2024-04-25T00:14:55.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.256+0000 7f2b71e79700 1 -- 172.21.15.27:0/2786582677 wait complete. 2024-04-25T00:14:55.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.257+0000 7f2b71e79700 1 Processor -- start 2024-04-25T00:14:55.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.257+0000 7f2b71e79700 1 -- start start 2024-04-25T00:14:55.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.257+0000 7f2b71e79700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6c101130 0x7f2b6c102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:55.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.257+0000 7f2b71e79700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b6c101af0 0x7f2b6c103350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:55.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.257+0000 7f2b71e79700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b6c1071a0 0x7f2b6c103890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:55.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.257+0000 7f2b71e79700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6c111460 con 0x7f2b6c101af0 2024-04-25T00:14:55.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.257+0000 7f2b71e79700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6c1112e0 con 0x7f2b6c101130 2024-04-25T00:14:55.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.257+0000 7f2b71e79700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6c1115e0 con 0x7f2b6c1071a0 2024-04-25T00:14:55.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.257+0000 7f2b6affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b6c101af0 0x7f2b6c103350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:55.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.257+0000 7f2b6affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b6c101af0 0x7f2b6c103350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45310/0 (socket says 172.21.15.27:45310) 2024-04-25T00:14:55.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.257+0000 7f2b6b7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6c101130 0x7f2b6c102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:55.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.257+0000 7f2b6affd700 1 -- 172.21.15.27:0/2807046015 learned_addr learned my addr 172.21.15.27:0/2807046015 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:55.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.257+0000 7f2b6bfff700 1 --2- 172.21.15.27:0/2807046015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b6c1071a0 0x7f2b6c103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:55.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.258+0000 7f2b6b7fe700 1 -- 172.21.15.27:0/2807046015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b6c1071a0 msgr2=0x7f2b6c103890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:55.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.258+0000 7f2b6b7fe700 1 --2- 172.21.15.27:0/2807046015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b6c1071a0 0x7f2b6c103890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:55.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.258+0000 7f2b6b7fe700 1 -- 172.21.15.27:0/2807046015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b6c101af0 msgr2=0x7f2b6c103350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:55.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.258+0000 7f2b6b7fe700 1 --2- 172.21.15.27:0/2807046015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b6c101af0 0x7f2b6c103350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:55.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.258+0000 7f2b6b7fe700 1 -- 172.21.15.27:0/2807046015 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b6c1a6e80 con 0x7f2b6c101130 2024-04-25T00:14:55.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.258+0000 7f2b6affd700 1 --2- 172.21.15.27:0/2807046015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b6c101af0 0x7f2b6c103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:14:55.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.258+0000 7f2b6b7fe700 1 --2- 172.21.15.27:0/2807046015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6c101130 0x7f2b6c102e10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c007ec0 tx=0x7f2b5c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:55.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.258+0000 7f2b68ff9700 1 -- 172.21.15.27:0/2807046015 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c004db0 con 0x7f2b6c101130 2024-04-25T00:14:55.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.258+0000 7f2b71e79700 1 -- 172.21.15.27:0/2807046015 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b6c1a7170 con 0x7f2b6c101130 2024-04-25T00:14:55.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.258+0000 7f2b68ff9700 1 -- 172.21.15.27:0/2807046015 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b5c005020 con 0x7f2b6c101130 2024-04-25T00:14:55.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.258+0000 7f2b68ff9700 1 -- 172.21.15.27:0/2807046015 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c015660 con 0x7f2b6c101130 2024-04-25T00:14:55.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.258+0000 7f2b71e79700 1 -- 172.21.15.27:0/2807046015 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b6c1a76b0 con 0x7f2b6c101130 2024-04-25T00:14:55.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.259+0000 7f2b71e79700 1 -- 172.21.15.27:0/2807046015 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b6c068f20 con 0x7f2b6c101130 2024-04-25T00:14:55.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.261+0000 7f2b68ff9700 1 -- 172.21.15.27:0/2807046015 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2b5c005540 con 0x7f2b6c101130 2024-04-25T00:14:55.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.261+0000 7f2b68ff9700 1 --2- 172.21.15.27:0/2807046015 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2b580a2cb0 0x7f2b580a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:55.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.261+0000 7f2b6affd700 1 --2- 172.21.15.27:0/2807046015 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2b580a2cb0 0x7f2b580a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:55.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.261+0000 7f2b68ff9700 1 -- 172.21.15.27:0/2807046015 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2b5c0cf690 con 0x7f2b6c101130 2024-04-25T00:14:55.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.262+0000 7f2b6affd700 1 --2- 172.21.15.27:0/2807046015 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2b580a2cb0 0x7f2b580a5170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c0ff850 tx=0x7f2b54009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:55.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.265+0000 7f2b68ff9700 1 -- 172.21.15.27:0/2807046015 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2b5c010070 con 0x7f2b6c101130 2024-04-25T00:14:55.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:55 smithi027 ceph-mon[29208]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-25T00:14:55.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.476+0000 7f2b71e79700 1 -- 172.21.15.27:0/2807046015 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f2b6c1a79b0 con 0x7f2b6c101130 2024-04-25T00:14:55.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.477+0000 7f2b68ff9700 1 -- 172.21.15.27:0/2807046015 <== mon.2 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2b5c09c0d0 con 0x7f2b6c101130 2024-04-25T00:14:55.478 INFO:teuthology.orchestra.run.smithi027.stdout:107374182473 2024-04-25T00:14:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.479+0000 7f2b71e79700 1 -- 172.21.15.27:0/2807046015 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2b580a2cb0 msgr2=0x7f2b580a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.479+0000 7f2b71e79700 1 --2- 172.21.15.27:0/2807046015 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2b580a2cb0 0x7f2b580a5170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c0ff850 tx=0x7f2b54009260 comp rx=0 tx=0).stop 2024-04-25T00:14:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.479+0000 7f2b71e79700 1 -- 172.21.15.27:0/2807046015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6c101130 msgr2=0x7f2b6c102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.479+0000 7f2b71e79700 1 --2- 172.21.15.27:0/2807046015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6c101130 0x7f2b6c102e10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c007ec0 tx=0x7f2b5c00d3b0 comp rx=0 tx=0).stop 2024-04-25T00:14:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.479+0000 7f2b71e79700 1 -- 172.21.15.27:0/2807046015 shutdown_connections 2024-04-25T00:14:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.479+0000 7f2b71e79700 1 --2- 172.21.15.27:0/2807046015 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f2b580a2cb0 0x7f2b580a5170 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.479+0000 7f2b71e79700 1 --2- 172.21.15.27:0/2807046015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b6c1071a0 0x7f2b6c103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.479+0000 7f2b71e79700 1 --2- 172.21.15.27:0/2807046015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6c101130 0x7f2b6c102e10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.479+0000 7f2b71e79700 1 --2- 172.21.15.27:0/2807046015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b6c101af0 0x7f2b6c103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.479+0000 7f2b71e79700 1 -- 172.21.15.27:0/2807046015 >> 172.21.15.27:0/2807046015 conn(0x7f2b6c0780e0 msgr2=0x7f2b6c1002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.480+0000 7f2b71e79700 1 -- 172.21.15.27:0/2807046015 shutdown_connections 2024-04-25T00:14:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:55.480+0000 7f2b71e79700 1 -- 172.21.15.27:0/2807046015 wait complete. 2024-04-25T00:14:55.680 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332142 got 219043332148 for osd.7 2024-04-25T00:14:55.680 DEBUG:teuthology.parallel:result is None 2024-04-25T00:14:55.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:55 smithi139 ceph-mon[32833]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-25T00:14:55.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:55 smithi153 ceph-mon[32896]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-25T00:14:56.375 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182469 got 107374182473 for osd.3 2024-04-25T00:14:56.375 DEBUG:teuthology.parallel:result is None 2024-04-25T00:14:56.375 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T00:14:56.376 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T00:14:56.633 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:56.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:56 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2807046015' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T00:14:56.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:56 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2807046015' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T00:14:56.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:56 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2807046015' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T00:14:57.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:57 smithi027 ceph-mon[29208]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T00:14:57.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:57 smithi139 ceph-mon[32833]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T00:14:57.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:57 smithi153 ceph-mon[32896]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T00:14:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.297+0000 7f73e2d0b700 1 -- 172.21.15.27:0/2903209299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73dc1014c0 msgr2=0x7f73dc1018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.297+0000 7f73e2d0b700 1 --2- 172.21.15.27:0/2903209299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73dc1014c0 0x7f73dc1018a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f73cc0099c0 tx=0x7f73cc009cd0 comp rx=0 tx=0).stop 2024-04-25T00:14:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.298+0000 7f73e2d0b700 1 -- 172.21.15.27:0/2903209299 shutdown_connections 2024-04-25T00:14:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.298+0000 7f73e2d0b700 1 --2- 172.21.15.27:0/2903209299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73dc1064c0 0x7f73dc10e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.298+0000 7f73e2d0b700 1 --2- 172.21.15.27:0/2903209299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73dc1014c0 0x7f73dc1018a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.298+0000 7f73e2d0b700 1 --2- 172.21.15.27:0/2903209299 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73dc101e70 0x7f73dc105de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.298+0000 7f73e2d0b700 1 -- 172.21.15.27:0/2903209299 >> 172.21.15.27:0/2903209299 conn(0x7f73dc077040 msgr2=0x7f73dc077450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.298+0000 7f73e2d0b700 1 -- 172.21.15.27:0/2903209299 shutdown_connections 2024-04-25T00:14:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.298+0000 7f73e2d0b700 1 -- 172.21.15.27:0/2903209299 wait complete. 2024-04-25T00:14:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.298+0000 7f73e2d0b700 1 Processor -- start 2024-04-25T00:14:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.298+0000 7f73e2d0b700 1 -- start start 2024-04-25T00:14:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.298+0000 7f73e2d0b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73dc1014c0 0x7f73dc19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73e2d0b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73dc101e70 0x7f73dc19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73e2d0b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73dc1064c0 0x7f73dc1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73e2d0b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc112620 con 0x7f73dc1064c0 2024-04-25T00:14:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73e2d0b700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc1124a0 con 0x7f73dc101e70 2024-04-25T00:14:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73e2d0b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc1127a0 con 0x7f73dc1014c0 2024-04-25T00:14:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73e12a8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73dc1064c0 0x7f73dc1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73e12a8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73dc1064c0 0x7f73dc1988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48658/0 (socket says 172.21.15.27:48658) 2024-04-25T00:14:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73e12a8700 1 -- 172.21.15.27:0/1033528703 learned_addr learned my addr 172.21.15.27:0/1033528703 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:14:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73e0aa7700 1 --2- 172.21.15.27:0/1033528703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73dc1014c0 0x7f73dc19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73dbfff700 1 --2- 172.21.15.27:0/1033528703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73dc101e70 0x7f73dc19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73e12a8700 1 -- 172.21.15.27:0/1033528703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73dc1014c0 msgr2=0x7f73dc19e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73e12a8700 1 --2- 172.21.15.27:0/1033528703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73dc1014c0 0x7f73dc19e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73e12a8700 1 -- 172.21.15.27:0/1033528703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73dc101e70 msgr2=0x7f73dc19ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73e12a8700 1 --2- 172.21.15.27:0/1033528703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73dc101e70 0x7f73dc19ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73e12a8700 1 -- 172.21.15.27:0/1033528703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73dc199020 con 0x7f73dc1064c0 2024-04-25T00:14:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73e12a8700 1 --2- 172.21.15.27:0/1033528703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73dc1064c0 0x7f73dc1988c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f73d400b970 tx=0x7f73d400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73d9ffb700 1 -- 172.21.15.27:0/1033528703 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d4019070 con 0x7f73dc1064c0 2024-04-25T00:14:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73d9ffb700 1 -- 172.21.15.27:0/1033528703 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73d400ce10 con 0x7f73dc1064c0 2024-04-25T00:14:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73d9ffb700 1 -- 172.21.15.27:0/1033528703 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d4007370 con 0x7f73dc1064c0 2024-04-25T00:14:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.299+0000 7f73e2d0b700 1 -- 172.21.15.27:0/1033528703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73dc199310 con 0x7f73dc1064c0 2024-04-25T00:14:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.300+0000 7f73e2d0b700 1 -- 172.21.15.27:0/1033528703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73dc10e9a0 con 0x7f73dc1064c0 2024-04-25T00:14:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.301+0000 7f73e2d0b700 1 -- 172.21.15.27:0/1033528703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73dc068f20 con 0x7f73dc1064c0 2024-04-25T00:14:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.302+0000 7f73d9ffb700 1 -- 172.21.15.27:0/1033528703 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f73d401c070 con 0x7f73dc1064c0 2024-04-25T00:14:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.302+0000 7f73d9ffb700 1 --2- 172.21.15.27:0/1033528703 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f73c40a2d30 0x7f73c40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.303+0000 7f73e0aa7700 1 --2- 172.21.15.27:0/1033528703 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f73c40a2d30 0x7f73c40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.303+0000 7f73d9ffb700 1 -- 172.21.15.27:0/1033528703 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f73d40ce780 con 0x7f73dc1064c0 2024-04-25T00:14:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.303+0000 7f73e0aa7700 1 --2- 172.21.15.27:0/1033528703 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f73c40a2d30 0x7f73c40a51f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f73cc000c00 tx=0x7f73cc005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.307+0000 7f73d9ffb700 1 -- 172.21.15.27:0/1033528703 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f73d409b330 con 0x7f73dc1064c0 2024-04-25T00:14:58.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.511+0000 7f73e2d0b700 1 -- 172.21.15.27:0/1033528703 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f73dc199df0 con 0x7f73c40a2d30 2024-04-25T00:14:58.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.521+0000 7f73d9ffb700 1 -- 172.21.15.27:0/1033528703 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340678 (secure 0 0 0) 0x7f73dc199df0 con 0x7f73c40a2d30 2024-04-25T00:14:58.527 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:14:58.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.524+0000 7f73e2d0b700 1 -- 172.21.15.27:0/1033528703 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f73c40a2d30 msgr2=0x7f73c40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:58.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.524+0000 7f73e2d0b700 1 --2- 172.21.15.27:0/1033528703 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f73c40a2d30 0x7f73c40a51f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f73cc000c00 tx=0x7f73cc005fb0 comp rx=0 tx=0).stop 2024-04-25T00:14:58.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.524+0000 7f73e2d0b700 1 -- 172.21.15.27:0/1033528703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73dc1064c0 msgr2=0x7f73dc1988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:58.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.524+0000 7f73e2d0b700 1 --2- 172.21.15.27:0/1033528703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73dc1064c0 0x7f73dc1988c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f73d400b970 tx=0x7f73d400bc80 comp rx=0 tx=0).stop 2024-04-25T00:14:58.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.524+0000 7f73e2d0b700 1 -- 172.21.15.27:0/1033528703 shutdown_connections 2024-04-25T00:14:58.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.524+0000 7f73e2d0b700 1 --2- 172.21.15.27:0/1033528703 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f73c40a2d30 0x7f73c40a51f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:58.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.524+0000 7f73e2d0b700 1 --2- 172.21.15.27:0/1033528703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73dc1064c0 0x7f73dc1988c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:58.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.524+0000 7f73e2d0b700 1 --2- 172.21.15.27:0/1033528703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73dc1014c0 0x7f73dc19e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:58.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.524+0000 7f73e2d0b700 1 --2- 172.21.15.27:0/1033528703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73dc101e70 0x7f73dc19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:58.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.524+0000 7f73e2d0b700 1 -- 172.21.15.27:0/1033528703 >> 172.21.15.27:0/1033528703 conn(0x7f73dc077040 msgr2=0x7f73dc0ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:58.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.524+0000 7f73e2d0b700 1 -- 172.21.15.27:0/1033528703 shutdown_connections 2024-04-25T00:14:58.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:14:58.524+0000 7f73e2d0b700 1 -- 172.21.15.27:0/1033528703 wait complete. 2024-04-25T00:14:58.531 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T00:14:59.352 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-04-25T00:14:57.595655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658566,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":849,"num_read_kb":549,"num_write":421,"num_write_kb":3076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":346,"ondisk_log_size":346,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":349028,"kb_used_data":12284,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124592796,"statfs":{"total":1151940427776,"available":1151583023104,"internally_reserved":0,"allocated":12578816,"data_stored":7961504,"data_compressed":359538,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":344634681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":100,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003424"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263116+0000","last_change":"2024-04-25T00:13:19.429658+0000","last_active":"2024-04-25T00:14:11.263116+0000","last_peered":"2024-04-25T00:14:11.263116+0000","last_clean":"2024-04-25T00:14:11.263116+0000","last_became_active":"2024-04-25T00:13:19.428426+0000","last_became_peered":"2024-04-25T00:13:19.428426+0000","last_unstale":"2024-04-25T00:14:11.263116+0000","last_undegraded":"2024-04-25T00:14:11.263116+0000","last_fullsized":"2024-04-25T00:14:11.263116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:10.814361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263098+0000","last_change":"2024-04-25T00:13:19.425504+0000","last_active":"2024-04-25T00:14:11.263098+0000","last_peered":"2024-04-25T00:14:11.263098+0000","last_clean":"2024-04-25T00:14:11.263098+0000","last_became_active":"2024-04-25T00:13:19.425170+0000","last_became_peered":"2024-04-25T00:13:19.425170+0000","last_unstale":"2024-04-25T00:14:11.263098+0000","last_undegraded":"2024-04-25T00:14:11.263098+0000","last_fullsized":"2024-04-25T00:14:11.263098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:31:13.826981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263190+0000","last_change":"2024-04-25T00:13:19.420842+0000","last_active":"2024-04-25T00:14:11.263190+0000","last_peered":"2024-04-25T00:14:11.263190+0000","last_clean":"2024-04-25T00:14:11.263190+0000","last_became_active":"2024-04-25T00:13:19.420241+0000","last_became_peered":"2024-04-25T00:13:19.420241+0000","last_unstale":"2024-04-25T00:14:11.263190+0000","last_undegraded":"2024-04-25T00:14:11.263190+0000","last_fullsized":"2024-04-25T00:14:11.263190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:52:39.500432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.268192+0000","last_change":"2024-04-25T00:14:11.267688+0000","last_active":"2024-04-25T00:14:11.268192+0000","last_peered":"2024-04-25T00:14:11.268192+0000","last_clean":"2024-04-25T00:14:11.268192+0000","last_became_active":"2024-04-25T00:14:11.267225+0000","last_became_peered":"2024-04-25T00:14:11.267225+0000","last_unstale":"2024-04-25T00:14:11.268192+0000","last_undegraded":"2024-04-25T00:14:11.268192+0000","last_fullsized":"2024-04-25T00:14:11.268192+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:15.867489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.267718+0000","last_change":"2024-04-25T00:13:19.416519+0000","last_active":"2024-04-25T00:14:11.267718+0000","last_peered":"2024-04-25T00:14:11.267718+0000","last_clean":"2024-04-25T00:14:11.267718+0000","last_became_active":"2024-04-25T00:13:19.416080+0000","last_became_peered":"2024-04-25T00:13:19.416080+0000","last_unstale":"2024-04-25T00:14:11.267718+0000","last_undegraded":"2024-04-25T00:14:11.267718+0000","last_fullsized":"2024-04-25T00:14:11.267718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:04.450160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.270437+0000","last_change":"2024-04-25T00:13:19.424378+0000","last_active":"2024-04-25T00:14:11.270437+0000","last_peered":"2024-04-25T00:14:11.270437+0000","last_clean":"2024-04-25T00:14:11.270437+0000","last_became_active":"2024-04-25T00:13:19.423918+0000","last_became_peered":"2024-04-25T00:13:19.423918+0000","last_unstale":"2024-04-25T00:14:11.270437+0000","last_undegraded":"2024-04-25T00:14:11.270437+0000","last_fullsized":"2024-04-25T00:14:11.270437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:01:11.397328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265344+0000","last_change":"2024-04-25T00:13:19.424729+0000","last_active":"2024-04-25T00:14:11.265344+0000","last_peered":"2024-04-25T00:14:11.265344+0000","last_clean":"2024-04-25T00:14:11.265344+0000","last_became_active":"2024-04-25T00:13:19.422874+0000","last_became_peered":"2024-04-25T00:13:19.422874+0000","last_unstale":"2024-04-25T00:14:11.265344+0000","last_undegraded":"2024-04-25T00:14:11.265344+0000","last_fullsized":"2024-04-25T00:14:11.265344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:52:39.483589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.266615+0000","last_change":"2024-04-25T00:13:19.418043+0000","last_active":"2024-04-25T00:14:11.266615+0000","last_peered":"2024-04-25T00:14:11.266615+0000","last_clean":"2024-04-25T00:14:11.266615+0000","last_became_active":"2024-04-25T00:13:19.417443+0000","last_became_peered":"2024-04-25T00:13:19.417443+0000","last_unstale":"2024-04-25T00:14:11.266615+0000","last_undegraded":"2024-04-25T00:14:11.266615+0000","last_fullsized":"2024-04-25T00:14:11.266615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:47:53.141744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260673+0000","last_change":"2024-04-25T00:13:19.420989+0000","last_active":"2024-04-25T00:14:11.260673+0000","last_peered":"2024-04-25T00:14:11.260673+0000","last_clean":"2024-04-25T00:14:11.260673+0000","last_became_active":"2024-04-25T00:13:19.420669+0000","last_became_peered":"2024-04-25T00:13:19.420669+0000","last_unstale":"2024-04-25T00:14:11.260673+0000","last_undegraded":"2024-04-25T00:14:11.260673+0000","last_fullsized":"2024-04-25T00:14:11.260673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:18.236132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.271184+0000","last_change":"2024-04-25T00:13:19.424176+0000","last_active":"2024-04-25T00:14:11.271184+0000","last_peered":"2024-04-25T00:14:11.271184+0000","last_clean":"2024-04-25T00:14:11.271184+0000","last_became_active":"2024-04-25T00:13:19.423656+0000","last_became_peered":"2024-04-25T00:13:19.423656+0000","last_unstale":"2024-04-25T00:14:11.271184+0000","last_undegraded":"2024-04-25T00:14:11.271184+0000","last_fullsized":"2024-04-25T00:14:11.271184+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:24:41.697077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265720+0000","last_change":"2024-04-25T00:13:19.419624+0000","last_active":"2024-04-25T00:14:11.265720+0000","last_peered":"2024-04-25T00:14:11.265720+0000","last_clean":"2024-04-25T00:14:11.265720+0000","last_became_active":"2024-04-25T00:13:19.419263+0000","last_became_peered":"2024-04-25T00:13:19.419263+0000","last_unstale":"2024-04-25T00:14:11.265720+0000","last_undegraded":"2024-04-25T00:14:11.265720+0000","last_fullsized":"2024-04-25T00:14:11.265720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:29.130733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264264+0000","last_change":"2024-04-25T00:13:19.427865+0000","last_active":"2024-04-25T00:14:11.264264+0000","last_peered":"2024-04-25T00:14:11.264264+0000","last_clean":"2024-04-25T00:14:11.264264+0000","last_became_active":"2024-04-25T00:13:19.427538+0000","last_became_peered":"2024-04-25T00:13:19.427538+0000","last_unstale":"2024-04-25T00:14:11.264264+0000","last_undegraded":"2024-04-25T00:14:11.264264+0000","last_fullsized":"2024-04-25T00:14:11.264264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:05.136948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.266256+0000","last_change":"2024-04-25T00:13:19.419261+0000","last_active":"2024-04-25T00:14:11.266256+0000","last_peered":"2024-04-25T00:14:11.266256+0000","last_clean":"2024-04-25T00:14:11.266256+0000","last_became_active":"2024-04-25T00:13:19.418938+0000","last_became_peered":"2024-04-25T00:13:19.418938+0000","last_unstale":"2024-04-25T00:14:11.266256+0000","last_undegraded":"2024-04-25T00:14:11.266256+0000","last_fullsized":"2024-04-25T00:14:11.266256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:44.971640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262686+0000","last_change":"2024-04-25T00:13:19.417654+0000","last_active":"2024-04-25T00:14:11.262686+0000","last_peered":"2024-04-25T00:14:11.262686+0000","last_clean":"2024-04-25T00:14:11.262686+0000","last_became_active":"2024-04-25T00:13:19.417319+0000","last_became_peered":"2024-04-25T00:13:19.417319+0000","last_unstale":"2024-04-25T00:14:11.262686+0000","last_undegraded":"2024-04-25T00:14:11.262686+0000","last_fullsized":"2024-04-25T00:14:11.262686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:43:54.447937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262540+0000","last_change":"2024-04-25T00:13:19.420761+0000","last_active":"2024-04-25T00:14:11.262540+0000","last_peered":"2024-04-25T00:14:11.262540+0000","last_clean":"2024-04-25T00:14:11.262540+0000","last_became_active":"2024-04-25T00:13:19.420334+0000","last_became_peered":"2024-04-25T00:13:19.420334+0000","last_unstale":"2024-04-25T00:14:11.262540+0000","last_undegraded":"2024-04-25T00:14:11.262540+0000","last_fullsized":"2024-04-25T00:14:11.262540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:20.136470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260093+0000","last_change":"2024-04-25T00:13:19.420132+0000","last_active":"2024-04-25T00:14:11.260093+0000","last_peered":"2024-04-25T00:14:11.260093+0000","last_clean":"2024-04-25T00:14:11.260093+0000","last_became_active":"2024-04-25T00:13:19.418910+0000","last_became_peered":"2024-04-25T00:13:19.418910+0000","last_unstale":"2024-04-25T00:14:11.260093+0000","last_undegraded":"2024-04-25T00:14:11.260093+0000","last_fullsized":"2024-04-25T00:14:11.260093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:17.895412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262869+0000","last_change":"2024-04-25T00:13:19.430242+0000","last_active":"2024-04-25T00:14:11.262869+0000","last_peered":"2024-04-25T00:14:11.262869+0000","last_clean":"2024-04-25T00:14:11.262869+0000","last_became_active":"2024-04-25T00:13:19.429077+0000","last_became_peered":"2024-04-25T00:13:19.429077+0000","last_unstale":"2024-04-25T00:14:11.262869+0000","last_undegraded":"2024-04-25T00:14:11.262869+0000","last_fullsized":"2024-04-25T00:14:11.262869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:34:14.351157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262098+0000","last_change":"2024-04-25T00:13:19.414375+0000","last_active":"2024-04-25T00:14:11.262098+0000","last_peered":"2024-04-25T00:14:11.262098+0000","last_clean":"2024-04-25T00:14:11.262098+0000","last_became_active":"2024-04-25T00:13:19.414003+0000","last_became_peered":"2024-04-25T00:13:19.414003+0000","last_unstale":"2024-04-25T00:14:11.262098+0000","last_undegraded":"2024-04-25T00:14:11.262098+0000","last_fullsized":"2024-04-25T00:14:11.262098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:43:50.612743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260516+0000","last_change":"2024-04-25T00:13:19.415322+0000","last_active":"2024-04-25T00:14:11.260516+0000","last_peered":"2024-04-25T00:14:11.260516+0000","last_clean":"2024-04-25T00:14:11.260516+0000","last_became_active":"2024-04-25T00:13:19.414925+0000","last_became_peered":"2024-04-25T00:13:19.414925+0000","last_unstale":"2024-04-25T00:14:11.260516+0000","last_undegraded":"2024-04-25T00:14:11.260516+0000","last_fullsized":"2024-04-25T00:14:11.260516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:58.140640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.259607+0000","last_change":"2024-04-25T00:13:23.584763+0000","last_active":"2024-04-25T00:14:11.259607+0000","last_peered":"2024-04-25T00:14:11.259607+0000","last_clean":"2024-04-25T00:14:11.259607+0000","last_became_active":"2024-04-25T00:13:23.584303+0000","last_became_peered":"2024-04-25T00:13:23.584303+0000","last_unstale":"2024-04-25T00:14:11.259607+0000","last_undegraded":"2024-04-25T00:14:11.259607+0000","last_fullsized":"2024-04-25T00:14:11.259607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:54:11.447867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":107,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:51.677320+0000","last_change":"2024-04-25T00:13:18.519576+0000","last_active":"2024-04-25T00:14:51.677320+0000","last_peered":"2024-04-25T00:14:51.677320+0000","last_clean":"2024-04-25T00:14:51.677320+0000","last_became_active":"2024-04-25T00:13:18.519201+0000","last_became_peered":"2024-04-25T00:13:18.519201+0000","last_unstale":"2024-04-25T00:14:51.677320+0000","last_undegraded":"2024-04-25T00:14:51.677320+0000","last_fullsized":"2024-04-25T00:14:51.677320+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:19.681096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":67,"num_read_kb":67,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.267035+0000","last_change":"2024-04-25T00:13:19.419098+0000","last_active":"2024-04-25T00:14:11.267035+0000","last_peered":"2024-04-25T00:14:11.267035+0000","last_clean":"2024-04-25T00:14:11.267035+0000","last_became_active":"2024-04-25T00:13:19.418649+0000","last_became_peered":"2024-04-25T00:13:19.418649+0000","last_unstale":"2024-04-25T00:14:11.267035+0000","last_undegraded":"2024-04-25T00:14:11.267035+0000","last_fullsized":"2024-04-25T00:14:11.267035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:04.561470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263180+0000","last_change":"2024-04-25T00:13:23.580354+0000","last_active":"2024-04-25T00:14:11.263180+0000","last_peered":"2024-04-25T00:14:11.263180+0000","last_clean":"2024-04-25T00:14:11.263180+0000","last_became_active":"2024-04-25T00:13:23.579944+0000","last_became_peered":"2024-04-25T00:13:23.579944+0000","last_unstale":"2024-04-25T00:14:11.263180+0000","last_undegraded":"2024-04-25T00:14:11.263180+0000","last_fullsized":"2024-04-25T00:14:11.263180+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:48:57.788974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.267134+0000","last_change":"2024-04-25T00:13:18.527471+0000","last_active":"2024-04-25T00:14:11.267134+0000","last_peered":"2024-04-25T00:14:11.267134+0000","last_clean":"2024-04-25T00:14:11.267134+0000","last_became_active":"2024-04-25T00:13:18.527138+0000","last_became_peered":"2024-04-25T00:13:18.527138+0000","last_unstale":"2024-04-25T00:14:11.267134+0000","last_undegraded":"2024-04-25T00:14:11.267134+0000","last_fullsized":"2024-04-25T00:14:11.267134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:38.003719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262922+0000","last_change":"2024-04-25T00:13:19.420563+0000","last_active":"2024-04-25T00:14:11.262922+0000","last_peered":"2024-04-25T00:14:11.262922+0000","last_clean":"2024-04-25T00:14:11.262922+0000","last_became_active":"2024-04-25T00:13:19.420142+0000","last_became_peered":"2024-04-25T00:13:19.420142+0000","last_unstale":"2024-04-25T00:14:11.262922+0000","last_undegraded":"2024-04-25T00:14:11.262922+0000","last_fullsized":"2024-04-25T00:14:11.262922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:23:08.423795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.271321+0000","last_change":"2024-04-25T00:14:11.271321+0000","last_active":"2024-04-25T00:14:11.271321+0000","last_peered":"2024-04-25T00:14:11.271321+0000","last_clean":"2024-04-25T00:14:11.271321+0000","last_became_active":"2024-04-25T00:14:11.270478+0000","last_became_peered":"2024-04-25T00:14:11.270478+0000","last_unstale":"2024-04-25T00:14:11.271321+0000","last_undegraded":"2024-04-25T00:14:11.271321+0000","last_fullsized":"2024-04-25T00:14:11.271321+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:41.796155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265434+0000","last_change":"2024-04-25T00:13:18.520509+0000","last_active":"2024-04-25T00:14:11.265434+0000","last_peered":"2024-04-25T00:14:11.265434+0000","last_clean":"2024-04-25T00:14:11.265434+0000","last_became_active":"2024-04-25T00:13:18.520139+0000","last_became_peered":"2024-04-25T00:13:18.520139+0000","last_unstale":"2024-04-25T00:14:11.265434+0000","last_undegraded":"2024-04-25T00:14:11.265434+0000","last_fullsized":"2024-04-25T00:14:11.265434+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:08.797670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260383+0000","last_change":"2024-04-25T00:13:19.419115+0000","last_active":"2024-04-25T00:14:11.260383+0000","last_peered":"2024-04-25T00:14:11.260383+0000","last_clean":"2024-04-25T00:14:11.260383+0000","last_became_active":"2024-04-25T00:13:19.417865+0000","last_became_peered":"2024-04-25T00:13:19.417865+0000","last_unstale":"2024-04-25T00:14:11.260383+0000","last_undegraded":"2024-04-25T00:14:11.260383+0000","last_fullsized":"2024-04-25T00:14:11.260383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:20:11.173803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263726+0000","last_change":"2024-04-25T00:13:23.585052+0000","last_active":"2024-04-25T00:14:11.263726+0000","last_peered":"2024-04-25T00:14:11.263726+0000","last_clean":"2024-04-25T00:14:11.263726+0000","last_became_active":"2024-04-25T00:13:23.584696+0000","last_became_peered":"2024-04-25T00:13:23.584696+0000","last_unstale":"2024-04-25T00:14:11.263726+0000","last_undegraded":"2024-04-25T00:14:11.263726+0000","last_fullsized":"2024-04-25T00:14:11.263726+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:26.894183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264908+0000","last_change":"2024-04-25T00:13:18.522793+0000","last_active":"2024-04-25T00:14:11.264908+0000","last_peered":"2024-04-25T00:14:11.264908+0000","last_clean":"2024-04-25T00:14:11.264908+0000","last_became_active":"2024-04-25T00:13:18.522453+0000","last_became_peered":"2024-04-25T00:13:18.522453+0000","last_unstale":"2024-04-25T00:14:11.264908+0000","last_undegraded":"2024-04-25T00:14:11.264908+0000","last_fullsized":"2024-04-25T00:14:11.264908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:08.368214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263127+0000","last_change":"2024-04-25T00:13:19.418573+0000","last_active":"2024-04-25T00:14:11.263127+0000","last_peered":"2024-04-25T00:14:11.263127+0000","last_clean":"2024-04-25T00:14:11.263127+0000","last_became_active":"2024-04-25T00:13:19.418219+0000","last_became_peered":"2024-04-25T00:13:19.418219+0000","last_unstale":"2024-04-25T00:14:11.263127+0000","last_undegraded":"2024-04-25T00:14:11.263127+0000","last_fullsized":"2024-04-25T00:14:11.263127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:08:44.453017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260123+0000","last_change":"2024-04-25T00:13:23.578716+0000","last_active":"2024-04-25T00:14:11.260123+0000","last_peered":"2024-04-25T00:14:11.260123+0000","last_clean":"2024-04-25T00:14:11.260123+0000","last_became_active":"2024-04-25T00:13:23.578325+0000","last_became_peered":"2024-04-25T00:13:23.578325+0000","last_unstale":"2024-04-25T00:14:11.260123+0000","last_undegraded":"2024-04-25T00:14:11.260123+0000","last_fullsized":"2024-04-25T00:14:11.260123+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:39:57.145656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":101,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:51.656790+0000","last_change":"2024-04-25T00:13:18.527369+0000","last_active":"2024-04-25T00:14:51.656790+0000","last_peered":"2024-04-25T00:14:51.656790+0000","last_clean":"2024-04-25T00:14:51.656790+0000","last_became_active":"2024-04-25T00:13:18.526032+0000","last_became_peered":"2024-04-25T00:13:18.526032+0000","last_unstale":"2024-04-25T00:14:51.656790+0000","last_undegraded":"2024-04-25T00:14:51.656790+0000","last_fullsized":"2024-04-25T00:14:51.656790+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:52.635184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264866+0000","last_change":"2024-04-25T00:13:19.413305+0000","last_active":"2024-04-25T00:14:11.264866+0000","last_peered":"2024-04-25T00:14:11.264866+0000","last_clean":"2024-04-25T00:14:11.264866+0000","last_became_active":"2024-04-25T00:13:19.412956+0000","last_became_peered":"2024-04-25T00:13:19.412956+0000","last_unstale":"2024-04-25T00:14:11.264866+0000","last_undegraded":"2024-04-25T00:14:11.264866+0000","last_fullsized":"2024-04-25T00:14:11.264866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:54.436470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.273057+0000","last_change":"2024-04-25T00:14:11.273057+0000","last_active":"2024-04-25T00:14:11.273057+0000","last_peered":"2024-04-25T00:14:11.273057+0000","last_clean":"2024-04-25T00:14:11.273057+0000","last_became_active":"2024-04-25T00:14:11.272230+0000","last_became_peered":"2024-04-25T00:14:11.272230+0000","last_unstale":"2024-04-25T00:14:11.273057+0000","last_undegraded":"2024-04-25T00:14:11.273057+0000","last_fullsized":"2024-04-25T00:14:11.273057+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:37.695139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.270417+0000","last_change":"2024-04-25T00:13:18.517507+0000","last_active":"2024-04-25T00:14:11.270417+0000","last_peered":"2024-04-25T00:14:11.270417+0000","last_clean":"2024-04-25T00:14:11.270417+0000","last_became_active":"2024-04-25T00:13:18.517099+0000","last_became_peered":"2024-04-25T00:13:18.517099+0000","last_unstale":"2024-04-25T00:14:11.270417+0000","last_undegraded":"2024-04-25T00:14:11.270417+0000","last_fullsized":"2024-04-25T00:14:11.270417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:42:17.357610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.271691+0000","last_change":"2024-04-25T00:14:11.271691+0000","last_active":"2024-04-25T00:14:11.271691+0000","last_peered":"2024-04-25T00:14:11.271691+0000","last_clean":"2024-04-25T00:14:11.271691+0000","last_became_active":"2024-04-25T00:14:11.271160+0000","last_became_peered":"2024-04-25T00:14:11.271160+0000","last_unstale":"2024-04-25T00:14:11.271691+0000","last_undegraded":"2024-04-25T00:14:11.271691+0000","last_fullsized":"2024-04-25T00:14:11.271691+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:24.416810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262664+0000","last_change":"2024-04-25T00:13:23.582526+0000","last_active":"2024-04-25T00:14:11.262664+0000","last_peered":"2024-04-25T00:14:11.262664+0000","last_clean":"2024-04-25T00:14:11.262664+0000","last_became_active":"2024-04-25T00:13:23.581221+0000","last_became_peered":"2024-04-25T00:13:23.581221+0000","last_unstale":"2024-04-25T00:14:11.262664+0000","last_undegraded":"2024-04-25T00:14:11.262664+0000","last_fullsized":"2024-04-25T00:14:11.262664+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:46.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.266648+0000","last_change":"2024-04-25T00:13:18.522245+0000","last_active":"2024-04-25T00:14:11.266648+0000","last_peered":"2024-04-25T00:14:11.266648+0000","last_clean":"2024-04-25T00:14:11.266648+0000","last_became_active":"2024-04-25T00:13:18.521440+0000","last_became_peered":"2024-04-25T00:13:18.521440+0000","last_unstale":"2024-04-25T00:14:11.266648+0000","last_undegraded":"2024-04-25T00:14:11.266648+0000","last_fullsized":"2024-04-25T00:14:11.266648+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:21:37.847964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.266511+0000","last_change":"2024-04-25T00:13:19.422771+0000","last_active":"2024-04-25T00:14:11.266511+0000","last_peered":"2024-04-25T00:14:11.266511+0000","last_clean":"2024-04-25T00:14:11.266511+0000","last_became_active":"2024-04-25T00:13:19.422387+0000","last_became_peered":"2024-04-25T00:13:19.422387+0000","last_unstale":"2024-04-25T00:14:11.266511+0000","last_undegraded":"2024-04-25T00:14:11.266511+0000","last_fullsized":"2024-04-25T00:14:11.266511+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:15.647186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262943+0000","last_change":"2024-04-25T00:13:23.586235+0000","last_active":"2024-04-25T00:14:11.262943+0000","last_peered":"2024-04-25T00:14:11.262943+0000","last_clean":"2024-04-25T00:14:11.262943+0000","last_became_active":"2024-04-25T00:13:23.585803+0000","last_became_peered":"2024-04-25T00:13:23.585803+0000","last_unstale":"2024-04-25T00:14:11.262943+0000","last_undegraded":"2024-04-25T00:14:11.262943+0000","last_fullsized":"2024-04-25T00:14:11.262943+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:46.235435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264403+0000","last_change":"2024-04-25T00:13:18.525283+0000","last_active":"2024-04-25T00:14:11.264403+0000","last_peered":"2024-04-25T00:14:11.264403+0000","last_clean":"2024-04-25T00:14:11.264403+0000","last_became_active":"2024-04-25T00:13:18.524939+0000","last_became_peered":"2024-04-25T00:13:18.524939+0000","last_unstale":"2024-04-25T00:14:11.264403+0000","last_undegraded":"2024-04-25T00:14:11.264403+0000","last_fullsized":"2024-04-25T00:14:11.264403+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:09.258498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261187+0000","last_change":"2024-04-25T00:13:19.415746+0000","last_active":"2024-04-25T00:14:11.261187+0000","last_peered":"2024-04-25T00:14:11.261187+0000","last_clean":"2024-04-25T00:14:11.261187+0000","last_became_active":"2024-04-25T00:13:19.415325+0000","last_became_peered":"2024-04-25T00:13:19.415325+0000","last_unstale":"2024-04-25T00:14:11.261187+0000","last_undegraded":"2024-04-25T00:14:11.261187+0000","last_fullsized":"2024-04-25T00:14:11.261187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:17:48.268845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265732+0000","last_change":"2024-04-25T00:13:23.583153+0000","last_active":"2024-04-25T00:14:11.265732+0000","last_peered":"2024-04-25T00:14:11.265732+0000","last_clean":"2024-04-25T00:14:11.265732+0000","last_became_active":"2024-04-25T00:13:23.580257+0000","last_became_peered":"2024-04-25T00:13:23.580257+0000","last_unstale":"2024-04-25T00:14:11.265732+0000","last_undegraded":"2024-04-25T00:14:11.265732+0000","last_fullsized":"2024-04-25T00:14:11.265732+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:09:32.990253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":165,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:52.657561+0000","last_change":"2024-04-25T00:13:18.520365+0000","last_active":"2024-04-25T00:14:52.657561+0000","last_peered":"2024-04-25T00:14:52.657561+0000","last_clean":"2024-04-25T00:14:52.657561+0000","last_became_active":"2024-04-25T00:13:18.519918+0000","last_became_peered":"2024-04-25T00:13:18.519918+0000","last_unstale":"2024-04-25T00:14:52.657561+0000","last_undegraded":"2024-04-25T00:14:52.657561+0000","last_fullsized":"2024-04-25T00:14:52.657561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:46.415197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":134,"num_read_kb":134,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260684+0000","last_change":"2024-04-25T00:13:19.414341+0000","last_active":"2024-04-25T00:14:11.260684+0000","last_peered":"2024-04-25T00:14:11.260684+0000","last_clean":"2024-04-25T00:14:11.260684+0000","last_became_active":"2024-04-25T00:13:19.413793+0000","last_became_peered":"2024-04-25T00:13:19.413793+0000","last_unstale":"2024-04-25T00:14:11.260684+0000","last_undegraded":"2024-04-25T00:14:11.260684+0000","last_fullsized":"2024-04-25T00:14:11.260684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:55.927082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261020+0000","last_change":"2024-04-25T00:13:23.590631+0000","last_active":"2024-04-25T00:14:11.261020+0000","last_peered":"2024-04-25T00:14:11.261020+0000","last_clean":"2024-04-25T00:14:11.261020+0000","last_became_active":"2024-04-25T00:13:23.589980+0000","last_became_peered":"2024-04-25T00:13:23.589980+0000","last_unstale":"2024-04-25T00:14:11.261020+0000","last_undegraded":"2024-04-25T00:14:11.261020+0000","last_fullsized":"2024-04-25T00:14:11.261020+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:18.234048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265367+0000","last_change":"2024-04-25T00:13:18.526353+0000","last_active":"2024-04-25T00:14:11.265367+0000","last_peered":"2024-04-25T00:14:11.265367+0000","last_clean":"2024-04-25T00:14:11.265367+0000","last_became_active":"2024-04-25T00:13:18.525951+0000","last_became_peered":"2024-04-25T00:13:18.525951+0000","last_unstale":"2024-04-25T00:14:11.265367+0000","last_undegraded":"2024-04-25T00:14:11.265367+0000","last_fullsized":"2024-04-25T00:14:11.265367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:10:29.597797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262851+0000","last_change":"2024-04-25T00:13:19.425486+0000","last_active":"2024-04-25T00:14:11.262851+0000","last_peered":"2024-04-25T00:14:11.262851+0000","last_clean":"2024-04-25T00:14:11.262851+0000","last_became_active":"2024-04-25T00:13:19.425087+0000","last_became_peered":"2024-04-25T00:13:19.425087+0000","last_unstale":"2024-04-25T00:14:11.262851+0000","last_undegraded":"2024-04-25T00:14:11.262851+0000","last_fullsized":"2024-04-25T00:14:11.262851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:00.216683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264036+0000","last_change":"2024-04-25T00:13:23.579747+0000","last_active":"2024-04-25T00:14:11.264036+0000","last_peered":"2024-04-25T00:14:11.264036+0000","last_clean":"2024-04-25T00:14:11.264036+0000","last_became_active":"2024-04-25T00:13:23.579346+0000","last_became_peered":"2024-04-25T00:13:23.579346+0000","last_unstale":"2024-04-25T00:14:11.264036+0000","last_undegraded":"2024-04-25T00:14:11.264036+0000","last_fullsized":"2024-04-25T00:14:11.264036+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:10:18.969942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262985+0000","last_change":"2024-04-25T00:13:18.524707+0000","last_active":"2024-04-25T00:14:11.262985+0000","last_peered":"2024-04-25T00:14:11.262985+0000","last_clean":"2024-04-25T00:14:11.262985+0000","last_became_active":"2024-04-25T00:13:18.524352+0000","last_became_peered":"2024-04-25T00:13:18.524352+0000","last_unstale":"2024-04-25T00:14:11.262985+0000","last_undegraded":"2024-04-25T00:14:11.262985+0000","last_fullsized":"2024-04-25T00:14:11.262985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:29.992843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260591+0000","last_change":"2024-04-25T00:13:19.421845+0000","last_active":"2024-04-25T00:14:11.260591+0000","last_peered":"2024-04-25T00:14:11.260591+0000","last_clean":"2024-04-25T00:14:11.260591+0000","last_became_active":"2024-04-25T00:13:19.421458+0000","last_became_peered":"2024-04-25T00:13:19.421458+0000","last_unstale":"2024-04-25T00:14:11.260591+0000","last_undegraded":"2024-04-25T00:14:11.260591+0000","last_fullsized":"2024-04-25T00:14:11.260591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:09.286604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261234+0000","last_change":"2024-04-25T00:13:23.583721+0000","last_active":"2024-04-25T00:14:11.261234+0000","last_peered":"2024-04-25T00:14:11.261234+0000","last_clean":"2024-04-25T00:14:11.261234+0000","last_became_active":"2024-04-25T00:13:23.583340+0000","last_became_peered":"2024-04-25T00:13:23.583340+0000","last_unstale":"2024-04-25T00:14:11.261234+0000","last_undegraded":"2024-04-25T00:14:11.261234+0000","last_fullsized":"2024-04-25T00:14:11.261234+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:26.446468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'32","reported_seq":202,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:52.657321+0000","last_change":"2024-04-25T00:13:18.520653+0000","last_active":"2024-04-25T00:14:52.657321+0000","last_peered":"2024-04-25T00:14:52.657321+0000","last_clean":"2024-04-25T00:14:52.657321+0000","last_became_active":"2024-04-25T00:13:18.520278+0000","last_became_peered":"2024-04-25T00:13:18.520278+0000","last_unstale":"2024-04-25T00:14:52.657321+0000","last_undegraded":"2024-04-25T00:14:52.657321+0000","last_fullsized":"2024-04-25T00:14:52.657321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:58.311179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61151,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":143,"num_read_kb":131,"num_write":34,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264723+0000","last_change":"2024-04-25T00:13:19.421599+0000","last_active":"2024-04-25T00:14:11.264723+0000","last_peered":"2024-04-25T00:14:11.264723+0000","last_clean":"2024-04-25T00:14:11.264723+0000","last_became_active":"2024-04-25T00:13:19.421107+0000","last_became_peered":"2024-04-25T00:13:19.421107+0000","last_unstale":"2024-04-25T00:14:11.264723+0000","last_undegraded":"2024-04-25T00:14:11.264723+0000","last_fullsized":"2024-04-25T00:14:11.264723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:59.577987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263744+0000","last_change":"2024-04-25T00:13:23.580538+0000","last_active":"2024-04-25T00:14:11.263744+0000","last_peered":"2024-04-25T00:14:11.263744+0000","last_clean":"2024-04-25T00:14:11.263744+0000","last_became_active":"2024-04-25T00:13:23.580095+0000","last_became_peered":"2024-04-25T00:13:23.580095+0000","last_unstale":"2024-04-25T00:14:11.263744+0000","last_undegraded":"2024-04-25T00:14:11.263744+0000","last_fullsized":"2024-04-25T00:14:11.263744+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:04.189530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.259946+0000","last_change":"2024-04-25T00:13:18.522501+0000","last_active":"2024-04-25T00:14:11.259946+0000","last_peered":"2024-04-25T00:14:11.259946+0000","last_clean":"2024-04-25T00:14:11.259946+0000","last_became_active":"2024-04-25T00:13:18.522160+0000","last_became_peered":"2024-04-25T00:13:18.522160+0000","last_unstale":"2024-04-25T00:14:11.259946+0000","last_undegraded":"2024-04-25T00:14:11.259946+0000","last_fullsized":"2024-04-25T00:14:11.259946+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:31.876731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.266027+0000","last_change":"2024-04-25T00:13:19.417957+0000","last_active":"2024-04-25T00:14:11.266027+0000","last_peered":"2024-04-25T00:14:11.266027+0000","last_clean":"2024-04-25T00:14:11.266027+0000","last_became_active":"2024-04-25T00:13:19.417600+0000","last_became_peered":"2024-04-25T00:13:19.417600+0000","last_unstale":"2024-04-25T00:14:11.266027+0000","last_undegraded":"2024-04-25T00:14:11.266027+0000","last_fullsized":"2024-04-25T00:14:11.266027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:24.600965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262047+0000","last_change":"2024-04-25T00:13:23.582279+0000","last_active":"2024-04-25T00:14:11.262047+0000","last_peered":"2024-04-25T00:14:11.262047+0000","last_clean":"2024-04-25T00:14:11.262047+0000","last_became_active":"2024-04-25T00:13:23.581766+0000","last_became_peered":"2024-04-25T00:13:23.581766+0000","last_unstale":"2024-04-25T00:14:11.262047+0000","last_undegraded":"2024-04-25T00:14:11.262047+0000","last_fullsized":"2024-04-25T00:14:11.262047+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:36.115870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263249+0000","last_change":"2024-04-25T00:13:18.527443+0000","last_active":"2024-04-25T00:14:11.263249+0000","last_peered":"2024-04-25T00:14:11.263249+0000","last_clean":"2024-04-25T00:14:11.263249+0000","last_became_active":"2024-04-25T00:13:18.526919+0000","last_became_peered":"2024-04-25T00:13:18.526919+0000","last_unstale":"2024-04-25T00:14:11.263249+0000","last_undegraded":"2024-04-25T00:14:11.263249+0000","last_fullsized":"2024-04-25T00:14:11.263249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:32.737580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265626+0000","last_change":"2024-04-25T00:13:19.425947+0000","last_active":"2024-04-25T00:14:11.265626+0000","last_peered":"2024-04-25T00:14:11.265626+0000","last_clean":"2024-04-25T00:14:11.265626+0000","last_became_active":"2024-04-25T00:13:19.425613+0000","last_became_peered":"2024-04-25T00:13:19.425613+0000","last_unstale":"2024-04-25T00:14:11.265626+0000","last_undegraded":"2024-04-25T00:14:11.265626+0000","last_fullsized":"2024-04-25T00:14:11.265626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:28:34.729375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261732+0000","last_change":"2024-04-25T00:13:23.587794+0000","last_active":"2024-04-25T00:14:11.261732+0000","last_peered":"2024-04-25T00:14:11.261732+0000","last_clean":"2024-04-25T00:14:11.261732+0000","last_became_active":"2024-04-25T00:13:23.587334+0000","last_became_peered":"2024-04-25T00:13:23.587334+0000","last_unstale":"2024-04-25T00:14:11.261732+0000","last_undegraded":"2024-04-25T00:14:11.261732+0000","last_fullsized":"2024-04-25T00:14:11.261732+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:58.460439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:52.657350+0000","last_change":"2024-04-25T00:13:18.521701+0000","last_active":"2024-04-25T00:14:52.657350+0000","last_peered":"2024-04-25T00:14:52.657350+0000","last_clean":"2024-04-25T00:14:52.657350+0000","last_became_active":"2024-04-25T00:13:18.521360+0000","last_became_peered":"2024-04-25T00:13:18.521360+0000","last_unstale":"2024-04-25T00:14:52.657350+0000","last_undegraded":"2024-04-25T00:14:52.657350+0000","last_fullsized":"2024-04-25T00:14:52.657350+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:18:11.567426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265853+0000","last_change":"2024-04-25T00:13:19.424162+0000","last_active":"2024-04-25T00:14:11.265853+0000","last_peered":"2024-04-25T00:14:11.265853+0000","last_clean":"2024-04-25T00:14:11.265853+0000","last_became_active":"2024-04-25T00:13:19.423845+0000","last_became_peered":"2024-04-25T00:13:19.423845+0000","last_unstale":"2024-04-25T00:14:11.265853+0000","last_undegraded":"2024-04-25T00:14:11.265853+0000","last_fullsized":"2024-04-25T00:14:11.265853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:33.887577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.267758+0000","last_change":"2024-04-25T00:13:23.582102+0000","last_active":"2024-04-25T00:14:11.267758+0000","last_peered":"2024-04-25T00:14:11.267758+0000","last_clean":"2024-04-25T00:14:11.267758+0000","last_became_active":"2024-04-25T00:13:23.581679+0000","last_became_peered":"2024-04-25T00:13:23.581679+0000","last_unstale":"2024-04-25T00:14:11.267758+0000","last_undegraded":"2024-04-25T00:14:11.267758+0000","last_fullsized":"2024-04-25T00:14:11.267758+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:05:39.518634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265885+0000","last_change":"2024-04-25T00:13:18.518285+0000","last_active":"2024-04-25T00:14:11.265885+0000","last_peered":"2024-04-25T00:14:11.265885+0000","last_clean":"2024-04-25T00:14:11.265885+0000","last_became_active":"2024-04-25T00:13:18.517826+0000","last_became_peered":"2024-04-25T00:13:18.517826+0000","last_unstale":"2024-04-25T00:14:11.265885+0000","last_undegraded":"2024-04-25T00:14:11.265885+0000","last_fullsized":"2024-04-25T00:14:11.265885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:12.901225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.268444+0000","last_change":"2024-04-25T00:14:11.268444+0000","last_active":"2024-04-25T00:14:11.268444+0000","last_peered":"2024-04-25T00:14:11.268444+0000","last_clean":"2024-04-25T00:14:11.268444+0000","last_became_active":"2024-04-25T00:14:11.267934+0000","last_became_peered":"2024-04-25T00:14:11.267934+0000","last_unstale":"2024-04-25T00:14:11.268444+0000","last_undegraded":"2024-04-25T00:14:11.268444+0000","last_fullsized":"2024-04-25T00:14:11.268444+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:12:16.209890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265284+0000","last_change":"2024-04-25T00:13:23.578445+0000","last_active":"2024-04-25T00:14:11.265284+0000","last_peered":"2024-04-25T00:14:11.265284+0000","last_clean":"2024-04-25T00:14:11.265284+0000","last_became_active":"2024-04-25T00:13:23.578048+0000","last_became_peered":"2024-04-25T00:13:23.578048+0000","last_unstale":"2024-04-25T00:14:11.265284+0000","last_undegraded":"2024-04-25T00:14:11.265284+0000","last_fullsized":"2024-04-25T00:14:11.265284+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:55.072460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264494+0000","last_change":"2024-04-25T00:13:18.515939+0000","last_active":"2024-04-25T00:14:11.264494+0000","last_peered":"2024-04-25T00:14:11.264494+0000","last_clean":"2024-04-25T00:14:11.264494+0000","last_became_active":"2024-04-25T00:13:18.515597+0000","last_became_peered":"2024-04-25T00:13:18.515597+0000","last_unstale":"2024-04-25T00:14:11.264494+0000","last_undegraded":"2024-04-25T00:14:11.264494+0000","last_fullsized":"2024-04-25T00:14:11.264494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:03:30.266404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.269143+0000","last_change":"2024-04-25T00:14:11.269143+0000","last_active":"2024-04-25T00:14:11.269143+0000","last_peered":"2024-04-25T00:14:11.269143+0000","last_clean":"2024-04-25T00:14:11.269143+0000","last_became_active":"2024-04-25T00:14:11.268736+0000","last_became_peered":"2024-04-25T00:14:11.268736+0000","last_unstale":"2024-04-25T00:14:11.269143+0000","last_undegraded":"2024-04-25T00:14:11.269143+0000","last_fullsized":"2024-04-25T00:14:11.269143+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:19.559974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264834+0000","last_change":"2024-04-25T00:13:23.862576+0000","last_active":"2024-04-25T00:14:11.264834+0000","last_peered":"2024-04-25T00:14:11.264834+0000","last_clean":"2024-04-25T00:14:11.264834+0000","last_became_active":"2024-04-25T00:13:23.862024+0000","last_became_peered":"2024-04-25T00:13:23.862024+0000","last_unstale":"2024-04-25T00:14:11.264834+0000","last_undegraded":"2024-04-25T00:14:11.264834+0000","last_fullsized":"2024-04-25T00:14:11.264834+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:17.249680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261518+0000","last_change":"2024-04-25T00:13:18.522503+0000","last_active":"2024-04-25T00:14:11.261518+0000","last_peered":"2024-04-25T00:14:11.261518+0000","last_clean":"2024-04-25T00:14:11.261518+0000","last_became_active":"2024-04-25T00:13:18.522126+0000","last_became_peered":"2024-04-25T00:13:18.522126+0000","last_unstale":"2024-04-25T00:14:11.261518+0000","last_undegraded":"2024-04-25T00:14:11.261518+0000","last_fullsized":"2024-04-25T00:14:11.261518+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:47:57.530098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.271543+0000","last_change":"2024-04-25T00:13:19.416862+0000","last_active":"2024-04-25T00:14:11.271543+0000","last_peered":"2024-04-25T00:14:11.271543+0000","last_clean":"2024-04-25T00:14:11.271543+0000","last_became_active":"2024-04-25T00:13:19.416375+0000","last_became_peered":"2024-04-25T00:13:19.416375+0000","last_unstale":"2024-04-25T00:14:11.271543+0000","last_undegraded":"2024-04-25T00:14:11.271543+0000","last_fullsized":"2024-04-25T00:14:11.271543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:43:32.031623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.267401+0000","last_change":"2024-04-25T00:13:23.582387+0000","last_active":"2024-04-25T00:14:11.267401+0000","last_peered":"2024-04-25T00:14:11.267401+0000","last_clean":"2024-04-25T00:14:11.267401+0000","last_became_active":"2024-04-25T00:13:23.581984+0000","last_became_peered":"2024-04-25T00:13:23.581984+0000","last_unstale":"2024-04-25T00:14:11.267401+0000","last_undegraded":"2024-04-25T00:14:11.267401+0000","last_fullsized":"2024-04-25T00:14:11.267401+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:07:16.388293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":100,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:53.680660+0000","last_change":"2024-04-25T00:13:18.524121+0000","last_active":"2024-04-25T00:14:53.680660+0000","last_peered":"2024-04-25T00:14:53.680660+0000","last_clean":"2024-04-25T00:14:53.680660+0000","last_became_active":"2024-04-25T00:13:18.523731+0000","last_became_peered":"2024-04-25T00:13:18.523731+0000","last_unstale":"2024-04-25T00:14:53.680660+0000","last_undegraded":"2024-04-25T00:14:53.680660+0000","last_fullsized":"2024-04-25T00:14:53.680660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:22.105952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.270799+0000","last_change":"2024-04-25T00:13:19.424753+0000","last_active":"2024-04-25T00:14:11.270799+0000","last_peered":"2024-04-25T00:14:11.270799+0000","last_clean":"2024-04-25T00:14:11.270799+0000","last_became_active":"2024-04-25T00:13:19.423214+0000","last_became_peered":"2024-04-25T00:13:19.423214+0000","last_unstale":"2024-04-25T00:14:11.270799+0000","last_undegraded":"2024-04-25T00:14:11.270799+0000","last_fullsized":"2024-04-25T00:14:11.270799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:41.627442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.272431+0000","last_change":"2024-04-25T00:14:11.272431+0000","last_active":"2024-04-25T00:14:11.272431+0000","last_peered":"2024-04-25T00:14:11.272431+0000","last_clean":"2024-04-25T00:14:11.272431+0000","last_became_active":"2024-04-25T00:14:11.271997+0000","last_became_peered":"2024-04-25T00:14:11.271997+0000","last_unstale":"2024-04-25T00:14:11.272431+0000","last_undegraded":"2024-04-25T00:14:11.272431+0000","last_fullsized":"2024-04-25T00:14:11.272431+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:18.768009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264624+0000","last_change":"2024-04-25T00:13:19.420617+0000","last_active":"2024-04-25T00:14:11.264624+0000","last_peered":"2024-04-25T00:14:11.264624+0000","last_clean":"2024-04-25T00:14:11.264624+0000","last_became_active":"2024-04-25T00:13:19.420274+0000","last_became_peered":"2024-04-25T00:13:19.420274+0000","last_unstale":"2024-04-25T00:14:11.264624+0000","last_undegraded":"2024-04-25T00:14:11.264624+0000","last_fullsized":"2024-04-25T00:14:11.264624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:07.515196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:53.658208+0000","last_change":"2024-04-25T00:13:18.529753+0000","last_active":"2024-04-25T00:14:53.658208+0000","last_peered":"2024-04-25T00:14:53.658208+0000","last_clean":"2024-04-25T00:14:53.658208+0000","last_became_active":"2024-04-25T00:13:18.525521+0000","last_became_peered":"2024-04-25T00:13:18.525521+0000","last_unstale":"2024-04-25T00:14:53.658208+0000","last_undegraded":"2024-04-25T00:14:53.658208+0000","last_fullsized":"2024-04-25T00:14:53.658208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:17.044367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265467+0000","last_change":"2024-04-25T00:13:23.589613+0000","last_active":"2024-04-25T00:14:11.265467+0000","last_peered":"2024-04-25T00:14:11.265467+0000","last_clean":"2024-04-25T00:14:11.265467+0000","last_became_active":"2024-04-25T00:13:23.589323+0000","last_became_peered":"2024-04-25T00:13:23.589323+0000","last_unstale":"2024-04-25T00:14:11.265467+0000","last_undegraded":"2024-04-25T00:14:11.265467+0000","last_fullsized":"2024-04-25T00:14:11.265467+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:34:42.272706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.266378+0000","last_change":"2024-04-25T00:13:19.417775+0000","last_active":"2024-04-25T00:14:11.266378+0000","last_peered":"2024-04-25T00:14:11.266378+0000","last_clean":"2024-04-25T00:14:11.266378+0000","last_became_active":"2024-04-25T00:13:19.415096+0000","last_became_peered":"2024-04-25T00:13:19.415096+0000","last_unstale":"2024-04-25T00:14:11.266378+0000","last_undegraded":"2024-04-25T00:14:11.266378+0000","last_fullsized":"2024-04-25T00:14:11.266378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:54.975923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263907+0000","last_change":"2024-04-25T00:13:18.514560+0000","last_active":"2024-04-25T00:14:11.263907+0000","last_peered":"2024-04-25T00:14:11.263907+0000","last_clean":"2024-04-25T00:14:11.263907+0000","last_became_active":"2024-04-25T00:13:18.514034+0000","last_became_peered":"2024-04-25T00:13:18.514034+0000","last_unstale":"2024-04-25T00:14:11.263907+0000","last_undegraded":"2024-04-25T00:14:11.263907+0000","last_fullsized":"2024-04-25T00:14:11.263907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:35:18.110206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":359,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265210+0000","last_change":"2024-04-25T00:12:01.540807+0000","last_active":"2024-04-25T00:14:11.265210+0000","last_peered":"2024-04-25T00:14:11.265210+0000","last_clean":"2024-04-25T00:14:11.265210+0000","last_became_active":"2024-04-25T00:12:01.540272+0000","last_became_peered":"2024-04-25T00:12:01.540272+0000","last_unstale":"2024-04-25T00:14:11.265210+0000","last_undegraded":"2024-04-25T00:14:11.265210+0000","last_fullsized":"2024-04-25T00:14:11.265210+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:08:34.007429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:08:34.007429+0000","last_clean_scrub_stamp":"2024-04-25T00:08:34.007429+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:50:14.642604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263851+0000","last_change":"2024-04-25T00:13:23.862493+0000","last_active":"2024-04-25T00:14:11.263851+0000","last_peered":"2024-04-25T00:14:11.263851+0000","last_clean":"2024-04-25T00:14:11.263851+0000","last_became_active":"2024-04-25T00:13:23.861972+0000","last_became_peered":"2024-04-25T00:13:23.861972+0000","last_unstale":"2024-04-25T00:14:11.263851+0000","last_undegraded":"2024-04-25T00:14:11.263851+0000","last_fullsized":"2024-04-25T00:14:11.263851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:49:13.507357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265066+0000","last_change":"2024-04-25T00:13:19.421920+0000","last_active":"2024-04-25T00:14:11.265066+0000","last_peered":"2024-04-25T00:14:11.265066+0000","last_clean":"2024-04-25T00:14:11.265066+0000","last_became_active":"2024-04-25T00:13:19.421585+0000","last_became_peered":"2024-04-25T00:13:19.421585+0000","last_unstale":"2024-04-25T00:14:11.265066+0000","last_undegraded":"2024-04-25T00:14:11.265066+0000","last_fullsized":"2024-04-25T00:14:11.265066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:53:24.937520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265734+0000","last_change":"2024-04-25T00:13:18.527062+0000","last_active":"2024-04-25T00:14:11.265734+0000","last_peered":"2024-04-25T00:14:11.265734+0000","last_clean":"2024-04-25T00:14:11.265734+0000","last_became_active":"2024-04-25T00:13:18.526338+0000","last_became_peered":"2024-04-25T00:13:18.526338+0000","last_unstale":"2024-04-25T00:14:11.265734+0000","last_undegraded":"2024-04-25T00:14:11.265734+0000","last_fullsized":"2024-04-25T00:14:11.265734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:21.756728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263018+0000","last_change":"2024-04-25T00:13:23.575162+0000","last_active":"2024-04-25T00:14:11.263018+0000","last_peered":"2024-04-25T00:14:11.263018+0000","last_clean":"2024-04-25T00:14:11.263018+0000","last_became_active":"2024-04-25T00:13:23.574722+0000","last_became_peered":"2024-04-25T00:13:23.574722+0000","last_unstale":"2024-04-25T00:14:11.263018+0000","last_undegraded":"2024-04-25T00:14:11.263018+0000","last_fullsized":"2024-04-25T00:14:11.263018+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:52.958539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264851+0000","last_change":"2024-04-25T00:13:18.523019+0000","last_active":"2024-04-25T00:14:11.264851+0000","last_peered":"2024-04-25T00:14:11.264851+0000","last_clean":"2024-04-25T00:14:11.264851+0000","last_became_active":"2024-04-25T00:13:18.522677+0000","last_became_peered":"2024-04-25T00:13:18.522677+0000","last_unstale":"2024-04-25T00:14:11.264851+0000","last_undegraded":"2024-04-25T00:14:11.264851+0000","last_fullsized":"2024-04-25T00:14:11.264851+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:14.470154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.267567+0000","last_change":"2024-04-25T00:13:19.417672+0000","last_active":"2024-04-25T00:14:11.267567+0000","last_peered":"2024-04-25T00:14:11.267567+0000","last_clean":"2024-04-25T00:14:11.267567+0000","last_became_active":"2024-04-25T00:13:19.417323+0000","last_became_peered":"2024-04-25T00:13:19.417323+0000","last_unstale":"2024-04-25T00:14:11.267567+0000","last_undegraded":"2024-04-25T00:14:11.267567+0000","last_fullsized":"2024-04-25T00:14:11.267567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:53.734784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.266386+0000","last_change":"2024-04-25T00:13:23.862320+0000","last_active":"2024-04-25T00:14:11.266386+0000","last_peered":"2024-04-25T00:14:11.266386+0000","last_clean":"2024-04-25T00:14:11.266386+0000","last_became_active":"2024-04-25T00:13:23.861895+0000","last_became_peered":"2024-04-25T00:13:23.861895+0000","last_unstale":"2024-04-25T00:14:11.266386+0000","last_undegraded":"2024-04-25T00:14:11.266386+0000","last_fullsized":"2024-04-25T00:14:11.266386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:16:21.324727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261793+0000","last_change":"2024-04-25T00:13:18.520989+0000","last_active":"2024-04-25T00:14:11.261793+0000","last_peered":"2024-04-25T00:14:11.261793+0000","last_clean":"2024-04-25T00:14:11.261793+0000","last_became_active":"2024-04-25T00:13:18.520566+0000","last_became_peered":"2024-04-25T00:13:18.520566+0000","last_unstale":"2024-04-25T00:14:11.261793+0000","last_undegraded":"2024-04-25T00:14:11.261793+0000","last_fullsized":"2024-04-25T00:14:11.261793+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:22:32.532352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265660+0000","last_change":"2024-04-25T00:13:19.422741+0000","last_active":"2024-04-25T00:14:11.265660+0000","last_peered":"2024-04-25T00:14:11.265660+0000","last_clean":"2024-04-25T00:14:11.265660+0000","last_became_active":"2024-04-25T00:13:19.422379+0000","last_became_peered":"2024-04-25T00:13:19.422379+0000","last_unstale":"2024-04-25T00:14:11.265660+0000","last_undegraded":"2024-04-25T00:14:11.265660+0000","last_fullsized":"2024-04-25T00:14:11.265660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:41.337641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.259565+0000","last_change":"2024-04-25T00:13:23.577888+0000","last_active":"2024-04-25T00:14:11.259565+0000","last_peered":"2024-04-25T00:14:11.259565+0000","last_clean":"2024-04-25T00:14:11.259565+0000","last_became_active":"2024-04-25T00:13:23.577440+0000","last_became_peered":"2024-04-25T00:13:23.577440+0000","last_unstale":"2024-04-25T00:14:11.259565+0000","last_undegraded":"2024-04-25T00:14:11.259565+0000","last_fullsized":"2024-04-25T00:14:11.259565+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:03.760277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264429+0000","last_change":"2024-04-25T00:13:18.527996+0000","last_active":"2024-04-25T00:14:11.264429+0000","last_peered":"2024-04-25T00:14:11.264429+0000","last_clean":"2024-04-25T00:14:11.264429+0000","last_became_active":"2024-04-25T00:13:18.527636+0000","last_became_peered":"2024-04-25T00:13:18.527636+0000","last_unstale":"2024-04-25T00:14:11.264429+0000","last_undegraded":"2024-04-25T00:14:11.264429+0000","last_fullsized":"2024-04-25T00:14:11.264429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:52.137108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263053+0000","last_change":"2024-04-25T00:13:19.425538+0000","last_active":"2024-04-25T00:14:11.263053+0000","last_peered":"2024-04-25T00:14:11.263053+0000","last_clean":"2024-04-25T00:14:11.263053+0000","last_became_active":"2024-04-25T00:13:19.420790+0000","last_became_peered":"2024-04-25T00:13:19.420790+0000","last_unstale":"2024-04-25T00:14:11.263053+0000","last_undegraded":"2024-04-25T00:14:11.263053+0000","last_fullsized":"2024-04-25T00:14:11.263053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:04.319759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264168+0000","last_change":"2024-04-25T00:13:23.863187+0000","last_active":"2024-04-25T00:14:11.264168+0000","last_peered":"2024-04-25T00:14:11.264168+0000","last_clean":"2024-04-25T00:14:11.264168+0000","last_became_active":"2024-04-25T00:13:23.862206+0000","last_became_peered":"2024-04-25T00:13:23.862206+0000","last_unstale":"2024-04-25T00:14:11.264168+0000","last_undegraded":"2024-04-25T00:14:11.264168+0000","last_fullsized":"2024-04-25T00:14:11.264168+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:57:17.395007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260093+0000","last_change":"2024-04-25T00:13:18.529224+0000","last_active":"2024-04-25T00:14:11.260093+0000","last_peered":"2024-04-25T00:14:11.260093+0000","last_clean":"2024-04-25T00:14:11.260093+0000","last_became_active":"2024-04-25T00:13:18.528908+0000","last_became_peered":"2024-04-25T00:13:18.528908+0000","last_unstale":"2024-04-25T00:14:11.260093+0000","last_undegraded":"2024-04-25T00:14:11.260093+0000","last_fullsized":"2024-04-25T00:14:11.260093+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:27.035726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264608+0000","last_change":"2024-04-25T00:13:19.422223+0000","last_active":"2024-04-25T00:14:11.264608+0000","last_peered":"2024-04-25T00:14:11.264608+0000","last_clean":"2024-04-25T00:14:11.264608+0000","last_became_active":"2024-04-25T00:13:19.421524+0000","last_became_peered":"2024-04-25T00:13:19.421524+0000","last_unstale":"2024-04-25T00:14:11.264608+0000","last_undegraded":"2024-04-25T00:14:11.264608+0000","last_fullsized":"2024-04-25T00:14:11.264608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:17:01.849503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261900+0000","last_change":"2024-04-25T00:13:23.583015+0000","last_active":"2024-04-25T00:14:11.261900+0000","last_peered":"2024-04-25T00:14:11.261900+0000","last_clean":"2024-04-25T00:14:11.261900+0000","last_became_active":"2024-04-25T00:13:23.582612+0000","last_became_peered":"2024-04-25T00:13:23.582612+0000","last_unstale":"2024-04-25T00:14:11.261900+0000","last_undegraded":"2024-04-25T00:14:11.261900+0000","last_fullsized":"2024-04-25T00:14:11.261900+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:36.872281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265958+0000","last_change":"2024-04-25T00:13:18.530027+0000","last_active":"2024-04-25T00:14:11.265958+0000","last_peered":"2024-04-25T00:14:11.265958+0000","last_clean":"2024-04-25T00:14:11.265958+0000","last_became_active":"2024-04-25T00:13:18.529717+0000","last_became_peered":"2024-04-25T00:13:18.529717+0000","last_unstale":"2024-04-25T00:14:11.265958+0000","last_undegraded":"2024-04-25T00:14:11.265958+0000","last_fullsized":"2024-04-25T00:14:11.265958+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:23:31.020637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261938+0000","last_change":"2024-04-25T00:13:19.428650+0000","last_active":"2024-04-25T00:14:11.261938+0000","last_peered":"2024-04-25T00:14:11.261938+0000","last_clean":"2024-04-25T00:14:11.261938+0000","last_became_active":"2024-04-25T00:13:19.428156+0000","last_became_peered":"2024-04-25T00:13:19.428156+0000","last_unstale":"2024-04-25T00:14:11.261938+0000","last_undegraded":"2024-04-25T00:14:11.261938+0000","last_fullsized":"2024-04-25T00:14:11.261938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:43.094529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264014+0000","last_change":"2024-04-25T00:13:23.579125+0000","last_active":"2024-04-25T00:14:11.264014+0000","last_peered":"2024-04-25T00:14:11.264014+0000","last_clean":"2024-04-25T00:14:11.264014+0000","last_became_active":"2024-04-25T00:13:23.578728+0000","last_became_peered":"2024-04-25T00:13:23.578728+0000","last_unstale":"2024-04-25T00:14:11.264014+0000","last_undegraded":"2024-04-25T00:14:11.264014+0000","last_fullsized":"2024-04-25T00:14:11.264014+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:16:16.834399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.278099+0000","last_change":"2024-04-25T00:14:11.278099+0000","last_active":"2024-04-25T00:14:11.278099+0000","last_peered":"2024-04-25T00:14:11.278099+0000","last_clean":"2024-04-25T00:14:11.278099+0000","last_became_active":"2024-04-25T00:14:11.269977+0000","last_became_peered":"2024-04-25T00:14:11.269977+0000","last_unstale":"2024-04-25T00:14:11.278099+0000","last_undegraded":"2024-04-25T00:14:11.278099+0000","last_fullsized":"2024-04-25T00:14:11.278099+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:48:59.261854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.266793+0000","last_change":"2024-04-25T00:13:19.414848+0000","last_active":"2024-04-25T00:14:11.266793+0000","last_peered":"2024-04-25T00:14:11.266793+0000","last_clean":"2024-04-25T00:14:11.266793+0000","last_became_active":"2024-04-25T00:13:19.414370+0000","last_became_peered":"2024-04-25T00:13:19.414370+0000","last_unstale":"2024-04-25T00:14:11.266793+0000","last_undegraded":"2024-04-25T00:14:11.266793+0000","last_fullsized":"2024-04-25T00:14:11.266793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:41:08.779171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264049+0000","last_change":"2024-04-25T00:13:23.862577+0000","last_active":"2024-04-25T00:14:11.264049+0000","last_peered":"2024-04-25T00:14:11.264049+0000","last_clean":"2024-04-25T00:14:11.264049+0000","last_became_active":"2024-04-25T00:13:23.862098+0000","last_became_peered":"2024-04-25T00:13:23.862098+0000","last_unstale":"2024-04-25T00:14:11.264049+0000","last_undegraded":"2024-04-25T00:14:11.264049+0000","last_fullsized":"2024-04-25T00:14:11.264049+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:08:00.344041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.274790+0000","last_change":"2024-04-25T00:14:11.274790+0000","last_active":"2024-04-25T00:14:11.274790+0000","last_peered":"2024-04-25T00:14:11.274790+0000","last_clean":"2024-04-25T00:14:11.274790+0000","last_became_active":"2024-04-25T00:14:11.267132+0000","last_became_peered":"2024-04-25T00:14:11.267132+0000","last_unstale":"2024-04-25T00:14:11.274790+0000","last_undegraded":"2024-04-25T00:14:11.274790+0000","last_fullsized":"2024-04-25T00:14:11.274790+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:58.302788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265720+0000","last_change":"2024-04-25T00:13:19.427992+0000","last_active":"2024-04-25T00:14:11.265720+0000","last_peered":"2024-04-25T00:14:11.265720+0000","last_clean":"2024-04-25T00:14:11.265720+0000","last_became_active":"2024-04-25T00:13:19.427665+0000","last_became_peered":"2024-04-25T00:13:19.427665+0000","last_unstale":"2024-04-25T00:14:11.265720+0000","last_undegraded":"2024-04-25T00:14:11.265720+0000","last_fullsized":"2024-04-25T00:14:11.265720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:12:31.582185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260101+0000","last_change":"2024-04-25T00:13:23.582430+0000","last_active":"2024-04-25T00:14:11.260101+0000","last_peered":"2024-04-25T00:14:11.260101+0000","last_clean":"2024-04-25T00:14:11.260101+0000","last_became_active":"2024-04-25T00:13:23.582008+0000","last_became_peered":"2024-04-25T00:13:23.582008+0000","last_unstale":"2024-04-25T00:14:11.260101+0000","last_undegraded":"2024-04-25T00:14:11.260101+0000","last_fullsized":"2024-04-25T00:14:11.260101+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:11.068720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":100,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:51.675775+0000","last_change":"2024-04-25T00:13:18.528030+0000","last_active":"2024-04-25T00:14:51.675775+0000","last_peered":"2024-04-25T00:14:51.675775+0000","last_clean":"2024-04-25T00:14:51.675775+0000","last_became_active":"2024-04-25T00:13:18.527636+0000","last_became_peered":"2024-04-25T00:13:18.527636+0000","last_unstale":"2024-04-25T00:14:51.675775+0000","last_undegraded":"2024-04-25T00:14:51.675775+0000","last_fullsized":"2024-04-25T00:14:51.675775+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:36.311810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265767+0000","last_change":"2024-04-25T00:13:19.428181+0000","last_active":"2024-04-25T00:14:11.265767+0000","last_peered":"2024-04-25T00:14:11.265767+0000","last_clean":"2024-04-25T00:14:11.265767+0000","last_became_active":"2024-04-25T00:13:19.427785+0000","last_became_peered":"2024-04-25T00:13:19.427785+0000","last_unstale":"2024-04-25T00:14:11.265767+0000","last_undegraded":"2024-04-25T00:14:11.265767+0000","last_fullsized":"2024-04-25T00:14:11.265767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:31.745679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.271507+0000","last_change":"2024-04-25T00:14:11.271507+0000","last_active":"2024-04-25T00:14:11.271507+0000","last_peered":"2024-04-25T00:14:11.271507+0000","last_clean":"2024-04-25T00:14:11.271507+0000","last_became_active":"2024-04-25T00:14:11.270452+0000","last_became_peered":"2024-04-25T00:14:11.270452+0000","last_unstale":"2024-04-25T00:14:11.271507+0000","last_undegraded":"2024-04-25T00:14:11.271507+0000","last_fullsized":"2024-04-25T00:14:11.271507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:19.955242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263271+0000","last_change":"2024-04-25T00:13:18.529622+0000","last_active":"2024-04-25T00:14:11.263271+0000","last_peered":"2024-04-25T00:14:11.263271+0000","last_clean":"2024-04-25T00:14:11.263271+0000","last_became_active":"2024-04-25T00:13:18.529305+0000","last_became_peered":"2024-04-25T00:13:18.529305+0000","last_unstale":"2024-04-25T00:14:11.263271+0000","last_undegraded":"2024-04-25T00:14:11.263271+0000","last_fullsized":"2024-04-25T00:14:11.263271+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:10:43.334712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265609+0000","last_change":"2024-04-25T00:13:19.421783+0000","last_active":"2024-04-25T00:14:11.265609+0000","last_peered":"2024-04-25T00:14:11.265609+0000","last_clean":"2024-04-25T00:14:11.265609+0000","last_became_active":"2024-04-25T00:13:19.421447+0000","last_became_peered":"2024-04-25T00:13:19.421447+0000","last_unstale":"2024-04-25T00:14:11.265609+0000","last_undegraded":"2024-04-25T00:14:11.265609+0000","last_fullsized":"2024-04-25T00:14:11.265609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:16.107220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264096+0000","last_change":"2024-04-25T00:13:23.578252+0000","last_active":"2024-04-25T00:14:11.264096+0000","last_peered":"2024-04-25T00:14:11.264096+0000","last_clean":"2024-04-25T00:14:11.264096+0000","last_became_active":"2024-04-25T00:13:23.577779+0000","last_became_peered":"2024-04-25T00:13:23.577779+0000","last_unstale":"2024-04-25T00:14:11.264096+0000","last_undegraded":"2024-04-25T00:14:11.264096+0000","last_fullsized":"2024-04-25T00:14:11.264096+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:43.822447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.267645+0000","last_change":"2024-04-25T00:13:18.523583+0000","last_active":"2024-04-25T00:14:11.267645+0000","last_peered":"2024-04-25T00:14:11.267645+0000","last_clean":"2024-04-25T00:14:11.267645+0000","last_became_active":"2024-04-25T00:13:18.523238+0000","last_became_peered":"2024-04-25T00:13:18.523238+0000","last_unstale":"2024-04-25T00:14:11.267645+0000","last_undegraded":"2024-04-25T00:14:11.267645+0000","last_fullsized":"2024-04-25T00:14:11.267645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:03.869276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265969+0000","last_change":"2024-04-25T00:13:19.426021+0000","last_active":"2024-04-25T00:14:11.265969+0000","last_peered":"2024-04-25T00:14:11.265969+0000","last_clean":"2024-04-25T00:14:11.265969+0000","last_became_active":"2024-04-25T00:13:19.425676+0000","last_became_peered":"2024-04-25T00:13:19.425676+0000","last_unstale":"2024-04-25T00:14:11.265969+0000","last_undegraded":"2024-04-25T00:14:11.265969+0000","last_fullsized":"2024-04-25T00:14:11.265969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:10.044202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261982+0000","last_change":"2024-04-25T00:13:19.422402+0000","last_active":"2024-04-25T00:14:11.261982+0000","last_peered":"2024-04-25T00:14:11.261982+0000","last_clean":"2024-04-25T00:14:11.261982+0000","last_became_active":"2024-04-25T00:13:19.422057+0000","last_became_peered":"2024-04-25T00:13:19.422057+0000","last_unstale":"2024-04-25T00:14:11.261982+0000","last_undegraded":"2024-04-25T00:14:11.261982+0000","last_fullsized":"2024-04-25T00:14:11.261982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:31:43.703030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.259941+0000","last_change":"2024-04-25T00:13:19.425434+0000","last_active":"2024-04-25T00:14:11.259941+0000","last_peered":"2024-04-25T00:14:11.259941+0000","last_clean":"2024-04-25T00:14:11.259941+0000","last_became_active":"2024-04-25T00:13:19.425051+0000","last_became_peered":"2024-04-25T00:13:19.425051+0000","last_unstale":"2024-04-25T00:14:11.259941+0000","last_undegraded":"2024-04-25T00:14:11.259941+0000","last_fullsized":"2024-04-25T00:14:11.259941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:17.568415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.259653+0000","last_change":"2024-04-25T00:13:19.413835+0000","last_active":"2024-04-25T00:14:11.259653+0000","last_peered":"2024-04-25T00:14:11.259653+0000","last_clean":"2024-04-25T00:14:11.259653+0000","last_became_active":"2024-04-25T00:13:19.413406+0000","last_became_peered":"2024-04-25T00:13:19.413406+0000","last_unstale":"2024-04-25T00:14:11.259653+0000","last_undegraded":"2024-04-25T00:14:11.259653+0000","last_fullsized":"2024-04-25T00:14:11.259653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:33.852391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262898+0000","last_change":"2024-04-25T00:13:19.426875+0000","last_active":"2024-04-25T00:14:11.262898+0000","last_peered":"2024-04-25T00:14:11.262898+0000","last_clean":"2024-04-25T00:14:11.262898+0000","last_became_active":"2024-04-25T00:13:19.425828+0000","last_became_peered":"2024-04-25T00:13:19.425828+0000","last_unstale":"2024-04-25T00:14:11.262898+0000","last_undegraded":"2024-04-25T00:14:11.262898+0000","last_fullsized":"2024-04-25T00:14:11.262898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:21.119551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.267662+0000","last_change":"2024-04-25T00:13:19.423637+0000","last_active":"2024-04-25T00:14:11.267662+0000","last_peered":"2024-04-25T00:14:11.267662+0000","last_clean":"2024-04-25T00:14:11.267662+0000","last_became_active":"2024-04-25T00:13:19.423272+0000","last_became_peered":"2024-04-25T00:13:19.423272+0000","last_unstale":"2024-04-25T00:14:11.267662+0000","last_undegraded":"2024-04-25T00:14:11.267662+0000","last_fullsized":"2024-04-25T00:14:11.267662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:32:38.794966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262672+0000","last_change":"2024-04-25T00:13:19.423538+0000","last_active":"2024-04-25T00:14:11.262672+0000","last_peered":"2024-04-25T00:14:11.262672+0000","last_clean":"2024-04-25T00:14:11.262672+0000","last_became_active":"2024-04-25T00:13:19.423028+0000","last_became_peered":"2024-04-25T00:13:19.423028+0000","last_unstale":"2024-04-25T00:14:11.262672+0000","last_undegraded":"2024-04-25T00:14:11.262672+0000","last_fullsized":"2024-04-25T00:14:11.262672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:29:41.975044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260453+0000","last_change":"2024-04-25T00:13:19.416899+0000","last_active":"2024-04-25T00:14:11.260453+0000","last_peered":"2024-04-25T00:14:11.260453+0000","last_clean":"2024-04-25T00:14:11.260453+0000","last_became_active":"2024-04-25T00:13:19.416505+0000","last_became_peered":"2024-04-25T00:13:19.416505+0000","last_unstale":"2024-04-25T00:14:11.260453+0000","last_undegraded":"2024-04-25T00:14:11.260453+0000","last_fullsized":"2024-04-25T00:14:11.260453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:37:40.204119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265830+0000","last_change":"2024-04-25T00:13:19.426215+0000","last_active":"2024-04-25T00:14:11.265830+0000","last_peered":"2024-04-25T00:14:11.265830+0000","last_clean":"2024-04-25T00:14:11.265830+0000","last_became_active":"2024-04-25T00:13:19.422804+0000","last_became_peered":"2024-04-25T00:13:19.422804+0000","last_unstale":"2024-04-25T00:14:11.265830+0000","last_undegraded":"2024-04-25T00:14:11.265830+0000","last_fullsized":"2024-04-25T00:14:11.265830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:24:44.630891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264522+0000","last_change":"2024-04-25T00:13:19.413564+0000","last_active":"2024-04-25T00:14:11.264522+0000","last_peered":"2024-04-25T00:14:11.264522+0000","last_clean":"2024-04-25T00:14:11.264522+0000","last_became_active":"2024-04-25T00:13:19.413223+0000","last_became_peered":"2024-04-25T00:13:19.413223+0000","last_unstale":"2024-04-25T00:14:11.264522+0000","last_undegraded":"2024-04-25T00:14:11.264522+0000","last_fullsized":"2024-04-25T00:14:11.264522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:45.629018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265855+0000","last_change":"2024-04-25T00:13:19.425945+0000","last_active":"2024-04-25T00:14:11.265855+0000","last_peered":"2024-04-25T00:14:11.265855+0000","last_clean":"2024-04-25T00:14:11.265855+0000","last_became_active":"2024-04-25T00:13:19.422499+0000","last_became_peered":"2024-04-25T00:13:19.422499+0000","last_unstale":"2024-04-25T00:14:11.265855+0000","last_undegraded":"2024-04-25T00:14:11.265855+0000","last_fullsized":"2024-04-25T00:14:11.265855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:45.299300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260372+0000","last_change":"2024-04-25T00:13:19.418857+0000","last_active":"2024-04-25T00:14:11.260372+0000","last_peered":"2024-04-25T00:14:11.260372+0000","last_clean":"2024-04-25T00:14:11.260372+0000","last_became_active":"2024-04-25T00:13:19.418535+0000","last_became_peered":"2024-04-25T00:13:19.418535+0000","last_unstale":"2024-04-25T00:14:11.260372+0000","last_undegraded":"2024-04-25T00:14:11.260372+0000","last_fullsized":"2024-04-25T00:14:11.260372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:44.288129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265861+0000","last_change":"2024-04-25T00:13:19.420550+0000","last_active":"2024-04-25T00:14:11.265861+0000","last_peered":"2024-04-25T00:14:11.265861+0000","last_clean":"2024-04-25T00:14:11.265861+0000","last_became_active":"2024-04-25T00:13:19.420235+0000","last_became_peered":"2024-04-25T00:13:19.420235+0000","last_unstale":"2024-04-25T00:14:11.265861+0000","last_undegraded":"2024-04-25T00:14:11.265861+0000","last_fullsized":"2024-04-25T00:14:11.265861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:59:36.543776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265433+0000","last_change":"2024-04-25T00:13:19.429995+0000","last_active":"2024-04-25T00:14:11.265433+0000","last_peered":"2024-04-25T00:14:11.265433+0000","last_clean":"2024-04-25T00:14:11.265433+0000","last_became_active":"2024-04-25T00:13:19.429185+0000","last_became_peered":"2024-04-25T00:13:19.429185+0000","last_unstale":"2024-04-25T00:14:11.265433+0000","last_undegraded":"2024-04-25T00:14:11.265433+0000","last_fullsized":"2024-04-25T00:14:11.265433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:29:27.974479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":213064,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67964,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":713,"num_read_kb":430,"num_write":238,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":203892,"data_compressed":4806,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":501433,"data_compressed":20615,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645349,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28140,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717012,"statfs":{"total":95995035648,"available":95966220288,"internally_reserved":0,"allocated":1146880,"data_stored":1149791,"data_compressed":58001,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808875,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":505459,"data_compressed":20615,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070513,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":501741,"data_compressed":20615,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":1098238,"data_compressed":56399,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495675,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":496867,"data_compressed":20615,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":559048,"data_compressed":22217,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":498323,"data_compressed":20615,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":559268,"data_compressed":22217,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":496575,"data_compressed":20615,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":501332,"data_compressed":20615,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1085440,"data_stored":1093429,"data_compressed":56399,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62903,"data_compressed":1602,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":63123,"data_compressed":1602,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":63278,"data_compressed":1602,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:14:59.359 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T00:14:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:14:59.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:14:59 smithi027 ceph-mon[29208]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:14:59.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:14:59 smithi139 ceph-mon[32833]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:14:59.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:14:59 smithi153 ceph-mon[32896]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:15:00.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:00 smithi027 ceph-mon[29208]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:15:00.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:00 smithi139 ceph-mon[32833]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:15:00.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:00 smithi153 ceph-mon[32896]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:15:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.382+0000 7f93f04f1700 1 -- 172.21.15.27:0/2119015863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93e8101120 msgr2=0x7f93e81015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.382+0000 7f93f04f1700 1 --2- 172.21.15.27:0/2119015863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93e8101120 0x7f93e81015a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f93d8009990 tx=0x7f93d8009ca0 comp rx=0 tx=0).stop 2024-04-25T00:15:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.382+0000 7f93f04f1700 1 -- 172.21.15.27:0/2119015863 shutdown_connections 2024-04-25T00:15:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.382+0000 7f93f04f1700 1 --2- 172.21.15.27:0/2119015863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93e8101ae0 0x7f93e810dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.382+0000 7f93f04f1700 1 --2- 172.21.15.27:0/2119015863 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f93e8107190 0x7f93e8107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.382+0000 7f93f04f1700 1 --2- 172.21.15.27:0/2119015863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93e8101120 0x7f93e81015a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.382+0000 7f93f04f1700 1 -- 172.21.15.27:0/2119015863 >> 172.21.15.27:0/2119015863 conn(0x7f93e80780a0 msgr2=0x7f93e80784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.382+0000 7f93f04f1700 1 -- 172.21.15.27:0/2119015863 shutdown_connections 2024-04-25T00:15:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.382+0000 7f93f04f1700 1 -- 172.21.15.27:0/2119015863 wait complete. 2024-04-25T00:15:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.383+0000 7f93f04f1700 1 Processor -- start 2024-04-25T00:15:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.383+0000 7f93f04f1700 1 -- start start 2024-04-25T00:15:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.383+0000 7f93f04f1700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f93e8101120 0x7f93e819d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.383+0000 7f93f04f1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93e8101ae0 0x7f93e819db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.383+0000 7f93f04f1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93e8107190 0x7f93e8197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.383+0000 7f93f04f1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f93e8111430 con 0x7f93e8101ae0 2024-04-25T00:15:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.383+0000 7f93f04f1700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f93e81112b0 con 0x7f93e8101120 2024-04-25T00:15:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.383+0000 7f93f04f1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f93e81115b0 con 0x7f93e8107190 2024-04-25T00:15:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.383+0000 7f93eda8c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93e8101ae0 0x7f93e819db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.383+0000 7f93eda8c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93e8101ae0 0x7f93e819db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48686/0 (socket says 172.21.15.27:48686) 2024-04-25T00:15:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.383+0000 7f93eda8c700 1 -- 172.21.15.27:0/1942038891 learned_addr learned my addr 172.21.15.27:0/1942038891 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.383+0000 7f93ee28d700 1 --2- 172.21.15.27:0/1942038891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f93e8101120 0x7f93e819d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.383+0000 7f93eea8e700 1 --2- 172.21.15.27:0/1942038891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93e8107190 0x7f93e8197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.384+0000 7f93eda8c700 1 -- 172.21.15.27:0/1942038891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93e8107190 msgr2=0x7f93e8197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.384+0000 7f93eda8c700 1 --2- 172.21.15.27:0/1942038891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93e8107190 0x7f93e8197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.384+0000 7f93eda8c700 1 -- 172.21.15.27:0/1942038891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f93e8101120 msgr2=0x7f93e819d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.384+0000 7f93eda8c700 1 --2- 172.21.15.27:0/1942038891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f93e8101120 0x7f93e819d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.384+0000 7f93eda8c700 1 -- 172.21.15.27:0/1942038891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e8197ff0 con 0x7f93e8101ae0 2024-04-25T00:15:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.384+0000 7f93eda8c700 1 --2- 172.21.15.27:0/1942038891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93e8101ae0 0x7f93e819db30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f93d80038b0 tx=0x7f93d8009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.384+0000 7f93df7fe700 1 -- 172.21.15.27:0/1942038891 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d8027070 con 0x7f93e8101ae0 2024-04-25T00:15:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.384+0000 7f93f04f1700 1 -- 172.21.15.27:0/1942038891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93e8198280 con 0x7f93e8101ae0 2024-04-25T00:15:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.384+0000 7f93df7fe700 1 -- 172.21.15.27:0/1942038891 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93d8004050 con 0x7f93e8101ae0 2024-04-25T00:15:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.384+0000 7f93df7fe700 1 -- 172.21.15.27:0/1942038891 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d8022cd0 con 0x7f93e8101ae0 2024-04-25T00:15:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.384+0000 7f93f04f1700 1 -- 172.21.15.27:0/1942038891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93e81a43a0 con 0x7f93e8101ae0 2024-04-25T00:15:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.385+0000 7f93f04f1700 1 -- 172.21.15.27:0/1942038891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93e8102840 con 0x7f93e8101ae0 2024-04-25T00:15:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.386+0000 7f93df7fe700 1 -- 172.21.15.27:0/1942038891 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f93d802a070 con 0x7f93e8101ae0 2024-04-25T00:15:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.387+0000 7f93df7fe700 1 --2- 172.21.15.27:0/1942038891 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f93d40a2cb0 0x7f93d40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.387+0000 7f93ee28d700 1 --2- 172.21.15.27:0/1942038891 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f93d40a2cb0 0x7f93d40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.387+0000 7f93df7fe700 1 -- 172.21.15.27:0/1942038891 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f93d80d0330 con 0x7f93e8101ae0 2024-04-25T00:15:01.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.387+0000 7f93ee28d700 1 --2- 172.21.15.27:0/1942038891 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f93d40a2cb0 0x7f93d40a5170 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f93e4005950 tx=0x7f93e400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:01.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.391+0000 7f93df7fe700 1 -- 172.21.15.27:0/1942038891 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f93d805e740 con 0x7f93e8101ae0 2024-04-25T00:15:01.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.594+0000 7f93f04f1700 1 -- 172.21.15.27:0/1942038891 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f93e8198bf0 con 0x7f93d40a2cb0 2024-04-25T00:15:01.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.604+0000 7f93df7fe700 1 -- 172.21.15.27:0/1942038891 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340678 (secure 0 0 0) 0x7f93e8198bf0 con 0x7f93d40a2cb0 2024-04-25T00:15:01.608 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:01.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.607+0000 7f93f04f1700 1 -- 172.21.15.27:0/1942038891 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f93d40a2cb0 msgr2=0x7f93d40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:01.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.607+0000 7f93f04f1700 1 --2- 172.21.15.27:0/1942038891 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f93d40a2cb0 0x7f93d40a5170 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f93e4005950 tx=0x7f93e400b410 comp rx=0 tx=0).stop 2024-04-25T00:15:01.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.607+0000 7f93f04f1700 1 -- 172.21.15.27:0/1942038891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93e8101ae0 msgr2=0x7f93e819db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:01.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.607+0000 7f93f04f1700 1 --2- 172.21.15.27:0/1942038891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93e8101ae0 0x7f93e819db30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f93d80038b0 tx=0x7f93d8009f50 comp rx=0 tx=0).stop 2024-04-25T00:15:01.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.607+0000 7f93f04f1700 1 -- 172.21.15.27:0/1942038891 shutdown_connections 2024-04-25T00:15:01.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.607+0000 7f93f04f1700 1 --2- 172.21.15.27:0/1942038891 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f93d40a2cb0 0x7f93d40a5170 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:01.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.607+0000 7f93f04f1700 1 --2- 172.21.15.27:0/1942038891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93e8107190 0x7f93e8197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:01.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.607+0000 7f93f04f1700 1 --2- 172.21.15.27:0/1942038891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f93e8101120 0x7f93e819d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:01.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.607+0000 7f93f04f1700 1 --2- 172.21.15.27:0/1942038891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93e8101ae0 0x7f93e819db30 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:01.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.607+0000 7f93f04f1700 1 -- 172.21.15.27:0/1942038891 >> 172.21.15.27:0/1942038891 conn(0x7f93e80780a0 msgr2=0x7f93e80ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:01.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.607+0000 7f93f04f1700 1 -- 172.21.15.27:0/1942038891 shutdown_connections 2024-04-25T00:15:01.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:01.607+0000 7f93f04f1700 1 -- 172.21.15.27:0/1942038891 wait complete. 2024-04-25T00:15:01.612 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T00:15:01.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:01 smithi027 ceph-mon[29208]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:15:01.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:01 smithi139 ceph-mon[32833]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:15:01.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:01 smithi153 ceph-mon[32896]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:15:02.371 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-04-25T00:14:59.596247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658566,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":889,"num_read_kb":569,"num_write":421,"num_write_kb":3076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":346,"ondisk_log_size":346,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":349028,"kb_used_data":12284,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124592796,"statfs":{"total":1151940427776,"available":1151583023104,"internally_reserved":0,"allocated":12578816,"data_stored":7961504,"data_compressed":359538,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":344634681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":70,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003432"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263116+0000","last_change":"2024-04-25T00:13:19.429658+0000","last_active":"2024-04-25T00:14:11.263116+0000","last_peered":"2024-04-25T00:14:11.263116+0000","last_clean":"2024-04-25T00:14:11.263116+0000","last_became_active":"2024-04-25T00:13:19.428426+0000","last_became_peered":"2024-04-25T00:13:19.428426+0000","last_unstale":"2024-04-25T00:14:11.263116+0000","last_undegraded":"2024-04-25T00:14:11.263116+0000","last_fullsized":"2024-04-25T00:14:11.263116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:10.814361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263098+0000","last_change":"2024-04-25T00:13:19.425504+0000","last_active":"2024-04-25T00:14:11.263098+0000","last_peered":"2024-04-25T00:14:11.263098+0000","last_clean":"2024-04-25T00:14:11.263098+0000","last_became_active":"2024-04-25T00:13:19.425170+0000","last_became_peered":"2024-04-25T00:13:19.425170+0000","last_unstale":"2024-04-25T00:14:11.263098+0000","last_undegraded":"2024-04-25T00:14:11.263098+0000","last_fullsized":"2024-04-25T00:14:11.263098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:31:13.826981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263190+0000","last_change":"2024-04-25T00:13:19.420842+0000","last_active":"2024-04-25T00:14:11.263190+0000","last_peered":"2024-04-25T00:14:11.263190+0000","last_clean":"2024-04-25T00:14:11.263190+0000","last_became_active":"2024-04-25T00:13:19.420241+0000","last_became_peered":"2024-04-25T00:13:19.420241+0000","last_unstale":"2024-04-25T00:14:11.263190+0000","last_undegraded":"2024-04-25T00:14:11.263190+0000","last_fullsized":"2024-04-25T00:14:11.263190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:52:39.500432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.268192+0000","last_change":"2024-04-25T00:14:11.267688+0000","last_active":"2024-04-25T00:14:11.268192+0000","last_peered":"2024-04-25T00:14:11.268192+0000","last_clean":"2024-04-25T00:14:11.268192+0000","last_became_active":"2024-04-25T00:14:11.267225+0000","last_became_peered":"2024-04-25T00:14:11.267225+0000","last_unstale":"2024-04-25T00:14:11.268192+0000","last_undegraded":"2024-04-25T00:14:11.268192+0000","last_fullsized":"2024-04-25T00:14:11.268192+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:15.867489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.267718+0000","last_change":"2024-04-25T00:13:19.416519+0000","last_active":"2024-04-25T00:14:11.267718+0000","last_peered":"2024-04-25T00:14:11.267718+0000","last_clean":"2024-04-25T00:14:11.267718+0000","last_became_active":"2024-04-25T00:13:19.416080+0000","last_became_peered":"2024-04-25T00:13:19.416080+0000","last_unstale":"2024-04-25T00:14:11.267718+0000","last_undegraded":"2024-04-25T00:14:11.267718+0000","last_fullsized":"2024-04-25T00:14:11.267718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:04.450160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.270437+0000","last_change":"2024-04-25T00:13:19.424378+0000","last_active":"2024-04-25T00:14:11.270437+0000","last_peered":"2024-04-25T00:14:11.270437+0000","last_clean":"2024-04-25T00:14:11.270437+0000","last_became_active":"2024-04-25T00:13:19.423918+0000","last_became_peered":"2024-04-25T00:13:19.423918+0000","last_unstale":"2024-04-25T00:14:11.270437+0000","last_undegraded":"2024-04-25T00:14:11.270437+0000","last_fullsized":"2024-04-25T00:14:11.270437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:01:11.397328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265344+0000","last_change":"2024-04-25T00:13:19.424729+0000","last_active":"2024-04-25T00:14:11.265344+0000","last_peered":"2024-04-25T00:14:11.265344+0000","last_clean":"2024-04-25T00:14:11.265344+0000","last_became_active":"2024-04-25T00:13:19.422874+0000","last_became_peered":"2024-04-25T00:13:19.422874+0000","last_unstale":"2024-04-25T00:14:11.265344+0000","last_undegraded":"2024-04-25T00:14:11.265344+0000","last_fullsized":"2024-04-25T00:14:11.265344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:52:39.483589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.266615+0000","last_change":"2024-04-25T00:13:19.418043+0000","last_active":"2024-04-25T00:14:11.266615+0000","last_peered":"2024-04-25T00:14:11.266615+0000","last_clean":"2024-04-25T00:14:11.266615+0000","last_became_active":"2024-04-25T00:13:19.417443+0000","last_became_peered":"2024-04-25T00:13:19.417443+0000","last_unstale":"2024-04-25T00:14:11.266615+0000","last_undegraded":"2024-04-25T00:14:11.266615+0000","last_fullsized":"2024-04-25T00:14:11.266615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:47:53.141744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260673+0000","last_change":"2024-04-25T00:13:19.420989+0000","last_active":"2024-04-25T00:14:11.260673+0000","last_peered":"2024-04-25T00:14:11.260673+0000","last_clean":"2024-04-25T00:14:11.260673+0000","last_became_active":"2024-04-25T00:13:19.420669+0000","last_became_peered":"2024-04-25T00:13:19.420669+0000","last_unstale":"2024-04-25T00:14:11.260673+0000","last_undegraded":"2024-04-25T00:14:11.260673+0000","last_fullsized":"2024-04-25T00:14:11.260673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:18.236132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.271184+0000","last_change":"2024-04-25T00:13:19.424176+0000","last_active":"2024-04-25T00:14:11.271184+0000","last_peered":"2024-04-25T00:14:11.271184+0000","last_clean":"2024-04-25T00:14:11.271184+0000","last_became_active":"2024-04-25T00:13:19.423656+0000","last_became_peered":"2024-04-25T00:13:19.423656+0000","last_unstale":"2024-04-25T00:14:11.271184+0000","last_undegraded":"2024-04-25T00:14:11.271184+0000","last_fullsized":"2024-04-25T00:14:11.271184+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:24:41.697077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265720+0000","last_change":"2024-04-25T00:13:19.419624+0000","last_active":"2024-04-25T00:14:11.265720+0000","last_peered":"2024-04-25T00:14:11.265720+0000","last_clean":"2024-04-25T00:14:11.265720+0000","last_became_active":"2024-04-25T00:13:19.419263+0000","last_became_peered":"2024-04-25T00:13:19.419263+0000","last_unstale":"2024-04-25T00:14:11.265720+0000","last_undegraded":"2024-04-25T00:14:11.265720+0000","last_fullsized":"2024-04-25T00:14:11.265720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:29.130733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264264+0000","last_change":"2024-04-25T00:13:19.427865+0000","last_active":"2024-04-25T00:14:11.264264+0000","last_peered":"2024-04-25T00:14:11.264264+0000","last_clean":"2024-04-25T00:14:11.264264+0000","last_became_active":"2024-04-25T00:13:19.427538+0000","last_became_peered":"2024-04-25T00:13:19.427538+0000","last_unstale":"2024-04-25T00:14:11.264264+0000","last_undegraded":"2024-04-25T00:14:11.264264+0000","last_fullsized":"2024-04-25T00:14:11.264264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:05.136948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.266256+0000","last_change":"2024-04-25T00:13:19.419261+0000","last_active":"2024-04-25T00:14:11.266256+0000","last_peered":"2024-04-25T00:14:11.266256+0000","last_clean":"2024-04-25T00:14:11.266256+0000","last_became_active":"2024-04-25T00:13:19.418938+0000","last_became_peered":"2024-04-25T00:13:19.418938+0000","last_unstale":"2024-04-25T00:14:11.266256+0000","last_undegraded":"2024-04-25T00:14:11.266256+0000","last_fullsized":"2024-04-25T00:14:11.266256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:44.971640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262686+0000","last_change":"2024-04-25T00:13:19.417654+0000","last_active":"2024-04-25T00:14:11.262686+0000","last_peered":"2024-04-25T00:14:11.262686+0000","last_clean":"2024-04-25T00:14:11.262686+0000","last_became_active":"2024-04-25T00:13:19.417319+0000","last_became_peered":"2024-04-25T00:13:19.417319+0000","last_unstale":"2024-04-25T00:14:11.262686+0000","last_undegraded":"2024-04-25T00:14:11.262686+0000","last_fullsized":"2024-04-25T00:14:11.262686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:43:54.447937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262540+0000","last_change":"2024-04-25T00:13:19.420761+0000","last_active":"2024-04-25T00:14:11.262540+0000","last_peered":"2024-04-25T00:14:11.262540+0000","last_clean":"2024-04-25T00:14:11.262540+0000","last_became_active":"2024-04-25T00:13:19.420334+0000","last_became_peered":"2024-04-25T00:13:19.420334+0000","last_unstale":"2024-04-25T00:14:11.262540+0000","last_undegraded":"2024-04-25T00:14:11.262540+0000","last_fullsized":"2024-04-25T00:14:11.262540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:20.136470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260093+0000","last_change":"2024-04-25T00:13:19.420132+0000","last_active":"2024-04-25T00:14:11.260093+0000","last_peered":"2024-04-25T00:14:11.260093+0000","last_clean":"2024-04-25T00:14:11.260093+0000","last_became_active":"2024-04-25T00:13:19.418910+0000","last_became_peered":"2024-04-25T00:13:19.418910+0000","last_unstale":"2024-04-25T00:14:11.260093+0000","last_undegraded":"2024-04-25T00:14:11.260093+0000","last_fullsized":"2024-04-25T00:14:11.260093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:17.895412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262869+0000","last_change":"2024-04-25T00:13:19.430242+0000","last_active":"2024-04-25T00:14:11.262869+0000","last_peered":"2024-04-25T00:14:11.262869+0000","last_clean":"2024-04-25T00:14:11.262869+0000","last_became_active":"2024-04-25T00:13:19.429077+0000","last_became_peered":"2024-04-25T00:13:19.429077+0000","last_unstale":"2024-04-25T00:14:11.262869+0000","last_undegraded":"2024-04-25T00:14:11.262869+0000","last_fullsized":"2024-04-25T00:14:11.262869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:34:14.351157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262098+0000","last_change":"2024-04-25T00:13:19.414375+0000","last_active":"2024-04-25T00:14:11.262098+0000","last_peered":"2024-04-25T00:14:11.262098+0000","last_clean":"2024-04-25T00:14:11.262098+0000","last_became_active":"2024-04-25T00:13:19.414003+0000","last_became_peered":"2024-04-25T00:13:19.414003+0000","last_unstale":"2024-04-25T00:14:11.262098+0000","last_undegraded":"2024-04-25T00:14:11.262098+0000","last_fullsized":"2024-04-25T00:14:11.262098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:43:50.612743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260516+0000","last_change":"2024-04-25T00:13:19.415322+0000","last_active":"2024-04-25T00:14:11.260516+0000","last_peered":"2024-04-25T00:14:11.260516+0000","last_clean":"2024-04-25T00:14:11.260516+0000","last_became_active":"2024-04-25T00:13:19.414925+0000","last_became_peered":"2024-04-25T00:13:19.414925+0000","last_unstale":"2024-04-25T00:14:11.260516+0000","last_undegraded":"2024-04-25T00:14:11.260516+0000","last_fullsized":"2024-04-25T00:14:11.260516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:58.140640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.259607+0000","last_change":"2024-04-25T00:13:23.584763+0000","last_active":"2024-04-25T00:14:11.259607+0000","last_peered":"2024-04-25T00:14:11.259607+0000","last_clean":"2024-04-25T00:14:11.259607+0000","last_became_active":"2024-04-25T00:13:23.584303+0000","last_became_peered":"2024-04-25T00:13:23.584303+0000","last_unstale":"2024-04-25T00:14:11.259607+0000","last_undegraded":"2024-04-25T00:14:11.259607+0000","last_fullsized":"2024-04-25T00:14:11.259607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:54:11.447867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":112,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:56.683932+0000","last_change":"2024-04-25T00:13:18.519576+0000","last_active":"2024-04-25T00:14:56.683932+0000","last_peered":"2024-04-25T00:14:56.683932+0000","last_clean":"2024-04-25T00:14:56.683932+0000","last_became_active":"2024-04-25T00:13:18.519201+0000","last_became_peered":"2024-04-25T00:13:18.519201+0000","last_unstale":"2024-04-25T00:14:56.683932+0000","last_undegraded":"2024-04-25T00:14:56.683932+0000","last_fullsized":"2024-04-25T00:14:56.683932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:19.681096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.267035+0000","last_change":"2024-04-25T00:13:19.419098+0000","last_active":"2024-04-25T00:14:11.267035+0000","last_peered":"2024-04-25T00:14:11.267035+0000","last_clean":"2024-04-25T00:14:11.267035+0000","last_became_active":"2024-04-25T00:13:19.418649+0000","last_became_peered":"2024-04-25T00:13:19.418649+0000","last_unstale":"2024-04-25T00:14:11.267035+0000","last_undegraded":"2024-04-25T00:14:11.267035+0000","last_fullsized":"2024-04-25T00:14:11.267035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:04.561470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263180+0000","last_change":"2024-04-25T00:13:23.580354+0000","last_active":"2024-04-25T00:14:11.263180+0000","last_peered":"2024-04-25T00:14:11.263180+0000","last_clean":"2024-04-25T00:14:11.263180+0000","last_became_active":"2024-04-25T00:13:23.579944+0000","last_became_peered":"2024-04-25T00:13:23.579944+0000","last_unstale":"2024-04-25T00:14:11.263180+0000","last_undegraded":"2024-04-25T00:14:11.263180+0000","last_fullsized":"2024-04-25T00:14:11.263180+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:48:57.788974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.267134+0000","last_change":"2024-04-25T00:13:18.527471+0000","last_active":"2024-04-25T00:14:11.267134+0000","last_peered":"2024-04-25T00:14:11.267134+0000","last_clean":"2024-04-25T00:14:11.267134+0000","last_became_active":"2024-04-25T00:13:18.527138+0000","last_became_peered":"2024-04-25T00:13:18.527138+0000","last_unstale":"2024-04-25T00:14:11.267134+0000","last_undegraded":"2024-04-25T00:14:11.267134+0000","last_fullsized":"2024-04-25T00:14:11.267134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:38.003719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262922+0000","last_change":"2024-04-25T00:13:19.420563+0000","last_active":"2024-04-25T00:14:11.262922+0000","last_peered":"2024-04-25T00:14:11.262922+0000","last_clean":"2024-04-25T00:14:11.262922+0000","last_became_active":"2024-04-25T00:13:19.420142+0000","last_became_peered":"2024-04-25T00:13:19.420142+0000","last_unstale":"2024-04-25T00:14:11.262922+0000","last_undegraded":"2024-04-25T00:14:11.262922+0000","last_fullsized":"2024-04-25T00:14:11.262922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:23:08.423795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.271321+0000","last_change":"2024-04-25T00:14:11.271321+0000","last_active":"2024-04-25T00:14:11.271321+0000","last_peered":"2024-04-25T00:14:11.271321+0000","last_clean":"2024-04-25T00:14:11.271321+0000","last_became_active":"2024-04-25T00:14:11.270478+0000","last_became_peered":"2024-04-25T00:14:11.270478+0000","last_unstale":"2024-04-25T00:14:11.271321+0000","last_undegraded":"2024-04-25T00:14:11.271321+0000","last_fullsized":"2024-04-25T00:14:11.271321+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:41.796155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265434+0000","last_change":"2024-04-25T00:13:18.520509+0000","last_active":"2024-04-25T00:14:11.265434+0000","last_peered":"2024-04-25T00:14:11.265434+0000","last_clean":"2024-04-25T00:14:11.265434+0000","last_became_active":"2024-04-25T00:13:18.520139+0000","last_became_peered":"2024-04-25T00:13:18.520139+0000","last_unstale":"2024-04-25T00:14:11.265434+0000","last_undegraded":"2024-04-25T00:14:11.265434+0000","last_fullsized":"2024-04-25T00:14:11.265434+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:08.797670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260383+0000","last_change":"2024-04-25T00:13:19.419115+0000","last_active":"2024-04-25T00:14:11.260383+0000","last_peered":"2024-04-25T00:14:11.260383+0000","last_clean":"2024-04-25T00:14:11.260383+0000","last_became_active":"2024-04-25T00:13:19.417865+0000","last_became_peered":"2024-04-25T00:13:19.417865+0000","last_unstale":"2024-04-25T00:14:11.260383+0000","last_undegraded":"2024-04-25T00:14:11.260383+0000","last_fullsized":"2024-04-25T00:14:11.260383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:20:11.173803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263726+0000","last_change":"2024-04-25T00:13:23.585052+0000","last_active":"2024-04-25T00:14:11.263726+0000","last_peered":"2024-04-25T00:14:11.263726+0000","last_clean":"2024-04-25T00:14:11.263726+0000","last_became_active":"2024-04-25T00:13:23.584696+0000","last_became_peered":"2024-04-25T00:13:23.584696+0000","last_unstale":"2024-04-25T00:14:11.263726+0000","last_undegraded":"2024-04-25T00:14:11.263726+0000","last_fullsized":"2024-04-25T00:14:11.263726+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:26.894183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264908+0000","last_change":"2024-04-25T00:13:18.522793+0000","last_active":"2024-04-25T00:14:11.264908+0000","last_peered":"2024-04-25T00:14:11.264908+0000","last_clean":"2024-04-25T00:14:11.264908+0000","last_became_active":"2024-04-25T00:13:18.522453+0000","last_became_peered":"2024-04-25T00:13:18.522453+0000","last_unstale":"2024-04-25T00:14:11.264908+0000","last_undegraded":"2024-04-25T00:14:11.264908+0000","last_fullsized":"2024-04-25T00:14:11.264908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:08.368214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263127+0000","last_change":"2024-04-25T00:13:19.418573+0000","last_active":"2024-04-25T00:14:11.263127+0000","last_peered":"2024-04-25T00:14:11.263127+0000","last_clean":"2024-04-25T00:14:11.263127+0000","last_became_active":"2024-04-25T00:13:19.418219+0000","last_became_peered":"2024-04-25T00:13:19.418219+0000","last_unstale":"2024-04-25T00:14:11.263127+0000","last_undegraded":"2024-04-25T00:14:11.263127+0000","last_fullsized":"2024-04-25T00:14:11.263127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:08:44.453017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260123+0000","last_change":"2024-04-25T00:13:23.578716+0000","last_active":"2024-04-25T00:14:11.260123+0000","last_peered":"2024-04-25T00:14:11.260123+0000","last_clean":"2024-04-25T00:14:11.260123+0000","last_became_active":"2024-04-25T00:13:23.578325+0000","last_became_peered":"2024-04-25T00:13:23.578325+0000","last_unstale":"2024-04-25T00:14:11.260123+0000","last_undegraded":"2024-04-25T00:14:11.260123+0000","last_fullsized":"2024-04-25T00:14:11.260123+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:39:57.145656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:56.664015+0000","last_change":"2024-04-25T00:13:18.527369+0000","last_active":"2024-04-25T00:14:56.664015+0000","last_peered":"2024-04-25T00:14:56.664015+0000","last_clean":"2024-04-25T00:14:56.664015+0000","last_became_active":"2024-04-25T00:13:18.526032+0000","last_became_peered":"2024-04-25T00:13:18.526032+0000","last_unstale":"2024-04-25T00:14:56.664015+0000","last_undegraded":"2024-04-25T00:14:56.664015+0000","last_fullsized":"2024-04-25T00:14:56.664015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:52.635184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264866+0000","last_change":"2024-04-25T00:13:19.413305+0000","last_active":"2024-04-25T00:14:11.264866+0000","last_peered":"2024-04-25T00:14:11.264866+0000","last_clean":"2024-04-25T00:14:11.264866+0000","last_became_active":"2024-04-25T00:13:19.412956+0000","last_became_peered":"2024-04-25T00:13:19.412956+0000","last_unstale":"2024-04-25T00:14:11.264866+0000","last_undegraded":"2024-04-25T00:14:11.264866+0000","last_fullsized":"2024-04-25T00:14:11.264866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:54.436470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.273057+0000","last_change":"2024-04-25T00:14:11.273057+0000","last_active":"2024-04-25T00:14:11.273057+0000","last_peered":"2024-04-25T00:14:11.273057+0000","last_clean":"2024-04-25T00:14:11.273057+0000","last_became_active":"2024-04-25T00:14:11.272230+0000","last_became_peered":"2024-04-25T00:14:11.272230+0000","last_unstale":"2024-04-25T00:14:11.273057+0000","last_undegraded":"2024-04-25T00:14:11.273057+0000","last_fullsized":"2024-04-25T00:14:11.273057+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:37.695139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.270417+0000","last_change":"2024-04-25T00:13:18.517507+0000","last_active":"2024-04-25T00:14:11.270417+0000","last_peered":"2024-04-25T00:14:11.270417+0000","last_clean":"2024-04-25T00:14:11.270417+0000","last_became_active":"2024-04-25T00:13:18.517099+0000","last_became_peered":"2024-04-25T00:13:18.517099+0000","last_unstale":"2024-04-25T00:14:11.270417+0000","last_undegraded":"2024-04-25T00:14:11.270417+0000","last_fullsized":"2024-04-25T00:14:11.270417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:42:17.357610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.271691+0000","last_change":"2024-04-25T00:14:11.271691+0000","last_active":"2024-04-25T00:14:11.271691+0000","last_peered":"2024-04-25T00:14:11.271691+0000","last_clean":"2024-04-25T00:14:11.271691+0000","last_became_active":"2024-04-25T00:14:11.271160+0000","last_became_peered":"2024-04-25T00:14:11.271160+0000","last_unstale":"2024-04-25T00:14:11.271691+0000","last_undegraded":"2024-04-25T00:14:11.271691+0000","last_fullsized":"2024-04-25T00:14:11.271691+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:24.416810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262664+0000","last_change":"2024-04-25T00:13:23.582526+0000","last_active":"2024-04-25T00:14:11.262664+0000","last_peered":"2024-04-25T00:14:11.262664+0000","last_clean":"2024-04-25T00:14:11.262664+0000","last_became_active":"2024-04-25T00:13:23.581221+0000","last_became_peered":"2024-04-25T00:13:23.581221+0000","last_unstale":"2024-04-25T00:14:11.262664+0000","last_undegraded":"2024-04-25T00:14:11.262664+0000","last_fullsized":"2024-04-25T00:14:11.262664+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:46.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.266648+0000","last_change":"2024-04-25T00:13:18.522245+0000","last_active":"2024-04-25T00:14:11.266648+0000","last_peered":"2024-04-25T00:14:11.266648+0000","last_clean":"2024-04-25T00:14:11.266648+0000","last_became_active":"2024-04-25T00:13:18.521440+0000","last_became_peered":"2024-04-25T00:13:18.521440+0000","last_unstale":"2024-04-25T00:14:11.266648+0000","last_undegraded":"2024-04-25T00:14:11.266648+0000","last_fullsized":"2024-04-25T00:14:11.266648+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:21:37.847964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.266511+0000","last_change":"2024-04-25T00:13:19.422771+0000","last_active":"2024-04-25T00:14:11.266511+0000","last_peered":"2024-04-25T00:14:11.266511+0000","last_clean":"2024-04-25T00:14:11.266511+0000","last_became_active":"2024-04-25T00:13:19.422387+0000","last_became_peered":"2024-04-25T00:13:19.422387+0000","last_unstale":"2024-04-25T00:14:11.266511+0000","last_undegraded":"2024-04-25T00:14:11.266511+0000","last_fullsized":"2024-04-25T00:14:11.266511+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:15.647186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262943+0000","last_change":"2024-04-25T00:13:23.586235+0000","last_active":"2024-04-25T00:14:11.262943+0000","last_peered":"2024-04-25T00:14:11.262943+0000","last_clean":"2024-04-25T00:14:11.262943+0000","last_became_active":"2024-04-25T00:13:23.585803+0000","last_became_peered":"2024-04-25T00:13:23.585803+0000","last_unstale":"2024-04-25T00:14:11.262943+0000","last_undegraded":"2024-04-25T00:14:11.262943+0000","last_fullsized":"2024-04-25T00:14:11.262943+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:46.235435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264403+0000","last_change":"2024-04-25T00:13:18.525283+0000","last_active":"2024-04-25T00:14:11.264403+0000","last_peered":"2024-04-25T00:14:11.264403+0000","last_clean":"2024-04-25T00:14:11.264403+0000","last_became_active":"2024-04-25T00:13:18.524939+0000","last_became_peered":"2024-04-25T00:13:18.524939+0000","last_unstale":"2024-04-25T00:14:11.264403+0000","last_undegraded":"2024-04-25T00:14:11.264403+0000","last_fullsized":"2024-04-25T00:14:11.264403+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:09.258498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261187+0000","last_change":"2024-04-25T00:13:19.415746+0000","last_active":"2024-04-25T00:14:11.261187+0000","last_peered":"2024-04-25T00:14:11.261187+0000","last_clean":"2024-04-25T00:14:11.261187+0000","last_became_active":"2024-04-25T00:13:19.415325+0000","last_became_peered":"2024-04-25T00:13:19.415325+0000","last_unstale":"2024-04-25T00:14:11.261187+0000","last_undegraded":"2024-04-25T00:14:11.261187+0000","last_fullsized":"2024-04-25T00:14:11.261187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:17:48.268845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265732+0000","last_change":"2024-04-25T00:13:23.583153+0000","last_active":"2024-04-25T00:14:11.265732+0000","last_peered":"2024-04-25T00:14:11.265732+0000","last_clean":"2024-04-25T00:14:11.265732+0000","last_became_active":"2024-04-25T00:13:23.580257+0000","last_became_peered":"2024-04-25T00:13:23.580257+0000","last_unstale":"2024-04-25T00:14:11.265732+0000","last_undegraded":"2024-04-25T00:14:11.265732+0000","last_fullsized":"2024-04-25T00:14:11.265732+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:09:32.990253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":175,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:57.664767+0000","last_change":"2024-04-25T00:13:18.520365+0000","last_active":"2024-04-25T00:14:57.664767+0000","last_peered":"2024-04-25T00:14:57.664767+0000","last_clean":"2024-04-25T00:14:57.664767+0000","last_became_active":"2024-04-25T00:13:18.519918+0000","last_became_peered":"2024-04-25T00:13:18.519918+0000","last_unstale":"2024-04-25T00:14:57.664767+0000","last_undegraded":"2024-04-25T00:14:57.664767+0000","last_fullsized":"2024-04-25T00:14:57.664767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:46.415197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260684+0000","last_change":"2024-04-25T00:13:19.414341+0000","last_active":"2024-04-25T00:14:11.260684+0000","last_peered":"2024-04-25T00:14:11.260684+0000","last_clean":"2024-04-25T00:14:11.260684+0000","last_became_active":"2024-04-25T00:13:19.413793+0000","last_became_peered":"2024-04-25T00:13:19.413793+0000","last_unstale":"2024-04-25T00:14:11.260684+0000","last_undegraded":"2024-04-25T00:14:11.260684+0000","last_fullsized":"2024-04-25T00:14:11.260684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:55.927082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261020+0000","last_change":"2024-04-25T00:13:23.590631+0000","last_active":"2024-04-25T00:14:11.261020+0000","last_peered":"2024-04-25T00:14:11.261020+0000","last_clean":"2024-04-25T00:14:11.261020+0000","last_became_active":"2024-04-25T00:13:23.589980+0000","last_became_peered":"2024-04-25T00:13:23.589980+0000","last_unstale":"2024-04-25T00:14:11.261020+0000","last_undegraded":"2024-04-25T00:14:11.261020+0000","last_fullsized":"2024-04-25T00:14:11.261020+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:18.234048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265367+0000","last_change":"2024-04-25T00:13:18.526353+0000","last_active":"2024-04-25T00:14:11.265367+0000","last_peered":"2024-04-25T00:14:11.265367+0000","last_clean":"2024-04-25T00:14:11.265367+0000","last_became_active":"2024-04-25T00:13:18.525951+0000","last_became_peered":"2024-04-25T00:13:18.525951+0000","last_unstale":"2024-04-25T00:14:11.265367+0000","last_undegraded":"2024-04-25T00:14:11.265367+0000","last_fullsized":"2024-04-25T00:14:11.265367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:10:29.597797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262851+0000","last_change":"2024-04-25T00:13:19.425486+0000","last_active":"2024-04-25T00:14:11.262851+0000","last_peered":"2024-04-25T00:14:11.262851+0000","last_clean":"2024-04-25T00:14:11.262851+0000","last_became_active":"2024-04-25T00:13:19.425087+0000","last_became_peered":"2024-04-25T00:13:19.425087+0000","last_unstale":"2024-04-25T00:14:11.262851+0000","last_undegraded":"2024-04-25T00:14:11.262851+0000","last_fullsized":"2024-04-25T00:14:11.262851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:00.216683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264036+0000","last_change":"2024-04-25T00:13:23.579747+0000","last_active":"2024-04-25T00:14:11.264036+0000","last_peered":"2024-04-25T00:14:11.264036+0000","last_clean":"2024-04-25T00:14:11.264036+0000","last_became_active":"2024-04-25T00:13:23.579346+0000","last_became_peered":"2024-04-25T00:13:23.579346+0000","last_unstale":"2024-04-25T00:14:11.264036+0000","last_undegraded":"2024-04-25T00:14:11.264036+0000","last_fullsized":"2024-04-25T00:14:11.264036+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:10:18.969942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262985+0000","last_change":"2024-04-25T00:13:18.524707+0000","last_active":"2024-04-25T00:14:11.262985+0000","last_peered":"2024-04-25T00:14:11.262985+0000","last_clean":"2024-04-25T00:14:11.262985+0000","last_became_active":"2024-04-25T00:13:18.524352+0000","last_became_peered":"2024-04-25T00:13:18.524352+0000","last_unstale":"2024-04-25T00:14:11.262985+0000","last_undegraded":"2024-04-25T00:14:11.262985+0000","last_fullsized":"2024-04-25T00:14:11.262985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:29.992843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260591+0000","last_change":"2024-04-25T00:13:19.421845+0000","last_active":"2024-04-25T00:14:11.260591+0000","last_peered":"2024-04-25T00:14:11.260591+0000","last_clean":"2024-04-25T00:14:11.260591+0000","last_became_active":"2024-04-25T00:13:19.421458+0000","last_became_peered":"2024-04-25T00:13:19.421458+0000","last_unstale":"2024-04-25T00:14:11.260591+0000","last_undegraded":"2024-04-25T00:14:11.260591+0000","last_fullsized":"2024-04-25T00:14:11.260591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:09.286604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261234+0000","last_change":"2024-04-25T00:13:23.583721+0000","last_active":"2024-04-25T00:14:11.261234+0000","last_peered":"2024-04-25T00:14:11.261234+0000","last_clean":"2024-04-25T00:14:11.261234+0000","last_became_active":"2024-04-25T00:13:23.583340+0000","last_became_peered":"2024-04-25T00:13:23.583340+0000","last_unstale":"2024-04-25T00:14:11.261234+0000","last_undegraded":"2024-04-25T00:14:11.261234+0000","last_fullsized":"2024-04-25T00:14:11.261234+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:26.446468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'32","reported_seq":212,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:57.664698+0000","last_change":"2024-04-25T00:13:18.520653+0000","last_active":"2024-04-25T00:14:57.664698+0000","last_peered":"2024-04-25T00:14:57.664698+0000","last_clean":"2024-04-25T00:14:57.664698+0000","last_became_active":"2024-04-25T00:13:18.520278+0000","last_became_peered":"2024-04-25T00:13:18.520278+0000","last_unstale":"2024-04-25T00:14:57.664698+0000","last_undegraded":"2024-04-25T00:14:57.664698+0000","last_fullsized":"2024-04-25T00:14:57.664698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:58.311179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61151,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":153,"num_read_kb":136,"num_write":34,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264723+0000","last_change":"2024-04-25T00:13:19.421599+0000","last_active":"2024-04-25T00:14:11.264723+0000","last_peered":"2024-04-25T00:14:11.264723+0000","last_clean":"2024-04-25T00:14:11.264723+0000","last_became_active":"2024-04-25T00:13:19.421107+0000","last_became_peered":"2024-04-25T00:13:19.421107+0000","last_unstale":"2024-04-25T00:14:11.264723+0000","last_undegraded":"2024-04-25T00:14:11.264723+0000","last_fullsized":"2024-04-25T00:14:11.264723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:59.577987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263744+0000","last_change":"2024-04-25T00:13:23.580538+0000","last_active":"2024-04-25T00:14:11.263744+0000","last_peered":"2024-04-25T00:14:11.263744+0000","last_clean":"2024-04-25T00:14:11.263744+0000","last_became_active":"2024-04-25T00:13:23.580095+0000","last_became_peered":"2024-04-25T00:13:23.580095+0000","last_unstale":"2024-04-25T00:14:11.263744+0000","last_undegraded":"2024-04-25T00:14:11.263744+0000","last_fullsized":"2024-04-25T00:14:11.263744+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:04.189530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.259946+0000","last_change":"2024-04-25T00:13:18.522501+0000","last_active":"2024-04-25T00:14:11.259946+0000","last_peered":"2024-04-25T00:14:11.259946+0000","last_clean":"2024-04-25T00:14:11.259946+0000","last_became_active":"2024-04-25T00:13:18.522160+0000","last_became_peered":"2024-04-25T00:13:18.522160+0000","last_unstale":"2024-04-25T00:14:11.259946+0000","last_undegraded":"2024-04-25T00:14:11.259946+0000","last_fullsized":"2024-04-25T00:14:11.259946+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:31.876731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.266027+0000","last_change":"2024-04-25T00:13:19.417957+0000","last_active":"2024-04-25T00:14:11.266027+0000","last_peered":"2024-04-25T00:14:11.266027+0000","last_clean":"2024-04-25T00:14:11.266027+0000","last_became_active":"2024-04-25T00:13:19.417600+0000","last_became_peered":"2024-04-25T00:13:19.417600+0000","last_unstale":"2024-04-25T00:14:11.266027+0000","last_undegraded":"2024-04-25T00:14:11.266027+0000","last_fullsized":"2024-04-25T00:14:11.266027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:24.600965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262047+0000","last_change":"2024-04-25T00:13:23.582279+0000","last_active":"2024-04-25T00:14:11.262047+0000","last_peered":"2024-04-25T00:14:11.262047+0000","last_clean":"2024-04-25T00:14:11.262047+0000","last_became_active":"2024-04-25T00:13:23.581766+0000","last_became_peered":"2024-04-25T00:13:23.581766+0000","last_unstale":"2024-04-25T00:14:11.262047+0000","last_undegraded":"2024-04-25T00:14:11.262047+0000","last_fullsized":"2024-04-25T00:14:11.262047+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:36.115870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263249+0000","last_change":"2024-04-25T00:13:18.527443+0000","last_active":"2024-04-25T00:14:11.263249+0000","last_peered":"2024-04-25T00:14:11.263249+0000","last_clean":"2024-04-25T00:14:11.263249+0000","last_became_active":"2024-04-25T00:13:18.526919+0000","last_became_peered":"2024-04-25T00:13:18.526919+0000","last_unstale":"2024-04-25T00:14:11.263249+0000","last_undegraded":"2024-04-25T00:14:11.263249+0000","last_fullsized":"2024-04-25T00:14:11.263249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:32.737580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265626+0000","last_change":"2024-04-25T00:13:19.425947+0000","last_active":"2024-04-25T00:14:11.265626+0000","last_peered":"2024-04-25T00:14:11.265626+0000","last_clean":"2024-04-25T00:14:11.265626+0000","last_became_active":"2024-04-25T00:13:19.425613+0000","last_became_peered":"2024-04-25T00:13:19.425613+0000","last_unstale":"2024-04-25T00:14:11.265626+0000","last_undegraded":"2024-04-25T00:14:11.265626+0000","last_fullsized":"2024-04-25T00:14:11.265626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:28:34.729375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261732+0000","last_change":"2024-04-25T00:13:23.587794+0000","last_active":"2024-04-25T00:14:11.261732+0000","last_peered":"2024-04-25T00:14:11.261732+0000","last_clean":"2024-04-25T00:14:11.261732+0000","last_became_active":"2024-04-25T00:13:23.587334+0000","last_became_peered":"2024-04-25T00:13:23.587334+0000","last_unstale":"2024-04-25T00:14:11.261732+0000","last_undegraded":"2024-04-25T00:14:11.261732+0000","last_fullsized":"2024-04-25T00:14:11.261732+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:58.460439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":110,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:57.664894+0000","last_change":"2024-04-25T00:13:18.521701+0000","last_active":"2024-04-25T00:14:57.664894+0000","last_peered":"2024-04-25T00:14:57.664894+0000","last_clean":"2024-04-25T00:14:57.664894+0000","last_became_active":"2024-04-25T00:13:18.521360+0000","last_became_peered":"2024-04-25T00:13:18.521360+0000","last_unstale":"2024-04-25T00:14:57.664894+0000","last_undegraded":"2024-04-25T00:14:57.664894+0000","last_fullsized":"2024-04-25T00:14:57.664894+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:18:11.567426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265853+0000","last_change":"2024-04-25T00:13:19.424162+0000","last_active":"2024-04-25T00:14:11.265853+0000","last_peered":"2024-04-25T00:14:11.265853+0000","last_clean":"2024-04-25T00:14:11.265853+0000","last_became_active":"2024-04-25T00:13:19.423845+0000","last_became_peered":"2024-04-25T00:13:19.423845+0000","last_unstale":"2024-04-25T00:14:11.265853+0000","last_undegraded":"2024-04-25T00:14:11.265853+0000","last_fullsized":"2024-04-25T00:14:11.265853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:33.887577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.267758+0000","last_change":"2024-04-25T00:13:23.582102+0000","last_active":"2024-04-25T00:14:11.267758+0000","last_peered":"2024-04-25T00:14:11.267758+0000","last_clean":"2024-04-25T00:14:11.267758+0000","last_became_active":"2024-04-25T00:13:23.581679+0000","last_became_peered":"2024-04-25T00:13:23.581679+0000","last_unstale":"2024-04-25T00:14:11.267758+0000","last_undegraded":"2024-04-25T00:14:11.267758+0000","last_fullsized":"2024-04-25T00:14:11.267758+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:05:39.518634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265885+0000","last_change":"2024-04-25T00:13:18.518285+0000","last_active":"2024-04-25T00:14:11.265885+0000","last_peered":"2024-04-25T00:14:11.265885+0000","last_clean":"2024-04-25T00:14:11.265885+0000","last_became_active":"2024-04-25T00:13:18.517826+0000","last_became_peered":"2024-04-25T00:13:18.517826+0000","last_unstale":"2024-04-25T00:14:11.265885+0000","last_undegraded":"2024-04-25T00:14:11.265885+0000","last_fullsized":"2024-04-25T00:14:11.265885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:12.901225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.268444+0000","last_change":"2024-04-25T00:14:11.268444+0000","last_active":"2024-04-25T00:14:11.268444+0000","last_peered":"2024-04-25T00:14:11.268444+0000","last_clean":"2024-04-25T00:14:11.268444+0000","last_became_active":"2024-04-25T00:14:11.267934+0000","last_became_peered":"2024-04-25T00:14:11.267934+0000","last_unstale":"2024-04-25T00:14:11.268444+0000","last_undegraded":"2024-04-25T00:14:11.268444+0000","last_fullsized":"2024-04-25T00:14:11.268444+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:12:16.209890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265284+0000","last_change":"2024-04-25T00:13:23.578445+0000","last_active":"2024-04-25T00:14:11.265284+0000","last_peered":"2024-04-25T00:14:11.265284+0000","last_clean":"2024-04-25T00:14:11.265284+0000","last_became_active":"2024-04-25T00:13:23.578048+0000","last_became_peered":"2024-04-25T00:13:23.578048+0000","last_unstale":"2024-04-25T00:14:11.265284+0000","last_undegraded":"2024-04-25T00:14:11.265284+0000","last_fullsized":"2024-04-25T00:14:11.265284+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:55.072460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264494+0000","last_change":"2024-04-25T00:13:18.515939+0000","last_active":"2024-04-25T00:14:11.264494+0000","last_peered":"2024-04-25T00:14:11.264494+0000","last_clean":"2024-04-25T00:14:11.264494+0000","last_became_active":"2024-04-25T00:13:18.515597+0000","last_became_peered":"2024-04-25T00:13:18.515597+0000","last_unstale":"2024-04-25T00:14:11.264494+0000","last_undegraded":"2024-04-25T00:14:11.264494+0000","last_fullsized":"2024-04-25T00:14:11.264494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:03:30.266404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.269143+0000","last_change":"2024-04-25T00:14:11.269143+0000","last_active":"2024-04-25T00:14:11.269143+0000","last_peered":"2024-04-25T00:14:11.269143+0000","last_clean":"2024-04-25T00:14:11.269143+0000","last_became_active":"2024-04-25T00:14:11.268736+0000","last_became_peered":"2024-04-25T00:14:11.268736+0000","last_unstale":"2024-04-25T00:14:11.269143+0000","last_undegraded":"2024-04-25T00:14:11.269143+0000","last_fullsized":"2024-04-25T00:14:11.269143+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:19.559974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264834+0000","last_change":"2024-04-25T00:13:23.862576+0000","last_active":"2024-04-25T00:14:11.264834+0000","last_peered":"2024-04-25T00:14:11.264834+0000","last_clean":"2024-04-25T00:14:11.264834+0000","last_became_active":"2024-04-25T00:13:23.862024+0000","last_became_peered":"2024-04-25T00:13:23.862024+0000","last_unstale":"2024-04-25T00:14:11.264834+0000","last_undegraded":"2024-04-25T00:14:11.264834+0000","last_fullsized":"2024-04-25T00:14:11.264834+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:17.249680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261518+0000","last_change":"2024-04-25T00:13:18.522503+0000","last_active":"2024-04-25T00:14:11.261518+0000","last_peered":"2024-04-25T00:14:11.261518+0000","last_clean":"2024-04-25T00:14:11.261518+0000","last_became_active":"2024-04-25T00:13:18.522126+0000","last_became_peered":"2024-04-25T00:13:18.522126+0000","last_unstale":"2024-04-25T00:14:11.261518+0000","last_undegraded":"2024-04-25T00:14:11.261518+0000","last_fullsized":"2024-04-25T00:14:11.261518+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:47:57.530098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.271543+0000","last_change":"2024-04-25T00:13:19.416862+0000","last_active":"2024-04-25T00:14:11.271543+0000","last_peered":"2024-04-25T00:14:11.271543+0000","last_clean":"2024-04-25T00:14:11.271543+0000","last_became_active":"2024-04-25T00:13:19.416375+0000","last_became_peered":"2024-04-25T00:13:19.416375+0000","last_unstale":"2024-04-25T00:14:11.271543+0000","last_undegraded":"2024-04-25T00:14:11.271543+0000","last_fullsized":"2024-04-25T00:14:11.271543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:43:32.031623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.267401+0000","last_change":"2024-04-25T00:13:23.582387+0000","last_active":"2024-04-25T00:14:11.267401+0000","last_peered":"2024-04-25T00:14:11.267401+0000","last_clean":"2024-04-25T00:14:11.267401+0000","last_became_active":"2024-04-25T00:13:23.581984+0000","last_became_peered":"2024-04-25T00:13:23.581984+0000","last_unstale":"2024-04-25T00:14:11.267401+0000","last_undegraded":"2024-04-25T00:14:11.267401+0000","last_fullsized":"2024-04-25T00:14:11.267401+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:07:16.388293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":100,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:53.680660+0000","last_change":"2024-04-25T00:13:18.524121+0000","last_active":"2024-04-25T00:14:53.680660+0000","last_peered":"2024-04-25T00:14:53.680660+0000","last_clean":"2024-04-25T00:14:53.680660+0000","last_became_active":"2024-04-25T00:13:18.523731+0000","last_became_peered":"2024-04-25T00:13:18.523731+0000","last_unstale":"2024-04-25T00:14:53.680660+0000","last_undegraded":"2024-04-25T00:14:53.680660+0000","last_fullsized":"2024-04-25T00:14:53.680660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:22.105952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.270799+0000","last_change":"2024-04-25T00:13:19.424753+0000","last_active":"2024-04-25T00:14:11.270799+0000","last_peered":"2024-04-25T00:14:11.270799+0000","last_clean":"2024-04-25T00:14:11.270799+0000","last_became_active":"2024-04-25T00:13:19.423214+0000","last_became_peered":"2024-04-25T00:13:19.423214+0000","last_unstale":"2024-04-25T00:14:11.270799+0000","last_undegraded":"2024-04-25T00:14:11.270799+0000","last_fullsized":"2024-04-25T00:14:11.270799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:41.627442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.272431+0000","last_change":"2024-04-25T00:14:11.272431+0000","last_active":"2024-04-25T00:14:11.272431+0000","last_peered":"2024-04-25T00:14:11.272431+0000","last_clean":"2024-04-25T00:14:11.272431+0000","last_became_active":"2024-04-25T00:14:11.271997+0000","last_became_peered":"2024-04-25T00:14:11.271997+0000","last_unstale":"2024-04-25T00:14:11.272431+0000","last_undegraded":"2024-04-25T00:14:11.272431+0000","last_fullsized":"2024-04-25T00:14:11.272431+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:18.768009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264624+0000","last_change":"2024-04-25T00:13:19.420617+0000","last_active":"2024-04-25T00:14:11.264624+0000","last_peered":"2024-04-25T00:14:11.264624+0000","last_clean":"2024-04-25T00:14:11.264624+0000","last_became_active":"2024-04-25T00:13:19.420274+0000","last_became_peered":"2024-04-25T00:13:19.420274+0000","last_unstale":"2024-04-25T00:14:11.264624+0000","last_undegraded":"2024-04-25T00:14:11.264624+0000","last_fullsized":"2024-04-25T00:14:11.264624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:07.515196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:53.658208+0000","last_change":"2024-04-25T00:13:18.529753+0000","last_active":"2024-04-25T00:14:53.658208+0000","last_peered":"2024-04-25T00:14:53.658208+0000","last_clean":"2024-04-25T00:14:53.658208+0000","last_became_active":"2024-04-25T00:13:18.525521+0000","last_became_peered":"2024-04-25T00:13:18.525521+0000","last_unstale":"2024-04-25T00:14:53.658208+0000","last_undegraded":"2024-04-25T00:14:53.658208+0000","last_fullsized":"2024-04-25T00:14:53.658208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:17.044367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265467+0000","last_change":"2024-04-25T00:13:23.589613+0000","last_active":"2024-04-25T00:14:11.265467+0000","last_peered":"2024-04-25T00:14:11.265467+0000","last_clean":"2024-04-25T00:14:11.265467+0000","last_became_active":"2024-04-25T00:13:23.589323+0000","last_became_peered":"2024-04-25T00:13:23.589323+0000","last_unstale":"2024-04-25T00:14:11.265467+0000","last_undegraded":"2024-04-25T00:14:11.265467+0000","last_fullsized":"2024-04-25T00:14:11.265467+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:34:42.272706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.266378+0000","last_change":"2024-04-25T00:13:19.417775+0000","last_active":"2024-04-25T00:14:11.266378+0000","last_peered":"2024-04-25T00:14:11.266378+0000","last_clean":"2024-04-25T00:14:11.266378+0000","last_became_active":"2024-04-25T00:13:19.415096+0000","last_became_peered":"2024-04-25T00:13:19.415096+0000","last_unstale":"2024-04-25T00:14:11.266378+0000","last_undegraded":"2024-04-25T00:14:11.266378+0000","last_fullsized":"2024-04-25T00:14:11.266378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:54.975923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263907+0000","last_change":"2024-04-25T00:13:18.514560+0000","last_active":"2024-04-25T00:14:11.263907+0000","last_peered":"2024-04-25T00:14:11.263907+0000","last_clean":"2024-04-25T00:14:11.263907+0000","last_became_active":"2024-04-25T00:13:18.514034+0000","last_became_peered":"2024-04-25T00:13:18.514034+0000","last_unstale":"2024-04-25T00:14:11.263907+0000","last_undegraded":"2024-04-25T00:14:11.263907+0000","last_fullsized":"2024-04-25T00:14:11.263907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:35:18.110206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":359,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265210+0000","last_change":"2024-04-25T00:12:01.540807+0000","last_active":"2024-04-25T00:14:11.265210+0000","last_peered":"2024-04-25T00:14:11.265210+0000","last_clean":"2024-04-25T00:14:11.265210+0000","last_became_active":"2024-04-25T00:12:01.540272+0000","last_became_peered":"2024-04-25T00:12:01.540272+0000","last_unstale":"2024-04-25T00:14:11.265210+0000","last_undegraded":"2024-04-25T00:14:11.265210+0000","last_fullsized":"2024-04-25T00:14:11.265210+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:08:34.007429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:08:34.007429+0000","last_clean_scrub_stamp":"2024-04-25T00:08:34.007429+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:50:14.642604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263851+0000","last_change":"2024-04-25T00:13:23.862493+0000","last_active":"2024-04-25T00:14:11.263851+0000","last_peered":"2024-04-25T00:14:11.263851+0000","last_clean":"2024-04-25T00:14:11.263851+0000","last_became_active":"2024-04-25T00:13:23.861972+0000","last_became_peered":"2024-04-25T00:13:23.861972+0000","last_unstale":"2024-04-25T00:14:11.263851+0000","last_undegraded":"2024-04-25T00:14:11.263851+0000","last_fullsized":"2024-04-25T00:14:11.263851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:49:13.507357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265066+0000","last_change":"2024-04-25T00:13:19.421920+0000","last_active":"2024-04-25T00:14:11.265066+0000","last_peered":"2024-04-25T00:14:11.265066+0000","last_clean":"2024-04-25T00:14:11.265066+0000","last_became_active":"2024-04-25T00:13:19.421585+0000","last_became_peered":"2024-04-25T00:13:19.421585+0000","last_unstale":"2024-04-25T00:14:11.265066+0000","last_undegraded":"2024-04-25T00:14:11.265066+0000","last_fullsized":"2024-04-25T00:14:11.265066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:53:24.937520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265734+0000","last_change":"2024-04-25T00:13:18.527062+0000","last_active":"2024-04-25T00:14:11.265734+0000","last_peered":"2024-04-25T00:14:11.265734+0000","last_clean":"2024-04-25T00:14:11.265734+0000","last_became_active":"2024-04-25T00:13:18.526338+0000","last_became_peered":"2024-04-25T00:13:18.526338+0000","last_unstale":"2024-04-25T00:14:11.265734+0000","last_undegraded":"2024-04-25T00:14:11.265734+0000","last_fullsized":"2024-04-25T00:14:11.265734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:21.756728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263018+0000","last_change":"2024-04-25T00:13:23.575162+0000","last_active":"2024-04-25T00:14:11.263018+0000","last_peered":"2024-04-25T00:14:11.263018+0000","last_clean":"2024-04-25T00:14:11.263018+0000","last_became_active":"2024-04-25T00:13:23.574722+0000","last_became_peered":"2024-04-25T00:13:23.574722+0000","last_unstale":"2024-04-25T00:14:11.263018+0000","last_undegraded":"2024-04-25T00:14:11.263018+0000","last_fullsized":"2024-04-25T00:14:11.263018+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:52.958539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264851+0000","last_change":"2024-04-25T00:13:18.523019+0000","last_active":"2024-04-25T00:14:11.264851+0000","last_peered":"2024-04-25T00:14:11.264851+0000","last_clean":"2024-04-25T00:14:11.264851+0000","last_became_active":"2024-04-25T00:13:18.522677+0000","last_became_peered":"2024-04-25T00:13:18.522677+0000","last_unstale":"2024-04-25T00:14:11.264851+0000","last_undegraded":"2024-04-25T00:14:11.264851+0000","last_fullsized":"2024-04-25T00:14:11.264851+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:14.470154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.267567+0000","last_change":"2024-04-25T00:13:19.417672+0000","last_active":"2024-04-25T00:14:11.267567+0000","last_peered":"2024-04-25T00:14:11.267567+0000","last_clean":"2024-04-25T00:14:11.267567+0000","last_became_active":"2024-04-25T00:13:19.417323+0000","last_became_peered":"2024-04-25T00:13:19.417323+0000","last_unstale":"2024-04-25T00:14:11.267567+0000","last_undegraded":"2024-04-25T00:14:11.267567+0000","last_fullsized":"2024-04-25T00:14:11.267567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:53.734784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.266386+0000","last_change":"2024-04-25T00:13:23.862320+0000","last_active":"2024-04-25T00:14:11.266386+0000","last_peered":"2024-04-25T00:14:11.266386+0000","last_clean":"2024-04-25T00:14:11.266386+0000","last_became_active":"2024-04-25T00:13:23.861895+0000","last_became_peered":"2024-04-25T00:13:23.861895+0000","last_unstale":"2024-04-25T00:14:11.266386+0000","last_undegraded":"2024-04-25T00:14:11.266386+0000","last_fullsized":"2024-04-25T00:14:11.266386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:16:21.324727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261793+0000","last_change":"2024-04-25T00:13:18.520989+0000","last_active":"2024-04-25T00:14:11.261793+0000","last_peered":"2024-04-25T00:14:11.261793+0000","last_clean":"2024-04-25T00:14:11.261793+0000","last_became_active":"2024-04-25T00:13:18.520566+0000","last_became_peered":"2024-04-25T00:13:18.520566+0000","last_unstale":"2024-04-25T00:14:11.261793+0000","last_undegraded":"2024-04-25T00:14:11.261793+0000","last_fullsized":"2024-04-25T00:14:11.261793+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:22:32.532352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265660+0000","last_change":"2024-04-25T00:13:19.422741+0000","last_active":"2024-04-25T00:14:11.265660+0000","last_peered":"2024-04-25T00:14:11.265660+0000","last_clean":"2024-04-25T00:14:11.265660+0000","last_became_active":"2024-04-25T00:13:19.422379+0000","last_became_peered":"2024-04-25T00:13:19.422379+0000","last_unstale":"2024-04-25T00:14:11.265660+0000","last_undegraded":"2024-04-25T00:14:11.265660+0000","last_fullsized":"2024-04-25T00:14:11.265660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:41.337641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.259565+0000","last_change":"2024-04-25T00:13:23.577888+0000","last_active":"2024-04-25T00:14:11.259565+0000","last_peered":"2024-04-25T00:14:11.259565+0000","last_clean":"2024-04-25T00:14:11.259565+0000","last_became_active":"2024-04-25T00:13:23.577440+0000","last_became_peered":"2024-04-25T00:13:23.577440+0000","last_unstale":"2024-04-25T00:14:11.259565+0000","last_undegraded":"2024-04-25T00:14:11.259565+0000","last_fullsized":"2024-04-25T00:14:11.259565+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:03.760277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264429+0000","last_change":"2024-04-25T00:13:18.527996+0000","last_active":"2024-04-25T00:14:11.264429+0000","last_peered":"2024-04-25T00:14:11.264429+0000","last_clean":"2024-04-25T00:14:11.264429+0000","last_became_active":"2024-04-25T00:13:18.527636+0000","last_became_peered":"2024-04-25T00:13:18.527636+0000","last_unstale":"2024-04-25T00:14:11.264429+0000","last_undegraded":"2024-04-25T00:14:11.264429+0000","last_fullsized":"2024-04-25T00:14:11.264429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:52.137108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263053+0000","last_change":"2024-04-25T00:13:19.425538+0000","last_active":"2024-04-25T00:14:11.263053+0000","last_peered":"2024-04-25T00:14:11.263053+0000","last_clean":"2024-04-25T00:14:11.263053+0000","last_became_active":"2024-04-25T00:13:19.420790+0000","last_became_peered":"2024-04-25T00:13:19.420790+0000","last_unstale":"2024-04-25T00:14:11.263053+0000","last_undegraded":"2024-04-25T00:14:11.263053+0000","last_fullsized":"2024-04-25T00:14:11.263053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:04.319759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264168+0000","last_change":"2024-04-25T00:13:23.863187+0000","last_active":"2024-04-25T00:14:11.264168+0000","last_peered":"2024-04-25T00:14:11.264168+0000","last_clean":"2024-04-25T00:14:11.264168+0000","last_became_active":"2024-04-25T00:13:23.862206+0000","last_became_peered":"2024-04-25T00:13:23.862206+0000","last_unstale":"2024-04-25T00:14:11.264168+0000","last_undegraded":"2024-04-25T00:14:11.264168+0000","last_fullsized":"2024-04-25T00:14:11.264168+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:57:17.395007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260093+0000","last_change":"2024-04-25T00:13:18.529224+0000","last_active":"2024-04-25T00:14:11.260093+0000","last_peered":"2024-04-25T00:14:11.260093+0000","last_clean":"2024-04-25T00:14:11.260093+0000","last_became_active":"2024-04-25T00:13:18.528908+0000","last_became_peered":"2024-04-25T00:13:18.528908+0000","last_unstale":"2024-04-25T00:14:11.260093+0000","last_undegraded":"2024-04-25T00:14:11.260093+0000","last_fullsized":"2024-04-25T00:14:11.260093+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:27.035726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264608+0000","last_change":"2024-04-25T00:13:19.422223+0000","last_active":"2024-04-25T00:14:11.264608+0000","last_peered":"2024-04-25T00:14:11.264608+0000","last_clean":"2024-04-25T00:14:11.264608+0000","last_became_active":"2024-04-25T00:13:19.421524+0000","last_became_peered":"2024-04-25T00:13:19.421524+0000","last_unstale":"2024-04-25T00:14:11.264608+0000","last_undegraded":"2024-04-25T00:14:11.264608+0000","last_fullsized":"2024-04-25T00:14:11.264608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:17:01.849503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261900+0000","last_change":"2024-04-25T00:13:23.583015+0000","last_active":"2024-04-25T00:14:11.261900+0000","last_peered":"2024-04-25T00:14:11.261900+0000","last_clean":"2024-04-25T00:14:11.261900+0000","last_became_active":"2024-04-25T00:13:23.582612+0000","last_became_peered":"2024-04-25T00:13:23.582612+0000","last_unstale":"2024-04-25T00:14:11.261900+0000","last_undegraded":"2024-04-25T00:14:11.261900+0000","last_fullsized":"2024-04-25T00:14:11.261900+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:36.872281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265958+0000","last_change":"2024-04-25T00:13:18.530027+0000","last_active":"2024-04-25T00:14:11.265958+0000","last_peered":"2024-04-25T00:14:11.265958+0000","last_clean":"2024-04-25T00:14:11.265958+0000","last_became_active":"2024-04-25T00:13:18.529717+0000","last_became_peered":"2024-04-25T00:13:18.529717+0000","last_unstale":"2024-04-25T00:14:11.265958+0000","last_undegraded":"2024-04-25T00:14:11.265958+0000","last_fullsized":"2024-04-25T00:14:11.265958+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:23:31.020637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261938+0000","last_change":"2024-04-25T00:13:19.428650+0000","last_active":"2024-04-25T00:14:11.261938+0000","last_peered":"2024-04-25T00:14:11.261938+0000","last_clean":"2024-04-25T00:14:11.261938+0000","last_became_active":"2024-04-25T00:13:19.428156+0000","last_became_peered":"2024-04-25T00:13:19.428156+0000","last_unstale":"2024-04-25T00:14:11.261938+0000","last_undegraded":"2024-04-25T00:14:11.261938+0000","last_fullsized":"2024-04-25T00:14:11.261938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:43.094529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264014+0000","last_change":"2024-04-25T00:13:23.579125+0000","last_active":"2024-04-25T00:14:11.264014+0000","last_peered":"2024-04-25T00:14:11.264014+0000","last_clean":"2024-04-25T00:14:11.264014+0000","last_became_active":"2024-04-25T00:13:23.578728+0000","last_became_peered":"2024-04-25T00:13:23.578728+0000","last_unstale":"2024-04-25T00:14:11.264014+0000","last_undegraded":"2024-04-25T00:14:11.264014+0000","last_fullsized":"2024-04-25T00:14:11.264014+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:16:16.834399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.278099+0000","last_change":"2024-04-25T00:14:11.278099+0000","last_active":"2024-04-25T00:14:11.278099+0000","last_peered":"2024-04-25T00:14:11.278099+0000","last_clean":"2024-04-25T00:14:11.278099+0000","last_became_active":"2024-04-25T00:14:11.269977+0000","last_became_peered":"2024-04-25T00:14:11.269977+0000","last_unstale":"2024-04-25T00:14:11.278099+0000","last_undegraded":"2024-04-25T00:14:11.278099+0000","last_fullsized":"2024-04-25T00:14:11.278099+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:48:59.261854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.266793+0000","last_change":"2024-04-25T00:13:19.414848+0000","last_active":"2024-04-25T00:14:11.266793+0000","last_peered":"2024-04-25T00:14:11.266793+0000","last_clean":"2024-04-25T00:14:11.266793+0000","last_became_active":"2024-04-25T00:13:19.414370+0000","last_became_peered":"2024-04-25T00:13:19.414370+0000","last_unstale":"2024-04-25T00:14:11.266793+0000","last_undegraded":"2024-04-25T00:14:11.266793+0000","last_fullsized":"2024-04-25T00:14:11.266793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:41:08.779171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264049+0000","last_change":"2024-04-25T00:13:23.862577+0000","last_active":"2024-04-25T00:14:11.264049+0000","last_peered":"2024-04-25T00:14:11.264049+0000","last_clean":"2024-04-25T00:14:11.264049+0000","last_became_active":"2024-04-25T00:13:23.862098+0000","last_became_peered":"2024-04-25T00:13:23.862098+0000","last_unstale":"2024-04-25T00:14:11.264049+0000","last_undegraded":"2024-04-25T00:14:11.264049+0000","last_fullsized":"2024-04-25T00:14:11.264049+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:08:00.344041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.274790+0000","last_change":"2024-04-25T00:14:11.274790+0000","last_active":"2024-04-25T00:14:11.274790+0000","last_peered":"2024-04-25T00:14:11.274790+0000","last_clean":"2024-04-25T00:14:11.274790+0000","last_became_active":"2024-04-25T00:14:11.267132+0000","last_became_peered":"2024-04-25T00:14:11.267132+0000","last_unstale":"2024-04-25T00:14:11.274790+0000","last_undegraded":"2024-04-25T00:14:11.274790+0000","last_fullsized":"2024-04-25T00:14:11.274790+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:58.302788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265720+0000","last_change":"2024-04-25T00:13:19.427992+0000","last_active":"2024-04-25T00:14:11.265720+0000","last_peered":"2024-04-25T00:14:11.265720+0000","last_clean":"2024-04-25T00:14:11.265720+0000","last_became_active":"2024-04-25T00:13:19.427665+0000","last_became_peered":"2024-04-25T00:13:19.427665+0000","last_unstale":"2024-04-25T00:14:11.265720+0000","last_undegraded":"2024-04-25T00:14:11.265720+0000","last_fullsized":"2024-04-25T00:14:11.265720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:12:31.582185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260101+0000","last_change":"2024-04-25T00:13:23.582430+0000","last_active":"2024-04-25T00:14:11.260101+0000","last_peered":"2024-04-25T00:14:11.260101+0000","last_clean":"2024-04-25T00:14:11.260101+0000","last_became_active":"2024-04-25T00:13:23.582008+0000","last_became_peered":"2024-04-25T00:13:23.582008+0000","last_unstale":"2024-04-25T00:14:11.260101+0000","last_undegraded":"2024-04-25T00:14:11.260101+0000","last_fullsized":"2024-04-25T00:14:11.260101+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:11.068720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:56.683982+0000","last_change":"2024-04-25T00:13:18.528030+0000","last_active":"2024-04-25T00:14:56.683982+0000","last_peered":"2024-04-25T00:14:56.683982+0000","last_clean":"2024-04-25T00:14:56.683982+0000","last_became_active":"2024-04-25T00:13:18.527636+0000","last_became_peered":"2024-04-25T00:13:18.527636+0000","last_unstale":"2024-04-25T00:14:56.683982+0000","last_undegraded":"2024-04-25T00:14:56.683982+0000","last_fullsized":"2024-04-25T00:14:56.683982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:36.311810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265767+0000","last_change":"2024-04-25T00:13:19.428181+0000","last_active":"2024-04-25T00:14:11.265767+0000","last_peered":"2024-04-25T00:14:11.265767+0000","last_clean":"2024-04-25T00:14:11.265767+0000","last_became_active":"2024-04-25T00:13:19.427785+0000","last_became_peered":"2024-04-25T00:13:19.427785+0000","last_unstale":"2024-04-25T00:14:11.265767+0000","last_undegraded":"2024-04-25T00:14:11.265767+0000","last_fullsized":"2024-04-25T00:14:11.265767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:31.745679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.271507+0000","last_change":"2024-04-25T00:14:11.271507+0000","last_active":"2024-04-25T00:14:11.271507+0000","last_peered":"2024-04-25T00:14:11.271507+0000","last_clean":"2024-04-25T00:14:11.271507+0000","last_became_active":"2024-04-25T00:14:11.270452+0000","last_became_peered":"2024-04-25T00:14:11.270452+0000","last_unstale":"2024-04-25T00:14:11.271507+0000","last_undegraded":"2024-04-25T00:14:11.271507+0000","last_fullsized":"2024-04-25T00:14:11.271507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:19.955242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.263271+0000","last_change":"2024-04-25T00:13:18.529622+0000","last_active":"2024-04-25T00:14:11.263271+0000","last_peered":"2024-04-25T00:14:11.263271+0000","last_clean":"2024-04-25T00:14:11.263271+0000","last_became_active":"2024-04-25T00:13:18.529305+0000","last_became_peered":"2024-04-25T00:13:18.529305+0000","last_unstale":"2024-04-25T00:14:11.263271+0000","last_undegraded":"2024-04-25T00:14:11.263271+0000","last_fullsized":"2024-04-25T00:14:11.263271+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:10:43.334712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265609+0000","last_change":"2024-04-25T00:13:19.421783+0000","last_active":"2024-04-25T00:14:11.265609+0000","last_peered":"2024-04-25T00:14:11.265609+0000","last_clean":"2024-04-25T00:14:11.265609+0000","last_became_active":"2024-04-25T00:13:19.421447+0000","last_became_peered":"2024-04-25T00:13:19.421447+0000","last_unstale":"2024-04-25T00:14:11.265609+0000","last_undegraded":"2024-04-25T00:14:11.265609+0000","last_fullsized":"2024-04-25T00:14:11.265609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:16.107220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264096+0000","last_change":"2024-04-25T00:13:23.578252+0000","last_active":"2024-04-25T00:14:11.264096+0000","last_peered":"2024-04-25T00:14:11.264096+0000","last_clean":"2024-04-25T00:14:11.264096+0000","last_became_active":"2024-04-25T00:13:23.577779+0000","last_became_peered":"2024-04-25T00:13:23.577779+0000","last_unstale":"2024-04-25T00:14:11.264096+0000","last_undegraded":"2024-04-25T00:14:11.264096+0000","last_fullsized":"2024-04-25T00:14:11.264096+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:43.822447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.267645+0000","last_change":"2024-04-25T00:13:18.523583+0000","last_active":"2024-04-25T00:14:11.267645+0000","last_peered":"2024-04-25T00:14:11.267645+0000","last_clean":"2024-04-25T00:14:11.267645+0000","last_became_active":"2024-04-25T00:13:18.523238+0000","last_became_peered":"2024-04-25T00:13:18.523238+0000","last_unstale":"2024-04-25T00:14:11.267645+0000","last_undegraded":"2024-04-25T00:14:11.267645+0000","last_fullsized":"2024-04-25T00:14:11.267645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:03.869276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265969+0000","last_change":"2024-04-25T00:13:19.426021+0000","last_active":"2024-04-25T00:14:11.265969+0000","last_peered":"2024-04-25T00:14:11.265969+0000","last_clean":"2024-04-25T00:14:11.265969+0000","last_became_active":"2024-04-25T00:13:19.425676+0000","last_became_peered":"2024-04-25T00:13:19.425676+0000","last_unstale":"2024-04-25T00:14:11.265969+0000","last_undegraded":"2024-04-25T00:14:11.265969+0000","last_fullsized":"2024-04-25T00:14:11.265969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:10.044202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.261982+0000","last_change":"2024-04-25T00:13:19.422402+0000","last_active":"2024-04-25T00:14:11.261982+0000","last_peered":"2024-04-25T00:14:11.261982+0000","last_clean":"2024-04-25T00:14:11.261982+0000","last_became_active":"2024-04-25T00:13:19.422057+0000","last_became_peered":"2024-04-25T00:13:19.422057+0000","last_unstale":"2024-04-25T00:14:11.261982+0000","last_undegraded":"2024-04-25T00:14:11.261982+0000","last_fullsized":"2024-04-25T00:14:11.261982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:31:43.703030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.259941+0000","last_change":"2024-04-25T00:13:19.425434+0000","last_active":"2024-04-25T00:14:11.259941+0000","last_peered":"2024-04-25T00:14:11.259941+0000","last_clean":"2024-04-25T00:14:11.259941+0000","last_became_active":"2024-04-25T00:13:19.425051+0000","last_became_peered":"2024-04-25T00:13:19.425051+0000","last_unstale":"2024-04-25T00:14:11.259941+0000","last_undegraded":"2024-04-25T00:14:11.259941+0000","last_fullsized":"2024-04-25T00:14:11.259941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:17.568415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.259653+0000","last_change":"2024-04-25T00:13:19.413835+0000","last_active":"2024-04-25T00:14:11.259653+0000","last_peered":"2024-04-25T00:14:11.259653+0000","last_clean":"2024-04-25T00:14:11.259653+0000","last_became_active":"2024-04-25T00:13:19.413406+0000","last_became_peered":"2024-04-25T00:13:19.413406+0000","last_unstale":"2024-04-25T00:14:11.259653+0000","last_undegraded":"2024-04-25T00:14:11.259653+0000","last_fullsized":"2024-04-25T00:14:11.259653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:33.852391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262898+0000","last_change":"2024-04-25T00:13:19.426875+0000","last_active":"2024-04-25T00:14:11.262898+0000","last_peered":"2024-04-25T00:14:11.262898+0000","last_clean":"2024-04-25T00:14:11.262898+0000","last_became_active":"2024-04-25T00:13:19.425828+0000","last_became_peered":"2024-04-25T00:13:19.425828+0000","last_unstale":"2024-04-25T00:14:11.262898+0000","last_undegraded":"2024-04-25T00:14:11.262898+0000","last_fullsized":"2024-04-25T00:14:11.262898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:21.119551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.267662+0000","last_change":"2024-04-25T00:13:19.423637+0000","last_active":"2024-04-25T00:14:11.267662+0000","last_peered":"2024-04-25T00:14:11.267662+0000","last_clean":"2024-04-25T00:14:11.267662+0000","last_became_active":"2024-04-25T00:13:19.423272+0000","last_became_peered":"2024-04-25T00:13:19.423272+0000","last_unstale":"2024-04-25T00:14:11.267662+0000","last_undegraded":"2024-04-25T00:14:11.267662+0000","last_fullsized":"2024-04-25T00:14:11.267662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:32:38.794966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.262672+0000","last_change":"2024-04-25T00:13:19.423538+0000","last_active":"2024-04-25T00:14:11.262672+0000","last_peered":"2024-04-25T00:14:11.262672+0000","last_clean":"2024-04-25T00:14:11.262672+0000","last_became_active":"2024-04-25T00:13:19.423028+0000","last_became_peered":"2024-04-25T00:13:19.423028+0000","last_unstale":"2024-04-25T00:14:11.262672+0000","last_undegraded":"2024-04-25T00:14:11.262672+0000","last_fullsized":"2024-04-25T00:14:11.262672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:29:41.975044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260453+0000","last_change":"2024-04-25T00:13:19.416899+0000","last_active":"2024-04-25T00:14:11.260453+0000","last_peered":"2024-04-25T00:14:11.260453+0000","last_clean":"2024-04-25T00:14:11.260453+0000","last_became_active":"2024-04-25T00:13:19.416505+0000","last_became_peered":"2024-04-25T00:13:19.416505+0000","last_unstale":"2024-04-25T00:14:11.260453+0000","last_undegraded":"2024-04-25T00:14:11.260453+0000","last_fullsized":"2024-04-25T00:14:11.260453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:37:40.204119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265830+0000","last_change":"2024-04-25T00:13:19.426215+0000","last_active":"2024-04-25T00:14:11.265830+0000","last_peered":"2024-04-25T00:14:11.265830+0000","last_clean":"2024-04-25T00:14:11.265830+0000","last_became_active":"2024-04-25T00:13:19.422804+0000","last_became_peered":"2024-04-25T00:13:19.422804+0000","last_unstale":"2024-04-25T00:14:11.265830+0000","last_undegraded":"2024-04-25T00:14:11.265830+0000","last_fullsized":"2024-04-25T00:14:11.265830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:24:44.630891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.264522+0000","last_change":"2024-04-25T00:13:19.413564+0000","last_active":"2024-04-25T00:14:11.264522+0000","last_peered":"2024-04-25T00:14:11.264522+0000","last_clean":"2024-04-25T00:14:11.264522+0000","last_became_active":"2024-04-25T00:13:19.413223+0000","last_became_peered":"2024-04-25T00:13:19.413223+0000","last_unstale":"2024-04-25T00:14:11.264522+0000","last_undegraded":"2024-04-25T00:14:11.264522+0000","last_fullsized":"2024-04-25T00:14:11.264522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:45.629018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265855+0000","last_change":"2024-04-25T00:13:19.425945+0000","last_active":"2024-04-25T00:14:11.265855+0000","last_peered":"2024-04-25T00:14:11.265855+0000","last_clean":"2024-04-25T00:14:11.265855+0000","last_became_active":"2024-04-25T00:13:19.422499+0000","last_became_peered":"2024-04-25T00:13:19.422499+0000","last_unstale":"2024-04-25T00:14:11.265855+0000","last_undegraded":"2024-04-25T00:14:11.265855+0000","last_fullsized":"2024-04-25T00:14:11.265855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:45.299300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.260372+0000","last_change":"2024-04-25T00:13:19.418857+0000","last_active":"2024-04-25T00:14:11.260372+0000","last_peered":"2024-04-25T00:14:11.260372+0000","last_clean":"2024-04-25T00:14:11.260372+0000","last_became_active":"2024-04-25T00:13:19.418535+0000","last_became_peered":"2024-04-25T00:13:19.418535+0000","last_unstale":"2024-04-25T00:14:11.260372+0000","last_undegraded":"2024-04-25T00:14:11.260372+0000","last_fullsized":"2024-04-25T00:14:11.260372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:44.288129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265861+0000","last_change":"2024-04-25T00:13:19.420550+0000","last_active":"2024-04-25T00:14:11.265861+0000","last_peered":"2024-04-25T00:14:11.265861+0000","last_clean":"2024-04-25T00:14:11.265861+0000","last_became_active":"2024-04-25T00:13:19.420235+0000","last_became_peered":"2024-04-25T00:13:19.420235+0000","last_unstale":"2024-04-25T00:14:11.265861+0000","last_undegraded":"2024-04-25T00:14:11.265861+0000","last_fullsized":"2024-04-25T00:14:11.265861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:59:36.543776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T00:14:11.265433+0000","last_change":"2024-04-25T00:13:19.429995+0000","last_active":"2024-04-25T00:14:11.265433+0000","last_peered":"2024-04-25T00:14:11.265433+0000","last_clean":"2024-04-25T00:14:11.265433+0000","last_became_active":"2024-04-25T00:13:19.429185+0000","last_became_peered":"2024-04-25T00:13:19.429185+0000","last_unstale":"2024-04-25T00:14:11.265433+0000","last_undegraded":"2024-04-25T00:14:11.265433+0000","last_fullsized":"2024-04-25T00:14:11.265433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:29:27.974479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":213064,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67964,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":753,"num_read_kb":450,"num_write":238,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":203892,"data_compressed":4806,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":501433,"data_compressed":20615,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28140,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717012,"statfs":{"total":95995035648,"available":95966220288,"internally_reserved":0,"allocated":1146880,"data_stored":1149791,"data_compressed":58001,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808876,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":505459,"data_compressed":20615,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070513,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":501741,"data_compressed":20615,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":1098238,"data_compressed":56399,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495675,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":496867,"data_compressed":20615,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":559048,"data_compressed":22217,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":498323,"data_compressed":20615,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":559268,"data_compressed":22217,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":496575,"data_compressed":20615,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920837,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":501332,"data_compressed":20615,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1085440,"data_stored":1093429,"data_compressed":56399,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62903,"data_compressed":1602,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":63123,"data_compressed":1602,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":63278,"data_compressed":1602,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:15:02.376 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T00:15:02.376 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T00:15:02.377 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-25T00:15:02.377 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-25T00:15:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:15:03.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:03 smithi027 ceph-mon[29208]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:15:03.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:03 smithi027 ceph-mon[29208]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:15:03.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:03 smithi139 ceph-mon[32833]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:15:03.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:03 smithi139 ceph-mon[32833]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:15:03.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:03 smithi153 ceph-mon[32896]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:15:03.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:03 smithi153 ceph-mon[32896]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:15:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.345+0000 7fc5549fd700 1 -- 172.21.15.27:0/1233790748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc550107570 msgr2=0x7fc55010f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.345+0000 7fc5549fd700 1 --2- 172.21.15.27:0/1233790748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc550107570 0x7fc55010f140 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc54800b0a0 tx=0x7fc54800b3b0 comp rx=0 tx=0).stop 2024-04-25T00:15:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.345+0000 7fc5549fd700 1 -- 172.21.15.27:0/1233790748 shutdown_connections 2024-04-25T00:15:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.345+0000 7fc5549fd700 1 --2- 172.21.15.27:0/1233790748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc550107570 0x7fc55010f140 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.345+0000 7fc5549fd700 1 --2- 172.21.15.27:0/1233790748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5501024e0 0x7fc5501028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.345+0000 7fc5549fd700 1 --2- 172.21.15.27:0/1233790748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc550102e90 0x7fc550106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.345+0000 7fc5549fd700 1 -- 172.21.15.27:0/1233790748 >> 172.21.15.27:0/1233790748 conn(0x7fc550079390 msgr2=0x7fc5500797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.345+0000 7fc5549fd700 1 -- 172.21.15.27:0/1233790748 shutdown_connections 2024-04-25T00:15:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.345+0000 7fc5549fd700 1 -- 172.21.15.27:0/1233790748 wait complete. 2024-04-25T00:15:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.346+0000 7fc5549fd700 1 Processor -- start 2024-04-25T00:15:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.346+0000 7fc5549fd700 1 -- start start 2024-04-25T00:15:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.346+0000 7fc5549fd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5501024e0 0x7fc55019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.346+0000 7fc5549fd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc550102e90 0x7fc55019eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.346+0000 7fc5549fd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc550107570 0x7fc5501987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.346+0000 7fc5549fd700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc550112550 con 0x7fc550107570 2024-04-25T00:15:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.346+0000 7fc5549fd700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc5501123d0 con 0x7fc5501024e0 2024-04-25T00:15:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.346+0000 7fc5549fd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5501126d0 con 0x7fc550102e90 2024-04-25T00:15:04.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.347+0000 7fc54e59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5501024e0 0x7fc55019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:04.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.347+0000 7fc54ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc550107570 0x7fc5501987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:04.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.347+0000 7fc54ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc550107570 0x7fc5501987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48698/0 (socket says 172.21.15.27:48698) 2024-04-25T00:15:04.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.347+0000 7fc54ed9d700 1 -- 172.21.15.27:0/1950314007 learned_addr learned my addr 172.21.15.27:0/1950314007 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:04.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.347+0000 7fc54e59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5501024e0 0x7fc55019e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:57378/0 (socket says 172.21.15.27:57378) 2024-04-25T00:15:04.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.347+0000 7fc54dd9b700 1 --2- 172.21.15.27:0/1950314007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc550102e90 0x7fc55019eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:04.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.347+0000 7fc54ed9d700 1 -- 172.21.15.27:0/1950314007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc550102e90 msgr2=0x7fc55019eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.347+0000 7fc54ed9d700 1 --2- 172.21.15.27:0/1950314007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc550102e90 0x7fc55019eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.347+0000 7fc54ed9d700 1 -- 172.21.15.27:0/1950314007 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5501024e0 msgr2=0x7fc55019e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.347+0000 7fc54ed9d700 1 --2- 172.21.15.27:0/1950314007 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5501024e0 0x7fc55019e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.347+0000 7fc54ed9d700 1 -- 172.21.15.27:0/1950314007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc550198fb0 con 0x7fc550107570 2024-04-25T00:15:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.347+0000 7fc54ed9d700 1 --2- 172.21.15.27:0/1950314007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc550107570 0x7fc5501987d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc5480062a0 tx=0x7fc548004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.347+0000 7fc53f7fe700 1 -- 172.21.15.27:0/1950314007 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc548021070 con 0x7fc550107570 2024-04-25T00:15:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.347+0000 7fc5549fd700 1 -- 172.21.15.27:0/1950314007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc550199240 con 0x7fc550107570 2024-04-25T00:15:04.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.347+0000 7fc53f7fe700 1 -- 172.21.15.27:0/1950314007 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc548003c00 con 0x7fc550107570 2024-04-25T00:15:04.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.347+0000 7fc53f7fe700 1 -- 172.21.15.27:0/1950314007 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc548004880 con 0x7fc550107570 2024-04-25T00:15:04.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.347+0000 7fc5549fd700 1 -- 172.21.15.27:0/1950314007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc550074460 con 0x7fc550107570 2024-04-25T00:15:04.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.348+0000 7fc5549fd700 1 -- 172.21.15.27:0/1950314007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc550103970 con 0x7fc550107570 2024-04-25T00:15:04.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.350+0000 7fc53f7fe700 1 -- 172.21.15.27:0/1950314007 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc548003db0 con 0x7fc550107570 2024-04-25T00:15:04.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.350+0000 7fc53f7fe700 1 --2- 172.21.15.27:0/1950314007 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc5380a2cb0 0x7fc5380a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:04.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.350+0000 7fc54e59c700 1 --2- 172.21.15.27:0/1950314007 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc5380a2cb0 0x7fc5380a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:04.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.350+0000 7fc53f7fe700 1 -- 172.21.15.27:0/1950314007 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc54800f020 con 0x7fc550107570 2024-04-25T00:15:04.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.351+0000 7fc54e59c700 1 --2- 172.21.15.27:0/1950314007 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc5380a2cb0 0x7fc5380a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc540004150 tx=0x7fc54000f7d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:04.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.354+0000 7fc53f7fe700 1 -- 172.21.15.27:0/1950314007 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc54802a020 con 0x7fc550107570 2024-04-25T00:15:04.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.611+0000 7fc5549fd700 1 -- 172.21.15.27:0/1950314007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fc55002db30 con 0x7fc550107570 2024-04-25T00:15:04.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.611+0000 7fc53f7fe700 1 -- 172.21.15.27:0/1950314007 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fc54809d500 con 0x7fc550107570 2024-04-25T00:15:04.611 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:04.611 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T00:15:04.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.613+0000 7fc5549fd700 1 -- 172.21.15.27:0/1950314007 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc5380a2cb0 msgr2=0x7fc5380a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:04.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.613+0000 7fc5549fd700 1 --2- 172.21.15.27:0/1950314007 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc5380a2cb0 0x7fc5380a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc540004150 tx=0x7fc54000f7d0 comp rx=0 tx=0).stop 2024-04-25T00:15:04.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.613+0000 7fc5549fd700 1 -- 172.21.15.27:0/1950314007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc550107570 msgr2=0x7fc5501987d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:04.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.613+0000 7fc5549fd700 1 --2- 172.21.15.27:0/1950314007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc550107570 0x7fc5501987d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc5480062a0 tx=0x7fc548004d70 comp rx=0 tx=0).stop 2024-04-25T00:15:04.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.613+0000 7fc5549fd700 1 -- 172.21.15.27:0/1950314007 shutdown_connections 2024-04-25T00:15:04.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.613+0000 7fc5549fd700 1 --2- 172.21.15.27:0/1950314007 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc5380a2cb0 0x7fc5380a5170 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:04.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.613+0000 7fc5549fd700 1 --2- 172.21.15.27:0/1950314007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc550107570 0x7fc5501987d0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:04.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.613+0000 7fc5549fd700 1 --2- 172.21.15.27:0/1950314007 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5501024e0 0x7fc55019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:04.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.613+0000 7fc5549fd700 1 --2- 172.21.15.27:0/1950314007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc550102e90 0x7fc55019eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:04.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.613+0000 7fc5549fd700 1 -- 172.21.15.27:0/1950314007 >> 172.21.15.27:0/1950314007 conn(0x7fc550079390 msgr2=0x7fc550100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:04.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.614+0000 7fc5549fd700 1 -- 172.21.15.27:0/1950314007 shutdown_connections 2024-04-25T00:15:04.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:04.614+0000 7fc5549fd700 1 -- 172.21.15.27:0/1950314007 wait complete. 2024-04-25T00:15:05.412 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-25T00:15:05.412 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:15:05.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.520+0000 7f8b87a6d640 1 -- 172.21.15.27:0/3483251637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b80101890 msgr2=0x7f8b80101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.520+0000 7f8b87a6d640 1 --2- 172.21.15.27:0/3483251637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b80101890 0x7f8b80101d10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009900 tx=0x7f8b7002a410 comp rx=0 tx=0).stop 2024-04-25T00:15:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.520+0000 7f8b87a6d640 1 -- 172.21.15.27:0/3483251637 shutdown_connections 2024-04-25T00:15:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.520+0000 7f8b87a6d640 1 --2- 172.21.15.27:0/3483251637 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b80102250 0x7f8b80108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.520+0000 7f8b87a6d640 1 --2- 172.21.15.27:0/3483251637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b80101890 0x7f8b80101d10 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.520+0000 7f8b87a6d640 1 --2- 172.21.15.27:0/3483251637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b801006a0 0x7f8b80100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.520+0000 7f8b87a6d640 1 -- 172.21.15.27:0/3483251637 >> 172.21.15.27:0/3483251637 conn(0x7f8b800fbc30 msgr2=0x7f8b800fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.520+0000 7f8b87a6d640 1 -- 172.21.15.27:0/3483251637 shutdown_connections 2024-04-25T00:15:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.520+0000 7f8b87a6d640 1 -- 172.21.15.27:0/3483251637 wait complete. 2024-04-25T00:15:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b87a6d640 1 Processor -- start 2024-04-25T00:15:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b87a6d640 1 -- start start 2024-04-25T00:15:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b87a6d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b801006a0 0x7f8b8006a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b87a6d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b80101890 0x7f8b8019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b87a6d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b80102250 0x7f8b8019eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b87a6d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8010c390 con 0x7f8b801006a0 2024-04-25T00:15:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b87a6d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8010c210 con 0x7f8b80102250 2024-04-25T00:15:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b87a6d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8010c510 con 0x7f8b80101890 2024-04-25T00:15:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b85809640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b801006a0 0x7f8b8006a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b85809640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b801006a0 0x7f8b8006a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48716/0 (socket says 172.21.15.27:48716) 2024-04-25T00:15:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b85809640 1 -- 172.21.15.27:0/943476827 learned_addr learned my addr 172.21.15.27:0/943476827 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b8600a640 1 --2- 172.21.15.27:0/943476827 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b80102250 0x7f8b8019eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b85008640 1 --2- 172.21.15.27:0/943476827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b80101890 0x7f8b8019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b85809640 1 -- 172.21.15.27:0/943476827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b80101890 msgr2=0x7f8b8019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b85809640 1 --2- 172.21.15.27:0/943476827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b80101890 0x7f8b8019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b85809640 1 -- 172.21.15.27:0/943476827 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b80102250 msgr2=0x7f8b8019eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b85809640 1 --2- 172.21.15.27:0/943476827 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b80102250 0x7f8b8019eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b85809640 1 -- 172.21.15.27:0/943476827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b8019f7b0 con 0x7f8b801006a0 2024-04-25T00:15:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.521+0000 7f8b8600a640 1 --2- 172.21.15.27:0/943476827 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b80102250 0x7f8b8019eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.522+0000 7f8b85809640 1 --2- 172.21.15.27:0/943476827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b801006a0 0x7f8b8006a910 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00b450 tx=0x7f8b7c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.522+0000 7f8b76ffd640 1 -- 172.21.15.27:0/943476827 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c015020 con 0x7f8b801006a0 2024-04-25T00:15:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.522+0000 7f8b87a6d640 1 -- 172.21.15.27:0/943476827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b8019fa40 con 0x7f8b801006a0 2024-04-25T00:15:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.522+0000 7f8b76ffd640 1 -- 172.21.15.27:0/943476827 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b7c011d70 con 0x7f8b801006a0 2024-04-25T00:15:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.522+0000 7f8b76ffd640 1 -- 172.21.15.27:0/943476827 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c00f340 con 0x7f8b801006a0 2024-04-25T00:15:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.522+0000 7f8b87a6d640 1 -- 172.21.15.27:0/943476827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b801a0030 con 0x7f8b801006a0 2024-04-25T00:15:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.523+0000 7f8b87a6d640 1 -- 172.21.15.27:0/943476827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b80107a40 con 0x7f8b801006a0 2024-04-25T00:15:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.524+0000 7f8b76ffd640 1 -- 172.21.15.27:0/943476827 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8b7c018020 con 0x7f8b801006a0 2024-04-25T00:15:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.525+0000 7f8b76ffd640 1 --2- 172.21.15.27:0/943476827 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8b6c0a2c20 0x7f8b6c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.525+0000 7f8b85008640 1 --2- 172.21.15.27:0/943476827 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8b6c0a2c20 0x7f8b6c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.525+0000 7f8b76ffd640 1 -- 172.21.15.27:0/943476827 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8b7c0cef30 con 0x7f8b801006a0 2024-04-25T00:15:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.525+0000 7f8b85008640 1 --2- 172.21.15.27:0/943476827 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8b6c0a2c20 0x7f8b6c0a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8b70000c00 tx=0x7f8b700315d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:05.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.529+0000 7f8b76ffd640 1 -- 172.21.15.27:0/943476827 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8b7c01f020 con 0x7f8b801006a0 2024-04-25T00:15:05.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:05 smithi027 ceph-mon[29208]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-25T00:15:05.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:05 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1950314007' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T00:15:05.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.715+0000 7f8b87a6d640 1 -- 172.21.15.27:0/943476827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b8002d320 con 0x7f8b801006a0 2024-04-25T00:15:05.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.717+0000 7f8b76ffd640 1 -- 172.21.15.27:0/943476827 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24643 (secure 0 0 0) 0x7f8b7c00d040 con 0x7f8b801006a0 2024-04-25T00:15:05.718 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:05.718 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":91,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:14:11.249732+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:15:05.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.719+0000 7f8b87a6d640 1 -- 172.21.15.27:0/943476827 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8b6c0a2c20 msgr2=0x7f8b6c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:05.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.719+0000 7f8b87a6d640 1 --2- 172.21.15.27:0/943476827 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8b6c0a2c20 0x7f8b6c0a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8b70000c00 tx=0x7f8b700315d0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.719+0000 7f8b87a6d640 1 -- 172.21.15.27:0/943476827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b801006a0 msgr2=0x7f8b8006a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:05.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.719+0000 7f8b87a6d640 1 --2- 172.21.15.27:0/943476827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b801006a0 0x7f8b8006a910 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00b450 tx=0x7f8b7c00b910 comp rx=0 tx=0).stop 2024-04-25T00:15:05.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.719+0000 7f8b87a6d640 1 -- 172.21.15.27:0/943476827 shutdown_connections 2024-04-25T00:15:05.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.719+0000 7f8b87a6d640 1 --2- 172.21.15.27:0/943476827 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f8b6c0a2c20 0x7f8b6c0a50e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.719+0000 7f8b87a6d640 1 --2- 172.21.15.27:0/943476827 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b80102250 0x7f8b8019eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.719+0000 7f8b87a6d640 1 --2- 172.21.15.27:0/943476827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b80101890 0x7f8b8019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.719+0000 7f8b87a6d640 1 --2- 172.21.15.27:0/943476827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b801006a0 0x7f8b8006a910 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.719+0000 7f8b87a6d640 1 -- 172.21.15.27:0/943476827 >> 172.21.15.27:0/943476827 conn(0x7f8b800fbc30 msgr2=0x7f8b80104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:05.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.719+0000 7f8b87a6d640 1 -- 172.21.15.27:0/943476827 shutdown_connections 2024-04-25T00:15:05.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.719+0000 7f8b87a6d640 1 -- 172.21.15.27:0/943476827 wait complete. 2024-04-25T00:15:05.732 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:08:34.006973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T00:13:17.337307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T00:13:18.253713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T00:13:21.995987+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T00:15:05.732 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:15:05.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:05 smithi139 ceph-mon[32833]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-25T00:15:05.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:05 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1950314007' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T00:15:05.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:05 smithi153 ceph-mon[32896]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-25T00:15:05.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:05 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1950314007' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T00:15:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.837+0000 7fb129dd5640 1 -- 172.21.15.27:0/3862095864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb124102980 msgr2=0x7fb124102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.837+0000 7fb129dd5640 1 --2- 172.21.15.27:0/3862095864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb124102980 0x7fb124102e00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb118009900 tx=0x7fb11802a410 comp rx=0 tx=0).stop 2024-04-25T00:15:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.837+0000 7fb129dd5640 1 -- 172.21.15.27:0/3862095864 shutdown_connections 2024-04-25T00:15:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.837+0000 7fb129dd5640 1 --2- 172.21.15.27:0/3862095864 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb124103340 0x7fb124109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.837+0000 7fb129dd5640 1 --2- 172.21.15.27:0/3862095864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb124102980 0x7fb124102e00 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.837+0000 7fb129dd5640 1 --2- 172.21.15.27:0/3862095864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb124101790 0x7fb124101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.837+0000 7fb129dd5640 1 -- 172.21.15.27:0/3862095864 >> 172.21.15.27:0/3862095864 conn(0x7fb1240fcd20 msgr2=0x7fb1240ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.837+0000 7fb129dd5640 1 -- 172.21.15.27:0/3862095864 shutdown_connections 2024-04-25T00:15:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.837+0000 7fb129dd5640 1 -- 172.21.15.27:0/3862095864 wait complete. 2024-04-25T00:15:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.838+0000 7fb129dd5640 1 Processor -- start 2024-04-25T00:15:05.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.838+0000 7fb129dd5640 1 -- start start 2024-04-25T00:15:05.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.838+0000 7fb129dd5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb124101790 0x7fb1240754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:05.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.838+0000 7fb129dd5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb124102980 0x7fb124073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:05.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.838+0000 7fb129dd5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb124103340 0x7fb124074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.838+0000 7fb129dd5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb12410d410 con 0x7fb124103340 2024-04-25T00:15:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.838+0000 7fb129dd5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb12410d290 con 0x7fb124101790 2024-04-25T00:15:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.838+0000 7fb129dd5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb12410d590 con 0x7fb124102980 2024-04-25T00:15:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.839+0000 7fb123fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb124103340 0x7fb124074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.839+0000 7fb123fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb124103340 0x7fb124074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48732/0 (socket says 172.21.15.27:48732) 2024-04-25T00:15:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.839+0000 7fb123fff640 1 -- 172.21.15.27:0/1624726945 learned_addr learned my addr 172.21.15.27:0/1624726945 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.839+0000 7fb122ffd640 1 --2- 172.21.15.27:0/1624726945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb124102980 0x7fb124073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.839+0000 7fb1237fe640 1 --2- 172.21.15.27:0/1624726945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb124101790 0x7fb1240754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.839+0000 7fb123fff640 1 -- 172.21.15.27:0/1624726945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb124102980 msgr2=0x7fb124073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.839+0000 7fb123fff640 1 --2- 172.21.15.27:0/1624726945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb124102980 0x7fb124073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.839+0000 7fb123fff640 1 -- 172.21.15.27:0/1624726945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb124101790 msgr2=0x7fb1240754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.839+0000 7fb123fff640 1 --2- 172.21.15.27:0/1624726945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb124101790 0x7fb1240754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.839+0000 7fb123fff640 1 -- 172.21.15.27:0/1624726945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb124074900 con 0x7fb124103340 2024-04-25T00:15:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.839+0000 7fb123fff640 1 --2- 172.21.15.27:0/1624726945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb124103340 0x7fb124074070 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb11c00eaa0 tx=0x7fb11c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:05.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.839+0000 7fb120ff9640 1 -- 172.21.15.27:0/1624726945 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c00cdb0 con 0x7fb124103340 2024-04-25T00:15:05.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.839+0000 7fb120ff9640 1 -- 172.21.15.27:0/1624726945 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb11c005020 con 0x7fb124103340 2024-04-25T00:15:05.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.839+0000 7fb120ff9640 1 -- 172.21.15.27:0/1624726945 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c0116b0 con 0x7fb124103340 2024-04-25T00:15:05.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.839+0000 7fb129dd5640 1 -- 172.21.15.27:0/1624726945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1241a1bc0 con 0x7fb124103340 2024-04-25T00:15:05.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.839+0000 7fb129dd5640 1 -- 172.21.15.27:0/1624726945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1241a20e0 con 0x7fb124103340 2024-04-25T00:15:05.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.840+0000 7fb129dd5640 1 -- 172.21.15.27:0/1624726945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1240689b0 con 0x7fb124103340 2024-04-25T00:15:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.841+0000 7fb120ff9640 1 -- 172.21.15.27:0/1624726945 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb11c005560 con 0x7fb124103340 2024-04-25T00:15:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.842+0000 7fb120ff9640 1 --2- 172.21.15.27:0/1624726945 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb10c0a2ca0 0x7fb10c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.842+0000 7fb120ff9640 1 -- 172.21.15.27:0/1624726945 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb11c0cfb20 con 0x7fb124103340 2024-04-25T00:15:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.842+0000 7fb1237fe640 1 --2- 172.21.15.27:0/1624726945 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb10c0a2ca0 0x7fb10c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.843+0000 7fb1237fe640 1 --2- 172.21.15.27:0/1624726945 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb10c0a2ca0 0x7fb10c0a5160 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb114005fd0 tx=0x7fb114005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:05.846+0000 7fb120ff9640 1 -- 172.21.15.27:0/1624726945 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb11c016030 con 0x7fb124103340 2024-04-25T00:15:06.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.032+0000 7fb129dd5640 1 -- 172.21.15.27:0/1624726945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb12402d1a0 con 0x7fb124103340 2024-04-25T00:15:06.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.032+0000 7fb120ff9640 1 -- 172.21.15.27:0/1624726945 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7fb11c09c640 con 0x7fb124103340 2024-04-25T00:15:06.032 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T00:15:06.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.034+0000 7fb129dd5640 1 -- 172.21.15.27:0/1624726945 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb10c0a2ca0 msgr2=0x7fb10c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.034+0000 7fb129dd5640 1 --2- 172.21.15.27:0/1624726945 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb10c0a2ca0 0x7fb10c0a5160 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb114005fd0 tx=0x7fb114005850 comp rx=0 tx=0).stop 2024-04-25T00:15:06.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.034+0000 7fb129dd5640 1 -- 172.21.15.27:0/1624726945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb124103340 msgr2=0x7fb124074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.034+0000 7fb129dd5640 1 --2- 172.21.15.27:0/1624726945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb124103340 0x7fb124074070 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb11c00eaa0 tx=0x7fb11c00ef60 comp rx=0 tx=0).stop 2024-04-25T00:15:06.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.034+0000 7fb129dd5640 1 -- 172.21.15.27:0/1624726945 shutdown_connections 2024-04-25T00:15:06.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.034+0000 7fb129dd5640 1 --2- 172.21.15.27:0/1624726945 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb10c0a2ca0 0x7fb10c0a5160 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.034+0000 7fb129dd5640 1 --2- 172.21.15.27:0/1624726945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb124103340 0x7fb124074070 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.034+0000 7fb129dd5640 1 --2- 172.21.15.27:0/1624726945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb124102980 0x7fb124073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.034+0000 7fb129dd5640 1 --2- 172.21.15.27:0/1624726945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb124101790 0x7fb1240754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.034+0000 7fb129dd5640 1 -- 172.21.15.27:0/1624726945 >> 172.21.15.27:0/1624726945 conn(0x7fb1240fcd20 msgr2=0x7fb124105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:06.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.034+0000 7fb129dd5640 1 -- 172.21.15.27:0/1624726945 shutdown_connections 2024-04-25T00:15:06.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.034+0000 7fb129dd5640 1 -- 172.21.15.27:0/1624726945 wait complete. 2024-04-25T00:15:06.046 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T00:15:06.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.152+0000 7f00d63a5640 1 -- 172.21.15.27:0/3345472592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00d0102230 msgr2=0x7f00d0108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.152+0000 7f00d63a5640 1 --2- 172.21.15.27:0/3345472592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00d0102230 0x7f00d0108ac0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f00c8009ca0 tx=0x7f00c802a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.152+0000 7f00d63a5640 1 -- 172.21.15.27:0/3345472592 shutdown_connections 2024-04-25T00:15:06.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.152+0000 7f00d63a5640 1 --2- 172.21.15.27:0/3345472592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00d0102230 0x7f00d0108ac0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.152+0000 7f00d63a5640 1 --2- 172.21.15.27:0/3345472592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00d0101870 0x7f00d0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.152+0000 7f00d63a5640 1 --2- 172.21.15.27:0/3345472592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00d0100680 0x7f00d0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.152+0000 7f00d63a5640 1 -- 172.21.15.27:0/3345472592 >> 172.21.15.27:0/3345472592 conn(0x7f00d00fbbf0 msgr2=0x7f00d00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:06.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.152+0000 7f00d63a5640 1 -- 172.21.15.27:0/3345472592 shutdown_connections 2024-04-25T00:15:06.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.152+0000 7f00d63a5640 1 -- 172.21.15.27:0/3345472592 wait complete. 2024-04-25T00:15:06.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.152+0000 7f00d63a5640 1 Processor -- start 2024-04-25T00:15:06.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.152+0000 7f00d63a5640 1 -- start start 2024-04-25T00:15:06.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.153+0000 7f00d63a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00d0100680 0x7f00d0199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:06.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.153+0000 7f00d63a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00d0101870 0x7f00d019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:06.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.153+0000 7f00d63a5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00d0102230 0x7f00d01a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:06.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.153+0000 7f00d63a5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f00d010c300 con 0x7f00d0101870 2024-04-25T00:15:06.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.153+0000 7f00d63a5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f00d010c180 con 0x7f00d0102230 2024-04-25T00:15:06.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.153+0000 7f00d63a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f00d010c480 con 0x7f00d0100680 2024-04-25T00:15:06.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.153+0000 7f00cf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00d0101870 0x7f00d019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:06.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.153+0000 7f00cf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00d0101870 0x7f00d019a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48758/0 (socket says 172.21.15.27:48758) 2024-04-25T00:15:06.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.153+0000 7f00cf7fe640 1 -- 172.21.15.27:0/3758937458 learned_addr learned my addr 172.21.15.27:0/3758937458 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:06.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.153+0000 7f00cffff640 1 --2- 172.21.15.27:0/3758937458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00d0100680 0x7f00d0199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:06.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.153+0000 7f00d4942640 1 --2- 172.21.15.27:0/3758937458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00d0102230 0x7f00d01a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:06.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.153+0000 7f00cf7fe640 1 -- 172.21.15.27:0/3758937458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00d0100680 msgr2=0x7f00d0199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.153+0000 7f00cf7fe640 1 --2- 172.21.15.27:0/3758937458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00d0100680 0x7f00d0199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.153+0000 7f00cf7fe640 1 -- 172.21.15.27:0/3758937458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00d0102230 msgr2=0x7f00d01a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.153+0000 7f00cf7fe640 1 --2- 172.21.15.27:0/3758937458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00d0102230 0x7f00d01a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.153+0000 7f00cf7fe640 1 -- 172.21.15.27:0/3758937458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d01a18b0 con 0x7f00d0101870 2024-04-25T00:15:06.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.154+0000 7f00cf7fe640 1 --2- 172.21.15.27:0/3758937458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00d0101870 0x7f00d019a1b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f00c400d950 tx=0x7f00c400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:06.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.154+0000 7f00cd7fa640 1 -- 172.21.15.27:0/3758937458 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c4017070 con 0x7f00d0101870 2024-04-25T00:15:06.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.154+0000 7f00cd7fa640 1 -- 172.21.15.27:0/3758937458 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00c4009c30 con 0x7f00d0101870 2024-04-25T00:15:06.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.154+0000 7f00d63a5640 1 -- 172.21.15.27:0/3758937458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00d01a1ba0 con 0x7f00d0101870 2024-04-25T00:15:06.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.154+0000 7f00cd7fa640 1 -- 172.21.15.27:0/3758937458 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c400f6b0 con 0x7f00d0101870 2024-04-25T00:15:06.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.154+0000 7f00d63a5640 1 -- 172.21.15.27:0/3758937458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00d01a20f0 con 0x7f00d0101870 2024-04-25T00:15:06.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.155+0000 7f00d63a5640 1 -- 172.21.15.27:0/3758937458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00d01a22c0 con 0x7f00d0101870 2024-04-25T00:15:06.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.156+0000 7f00cd7fa640 1 -- 172.21.15.27:0/3758937458 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f00c40109d0 con 0x7f00d0101870 2024-04-25T00:15:06.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.156+0000 7f00cd7fa640 1 --2- 172.21.15.27:0/3758937458 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f00b80a2c20 0x7f00b80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:06.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.157+0000 7f00cd7fa640 1 -- 172.21.15.27:0/3758937458 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f00c40cf0d0 con 0x7f00d0101870 2024-04-25T00:15:06.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.157+0000 7f00cffff640 1 --2- 172.21.15.27:0/3758937458 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f00b80a2c20 0x7f00b80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:06.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.157+0000 7f00cffff640 1 --2- 172.21.15.27:0/3758937458 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f00b80a2c20 0x7f00b80a50e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f00d019ab40 tx=0x7f00c0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:06.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.161+0000 7f00cd7fa640 1 -- 172.21.15.27:0/3758937458 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f00c4009de0 con 0x7f00d0101870 2024-04-25T00:15:06.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.346+0000 7f00d63a5640 1 -- 172.21.15.27:0/3758937458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f00d01a2490 con 0x7f00d0101870 2024-04-25T00:15:06.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.347+0000 7f00cd7fa640 1 -- 172.21.15.27:0/3758937458 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f00c409b520 con 0x7f00d0101870 2024-04-25T00:15:06.347 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:15:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.349+0000 7f00d63a5640 1 -- 172.21.15.27:0/3758937458 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f00b80a2c20 msgr2=0x7f00b80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.349+0000 7f00d63a5640 1 --2- 172.21.15.27:0/3758937458 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f00b80a2c20 0x7f00b80a50e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f00d019ab40 tx=0x7f00c0005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.349+0000 7f00d63a5640 1 -- 172.21.15.27:0/3758937458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00d0101870 msgr2=0x7f00d019a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.349+0000 7f00d63a5640 1 --2- 172.21.15.27:0/3758937458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00d0101870 0x7f00d019a1b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f00c400d950 tx=0x7f00c400de10 comp rx=0 tx=0).stop 2024-04-25T00:15:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.349+0000 7f00d63a5640 1 -- 172.21.15.27:0/3758937458 shutdown_connections 2024-04-25T00:15:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.349+0000 7f00d63a5640 1 --2- 172.21.15.27:0/3758937458 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f00b80a2c20 0x7f00b80a50e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.349+0000 7f00d63a5640 1 --2- 172.21.15.27:0/3758937458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00d0102230 0x7f00d01a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.349+0000 7f00d63a5640 1 --2- 172.21.15.27:0/3758937458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00d0101870 0x7f00d019a1b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.349+0000 7f00d63a5640 1 --2- 172.21.15.27:0/3758937458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00d0100680 0x7f00d0199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.349+0000 7f00d63a5640 1 -- 172.21.15.27:0/3758937458 >> 172.21.15.27:0/3758937458 conn(0x7f00d00fbbf0 msgr2=0x7f00d00fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:06.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.349+0000 7f00d63a5640 1 -- 172.21.15.27:0/3758937458 shutdown_connections 2024-04-25T00:15:06.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.349+0000 7f00d63a5640 1 -- 172.21.15.27:0/3758937458 wait complete. 2024-04-25T00:15:06.360 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T00:15:06.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.470+0000 7fe0c4b09640 1 -- 172.21.15.27:0/1522785162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0c01006a0 msgr2=0x7fe0c0100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.470+0000 7fe0c4b09640 1 --2- 172.21.15.27:0/1522785162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0c01006a0 0x7fe0c0100ac0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8009900 tx=0x7fe0b802a3f0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.470+0000 7fe0c4b09640 1 -- 172.21.15.27:0/1522785162 shutdown_connections 2024-04-25T00:15:06.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.470+0000 7fe0c4b09640 1 --2- 172.21.15.27:0/1522785162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0c0102250 0x7fe0c0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.470+0000 7fe0c4b09640 1 --2- 172.21.15.27:0/1522785162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0c0101890 0x7fe0c0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.470+0000 7fe0c4b09640 1 --2- 172.21.15.27:0/1522785162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0c01006a0 0x7fe0c0100ac0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.470+0000 7fe0c4b09640 1 -- 172.21.15.27:0/1522785162 >> 172.21.15.27:0/1522785162 conn(0x7fe0c00fbc30 msgr2=0x7fe0c00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:06.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.470+0000 7fe0c4b09640 1 -- 172.21.15.27:0/1522785162 shutdown_connections 2024-04-25T00:15:06.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.470+0000 7fe0c4b09640 1 -- 172.21.15.27:0/1522785162 wait complete. 2024-04-25T00:15:06.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0c4b09640 1 Processor -- start 2024-04-25T00:15:06.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0c4b09640 1 -- start start 2024-04-25T00:15:06.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0c4b09640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0c01006a0 0x7fe0c0197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:06.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0c4b09640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0c0101890 0x7fe0c01980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:06.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0c4b09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0c0102250 0x7fe0c019f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:06.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0c4b09640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c010d460 con 0x7fe0c01006a0 2024-04-25T00:15:06.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0c4b09640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c010d2e0 con 0x7fe0c0101890 2024-04-25T00:15:06.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0c4b09640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c010d5e0 con 0x7fe0c0102250 2024-04-25T00:15:06.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0be59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0c01006a0 0x7fe0c0197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:06.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0be59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0c01006a0 0x7fe0c0197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48770/0 (socket says 172.21.15.27:48770) 2024-04-25T00:15:06.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0be59c640 1 -- 172.21.15.27:0/3077760542 learned_addr learned my addr 172.21.15.27:0/3077760542 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:06.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0bdd9b640 1 --2- 172.21.15.27:0/3077760542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0c0101890 0x7fe0c01980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:06.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0be59c640 1 -- 172.21.15.27:0/3077760542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0c0102250 msgr2=0x7fe0c019f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0bed9d640 1 --2- 172.21.15.27:0/3077760542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0c0102250 0x7fe0c019f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:06.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0be59c640 1 --2- 172.21.15.27:0/3077760542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0c0102250 0x7fe0c019f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0be59c640 1 -- 172.21.15.27:0/3077760542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0c0101890 msgr2=0x7fe0c01980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0be59c640 1 --2- 172.21.15.27:0/3077760542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0c0101890 0x7fe0c01980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0be59c640 1 -- 172.21.15.27:0/3077760542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c019f780 con 0x7fe0c01006a0 2024-04-25T00:15:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.471+0000 7fe0bdd9b640 1 --2- 172.21.15.27:0/3077760542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0c0101890 0x7fe0c01980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.472+0000 7fe0be59c640 1 --2- 172.21.15.27:0/3077760542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0c01006a0 0x7fe0c0197b60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8031610 tx=0x7fe0b8032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.472+0000 7fe0b77fe640 1 -- 172.21.15.27:0/3077760542 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b803f030 con 0x7fe0c01006a0 2024-04-25T00:15:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.472+0000 7fe0b77fe640 1 -- 172.21.15.27:0/3077760542 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0b8003ce0 con 0x7fe0c01006a0 2024-04-25T00:15:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.472+0000 7fe0b77fe640 1 -- 172.21.15.27:0/3077760542 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8035070 con 0x7fe0c01006a0 2024-04-25T00:15:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.472+0000 7fe0c4b09640 1 -- 172.21.15.27:0/3077760542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0c019fa10 con 0x7fe0c01006a0 2024-04-25T00:15:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.472+0000 7fe0c4b09640 1 -- 172.21.15.27:0/3077760542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0c019feb0 con 0x7fe0c01006a0 2024-04-25T00:15:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.473+0000 7fe0c4b09640 1 -- 172.21.15.27:0/3077760542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0c0107a90 con 0x7fe0c01006a0 2024-04-25T00:15:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.474+0000 7fe0b77fe640 1 -- 172.21.15.27:0/3077760542 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe0b8042030 con 0x7fe0c01006a0 2024-04-25T00:15:06.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.475+0000 7fe0b77fe640 1 --2- 172.21.15.27:0/3077760542 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe0ac0a2c20 0x7fe0ac0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:06.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.475+0000 7fe0b77fe640 1 -- 172.21.15.27:0/3077760542 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe0b80efa50 con 0x7fe0c01006a0 2024-04-25T00:15:06.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.475+0000 7fe0bdd9b640 1 --2- 172.21.15.27:0/3077760542 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe0ac0a2c20 0x7fe0ac0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:06.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.475+0000 7fe0bdd9b640 1 --2- 172.21.15.27:0/3077760542 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe0ac0a2c20 0x7fe0ac0a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0199070 tx=0x7fe0a8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:06.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.479+0000 7fe0b77fe640 1 -- 172.21.15.27:0/3077760542 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe0b80bc4f0 con 0x7fe0c01006a0 2024-04-25T00:15:06.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.667+0000 7fe0c4b09640 1 -- 172.21.15.27:0/3077760542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe0c00623e0 con 0x7fe0c01006a0 2024-04-25T00:15:06.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.667+0000 7fe0b77fe640 1 -- 172.21.15.27:0/3077760542 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fe0b80bbdf0 con 0x7fe0c01006a0 2024-04-25T00:15:06.667 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T00:15:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.670+0000 7fe0c4b09640 1 -- 172.21.15.27:0/3077760542 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe0ac0a2c20 msgr2=0x7fe0ac0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.670+0000 7fe0c4b09640 1 --2- 172.21.15.27:0/3077760542 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe0ac0a2c20 0x7fe0ac0a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0199070 tx=0x7fe0a8005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.670+0000 7fe0c4b09640 1 -- 172.21.15.27:0/3077760542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0c01006a0 msgr2=0x7fe0c0197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.670+0000 7fe0c4b09640 1 --2- 172.21.15.27:0/3077760542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0c01006a0 0x7fe0c0197b60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8031610 tx=0x7fe0b8032ce0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.670+0000 7fe0c4b09640 1 -- 172.21.15.27:0/3077760542 shutdown_connections 2024-04-25T00:15:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.670+0000 7fe0c4b09640 1 --2- 172.21.15.27:0/3077760542 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fe0ac0a2c20 0x7fe0ac0a50e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.670+0000 7fe0c4b09640 1 --2- 172.21.15.27:0/3077760542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0c0102250 0x7fe0c019f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.670+0000 7fe0c4b09640 1 --2- 172.21.15.27:0/3077760542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0c0101890 0x7fe0c01980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.670+0000 7fe0c4b09640 1 --2- 172.21.15.27:0/3077760542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0c01006a0 0x7fe0c0197b60 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.670+0000 7fe0c4b09640 1 -- 172.21.15.27:0/3077760542 >> 172.21.15.27:0/3077760542 conn(0x7fe0c00fbc30 msgr2=0x7fe0c0104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.670+0000 7fe0c4b09640 1 -- 172.21.15.27:0/3077760542 shutdown_connections 2024-04-25T00:15:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.670+0000 7fe0c4b09640 1 -- 172.21.15.27:0/3077760542 wait complete. 2024-04-25T00:15:06.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:06 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/943476827' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:06.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:06 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1624726945' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:06.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:06 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3758937458' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:06.683 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T00:15:06.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:06 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/943476827' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:06.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:06 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1624726945' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:06.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:06 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3758937458' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:06.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:06 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/943476827' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:06.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:06 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1624726945' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:06.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:06 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3758937458' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.790+0000 7f154a2ae640 1 -- 172.21.15.27:0/1700223265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1544104b90 msgr2=0x7f1544106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.790+0000 7f154a2ae640 1 --2- 172.21.15.27:0/1700223265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1544104b90 0x7f1544106f80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1538009900 tx=0x7f153802a410 comp rx=0 tx=0).stop 2024-04-25T00:15:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.790+0000 7f154a2ae640 1 -- 172.21.15.27:0/1700223265 shutdown_connections 2024-04-25T00:15:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.790+0000 7f154a2ae640 1 --2- 172.21.15.27:0/1700223265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15441074c0 0x7f15441098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.790+0000 7f154a2ae640 1 --2- 172.21.15.27:0/1700223265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1544104b90 0x7f1544106f80 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.790+0000 7f154a2ae640 1 --2- 172.21.15.27:0/1700223265 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1544102260 0x7f1544104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.790+0000 7f154a2ae640 1 -- 172.21.15.27:0/1700223265 >> 172.21.15.27:0/1700223265 conn(0x7f15440fbbf0 msgr2=0x7f15440fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.790+0000 7f154a2ae640 1 -- 172.21.15.27:0/1700223265 shutdown_connections 2024-04-25T00:15:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.790+0000 7f154a2ae640 1 -- 172.21.15.27:0/1700223265 wait complete. 2024-04-25T00:15:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.791+0000 7f154a2ae640 1 Processor -- start 2024-04-25T00:15:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.791+0000 7f154a2ae640 1 -- start start 2024-04-25T00:15:06.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.791+0000 7f154a2ae640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1544102260 0x7f1544101bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:06.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.791+0000 7f154a2ae640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1544104b90 0x7f1544100210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.791+0000 7f154a2ae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15441074c0 0x7f1544100750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.791+0000 7f154a2ae640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f154410d420 con 0x7f15441074c0 2024-04-25T00:15:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.791+0000 7f154a2ae640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f154410d2a0 con 0x7f1544102260 2024-04-25T00:15:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.791+0000 7f154a2ae640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f154410d5a0 con 0x7f1544104b90 2024-04-25T00:15:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.792+0000 7f154884b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15441074c0 0x7f1544100750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.792+0000 7f154884b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15441074c0 0x7f1544100750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47412/0 (socket says 172.21.15.27:47412) 2024-04-25T00:15:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.792+0000 7f154884b640 1 -- 172.21.15.27:0/3049822835 learned_addr learned my addr 172.21.15.27:0/3049822835 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.792+0000 7f1543fff640 1 --2- 172.21.15.27:0/3049822835 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1544102260 0x7f1544101bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.792+0000 7f15437fe640 1 --2- 172.21.15.27:0/3049822835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1544104b90 0x7f1544100210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.792+0000 7f154884b640 1 -- 172.21.15.27:0/3049822835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1544104b90 msgr2=0x7f1544100210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.792+0000 7f154884b640 1 --2- 172.21.15.27:0/3049822835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1544104b90 0x7f1544100210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.792+0000 7f154884b640 1 -- 172.21.15.27:0/3049822835 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1544102260 msgr2=0x7f1544101bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.792+0000 7f154884b640 1 --2- 172.21.15.27:0/3049822835 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1544102260 0x7f1544101bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.792+0000 7f154884b640 1 -- 172.21.15.27:0/3049822835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1544100fe0 con 0x7f15441074c0 2024-04-25T00:15:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.792+0000 7f15437fe640 1 --2- 172.21.15.27:0/3049822835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1544104b90 0x7f1544100210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.792+0000 7f154884b640 1 --2- 172.21.15.27:0/3049822835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15441074c0 0x7f1544100750 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f153c00cce0 tx=0x7f153c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.792+0000 7f15417fa640 1 -- 172.21.15.27:0/3049822835 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153c00f450 con 0x7f15441074c0 2024-04-25T00:15:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.792+0000 7f15417fa640 1 -- 172.21.15.27:0/3049822835 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f153c00fad0 con 0x7f15441074c0 2024-04-25T00:15:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.792+0000 7f154a2ae640 1 -- 172.21.15.27:0/3049822835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15441a1b80 con 0x7f15441074c0 2024-04-25T00:15:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.792+0000 7f15417fa640 1 -- 172.21.15.27:0/3049822835 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153c018c10 con 0x7f15441074c0 2024-04-25T00:15:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.792+0000 7f154a2ae640 1 -- 172.21.15.27:0/3049822835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15441a2050 con 0x7f15441074c0 2024-04-25T00:15:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.793+0000 7f154a2ae640 1 -- 172.21.15.27:0/3049822835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1544068560 con 0x7f15441074c0 2024-04-25T00:15:06.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.795+0000 7f15417fa640 1 -- 172.21.15.27:0/3049822835 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f153c00f5f0 con 0x7f15441074c0 2024-04-25T00:15:06.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.796+0000 7f15417fa640 1 --2- 172.21.15.27:0/3049822835 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f152c0a2c20 0x7f152c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:06.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.796+0000 7f1543fff640 1 --2- 172.21.15.27:0/3049822835 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f152c0a2c20 0x7f152c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:06.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.796+0000 7f15417fa640 1 -- 172.21.15.27:0/3049822835 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f153c0cfc70 con 0x7f15441074c0 2024-04-25T00:15:06.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.796+0000 7f1543fff640 1 --2- 172.21.15.27:0/3049822835 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f152c0a2c20 0x7f152c0a50e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f153400bbe0 tx=0x7f153400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.800+0000 7f15417fa640 1 -- 172.21.15.27:0/3049822835 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f153c014030 con 0x7f15441074c0 2024-04-25T00:15:06.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.985+0000 7f154a2ae640 1 -- 172.21.15.27:0/3049822835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f154402d1a0 con 0x7f15441074c0 2024-04-25T00:15:06.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.986+0000 7f15417fa640 1 -- 172.21.15.27:0/3049822835 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f153c09c710 con 0x7f15441074c0 2024-04-25T00:15:06.986 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:15:06.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.987+0000 7f154a2ae640 1 -- 172.21.15.27:0/3049822835 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f152c0a2c20 msgr2=0x7f152c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.987+0000 7f154a2ae640 1 --2- 172.21.15.27:0/3049822835 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f152c0a2c20 0x7f152c0a50e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f153400bbe0 tx=0x7f153400b390 comp rx=0 tx=0).stop 2024-04-25T00:15:06.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.987+0000 7f154a2ae640 1 -- 172.21.15.27:0/3049822835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15441074c0 msgr2=0x7f1544100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.987+0000 7f154a2ae640 1 --2- 172.21.15.27:0/3049822835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15441074c0 0x7f1544100750 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f153c00cce0 tx=0x7f153c00a620 comp rx=0 tx=0).stop 2024-04-25T00:15:06.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.988+0000 7f154a2ae640 1 -- 172.21.15.27:0/3049822835 shutdown_connections 2024-04-25T00:15:06.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.988+0000 7f154a2ae640 1 --2- 172.21.15.27:0/3049822835 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f152c0a2c20 0x7f152c0a50e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.988+0000 7f154a2ae640 1 --2- 172.21.15.27:0/3049822835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15441074c0 0x7f1544100750 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.988+0000 7f154a2ae640 1 --2- 172.21.15.27:0/3049822835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1544104b90 0x7f1544100210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.988+0000 7f154a2ae640 1 --2- 172.21.15.27:0/3049822835 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1544102260 0x7f1544101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.988+0000 7f154a2ae640 1 -- 172.21.15.27:0/3049822835 >> 172.21.15.27:0/3049822835 conn(0x7f15440fbbf0 msgr2=0x7f1544102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:06.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.988+0000 7f154a2ae640 1 -- 172.21.15.27:0/3049822835 shutdown_connections 2024-04-25T00:15:06.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:06.988+0000 7f154a2ae640 1 -- 172.21.15.27:0/3049822835 wait complete. 2024-04-25T00:15:06.999 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:07.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.101+0000 7f167b62a640 1 -- 172.21.15.27:0/1456042968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16741029a0 msgr2=0x7f1674102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.101+0000 7f167b62a640 1 --2- 172.21.15.27:0/1456042968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16741029a0 0x7f1674102e20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1670009900 tx=0x7f167002a410 comp rx=0 tx=0).stop 2024-04-25T00:15:07.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.101+0000 7f167b62a640 1 -- 172.21.15.27:0/1456042968 shutdown_connections 2024-04-25T00:15:07.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.101+0000 7f167b62a640 1 --2- 172.21.15.27:0/1456042968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1674103360 0x7f1674109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.101+0000 7f167b62a640 1 --2- 172.21.15.27:0/1456042968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16741029a0 0x7f1674102e20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.101+0000 7f167b62a640 1 --2- 172.21.15.27:0/1456042968 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16741017b0 0x7f1674101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.101+0000 7f167b62a640 1 -- 172.21.15.27:0/1456042968 >> 172.21.15.27:0/1456042968 conn(0x7f16740fcd20 msgr2=0x7f16740ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:07.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.102+0000 7f167b62a640 1 -- 172.21.15.27:0/1456042968 shutdown_connections 2024-04-25T00:15:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.102+0000 7f167b62a640 1 -- 172.21.15.27:0/1456042968 wait complete. 2024-04-25T00:15:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.102+0000 7f167b62a640 1 Processor -- start 2024-04-25T00:15:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.102+0000 7f167b62a640 1 -- start start 2024-04-25T00:15:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.102+0000 7f167b62a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16741017b0 0x7f1674199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.102+0000 7f167b62a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16741029a0 0x7f167419a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.102+0000 7f167b62a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1674103360 0x7f16741a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.102+0000 7f167b62a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f167410d430 con 0x7f1674103360 2024-04-25T00:15:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.102+0000 7f167b62a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f167410d2b0 con 0x7f16741029a0 2024-04-25T00:15:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.102+0000 7f167b62a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f167410d5b0 con 0x7f16741017b0 2024-04-25T00:15:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.102+0000 7f1679bc7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1674103360 0x7f16741a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.102+0000 7f1679bc7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1674103360 0x7f16741a12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47426/0 (socket says 172.21.15.27:47426) 2024-04-25T00:15:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.102+0000 7f1679bc7640 1 -- 172.21.15.27:0/3074811167 learned_addr learned my addr 172.21.15.27:0/3074811167 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.103+0000 7f16793c6640 1 --2- 172.21.15.27:0/3074811167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16741017b0 0x7f1674199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.103+0000 7f1678bc5640 1 --2- 172.21.15.27:0/3074811167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16741029a0 0x7f167419a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:07.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.103+0000 7f1679bc7640 1 -- 172.21.15.27:0/3074811167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16741017b0 msgr2=0x7f1674199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.103+0000 7f1679bc7640 1 --2- 172.21.15.27:0/3074811167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16741017b0 0x7f1674199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.103+0000 7f1679bc7640 1 -- 172.21.15.27:0/3074811167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16741029a0 msgr2=0x7f167419a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.103+0000 7f1679bc7640 1 --2- 172.21.15.27:0/3074811167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16741029a0 0x7f167419a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.103+0000 7f1679bc7640 1 -- 172.21.15.27:0/3074811167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16741a1940 con 0x7f1674103360 2024-04-25T00:15:07.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.103+0000 7f1678bc5640 1 --2- 172.21.15.27:0/3074811167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16741029a0 0x7f167419a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:07.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.103+0000 7f1679bc7640 1 --2- 172.21.15.27:0/3074811167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1674103360 0x7f16741a12a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f166c00eaa0 tx=0x7f166c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:07.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.103+0000 7f166a7fc640 1 -- 172.21.15.27:0/3074811167 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c00cdb0 con 0x7f1674103360 2024-04-25T00:15:07.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.103+0000 7f166a7fc640 1 -- 172.21.15.27:0/3074811167 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f166c005020 con 0x7f1674103360 2024-04-25T00:15:07.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.103+0000 7f167b62a640 1 -- 172.21.15.27:0/3074811167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16741a1c30 con 0x7f1674103360 2024-04-25T00:15:07.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.103+0000 7f166a7fc640 1 -- 172.21.15.27:0/3074811167 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c0116b0 con 0x7f1674103360 2024-04-25T00:15:07.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.104+0000 7f167b62a640 1 -- 172.21.15.27:0/3074811167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16741a2180 con 0x7f1674103360 2024-04-25T00:15:07.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.105+0000 7f167b62a640 1 -- 172.21.15.27:0/3074811167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16741a2350 con 0x7f1674103360 2024-04-25T00:15:07.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.106+0000 7f166a7fc640 1 -- 172.21.15.27:0/3074811167 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f166c005560 con 0x7f1674103360 2024-04-25T00:15:07.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.106+0000 7f166a7fc640 1 --2- 172.21.15.27:0/3074811167 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16600a2c20 0x7f16600a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.106+0000 7f16793c6640 1 --2- 172.21.15.27:0/3074811167 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16600a2c20 0x7f16600a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:07.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.106+0000 7f166a7fc640 1 -- 172.21.15.27:0/3074811167 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f166c0cf480 con 0x7f1674103360 2024-04-25T00:15:07.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.107+0000 7f16793c6640 1 --2- 172.21.15.27:0/3074811167 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16600a2c20 0x7f16600a50e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f167419b200 tx=0x7f1664005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:07.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.115+0000 7f166a7fc640 1 -- 172.21.15.27:0/3074811167 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f166c016030 con 0x7f1674103360 2024-04-25T00:15:07.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.340+0000 7f167b62a640 1 -- 172.21.15.27:0/3074811167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16741a2520 con 0x7f1674103360 2024-04-25T00:15:07.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.341+0000 7f166a7fc640 1 -- 172.21.15.27:0/3074811167 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11944 (secure 0 0 0) 0x7f166c09bf20 con 0x7f1674103360 2024-04-25T00:15:07.341 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:07.341 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:13:47.575422+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24428,24496,24521,24430,24514]},"id":1}]} 2024-04-25T00:15:07.341 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 28 2024-04-25T00:15:07.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.343+0000 7f167b62a640 1 -- 172.21.15.27:0/3074811167 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16600a2c20 msgr2=0x7f16600a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.343+0000 7f167b62a640 1 --2- 172.21.15.27:0/3074811167 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16600a2c20 0x7f16600a50e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f167419b200 tx=0x7f1664005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:07.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.343+0000 7f167b62a640 1 -- 172.21.15.27:0/3074811167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1674103360 msgr2=0x7f16741a12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.343+0000 7f167b62a640 1 --2- 172.21.15.27:0/3074811167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1674103360 0x7f16741a12a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f166c00eaa0 tx=0x7f166c00ef60 comp rx=0 tx=0).stop 2024-04-25T00:15:07.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.343+0000 7f167b62a640 1 -- 172.21.15.27:0/3074811167 shutdown_connections 2024-04-25T00:15:07.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.343+0000 7f167b62a640 1 --2- 172.21.15.27:0/3074811167 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16600a2c20 0x7f16600a50e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.343+0000 7f167b62a640 1 --2- 172.21.15.27:0/3074811167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1674103360 0x7f16741a12a0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.343+0000 7f167b62a640 1 --2- 172.21.15.27:0/3074811167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16741029a0 0x7f167419a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.343+0000 7f167b62a640 1 --2- 172.21.15.27:0/3074811167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16741017b0 0x7f1674199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.343+0000 7f167b62a640 1 -- 172.21.15.27:0/3074811167 >> 172.21.15.27:0/3074811167 conn(0x7f16740fcd20 msgr2=0x7f1674073970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:07.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.343+0000 7f167b62a640 1 -- 172.21.15.27:0/3074811167 shutdown_connections 2024-04-25T00:15:07.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.343+0000 7f167b62a640 1 -- 172.21.15.27:0/3074811167 wait complete. 2024-04-25T00:15:07.354 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:15:07.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.457+0000 7f4315210640 1 -- 172.21.15.27:0/780246965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43101006a0 msgr2=0x7f4310100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.457+0000 7f4315210640 1 --2- 172.21.15.27:0/780246965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43101006a0 0x7f4310100ac0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4300009ca0 tx=0x7f430002a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.457+0000 7f4315210640 1 -- 172.21.15.27:0/780246965 shutdown_connections 2024-04-25T00:15:07.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.457+0000 7f4315210640 1 --2- 172.21.15.27:0/780246965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4310102250 0x7f4310108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.457+0000 7f4315210640 1 --2- 172.21.15.27:0/780246965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4310101890 0x7f4310101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.457+0000 7f4315210640 1 --2- 172.21.15.27:0/780246965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43101006a0 0x7f4310100ac0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.457+0000 7f4315210640 1 -- 172.21.15.27:0/780246965 >> 172.21.15.27:0/780246965 conn(0x7f43100fbc30 msgr2=0x7f43100fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:07.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.457+0000 7f4315210640 1 -- 172.21.15.27:0/780246965 shutdown_connections 2024-04-25T00:15:07.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.457+0000 7f4315210640 1 -- 172.21.15.27:0/780246965 wait complete. 2024-04-25T00:15:07.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.457+0000 7f4315210640 1 Processor -- start 2024-04-25T00:15:07.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f4315210640 1 -- start start 2024-04-25T00:15:07.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f4315210640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43101006a0 0x7f431006a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f4315210640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4310101890 0x7f431019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f4315210640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4310102250 0x7f431019edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f4315210640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f431010c320 con 0x7f4310101890 2024-04-25T00:15:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f4315210640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f431010c1a0 con 0x7f43101006a0 2024-04-25T00:15:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f4315210640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f431010c4a0 con 0x7f4310102250 2024-04-25T00:15:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f430e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4310101890 0x7f431019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f430e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4310101890 0x7f431019c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47452/0 (socket says 172.21.15.27:47452) 2024-04-25T00:15:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f430e59c640 1 -- 172.21.15.27:0/1354701673 learned_addr learned my addr 172.21.15.27:0/1354701673 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f430f59e640 1 --2- 172.21.15.27:0/1354701673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4310102250 0x7f431019edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f430ed9d640 1 --2- 172.21.15.27:0/1354701673 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43101006a0 0x7f431006a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f430e59c640 1 -- 172.21.15.27:0/1354701673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4310102250 msgr2=0x7f431019edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f430e59c640 1 --2- 172.21.15.27:0/1354701673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4310102250 0x7f431019edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f430e59c640 1 -- 172.21.15.27:0/1354701673 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43101006a0 msgr2=0x7f431006a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f430e59c640 1 --2- 172.21.15.27:0/1354701673 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43101006a0 0x7f431006a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f430e59c640 1 -- 172.21.15.27:0/1354701673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f431019f6b0 con 0x7f4310101890 2024-04-25T00:15:07.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.458+0000 7f430e59c640 1 --2- 172.21.15.27:0/1354701673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4310101890 0x7f431019c7b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f430400c920 tx=0x7f430400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:07.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.459+0000 7f42f7fff640 1 -- 172.21.15.27:0/1354701673 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4304007c70 con 0x7f4310101890 2024-04-25T00:15:07.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.459+0000 7f42f7fff640 1 -- 172.21.15.27:0/1354701673 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4304007e10 con 0x7f4310101890 2024-04-25T00:15:07.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.459+0000 7f4315210640 1 -- 172.21.15.27:0/1354701673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f431019f9a0 con 0x7f4310101890 2024-04-25T00:15:07.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.459+0000 7f42f7fff640 1 -- 172.21.15.27:0/1354701673 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4304018890 con 0x7f4310101890 2024-04-25T00:15:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.459+0000 7f4315210640 1 -- 172.21.15.27:0/1354701673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f431019fef0 con 0x7f4310101890 2024-04-25T00:15:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.460+0000 7f4315210640 1 -- 172.21.15.27:0/1354701673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4310107960 con 0x7f4310101890 2024-04-25T00:15:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.461+0000 7f42f7fff640 1 -- 172.21.15.27:0/1354701673 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4304018a70 con 0x7f4310101890 2024-04-25T00:15:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.461+0000 7f42f7fff640 1 --2- 172.21.15.27:0/1354701673 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f42f80a2c20 0x7f42f80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.462+0000 7f430ed9d640 1 --2- 172.21.15.27:0/1354701673 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f42f80a2c20 0x7f42f80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.462+0000 7f42f7fff640 1 -- 172.21.15.27:0/1354701673 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f43040cf690 con 0x7f4310101890 2024-04-25T00:15:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.462+0000 7f430ed9d640 1 --2- 172.21.15.27:0/1354701673 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f42f80a2c20 0x7f42f80a50e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4300000f80 tx=0x7f430002f470 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.466+0000 7f42f7fff640 1 -- 172.21.15.27:0/1354701673 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4304014030 con 0x7f4310101890 2024-04-25T00:15:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.650+0000 7f4315210640 1 -- 172.21.15.27:0/1354701673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f431002d2f0 con 0x7f4310101890 2024-04-25T00:15:07.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.652+0000 7f42f7fff640 1 -- 172.21.15.27:0/1354701673 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24643 (secure 0 0 0) 0x7f430409c130 con 0x7f4310101890 2024-04-25T00:15:07.652 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:07.652 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":91,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:14:11.249732+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:15:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.654+0000 7f4315210640 1 -- 172.21.15.27:0/1354701673 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f42f80a2c20 msgr2=0x7f42f80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.654+0000 7f4315210640 1 --2- 172.21.15.27:0/1354701673 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f42f80a2c20 0x7f42f80a50e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4300000f80 tx=0x7f430002f470 comp rx=0 tx=0).stop 2024-04-25T00:15:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.654+0000 7f4315210640 1 -- 172.21.15.27:0/1354701673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4310101890 msgr2=0x7f431019c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.654+0000 7f4315210640 1 --2- 172.21.15.27:0/1354701673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4310101890 0x7f431019c7b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f430400c920 tx=0x7f430400cde0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.655+0000 7f4315210640 1 -- 172.21.15.27:0/1354701673 shutdown_connections 2024-04-25T00:15:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.655+0000 7f4315210640 1 --2- 172.21.15.27:0/1354701673 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f42f80a2c20 0x7f42f80a50e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.655+0000 7f4315210640 1 --2- 172.21.15.27:0/1354701673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4310102250 0x7f431019edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.655+0000 7f4315210640 1 --2- 172.21.15.27:0/1354701673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4310101890 0x7f431019c7b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.655+0000 7f4315210640 1 --2- 172.21.15.27:0/1354701673 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43101006a0 0x7f431006a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.655+0000 7f4315210640 1 -- 172.21.15.27:0/1354701673 >> 172.21.15.27:0/1354701673 conn(0x7f43100fbc30 msgr2=0x7f4310104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.655+0000 7f4315210640 1 -- 172.21.15.27:0/1354701673 shutdown_connections 2024-04-25T00:15:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.655+0000 7f4315210640 1 -- 172.21.15.27:0/1354701673 wait complete. 2024-04-25T00:15:07.665 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:07 smithi027 ceph-mon[29208]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:15:07.665 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:07 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3077760542' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:07.665 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:07 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3049822835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:07.665 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:07 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3074811167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:07.666 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:08:34.006973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T00:13:17.337307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T00:13:18.253713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T00:13:21.995987+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T00:15:07.667 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:15:07.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:07 smithi139 ceph-mon[32833]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:15:07.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:07 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3077760542' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:07.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:07 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3049822835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:07.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:07 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3074811167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.772+0000 7fb9e3658640 1 -- 172.21.15.27:0/1696513890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9dc1012a0 msgr2=0x7fb9dc108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.772+0000 7fb9e3658640 1 --2- 172.21.15.27:0/1696513890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9dc1012a0 0x7fb9dc108b40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4009ca0 tx=0x7fb9d402a8b0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.772+0000 7fb9e3658640 1 -- 172.21.15.27:0/1696513890 shutdown_connections 2024-04-25T00:15:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.772+0000 7fb9e3658640 1 --2- 172.21.15.27:0/1696513890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9dc1012a0 0x7fb9dc108b40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.772+0000 7fb9e3658640 1 --2- 172.21.15.27:0/1696513890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9dc1008e0 0x7fb9dc100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.772+0000 7fb9e3658640 1 --2- 172.21.15.27:0/1696513890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9dc101d20 0x7fb9dc102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.772+0000 7fb9e3658640 1 -- 172.21.15.27:0/1696513890 >> 172.21.15.27:0/1696513890 conn(0x7fb9dc0fb370 msgr2=0x7fb9dc0fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.772+0000 7fb9e3658640 1 -- 172.21.15.27:0/1696513890 shutdown_connections 2024-04-25T00:15:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.772+0000 7fb9e3658640 1 -- 172.21.15.27:0/1696513890 wait complete. 2024-04-25T00:15:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.772+0000 7fb9e3658640 1 Processor -- start 2024-04-25T00:15:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.772+0000 7fb9e3658640 1 -- start start 2024-04-25T00:15:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e3658640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9dc1008e0 0x7fb9dc197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e3658640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9dc1012a0 0x7fb9dc197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e3658640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9dc101d20 0x7fb9dc19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e3658640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb9dc10c330 con 0x7fb9dc101d20 2024-04-25T00:15:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e3658640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb9dc10c1b0 con 0x7fb9dc1008e0 2024-04-25T00:15:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e3658640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb9dc10c4b0 con 0x7fb9dc1012a0 2024-04-25T00:15:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e1bf5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9dc101d20 0x7fb9dc19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e1bf5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9dc101d20 0x7fb9dc19f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47466/0 (socket says 172.21.15.27:47466) 2024-04-25T00:15:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e1bf5640 1 -- 172.21.15.27:0/4140707601 learned_addr learned my addr 172.21.15.27:0/4140707601 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e13f4640 1 --2- 172.21.15.27:0/4140707601 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9dc1008e0 0x7fb9dc197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e1bf5640 1 -- 172.21.15.27:0/4140707601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9dc1012a0 msgr2=0x7fb9dc197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e1bf5640 1 --2- 172.21.15.27:0/4140707601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9dc1012a0 0x7fb9dc197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e1bf5640 1 -- 172.21.15.27:0/4140707601 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9dc1008e0 msgr2=0x7fb9dc197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e1bf5640 1 --2- 172.21.15.27:0/4140707601 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9dc1008e0 0x7fb9dc197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e1bf5640 1 -- 172.21.15.27:0/4140707601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9dc19f690 con 0x7fb9dc101d20 2024-04-25T00:15:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e1bf5640 1 --2- 172.21.15.27:0/4140707601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9dc101d20 0x7fb9dc19f020 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4009dc0 tx=0x7fb9d4003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9d27fc640 1 -- 172.21.15.27:0/4140707601 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d4004450 con 0x7fb9dc101d20 2024-04-25T00:15:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9d27fc640 1 -- 172.21.15.27:0/4140707601 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9d4040070 con 0x7fb9dc101d20 2024-04-25T00:15:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e3658640 1 -- 172.21.15.27:0/4140707601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9dc19f920 con 0x7fb9dc101d20 2024-04-25T00:15:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9d27fc640 1 -- 172.21.15.27:0/4140707601 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d4045e20 con 0x7fb9dc101d20 2024-04-25T00:15:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.773+0000 7fb9e3658640 1 -- 172.21.15.27:0/4140707601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9dc19fec0 con 0x7fb9dc101d20 2024-04-25T00:15:07.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:07 smithi153 ceph-mon[32896]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:15:07.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:07 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3077760542' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:07.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:07 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3049822835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:07.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:07 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3074811167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:07.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.774+0000 7fb9e3658640 1 -- 172.21.15.27:0/4140707601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9dc1a00f0 con 0x7fb9dc101d20 2024-04-25T00:15:07.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.775+0000 7fb9d27fc640 1 -- 172.21.15.27:0/4140707601 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb9d40045f0 con 0x7fb9dc101d20 2024-04-25T00:15:07.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.776+0000 7fb9d27fc640 1 --2- 172.21.15.27:0/4140707601 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb9c80a2ca0 0x7fb9c80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.776+0000 7fb9e13f4640 1 --2- 172.21.15.27:0/4140707601 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb9c80a2ca0 0x7fb9c80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:07.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.776+0000 7fb9d27fc640 1 -- 172.21.15.27:0/4140707601 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb9d40f0aa0 con 0x7fb9dc101d20 2024-04-25T00:15:07.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.776+0000 7fb9e13f4640 1 --2- 172.21.15.27:0/4140707601 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb9c80a2ca0 0x7fb9c80a5160 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8005fd0 tx=0x7fb9d800b510 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:07.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.780+0000 7fb9d27fc640 1 -- 172.21.15.27:0/4140707601 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb9d40bd6e0 con 0x7fb9dc101d20 2024-04-25T00:15:07.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.965+0000 7fb9e3658640 1 -- 172.21.15.27:0/4140707601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb9dc1a03e0 con 0x7fb9dc101d20 2024-04-25T00:15:07.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.965+0000 7fb9d27fc640 1 -- 172.21.15.27:0/4140707601 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7fb9d40bcfe0 con 0x7fb9dc101d20 2024-04-25T00:15:07.965 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T00:15:07.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.967+0000 7fb9e3658640 1 -- 172.21.15.27:0/4140707601 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb9c80a2ca0 msgr2=0x7fb9c80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.967+0000 7fb9e3658640 1 --2- 172.21.15.27:0/4140707601 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb9c80a2ca0 0x7fb9c80a5160 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8005fd0 tx=0x7fb9d800b510 comp rx=0 tx=0).stop 2024-04-25T00:15:07.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.967+0000 7fb9e3658640 1 -- 172.21.15.27:0/4140707601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9dc101d20 msgr2=0x7fb9dc19f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.967+0000 7fb9e3658640 1 --2- 172.21.15.27:0/4140707601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9dc101d20 0x7fb9dc19f020 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4009dc0 tx=0x7fb9d4003c20 comp rx=0 tx=0).stop 2024-04-25T00:15:07.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.967+0000 7fb9e3658640 1 -- 172.21.15.27:0/4140707601 shutdown_connections 2024-04-25T00:15:07.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.967+0000 7fb9e3658640 1 --2- 172.21.15.27:0/4140707601 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb9c80a2ca0 0x7fb9c80a5160 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.967+0000 7fb9e3658640 1 --2- 172.21.15.27:0/4140707601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9dc101d20 0x7fb9dc19f020 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.967+0000 7fb9e3658640 1 --2- 172.21.15.27:0/4140707601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9dc1012a0 0x7fb9dc197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.967+0000 7fb9e3658640 1 --2- 172.21.15.27:0/4140707601 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9dc1008e0 0x7fb9dc197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.967+0000 7fb9e3658640 1 -- 172.21.15.27:0/4140707601 >> 172.21.15.27:0/4140707601 conn(0x7fb9dc0fb370 msgr2=0x7fb9dc0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:07.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.967+0000 7fb9e3658640 1 -- 172.21.15.27:0/4140707601 shutdown_connections 2024-04-25T00:15:07.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:07.967+0000 7fb9e3658640 1 -- 172.21.15.27:0/4140707601 wait complete. 2024-04-25T00:15:07.978 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T00:15:08.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.083+0000 7f03eabb7640 1 -- 172.21.15.27:0/4002192700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e41006a0 msgr2=0x7f03e4100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.083+0000 7f03eabb7640 1 --2- 172.21.15.27:0/4002192700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e41006a0 0x7f03e4100ac0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009900 tx=0x7f03d802a410 comp rx=0 tx=0).stop 2024-04-25T00:15:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.083+0000 7f03eabb7640 1 -- 172.21.15.27:0/4002192700 shutdown_connections 2024-04-25T00:15:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.083+0000 7f03eabb7640 1 --2- 172.21.15.27:0/4002192700 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f03e4102250 0x7f03e4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.083+0000 7f03eabb7640 1 --2- 172.21.15.27:0/4002192700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4101890 0x7f03e4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.083+0000 7f03eabb7640 1 --2- 172.21.15.27:0/4002192700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e41006a0 0x7f03e4100ac0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.083+0000 7f03eabb7640 1 -- 172.21.15.27:0/4002192700 >> 172.21.15.27:0/4002192700 conn(0x7f03e40fbc30 msgr2=0x7f03e40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.083+0000 7f03eabb7640 1 -- 172.21.15.27:0/4002192700 shutdown_connections 2024-04-25T00:15:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.083+0000 7f03eabb7640 1 -- 172.21.15.27:0/4002192700 wait complete. 2024-04-25T00:15:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03eabb7640 1 Processor -- start 2024-04-25T00:15:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03eabb7640 1 -- start start 2024-04-25T00:15:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03eabb7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e41006a0 0x7f03e4196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03eabb7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f03e4101890 0x7f03e4196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03eabb7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e4102250 0x7f03e419df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:08.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03eabb7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f03e410c360 con 0x7f03e4102250 2024-04-25T00:15:08.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03eabb7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f03e410c1e0 con 0x7f03e4101890 2024-04-25T00:15:08.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03eabb7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f03e410c4e0 con 0x7f03e41006a0 2024-04-25T00:15:08.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03e9154640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e4102250 0x7f03e419df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:08.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03e9154640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e4102250 0x7f03e419df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47484/0 (socket says 172.21.15.27:47484) 2024-04-25T00:15:08.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03e9154640 1 -- 172.21.15.27:0/1176152468 learned_addr learned my addr 172.21.15.27:0/1176152468 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03e8953640 1 --2- 172.21.15.27:0/1176152468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e41006a0 0x7f03e4196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03e3fff640 1 --2- 172.21.15.27:0/1176152468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f03e4101890 0x7f03e4196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03e9154640 1 -- 172.21.15.27:0/1176152468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e41006a0 msgr2=0x7f03e4196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03e9154640 1 --2- 172.21.15.27:0/1176152468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e41006a0 0x7f03e4196980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03e9154640 1 -- 172.21.15.27:0/1176152468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f03e4101890 msgr2=0x7f03e4196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03e9154640 1 --2- 172.21.15.27:0/1176152468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f03e4101890 0x7f03e4196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03e9154640 1 -- 172.21.15.27:0/1176152468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e4069e40 con 0x7f03e4102250 2024-04-25T00:15:08.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.084+0000 7f03e9154640 1 --2- 172.21.15.27:0/1176152468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e4102250 0x7f03e419df30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f03dc00cce0 tx=0x7f03dc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:08.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.085+0000 7f03e1ffb640 1 -- 172.21.15.27:0/1176152468 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc00f450 con 0x7f03e4102250 2024-04-25T00:15:08.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.085+0000 7f03e1ffb640 1 -- 172.21.15.27:0/1176152468 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03dc00fad0 con 0x7f03e4102250 2024-04-25T00:15:08.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.085+0000 7f03eabb7640 1 -- 172.21.15.27:0/1176152468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03e406a130 con 0x7f03e4102250 2024-04-25T00:15:08.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.085+0000 7f03e1ffb640 1 -- 172.21.15.27:0/1176152468 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc018c10 con 0x7f03e4102250 2024-04-25T00:15:08.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.085+0000 7f03eabb7640 1 -- 172.21.15.27:0/1176152468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03e406a630 con 0x7f03e4102250 2024-04-25T00:15:08.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.086+0000 7f03eabb7640 1 -- 172.21.15.27:0/1176152468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03e4107960 con 0x7f03e4102250 2024-04-25T00:15:08.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.087+0000 7f03e1ffb640 1 -- 172.21.15.27:0/1176152468 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f03dc00f5f0 con 0x7f03e4102250 2024-04-25T00:15:08.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.087+0000 7f03e1ffb640 1 --2- 172.21.15.27:0/1176152468 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03d40a2c20 0x7f03d40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:08.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.087+0000 7f03e8953640 1 --2- 172.21.15.27:0/1176152468 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03d40a2c20 0x7f03d40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:08.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.088+0000 7f03e1ffb640 1 -- 172.21.15.27:0/1176152468 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f03dc0cfcc0 con 0x7f03e4102250 2024-04-25T00:15:08.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.088+0000 7f03e8953640 1 --2- 172.21.15.27:0/1176152468 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03d40a2c20 0x7f03d40a50e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f03d802aa00 tx=0x7f03d8031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:08.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.092+0000 7f03e1ffb640 1 -- 172.21.15.27:0/1176152468 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f03dc014030 con 0x7f03e4102250 2024-04-25T00:15:08.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.277+0000 7f03eabb7640 1 -- 172.21.15.27:0/1176152468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f03e41073d0 con 0x7f03e4102250 2024-04-25T00:15:08.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.277+0000 7f03e1ffb640 1 -- 172.21.15.27:0/1176152468 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f03dc09c760 con 0x7f03e4102250 2024-04-25T00:15:08.277 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:15:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.280+0000 7f03eabb7640 1 -- 172.21.15.27:0/1176152468 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03d40a2c20 msgr2=0x7f03d40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.280+0000 7f03eabb7640 1 --2- 172.21.15.27:0/1176152468 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03d40a2c20 0x7f03d40a50e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f03d802aa00 tx=0x7f03d8031400 comp rx=0 tx=0).stop 2024-04-25T00:15:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.280+0000 7f03eabb7640 1 -- 172.21.15.27:0/1176152468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e4102250 msgr2=0x7f03e419df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.280+0000 7f03eabb7640 1 --2- 172.21.15.27:0/1176152468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e4102250 0x7f03e419df30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f03dc00cce0 tx=0x7f03dc00a620 comp rx=0 tx=0).stop 2024-04-25T00:15:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.280+0000 7f03eabb7640 1 -- 172.21.15.27:0/1176152468 shutdown_connections 2024-04-25T00:15:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.280+0000 7f03eabb7640 1 --2- 172.21.15.27:0/1176152468 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03d40a2c20 0x7f03d40a50e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.280+0000 7f03eabb7640 1 --2- 172.21.15.27:0/1176152468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e4102250 0x7f03e419df30 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.280+0000 7f03eabb7640 1 --2- 172.21.15.27:0/1176152468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f03e4101890 0x7f03e4196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.280+0000 7f03eabb7640 1 --2- 172.21.15.27:0/1176152468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e41006a0 0x7f03e4196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.280+0000 7f03eabb7640 1 -- 172.21.15.27:0/1176152468 >> 172.21.15.27:0/1176152468 conn(0x7f03e40fbc30 msgr2=0x7f03e4104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.280+0000 7f03eabb7640 1 -- 172.21.15.27:0/1176152468 shutdown_connections 2024-04-25T00:15:08.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.280+0000 7f03eabb7640 1 -- 172.21.15.27:0/1176152468 wait complete. 2024-04-25T00:15:08.291 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T00:15:08.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.396+0000 7fb287ff9640 1 -- 172.21.15.27:0/2062307738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2801001b0 msgr2=0x7fb280109d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:08.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.396+0000 7fb287ff9640 1 --2- 172.21.15.27:0/2062307738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2801001b0 0x7fb280109d60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb278009ca0 tx=0x7fb27802a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.396+0000 7fb287ff9640 1 -- 172.21.15.27:0/2062307738 shutdown_connections 2024-04-25T00:15:08.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.396+0000 7fb287ff9640 1 --2- 172.21.15.27:0/2062307738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2801001b0 0x7fb280109d60 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.396+0000 7fb287ff9640 1 --2- 172.21.15.27:0/2062307738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2800ff7f0 0x7fb2800ffc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.396+0000 7fb287ff9640 1 --2- 172.21.15.27:0/2062307738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb280102d80 0x7fb2801031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.396+0000 7fb287ff9640 1 -- 172.21.15.27:0/2062307738 >> 172.21.15.27:0/2062307738 conn(0x7fb2800fb370 msgr2=0x7fb2800fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.396+0000 7fb287ff9640 1 -- 172.21.15.27:0/2062307738 shutdown_connections 2024-04-25T00:15:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.396+0000 7fb287ff9640 1 -- 172.21.15.27:0/2062307738 wait complete. 2024-04-25T00:15:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.396+0000 7fb287ff9640 1 Processor -- start 2024-04-25T00:15:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.396+0000 7fb287ff9640 1 -- start start 2024-04-25T00:15:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.397+0000 7fb287ff9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2800ff7f0 0x7fb280102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.397+0000 7fb287ff9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2801001b0 0x7fb280100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.397+0000 7fb287ff9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb280102d80 0x7fb280101000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.397+0000 7fb287ff9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb28010d400 con 0x7fb280102d80 2024-04-25T00:15:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.397+0000 7fb287ff9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb28010d280 con 0x7fb2800ff7f0 2024-04-25T00:15:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.397+0000 7fb287ff9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb28010d580 con 0x7fb2801001b0 2024-04-25T00:15:08.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.397+0000 7fb286596640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb280102d80 0x7fb280101000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:08.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.397+0000 7fb286596640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb280102d80 0x7fb280101000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47508/0 (socket says 172.21.15.27:47508) 2024-04-25T00:15:08.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.397+0000 7fb286596640 1 -- 172.21.15.27:0/2846681843 learned_addr learned my addr 172.21.15.27:0/2846681843 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:08.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.397+0000 7fb285594640 1 --2- 172.21.15.27:0/2846681843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2801001b0 0x7fb280100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:08.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.397+0000 7fb285d95640 1 --2- 172.21.15.27:0/2846681843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2800ff7f0 0x7fb280102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:08.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.397+0000 7fb286596640 1 -- 172.21.15.27:0/2846681843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2801001b0 msgr2=0x7fb280100ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:08.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.397+0000 7fb286596640 1 --2- 172.21.15.27:0/2846681843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2801001b0 0x7fb280100ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.397+0000 7fb286596640 1 -- 172.21.15.27:0/2846681843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2800ff7f0 msgr2=0x7fb280102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:08.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.397+0000 7fb286596640 1 --2- 172.21.15.27:0/2846681843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2800ff7f0 0x7fb280102480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.397+0000 7fb286596640 1 -- 172.21.15.27:0/2846681843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb280101890 con 0x7fb280102d80 2024-04-25T00:15:08.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.397+0000 7fb286596640 1 --2- 172.21.15.27:0/2846681843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb280102d80 0x7fb280101000 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb278031ae0 tx=0x7fb278031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:08.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.399+0000 7fb272ffd640 1 -- 172.21.15.27:0/2846681843 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb278003c30 con 0x7fb280102d80 2024-04-25T00:15:08.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.399+0000 7fb287ff9640 1 -- 172.21.15.27:0/2846681843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2801a1b40 con 0x7fb280102d80 2024-04-25T00:15:08.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.399+0000 7fb272ffd640 1 -- 172.21.15.27:0/2846681843 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb278003dd0 con 0x7fb280102d80 2024-04-25T00:15:08.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.399+0000 7fb272ffd640 1 -- 172.21.15.27:0/2846681843 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27802fc70 con 0x7fb280102d80 2024-04-25T00:15:08.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.399+0000 7fb287ff9640 1 -- 172.21.15.27:0/2846681843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2801a20e0 con 0x7fb280102d80 2024-04-25T00:15:08.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.400+0000 7fb287ff9640 1 -- 172.21.15.27:0/2846681843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb280108c40 con 0x7fb280102d80 2024-04-25T00:15:08.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.401+0000 7fb272ffd640 1 -- 172.21.15.27:0/2846681843 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb278004050 con 0x7fb280102d80 2024-04-25T00:15:08.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.401+0000 7fb272ffd640 1 --2- 172.21.15.27:0/2846681843 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb26c0a2c20 0x7fb26c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:08.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.402+0000 7fb285d95640 1 --2- 172.21.15.27:0/2846681843 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb26c0a2c20 0x7fb26c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:08.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.402+0000 7fb272ffd640 1 -- 172.21.15.27:0/2846681843 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb2780f0660 con 0x7fb280102d80 2024-04-25T00:15:08.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.402+0000 7fb285d95640 1 --2- 172.21.15.27:0/2846681843 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb26c0a2c20 0x7fb26c0a50e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb280069a60 tx=0x7fb27c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:08.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.406+0000 7fb272ffd640 1 -- 172.21.15.27:0/2846681843 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb2780bd330 con 0x7fb280102d80 2024-04-25T00:15:08.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.591+0000 7fb287ff9640 1 -- 172.21.15.27:0/2846681843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb280062340 con 0x7fb280102d80 2024-04-25T00:15:08.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.592+0000 7fb272ffd640 1 -- 172.21.15.27:0/2846681843 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fb2780bcc30 con 0x7fb280102d80 2024-04-25T00:15:08.592 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T00:15:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.593+0000 7fb287ff9640 1 -- 172.21.15.27:0/2846681843 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb26c0a2c20 msgr2=0x7fb26c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.593+0000 7fb287ff9640 1 --2- 172.21.15.27:0/2846681843 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb26c0a2c20 0x7fb26c0a50e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb280069a60 tx=0x7fb27c005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.593+0000 7fb287ff9640 1 -- 172.21.15.27:0/2846681843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb280102d80 msgr2=0x7fb280101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.593+0000 7fb287ff9640 1 --2- 172.21.15.27:0/2846681843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb280102d80 0x7fb280101000 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb278031ae0 tx=0x7fb278031ee0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.594+0000 7fb287ff9640 1 -- 172.21.15.27:0/2846681843 shutdown_connections 2024-04-25T00:15:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.594+0000 7fb287ff9640 1 --2- 172.21.15.27:0/2846681843 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb26c0a2c20 0x7fb26c0a50e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.594+0000 7fb287ff9640 1 --2- 172.21.15.27:0/2846681843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb280102d80 0x7fb280101000 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.594+0000 7fb287ff9640 1 --2- 172.21.15.27:0/2846681843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2801001b0 0x7fb280100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.594+0000 7fb287ff9640 1 --2- 172.21.15.27:0/2846681843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2800ff7f0 0x7fb280102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.594+0000 7fb287ff9640 1 -- 172.21.15.27:0/2846681843 >> 172.21.15.27:0/2846681843 conn(0x7fb2800fb370 msgr2=0x7fb280105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.594+0000 7fb287ff9640 1 -- 172.21.15.27:0/2846681843 shutdown_connections 2024-04-25T00:15:08.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.594+0000 7fb287ff9640 1 -- 172.21.15.27:0/2846681843 wait complete. 2024-04-25T00:15:08.603 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:08 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1354701673' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:08.603 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:08 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4140707601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:08.603 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:08 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1176152468' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:08.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T00:15:08.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.710+0000 7f7601d61640 1 -- 172.21.15.27:0/3392287033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75fc101790 msgr2=0x7f75fc101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:08.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.710+0000 7f7601d61640 1 --2- 172.21.15.27:0/3392287033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75fc101790 0x7f75fc101bb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f75e4009900 tx=0x7f75e402a410 comp rx=0 tx=0).stop 2024-04-25T00:15:08.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.710+0000 7f7601d61640 1 -- 172.21.15.27:0/3392287033 shutdown_connections 2024-04-25T00:15:08.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.710+0000 7f7601d61640 1 --2- 172.21.15.27:0/3392287033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75fc103340 0x7f75fc109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.710+0000 7f7601d61640 1 --2- 172.21.15.27:0/3392287033 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75fc102980 0x7f75fc102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.710+0000 7f7601d61640 1 --2- 172.21.15.27:0/3392287033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75fc101790 0x7f75fc101bb0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.710+0000 7f7601d61640 1 -- 172.21.15.27:0/3392287033 >> 172.21.15.27:0/3392287033 conn(0x7f75fc0fcd20 msgr2=0x7f75fc0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:08.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.710+0000 7f7601d61640 1 -- 172.21.15.27:0/3392287033 shutdown_connections 2024-04-25T00:15:08.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.710+0000 7f7601d61640 1 -- 172.21.15.27:0/3392287033 wait complete. 2024-04-25T00:15:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.710+0000 7f7601d61640 1 Processor -- start 2024-04-25T00:15:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.710+0000 7f7601d61640 1 -- start start 2024-04-25T00:15:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f7601d61640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75fc101790 0x7f75fc197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f7601d61640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75fc102980 0x7f75fc198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f7601d61640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75fc103340 0x7f75fc19f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f7601d61640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f75fc10d410 con 0x7f75fc102980 2024-04-25T00:15:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f7601d61640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f75fc10d290 con 0x7f75fc101790 2024-04-25T00:15:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f7601d61640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f75fc10d590 con 0x7f75fc103340 2024-04-25T00:15:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f75faffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75fc102980 0x7f75fc198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f75faffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75fc102980 0x7f75fc198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47524/0 (socket says 172.21.15.27:47524) 2024-04-25T00:15:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f75faffd640 1 -- 172.21.15.27:0/3276151716 learned_addr learned my addr 172.21.15.27:0/3276151716 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f75fb7fe640 1 --2- 172.21.15.27:0/3276151716 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75fc101790 0x7f75fc197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f75fbfff640 1 --2- 172.21.15.27:0/3276151716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75fc103340 0x7f75fc19f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f75faffd640 1 -- 172.21.15.27:0/3276151716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75fc103340 msgr2=0x7f75fc19f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f75faffd640 1 --2- 172.21.15.27:0/3276151716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75fc103340 0x7f75fc19f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f75faffd640 1 -- 172.21.15.27:0/3276151716 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75fc101790 msgr2=0x7f75fc197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f75faffd640 1 --2- 172.21.15.27:0/3276151716 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75fc101790 0x7f75fc197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f75faffd640 1 -- 172.21.15.27:0/3276151716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75fc19f730 con 0x7f75fc102980 2024-04-25T00:15:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f75faffd640 1 --2- 172.21.15.27:0/3276151716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75fc102980 0x7f75fc198050 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00e9f0 tx=0x7f75ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f75f8ff9640 1 -- 172.21.15.27:0/3276151716 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec00cd80 con 0x7f75fc102980 2024-04-25T00:15:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f7601d61640 1 -- 172.21.15.27:0/3276151716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75fc19fa20 con 0x7f75fc102980 2024-04-25T00:15:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f75f8ff9640 1 -- 172.21.15.27:0/3276151716 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75ec010910 con 0x7f75fc102980 2024-04-25T00:15:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f75f8ff9640 1 -- 172.21.15.27:0/3276151716 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec0188f0 con 0x7f75fc102980 2024-04-25T00:15:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.711+0000 7f7601d61640 1 -- 172.21.15.27:0/3276151716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75fc19ffc0 con 0x7f75fc102980 2024-04-25T00:15:08.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.713+0000 7f7601d61640 1 -- 172.21.15.27:0/3276151716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75fc108b30 con 0x7f75fc102980 2024-04-25T00:15:08.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.713+0000 7f75f8ff9640 1 -- 172.21.15.27:0/3276151716 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f75ec010ac0 con 0x7f75fc102980 2024-04-25T00:15:08.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.714+0000 7f75f8ff9640 1 --2- 172.21.15.27:0/3276151716 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f75e80a2c20 0x7f75e80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:08.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.714+0000 7f75fb7fe640 1 --2- 172.21.15.27:0/3276151716 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f75e80a2c20 0x7f75e80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:08.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.714+0000 7f75f8ff9640 1 -- 172.21.15.27:0/3276151716 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f75ec0cfa20 con 0x7f75fc102980 2024-04-25T00:15:08.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.715+0000 7f75fb7fe640 1 --2- 172.21.15.27:0/3276151716 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f75e80a2c20 0x7f75e80a50e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f75e4006010 tx=0x7f75e4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:08.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.718+0000 7f75f8ff9640 1 -- 172.21.15.27:0/3276151716 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f75ec09c4c0 con 0x7f75fc102980 2024-04-25T00:15:08.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:08 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1354701673' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:08.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:08 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4140707601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:08.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:08 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1176152468' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:08.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:08 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1354701673' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:08.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:08 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4140707601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:08.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:08 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1176152468' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:08.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.904+0000 7f7601d61640 1 -- 172.21.15.27:0/3276151716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f75fc02d320 con 0x7f75fc102980 2024-04-25T00:15:08.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.905+0000 7f75f8ff9640 1 -- 172.21.15.27:0/3276151716 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f75ec09bdc0 con 0x7f75fc102980 2024-04-25T00:15:08.905 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:15:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.907+0000 7f7601d61640 1 -- 172.21.15.27:0/3276151716 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f75e80a2c20 msgr2=0x7f75e80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.907+0000 7f7601d61640 1 --2- 172.21.15.27:0/3276151716 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f75e80a2c20 0x7f75e80a50e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f75e4006010 tx=0x7f75e4000bc0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.907+0000 7f7601d61640 1 -- 172.21.15.27:0/3276151716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75fc102980 msgr2=0x7f75fc198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.907+0000 7f7601d61640 1 --2- 172.21.15.27:0/3276151716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75fc102980 0x7f75fc198050 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00e9f0 tx=0x7f75ec00eeb0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.907+0000 7f7601d61640 1 -- 172.21.15.27:0/3276151716 shutdown_connections 2024-04-25T00:15:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.907+0000 7f7601d61640 1 --2- 172.21.15.27:0/3276151716 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f75e80a2c20 0x7f75e80a50e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.907+0000 7f7601d61640 1 --2- 172.21.15.27:0/3276151716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75fc103340 0x7f75fc19f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.907+0000 7f7601d61640 1 --2- 172.21.15.27:0/3276151716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75fc102980 0x7f75fc198050 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.907+0000 7f7601d61640 1 --2- 172.21.15.27:0/3276151716 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75fc101790 0x7f75fc197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.907+0000 7f7601d61640 1 -- 172.21.15.27:0/3276151716 >> 172.21.15.27:0/3276151716 conn(0x7f75fc0fcd20 msgr2=0x7f75fc105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.907+0000 7f7601d61640 1 -- 172.21.15.27:0/3276151716 shutdown_connections 2024-04-25T00:15:08.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:08.907+0000 7f7601d61640 1 -- 172.21.15.27:0/3276151716 wait complete. 2024-04-25T00:15:08.918 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:09.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.019+0000 7f0c21bdc640 1 -- 172.21.15.27:0/1348180670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c1006a0 msgr2=0x7f0c1c100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:09.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.019+0000 7f0c21bdc640 1 --2- 172.21.15.27:0/1348180670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c1006a0 0x7f0c1c100ac0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009900 tx=0x7f0c0c02a410 comp rx=0 tx=0).stop 2024-04-25T00:15:09.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.019+0000 7f0c21bdc640 1 -- 172.21.15.27:0/1348180670 shutdown_connections 2024-04-25T00:15:09.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.019+0000 7f0c21bdc640 1 --2- 172.21.15.27:0/1348180670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c1c102250 0x7f0c1c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.019+0000 7f0c21bdc640 1 --2- 172.21.15.27:0/1348180670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c1c101890 0x7f0c1c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.019+0000 7f0c21bdc640 1 --2- 172.21.15.27:0/1348180670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c1006a0 0x7f0c1c100ac0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.019+0000 7f0c21bdc640 1 -- 172.21.15.27:0/1348180670 >> 172.21.15.27:0/1348180670 conn(0x7f0c1c0fbc30 msgr2=0x7f0c1c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:09.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.019+0000 7f0c21bdc640 1 -- 172.21.15.27:0/1348180670 shutdown_connections 2024-04-25T00:15:09.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.019+0000 7f0c21bdc640 1 -- 172.21.15.27:0/1348180670 wait complete. 2024-04-25T00:15:09.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.020+0000 7f0c21bdc640 1 Processor -- start 2024-04-25T00:15:09.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.020+0000 7f0c21bdc640 1 -- start start 2024-04-25T00:15:09.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.020+0000 7f0c21bdc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c1c1006a0 0x7f0c1c1045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:09.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.020+0000 7f0c21bdc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c1c101890 0x7f0c1c102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:09.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.020+0000 7f0c21bdc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c102250 0x7f0c1c1031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:09.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.020+0000 7f0c21bdc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c10c340 con 0x7f0c1c102250 2024-04-25T00:15:09.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.020+0000 7f0c21bdc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c10c1c0 con 0x7f0c1c101890 2024-04-25T00:15:09.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.020+0000 7f0c21bdc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c10c4c0 con 0x7f0c1c1006a0 2024-04-25T00:15:09.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.020+0000 7f0c1bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c102250 0x7f0c1c1031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:09.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.020+0000 7f0c1bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c102250 0x7f0c1c1031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47548/0 (socket says 172.21.15.27:47548) 2024-04-25T00:15:09.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.020+0000 7f0c1bfff640 1 -- 172.21.15.27:0/4292462975 learned_addr learned my addr 172.21.15.27:0/4292462975 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:09.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.021+0000 7f0c1b7fe640 1 --2- 172.21.15.27:0/4292462975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c1c1006a0 0x7f0c1c1045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:09.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.021+0000 7f0c1affd640 1 --2- 172.21.15.27:0/4292462975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c1c101890 0x7f0c1c102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:09.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.021+0000 7f0c1bfff640 1 -- 172.21.15.27:0/4292462975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c1c1006a0 msgr2=0x7f0c1c1045e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:09.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.021+0000 7f0c1bfff640 1 --2- 172.21.15.27:0/4292462975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c1c1006a0 0x7f0c1c1045e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.021+0000 7f0c1bfff640 1 -- 172.21.15.27:0/4292462975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c1c101890 msgr2=0x7f0c1c102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:09.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.021+0000 7f0c1bfff640 1 --2- 172.21.15.27:0/4292462975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c1c101890 0x7f0c1c102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.021+0000 7f0c1bfff640 1 -- 172.21.15.27:0/4292462975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c1c103a70 con 0x7f0c1c102250 2024-04-25T00:15:09.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.021+0000 7f0c1b7fe640 1 --2- 172.21.15.27:0/4292462975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c1c1006a0 0x7f0c1c1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:09.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.021+0000 7f0c1bfff640 1 --2- 172.21.15.27:0/4292462975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c102250 0x7f0c1c1031b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0c1400eaa0 tx=0x7f0c1400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:09.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.021+0000 7f0c18ff9640 1 -- 172.21.15.27:0/4292462975 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1400cdb0 con 0x7f0c1c102250 2024-04-25T00:15:09.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.021+0000 7f0c21bdc640 1 -- 172.21.15.27:0/4292462975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c1c1a1b60 con 0x7f0c1c102250 2024-04-25T00:15:09.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.021+0000 7f0c18ff9640 1 -- 172.21.15.27:0/4292462975 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c14005020 con 0x7f0c1c102250 2024-04-25T00:15:09.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.021+0000 7f0c18ff9640 1 -- 172.21.15.27:0/4292462975 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c140116b0 con 0x7f0c1c102250 2024-04-25T00:15:09.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.021+0000 7f0c21bdc640 1 -- 172.21.15.27:0/4292462975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c1c1a2080 con 0x7f0c1c102250 2024-04-25T00:15:09.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.022+0000 7f0c21bdc640 1 -- 172.21.15.27:0/4292462975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c1c1a22b0 con 0x7f0c1c102250 2024-04-25T00:15:09.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.023+0000 7f0c18ff9640 1 -- 172.21.15.27:0/4292462975 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f0c14005560 con 0x7f0c1c102250 2024-04-25T00:15:09.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.024+0000 7f0c18ff9640 1 --2- 172.21.15.27:0/4292462975 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f0c040a2c20 0x7f0c040a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:09.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.024+0000 7f0c1b7fe640 1 --2- 172.21.15.27:0/4292462975 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f0c040a2c20 0x7f0c040a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:09.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.024+0000 7f0c18ff9640 1 -- 172.21.15.27:0/4292462975 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0c140cf8c0 con 0x7f0c1c102250 2024-04-25T00:15:09.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.024+0000 7f0c1b7fe640 1 --2- 172.21.15.27:0/4292462975 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f0c040a2c20 0x7f0c040a50e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c02a930 tx=0x7f0c0c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:09.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.028+0000 7f0c18ff9640 1 -- 172.21.15.27:0/4292462975 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0c14016030 con 0x7f0c1c102250 2024-04-25T00:15:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.252+0000 7f0c21bdc640 1 -- 172.21.15.27:0/4292462975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c1c1a2640 con 0x7f0c1c102250 2024-04-25T00:15:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.252+0000 7f0c18ff9640 1 -- 172.21.15.27:0/4292462975 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11944 (secure 0 0 0) 0x7f0c1409c300 con 0x7f0c1c102250 2024-04-25T00:15:09.252 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:09.253 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:13:47.575422+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24428,24496,24521,24430,24514]},"id":1}]} 2024-04-25T00:15:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 28 2024-04-25T00:15:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.254+0000 7f0c21bdc640 1 -- 172.21.15.27:0/4292462975 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f0c040a2c20 msgr2=0x7f0c040a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.254+0000 7f0c21bdc640 1 --2- 172.21.15.27:0/4292462975 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f0c040a2c20 0x7f0c040a50e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c02a930 tx=0x7f0c0c000bc0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.254+0000 7f0c21bdc640 1 -- 172.21.15.27:0/4292462975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c102250 msgr2=0x7f0c1c1031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.254+0000 7f0c21bdc640 1 --2- 172.21.15.27:0/4292462975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c102250 0x7f0c1c1031b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0c1400eaa0 tx=0x7f0c1400ef60 comp rx=0 tx=0).stop 2024-04-25T00:15:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.254+0000 7f0c21bdc640 1 -- 172.21.15.27:0/4292462975 shutdown_connections 2024-04-25T00:15:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.254+0000 7f0c21bdc640 1 --2- 172.21.15.27:0/4292462975 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f0c040a2c20 0x7f0c040a50e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.254+0000 7f0c21bdc640 1 --2- 172.21.15.27:0/4292462975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c102250 0x7f0c1c1031b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.254+0000 7f0c21bdc640 1 --2- 172.21.15.27:0/4292462975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c1c101890 0x7f0c1c102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.254+0000 7f0c21bdc640 1 --2- 172.21.15.27:0/4292462975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c1c1006a0 0x7f0c1c1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.254+0000 7f0c21bdc640 1 -- 172.21.15.27:0/4292462975 >> 172.21.15.27:0/4292462975 conn(0x7f0c1c0fbc30 msgr2=0x7f0c1c0fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.255+0000 7f0c21bdc640 1 -- 172.21.15.27:0/4292462975 shutdown_connections 2024-04-25T00:15:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.255+0000 7f0c21bdc640 1 -- 172.21.15.27:0/4292462975 wait complete. 2024-04-25T00:15:09.266 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:15:09.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.367+0000 7fabc57ed640 1 -- 172.21.15.27:0/2528366559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc0101890 msgr2=0x7fabc0101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:09.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.367+0000 7fabc57ed640 1 --2- 172.21.15.27:0/2528366559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc0101890 0x7fabc0101d10 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009900 tx=0x7fabb402a410 comp rx=0 tx=0).stop 2024-04-25T00:15:09.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.367+0000 7fabc57ed640 1 -- 172.21.15.27:0/2528366559 shutdown_connections 2024-04-25T00:15:09.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.367+0000 7fabc57ed640 1 --2- 172.21.15.27:0/2528366559 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fabc0102250 0x7fabc0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.367+0000 7fabc57ed640 1 --2- 172.21.15.27:0/2528366559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc0101890 0x7fabc0101d10 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.367+0000 7fabc57ed640 1 --2- 172.21.15.27:0/2528366559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabc01006a0 0x7fabc0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.367+0000 7fabc57ed640 1 -- 172.21.15.27:0/2528366559 >> 172.21.15.27:0/2528366559 conn(0x7fabc00fbc30 msgr2=0x7fabc00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:09.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.367+0000 7fabc57ed640 1 -- 172.21.15.27:0/2528366559 shutdown_connections 2024-04-25T00:15:09.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.367+0000 7fabc57ed640 1 -- 172.21.15.27:0/2528366559 wait complete. 2024-04-25T00:15:09.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.368+0000 7fabc57ed640 1 Processor -- start 2024-04-25T00:15:09.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.368+0000 7fabc57ed640 1 -- start start 2024-04-25T00:15:09.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.368+0000 7fabc57ed640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc01006a0 0x7fabc0197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:09.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.368+0000 7fabc57ed640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabc0101890 0x7fabc0198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:09.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.368+0000 7fabc57ed640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fabc0102250 0x7fabc019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:09.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.368+0000 7fabc57ed640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fabc010d440 con 0x7fabc01006a0 2024-04-25T00:15:09.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.368+0000 7fabc57ed640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fabc010d2c0 con 0x7fabc0102250 2024-04-25T00:15:09.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.368+0000 7fabc57ed640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fabc010d5c0 con 0x7fabc0101890 2024-04-25T00:15:09.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.368+0000 7fabbeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc01006a0 0x7fabc0197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:09.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.368+0000 7fabbeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc01006a0 0x7fabc0197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47580/0 (socket says 172.21.15.27:47580) 2024-04-25T00:15:09.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.368+0000 7fabbeffd640 1 -- 172.21.15.27:0/372645368 learned_addr learned my addr 172.21.15.27:0/372645368 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:09.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.368+0000 7fabbe7fc640 1 --2- 172.21.15.27:0/372645368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabc0101890 0x7fabc0198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:09.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.369+0000 7fabbf7fe640 1 --2- 172.21.15.27:0/372645368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fabc0102250 0x7fabc019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:09.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.369+0000 7fabbeffd640 1 -- 172.21.15.27:0/372645368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabc0101890 msgr2=0x7fabc0198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:09.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.369+0000 7fabbeffd640 1 --2- 172.21.15.27:0/372645368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabc0101890 0x7fabc0198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.369+0000 7fabbeffd640 1 -- 172.21.15.27:0/372645368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fabc0102250 msgr2=0x7fabc019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:09.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.369+0000 7fabbeffd640 1 --2- 172.21.15.27:0/372645368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fabc0102250 0x7fabc019f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.369+0000 7fabbeffd640 1 -- 172.21.15.27:0/372645368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc019f710 con 0x7fabc01006a0 2024-04-25T00:15:09.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.369+0000 7fabbeffd640 1 --2- 172.21.15.27:0/372645368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc01006a0 0x7fabc0197af0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fabb000e9c0 tx=0x7fabb000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:09.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.369+0000 7faba7fff640 1 -- 172.21.15.27:0/372645368 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb000cd80 con 0x7fabc01006a0 2024-04-25T00:15:09.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.369+0000 7fabc57ed640 1 -- 172.21.15.27:0/372645368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc019fa00 con 0x7fabc01006a0 2024-04-25T00:15:09.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.369+0000 7faba7fff640 1 -- 172.21.15.27:0/372645368 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabb00035f0 con 0x7fabc01006a0 2024-04-25T00:15:09.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.369+0000 7faba7fff640 1 -- 172.21.15.27:0/372645368 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb0010690 con 0x7fabc01006a0 2024-04-25T00:15:09.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.369+0000 7fabc57ed640 1 -- 172.21.15.27:0/372645368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc019fed0 con 0x7fabc01006a0 2024-04-25T00:15:09.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.370+0000 7fabc57ed640 1 -- 172.21.15.27:0/372645368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabc01079e0 con 0x7fabc01006a0 2024-04-25T00:15:09.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.371+0000 7faba7fff640 1 -- 172.21.15.27:0/372645368 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fabb00037a0 con 0x7fabc01006a0 2024-04-25T00:15:09.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.372+0000 7faba7fff640 1 --2- 172.21.15.27:0/372645368 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7faba80a2c20 0x7faba80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:09.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.372+0000 7fabbe7fc640 1 --2- 172.21.15.27:0/372645368 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7faba80a2c20 0x7faba80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:09.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.372+0000 7faba7fff640 1 -- 172.21.15.27:0/372645368 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fabb00cf9d0 con 0x7fabc01006a0 2024-04-25T00:15:09.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.373+0000 7fabbe7fc640 1 --2- 172.21.15.27:0/372645368 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7faba80a2c20 0x7faba80a50e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fabb40314e0 tx=0x7fabb4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:09.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.376+0000 7faba7fff640 1 -- 172.21.15.27:0/372645368 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fabb009c470 con 0x7fabc01006a0 2024-04-25T00:15:09.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.577+0000 7fabc57ed640 1 -- 172.21.15.27:0/372645368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fabc002d360 con 0x7fabc01006a0 2024-04-25T00:15:09.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.580+0000 7faba7fff640 1 -- 172.21.15.27:0/372645368 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24643 (secure 0 0 0) 0x7fabb009bd70 con 0x7fabc01006a0 2024-04-25T00:15:09.581 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:09.581 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":91,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:14:11.249732+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:15:09.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.585+0000 7fabc57ed640 1 -- 172.21.15.27:0/372645368 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7faba80a2c20 msgr2=0x7faba80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:09.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.585+0000 7fabc57ed640 1 --2- 172.21.15.27:0/372645368 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7faba80a2c20 0x7faba80a50e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fabb40314e0 tx=0x7fabb4000bc0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.585+0000 7fabc57ed640 1 -- 172.21.15.27:0/372645368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc01006a0 msgr2=0x7fabc0197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:09.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.585+0000 7fabc57ed640 1 --2- 172.21.15.27:0/372645368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc01006a0 0x7fabc0197af0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fabb000e9c0 tx=0x7fabb000ee80 comp rx=0 tx=0).stop 2024-04-25T00:15:09.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.585+0000 7fabc57ed640 1 -- 172.21.15.27:0/372645368 shutdown_connections 2024-04-25T00:15:09.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.585+0000 7fabc57ed640 1 --2- 172.21.15.27:0/372645368 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7faba80a2c20 0x7faba80a50e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.585+0000 7fabc57ed640 1 --2- 172.21.15.27:0/372645368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fabc0102250 0x7fabc019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.585+0000 7fabc57ed640 1 --2- 172.21.15.27:0/372645368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabc0101890 0x7fabc0198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.585+0000 7fabc57ed640 1 --2- 172.21.15.27:0/372645368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc01006a0 0x7fabc0197af0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.585+0000 7fabc57ed640 1 -- 172.21.15.27:0/372645368 >> 172.21.15.27:0/372645368 conn(0x7fabc00fbc30 msgr2=0x7fabc0104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:09.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.587+0000 7fabc57ed640 1 -- 172.21.15.27:0/372645368 shutdown_connections 2024-04-25T00:15:09.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.587+0000 7fabc57ed640 1 -- 172.21.15.27:0/372645368 wait complete. 2024-04-25T00:15:09.604 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:09 smithi027 ceph-mon[29208]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:15:09.604 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:09 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2846681843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:09.604 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:09 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3276151716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:09.604 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:09 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4292462975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:09.606 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:09.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.708+0000 7f03ead1b640 1 -- 172.21.15.27:0/4043479907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e4102250 msgr2=0x7f03e4108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:09.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.708+0000 7f03ead1b640 1 --2- 172.21.15.27:0/4043479907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e4102250 0x7f03e4108ae0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f03dc009ca0 tx=0x7f03dc02a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.708+0000 7f03ead1b640 1 -- 172.21.15.27:0/4043479907 shutdown_connections 2024-04-25T00:15:09.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.708+0000 7f03ead1b640 1 --2- 172.21.15.27:0/4043479907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e4102250 0x7f03e4108ae0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.708+0000 7f03ead1b640 1 --2- 172.21.15.27:0/4043479907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4101890 0x7f03e4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.708+0000 7f03ead1b640 1 --2- 172.21.15.27:0/4043479907 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f03e41006a0 0x7f03e4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.708+0000 7f03ead1b640 1 -- 172.21.15.27:0/4043479907 >> 172.21.15.27:0/4043479907 conn(0x7f03e40fbc30 msgr2=0x7f03e40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.708+0000 7f03ead1b640 1 -- 172.21.15.27:0/4043479907 shutdown_connections 2024-04-25T00:15:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.708+0000 7f03ead1b640 1 -- 172.21.15.27:0/4043479907 wait complete. 2024-04-25T00:15:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.709+0000 7f03ead1b640 1 Processor -- start 2024-04-25T00:15:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.709+0000 7f03ead1b640 1 -- start start 2024-04-25T00:15:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.709+0000 7f03ead1b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e41006a0 0x7f03e410b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.709+0000 7f03ead1b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f03e4101890 0x7f03e4109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.709+0000 7f03ead1b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4102250 0x7f03e4109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.709+0000 7f03ead1b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f03e410d440 con 0x7f03e41006a0 2024-04-25T00:15:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.709+0000 7f03ead1b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f03e410d2c0 con 0x7f03e4101890 2024-04-25T00:15:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.709+0000 7f03ead1b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f03e410d5c0 con 0x7f03e4102250 2024-04-25T00:15:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.709+0000 7f03e8ab7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e41006a0 0x7f03e410b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.709+0000 7f03e8ab7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e41006a0 0x7f03e410b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47604/0 (socket says 172.21.15.27:47604) 2024-04-25T00:15:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.709+0000 7f03e8ab7640 1 -- 172.21.15.27:0/3926014226 learned_addr learned my addr 172.21.15.27:0/3926014226 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.709+0000 7f03e92b8640 1 --2- 172.21.15.27:0/3926014226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4102250 0x7f03e4109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.709+0000 7f03e3fff640 1 --2- 172.21.15.27:0/3926014226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f03e4101890 0x7f03e4109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.710+0000 7f03e8ab7640 1 -- 172.21.15.27:0/3926014226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4102250 msgr2=0x7f03e4109ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.710+0000 7f03e8ab7640 1 --2- 172.21.15.27:0/3926014226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4102250 0x7f03e4109ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.710+0000 7f03e8ab7640 1 -- 172.21.15.27:0/3926014226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f03e4101890 msgr2=0x7f03e4109660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.710+0000 7f03e8ab7640 1 --2- 172.21.15.27:0/3926014226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f03e4101890 0x7f03e4109660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.710+0000 7f03e8ab7640 1 -- 172.21.15.27:0/3926014226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e410a430 con 0x7f03e41006a0 2024-04-25T00:15:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.710+0000 7f03e3fff640 1 --2- 172.21.15.27:0/3926014226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f03e4101890 0x7f03e4109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.710+0000 7f03e8ab7640 1 --2- 172.21.15.27:0/3926014226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e41006a0 0x7f03e410b020 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f03d800d770 tx=0x7f03d800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.710+0000 7f03e1ffb640 1 -- 172.21.15.27:0/3926014226 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d8017070 con 0x7f03e41006a0 2024-04-25T00:15:09.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.710+0000 7f03e1ffb640 1 -- 172.21.15.27:0/3926014226 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03d8009c30 con 0x7f03e41006a0 2024-04-25T00:15:09.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.710+0000 7f03ead1b640 1 -- 172.21.15.27:0/3926014226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03e41a1bd0 con 0x7f03e41006a0 2024-04-25T00:15:09.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.710+0000 7f03e1ffb640 1 -- 172.21.15.27:0/3926014226 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d8003dd0 con 0x7f03e41006a0 2024-04-25T00:15:09.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.710+0000 7f03ead1b640 1 -- 172.21.15.27:0/3926014226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03e41a2170 con 0x7f03e41006a0 2024-04-25T00:15:09.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.711+0000 7f03ead1b640 1 -- 172.21.15.27:0/3926014226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03e4107960 con 0x7f03e41006a0 2024-04-25T00:15:09.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.712+0000 7f03e1ffb640 1 -- 172.21.15.27:0/3926014226 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f03d8003fb0 con 0x7f03e41006a0 2024-04-25T00:15:09.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.713+0000 7f03e1ffb640 1 --2- 172.21.15.27:0/3926014226 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03d40a2ca0 0x7f03d40a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:09.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.713+0000 7f03e3fff640 1 --2- 172.21.15.27:0/3926014226 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03d40a2ca0 0x7f03d40a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:09.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.713+0000 7f03e1ffb640 1 -- 172.21.15.27:0/3926014226 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f03d80ceec0 con 0x7f03e41006a0 2024-04-25T00:15:09.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.713+0000 7f03e3fff640 1 --2- 172.21.15.27:0/3926014226 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03d40a2ca0 0x7f03d40a5160 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f03d00058c0 tx=0x7f03d0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.717+0000 7f03e1ffb640 1 -- 172.21.15.27:0/3926014226 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f03d8009de0 con 0x7f03e41006a0 2024-04-25T00:15:09.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:09 smithi139 ceph-mon[32833]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:15:09.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:09 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2846681843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:09.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:09 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3276151716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:09.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:09 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4292462975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:09.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:09 smithi153 ceph-mon[32896]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:15:09.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:09 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2846681843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:09.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:09 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3276151716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:09.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:09 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4292462975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.943+0000 7f03ead1b640 1 -- 172.21.15.27:0/3926014226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03e402d290 con 0x7f03e41006a0 2024-04-25T00:15:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.943+0000 7f03e1ffb640 1 -- 172.21.15.27:0/3926014226 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11944 (secure 0 0 0) 0x7f03d809b960 con 0x7f03e41006a0 2024-04-25T00:15:09.943 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:09.943 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:13:47.575422+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24428,24496,24521,24430,24514]},"id":1}]} 2024-04-25T00:15:09.944 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 28 2024-04-25T00:15:09.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.945+0000 7f03ead1b640 1 -- 172.21.15.27:0/3926014226 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03d40a2ca0 msgr2=0x7f03d40a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:09.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.945+0000 7f03ead1b640 1 --2- 172.21.15.27:0/3926014226 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03d40a2ca0 0x7f03d40a5160 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f03d00058c0 tx=0x7f03d0005850 comp rx=0 tx=0).stop 2024-04-25T00:15:09.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.945+0000 7f03ead1b640 1 -- 172.21.15.27:0/3926014226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e41006a0 msgr2=0x7f03e410b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:09.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.945+0000 7f03ead1b640 1 --2- 172.21.15.27:0/3926014226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e41006a0 0x7f03e410b020 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f03d800d770 tx=0x7f03d800dc30 comp rx=0 tx=0).stop 2024-04-25T00:15:09.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.945+0000 7f03ead1b640 1 -- 172.21.15.27:0/3926014226 shutdown_connections 2024-04-25T00:15:09.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.945+0000 7f03ead1b640 1 --2- 172.21.15.27:0/3926014226 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f03d40a2ca0 0x7f03d40a5160 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.945+0000 7f03ead1b640 1 --2- 172.21.15.27:0/3926014226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4102250 0x7f03e4109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.945+0000 7f03ead1b640 1 --2- 172.21.15.27:0/3926014226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f03e4101890 0x7f03e4109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.945+0000 7f03ead1b640 1 --2- 172.21.15.27:0/3926014226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e41006a0 0x7f03e410b020 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:09.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.945+0000 7f03ead1b640 1 -- 172.21.15.27:0/3926014226 >> 172.21.15.27:0/3926014226 conn(0x7f03e40fbc30 msgr2=0x7f03e4104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:09.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.945+0000 7f03ead1b640 1 -- 172.21.15.27:0/3926014226 shutdown_connections 2024-04-25T00:15:09.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:09.945+0000 7f03ead1b640 1 -- 172.21.15.27:0/3926014226 wait complete. 2024-04-25T00:15:09.957 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T00:13:18.872210+0000', 'modified': '2024-04-25T00:13:47.575422+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24521, 'mds_1': 24514, 'mds_2': 24496, 'mds_3': 24430, 'mds_4': 24428}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14652': {'gid': 14652, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.27:6839/3286639641', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6838', 'nonce': 3286639641}, {'type': 'v1', 'addr': '172.21.15.27:6839', 'nonce': 3286639641}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14664': {'gid': 14664, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.27:6841/2271898034', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6840', 'nonce': 2271898034}, {'type': 'v1', 'addr': '172.21.15.27:6841', 'nonce': 2271898034}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24428': {'gid': 24428, 'name': 'k', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.139:6833/3964352148', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6832', 'nonce': 3964352148}, {'type': 'v1', 'addr': '172.21.15.139:6833', 'nonce': 3964352148}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24430': {'gid': 24430, 'name': 'l', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.153:6833/2700717364', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6832', 'nonce': 2700717364}, {'type': 'v1', 'addr': '172.21.15.153:6833', 'nonce': 2700717364}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24461': {'gid': 24461, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.139:6835/3731284556', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6834', 'nonce': 3731284556}, {'type': 'v1', 'addr': '172.21.15.139:6835', 'nonce': 3731284556}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6835/584718807', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6834', 'nonce': 584718807}, {'type': 'v1', 'addr': '172.21.15.153:6835', 'nonce': 584718807}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.139:6837/2477304919', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6836', 'nonce': 2477304919}, {'type': 'v1', 'addr': '172.21.15.139:6837', 'nonce': 2477304919}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24496': {'gid': 24496, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.153:6837/4066071541', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6836', 'nonce': 4066071541}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 4066071541}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24514': {'gid': 24514, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.153:6839/1144169732', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 1144169732}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 1144169732}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24521': {'gid': 24521, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.139:6839/2728523692', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6838', 'nonce': 2728523692}, {'type': 'v1', 'addr': '172.21.15.139:6839', 'nonce': 2728523692}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24521, 'qdb_cluster': [24428, 24496, 24521, 24430, 24514]} 2024-04-25T00:15:09.957 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-25T00:15:09.957 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-04-25T00:15:10.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.060+0000 7f9c50fc8640 1 -- 172.21.15.27:0/2286086984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c4c102990 msgr2=0x7f9c4c102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.060+0000 7f9c50fc8640 1 --2- 172.21.15.27:0/2286086984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c4c102990 0x7f9c4c102e10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9c34009900 tx=0x7f9c3402a410 comp rx=0 tx=0).stop 2024-04-25T00:15:10.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.060+0000 7f9c50fc8640 1 -- 172.21.15.27:0/2286086984 shutdown_connections 2024-04-25T00:15:10.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.060+0000 7f9c50fc8640 1 --2- 172.21.15.27:0/2286086984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c4c103350 0x7f9c4c109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.060+0000 7f9c50fc8640 1 --2- 172.21.15.27:0/2286086984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c4c102990 0x7f9c4c102e10 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.060+0000 7f9c50fc8640 1 --2- 172.21.15.27:0/2286086984 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c4c1017a0 0x7f9c4c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.060+0000 7f9c50fc8640 1 -- 172.21.15.27:0/2286086984 >> 172.21.15.27:0/2286086984 conn(0x7f9c4c0fcd10 msgr2=0x7f9c4c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.060+0000 7f9c50fc8640 1 -- 172.21.15.27:0/2286086984 shutdown_connections 2024-04-25T00:15:10.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.061+0000 7f9c50fc8640 1 -- 172.21.15.27:0/2286086984 wait complete. 2024-04-25T00:15:10.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.061+0000 7f9c50fc8640 1 Processor -- start 2024-04-25T00:15:10.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.061+0000 7f9c50fc8640 1 -- start start 2024-04-25T00:15:10.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.061+0000 7f9c50fc8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c4c1017a0 0x7f9c4c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.061+0000 7f9c50fc8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c4c102990 0x7f9c4c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.061+0000 7f9c50fc8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c4c103350 0x7f9c4c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.061+0000 7f9c50fc8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4c10d420 con 0x7f9c4c1017a0 2024-04-25T00:15:10.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.061+0000 7f9c50fc8640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4c10d2a0 con 0x7f9c4c102990 2024-04-25T00:15:10.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.061+0000 7f9c50fc8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4c10d5a0 con 0x7f9c4c103350 2024-04-25T00:15:10.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.061+0000 7f9c4a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c4c1017a0 0x7f9c4c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.061+0000 7f9c4a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c4c1017a0 0x7f9c4c197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47608/0 (socket says 172.21.15.27:47608) 2024-04-25T00:15:10.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.061+0000 7f9c4a59c640 1 -- 172.21.15.27:0/2057481190 learned_addr learned my addr 172.21.15.27:0/2057481190 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.062+0000 7f9c4ad9d640 1 --2- 172.21.15.27:0/2057481190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c4c103350 0x7f9c4c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.062+0000 7f9c43fff640 1 --2- 172.21.15.27:0/2057481190 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c4c102990 0x7f9c4c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.062+0000 7f9c4a59c640 1 -- 172.21.15.27:0/2057481190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c4c103350 msgr2=0x7f9c4c19f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.062+0000 7f9c4a59c640 1 --2- 172.21.15.27:0/2057481190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c4c103350 0x7f9c4c19f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.062+0000 7f9c4a59c640 1 -- 172.21.15.27:0/2057481190 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c4c102990 msgr2=0x7f9c4c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.062+0000 7f9c4a59c640 1 --2- 172.21.15.27:0/2057481190 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c4c102990 0x7f9c4c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.062+0000 7f9c4a59c640 1 -- 172.21.15.27:0/2057481190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c4c19f6b0 con 0x7f9c4c1017a0 2024-04-25T00:15:10.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.062+0000 7f9c43fff640 1 --2- 172.21.15.27:0/2057481190 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c4c102990 0x7f9c4c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:10.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.062+0000 7f9c4a59c640 1 --2- 172.21.15.27:0/2057481190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c4c1017a0 0x7f9c4c197a90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c00d770 tx=0x7f9c3c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.062+0000 7f9c437fe640 1 -- 172.21.15.27:0/2057481190 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c017070 con 0x7f9c4c1017a0 2024-04-25T00:15:10.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.062+0000 7f9c50fc8640 1 -- 172.21.15.27:0/2057481190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c4c19f9a0 con 0x7f9c4c1017a0 2024-04-25T00:15:10.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.062+0000 7f9c437fe640 1 -- 172.21.15.27:0/2057481190 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c3c009c30 con 0x7f9c4c1017a0 2024-04-25T00:15:10.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.062+0000 7f9c437fe640 1 -- 172.21.15.27:0/2057481190 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c003dd0 con 0x7f9c4c1017a0 2024-04-25T00:15:10.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.062+0000 7f9c50fc8640 1 -- 172.21.15.27:0/2057481190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c4c19ff10 con 0x7f9c4c1017a0 2024-04-25T00:15:10.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.065+0000 7f9c53db5740 1 Processor -- start 2024-04-25T00:15:10.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.065+0000 7f9c53db5740 1 -- start start 2024-04-25T00:15:10.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.065+0000 7f9c437fe640 1 -- 172.21.15.27:0/2057481190 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9c3c003f70 con 0x7f9c4c1017a0 2024-04-25T00:15:10.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c53db5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3440 0x5646af1a3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c53db5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a3e30 0x5646af1a8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c53db5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a8fa0 0x5646af1ab430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c53db5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5646af106f00 con 0x5646af1a8fa0 2024-04-25T00:15:10.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c53db5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5646af0af750 con 0x5646af1a3e30 2024-04-25T00:15:10.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c53db5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5646af102ec0 con 0x5646af1a3440 2024-04-25T00:15:10.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c43fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a8fa0 0x5646af1ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c43fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a8fa0 0x5646af1ab430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47612/0 (socket says 172.21.15.27:47612) 2024-04-25T00:15:10.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c43fff640 1 -- 172.21.15.27:0/2925064834 learned_addr learned my addr 172.21.15.27:0/2925064834 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c4a59c640 1 --2- 172.21.15.27:0/2925064834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a3e30 0x5646af1a8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c4ad9d640 1 --2- 172.21.15.27:0/2925064834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3440 0x5646af1a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c43fff640 1 -- 172.21.15.27:0/2925064834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3440 msgr2=0x5646af1a3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c43fff640 1 --2- 172.21.15.27:0/2925064834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3440 0x5646af1a3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c43fff640 1 -- 172.21.15.27:0/2925064834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a3e30 msgr2=0x5646af1a8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c43fff640 1 --2- 172.21.15.27:0/2925064834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a3e30 0x5646af1a8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c43fff640 1 -- 172.21.15.27:0/2925064834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646aee1b2d0 con 0x5646af1a8fa0 2024-04-25T00:15:10.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c4ad9d640 1 --2- 172.21.15.27:0/2925064834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3440 0x5646af1a3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:10.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c43fff640 1 --2- 172.21.15.27:0/2925064834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a8fa0 0x5646af1ab430 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9c34031610 tx=0x7f9c34032d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb9563aa4b37d2c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c437fe640 1 --2- 172.21.15.27:0/2057481190 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9c380a2c20 0x7f9c380a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c4ad9d640 1 --2- 172.21.15.27:0/2057481190 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9c380a2c20 0x7f9c380a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c40ff9640 1 -- 172.21.15.27:0/2925064834 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3403f070 con 0x5646af1a8fa0 2024-04-25T00:15:10.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.066+0000 7f9c437fe640 1 -- 172.21.15.27:0/2057481190 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9c3c0ceef0 con 0x7f9c4c1017a0 2024-04-25T00:15:10.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.067+0000 7f9c40ff9640 1 -- 172.21.15.27:0/2925064834 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c34031970 con 0x5646af1a8fa0 2024-04-25T00:15:10.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.067+0000 7f9c40ff9640 1 -- 172.21.15.27:0/2925064834 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c34035070 con 0x5646af1a8fa0 2024-04-25T00:15:10.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.067+0000 7f9c4ad9d640 1 --2- 172.21.15.27:0/2057481190 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9c380a2c20 0x7f9c380a50e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x5646af1a4690 tx=0x7f9c44009600 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.067+0000 7f9c53db5740 1 -- 172.21.15.27:0/2925064834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a8fa0 msgr2=0x5646af1ab430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.067+0000 7f9c53db5740 1 --2- 172.21.15.27:0/2925064834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a8fa0 0x5646af1ab430 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9c34031610 tx=0x7f9c34032d80 comp rx=0 tx=0).stop 2024-04-25T00:15:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.067+0000 7f9c53db5740 1 -- 172.21.15.27:0/2925064834 shutdown_connections 2024-04-25T00:15:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.067+0000 7f9c53db5740 1 --2- 172.21.15.27:0/2925064834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a8fa0 0x5646af1ab430 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.067+0000 7f9c53db5740 1 --2- 172.21.15.27:0/2925064834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a3e30 0x5646af1a8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.067+0000 7f9c53db5740 1 --2- 172.21.15.27:0/2925064834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3440 0x5646af1a3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.067+0000 7f9c53db5740 1 -- 172.21.15.27:0/2925064834 >> 172.21.15.27:0/2925064834 conn(0x5646af19eab0 msgr2=0x5646af19eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.067+0000 7f9c53db5740 1 -- 172.21.15.27:0/2925064834 shutdown_connections 2024-04-25T00:15:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.067+0000 7f9c53db5740 1 -- 172.21.15.27:0/2925064834 wait complete. 2024-04-25T00:15:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c53db5740 1 -- start start 2024-04-25T00:15:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c53db5740 1 Processor -- start 2024-04-25T00:15:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c53db5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a3440 0x5646af1ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c53db5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3e30 0x5646af1add10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c53db5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a8fa0 0x5646af1b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c53db5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5646af1abef0 con 0x5646af1a3440 2024-04-25T00:15:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c53db5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5646af0c2cf0 con 0x5646af1a8fa0 2024-04-25T00:15:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c53db5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5646af1ac070 con 0x5646af1a3e30 2024-04-25T00:15:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c53db5740 10 client.0 _ll_register_callbacks cb 0x5646af0c26e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c4a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a3440 0x5646af1ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c4a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a3440 0x5646af1ad7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47618/0 (socket says 172.21.15.27:47618) 2024-04-25T00:15:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c4a59c640 1 -- 172.21.15.27:0/3774109896 learned_addr learned my addr 172.21.15.27:0/3774109896 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c43fff640 1 --2- 172.21.15.27:0/3774109896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3e30 0x5646af1add10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c4a59c640 1 -- 172.21.15.27:0/3774109896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3e30 msgr2=0x5646af1add10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c4a59c640 1 --2- 172.21.15.27:0/3774109896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3e30 0x5646af1add10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c4a59c640 1 -- 172.21.15.27:0/3774109896 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a8fa0 msgr2=0x5646af1b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c4ad9d640 1 --2- 172.21.15.27:0/3774109896 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a8fa0 0x5646af1b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c4a59c640 1 --2- 172.21.15.27:0/3774109896 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a8fa0 0x5646af1b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c4a59c640 1 -- 172.21.15.27:0/3774109896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646aeee56b0 con 0x5646af1a3440 2024-04-25T00:15:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c43fff640 1 --2- 172.21.15.27:0/3774109896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3e30 0x5646af1add10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.068+0000 7f9c4ad9d640 1 --2- 172.21.15.27:0/3774109896 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a8fa0 0x5646af1b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.069+0000 7f9c4a59c640 1 --2- 172.21.15.27:0/3774109896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a3440 0x5646af1ad7d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c09af90 tx=0x7f9c3c0036b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf9ed7ab2c49ebd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.069+0000 7f9c31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.069+0000 7f9c31ffb640 1 -- 172.21.15.27:0/3774109896 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c020050 con 0x5646af1a3440 2024-04-25T00:15:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.069+0000 7f9c31ffb640 1 -- 172.21.15.27:0/3774109896 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9c3c004260 con 0x5646af1a3440 2024-04-25T00:15:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.069+0000 7f9c53db5740 1 -- 172.21.15.27:0/3774109896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c34031970 con 0x5646af1a3440 2024-04-25T00:15:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.069+0000 7f9c31ffb640 1 -- 172.21.15.27:0/3774109896 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9c3c09add0 con 0x5646af1a3440 2024-04-25T00:15:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.069+0000 7f9c53db5740 10 client.15255 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.069+0000 7f9c53db5740 1 -- 172.21.15.27:0/3774109896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5646af1b7dc0 con 0x5646af1a3440 2024-04-25T00:15:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.069+0000 7f9c31ffb640 1 -- 172.21.15.27:0/3774109896 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c34031970 con 0x5646af1a3440 2024-04-25T00:15:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.069+0000 7f9c31ffb640 1 -- 172.21.15.27:0/3774109896 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9c3c097900 con 0x5646af1a3440 2024-04-25T00:15:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.069+0000 7f9c53db5740 10 client.15255 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.069+0000 7f9c53db5740 10 client.15255 resolve_mds: resolved name 'k' to daemon mds.k 2024-04-25T00:15:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.069+0000 7f9c53db5740 20 client.15255 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:10.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.069+0000 7f9c53db5740 1 --2- 172.21.15.27:0/3774109896 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5646af1ba4e0 0x5646af1bb950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.069+0000 7f9c53db5740 4 client.15255 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:15:10.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.069+0000 7f9c53db5740 1 -- 172.21.15.27:0/3774109896 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646af114440 con 0x5646af1ba4e0 2024-04-25T00:15:10.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.070+0000 7f9c43fff640 1 --2- 172.21.15.27:0/3774109896 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5646af1ba4e0 0x5646af1bb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.070+0000 7f9c43fff640 1 --2- 172.21.15.27:0/3774109896 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5646af1ba4e0 0x5646af1bb950 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6946cadc2846ebd3 server_cookie=bedf172b555b07c in_seq=0 out_seq=0 2024-04-25T00:15:10.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.070+0000 7f9c31ffb640 10 client.15255 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:15:10.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.071+0000 7f9c31ffb640 1 -- 172.21.15.27:0/3774109896 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5646af114440 con 0x5646af1ba4e0 2024-04-25T00:15:10.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.071+0000 7f9c31ffb640 10 client.15255 handle_command_reply: tid=0 2024-04-25T00:15:10.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.071+0000 7f9c53db5740 1 client.15255 shutdown 2024-04-25T00:15:10.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.071+0000 7f9c53db5740 1 -- 172.21.15.27:0/3774109896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a3440 msgr2=0x5646af1ad7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.071+0000 7f9c53db5740 1 --2- 172.21.15.27:0/3774109896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a3440 0x5646af1ad7d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c09af90 tx=0x7f9c3c0036b0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.071+0000 7f9c53db5740 1 -- 172.21.15.27:0/3774109896 shutdown_connections 2024-04-25T00:15:10.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.071+0000 7f9c53db5740 1 --2- 172.21.15.27:0/3774109896 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5646af1ba4e0 0x5646af1bb950 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.071+0000 7f9c53db5740 1 --2- 172.21.15.27:0/3774109896 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a8fa0 0x5646af1b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.071+0000 7f9c53db5740 1 --2- 172.21.15.27:0/3774109896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3e30 0x5646af1add10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.071+0000 7f9c31ffb640 0 client.15255 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:15:10.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.071+0000 7f9c53db5740 1 --2- 172.21.15.27:0/3774109896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a3440 0x5646af1ad7d0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.071+0000 7f9c53db5740 1 -- 172.21.15.27:0/3774109896 >> 172.21.15.27:0/3774109896 conn(0x5646af19eab0 msgr2=0x5646af19dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.071+0000 7f9c53db5740 1 -- 172.21.15.27:0/3774109896 shutdown_connections 2024-04-25T00:15:10.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.071+0000 7f9c53db5740 1 -- 172.21.15.27:0/3774109896 wait complete. 2024-04-25T00:15:10.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.071+0000 7f9c53db5740 20 client.15255 trim_cache size 0 max 16384 2024-04-25T00:15:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.089+0000 7f9c53db5740 1 Processor -- start 2024-04-25T00:15:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.089+0000 7f9c53db5740 1 -- start start 2024-04-25T00:15:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.089+0000 7f9c53db5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a3440 0x5646af1acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.089+0000 7f9c53db5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a3e30 0x5646af1b4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.089+0000 7f9c53db5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a8fa0 0x5646af1ad330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.090+0000 7f9c53db5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5646af1b9c90 con 0x5646af1a3e30 2024-04-25T00:15:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.090+0000 7f9c53db5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5646af1ba360 con 0x5646af1a3440 2024-04-25T00:15:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.090+0000 7f9c53db5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5646af1baa30 con 0x5646af1a8fa0 2024-04-25T00:15:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.090+0000 7f9c43fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a3e30 0x5646af1b4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.090+0000 7f9c43fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a3e30 0x5646af1b4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47624/0 (socket says 172.21.15.27:47624) 2024-04-25T00:15:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.090+0000 7f9c43fff640 1 -- 172.21.15.27:0/3974520519 learned_addr learned my addr 172.21.15.27:0/3974520519 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.090+0000 7f9c43fff640 1 -- 172.21.15.27:0/3974520519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a8fa0 msgr2=0x5646af1ad330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.090+0000 7f9c43fff640 1 --2- 172.21.15.27:0/3974520519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a8fa0 0x5646af1ad330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.090+0000 7f9c4a59c640 1 --2- 172.21.15.27:0/3974520519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a3440 0x5646af1acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.090+0000 7f9c43fff640 1 -- 172.21.15.27:0/3974520519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a3440 msgr2=0x5646af1acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.090+0000 7f9c43fff640 1 --2- 172.21.15.27:0/3974520519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a3440 0x5646af1acdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.090+0000 7f9c43fff640 1 -- 172.21.15.27:0/3974520519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646af1a4210 con 0x5646af1a3e30 2024-04-25T00:15:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.090+0000 7f9c4a59c640 1 --2- 172.21.15.27:0/3974520519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a3440 0x5646af1acdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.090+0000 7f9c43fff640 1 --2- 172.21.15.27:0/3974520519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a3e30 0x5646af1b4d10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9c34006010 tx=0x7f9c3402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30df8c6e857bd8df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.090+0000 7f9c33fff640 1 -- 172.21.15.27:0/3974520519 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c34050020 con 0x5646af1a3e30 2024-04-25T00:15:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.090+0000 7f9c33fff640 1 -- 172.21.15.27:0/3974520519 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c3404cdc0 con 0x5646af1a3e30 2024-04-25T00:15:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.090+0000 7f9c33fff640 1 -- 172.21.15.27:0/3974520519 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c34035040 con 0x5646af1a3e30 2024-04-25T00:15:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 1 -- 172.21.15.27:0/3974520519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a3e30 msgr2=0x5646af1b4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 1 --2- 172.21.15.27:0/3974520519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a3e30 0x5646af1b4d10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9c34006010 tx=0x7f9c3402a410 comp rx=0 tx=0).stop 2024-04-25T00:15:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 1 -- 172.21.15.27:0/3974520519 shutdown_connections 2024-04-25T00:15:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 1 --2- 172.21.15.27:0/3974520519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a8fa0 0x5646af1ad330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 1 --2- 172.21.15.27:0/3974520519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a3e30 0x5646af1b4d10 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 1 --2- 172.21.15.27:0/3974520519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a3440 0x5646af1acdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 1 -- 172.21.15.27:0/3974520519 >> 172.21.15.27:0/3974520519 conn(0x5646af19eab0 msgr2=0x5646af19ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 1 -- 172.21.15.27:0/3974520519 shutdown_connections 2024-04-25T00:15:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 1 -- 172.21.15.27:0/3974520519 wait complete. 2024-04-25T00:15:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 1 -- start start 2024-04-25T00:15:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 1 Processor -- start 2024-04-25T00:15:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a3440 0x5646af1b0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3e30 0x5646af1b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a8fa0 0x5646af1b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5646af1bb7d0 con 0x5646af1a8fa0 2024-04-25T00:15:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5646af1bb100 con 0x5646af1a3440 2024-04-25T00:15:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5646af1bbea0 con 0x5646af1a3e30 2024-04-25T00:15:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c53db5740 10 client.0 _ll_register_callbacks cb 0x7f9c3c0046e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c4ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a8fa0 0x5646af1b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c4ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a8fa0 0x5646af1b3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47628/0 (socket says 172.21.15.27:47628) 2024-04-25T00:15:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c4ad9d640 1 -- 172.21.15.27:0/563718369 learned_addr learned my addr 172.21.15.27:0/563718369 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.091+0000 7f9c43fff640 1 --2- 172.21.15.27:0/563718369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3e30 0x5646af1b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.092+0000 7f9c4a59c640 1 --2- 172.21.15.27:0/563718369 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a3440 0x5646af1b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.092+0000 7f9c4ad9d640 1 -- 172.21.15.27:0/563718369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3e30 msgr2=0x5646af1b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.092+0000 7f9c4ad9d640 1 --2- 172.21.15.27:0/563718369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3e30 0x5646af1b0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.092+0000 7f9c4ad9d640 1 -- 172.21.15.27:0/563718369 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a3440 msgr2=0x5646af1b0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.092+0000 7f9c4ad9d640 1 --2- 172.21.15.27:0/563718369 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a3440 0x5646af1b0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.092+0000 7f9c4ad9d640 1 -- 172.21.15.27:0/563718369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646af1b43b0 con 0x5646af1a8fa0 2024-04-25T00:15:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.092+0000 7f9c43fff640 1 --2- 172.21.15.27:0/563718369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3e30 0x5646af1b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.092+0000 7f9c4ad9d640 1 --2- 172.21.15.27:0/563718369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a8fa0 0x5646af1b3dc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9c44012ac0 tx=0x7f9c44004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80bbb26d4291aa19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.092+0000 7f9c31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.092+0000 7f9c31ffb640 1 -- 172.21.15.27:0/563718369 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c44003e40 con 0x5646af1a8fa0 2024-04-25T00:15:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.092+0000 7f9c31ffb640 1 -- 172.21.15.27:0/563718369 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9c4401e8f0 con 0x5646af1a8fa0 2024-04-25T00:15:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.092+0000 7f9c53db5740 1 -- 172.21.15.27:0/563718369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c3404cdc0 con 0x5646af1a8fa0 2024-04-25T00:15:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.092+0000 7f9c31ffb640 1 -- 172.21.15.27:0/563718369 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9c44031050 con 0x5646af1a8fa0 2024-04-25T00:15:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.092+0000 7f9c53db5740 10 client.15261 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.092+0000 7f9c53db5740 1 -- 172.21.15.27:0/563718369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5646af220930 con 0x5646af1a8fa0 2024-04-25T00:15:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.092+0000 7f9c31ffb640 1 -- 172.21.15.27:0/563718369 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c3404cdc0 con 0x5646af1a8fa0 2024-04-25T00:15:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.092+0000 7f9c31ffb640 1 -- 172.21.15.27:0/563718369 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9c4400f040 con 0x5646af1a8fa0 2024-04-25T00:15:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.093+0000 7f9c53db5740 10 client.15261 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.093+0000 7f9c53db5740 10 client.15261 resolve_mds: resolved name 'k' to daemon mds.k 2024-04-25T00:15:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.093+0000 7f9c53db5740 20 client.15261 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.093+0000 7f9c53db5740 1 --2- 172.21.15.27:0/563718369 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5646af2255b0 0x5646af2242c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.093+0000 7f9c53db5740 4 client.15261 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T00:15:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.093+0000 7f9c53db5740 1 -- 172.21.15.27:0/563718369 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "status"}) -- 0x5646af10e790 con 0x5646af2255b0 2024-04-25T00:15:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.093+0000 7f9c4a59c640 1 --2- 172.21.15.27:0/563718369 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5646af2255b0 0x5646af2242c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.093+0000 7f9c4a59c640 1 --2- 172.21.15.27:0/563718369 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5646af2255b0 0x5646af2242c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a61a9937201a2678 server_cookie=fd2381d4d7b75c5d in_seq=0 out_seq=0 2024-04-25T00:15:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.093+0000 7f9c31ffb640 10 client.15261 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:15:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.093+0000 7f9c31ffb640 1 -- 172.21.15.27:0/563718369 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5646af10e790 con 0x5646af2255b0 2024-04-25T00:15:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.093+0000 7f9c31ffb640 10 client.15261 handle_command_reply: tid=0 2024-04-25T00:15:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.093+0000 7f9c53db5740 1 client.15261 shutdown 2024-04-25T00:15:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c53db5740 1 -- 172.21.15.27:0/563718369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a8fa0 msgr2=0x5646af1b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c53db5740 1 --2- 172.21.15.27:0/563718369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a8fa0 0x5646af1b3dc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9c44012ac0 tx=0x7f9c44004ab0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c53db5740 1 -- 172.21.15.27:0/563718369 shutdown_connections 2024-04-25T00:15:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c53db5740 1 --2- 172.21.15.27:0/563718369 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5646af2255b0 0x5646af2242c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c53db5740 1 --2- 172.21.15.27:0/563718369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646af1a8fa0 0x5646af1b3dc0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c53db5740 1 --2- 172.21.15.27:0/563718369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646af1a3e30 0x5646af1b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c31ffb640 0 client.15261 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:15:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c53db5740 1 --2- 172.21.15.27:0/563718369 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646af1a3440 0x5646af1b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c53db5740 1 -- 172.21.15.27:0/563718369 >> 172.21.15.27:0/563718369 conn(0x5646af19eab0 msgr2=0x5646af1a19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c53db5740 1 -- 172.21.15.27:0/563718369 shutdown_connections 2024-04-25T00:15:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c53db5740 1 -- 172.21.15.27:0/563718369 wait complete. 2024-04-25T00:15:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c53db5740 20 client.15261 trim_cache size 0 max 16384 2024-04-25T00:15:10.102 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:15:10.103 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "518857f8-0297-11ef-bc93-c7b262605968", 2024-04-25T00:15:10.103 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 4, 2024-04-25T00:15:10.103 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24428, 2024-04-25T00:15:10.103 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T00:15:10.103 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T00:15:10.103 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T00:15:10.103 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 83.521980991000007, 2024-04-25T00:15:10.103 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 28, 2024-04-25T00:15:10.103 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 91, 2024-04-25T00:15:10.103 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T00:15:10.103 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 145.648561269 2024-04-25T00:15:10.103 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:15:10.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c50fc8640 1 -- 172.21.15.27:0/2057481190 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9c380a2c20 msgr2=0x7f9c380a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c50fc8640 1 --2- 172.21.15.27:0/2057481190 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9c380a2c20 0x7f9c380a50e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x5646af1a4690 tx=0x7f9c44009600 comp rx=0 tx=0).stop 2024-04-25T00:15:10.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c50fc8640 1 -- 172.21.15.27:0/2057481190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c4c1017a0 msgr2=0x7f9c4c197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c50fc8640 1 --2- 172.21.15.27:0/2057481190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c4c1017a0 0x7f9c4c197a90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c00d770 tx=0x7f9c3c00dc30 comp rx=0 tx=0).stop 2024-04-25T00:15:10.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c50fc8640 1 -- 172.21.15.27:0/2057481190 shutdown_connections 2024-04-25T00:15:10.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c50fc8640 1 --2- 172.21.15.27:0/2057481190 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f9c380a2c20 0x7f9c380a50e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c50fc8640 1 --2- 172.21.15.27:0/2057481190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c4c103350 0x7f9c4c19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c50fc8640 1 --2- 172.21.15.27:0/2057481190 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c4c102990 0x7f9c4c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c50fc8640 1 --2- 172.21.15.27:0/2057481190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c4c1017a0 0x7f9c4c197a90 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c50fc8640 1 -- 172.21.15.27:0/2057481190 >> 172.21.15.27:0/2057481190 conn(0x7f9c4c0fcd10 msgr2=0x7f9c4c105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c50fc8640 1 -- 172.21.15.27:0/2057481190 shutdown_connections 2024-04-25T00:15:10.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.094+0000 7f9c50fc8640 1 -- 172.21.15.27:0/2057481190 wait complete. 2024-04-25T00:15:10.106 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2024-04-25T00:15:10.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.209+0000 7f1663ada640 1 -- 172.21.15.27:0/4244661406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f165c100680 msgr2=0x7f165c100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.209+0000 7f1663ada640 1 --2- 172.21.15.27:0/4244661406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f165c100680 0x7f165c100aa0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1654009900 tx=0x7f165402a410 comp rx=0 tx=0).stop 2024-04-25T00:15:10.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.210+0000 7f1663ada640 1 -- 172.21.15.27:0/4244661406 shutdown_connections 2024-04-25T00:15:10.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.210+0000 7f1663ada640 1 --2- 172.21.15.27:0/4244661406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f165c102230 0x7f165c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.210+0000 7f1663ada640 1 --2- 172.21.15.27:0/4244661406 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f165c101870 0x7f165c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.210+0000 7f1663ada640 1 --2- 172.21.15.27:0/4244661406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f165c100680 0x7f165c100aa0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.210+0000 7f1663ada640 1 -- 172.21.15.27:0/4244661406 >> 172.21.15.27:0/4244661406 conn(0x7f165c0fbbf0 msgr2=0x7f165c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.210+0000 7f1663ada640 1 -- 172.21.15.27:0/4244661406 shutdown_connections 2024-04-25T00:15:10.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.210+0000 7f1663ada640 1 -- 172.21.15.27:0/4244661406 wait complete. 2024-04-25T00:15:10.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.210+0000 7f1663ada640 1 Processor -- start 2024-04-25T00:15:10.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.210+0000 7f1663ada640 1 -- start start 2024-04-25T00:15:10.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.210+0000 7f1663ada640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f165c100680 0x7f165c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.210+0000 7f1663ada640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f165c101870 0x7f165c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.210+0000 7f1663ada640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f165c102230 0x7f165c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.210+0000 7f1663ada640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f165c10c350 con 0x7f165c101870 2024-04-25T00:15:10.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.210+0000 7f1663ada640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f165c10c1d0 con 0x7f165c102230 2024-04-25T00:15:10.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.210+0000 7f1663ada640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f165c10c4d0 con 0x7f165c100680 2024-04-25T00:15:10.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.211+0000 7f1661075640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f165c101870 0x7f165c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.211+0000 7f1661075640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f165c101870 0x7f165c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47648/0 (socket says 172.21.15.27:47648) 2024-04-25T00:15:10.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.211+0000 7f1661075640 1 -- 172.21.15.27:0/2051647706 learned_addr learned my addr 172.21.15.27:0/2051647706 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.211+0000 7f1661876640 1 --2- 172.21.15.27:0/2051647706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f165c100680 0x7f165c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.211+0000 7f1662077640 1 --2- 172.21.15.27:0/2051647706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f165c102230 0x7f165c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.211+0000 7f1661075640 1 -- 172.21.15.27:0/2051647706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f165c100680 msgr2=0x7f165c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.211+0000 7f1661075640 1 --2- 172.21.15.27:0/2051647706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f165c100680 0x7f165c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.211+0000 7f1661075640 1 -- 172.21.15.27:0/2051647706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f165c102230 msgr2=0x7f165c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.211+0000 7f1661075640 1 --2- 172.21.15.27:0/2051647706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f165c102230 0x7f165c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.211+0000 7f1661075640 1 -- 172.21.15.27:0/2051647706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f165c19f650 con 0x7f165c101870 2024-04-25T00:15:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.211+0000 7f1661075640 1 --2- 172.21.15.27:0/2051647706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f165c101870 0x7f165c197f70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f164c00c920 tx=0x7f164c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.211+0000 7f1652ffd640 1 -- 172.21.15.27:0/2051647706 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f164c007c70 con 0x7f165c101870 2024-04-25T00:15:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.211+0000 7f1652ffd640 1 -- 172.21.15.27:0/2051647706 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f164c007e10 con 0x7f165c101870 2024-04-25T00:15:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.211+0000 7f1663ada640 1 -- 172.21.15.27:0/2051647706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f165c19f940 con 0x7f165c101870 2024-04-25T00:15:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.211+0000 7f1652ffd640 1 -- 172.21.15.27:0/2051647706 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f164c018890 con 0x7f165c101870 2024-04-25T00:15:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.211+0000 7f1663ada640 1 -- 172.21.15.27:0/2051647706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f165c19fee0 con 0x7f165c101870 2024-04-25T00:15:10.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.214+0000 7f1652ffd640 1 -- 172.21.15.27:0/2051647706 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f164c018a70 con 0x7f165c101870 2024-04-25T00:15:10.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.214+0000 7f16667ec740 1 Processor -- start 2024-04-25T00:15:10.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.214+0000 7f16667ec740 1 -- start start 2024-04-25T00:15:10.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.214+0000 7f16667ec740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9145f850 0x557f9145fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.214+0000 7f16667ec740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f91460240 0x557f91464d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.214+0000 7f16667ec740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f914653b0 0x557f91467840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.214+0000 7f16667ec740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557f913bef00 con 0x557f9145f850 2024-04-25T00:15:10.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.214+0000 7f16667ec740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557f91367750 con 0x557f914653b0 2024-04-25T00:15:10.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.214+0000 7f16667ec740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557f913baec0 con 0x557f91460240 2024-04-25T00:15:10.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.214+0000 7f1662077640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9145f850 0x557f9145fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.215+0000 7f1662077640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9145f850 0x557f9145fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47662/0 (socket says 172.21.15.27:47662) 2024-04-25T00:15:10.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.215+0000 7f1661876640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f91460240 0x557f91464d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.215+0000 7f1662077640 1 -- 172.21.15.27:0/4198881676 learned_addr learned my addr 172.21.15.27:0/4198881676 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.215+0000 7f1661075640 1 --2- 172.21.15.27:0/4198881676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f914653b0 0x557f91467840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.215+0000 7f1652ffd640 1 --2- 172.21.15.27:0/2051647706 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16480a2ca0 0x7f16480a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.215+0000 7f1662077640 1 -- 172.21.15.27:0/4198881676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f91460240 msgr2=0x557f91464d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.215+0000 7f1662077640 1 --2- 172.21.15.27:0/4198881676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f91460240 0x557f91464d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.215+0000 7f1662077640 1 -- 172.21.15.27:0/4198881676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f914653b0 msgr2=0x557f91467840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.215+0000 7f1662077640 1 --2- 172.21.15.27:0/4198881676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f914653b0 0x557f91467840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.215+0000 7f1662077640 1 -- 172.21.15.27:0/4198881676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f910d32d0 con 0x557f9145f850 2024-04-25T00:15:10.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.215+0000 7f1652ffd640 1 -- 172.21.15.27:0/2051647706 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f164c0cf540 con 0x7f165c101870 2024-04-25T00:15:10.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.215+0000 7f1662077640 1 --2- 172.21.15.27:0/2051647706 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16480a2ca0 0x7f16480a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.215+0000 7f1662077640 1 --2- 172.21.15.27:0/4198881676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9145f850 0x557f9145fc70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1658010370 tx=0x7f1658010770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4057ccb707241a7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.215+0000 7f1647fff640 1 -- 172.21.15.27:0/4198881676 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1658022070 con 0x557f9145f850 2024-04-25T00:15:10.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.215+0000 7f1647fff640 1 -- 172.21.15.27:0/4198881676 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16580113e0 con 0x557f9145f850 2024-04-25T00:15:10.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.215+0000 7f1647fff640 1 -- 172.21.15.27:0/4198881676 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1658016070 con 0x557f9145f850 2024-04-25T00:15:10.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.216+0000 7f16667ec740 1 -- 172.21.15.27:0/4198881676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9145f850 msgr2=0x557f9145fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.216+0000 7f1662077640 1 --2- 172.21.15.27:0/2051647706 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16480a2ca0 0x7f16480a5160 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f165801f000 tx=0x7f1658022000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.216+0000 7f16667ec740 1 --2- 172.21.15.27:0/4198881676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9145f850 0x557f9145fc70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1658010370 tx=0x7f1658010770 comp rx=0 tx=0).stop 2024-04-25T00:15:10.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.216+0000 7f16667ec740 1 -- 172.21.15.27:0/4198881676 shutdown_connections 2024-04-25T00:15:10.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.216+0000 7f16667ec740 1 --2- 172.21.15.27:0/4198881676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f914653b0 0x557f91467840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.216+0000 7f16667ec740 1 --2- 172.21.15.27:0/4198881676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f91460240 0x557f91464d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.216+0000 7f16667ec740 1 --2- 172.21.15.27:0/4198881676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9145f850 0x557f9145fc70 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.216+0000 7f16667ec740 1 -- 172.21.15.27:0/4198881676 >> 172.21.15.27:0/4198881676 conn(0x557f91456ab0 msgr2=0x557f91456eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.216+0000 7f16667ec740 1 -- 172.21.15.27:0/4198881676 shutdown_connections 2024-04-25T00:15:10.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.216+0000 7f16667ec740 1 -- 172.21.15.27:0/4198881676 wait complete. 2024-04-25T00:15:10.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.217+0000 7f16667ec740 1 -- start start 2024-04-25T00:15:10.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.217+0000 7f16667ec740 1 Processor -- start 2024-04-25T00:15:10.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.217+0000 7f16667ec740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9145f850 0x557f91469c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.217+0000 7f16667ec740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f91460240 0x557f9146a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.217+0000 7f16667ec740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f914653b0 0x557f914711f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.217+0000 7f16667ec740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557f91468360 con 0x557f9145f850 2024-04-25T00:15:10.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.217+0000 7f16667ec740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557f9137acf0 con 0x557f91460240 2024-04-25T00:15:10.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.217+0000 7f16667ec740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557f914684e0 con 0x557f914653b0 2024-04-25T00:15:10.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.217+0000 7f16667ec740 10 client.0 _ll_register_callbacks cb 0x557f9137a6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:10.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.217+0000 7f1661876640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9145f850 0x557f91469c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.217+0000 7f1661876640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9145f850 0x557f91469c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47670/0 (socket says 172.21.15.27:47670) 2024-04-25T00:15:10.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.217+0000 7f1661876640 1 -- 172.21.15.27:0/732598100 learned_addr learned my addr 172.21.15.27:0/732598100 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.217+0000 7f1661075640 1 --2- 172.21.15.27:0/732598100 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f91460240 0x557f9146a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.217+0000 7f1662077640 1 --2- 172.21.15.27:0/732598100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f914653b0 0x557f914711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.218+0000 7f1661876640 1 -- 172.21.15.27:0/732598100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f914653b0 msgr2=0x557f914711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.218+0000 7f1661876640 1 --2- 172.21.15.27:0/732598100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f914653b0 0x557f914711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.218+0000 7f1661876640 1 -- 172.21.15.27:0/732598100 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f91460240 msgr2=0x557f9146a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.218+0000 7f1661876640 1 --2- 172.21.15.27:0/732598100 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f91460240 0x557f9146a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.218+0000 7f1661876640 1 -- 172.21.15.27:0/732598100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f9119d6b0 con 0x557f9145f850 2024-04-25T00:15:10.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.218+0000 7f1661075640 1 --2- 172.21.15.27:0/732598100 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f91460240 0x557f9146a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:10.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.218+0000 7f1661876640 1 --2- 172.21.15.27:0/732598100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9145f850 0x557f91469c70 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1654009900 tx=0x7f16540045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9be64fefeb7d56d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.218+0000 7f16457fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:10.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.218+0000 7f16457fa640 1 -- 172.21.15.27:0/732598100 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1654035040 con 0x557f9145f850 2024-04-25T00:15:10.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.218+0000 7f16667ec740 1 -- 172.21.15.27:0/732598100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16580113e0 con 0x557f9145f850 2024-04-25T00:15:10.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.218+0000 7f16457fa640 1 -- 172.21.15.27:0/732598100 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1654004770 con 0x557f9145f850 2024-04-25T00:15:10.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.218+0000 7f16457fa640 1 -- 172.21.15.27:0/732598100 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f165404f050 con 0x557f9145f850 2024-04-25T00:15:10.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.218+0000 7f16667ec740 10 client.15273 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:10.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.218+0000 7f16667ec740 1 -- 172.21.15.27:0/732598100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557f91474290 con 0x557f9145f850 2024-04-25T00:15:10.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.218+0000 7f16457fa640 1 -- 172.21.15.27:0/732598100 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16580113e0 con 0x557f9145f850 2024-04-25T00:15:10.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.218+0000 7f16457fa640 1 -- 172.21.15.27:0/732598100 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1654031c30 con 0x557f9145f850 2024-04-25T00:15:10.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.218+0000 7f16667ec740 10 client.15273 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:10.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.219+0000 7f16667ec740 10 client.15273 resolve_mds: resolved name 'l' to daemon mds.l 2024-04-25T00:15:10.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.219+0000 7f16667ec740 20 client.15273 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:10.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.219+0000 7f16667ec740 1 --2- 172.21.15.27:0/732598100 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x557f914768f0 0x557f91477d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.219+0000 7f16667ec740 4 client.15273 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:15:10.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.219+0000 7f16667ec740 1 -- 172.21.15.27:0/732598100 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f913cc440 con 0x557f914768f0 2024-04-25T00:15:10.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.219+0000 7f1661075640 1 --2- 172.21.15.27:0/732598100 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x557f914768f0 0x557f91477d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.219+0000 7f1661075640 1 --2- 172.21.15.27:0/732598100 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x557f914768f0 0x557f91477d60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c06f498892cf70d6 server_cookie=5f8cfc0ac3bbe4ca in_seq=0 out_seq=0 2024-04-25T00:15:10.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.219+0000 7f16457fa640 10 client.15273 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:15:10.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.220+0000 7f16457fa640 1 -- 172.21.15.27:0/732598100 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557f913cc440 con 0x557f914768f0 2024-04-25T00:15:10.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.220+0000 7f16457fa640 10 client.15273 handle_command_reply: tid=0 2024-04-25T00:15:10.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.220+0000 7f16667ec740 1 client.15273 shutdown 2024-04-25T00:15:10.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.220+0000 7f16667ec740 1 -- 172.21.15.27:0/732598100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9145f850 msgr2=0x557f91469c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.220+0000 7f16667ec740 1 --2- 172.21.15.27:0/732598100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9145f850 0x557f91469c70 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1654009900 tx=0x7f16540045a0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.220+0000 7f16667ec740 1 -- 172.21.15.27:0/732598100 shutdown_connections 2024-04-25T00:15:10.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.220+0000 7f16667ec740 1 --2- 172.21.15.27:0/732598100 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x557f914768f0 0x557f91477d60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.220+0000 7f16667ec740 1 --2- 172.21.15.27:0/732598100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f914653b0 0x557f914711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.220+0000 7f16457fa640 0 client.15273 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:15:10.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.220+0000 7f16667ec740 1 --2- 172.21.15.27:0/732598100 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f91460240 0x557f9146a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.220+0000 7f16667ec740 1 --2- 172.21.15.27:0/732598100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9145f850 0x557f91469c70 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.220+0000 7f16667ec740 1 -- 172.21.15.27:0/732598100 >> 172.21.15.27:0/732598100 conn(0x557f91456ab0 msgr2=0x557f91455ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.220+0000 7f16667ec740 1 -- 172.21.15.27:0/732598100 shutdown_connections 2024-04-25T00:15:10.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.220+0000 7f16667ec740 1 -- 172.21.15.27:0/732598100 wait complete. 2024-04-25T00:15:10.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.220+0000 7f16667ec740 20 client.15273 trim_cache size 0 max 16384 2024-04-25T00:15:10.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f16667ec740 1 Processor -- start 2024-04-25T00:15:10.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f16667ec740 1 -- start start 2024-04-25T00:15:10.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f16667ec740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f9145f850 0x557f91451690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f16667ec740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f91460240 0x557f914711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f16667ec740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f914653b0 0x557f91460ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f16667ec740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557f914760a0 con 0x557f914653b0 2024-04-25T00:15:10.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f16667ec740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557f91476770 con 0x557f91460240 2024-04-25T00:15:10.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f16667ec740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557f91476e40 con 0x557f9145f850 2024-04-25T00:15:10.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f1662077640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f914653b0 0x557f91460ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f1662077640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f914653b0 0x557f91460ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47676/0 (socket says 172.21.15.27:47676) 2024-04-25T00:15:10.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f1662077640 1 -- 172.21.15.27:0/3986453873 learned_addr learned my addr 172.21.15.27:0/3986453873 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f1662077640 1 -- 172.21.15.27:0/3986453873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f9145f850 msgr2=0x557f91451690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f1662077640 1 --2- 172.21.15.27:0/3986453873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f9145f850 0x557f91451690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f1662077640 1 -- 172.21.15.27:0/3986453873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f91460240 msgr2=0x557f914711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f1661075640 1 --2- 172.21.15.27:0/3986453873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f91460240 0x557f914711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f1662077640 1 --2- 172.21.15.27:0/3986453873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f91460240 0x557f914711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f1662077640 1 -- 172.21.15.27:0/3986453873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f91464920 con 0x557f914653b0 2024-04-25T00:15:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.239+0000 7f1661075640 1 --2- 172.21.15.27:0/3986453873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f91460240 0x557f914711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.240+0000 7f1662077640 1 --2- 172.21.15.27:0/3986453873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f914653b0 0x557f91460ad0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1658010c70 tx=0x7f1658025000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f5a21632f260043 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.240+0000 7f16477fe640 1 -- 172.21.15.27:0/3986453873 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1658035070 con 0x557f914653b0 2024-04-25T00:15:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.240+0000 7f16477fe640 1 -- 172.21.15.27:0/3986453873 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1658012930 con 0x557f914653b0 2024-04-25T00:15:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.240+0000 7f16477fe640 1 -- 172.21.15.27:0/3986453873 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1658016040 con 0x557f914653b0 2024-04-25T00:15:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.240+0000 7f16667ec740 1 -- 172.21.15.27:0/3986453873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f914653b0 msgr2=0x557f91460ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.240+0000 7f16667ec740 1 --2- 172.21.15.27:0/3986453873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f914653b0 0x557f91460ad0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1658010c70 tx=0x7f1658025000 comp rx=0 tx=0).stop 2024-04-25T00:15:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.240+0000 7f16667ec740 1 -- 172.21.15.27:0/3986453873 shutdown_connections 2024-04-25T00:15:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.240+0000 7f16667ec740 1 --2- 172.21.15.27:0/3986453873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f914653b0 0x557f91460ad0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.240+0000 7f16667ec740 1 --2- 172.21.15.27:0/3986453873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f91460240 0x557f914711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.240+0000 7f16667ec740 1 --2- 172.21.15.27:0/3986453873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f9145f850 0x557f91451690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.240+0000 7f16667ec740 1 -- 172.21.15.27:0/3986453873 >> 172.21.15.27:0/3986453873 conn(0x557f91456ab0 msgr2=0x557f91456e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.240+0000 7f16667ec740 1 -- 172.21.15.27:0/3986453873 shutdown_connections 2024-04-25T00:15:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.241+0000 7f16667ec740 1 -- 172.21.15.27:0/3986453873 wait complete. 2024-04-25T00:15:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.241+0000 7f16667ec740 1 -- start start 2024-04-25T00:15:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.241+0000 7f16667ec740 1 Processor -- start 2024-04-25T00:15:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.241+0000 7f16667ec740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f9145f850 0x557f9146d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.241+0000 7f16667ec740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f91460240 0x557f9146d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.241+0000 7f16667ec740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f914653b0 0x557f91470360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.241+0000 7f16667ec740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557f91477be0 con 0x557f91460240 2024-04-25T00:15:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.241+0000 7f16667ec740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557f91477510 con 0x557f914653b0 2024-04-25T00:15:10.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.241+0000 7f16667ec740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557f914782b0 con 0x557f9145f850 2024-04-25T00:15:10.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.241+0000 7f16667ec740 10 client.0 _ll_register_callbacks cb 0x7f1654004230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:10.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.241+0000 7f1661075640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f91460240 0x557f9146d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.241+0000 7f1661075640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f91460240 0x557f9146d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47688/0 (socket says 172.21.15.27:47688) 2024-04-25T00:15:10.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.241+0000 7f1661075640 1 -- 172.21.15.27:0/3366717172 learned_addr learned my addr 172.21.15.27:0/3366717172 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f1662077640 1 --2- 172.21.15.27:0/3366717172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f914653b0 0x557f91470360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f1661876640 1 --2- 172.21.15.27:0/3366717172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f9145f850 0x557f9146d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f1661075640 1 -- 172.21.15.27:0/3366717172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f9145f850 msgr2=0x557f9146d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f1661075640 1 --2- 172.21.15.27:0/3366717172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f9145f850 0x557f9146d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f1661075640 1 -- 172.21.15.27:0/3366717172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f914653b0 msgr2=0x557f91470360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f1661075640 1 --2- 172.21.15.27:0/3366717172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f914653b0 0x557f91470360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f1661075640 1 -- 172.21.15.27:0/3366717172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f91470950 con 0x557f91460240 2024-04-25T00:15:10.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f1661075640 1 --2- 172.21.15.27:0/3366717172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f91460240 0x557f9146d880 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f164c007590 tx=0x7f164c0d5110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79079bc160e8ced3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f16457fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f16457fa640 1 -- 172.21.15.27:0/3366717172 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f164c09b8e0 con 0x557f91460240 2024-04-25T00:15:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f16457fa640 1 -- 172.21.15.27:0/3366717172 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f164c0cf2c0 con 0x557f91460240 2024-04-25T00:15:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f16667ec740 1 -- 172.21.15.27:0/3366717172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1658012930 con 0x557f91460240 2024-04-25T00:15:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f16457fa640 1 -- 172.21.15.27:0/3366717172 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f164c09fc90 con 0x557f91460240 2024-04-25T00:15:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f16667ec740 10 client.15279 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f16667ec740 1 -- 172.21.15.27:0/3366717172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557f914dcdf0 con 0x557f91460240 2024-04-25T00:15:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f16457fa640 1 -- 172.21.15.27:0/3366717172 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1658012930 con 0x557f91460240 2024-04-25T00:15:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f16457fa640 1 -- 172.21.15.27:0/3366717172 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f164c097f50 con 0x557f91460240 2024-04-25T00:15:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.242+0000 7f16667ec740 10 client.15279 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.243+0000 7f16667ec740 10 client.15279 resolve_mds: resolved name 'l' to daemon mds.l 2024-04-25T00:15:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.243+0000 7f16667ec740 20 client.15279 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.243+0000 7f16667ec740 1 --2- 172.21.15.27:0/3366717172 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x557f914e19b0 0x557f914dddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.243+0000 7f16667ec740 4 client.15279 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T00:15:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.243+0000 7f16667ec740 1 -- 172.21.15.27:0/3366717172 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "status"}) -- 0x557f913c6790 con 0x557f914e19b0 2024-04-25T00:15:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.243+0000 7f1661876640 1 --2- 172.21.15.27:0/3366717172 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x557f914e19b0 0x557f914dddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.243+0000 7f16457fa640 10 client.15279 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:15:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.243+0000 7f1661876640 1 --2- 172.21.15.27:0/3366717172 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x557f914e19b0 0x557f914dddf0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=50ce51286f2c88b1 server_cookie=b32a4c70ef5ab0a2 in_seq=0 out_seq=0 2024-04-25T00:15:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.243+0000 7f16457fa640 1 -- 172.21.15.27:0/3366717172 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557f913c6790 con 0x557f914e19b0 2024-04-25T00:15:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.243+0000 7f16457fa640 10 client.15279 handle_command_reply: tid=0 2024-04-25T00:15:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.243+0000 7f16667ec740 1 client.15279 shutdown 2024-04-25T00:15:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.243+0000 7f16667ec740 1 -- 172.21.15.27:0/3366717172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f91460240 msgr2=0x557f9146d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f16667ec740 1 --2- 172.21.15.27:0/3366717172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f91460240 0x557f9146d880 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f164c007590 tx=0x7f164c0d5110 comp rx=0 tx=0).stop 2024-04-25T00:15:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f16667ec740 1 -- 172.21.15.27:0/3366717172 shutdown_connections 2024-04-25T00:15:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f16667ec740 1 --2- 172.21.15.27:0/3366717172 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x557f914e19b0 0x557f914dddf0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f16667ec740 1 --2- 172.21.15.27:0/3366717172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557f914653b0 0x557f91470360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f16667ec740 1 --2- 172.21.15.27:0/3366717172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f91460240 0x557f9146d880 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f16667ec740 1 --2- 172.21.15.27:0/3366717172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557f9145f850 0x557f9146d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f16457fa640 0 client.15279 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:15:10.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f16667ec740 1 -- 172.21.15.27:0/3366717172 >> 172.21.15.27:0/3366717172 conn(0x557f91456ab0 msgr2=0x557f91451690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f16667ec740 1 -- 172.21.15.27:0/3366717172 shutdown_connections 2024-04-25T00:15:10.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f16667ec740 1 -- 172.21.15.27:0/3366717172 wait complete. 2024-04-25T00:15:10.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f16667ec740 20 client.15279 trim_cache size 0 max 16384 2024-04-25T00:15:10.251 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:15:10.252 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "518857f8-0297-11ef-bc93-c7b262605968", 2024-04-25T00:15:10.252 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 3, 2024-04-25T00:15:10.252 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24430, 2024-04-25T00:15:10.252 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T00:15:10.252 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T00:15:10.252 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T00:15:10.252 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 84.681269557999997, 2024-04-25T00:15:10.252 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 28, 2024-04-25T00:15:10.252 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 91, 2024-04-25T00:15:10.252 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T00:15:10.252 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 148.696286585 2024-04-25T00:15:10.252 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:15:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f1663ada640 1 -- 172.21.15.27:0/2051647706 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16480a2ca0 msgr2=0x7f16480a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f1663ada640 1 --2- 172.21.15.27:0/2051647706 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16480a2ca0 0x7f16480a5160 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f165801f000 tx=0x7f1658022000 comp rx=0 tx=0).stop 2024-04-25T00:15:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f1663ada640 1 -- 172.21.15.27:0/2051647706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f165c101870 msgr2=0x7f165c197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f1663ada640 1 --2- 172.21.15.27:0/2051647706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f165c101870 0x7f165c197f70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f164c00c920 tx=0x7f164c00cde0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f1663ada640 1 -- 172.21.15.27:0/2051647706 shutdown_connections 2024-04-25T00:15:10.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f1663ada640 1 --2- 172.21.15.27:0/2051647706 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f16480a2ca0 0x7f16480a5160 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f1663ada640 1 --2- 172.21.15.27:0/2051647706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f165c102230 0x7f165c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f1663ada640 1 --2- 172.21.15.27:0/2051647706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f165c101870 0x7f165c197f70 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f1663ada640 1 --2- 172.21.15.27:0/2051647706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f165c100680 0x7f165c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f1663ada640 1 -- 172.21.15.27:0/2051647706 >> 172.21.15.27:0/2051647706 conn(0x7f165c0fbbf0 msgr2=0x7f165c0fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f1663ada640 1 -- 172.21.15.27:0/2051647706 shutdown_connections 2024-04-25T00:15:10.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.244+0000 7f1663ada640 1 -- 172.21.15.27:0/2051647706 wait complete. 2024-04-25T00:15:10.255 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T00:15:10.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.358+0000 7f61443d5640 1 -- 172.21.15.27:0/529019118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f613c102200 msgr2=0x7f613c1045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.358+0000 7f61443d5640 1 --2- 172.21.15.27:0/529019118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f613c102200 0x7f613c1045f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f612c009ca0 tx=0x7f612c02a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.358+0000 7f61443d5640 1 -- 172.21.15.27:0/529019118 shutdown_connections 2024-04-25T00:15:10.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.358+0000 7f61443d5640 1 --2- 172.21.15.27:0/529019118 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f613c107460 0x7f613c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.358+0000 7f61443d5640 1 --2- 172.21.15.27:0/529019118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f613c104b30 0x7f613c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.358+0000 7f61443d5640 1 --2- 172.21.15.27:0/529019118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f613c102200 0x7f613c1045f0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.358+0000 7f61443d5640 1 -- 172.21.15.27:0/529019118 >> 172.21.15.27:0/529019118 conn(0x7f613c0fbbf0 msgr2=0x7f613c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.358+0000 7f61443d5640 1 -- 172.21.15.27:0/529019118 shutdown_connections 2024-04-25T00:15:10.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.358+0000 7f61443d5640 1 -- 172.21.15.27:0/529019118 wait complete. 2024-04-25T00:15:10.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.359+0000 7f61443d5640 1 Processor -- start 2024-04-25T00:15:10.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.359+0000 7f61443d5640 1 -- start start 2024-04-25T00:15:10.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.359+0000 7f61443d5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f613c102200 0x7f613c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.359+0000 7f61443d5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f613c104b30 0x7f613c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.359+0000 7f61443d5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f613c107460 0x7f613c19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.359+0000 7f61443d5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f613c10d430 con 0x7f613c102200 2024-04-25T00:15:10.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.359+0000 7f61443d5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f613c10d2b0 con 0x7f613c107460 2024-04-25T00:15:10.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.359+0000 7f61443d5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f613c10d5b0 con 0x7f613c104b30 2024-04-25T00:15:10.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.359+0000 7f6142171640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f613c102200 0x7f613c197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.359+0000 7f6142171640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f613c102200 0x7f613c197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47702/0 (socket says 172.21.15.27:47702) 2024-04-25T00:15:10.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.359+0000 7f6142171640 1 -- 172.21.15.27:0/3948103473 learned_addr learned my addr 172.21.15.27:0/3948103473 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.359+0000 7f6141970640 1 --2- 172.21.15.27:0/3948103473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f613c104b30 0x7f613c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.359+0000 7f6142171640 1 -- 172.21.15.27:0/3948103473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f613c104b30 msgr2=0x7f613c1980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.359+0000 7f6142171640 1 --2- 172.21.15.27:0/3948103473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f613c104b30 0x7f613c1980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.359+0000 7f6142171640 1 -- 172.21.15.27:0/3948103473 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f613c107460 msgr2=0x7f613c19f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.359+0000 7f6142972640 1 --2- 172.21.15.27:0/3948103473 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f613c107460 0x7f613c19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.359+0000 7f6142171640 1 --2- 172.21.15.27:0/3948103473 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f613c107460 0x7f613c19f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.360+0000 7f6142171640 1 -- 172.21.15.27:0/3948103473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f613c19f790 con 0x7f613c102200 2024-04-25T00:15:10.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.360+0000 7f6142972640 1 --2- 172.21.15.27:0/3948103473 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f613c107460 0x7f613c19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:10.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.360+0000 7f6142171640 1 --2- 172.21.15.27:0/3948103473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f613c102200 0x7f613c197b70 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f612c031a40 tx=0x7f612c031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.360+0000 7f613b7fe640 1 -- 172.21.15.27:0/3948103473 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c02fa20 con 0x7f613c102200 2024-04-25T00:15:10.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.360+0000 7f61443d5640 1 -- 172.21.15.27:0/3948103473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f613c19fa20 con 0x7f613c102200 2024-04-25T00:15:10.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.360+0000 7f613b7fe640 1 -- 172.21.15.27:0/3948103473 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f612c02fbc0 con 0x7f613c102200 2024-04-25T00:15:10.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.360+0000 7f61443d5640 1 -- 172.21.15.27:0/3948103473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f613c19ff70 con 0x7f613c102200 2024-04-25T00:15:10.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.360+0000 7f613b7fe640 1 -- 172.21.15.27:0/3948103473 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c03a720 con 0x7f613c102200 2024-04-25T00:15:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.362+0000 7f613b7fe640 1 -- 172.21.15.27:0/3948103473 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f612c03a8c0 con 0x7f613c102200 2024-04-25T00:15:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.362+0000 7f61471bf740 1 Processor -- start 2024-04-25T00:15:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f61471bf740 1 -- start start 2024-04-25T00:15:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f61471bf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4440 0x55baf89e4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f61471bf740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e4e30 0x55baf89e9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f61471bf740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e9fa0 0x55baf89ec430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f61471bf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55baf8947f00 con 0x55baf89e4440 2024-04-25T00:15:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f61471bf740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55baf88f0750 con 0x55baf89e4e30 2024-04-25T00:15:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f61471bf740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55baf8943ec0 con 0x55baf89e9fa0 2024-04-25T00:15:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f6142972640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4440 0x55baf89e4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f6142972640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4440 0x55baf89e4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47712/0 (socket says 172.21.15.27:47712) 2024-04-25T00:15:10.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f6142972640 1 -- 172.21.15.27:0/3654650830 learned_addr learned my addr 172.21.15.27:0/3654650830 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f6142171640 1 --2- 172.21.15.27:0/3654650830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e4e30 0x55baf89e9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f613b7fe640 1 --2- 172.21.15.27:0/3948103473 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f61240a2c20 0x7f61240a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f6141970640 1 --2- 172.21.15.27:0/3654650830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e9fa0 0x55baf89ec430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f6142972640 1 -- 172.21.15.27:0/3654650830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e9fa0 msgr2=0x55baf89ec430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f6142972640 1 --2- 172.21.15.27:0/3654650830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e9fa0 0x55baf89ec430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f6142972640 1 -- 172.21.15.27:0/3654650830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e4e30 msgr2=0x55baf89e9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f6142972640 1 --2- 172.21.15.27:0/3654650830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e4e30 0x55baf89e9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f6142972640 1 -- 172.21.15.27:0/3654650830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55baf865c2d0 con 0x55baf89e4440 2024-04-25T00:15:10.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f6142171640 1 --2- 172.21.15.27:0/3654650830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e4e30 0x55baf89e9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:10.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.363+0000 7f613b7fe640 1 -- 172.21.15.27:0/3948103473 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f612c0f0a60 con 0x7f613c102200 2024-04-25T00:15:10.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.364+0000 7f6142972640 1 --2- 172.21.15.27:0/3948103473 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f61240a2c20 0x7f61240a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.364+0000 7f6142972640 1 --2- 172.21.15.27:0/3654650830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4440 0x55baf89e4860 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6134005fd0 tx=0x7f613400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e90d6ea540d438 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.364+0000 7f6138bfe640 1 -- 172.21.15.27:0/3654650830 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613401a410 con 0x55baf89e4440 2024-04-25T00:15:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.364+0000 7f6138bfe640 1 -- 172.21.15.27:0/3654650830 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f613401aa90 con 0x55baf89e4440 2024-04-25T00:15:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.364+0000 7f6138bfe640 1 -- 172.21.15.27:0/3654650830 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6134012830 con 0x55baf89e4440 2024-04-25T00:15:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f61471bf740 1 -- 172.21.15.27:0/3654650830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4440 msgr2=0x55baf89e4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f6142972640 1 --2- 172.21.15.27:0/3948103473 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f61240a2c20 0x7f61240a50e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f613401f000 tx=0x7f6134020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f61471bf740 1 --2- 172.21.15.27:0/3654650830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4440 0x55baf89e4860 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6134005fd0 tx=0x7f613400e6c0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f61471bf740 1 -- 172.21.15.27:0/3654650830 shutdown_connections 2024-04-25T00:15:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f61471bf740 1 --2- 172.21.15.27:0/3654650830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e9fa0 0x55baf89ec430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f61471bf740 1 --2- 172.21.15.27:0/3654650830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e4e30 0x55baf89e9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f61471bf740 1 --2- 172.21.15.27:0/3654650830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4440 0x55baf89e4860 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f61471bf740 1 -- 172.21.15.27:0/3654650830 >> 172.21.15.27:0/3654650830 conn(0x55baf89dfab0 msgr2=0x55baf89dfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f61471bf740 1 -- 172.21.15.27:0/3654650830 shutdown_connections 2024-04-25T00:15:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f61471bf740 1 -- 172.21.15.27:0/3654650830 wait complete. 2024-04-25T00:15:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f61471bf740 1 -- start start 2024-04-25T00:15:10.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f61471bf740 1 Processor -- start 2024-04-25T00:15:10.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f61471bf740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4440 0x55baf89ee7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f61471bf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4e30 0x55baf89eed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f61471bf740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e9fa0 0x55baf89f5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f61471bf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55baf89ecf50 con 0x55baf89e4e30 2024-04-25T00:15:10.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f61471bf740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55baf8903cf0 con 0x55baf89e9fa0 2024-04-25T00:15:10.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.365+0000 7f61471bf740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55baf89ed0d0 con 0x55baf89e4440 2024-04-25T00:15:10.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f61471bf740 10 client.0 _ll_register_callbacks cb 0x55baf89036e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:10.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f6141970640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4e30 0x55baf89eed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f6142171640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4440 0x55baf89ee7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f6141970640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4e30 0x55baf89eed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47714/0 (socket says 172.21.15.27:47714) 2024-04-25T00:15:10.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f6141970640 1 -- 172.21.15.27:0/3389622075 learned_addr learned my addr 172.21.15.27:0/3389622075 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f6142972640 1 --2- 172.21.15.27:0/3389622075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e9fa0 0x55baf89f5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f6141970640 1 -- 172.21.15.27:0/3389622075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4440 msgr2=0x55baf89ee7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f6141970640 1 --2- 172.21.15.27:0/3389622075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4440 0x55baf89ee7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f6141970640 1 -- 172.21.15.27:0/3389622075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e9fa0 msgr2=0x55baf89f5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f6141970640 1 --2- 172.21.15.27:0/3389622075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e9fa0 0x55baf89f5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f6141970640 1 -- 172.21.15.27:0/3389622075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55baf87266b0 con 0x55baf89e4e30 2024-04-25T00:15:10.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f6142171640 1 --2- 172.21.15.27:0/3389622075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4440 0x55baf89ee7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:10.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f6141970640 1 --2- 172.21.15.27:0/3389622075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4e30 0x55baf89eed30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6130013710 tx=0x7f6130013bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6554e0e1b9d2663 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f6121ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:10.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f6121ffb640 1 -- 172.21.15.27:0/3389622075 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6130004460 con 0x55baf89e4e30 2024-04-25T00:15:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f6121ffb640 1 -- 172.21.15.27:0/3389622075 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6130011960 con 0x55baf89e4e30 2024-04-25T00:15:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f61471bf740 1 -- 172.21.15.27:0/3389622075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f613401aa90 con 0x55baf89e4e30 2024-04-25T00:15:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f6121ffb640 1 -- 172.21.15.27:0/3389622075 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6130011b60 con 0x55baf89e4e30 2024-04-25T00:15:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f61471bf740 10 client.15291 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.366+0000 7f61471bf740 1 -- 172.21.15.27:0/3389622075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55baf89f8e20 con 0x55baf89e4e30 2024-04-25T00:15:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.367+0000 7f6121ffb640 1 -- 172.21.15.27:0/3389622075 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f613401aa90 con 0x55baf89e4e30 2024-04-25T00:15:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.367+0000 7f6121ffb640 1 -- 172.21.15.27:0/3389622075 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f613001a070 con 0x55baf89e4e30 2024-04-25T00:15:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.367+0000 7f61471bf740 10 client.15291 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.367+0000 7f61471bf740 10 client.15291 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T00:15:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.367+0000 7f61471bf740 20 client.15291 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.367+0000 7f61471bf740 1 --2- 172.21.15.27:0/3389622075 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55baf89fb530 0x55baf89fc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.367+0000 7f61471bf740 4 client.15291 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:15:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.367+0000 7f61471bf740 1 -- 172.21.15.27:0/3389622075 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55baf8955440 con 0x55baf89fb530 2024-04-25T00:15:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.367+0000 7f6142171640 1 --2- 172.21.15.27:0/3389622075 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55baf89fb530 0x55baf89fc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.368+0000 7f6121ffb640 10 client.15291 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:15:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.368+0000 7f6142171640 1 --2- 172.21.15.27:0/3389622075 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55baf89fb530 0x55baf89fc9a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ade761322ba16e06 server_cookie=d2d865e8aeb6d778 in_seq=0 out_seq=0 2024-04-25T00:15:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.368+0000 7f6121ffb640 1 -- 172.21.15.27:0/3389622075 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55baf8955440 con 0x55baf89fb530 2024-04-25T00:15:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.368+0000 7f6121ffb640 10 client.15291 handle_command_reply: tid=0 2024-04-25T00:15:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.369+0000 7f61471bf740 1 client.15291 shutdown 2024-04-25T00:15:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.369+0000 7f61471bf740 1 -- 172.21.15.27:0/3389622075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4e30 msgr2=0x55baf89eed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.369+0000 7f61471bf740 1 --2- 172.21.15.27:0/3389622075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4e30 0x55baf89eed30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6130013710 tx=0x7f6130013bd0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.369+0000 7f61471bf740 1 -- 172.21.15.27:0/3389622075 shutdown_connections 2024-04-25T00:15:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.369+0000 7f61471bf740 1 --2- 172.21.15.27:0/3389622075 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55baf89fb530 0x55baf89fc9a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.369+0000 7f61471bf740 1 --2- 172.21.15.27:0/3389622075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e9fa0 0x55baf89f5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.369+0000 7f61471bf740 1 --2- 172.21.15.27:0/3389622075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4e30 0x55baf89eed30 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.369+0000 7f6121ffb640 0 client.15291 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:15:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.369+0000 7f61471bf740 1 --2- 172.21.15.27:0/3389622075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4440 0x55baf89ee7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.369+0000 7f61471bf740 1 -- 172.21.15.27:0/3389622075 >> 172.21.15.27:0/3389622075 conn(0x55baf89dfab0 msgr2=0x55baf89deae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.369+0000 7f61471bf740 1 -- 172.21.15.27:0/3389622075 shutdown_connections 2024-04-25T00:15:10.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.369+0000 7f61471bf740 1 -- 172.21.15.27:0/3389622075 wait complete. 2024-04-25T00:15:10.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.369+0000 7f61471bf740 20 client.15291 trim_cache size 0 max 16384 2024-04-25T00:15:10.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f61471bf740 1 Processor -- start 2024-04-25T00:15:10.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f61471bf740 1 -- start start 2024-04-25T00:15:10.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f61471bf740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e4440 0x55baf89e9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f61471bf740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4e30 0x55baf89f5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f61471bf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e9fa0 0x55baf89e56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f61471bf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55baf89face0 con 0x55baf89e9fa0 2024-04-25T00:15:10.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f61471bf740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55baf89fb3b0 con 0x55baf89e4440 2024-04-25T00:15:10.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f61471bf740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55baf89fba80 con 0x55baf89e4e30 2024-04-25T00:15:10.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f6142972640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e9fa0 0x55baf89e56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f6142972640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e9fa0 0x55baf89e56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47716/0 (socket says 172.21.15.27:47716) 2024-04-25T00:15:10.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f6142972640 1 -- 172.21.15.27:0/3791795242 learned_addr learned my addr 172.21.15.27:0/3791795242 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f6142171640 1 --2- 172.21.15.27:0/3791795242 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e4440 0x55baf89e9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f6141970640 1 --2- 172.21.15.27:0/3791795242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4e30 0x55baf89f5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f6142972640 1 -- 172.21.15.27:0/3791795242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4e30 msgr2=0x55baf89f5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f6142972640 1 --2- 172.21.15.27:0/3791795242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4e30 0x55baf89f5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f6142972640 1 -- 172.21.15.27:0/3791795242 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e4440 msgr2=0x55baf89e9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f6142972640 1 --2- 172.21.15.27:0/3791795242 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e4440 0x55baf89e9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f6142972640 1 -- 172.21.15.27:0/3791795242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55baf89e94e0 con 0x55baf89e9fa0 2024-04-25T00:15:10.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.387+0000 7f6141970640 1 --2- 172.21.15.27:0/3791795242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4e30 0x55baf89f5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:10.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.388+0000 7f6142972640 1 --2- 172.21.15.27:0/3791795242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e9fa0 0x55baf89e56c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x55baf89e5690 tx=0x7f6134011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9602b652c0179269 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.388+0000 7f6123fff640 1 -- 172.21.15.27:0/3791795242 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6134036070 con 0x55baf89e9fa0 2024-04-25T00:15:10.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.388+0000 7f6123fff640 1 -- 172.21.15.27:0/3791795242 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6134009440 con 0x55baf89e9fa0 2024-04-25T00:15:10.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.388+0000 7f6123fff640 1 -- 172.21.15.27:0/3791795242 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6134004be0 con 0x55baf89e9fa0 2024-04-25T00:15:10.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.388+0000 7f61471bf740 1 -- 172.21.15.27:0/3791795242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e9fa0 msgr2=0x55baf89e56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.388+0000 7f61471bf740 1 --2- 172.21.15.27:0/3791795242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e9fa0 0x55baf89e56c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x55baf89e5690 tx=0x7f6134011000 comp rx=0 tx=0).stop 2024-04-25T00:15:10.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.388+0000 7f61471bf740 1 -- 172.21.15.27:0/3791795242 shutdown_connections 2024-04-25T00:15:10.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.388+0000 7f61471bf740 1 --2- 172.21.15.27:0/3791795242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e9fa0 0x55baf89e56c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.388+0000 7f61471bf740 1 --2- 172.21.15.27:0/3791795242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4e30 0x55baf89f5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.388+0000 7f61471bf740 1 --2- 172.21.15.27:0/3791795242 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e4440 0x55baf89e9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.388+0000 7f61471bf740 1 -- 172.21.15.27:0/3791795242 >> 172.21.15.27:0/3791795242 conn(0x55baf89dfab0 msgr2=0x55baf89da690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.388+0000 7f61471bf740 1 -- 172.21.15.27:0/3791795242 shutdown_connections 2024-04-25T00:15:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.388+0000 7f61471bf740 1 -- 172.21.15.27:0/3791795242 wait complete. 2024-04-25T00:15:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f61471bf740 1 -- start start 2024-04-25T00:15:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f61471bf740 1 Processor -- start 2024-04-25T00:15:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f61471bf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4440 0x55baf89f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f61471bf740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4e30 0x55baf8a5d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f61471bf740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e9fa0 0x55baf8a5fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f61471bf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55baf89fc820 con 0x55baf89e4440 2024-04-25T00:15:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f61471bf740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55baf89fc150 con 0x55baf89e9fa0 2024-04-25T00:15:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f61471bf740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55baf89fcef0 con 0x55baf89e4e30 2024-04-25T00:15:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f61471bf740 10 client.0 _ll_register_callbacks cb 0x7f6130009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f6142171640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4440 0x55baf89f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f6142171640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4440 0x55baf89f3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47722/0 (socket says 172.21.15.27:47722) 2024-04-25T00:15:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f6142171640 1 -- 172.21.15.27:0/2614229438 learned_addr learned my addr 172.21.15.27:0/2614229438 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f6141970640 1 --2- 172.21.15.27:0/2614229438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4e30 0x55baf8a5d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f6142171640 1 -- 172.21.15.27:0/2614229438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4e30 msgr2=0x55baf8a5d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f6142171640 1 --2- 172.21.15.27:0/2614229438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4e30 0x55baf8a5d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f6142171640 1 -- 172.21.15.27:0/2614229438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e9fa0 msgr2=0x55baf8a5fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f6142171640 1 --2- 172.21.15.27:0/2614229438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e9fa0 0x55baf8a5fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f6142171640 1 -- 172.21.15.27:0/2614229438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55baf8a60050 con 0x55baf89e4440 2024-04-25T00:15:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f6141970640 1 --2- 172.21.15.27:0/2614229438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4e30 0x55baf8a5d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f6142171640 1 --2- 172.21.15.27:0/2614229438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4440 0x55baf89f3d20 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f612c0b98e0 tx=0x7f612c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae3db889f35504f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.389+0000 7f6121ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.390+0000 7f6121ffb640 1 -- 172.21.15.27:0/2614229438 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c0c5b80 con 0x55baf89e4440 2024-04-25T00:15:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.390+0000 7f6121ffb640 1 -- 172.21.15.27:0/2614229438 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f612c0ee300 con 0x55baf89e4440 2024-04-25T00:15:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.390+0000 7f61471bf740 1 -- 172.21.15.27:0/2614229438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6134009440 con 0x55baf89e4440 2024-04-25T00:15:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.390+0000 7f6121ffb640 1 -- 172.21.15.27:0/2614229438 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f612c0c12c0 con 0x55baf89e4440 2024-04-25T00:15:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.390+0000 7f61471bf740 10 client.15297 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.390+0000 7f61471bf740 1 -- 172.21.15.27:0/2614229438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55baf8a612c0 con 0x55baf89e4440 2024-04-25T00:15:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.390+0000 7f6121ffb640 1 -- 172.21.15.27:0/2614229438 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6134009440 con 0x55baf89e4440 2024-04-25T00:15:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.390+0000 7f6121ffb640 1 -- 172.21.15.27:0/2614229438 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f612c03abb0 con 0x55baf89e4440 2024-04-25T00:15:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.390+0000 7f61471bf740 10 client.15297 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.390+0000 7f61471bf740 10 client.15297 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T00:15:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.390+0000 7f61471bf740 20 client.15297 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.390+0000 7f61471bf740 1 --2- 172.21.15.27:0/2614229438 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55baf8a65f00 0x55baf8a64c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.390+0000 7f61471bf740 4 client.15297 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T00:15:10.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.390+0000 7f61471bf740 1 -- 172.21.15.27:0/2614229438 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "status"}) -- 0x55baf894f790 con 0x55baf8a65f00 2024-04-25T00:15:10.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.390+0000 7f6141970640 1 --2- 172.21.15.27:0/2614229438 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55baf8a65f00 0x55baf8a64c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.391+0000 7f6121ffb640 10 client.15297 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:15:10.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.391+0000 7f6141970640 1 --2- 172.21.15.27:0/2614229438 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55baf8a65f00 0x55baf8a64c10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d04c7c07599716d2 server_cookie=ea0152d7d1c9faf5 in_seq=0 out_seq=0 2024-04-25T00:15:10.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.391+0000 7f6121ffb640 1 -- 172.21.15.27:0/2614229438 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55baf894f790 con 0x55baf8a65f00 2024-04-25T00:15:10.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.391+0000 7f6121ffb640 10 client.15297 handle_command_reply: tid=0 2024-04-25T00:15:10.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.391+0000 7f61471bf740 1 client.15297 shutdown 2024-04-25T00:15:10.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.391+0000 7f61471bf740 1 -- 172.21.15.27:0/2614229438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4440 msgr2=0x55baf89f3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.391+0000 7f61471bf740 1 --2- 172.21.15.27:0/2614229438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4440 0x55baf89f3d20 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f612c0b98e0 tx=0x7f612c0095c0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.391+0000 7f61471bf740 1 -- 172.21.15.27:0/2614229438 shutdown_connections 2024-04-25T00:15:10.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.391+0000 7f61471bf740 1 --2- 172.21.15.27:0/2614229438 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55baf8a65f00 0x55baf8a64c10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.391+0000 7f61471bf740 1 --2- 172.21.15.27:0/2614229438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55baf89e9fa0 0x55baf8a5fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.391+0000 7f61471bf740 1 --2- 172.21.15.27:0/2614229438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baf89e4e30 0x55baf8a5d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.391+0000 7f6121ffb640 0 client.15297 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:15:10.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.391+0000 7f61471bf740 1 --2- 172.21.15.27:0/2614229438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55baf89e4440 0x55baf89f3d20 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.391+0000 7f61471bf740 1 -- 172.21.15.27:0/2614229438 >> 172.21.15.27:0/2614229438 conn(0x55baf89dfab0 msgr2=0x55baf89e56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.391+0000 7f61471bf740 1 -- 172.21.15.27:0/2614229438 shutdown_connections 2024-04-25T00:15:10.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.391+0000 7f61471bf740 1 -- 172.21.15.27:0/2614229438 wait complete. 2024-04-25T00:15:10.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.391+0000 7f61471bf740 20 client.15297 trim_cache size 0 max 16384 2024-04-25T00:15:10.399 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:15:10.399 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "518857f8-0297-11ef-bc93-c7b262605968", 2024-04-25T00:15:10.400 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 2, 2024-04-25T00:15:10.400 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24496, 2024-04-25T00:15:10.400 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T00:15:10.400 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T00:15:10.400 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T00:15:10.400 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 85.741659401999996, 2024-04-25T00:15:10.400 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 28, 2024-04-25T00:15:10.400 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 91, 2024-04-25T00:15:10.400 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T00:15:10.400 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 131.501556021 2024-04-25T00:15:10.400 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:15:10.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.392+0000 7f61443d5640 1 -- 172.21.15.27:0/3948103473 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f61240a2c20 msgr2=0x7f61240a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.392+0000 7f61443d5640 1 --2- 172.21.15.27:0/3948103473 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f61240a2c20 0x7f61240a50e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f613401f000 tx=0x7f6134020000 comp rx=0 tx=0).stop 2024-04-25T00:15:10.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.392+0000 7f61443d5640 1 -- 172.21.15.27:0/3948103473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f613c102200 msgr2=0x7f613c197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.392+0000 7f61443d5640 1 --2- 172.21.15.27:0/3948103473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f613c102200 0x7f613c197b70 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f612c031a40 tx=0x7f612c031ee0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.392+0000 7f61443d5640 1 -- 172.21.15.27:0/3948103473 shutdown_connections 2024-04-25T00:15:10.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.392+0000 7f61443d5640 1 --2- 172.21.15.27:0/3948103473 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f61240a2c20 0x7f61240a50e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.392+0000 7f61443d5640 1 --2- 172.21.15.27:0/3948103473 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f613c107460 0x7f613c19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.392+0000 7f61443d5640 1 --2- 172.21.15.27:0/3948103473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f613c104b30 0x7f613c1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.392+0000 7f61443d5640 1 --2- 172.21.15.27:0/3948103473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f613c102200 0x7f613c197b70 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.392+0000 7f61443d5640 1 -- 172.21.15.27:0/3948103473 >> 172.21.15.27:0/3948103473 conn(0x7f613c0fbbf0 msgr2=0x7f613c102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.392+0000 7f61443d5640 1 -- 172.21.15.27:0/3948103473 shutdown_connections 2024-04-25T00:15:10.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.392+0000 7f61443d5640 1 -- 172.21.15.27:0/3948103473 wait complete. 2024-04-25T00:15:10.403 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T00:15:10.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.511+0000 7fc6e08d2640 1 -- 172.21.15.27:0/2696559129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6dc103350 msgr2=0x7fc6dc109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.511+0000 7fc6e08d2640 1 --2- 172.21.15.27:0/2696559129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6dc103350 0x7fc6dc109be0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4009ca0 tx=0x7fc6d402a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.511+0000 7fc6e08d2640 1 -- 172.21.15.27:0/2696559129 shutdown_connections 2024-04-25T00:15:10.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.511+0000 7fc6e08d2640 1 --2- 172.21.15.27:0/2696559129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6dc103350 0x7fc6dc109be0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.511+0000 7fc6e08d2640 1 --2- 172.21.15.27:0/2696559129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6dc102990 0x7fc6dc102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.511+0000 7fc6e08d2640 1 --2- 172.21.15.27:0/2696559129 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6dc1017a0 0x7fc6dc101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.511+0000 7fc6e08d2640 1 -- 172.21.15.27:0/2696559129 >> 172.21.15.27:0/2696559129 conn(0x7fc6dc0fcd10 msgr2=0x7fc6dc0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.512+0000 7fc6e08d2640 1 -- 172.21.15.27:0/2696559129 shutdown_connections 2024-04-25T00:15:10.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.512+0000 7fc6e08d2640 1 -- 172.21.15.27:0/2696559129 wait complete. 2024-04-25T00:15:10.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.512+0000 7fc6e08d2640 1 Processor -- start 2024-04-25T00:15:10.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.512+0000 7fc6e08d2640 1 -- start start 2024-04-25T00:15:10.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.512+0000 7fc6e08d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6dc1017a0 0x7fc6dc197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.512+0000 7fc6e08d2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6dc102990 0x7fc6dc197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.512+0000 7fc6e08d2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6dc103350 0x7fc6dc19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.512+0000 7fc6e08d2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc6dc10d450 con 0x7fc6dc102990 2024-04-25T00:15:10.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.512+0000 7fc6e08d2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc6dc10d2d0 con 0x7fc6dc103350 2024-04-25T00:15:10.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.512+0000 7fc6e08d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc6dc10d5d0 con 0x7fc6dc1017a0 2024-04-25T00:15:10.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.513+0000 7fc6d9d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6dc102990 0x7fc6dc197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.513+0000 7fc6d9d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6dc102990 0x7fc6dc197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47738/0 (socket says 172.21.15.27:47738) 2024-04-25T00:15:10.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.513+0000 7fc6d9d9b640 1 -- 172.21.15.27:0/374402233 learned_addr learned my addr 172.21.15.27:0/374402233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.513+0000 7fc6da59c640 1 --2- 172.21.15.27:0/374402233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6dc1017a0 0x7fc6dc197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.513+0000 7fc6dad9d640 1 --2- 172.21.15.27:0/374402233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6dc103350 0x7fc6dc19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.513+0000 7fc6d9d9b640 1 -- 172.21.15.27:0/374402233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6dc1017a0 msgr2=0x7fc6dc197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.513+0000 7fc6d9d9b640 1 --2- 172.21.15.27:0/374402233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6dc1017a0 0x7fc6dc197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.513+0000 7fc6d9d9b640 1 -- 172.21.15.27:0/374402233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6dc103350 msgr2=0x7fc6dc19eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.513+0000 7fc6d9d9b640 1 --2- 172.21.15.27:0/374402233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6dc103350 0x7fc6dc19eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.513+0000 7fc6d9d9b640 1 -- 172.21.15.27:0/374402233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6dc19f530 con 0x7fc6dc102990 2024-04-25T00:15:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.513+0000 7fc6d9d9b640 1 --2- 172.21.15.27:0/374402233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6dc102990 0x7fc6dc197e50 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc6d000c920 tx=0x7fc6d000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.513+0000 7fc6cb7fe640 1 -- 172.21.15.27:0/374402233 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0007c70 con 0x7fc6dc102990 2024-04-25T00:15:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.513+0000 7fc6cb7fe640 1 -- 172.21.15.27:0/374402233 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6d0007e10 con 0x7fc6dc102990 2024-04-25T00:15:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.513+0000 7fc6e08d2640 1 -- 172.21.15.27:0/374402233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6dc19f820 con 0x7fc6dc102990 2024-04-25T00:15:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.513+0000 7fc6cb7fe640 1 -- 172.21.15.27:0/374402233 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0018890 con 0x7fc6dc102990 2024-04-25T00:15:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.513+0000 7fc6e08d2640 1 -- 172.21.15.27:0/374402233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6dc19fdc0 con 0x7fc6dc102990 2024-04-25T00:15:10.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.515+0000 7fc6cb7fe640 1 -- 172.21.15.27:0/374402233 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc6d0018a70 con 0x7fc6dc102990 2024-04-25T00:15:10.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.516+0000 7fc6e35dd740 1 Processor -- start 2024-04-25T00:15:10.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.516+0000 7fc6e35dd740 1 -- start start 2024-04-25T00:15:10.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.516+0000 7fc6e35dd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5440 0x55d0724f5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.516+0000 7fc6e35dd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724f5e30 0x55d0724fa930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.516+0000 7fc6e35dd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724fafa0 0x55d0724fd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.516+0000 7fc6e35dd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d072458f00 con 0x55d0724f5440 2024-04-25T00:15:10.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.516+0000 7fc6e35dd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d072401750 con 0x55d0724fafa0 2024-04-25T00:15:10.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.516+0000 7fc6e35dd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d072454ec0 con 0x55d0724f5e30 2024-04-25T00:15:10.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.516+0000 7fc6cb7fe640 1 --2- 172.21.15.27:0/374402233 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc6c40a2c10 0x7fc6c40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.516+0000 7fc6dad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5440 0x55d0724f5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.516+0000 7fc6d9d9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724fafa0 0x55d0724fd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.516+0000 7fc6dad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5440 0x55d0724f5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47742/0 (socket says 172.21.15.27:47742) 2024-04-25T00:15:10.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.516+0000 7fc6dad9d640 1 -- 172.21.15.27:0/2586182251 learned_addr learned my addr 172.21.15.27:0/2586182251 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.516+0000 7fc6dad9d640 1 --2- 172.21.15.27:0/374402233 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc6c40a2c10 0x7fc6c40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.516+0000 7fc6da59c640 1 --2- 172.21.15.27:0/2586182251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724f5e30 0x55d0724fa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.516+0000 7fc6cb7fe640 1 -- 172.21.15.27:0/374402233 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9150+0+0 (secure 0 0 0) 0x7fc6d00cf370 con 0x7fc6dc102990 2024-04-25T00:15:10.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.517+0000 7fc6dad9d640 1 -- 172.21.15.27:0/2586182251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724f5e30 msgr2=0x55d0724fa930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.517+0000 7fc6dad9d640 1 --2- 172.21.15.27:0/2586182251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724f5e30 0x55d0724fa930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.517+0000 7fc6dad9d640 1 -- 172.21.15.27:0/2586182251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724fafa0 msgr2=0x55d0724fd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.517+0000 7fc6dad9d640 1 --2- 172.21.15.27:0/2586182251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724fafa0 0x55d0724fd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.517+0000 7fc6dad9d640 1 -- 172.21.15.27:0/2586182251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d07216d2d0 con 0x55d0724f5440 2024-04-25T00:15:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.517+0000 7fc6dad9d640 1 --2- 172.21.15.27:0/2586182251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5440 0x55d0724f5860 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc6d402f540 tx=0x7fc6d4031850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d75ac9463540af45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.517+0000 7fc6dad9d640 1 --2- 172.21.15.27:0/374402233 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc6c40a2c10 0x7fc6c40a50d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4009a60 tx=0x7fc6d403b000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.517+0000 7fc6c8ff9640 1 -- 172.21.15.27:0/2586182251 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d404f070 con 0x55d0724f5440 2024-04-25T00:15:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.517+0000 7fc6c8ff9640 1 -- 172.21.15.27:0/2586182251 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6d4036040 con 0x55d0724f5440 2024-04-25T00:15:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.517+0000 7fc6c8ff9640 1 -- 172.21.15.27:0/2586182251 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d402fa40 con 0x55d0724f5440 2024-04-25T00:15:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.518+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2586182251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5440 msgr2=0x55d0724f5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.518+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/2586182251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5440 0x55d0724f5860 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc6d402f540 tx=0x7fc6d4031850 comp rx=0 tx=0).stop 2024-04-25T00:15:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.518+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2586182251 shutdown_connections 2024-04-25T00:15:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.518+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/2586182251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724fafa0 0x55d0724fd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.518+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/2586182251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724f5e30 0x55d0724fa930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.518+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/2586182251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5440 0x55d0724f5860 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.518+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2586182251 >> 172.21.15.27:0/2586182251 conn(0x55d0724f0ab0 msgr2=0x55d0724f0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.518+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2586182251 shutdown_connections 2024-04-25T00:15:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.518+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2586182251 wait complete. 2024-04-25T00:15:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.518+0000 7fc6e35dd740 1 -- start start 2024-04-25T00:15:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.518+0000 7fc6e35dd740 1 Processor -- start 2024-04-25T00:15:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.518+0000 7fc6e35dd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724f5440 0x55d0724ff7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.518+0000 7fc6e35dd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5e30 0x55d0724ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.518+0000 7fc6e35dd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724fafa0 0x55d072506d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.518+0000 7fc6e35dd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d0724fdd70 con 0x55d0724f5e30 2024-04-25T00:15:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.518+0000 7fc6e35dd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d072414cf0 con 0x55d0724f5440 2024-04-25T00:15:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.518+0000 7fc6e35dd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d0724fdef0 con 0x55d0724fafa0 2024-04-25T00:15:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.519+0000 7fc6e35dd740 10 client.0 _ll_register_callbacks cb 0x55d0724146e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.519+0000 7fc6d9d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5e30 0x55d0724ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.519+0000 7fc6d9d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5e30 0x55d0724ffcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47746/0 (socket says 172.21.15.27:47746) 2024-04-25T00:15:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.519+0000 7fc6d9d9b640 1 -- 172.21.15.27:0/1096764243 learned_addr learned my addr 172.21.15.27:0/1096764243 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.519+0000 7fc6d9d9b640 1 -- 172.21.15.27:0/1096764243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724fafa0 msgr2=0x55d072506d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.519+0000 7fc6d9d9b640 1 --2- 172.21.15.27:0/1096764243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724fafa0 0x55d072506d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.519+0000 7fc6d9d9b640 1 -- 172.21.15.27:0/1096764243 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724f5440 msgr2=0x55d0724ff7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.519+0000 7fc6d9d9b640 1 --2- 172.21.15.27:0/1096764243 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724f5440 0x55d0724ff7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.519+0000 7fc6d9d9b640 1 -- 172.21.15.27:0/1096764243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0722376b0 con 0x55d0724f5e30 2024-04-25T00:15:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.519+0000 7fc6d9d9b640 1 --2- 172.21.15.27:0/1096764243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5e30 0x55d0724ffcf0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc6d009b870 tx=0x7fc6d006b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b95aa1b1f4f39ec5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.519+0000 7fc6c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.519+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/1096764243 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d00cf1b0 con 0x55d0724f5e30 2024-04-25T00:15:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.519+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/1096764243 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9150+0+0 (secure 0 0 0) 0x7fc6d000fb30 con 0x55d0724f5e30 2024-04-25T00:15:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.519+0000 7fc6e35dd740 1 -- 172.21.15.27:0/1096764243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6d4036040 con 0x55d0724f5e30 2024-04-25T00:15:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.519+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/1096764243 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc6d009b8e0 con 0x55d0724f5e30 2024-04-25T00:15:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.519+0000 7fc6e35dd740 10 client.15309 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.519+0000 7fc6e35dd740 1 -- 172.21.15.27:0/1096764243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d072509d80 con 0x55d0724f5e30 2024-04-25T00:15:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.520+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/1096764243 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6d4036040 con 0x55d0724f5e30 2024-04-25T00:15:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.520+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/1096764243 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc6d00a40c0 con 0x55d0724f5e30 2024-04-25T00:15:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.520+0000 7fc6e35dd740 10 client.15309 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.520+0000 7fc6e35dd740 10 client.15309 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T00:15:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.520+0000 7fc6e35dd740 20 client.15309 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.520+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/1096764243 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55d07250c4c0 0x55d07250d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.520+0000 7fc6e35dd740 4 client.15309 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:15:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.520+0000 7fc6e35dd740 1 -- 172.21.15.27:0/1096764243 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d072466440 con 0x55d07250c4c0 2024-04-25T00:15:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.520+0000 7fc6da59c640 1 --2- 172.21.15.27:0/1096764243 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55d07250c4c0 0x55d07250d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.520+0000 7fc6da59c640 1 --2- 172.21.15.27:0/1096764243 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55d07250c4c0 0x55d07250d930 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d922333ec41f7c6d server_cookie=7d117bb2195be1a2 in_seq=0 out_seq=0 2024-04-25T00:15:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.520+0000 7fc6c1ffb640 10 client.15309 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:15:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.521+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/1096764243 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d072466440 con 0x55d07250c4c0 2024-04-25T00:15:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.521+0000 7fc6c1ffb640 10 client.15309 handle_command_reply: tid=0 2024-04-25T00:15:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.521+0000 7fc6e35dd740 1 client.15309 shutdown 2024-04-25T00:15:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.521+0000 7fc6e35dd740 1 -- 172.21.15.27:0/1096764243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5e30 msgr2=0x55d0724ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.521+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/1096764243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5e30 0x55d0724ffcf0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc6d009b870 tx=0x7fc6d006b790 comp rx=0 tx=0).stop 2024-04-25T00:15:10.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.522+0000 7fc6e35dd740 1 -- 172.21.15.27:0/1096764243 shutdown_connections 2024-04-25T00:15:10.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.522+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/1096764243 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55d07250c4c0 0x55d07250d930 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.522+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/1096764243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724fafa0 0x55d072506d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.522+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/1096764243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5e30 0x55d0724ffcf0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.522+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/1096764243 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724f5440 0x55d0724ff7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.522+0000 7fc6c1ffb640 0 client.15309 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:15:10.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.522+0000 7fc6e35dd740 1 -- 172.21.15.27:0/1096764243 >> 172.21.15.27:0/1096764243 conn(0x55d0724f0ab0 msgr2=0x55d0724efae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.522+0000 7fc6e35dd740 1 -- 172.21.15.27:0/1096764243 shutdown_connections 2024-04-25T00:15:10.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.522+0000 7fc6e35dd740 1 -- 172.21.15.27:0/1096764243 wait complete. 2024-04-25T00:15:10.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.522+0000 7fc6e35dd740 20 client.15309 trim_cache size 0 max 16384 2024-04-25T00:15:10.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6e35dd740 1 Processor -- start 2024-04-25T00:15:10.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6e35dd740 1 -- start start 2024-04-25T00:15:10.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6e35dd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5440 0x55d0724feaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6e35dd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724f5e30 0x55d072506cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6e35dd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724fafa0 0x55d0724fefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6e35dd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d07250bc70 con 0x55d0724f5440 2024-04-25T00:15:10.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6e35dd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d07250c340 con 0x55d0724fafa0 2024-04-25T00:15:10.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6e35dd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d07250ca10 con 0x55d0724f5e30 2024-04-25T00:15:10.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6da59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5440 0x55d0724feaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6da59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5440 0x55d0724feaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47758/0 (socket says 172.21.15.27:47758) 2024-04-25T00:15:10.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6da59c640 1 -- 172.21.15.27:0/2656989915 learned_addr learned my addr 172.21.15.27:0/2656989915 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6dad9d640 1 --2- 172.21.15.27:0/2656989915 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724fafa0 0x55d0724fefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6d9d9b640 1 --2- 172.21.15.27:0/2656989915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724f5e30 0x55d072506cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6da59c640 1 -- 172.21.15.27:0/2656989915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724f5e30 msgr2=0x55d072506cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6da59c640 1 --2- 172.21.15.27:0/2656989915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724f5e30 0x55d072506cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6da59c640 1 -- 172.21.15.27:0/2656989915 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724fafa0 msgr2=0x55d0724fefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6da59c640 1 --2- 172.21.15.27:0/2656989915 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724fafa0 0x55d0724fefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6da59c640 1 -- 172.21.15.27:0/2656989915 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0724f08e0 con 0x55d0724f5440 2024-04-25T00:15:10.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6d9d9b640 1 --2- 172.21.15.27:0/2656989915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724f5e30 0x55d072506cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:10.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.542+0000 7fc6da59c640 1 --2- 172.21.15.27:0/2656989915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5440 0x55d0724feaa0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc004700 tx=0x7fc6cc012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15831112499fbe91 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.543+0000 7fc6c3fff640 1 -- 172.21.15.27:0/2656989915 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc021030 con 0x55d0724f5440 2024-04-25T00:15:10.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.543+0000 7fc6c3fff640 1 -- 172.21.15.27:0/2656989915 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6cc01d910 con 0x55d0724f5440 2024-04-25T00:15:10.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.543+0000 7fc6c3fff640 1 -- 172.21.15.27:0/2656989915 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc004e50 con 0x55d0724f5440 2024-04-25T00:15:10.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.543+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2656989915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5440 msgr2=0x55d0724feaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.543+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/2656989915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5440 0x55d0724feaa0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc004700 tx=0x7fc6cc012820 comp rx=0 tx=0).stop 2024-04-25T00:15:10.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.543+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2656989915 shutdown_connections 2024-04-25T00:15:10.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.543+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/2656989915 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724fafa0 0x55d0724fefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.543+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/2656989915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724f5e30 0x55d072506cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.543+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/2656989915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724f5440 0x55d0724feaa0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.543+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2656989915 >> 172.21.15.27:0/2656989915 conn(0x55d0724f0ab0 msgr2=0x55d0724f7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.543+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2656989915 shutdown_connections 2024-04-25T00:15:10.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.543+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2656989915 wait complete. 2024-04-25T00:15:10.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6e35dd740 1 -- start start 2024-04-25T00:15:10.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6e35dd740 1 Processor -- start 2024-04-25T00:15:10.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6e35dd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724f5440 0x55d072502f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6e35dd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724f5e30 0x55d072503460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6e35dd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724fafa0 0x55d0725063c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6e35dd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d07250d7b0 con 0x55d0724fafa0 2024-04-25T00:15:10.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6e35dd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d07250d0e0 con 0x55d0724f5440 2024-04-25T00:15:10.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6e35dd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d07250de80 con 0x55d0724f5e30 2024-04-25T00:15:10.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6e35dd740 10 client.0 _ll_register_callbacks cb 0x7fc6d009b7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:10.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6dad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724fafa0 0x55d0725063c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6dad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724fafa0 0x55d0725063c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47764/0 (socket says 172.21.15.27:47764) 2024-04-25T00:15:10.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6dad9d640 1 -- 172.21.15.27:0/2703460766 learned_addr learned my addr 172.21.15.27:0/2703460766 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6d9d9b640 1 --2- 172.21.15.27:0/2703460766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724f5e30 0x55d072503460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6da59c640 1 --2- 172.21.15.27:0/2703460766 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724f5440 0x55d072502f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6dad9d640 1 -- 172.21.15.27:0/2703460766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724f5e30 msgr2=0x55d072503460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6dad9d640 1 --2- 172.21.15.27:0/2703460766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724f5e30 0x55d072503460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6dad9d640 1 -- 172.21.15.27:0/2703460766 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724f5440 msgr2=0x55d072502f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6dad9d640 1 --2- 172.21.15.27:0/2703460766 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724f5440 0x55d072502f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6dad9d640 1 -- 172.21.15.27:0/2703460766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d07250baa0 con 0x55d0724fafa0 2024-04-25T00:15:10.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6da59c640 1 --2- 172.21.15.27:0/2703460766 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724f5440 0x55d072502f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:10.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.544+0000 7fc6dad9d640 1 --2- 172.21.15.27:0/2703460766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724fafa0 0x55d0725063c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4031c10 tx=0x7fc6d4004480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ac305be887675b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.545+0000 7fc6c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:10.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.545+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/2703460766 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d405b070 con 0x55d0724fafa0 2024-04-25T00:15:10.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.545+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/2703460766 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9150+0+0 (secure 0 0 0) 0x7fc6d402f610 con 0x55d0724fafa0 2024-04-25T00:15:10.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.545+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2703460766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6cc01d910 con 0x55d0724fafa0 2024-04-25T00:15:10.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.545+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/2703460766 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc6d404f040 con 0x55d0724fafa0 2024-04-25T00:15:10.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.545+0000 7fc6e35dd740 10 client.15315 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:10.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.545+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2703460766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d072572c00 con 0x55d0724fafa0 2024-04-25T00:15:10.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.545+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/2703460766 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6cc01d910 con 0x55d0724fafa0 2024-04-25T00:15:10.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.545+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/2703460766 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc6d40305c0 con 0x55d0724fafa0 2024-04-25T00:15:10.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.545+0000 7fc6e35dd740 10 client.15315 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:10.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.545+0000 7fc6e35dd740 10 client.15315 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T00:15:10.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.545+0000 7fc6e35dd740 20 client.15315 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:10.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.545+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/2703460766 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55d0725775f0 0x55d072576300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.545+0000 7fc6e35dd740 4 client.15315 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T00:15:10.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.545+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2703460766 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "status"}) -- 0x55d072460790 con 0x55d0725775f0 2024-04-25T00:15:10.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.546+0000 7fc6da59c640 1 --2- 172.21.15.27:0/2703460766 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55d0725775f0 0x55d072576300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.546+0000 7fc6da59c640 1 --2- 172.21.15.27:0/2703460766 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55d0725775f0 0x55d072576300 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f0686850eb936326 server_cookie=69612a1b2ea4f97 in_seq=0 out_seq=0 2024-04-25T00:15:10.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.546+0000 7fc6c1ffb640 10 client.15315 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:15:10.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.546+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/2703460766 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55d072460790 con 0x55d0725775f0 2024-04-25T00:15:10.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.546+0000 7fc6c1ffb640 10 client.15315 handle_command_reply: tid=0 2024-04-25T00:15:10.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.546+0000 7fc6e35dd740 1 client.15315 shutdown 2024-04-25T00:15:10.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.546+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2703460766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724fafa0 msgr2=0x55d0725063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.546+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/2703460766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724fafa0 0x55d0725063c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4031c10 tx=0x7fc6d4004480 comp rx=0 tx=0).stop 2024-04-25T00:15:10.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.546+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2703460766 shutdown_connections 2024-04-25T00:15:10.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.546+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/2703460766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0724fafa0 0x55d0725063c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.546+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/2703460766 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55d0725775f0 0x55d072576300 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.546+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/2703460766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0724f5e30 0x55d072503460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.546+0000 7fc6e35dd740 1 --2- 172.21.15.27:0/2703460766 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d0724f5440 0x55d072502f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.546+0000 7fc6c1ffb640 0 client.15315 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:15:10.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.546+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2703460766 >> 172.21.15.27:0/2703460766 conn(0x55d0724f0ab0 msgr2=0x55d0724feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.546+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2703460766 shutdown_connections 2024-04-25T00:15:10.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.546+0000 7fc6e35dd740 1 -- 172.21.15.27:0/2703460766 wait complete. 2024-04-25T00:15:10.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.547+0000 7fc6e35dd740 20 client.15315 trim_cache size 0 max 16384 2024-04-25T00:15:10.553 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:15:10.553 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "518857f8-0297-11ef-bc93-c7b262605968", 2024-04-25T00:15:10.553 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 1, 2024-04-25T00:15:10.553 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24514, 2024-04-25T00:15:10.553 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T00:15:10.554 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T00:15:10.554 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T00:15:10.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 86.904851066000006, 2024-04-25T00:15:10.554 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 28, 2024-04-25T00:15:10.554 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 92, 2024-04-25T00:15:10.554 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T00:15:10.554 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 125.817540697 2024-04-25T00:15:10.554 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.547+0000 7fc6e08d2640 1 -- 172.21.15.27:0/374402233 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc6c40a2c10 msgr2=0x7fc6c40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.547+0000 7fc6e08d2640 1 --2- 172.21.15.27:0/374402233 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc6c40a2c10 0x7fc6c40a50d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4009a60 tx=0x7fc6d403b000 comp rx=0 tx=0).stop 2024-04-25T00:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.547+0000 7fc6e08d2640 1 -- 172.21.15.27:0/374402233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6dc102990 msgr2=0x7fc6dc197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.547+0000 7fc6e08d2640 1 --2- 172.21.15.27:0/374402233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6dc102990 0x7fc6dc197e50 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc6d000c920 tx=0x7fc6d000cde0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.547+0000 7fc6e08d2640 1 -- 172.21.15.27:0/374402233 shutdown_connections 2024-04-25T00:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.547+0000 7fc6e08d2640 1 --2- 172.21.15.27:0/374402233 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc6c40a2c10 0x7fc6c40a50d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.547+0000 7fc6e08d2640 1 --2- 172.21.15.27:0/374402233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6dc103350 0x7fc6dc19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.547+0000 7fc6e08d2640 1 --2- 172.21.15.27:0/374402233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6dc102990 0x7fc6dc197e50 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.547+0000 7fc6e08d2640 1 --2- 172.21.15.27:0/374402233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6dc1017a0 0x7fc6dc197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.547+0000 7fc6e08d2640 1 -- 172.21.15.27:0/374402233 >> 172.21.15.27:0/374402233 conn(0x7fc6dc0fcd10 msgr2=0x7fc6dc105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.547+0000 7fc6e08d2640 1 -- 172.21.15.27:0/374402233 shutdown_connections 2024-04-25T00:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.547+0000 7fc6e08d2640 1 -- 172.21.15.27:0/374402233 wait complete. 2024-04-25T00:15:10.556 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:10 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/372645368' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:10.557 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:10 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3926014226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:10.557 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-25T00:15:10.557 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:10 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-04-25T00:15:10.558 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T00:15:10.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.663+0000 7f6d5c109640 1 -- 172.21.15.27:0/2742359083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d54100f60 msgr2=0x7f6d541013e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.663+0000 7f6d5c109640 1 --2- 172.21.15.27:0/2742359083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d54100f60 0x7f6d541013e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400aa10 tx=0x7f6d4402b720 comp rx=0 tx=0).stop 2024-04-25T00:15:10.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.663+0000 7f6d5c109640 1 -- 172.21.15.27:0/2742359083 shutdown_connections 2024-04-25T00:15:10.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.663+0000 7f6d5c109640 1 --2- 172.21.15.27:0/2742359083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d54101920 0x7f6d54108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.663+0000 7f6d5c109640 1 --2- 172.21.15.27:0/2742359083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d54100f60 0x7f6d541013e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.663+0000 7f6d5c109640 1 --2- 172.21.15.27:0/2742359083 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d540ffe10 0x7f6d54100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.663+0000 7f6d5c109640 1 -- 172.21.15.27:0/2742359083 >> 172.21.15.27:0/2742359083 conn(0x7f6d540fb380 msgr2=0x7f6d540fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.663+0000 7f6d5c109640 1 -- 172.21.15.27:0/2742359083 shutdown_connections 2024-04-25T00:15:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.663+0000 7f6d5c109640 1 -- 172.21.15.27:0/2742359083 wait complete. 2024-04-25T00:15:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d5c109640 1 Processor -- start 2024-04-25T00:15:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d5c109640 1 -- start start 2024-04-25T00:15:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d5c109640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d540ffe10 0x7f6d54103d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d5c109640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d54100f60 0x7f6d541023d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d5c109640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d54101920 0x7f6d54102910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d5c109640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5410b230 con 0x7f6d54100f60 2024-04-25T00:15:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d5c109640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5410b0b0 con 0x7f6d540ffe10 2024-04-25T00:15:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d5c109640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5410b3b0 con 0x7f6d54101920 2024-04-25T00:15:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d596a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d54100f60 0x7f6d541023d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d596a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d54100f60 0x7f6d541023d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47786/0 (socket says 172.21.15.27:47786) 2024-04-25T00:15:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d596a4640 1 -- 172.21.15.27:0/2615366438 learned_addr learned my addr 172.21.15.27:0/2615366438 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d59ea5640 1 --2- 172.21.15.27:0/2615366438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d540ffe10 0x7f6d54103d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d5a6a6640 1 --2- 172.21.15.27:0/2615366438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d54101920 0x7f6d54102910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d596a4640 1 -- 172.21.15.27:0/2615366438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d54101920 msgr2=0x7f6d54102910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d596a4640 1 --2- 172.21.15.27:0/2615366438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d54101920 0x7f6d54102910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d596a4640 1 -- 172.21.15.27:0/2615366438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d540ffe10 msgr2=0x7f6d54103d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d596a4640 1 --2- 172.21.15.27:0/2615366438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d540ffe10 0x7f6d54103d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d596a4640 1 -- 172.21.15.27:0/2615366438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d541031a0 con 0x7f6d54100f60 2024-04-25T00:15:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d5a6a6640 1 --2- 172.21.15.27:0/2615366438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d54101920 0x7f6d54102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.664+0000 7f6d596a4640 1 --2- 172.21.15.27:0/2615366438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d54100f60 0x7f6d541023d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6d44006010 tx=0x7f6d440043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.665+0000 7f6d4affd640 1 -- 172.21.15.27:0/2615366438 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4403f070 con 0x7f6d54100f60 2024-04-25T00:15:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.665+0000 7f6d5c109640 1 -- 172.21.15.27:0/2615366438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d541a1c10 con 0x7f6d54100f60 2024-04-25T00:15:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.665+0000 7f6d4affd640 1 -- 172.21.15.27:0/2615366438 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d44032400 con 0x7f6d54100f60 2024-04-25T00:15:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.665+0000 7f6d4affd640 1 -- 172.21.15.27:0/2615366438 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d44036040 con 0x7f6d54100f60 2024-04-25T00:15:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.665+0000 7f6d5c109640 1 -- 172.21.15.27:0/2615366438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d541a2160 con 0x7f6d54100f60 2024-04-25T00:15:10.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.667+0000 7f6d4affd640 1 -- 172.21.15.27:0/2615366438 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6d44042070 con 0x7f6d54100f60 2024-04-25T00:15:10.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.667+0000 7f6d5ee0d740 1 Processor -- start 2024-04-25T00:15:10.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.667+0000 7f6d5ee0d740 1 -- start start 2024-04-25T00:15:10.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d5ee0d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f894435860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d5ee0d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435e30 0x55f89443a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d5ee0d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f89443afa0 0x55f89443d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d5ee0d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f894398f00 con 0x55f894435440 2024-04-25T00:15:10.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d5ee0d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f894341750 con 0x55f894435e30 2024-04-25T00:15:10.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d5ee0d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f894394ec0 con 0x55f89443afa0 2024-04-25T00:15:10.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d5a6a6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f894435860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d5a6a6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f894435860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47790/0 (socket says 172.21.15.27:47790) 2024-04-25T00:15:10.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d5a6a6640 1 -- 172.21.15.27:0/84005609 learned_addr learned my addr 172.21.15.27:0/84005609 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d4affd640 1 --2- 172.21.15.27:0/2615366438 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6d400a2b90 0x7f6d400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d59ea5640 1 --2- 172.21.15.27:0/84005609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435e30 0x55f89443a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d596a4640 1 --2- 172.21.15.27:0/84005609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f89443afa0 0x55f89443d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d5a6a6640 1 --2- 172.21.15.27:0/2615366438 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6d400a2b90 0x7f6d400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d4affd640 1 -- 172.21.15.27:0/2615366438 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9150+0+0 (secure 0 0 0) 0x7f6d440efc10 con 0x7f6d54100f60 2024-04-25T00:15:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d5a6a6640 1 -- 172.21.15.27:0/84005609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f89443afa0 msgr2=0x55f89443d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d5a6a6640 1 --2- 172.21.15.27:0/84005609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f89443afa0 0x55f89443d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d5a6a6640 1 -- 172.21.15.27:0/84005609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435e30 msgr2=0x55f89443a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d5a6a6640 1 --2- 172.21.15.27:0/84005609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435e30 0x55f89443a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d5a6a6640 1 -- 172.21.15.27:0/84005609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8940ad2d0 con 0x55f894435440 2024-04-25T00:15:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d5a6a6640 1 --2- 172.21.15.27:0/84005609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f894435860 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0129e0 tx=0x7f6d4c012ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7341cc917cb55507 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d3ffff640 1 -- 172.21.15.27:0/84005609 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c022070 con 0x55f894435440 2024-04-25T00:15:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.668+0000 7f6d5a6a6640 1 --2- 172.21.15.27:0/2615366438 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6d400a2b90 0x7f6d400a5050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c020000 tx=0x7f6d4c01f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.669+0000 7f6d3ffff640 1 -- 172.21.15.27:0/84005609 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d4c005500 con 0x55f894435440 2024-04-25T00:15:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.669+0000 7f6d3ffff640 1 -- 172.21.15.27:0/84005609 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c01e8c0 con 0x55f894435440 2024-04-25T00:15:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.669+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/84005609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 msgr2=0x55f894435860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.669+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/84005609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f894435860 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0129e0 tx=0x7f6d4c012ea0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.669+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/84005609 shutdown_connections 2024-04-25T00:15:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.669+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/84005609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f89443afa0 0x55f89443d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.669+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/84005609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435e30 0x55f89443a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.669+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/84005609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f894435860 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.669+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/84005609 >> 172.21.15.27:0/84005609 conn(0x55f894430ab0 msgr2=0x55f894430eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.669+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/84005609 shutdown_connections 2024-04-25T00:15:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.669+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/84005609 wait complete. 2024-04-25T00:15:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.670+0000 7f6d5ee0d740 1 -- start start 2024-04-25T00:15:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.670+0000 7f6d5ee0d740 1 Processor -- start 2024-04-25T00:15:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.670+0000 7f6d5ee0d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f89443f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.670+0000 7f6d5ee0d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435e30 0x55f89443fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.670+0000 7f6d5ee0d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f89443afa0 0x55f894446e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.670+0000 7f6d5ee0d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f89443df50 con 0x55f894435440 2024-04-25T00:15:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.670+0000 7f6d5ee0d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f894354cf0 con 0x55f894435e30 2024-04-25T00:15:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.670+0000 7f6d5ee0d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f89443e0d0 con 0x55f89443afa0 2024-04-25T00:15:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.670+0000 7f6d5ee0d740 10 client.0 _ll_register_callbacks cb 0x55f8943546e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.670+0000 7f6d59ea5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f89443f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.670+0000 7f6d59ea5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f89443f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47806/0 (socket says 172.21.15.27:47806) 2024-04-25T00:15:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.670+0000 7f6d59ea5640 1 -- 172.21.15.27:0/927209998 learned_addr learned my addr 172.21.15.27:0/927209998 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.670+0000 7f6d596a4640 1 --2- 172.21.15.27:0/927209998 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435e30 0x55f89443fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.671+0000 7f6d59ea5640 1 -- 172.21.15.27:0/927209998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f89443afa0 msgr2=0x55f894446e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.671+0000 7f6d59ea5640 1 --2- 172.21.15.27:0/927209998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f89443afa0 0x55f894446e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.671+0000 7f6d59ea5640 1 -- 172.21.15.27:0/927209998 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435e30 msgr2=0x55f89443fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.671+0000 7f6d59ea5640 1 --2- 172.21.15.27:0/927209998 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435e30 0x55f89443fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.671+0000 7f6d59ea5640 1 -- 172.21.15.27:0/927209998 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8941776b0 con 0x55f894435440 2024-04-25T00:15:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.671+0000 7f6d59ea5640 1 --2- 172.21.15.27:0/927209998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f89443f8a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000bad0 tx=0x7f6d5000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2895da346a2b16 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.671+0000 7f6d3d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.671+0000 7f6d3d7fa640 1 -- 172.21.15.27:0/927209998 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d50003da0 con 0x55f894435440 2024-04-25T00:15:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.671+0000 7f6d3d7fa640 1 -- 172.21.15.27:0/927209998 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9150+0+0 (secure 0 0 0) 0x7f6d50011440 con 0x55f894435440 2024-04-25T00:15:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.671+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/927209998 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d4c005500 con 0x55f894435440 2024-04-25T00:15:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.671+0000 7f6d3d7fa640 1 -- 172.21.15.27:0/927209998 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6d50011680 con 0x55f894435440 2024-04-25T00:15:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.671+0000 7f6d5ee0d740 10 client.15327 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.671+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/927209998 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f894449e40 con 0x55f894435440 2024-04-25T00:15:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.671+0000 7f6d3d7fa640 1 -- 172.21.15.27:0/927209998 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d4c005500 con 0x55f894435440 2024-04-25T00:15:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.671+0000 7f6d3d7fa640 1 -- 172.21.15.27:0/927209998 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6d5001ba40 con 0x55f894435440 2024-04-25T00:15:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.671+0000 7f6d5ee0d740 10 client.15327 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.671+0000 7f6d5ee0d740 10 client.15327 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T00:15:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.672+0000 7f6d5ee0d740 20 client.15327 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.672+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/927209998 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f89444c5e0 0x55f89444da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.672+0000 7f6d5ee0d740 4 client.15327 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:15:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.672+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/927209998 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8943a6440 con 0x55f89444c5e0 2024-04-25T00:15:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.672+0000 7f6d596a4640 1 --2- 172.21.15.27:0/927209998 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f89444c5e0 0x55f89444da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.672+0000 7f6d596a4640 1 --2- 172.21.15.27:0/927209998 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f89444c5e0 0x55f89444da50 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4431788886fbfd81 server_cookie=c62e069dd6b3ddcc in_seq=0 out_seq=0 2024-04-25T00:15:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.672+0000 7f6d3d7fa640 10 client.15327 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:15:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.673+0000 7f6d3d7fa640 1 -- 172.21.15.27:0/927209998 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f8943a6440 con 0x55f89444c5e0 2024-04-25T00:15:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.673+0000 7f6d3d7fa640 10 client.15327 handle_command_reply: tid=0 2024-04-25T00:15:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.673+0000 7f6d5ee0d740 1 client.15327 shutdown 2024-04-25T00:15:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.673+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/927209998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 msgr2=0x55f89443f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.673+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/927209998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f89443f8a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000bad0 tx=0x7f6d5000bf90 comp rx=0 tx=0).stop 2024-04-25T00:15:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.673+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/927209998 shutdown_connections 2024-04-25T00:15:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.673+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/927209998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f89443afa0 0x55f894446e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.673+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/927209998 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f89444c5e0 0x55f89444da50 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.673+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/927209998 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435e30 0x55f89443fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.673+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/927209998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f89443f8a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.673+0000 7f6d3d7fa640 0 client.15327 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:15:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.673+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/927209998 >> 172.21.15.27:0/927209998 conn(0x55f894430ab0 msgr2=0x55f89442fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.674+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/927209998 shutdown_connections 2024-04-25T00:15:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.674+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/927209998 wait complete. 2024-04-25T00:15:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.674+0000 7f6d5ee0d740 20 client.15327 trim_cache size 0 max 16384 2024-04-25T00:15:10.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.692+0000 7f6d5ee0d740 1 Processor -- start 2024-04-25T00:15:10.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.692+0000 7f6d5ee0d740 1 -- start start 2024-04-25T00:15:10.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.692+0000 7f6d5ee0d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f8944366c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.692+0000 7f6d5ee0d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435e30 0x55f894446de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.692+0000 7f6d5ee0d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f89443afa0 0x55f89442f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.692+0000 7f6d5ee0d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f89444bd90 con 0x55f894435440 2024-04-25T00:15:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.692+0000 7f6d5ee0d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f89444c460 con 0x55f894435e30 2024-04-25T00:15:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.692+0000 7f6d5ee0d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f89444cb30 con 0x55f89443afa0 2024-04-25T00:15:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.692+0000 7f6d59ea5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f8944366c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.692+0000 7f6d59ea5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f8944366c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47808/0 (socket says 172.21.15.27:47808) 2024-04-25T00:15:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.692+0000 7f6d59ea5640 1 -- 172.21.15.27:0/4181507763 learned_addr learned my addr 172.21.15.27:0/4181507763 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.693+0000 7f6d5a6a6640 1 --2- 172.21.15.27:0/4181507763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f89443afa0 0x55f89442f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.693+0000 7f6d59ea5640 1 -- 172.21.15.27:0/4181507763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f89443afa0 msgr2=0x55f89442f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.693+0000 7f6d59ea5640 1 --2- 172.21.15.27:0/4181507763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f89443afa0 0x55f89442f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.693+0000 7f6d59ea5640 1 -- 172.21.15.27:0/4181507763 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435e30 msgr2=0x55f894446de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.693+0000 7f6d59ea5640 1 --2- 172.21.15.27:0/4181507763 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435e30 0x55f894446de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.693+0000 7f6d59ea5640 1 -- 172.21.15.27:0/4181507763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f89442ef70 con 0x55f894435440 2024-04-25T00:15:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.693+0000 7f6d59ea5640 1 --2- 172.21.15.27:0/4181507763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f8944366c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6d5001be80 tx=0x7f6d500115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaa98fb89992b7ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.693+0000 7f6d3f7fe640 1 -- 172.21.15.27:0/4181507763 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d50000e60 con 0x55f894435440 2024-04-25T00:15:10.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.693+0000 7f6d3f7fe640 1 -- 172.21.15.27:0/4181507763 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d50005520 con 0x55f894435440 2024-04-25T00:15:10.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.693+0000 7f6d3f7fe640 1 -- 172.21.15.27:0/4181507763 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5001b420 con 0x55f894435440 2024-04-25T00:15:10.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.694+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/4181507763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 msgr2=0x55f8944366c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.694+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/4181507763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f8944366c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6d5001be80 tx=0x7f6d500115a0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.694+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/4181507763 shutdown_connections 2024-04-25T00:15:10.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.694+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/4181507763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f89443afa0 0x55f89442f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.694+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/4181507763 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435e30 0x55f894446de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.694+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/4181507763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f894435440 0x55f8944366c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.694+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/4181507763 >> 172.21.15.27:0/4181507763 conn(0x55f894430ab0 msgr2=0x55f89442b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.694+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/4181507763 shutdown_connections 2024-04-25T00:15:10.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.694+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/4181507763 wait complete. 2024-04-25T00:15:10.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.694+0000 7f6d5ee0d740 1 -- start start 2024-04-25T00:15:10.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.694+0000 7f6d5ee0d740 1 Processor -- start 2024-04-25T00:15:10.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.694+0000 7f6d5ee0d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435440 0x55f894442710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.694+0000 7f6d5ee0d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f894435e30 0x55f894442c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.694+0000 7f6d5ee0d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f89443afa0 0x55f894445c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.694+0000 7f6d5ee0d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f89444d8d0 con 0x55f89443afa0 2024-04-25T00:15:10.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.694+0000 7f6d5ee0d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f89444d200 con 0x55f894435440 2024-04-25T00:15:10.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.694+0000 7f6d5ee0d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f89444dfa0 con 0x55f894435e30 2024-04-25T00:15:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d5ee0d740 10 client.0 _ll_register_callbacks cb 0x7f6d50004400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d5a6a6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f89443afa0 0x55f894445c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d5a6a6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f89443afa0 0x55f894445c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47814/0 (socket says 172.21.15.27:47814) 2024-04-25T00:15:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d59ea5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435440 0x55f894442710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d5a6a6640 1 -- 172.21.15.27:0/3077263505 learned_addr learned my addr 172.21.15.27:0/3077263505 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d596a4640 1 --2- 172.21.15.27:0/3077263505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f894435e30 0x55f894442c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d5a6a6640 1 -- 172.21.15.27:0/3077263505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f894435e30 msgr2=0x55f894442c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d5a6a6640 1 --2- 172.21.15.27:0/3077263505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f894435e30 0x55f894442c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d5a6a6640 1 -- 172.21.15.27:0/3077263505 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435440 msgr2=0x55f894442710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d5a6a6640 1 --2- 172.21.15.27:0/3077263505 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435440 0x55f894442710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d5a6a6640 1 -- 172.21.15.27:0/3077263505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f894446260 con 0x55f89443afa0 2024-04-25T00:15:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d5a6a6640 1 --2- 172.21.15.27:0/3077263505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f89443afa0 0x55f894445c70 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x55f894436690 tx=0x7f6d4c01ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c5968af126b86f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d3d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d3d7fa640 1 -- 172.21.15.27:0/3077263505 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c037060 con 0x55f89443afa0 2024-04-25T00:15:10.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d3d7fa640 1 -- 172.21.15.27:0/3077263505 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9150+0+0 (secure 0 0 0) 0x7f6d4c026430 con 0x55f89443afa0 2024-04-25T00:15:10.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/3077263505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d50005520 con 0x55f89443afa0 2024-04-25T00:15:10.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d3d7fa640 1 -- 172.21.15.27:0/3077263505 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6d4c03e050 con 0x55f89443afa0 2024-04-25T00:15:10.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d5ee0d740 10 client.15333 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:10.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.695+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/3077263505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f8944b2830 con 0x55f89443afa0 2024-04-25T00:15:10.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.696+0000 7f6d3d7fa640 1 -- 172.21.15.27:0/3077263505 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d50005520 con 0x55f89443afa0 2024-04-25T00:15:10.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.696+0000 7f6d3d7fa640 1 -- 172.21.15.27:0/3077263505 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6d4c031030 con 0x55f89443afa0 2024-04-25T00:15:10.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.696+0000 7f6d5ee0d740 10 client.15333 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:10.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.696+0000 7f6d5ee0d740 10 client.15333 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T00:15:10.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.696+0000 7f6d5ee0d740 20 client.15333 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:10.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.696+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/3077263505 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f8944b7180 0x55f8944b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.696+0000 7f6d5ee0d740 4 client.15333 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T00:15:10.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.696+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/3077263505 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "status"}) -- 0x55f8943a0790 con 0x55f8944b7180 2024-04-25T00:15:10.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.696+0000 7f6d59ea5640 1 --2- 172.21.15.27:0/3077263505 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f8944b7180 0x55f8944b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.696+0000 7f6d59ea5640 1 --2- 172.21.15.27:0/3077263505 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f8944b7180 0x55f8944b5e90 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=928e1ec116d408a9 server_cookie=2d6134235bc7ee9c in_seq=0 out_seq=0 2024-04-25T00:15:10.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.696+0000 7f6d3d7fa640 10 client.15333 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:15:10.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.697+0000 7f6d3d7fa640 1 -- 172.21.15.27:0/3077263505 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+320 (crc 0 0 0) 0x55f8943a0790 con 0x55f8944b7180 2024-04-25T00:15:10.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.697+0000 7f6d3d7fa640 10 client.15333 handle_command_reply: tid=0 2024-04-25T00:15:10.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.697+0000 7f6d5ee0d740 1 client.15333 shutdown 2024-04-25T00:15:10.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.697+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/3077263505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f89443afa0 msgr2=0x55f894445c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.697+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/3077263505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f89443afa0 0x55f894445c70 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x55f894436690 tx=0x7f6d4c01ead0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.697+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/3077263505 shutdown_connections 2024-04-25T00:15:10.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.697+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/3077263505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f89443afa0 0x55f894445c70 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.697+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/3077263505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f894435e30 0x55f894442c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.697+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/3077263505 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f8944b7180 0x55f8944b5e90 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.697+0000 7f6d5ee0d740 1 --2- 172.21.15.27:0/3077263505 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f894435440 0x55f894442710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.697+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/3077263505 >> 172.21.15.27:0/3077263505 conn(0x55f894430ab0 msgr2=0x55f8944366c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.697+0000 7f6d3d7fa640 0 client.15333 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:15:10.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.697+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/3077263505 shutdown_connections 2024-04-25T00:15:10.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.697+0000 7f6d5ee0d740 1 -- 172.21.15.27:0/3077263505 wait complete. 2024-04-25T00:15:10.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.697+0000 7f6d5ee0d740 20 client.15333 trim_cache size 0 max 16384 2024-04-25T00:15:10.704 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:15:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "518857f8-0297-11ef-bc93-c7b262605968", 2024-04-25T00:15:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-04-25T00:15:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24521, 2024-04-25T00:15:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T00:15:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T00:15:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T00:15:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 111.19003395, 2024-04-25T00:15:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 28, 2024-04-25T00:15:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 92, 2024-04-25T00:15:10.705 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T00:15:10.705 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 117.32166893 2024-04-25T00:15:10.705 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:15:10.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.698+0000 7f6d5c109640 1 -- 172.21.15.27:0/2615366438 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6d400a2b90 msgr2=0x7f6d400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.698+0000 7f6d5c109640 1 --2- 172.21.15.27:0/2615366438 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6d400a2b90 0x7f6d400a5050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c020000 tx=0x7f6d4c01f000 comp rx=0 tx=0).stop 2024-04-25T00:15:10.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.698+0000 7f6d5c109640 1 -- 172.21.15.27:0/2615366438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d54100f60 msgr2=0x7f6d541023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.698+0000 7f6d5c109640 1 --2- 172.21.15.27:0/2615366438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d54100f60 0x7f6d541023d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6d44006010 tx=0x7f6d440043b0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.698+0000 7f6d5c109640 1 -- 172.21.15.27:0/2615366438 shutdown_connections 2024-04-25T00:15:10.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.698+0000 7f6d5c109640 1 --2- 172.21.15.27:0/2615366438 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6d400a2b90 0x7f6d400a5050 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.698+0000 7f6d5c109640 1 --2- 172.21.15.27:0/2615366438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d54101920 0x7f6d54102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.698+0000 7f6d5c109640 1 --2- 172.21.15.27:0/2615366438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d54100f60 0x7f6d541023d0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.698+0000 7f6d5c109640 1 --2- 172.21.15.27:0/2615366438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d540ffe10 0x7f6d54103d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.698+0000 7f6d5c109640 1 -- 172.21.15.27:0/2615366438 >> 172.21.15.27:0/2615366438 conn(0x7f6d540fb380 msgr2=0x7f6d541064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.698+0000 7f6d5c109640 1 -- 172.21.15.27:0/2615366438 shutdown_connections 2024-04-25T00:15:10.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:10.698+0000 7f6d5c109640 1 -- 172.21.15.27:0/2615366438 wait complete. 2024-04-25T00:15:10.708 INFO:tasks.cephadm:Setup complete, yielding 2024-04-25T00:15:10.709 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T00:15:10.719 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T00:15:10.719 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-25T00:15:10.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:10 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/372645368' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:10.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:10 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3926014226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:10.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-25T00:15:10.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:10 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-04-25T00:15:10.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:10 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/372645368' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:10.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:10 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3926014226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:10.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-25T00:15:10.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:10 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-04-25T00:15:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:15:11.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:11 smithi027 ceph-mon[29208]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:15:11.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-25T00:15:11.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:11 smithi027 ceph-mon[29208]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-04-25T00:15:11.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:11 smithi027 ceph-mon[29208]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T00:15:11.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:11 smithi027 ceph-mon[29208]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-25T00:15:11.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:11 smithi139 ceph-mon[32833]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:15:11.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-25T00:15:11.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:11 smithi139 ceph-mon[32833]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-04-25T00:15:11.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:11 smithi139 ceph-mon[32833]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T00:15:11.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:11 smithi139 ceph-mon[32833]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-25T00:15:11.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:11 smithi153 ceph-mon[32896]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:15:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-25T00:15:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:11 smithi153 ceph-mon[32896]: from='mgr.14150 172.21.15.27:0/2318192783' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-04-25T00:15:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:11 smithi153 ceph-mon[32896]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T00:15:11.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:11 smithi153 ceph-mon[32896]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-25T00:15:12.546 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:12 smithi027 ceph-mon[29208]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T00:15:12.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:12 smithi139 ceph-mon[32833]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T00:15:12.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.773+0000 7f78fc859700 1 -- 172.21.15.27:0/70097824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78f81034b0 msgr2=0x7f78f8103910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:12.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.773+0000 7f78fc859700 1 --2- 172.21.15.27:0/70097824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78f81034b0 0x7f78f8103910 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f78ec0099c0 tx=0x7f78ec009cd0 comp rx=0 tx=0).stop 2024-04-25T00:15:12.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.773+0000 7f78fc859700 1 -- 172.21.15.27:0/70097824 shutdown_connections 2024-04-25T00:15:12.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.773+0000 7f78fc859700 1 --2- 172.21.15.27:0/70097824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78f8103e50 0x7f78f810a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:12.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.773+0000 7f78fc859700 1 --2- 172.21.15.27:0/70097824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f81022b0 0x7f78f81026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:12.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.773+0000 7f78fc859700 1 --2- 172.21.15.27:0/70097824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78f81034b0 0x7f78f8103910 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:12.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.773+0000 7f78fc859700 1 -- 172.21.15.27:0/70097824 >> 172.21.15.27:0/70097824 conn(0x7f78f80fd850 msgr2=0x7f78f80ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:12.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.773+0000 7f78fc859700 1 -- 172.21.15.27:0/70097824 shutdown_connections 2024-04-25T00:15:12.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.773+0000 7f78fc859700 1 -- 172.21.15.27:0/70097824 wait complete. 2024-04-25T00:15:12.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78fc859700 1 Processor -- start 2024-04-25T00:15:12.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78fc859700 1 -- start start 2024-04-25T00:15:12.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78fc859700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f81022b0 0x7f78f8075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:12.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78fc859700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78f81034b0 0x7f78f8074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:12.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78fc859700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78f8103e50 0x7f78f8074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:12.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78fc859700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f78f810df10 con 0x7f78f81034b0 2024-04-25T00:15:12.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78fc859700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f78f810dd90 con 0x7f78f8103e50 2024-04-25T00:15:12.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78fc859700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f78f810e090 con 0x7f78f81022b0 2024-04-25T00:15:12.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78f5d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78f81034b0 0x7f78f8074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:12.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78f5d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78f81034b0 0x7f78f8074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47832/0 (socket says 172.21.15.27:47832) 2024-04-25T00:15:12.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78f5d9b700 1 -- 172.21.15.27:0/2094977636 learned_addr learned my addr 172.21.15.27:0/2094977636 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78f6d9d700 1 --2- 172.21.15.27:0/2094977636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78f8103e50 0x7f78f8074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78f659c700 1 --2- 172.21.15.27:0/2094977636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f81022b0 0x7f78f8075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78f5d9b700 1 -- 172.21.15.27:0/2094977636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f81022b0 msgr2=0x7f78f8075fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78f5d9b700 1 --2- 172.21.15.27:0/2094977636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f81022b0 0x7f78f8075fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78f5d9b700 1 -- 172.21.15.27:0/2094977636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78f8103e50 msgr2=0x7f78f8074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78f5d9b700 1 --2- 172.21.15.27:0/2094977636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78f8103e50 0x7f78f8074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.774+0000 7f78f5d9b700 1 -- 172.21.15.27:0/2094977636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f8075410 con 0x7f78f81034b0 2024-04-25T00:15:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.775+0000 7f78f5d9b700 1 --2- 172.21.15.27:0/2094977636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78f81034b0 0x7f78f8074620 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f78ec005910 tx=0x7f78ec009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.775+0000 7f78e77fe700 1 -- 172.21.15.27:0/2094977636 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ec027070 con 0x7f78f81034b0 2024-04-25T00:15:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.775+0000 7f78fc859700 1 -- 172.21.15.27:0/2094977636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f81a2680 con 0x7f78f81034b0 2024-04-25T00:15:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.775+0000 7f78e77fe700 1 -- 172.21.15.27:0/2094977636 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78ec004150 con 0x7f78f81034b0 2024-04-25T00:15:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.775+0000 7f78e77fe700 1 -- 172.21.15.27:0/2094977636 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ec022da0 con 0x7f78f81034b0 2024-04-25T00:15:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.775+0000 7f78fc859700 1 -- 172.21.15.27:0/2094977636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f81a2ad0 con 0x7f78f81034b0 2024-04-25T00:15:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.776+0000 7f78fc859700 1 -- 172.21.15.27:0/2094977636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f8068f20 con 0x7f78f81034b0 2024-04-25T00:15:12.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:12 smithi153 ceph-mon[32896]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T00:15:12.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.777+0000 7f78e77fe700 1 -- 172.21.15.27:0/2094977636 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f78ec02a070 con 0x7f78f81034b0 2024-04-25T00:15:12.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.777+0000 7f78e77fe700 1 --2- 172.21.15.27:0/2094977636 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78e00a2c20 0x7f78e00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:12.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.778+0000 7f78f659c700 1 --2- 172.21.15.27:0/2094977636 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78e00a2c20 0x7f78e00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:12.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.778+0000 7f78e77fe700 1 -- 172.21.15.27:0/2094977636 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f78ec0d01e0 con 0x7f78f81034b0 2024-04-25T00:15:12.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.778+0000 7f78f659c700 1 --2- 172.21.15.27:0/2094977636 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78e00a2c20 0x7f78e00a50e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f78f8103310 tx=0x7f78e8009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:12.782+0000 7f78e77fe700 1 -- 172.21.15.27:0/2094977636 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f78ec05e740 con 0x7f78f81034b0 2024-04-25T00:15:13.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:13.008+0000 7f78fc859700 1 -- 172.21.15.27:0/2094977636 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f78f8062670 con 0x7f78e00a2c20 2024-04-25T00:15:13.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:13.009+0000 7f78e77fe700 1 -- 172.21.15.27:0/2094977636 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f78f8062670 con 0x7f78e00a2c20 2024-04-25T00:15:13.009 INFO:teuthology.orchestra.run.smithi027.stdout:Backend: cephadm 2024-04-25T00:15:13.009 INFO:teuthology.orchestra.run.smithi027.stdout:Available: Yes 2024-04-25T00:15:13.009 INFO:teuthology.orchestra.run.smithi027.stdout:Paused: No 2024-04-25T00:15:13.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:13.011+0000 7f78fc859700 1 -- 172.21.15.27:0/2094977636 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78e00a2c20 msgr2=0x7f78e00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:13.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:13.011+0000 7f78fc859700 1 --2- 172.21.15.27:0/2094977636 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78e00a2c20 0x7f78e00a50e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f78f8103310 tx=0x7f78e8009190 comp rx=0 tx=0).stop 2024-04-25T00:15:13.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:13.011+0000 7f78fc859700 1 -- 172.21.15.27:0/2094977636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78f81034b0 msgr2=0x7f78f8074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:13.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:13.011+0000 7f78fc859700 1 --2- 172.21.15.27:0/2094977636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78f81034b0 0x7f78f8074620 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f78ec005910 tx=0x7f78ec009580 comp rx=0 tx=0).stop 2024-04-25T00:15:13.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:13.011+0000 7f78fc859700 1 -- 172.21.15.27:0/2094977636 shutdown_connections 2024-04-25T00:15:13.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:13.011+0000 7f78fc859700 1 --2- 172.21.15.27:0/2094977636 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78e00a2c20 0x7f78e00a50e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:13.011+0000 7f78fc859700 1 --2- 172.21.15.27:0/2094977636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78f8103e50 0x7f78f8074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:13.011+0000 7f78fc859700 1 --2- 172.21.15.27:0/2094977636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f81022b0 0x7f78f8075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:13.011+0000 7f78fc859700 1 --2- 172.21.15.27:0/2094977636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78f81034b0 0x7f78f8074620 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:13.011+0000 7f78fc859700 1 -- 172.21.15.27:0/2094977636 >> 172.21.15.27:0/2094977636 conn(0x7f78f80fd850 msgr2=0x7f78f81067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:13.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:13.011+0000 7f78fc859700 1 -- 172.21.15.27:0/2094977636 shutdown_connections 2024-04-25T00:15:13.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:13.011+0000 7f78fc859700 1 -- 172.21.15.27:0/2094977636 wait complete. 2024-04-25T00:15:13.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:13 smithi027 ceph-mon[29208]: pgmap v354: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 127 B/s wr, 8 op/s 2024-04-25T00:15:13.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:13 smithi139 ceph-mon[32833]: pgmap v354: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 127 B/s wr, 8 op/s 2024-04-25T00:15:13.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:13 smithi153 ceph-mon[32896]: pgmap v354: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 127 B/s wr, 8 op/s 2024-04-25T00:15:13.785 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-25T00:15:14.035 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:15:14.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:14 smithi027 ceph-mon[29208]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:14.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:14 smithi139 ceph-mon[32833]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:14.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:14 smithi153 ceph-mon[32896]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:15.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:15 smithi027 ceph-mon[29208]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-25T00:15:15.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.748+0000 7fee810d8700 1 -- 172.21.15.27:0/1764718673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee7c103e30 msgr2=0x7fee7c10a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:15.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.748+0000 7fee810d8700 1 --2- 172.21.15.27:0/1764718673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee7c103e30 0x7fee7c10a6e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fee6c0099c0 tx=0x7fee6c009cd0 comp rx=0 tx=0).stop 2024-04-25T00:15:15.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.749+0000 7fee810d8700 1 -- 172.21.15.27:0/1764718673 shutdown_connections 2024-04-25T00:15:15.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.749+0000 7fee810d8700 1 --2- 172.21.15.27:0/1764718673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee7c103e30 0x7fee7c10a6e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:15.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.749+0000 7fee810d8700 1 --2- 172.21.15.27:0/1764718673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee7c102290 0x7fee7c1026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:15.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.749+0000 7fee810d8700 1 --2- 172.21.15.27:0/1764718673 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee7c103490 0x7fee7c1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:15.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.749+0000 7fee810d8700 1 -- 172.21.15.27:0/1764718673 >> 172.21.15.27:0/1764718673 conn(0x7fee7c0fd830 msgr2=0x7fee7c0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:15.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.749+0000 7fee810d8700 1 -- 172.21.15.27:0/1764718673 shutdown_connections 2024-04-25T00:15:15.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.749+0000 7fee810d8700 1 -- 172.21.15.27:0/1764718673 wait complete. 2024-04-25T00:15:15.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.749+0000 7fee810d8700 1 Processor -- start 2024-04-25T00:15:15.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee810d8700 1 -- start start 2024-04-25T00:15:15.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee810d8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee7c102290 0x7fee7c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:15.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee810d8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee7c103490 0x7fee7c198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:15.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee810d8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee7c103e30 0x7fee7c19f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:15.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee810d8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fee7c10de50 con 0x7fee7c103e30 2024-04-25T00:15:15.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee810d8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fee7c10dcd0 con 0x7fee7c103490 2024-04-25T00:15:15.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee810d8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fee7c10dfd0 con 0x7fee7c102290 2024-04-25T00:15:15.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee7b59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee7c103e30 0x7fee7c19f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:15.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee7b59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee7c103e30 0x7fee7c19f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47852/0 (socket says 172.21.15.27:47852) 2024-04-25T00:15:15.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee7b59e700 1 -- 172.21.15.27:0/3736946167 learned_addr learned my addr 172.21.15.27:0/3736946167 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:15.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee7ad9d700 1 --2- 172.21.15.27:0/3736946167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee7c102290 0x7fee7c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:15.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee7a59c700 1 --2- 172.21.15.27:0/3736946167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee7c103490 0x7fee7c198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:15.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee7b59e700 1 -- 172.21.15.27:0/3736946167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee7c102290 msgr2=0x7fee7c1983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:15.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee7b59e700 1 --2- 172.21.15.27:0/3736946167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee7c102290 0x7fee7c1983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:15.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee7b59e700 1 -- 172.21.15.27:0/3736946167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee7c103490 msgr2=0x7fee7c198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:15.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee7b59e700 1 --2- 172.21.15.27:0/3736946167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee7c103490 0x7fee7c198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:15.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee7b59e700 1 -- 172.21.15.27:0/3736946167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee7c1a0030 con 0x7fee7c103e30 2024-04-25T00:15:15.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee7a59c700 1 --2- 172.21.15.27:0/3736946167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee7c103490 0x7fee7c198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:15.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.750+0000 7fee7b59e700 1 --2- 172.21.15.27:0/3736946167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee7c103e30 0x7fee7c19f990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fee6c009990 tx=0x7fee6c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:15.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.751+0000 7fee73fff700 1 -- 172.21.15.27:0/3736946167 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c027070 con 0x7fee7c103e30 2024-04-25T00:15:15.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.751+0000 7fee810d8700 1 -- 172.21.15.27:0/3736946167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee7c1a02c0 con 0x7fee7c103e30 2024-04-25T00:15:15.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.751+0000 7fee73fff700 1 -- 172.21.15.27:0/3736946167 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee6c004150 con 0x7fee7c103e30 2024-04-25T00:15:15.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.751+0000 7fee810d8700 1 -- 172.21.15.27:0/3736946167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee7c1a0770 con 0x7fee7c103e30 2024-04-25T00:15:15.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.751+0000 7fee73fff700 1 -- 172.21.15.27:0/3736946167 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c022e30 con 0x7fee7c103e30 2024-04-25T00:15:15.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.752+0000 7fee810d8700 1 -- 172.21.15.27:0/3736946167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee7c068f20 con 0x7fee7c103e30 2024-04-25T00:15:15.755 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:15 smithi139 ceph-mon[32833]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-25T00:15:15.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.753+0000 7fee73fff700 1 -- 172.21.15.27:0/3736946167 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fee6c02a070 con 0x7fee7c103e30 2024-04-25T00:15:15.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.754+0000 7fee73fff700 1 --2- 172.21.15.27:0/3736946167 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fee680a2c20 0x7fee680a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:15.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.754+0000 7fee73fff700 1 -- 172.21.15.27:0/3736946167 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fee6c0d00d0 con 0x7fee7c103e30 2024-04-25T00:15:15.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.754+0000 7fee7ad9d700 1 --2- 172.21.15.27:0/3736946167 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fee680a2c20 0x7fee680a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:15.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.754+0000 7fee7ad9d700 1 --2- 172.21.15.27:0/3736946167 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fee680a2c20 0x7fee680a50e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fee74009550 tx=0x7fee740091b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:15.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.758+0000 7fee73fff700 1 -- 172.21.15.27:0/3736946167 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fee6c013070 con 0x7fee7c103e30 2024-04-25T00:15:15.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:15 smithi153 ceph-mon[32896]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-25T00:15:15.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.973+0000 7fee810d8700 1 -- 172.21.15.27:0/3736946167 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fee7c062670 con 0x7fee680a2c20 2024-04-25T00:15:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.987+0000 7fee73fff700 1 -- 172.21.15.27:0/3736946167 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fee7c062670 con 0x7fee680a2c20 2024-04-25T00:15:15.988 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-25T00:15:15.988 INFO:teuthology.orchestra.run.smithi027.stdout:mds.a smithi027 running (2m) 25s ago 2m 18.4M - 19.0.0-2445-gd2c45d97 f68e37b807ef 08bfbb52e555 2024-04-25T00:15:15.988 INFO:teuthology.orchestra.run.smithi027.stdout:mds.b smithi139 running (2m) 56s ago 2m 26.1M - 19.0.0-2445-gd2c45d97 f68e37b807ef 3e317c4309d5 2024-04-25T00:15:15.988 INFO:teuthology.orchestra.run.smithi027.stdout:mds.c smithi153 running (2m) 56s ago 2m 17.1M - 19.0.0-2445-gd2c45d97 f68e37b807ef 1da4ca3353f4 2024-04-25T00:15:15.989 INFO:teuthology.orchestra.run.smithi027.stdout:mds.d smithi027 running (2m) 25s ago 2m 14.0M - 19.0.0-2445-gd2c45d97 f68e37b807ef def632097920 2024-04-25T00:15:15.989 INFO:teuthology.orchestra.run.smithi027.stdout:mds.e smithi139 running (2m) 56s ago 2m 15.1M - 19.0.0-2445-gd2c45d97 f68e37b807ef 8faab5e9836d 2024-04-25T00:15:15.989 INFO:teuthology.orchestra.run.smithi027.stdout:mds.f smithi153 running (2m) 56s ago 2m 17.5M - 19.0.0-2445-gd2c45d97 f68e37b807ef fe42e8756c60 2024-04-25T00:15:15.989 INFO:teuthology.orchestra.run.smithi027.stdout:mds.g smithi027 running (2m) 25s ago 2m 18.5M - 19.0.0-2445-gd2c45d97 f68e37b807ef 8ec929729bb6 2024-04-25T00:15:15.989 INFO:teuthology.orchestra.run.smithi027.stdout:mds.h smithi139 running (2m) 56s ago 2m 17.1M - 19.0.0-2445-gd2c45d97 f68e37b807ef 82223188b6cc 2024-04-25T00:15:15.989 INFO:teuthology.orchestra.run.smithi027.stdout:mds.i smithi153 running (2m) 56s ago 2m 17.2M - 19.0.0-2445-gd2c45d97 f68e37b807ef 4a115f16103e 2024-04-25T00:15:15.989 INFO:teuthology.orchestra.run.smithi027.stdout:mds.j smithi027 running (2m) 25s ago 2m 15.3M - 19.0.0-2445-gd2c45d97 f68e37b807ef 9926742db09d 2024-04-25T00:15:15.989 INFO:teuthology.orchestra.run.smithi027.stdout:mds.k smithi139 running (2m) 56s ago 2m 15.9M - 19.0.0-2445-gd2c45d97 f68e37b807ef 7024280520b2 2024-04-25T00:15:15.989 INFO:teuthology.orchestra.run.smithi027.stdout:mds.l smithi153 running (2m) 56s ago 2m 19.5M - 19.0.0-2445-gd2c45d97 f68e37b807ef c058c58a68b6 2024-04-25T00:15:15.989 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.x smithi027 *:9283,8765 running (9m) 25s ago 9m 514M - 19.0.0-2445-gd2c45d97 f68e37b807ef f7edeb76725e 2024-04-25T00:15:15.989 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.y smithi139 *:8443,8765 running (7m) 56s ago 7m 453M - 19.0.0-2445-gd2c45d97 f68e37b807ef 9ed73c881dff 2024-04-25T00:15:15.989 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.z smithi153 *:8443,8765 running (7m) 56s ago 7m 453M - 19.0.0-2445-gd2c45d97 f68e37b807ef 1b5a5bdb31ff 2024-04-25T00:15:15.989 INFO:teuthology.orchestra.run.smithi027.stdout:mon.a smithi027 running (10m) 25s ago 10m 50.2M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef f663a30b3ae1 2024-04-25T00:15:15.989 INFO:teuthology.orchestra.run.smithi027.stdout:mon.b smithi139 running (8m) 56s ago 8m 40.5M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef cb23fccae8ac 2024-04-25T00:15:15.990 INFO:teuthology.orchestra.run.smithi027.stdout:mon.c smithi153 running (8m) 56s ago 8m 39.8M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef d2ab362c9989 2024-04-25T00:15:15.990 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (7m) 25s ago 7m 46.3M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef b1fa48151373 2024-04-25T00:15:15.990 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi139 running (7m) 56s ago 7m 48.1M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 92c14b18aa89 2024-04-25T00:15:15.990 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi153 running (6m) 56s ago 6m 49.0M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 6782d8d0da7c 2024-04-25T00:15:15.990 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi027 running (6m) 25s ago 6m 55.3M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef d2ad0646c6da 2024-04-25T00:15:15.990 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi139 running (5m) 56s ago 5m 42.7M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef b63c667fe116 2024-04-25T00:15:15.990 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi153 running (5m) 56s ago 5m 54.5M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 301fa508b3ee 2024-04-25T00:15:15.990 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6 smithi027 running (5m) 25s ago 5m 42.2M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 3f295298506b 2024-04-25T00:15:15.990 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7 smithi139 running (4m) 56s ago 4m 44.3M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 4d6a184c52f3 2024-04-25T00:15:15.990 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8 smithi153 running (4m) 56s ago 4m 43.6M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef ab31b3015dc9 2024-04-25T00:15:15.990 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9 smithi027 running (3m) 25s ago 3m 42.5M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 9fde1ca8e12c 2024-04-25T00:15:15.990 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10 smithi139 running (3m) 56s ago 3m 43.1M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 263ff50ab294 2024-04-25T00:15:15.990 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11 smithi153 running (2m) 56s ago 2m 53.8M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef dd4490cb99aa 2024-04-25T00:15:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.990+0000 7fee810d8700 1 -- 172.21.15.27:0/3736946167 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fee680a2c20 msgr2=0x7fee680a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.990+0000 7fee810d8700 1 --2- 172.21.15.27:0/3736946167 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fee680a2c20 0x7fee680a50e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fee74009550 tx=0x7fee740091b0 comp rx=0 tx=0).stop 2024-04-25T00:15:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.990+0000 7fee810d8700 1 -- 172.21.15.27:0/3736946167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee7c103e30 msgr2=0x7fee7c19f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.990+0000 7fee810d8700 1 --2- 172.21.15.27:0/3736946167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee7c103e30 0x7fee7c19f990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fee6c009990 tx=0x7fee6c0047b0 comp rx=0 tx=0).stop 2024-04-25T00:15:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.990+0000 7fee810d8700 1 -- 172.21.15.27:0/3736946167 shutdown_connections 2024-04-25T00:15:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.990+0000 7fee810d8700 1 --2- 172.21.15.27:0/3736946167 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fee680a2c20 0x7fee680a50e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.990+0000 7fee810d8700 1 --2- 172.21.15.27:0/3736946167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee7c103e30 0x7fee7c19f990 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.990+0000 7fee810d8700 1 --2- 172.21.15.27:0/3736946167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee7c102290 0x7fee7c1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.990+0000 7fee810d8700 1 --2- 172.21.15.27:0/3736946167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee7c103490 0x7fee7c198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.990+0000 7fee810d8700 1 -- 172.21.15.27:0/3736946167 >> 172.21.15.27:0/3736946167 conn(0x7fee7c0fd830 msgr2=0x7fee7c106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.990+0000 7fee810d8700 1 -- 172.21.15.27:0/3736946167 shutdown_connections 2024-04-25T00:15:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:15.990+0000 7fee810d8700 1 -- 172.21.15.27:0/3736946167 wait complete. 2024-04-25T00:15:16.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:16 smithi027 ceph-mon[29208]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 639 B/s wr, 13 op/s 2024-04-25T00:15:16.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:16 smithi027 ceph-mon[29208]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:16.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:16 smithi139 ceph-mon[32833]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 639 B/s wr, 13 op/s 2024-04-25T00:15:16.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:16 smithi139 ceph-mon[32833]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:16.775 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-25T00:15:16.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:16 smithi153 ceph-mon[32896]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 639 B/s wr, 13 op/s 2024-04-25T00:15:16.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:16 smithi153 ceph-mon[32896]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:17.024 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:15:18.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.714+0000 7ff2100a0700 1 -- 172.21.15.27:0/2908927544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff208106060 msgr2=0x7ff20810c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:18.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.714+0000 7ff2100a0700 1 --2- 172.21.15.27:0/2908927544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff208106060 0x7ff20810c910 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff2000099c0 tx=0x7ff200009cd0 comp rx=0 tx=0).stop 2024-04-25T00:15:18.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.714+0000 7ff2100a0700 1 -- 172.21.15.27:0/2908927544 shutdown_connections 2024-04-25T00:15:18.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.714+0000 7ff2100a0700 1 --2- 172.21.15.27:0/2908927544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff208106060 0x7ff20810c910 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:18.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.714+0000 7ff2100a0700 1 --2- 172.21.15.27:0/2908927544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff2081044c0 0x7ff2081048e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:18.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.714+0000 7ff2100a0700 1 --2- 172.21.15.27:0/2908927544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2081056c0 0x7ff208105b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:18.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.714+0000 7ff2100a0700 1 -- 172.21.15.27:0/2908927544 >> 172.21.15.27:0/2908927544 conn(0x7ff2080ffa80 msgr2=0x7ff208101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:18.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.715+0000 7ff2100a0700 1 -- 172.21.15.27:0/2908927544 shutdown_connections 2024-04-25T00:15:18.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.715+0000 7ff2100a0700 1 -- 172.21.15.27:0/2908927544 wait complete. 2024-04-25T00:15:18.717 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:18 smithi027 ceph-mon[29208]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s 2024-04-25T00:15:18.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.715+0000 7ff2100a0700 1 Processor -- start 2024-04-25T00:15:18.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.715+0000 7ff2100a0700 1 -- start start 2024-04-25T00:15:18.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff2100a0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2081044c0 0x7ff20819a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:18.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff2100a0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2081056c0 0x7ff20819ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:18.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff2100a0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff208106060 0x7ff2081a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:18.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff2100a0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff2081100b0 con 0x7ff2081056c0 2024-04-25T00:15:18.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff2100a0700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff20810ff30 con 0x7ff208106060 2024-04-25T00:15:18.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff2100a0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff208110230 con 0x7ff2081044c0 2024-04-25T00:15:18.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff20d63b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2081056c0 0x7ff20819ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:18.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff20d63b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2081056c0 0x7ff20819ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54920/0 (socket says 172.21.15.27:54920) 2024-04-25T00:15:18.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff20d63b700 1 -- 172.21.15.27:0/3992839399 learned_addr learned my addr 172.21.15.27:0/3992839399 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:18.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff20de3c700 1 --2- 172.21.15.27:0/3992839399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2081044c0 0x7ff20819a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:18.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff20e63d700 1 --2- 172.21.15.27:0/3992839399 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff208106060 0x7ff2081a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:18.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff20d63b700 1 -- 172.21.15.27:0/3992839399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2081044c0 msgr2=0x7ff20819a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:18.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff20d63b700 1 --2- 172.21.15.27:0/3992839399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2081044c0 0x7ff20819a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:18.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff20d63b700 1 -- 172.21.15.27:0/3992839399 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff208106060 msgr2=0x7ff2081a1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:18.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff20d63b700 1 --2- 172.21.15.27:0/3992839399 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff208106060 0x7ff2081a1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:18.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff20d63b700 1 -- 172.21.15.27:0/3992839399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2081a2230 con 0x7ff2081056c0 2024-04-25T00:15:18.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff20e63d700 1 --2- 172.21.15.27:0/3992839399 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff208106060 0x7ff2081a1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:18.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff20d63b700 1 --2- 172.21.15.27:0/3992839399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2081056c0 0x7ff20819ab10 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc00e9f0 tx=0x7ff1fc00edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:18.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.716+0000 7ff206ffd700 1 -- 172.21.15.27:0/3992839399 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc00cbd0 con 0x7ff2081056c0 2024-04-25T00:15:18.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.717+0000 7ff2100a0700 1 -- 172.21.15.27:0/3992839399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2081a2520 con 0x7ff2081056c0 2024-04-25T00:15:18.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.717+0000 7ff206ffd700 1 -- 172.21.15.27:0/3992839399 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1fc00cd70 con 0x7ff2081056c0 2024-04-25T00:15:18.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.717+0000 7ff206ffd700 1 -- 172.21.15.27:0/3992839399 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc010430 con 0x7ff2081056c0 2024-04-25T00:15:18.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.717+0000 7ff2100a0700 1 -- 172.21.15.27:0/3992839399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2081a2a60 con 0x7ff2081056c0 2024-04-25T00:15:18.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.718+0000 7ff2100a0700 1 -- 172.21.15.27:0/3992839399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff208068f20 con 0x7ff2081056c0 2024-04-25T00:15:18.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.719+0000 7ff206ffd700 1 -- 172.21.15.27:0/3992839399 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff1fc004750 con 0x7ff2081056c0 2024-04-25T00:15:18.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.720+0000 7ff206ffd700 1 --2- 172.21.15.27:0/3992839399 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff1f00a2ca0 0x7ff1f00a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:18.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.720+0000 7ff20de3c700 1 --2- 172.21.15.27:0/3992839399 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff1f00a2ca0 0x7ff1f00a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:18.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.720+0000 7ff206ffd700 1 -- 172.21.15.27:0/3992839399 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff1fc0cf750 con 0x7ff2081056c0 2024-04-25T00:15:18.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.720+0000 7ff20de3c700 1 --2- 172.21.15.27:0/3992839399 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff1f00a2ca0 0x7ff1f00a5160 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff208105520 tx=0x7ff1f800b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:18.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.724+0000 7ff206ffd700 1 -- 172.21.15.27:0/3992839399 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff1fc09c320 con 0x7ff2081056c0 2024-04-25T00:15:18.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.940+0000 7ff2100a0700 1 -- 172.21.15.27:0/3992839399 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7ff208062670 con 0x7ff1f00a2ca0 2024-04-25T00:15:18.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.946+0000 7ff206ffd700 1 -- 172.21.15.27:0/3992839399 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7ff208062670 con 0x7ff1f00a2ca0 2024-04-25T00:15:18.948 INFO:teuthology.orchestra.run.smithi027.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-25T00:15:18.948 INFO:teuthology.orchestra.run.smithi027.stdout:mds.all 12/12 59s ago 2m smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l;count:12 2024-04-25T00:15:18.948 INFO:teuthology.orchestra.run.smithi027.stdout:mgr 3/3 59s ago 8m smithi027=x;smithi139=y;smithi153=z;count:3 2024-04-25T00:15:18.948 INFO:teuthology.orchestra.run.smithi027.stdout:mon 3/3 59s ago 8m smithi027:172.21.15.27=a;smithi139:172.21.15.139=b;smithi153:172.21.15.153=c;count:3 2024-04-25T00:15:18.948 INFO:teuthology.orchestra.run.smithi027.stdout:osd 12 59s ago - 2024-04-25T00:15:18.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.949+0000 7ff2100a0700 1 -- 172.21.15.27:0/3992839399 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff1f00a2ca0 msgr2=0x7ff1f00a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:18.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.949+0000 7ff2100a0700 1 --2- 172.21.15.27:0/3992839399 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff1f00a2ca0 0x7ff1f00a5160 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff208105520 tx=0x7ff1f800b040 comp rx=0 tx=0).stop 2024-04-25T00:15:18.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.949+0000 7ff2100a0700 1 -- 172.21.15.27:0/3992839399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2081056c0 msgr2=0x7ff20819ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:18.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.949+0000 7ff2100a0700 1 --2- 172.21.15.27:0/3992839399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2081056c0 0x7ff20819ab10 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc00e9f0 tx=0x7ff1fc00edb0 comp rx=0 tx=0).stop 2024-04-25T00:15:18.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.949+0000 7ff2100a0700 1 -- 172.21.15.27:0/3992839399 shutdown_connections 2024-04-25T00:15:18.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.949+0000 7ff2100a0700 1 --2- 172.21.15.27:0/3992839399 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7ff1f00a2ca0 0x7ff1f00a5160 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:18.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.949+0000 7ff2100a0700 1 --2- 172.21.15.27:0/3992839399 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff208106060 0x7ff2081a1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:18.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.949+0000 7ff2100a0700 1 --2- 172.21.15.27:0/3992839399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2081044c0 0x7ff20819a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:18.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.949+0000 7ff2100a0700 1 --2- 172.21.15.27:0/3992839399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2081056c0 0x7ff20819ab10 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:18.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.949+0000 7ff2100a0700 1 -- 172.21.15.27:0/3992839399 >> 172.21.15.27:0/3992839399 conn(0x7ff2080ffa80 msgr2=0x7ff2081088f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:18.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.949+0000 7ff2100a0700 1 -- 172.21.15.27:0/3992839399 shutdown_connections 2024-04-25T00:15:18.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:18.949+0000 7ff2100a0700 1 -- 172.21.15.27:0/3992839399 wait complete. 2024-04-25T00:15:19.005 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:18 smithi139 ceph-mon[32833]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s 2024-04-25T00:15:19.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:18 smithi153 ceph-mon[32896]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s 2024-04-25T00:15:19.661 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-25T00:15:19.914 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:15:19.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:19 smithi027 ceph-mon[29208]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:20.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:19 smithi139 ceph-mon[32833]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:20.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:19 smithi153 ceph-mon[32896]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:20.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:20 smithi027 ceph-mon[29208]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 445 B/s wr, 10 op/s 2024-04-25T00:15:21.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:20 smithi139 ceph-mon[32833]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 445 B/s wr, 10 op/s 2024-04-25T00:15:21.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:20 smithi153 ceph-mon[32896]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 445 B/s wr, 10 op/s 2024-04-25T00:15:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.665+0000 7fba41452700 1 -- 172.21.15.27:0/3799737598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba3c1022b0 msgr2=0x7fba3c1026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.665+0000 7fba41452700 1 --2- 172.21.15.27:0/3799737598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba3c1022b0 0x7fba3c1026d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0099c0 tx=0x7fba2c009cd0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.665+0000 7fba41452700 1 -- 172.21.15.27:0/3799737598 shutdown_connections 2024-04-25T00:15:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.665+0000 7fba41452700 1 --2- 172.21.15.27:0/3799737598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3c103e50 0x7fba3c10a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.665+0000 7fba41452700 1 --2- 172.21.15.27:0/3799737598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba3c1022b0 0x7fba3c1026d0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.665+0000 7fba41452700 1 --2- 172.21.15.27:0/3799737598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba3c1034b0 0x7fba3c103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.665+0000 7fba41452700 1 -- 172.21.15.27:0/3799737598 >> 172.21.15.27:0/3799737598 conn(0x7fba3c0fd850 msgr2=0x7fba3c0ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.665+0000 7fba41452700 1 -- 172.21.15.27:0/3799737598 shutdown_connections 2024-04-25T00:15:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.665+0000 7fba41452700 1 -- 172.21.15.27:0/3799737598 wait complete. 2024-04-25T00:15:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.665+0000 7fba41452700 1 Processor -- start 2024-04-25T00:15:21.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba41452700 1 -- start start 2024-04-25T00:15:21.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba41452700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba3c1022b0 0x7fba3c198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:21.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba41452700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba3c1034b0 0x7fba3c198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:21.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba41452700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3c103e50 0x7fba3c19f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:21.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba41452700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c10de70 con 0x7fba3c1034b0 2024-04-25T00:15:21.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba41452700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c10dcf0 con 0x7fba3c1022b0 2024-04-25T00:15:21.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba41452700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c10dff0 con 0x7fba3c103e50 2024-04-25T00:15:21.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba3a7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba3c1034b0 0x7fba3c198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:21.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba3a7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba3c1034b0 0x7fba3c198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54936/0 (socket says 172.21.15.27:54936) 2024-04-25T00:15:21.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba3a7fc700 1 -- 172.21.15.27:0/1988801310 learned_addr learned my addr 172.21.15.27:0/1988801310 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:21.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba3affd700 1 --2- 172.21.15.27:0/1988801310 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba3c1022b0 0x7fba3c198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:21.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba3b7fe700 1 --2- 172.21.15.27:0/1988801310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3c103e50 0x7fba3c19f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:21.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba3a7fc700 1 -- 172.21.15.27:0/1988801310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3c103e50 msgr2=0x7fba3c19f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:21.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba3a7fc700 1 --2- 172.21.15.27:0/1988801310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3c103e50 0x7fba3c19f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba3a7fc700 1 -- 172.21.15.27:0/1988801310 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba3c1022b0 msgr2=0x7fba3c198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:21.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba3a7fc700 1 --2- 172.21.15.27:0/1988801310 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba3c1022b0 0x7fba3c198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba3a7fc700 1 -- 172.21.15.27:0/1988801310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba3c1a0090 con 0x7fba3c1034b0 2024-04-25T00:15:21.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.666+0000 7fba3b7fe700 1 --2- 172.21.15.27:0/1988801310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3c103e50 0x7fba3c19f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:21.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.667+0000 7fba3a7fc700 1 --2- 172.21.15.27:0/1988801310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba3c1034b0 0x7fba3c198970 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fba3000d980 tx=0x7fba3000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:21.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.667+0000 7fba23fff700 1 -- 172.21.15.27:0/1988801310 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba30016070 con 0x7fba3c1034b0 2024-04-25T00:15:21.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.667+0000 7fba23fff700 1 -- 172.21.15.27:0/1988801310 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba3000de70 con 0x7fba3c1034b0 2024-04-25T00:15:21.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.667+0000 7fba41452700 1 -- 172.21.15.27:0/1988801310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba3c1a0380 con 0x7fba3c1034b0 2024-04-25T00:15:21.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.667+0000 7fba23fff700 1 -- 172.21.15.27:0/1988801310 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3000f3c0 con 0x7fba3c1034b0 2024-04-25T00:15:21.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.667+0000 7fba41452700 1 -- 172.21.15.27:0/1988801310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba3c1a0910 con 0x7fba3c1034b0 2024-04-25T00:15:21.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.668+0000 7fba41452700 1 -- 172.21.15.27:0/1988801310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba3c068f20 con 0x7fba3c1034b0 2024-04-25T00:15:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.670+0000 7fba23fff700 1 -- 172.21.15.27:0/1988801310 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fba30019070 con 0x7fba3c1034b0 2024-04-25T00:15:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.670+0000 7fba23fff700 1 --2- 172.21.15.27:0/1988801310 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fba240a2ca0 0x7fba240a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.670+0000 7fba3affd700 1 --2- 172.21.15.27:0/1988801310 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fba240a2ca0 0x7fba240a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.671+0000 7fba23fff700 1 -- 172.21.15.27:0/1988801310 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fba30009b40 con 0x7fba3c1034b0 2024-04-25T00:15:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.671+0000 7fba3affd700 1 --2- 172.21.15.27:0/1988801310 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fba240a2ca0 0x7fba240a5160 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0039c0 tx=0x7fba2c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:21.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.676+0000 7fba23fff700 1 -- 172.21.15.27:0/1988801310 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fba3009ba10 con 0x7fba3c1034b0 2024-04-25T00:15:21.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.892+0000 7fba41452700 1 -- 172.21.15.27:0/1988801310 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fba3c062670 con 0x7fba240a2ca0 2024-04-25T00:15:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.893+0000 7fba23fff700 1 -- 172.21.15.27:0/1988801310 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fba3c062670 con 0x7fba240a2ca0 2024-04-25T00:15:21.894 INFO:teuthology.orchestra.run.smithi027.stdout:HOST ADDR LABELS STATUS 2024-04-25T00:15:21.894 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 172.21.15.27 2024-04-25T00:15:21.894 INFO:teuthology.orchestra.run.smithi027.stdout:smithi139 172.21.15.139 2024-04-25T00:15:21.894 INFO:teuthology.orchestra.run.smithi027.stdout:smithi153 172.21.15.153 2024-04-25T00:15:21.894 INFO:teuthology.orchestra.run.smithi027.stdout:3 hosts in cluster 2024-04-25T00:15:21.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.895+0000 7fba41452700 1 -- 172.21.15.27:0/1988801310 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fba240a2ca0 msgr2=0x7fba240a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:21.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.895+0000 7fba41452700 1 --2- 172.21.15.27:0/1988801310 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fba240a2ca0 0x7fba240a5160 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0039c0 tx=0x7fba2c005fb0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.895+0000 7fba41452700 1 -- 172.21.15.27:0/1988801310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba3c1034b0 msgr2=0x7fba3c198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:21.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.895+0000 7fba41452700 1 --2- 172.21.15.27:0/1988801310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba3c1034b0 0x7fba3c198970 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fba3000d980 tx=0x7fba3000dc90 comp rx=0 tx=0).stop 2024-04-25T00:15:21.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.895+0000 7fba41452700 1 -- 172.21.15.27:0/1988801310 shutdown_connections 2024-04-25T00:15:21.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.895+0000 7fba41452700 1 --2- 172.21.15.27:0/1988801310 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fba240a2ca0 0x7fba240a5160 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.895+0000 7fba41452700 1 --2- 172.21.15.27:0/1988801310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3c103e50 0x7fba3c19f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.895+0000 7fba41452700 1 --2- 172.21.15.27:0/1988801310 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba3c1022b0 0x7fba3c198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.895+0000 7fba41452700 1 --2- 172.21.15.27:0/1988801310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba3c1034b0 0x7fba3c198970 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.895+0000 7fba41452700 1 -- 172.21.15.27:0/1988801310 >> 172.21.15.27:0/1988801310 conn(0x7fba3c0fd850 msgr2=0x7fba3c1067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:21.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.895+0000 7fba41452700 1 -- 172.21.15.27:0/1988801310 shutdown_connections 2024-04-25T00:15:21.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:21.895+0000 7fba41452700 1 -- 172.21.15.27:0/1988801310 wait complete. 2024-04-25T00:15:22.711 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-25T00:15:22.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:22 smithi027 ceph-mon[29208]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 402 B/s wr, 10 op/s 2024-04-25T00:15:22.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:22 smithi027 ceph-mon[29208]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:22.972 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:15:23.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:22 smithi139 ceph-mon[32833]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 402 B/s wr, 10 op/s 2024-04-25T00:15:23.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:22 smithi139 ceph-mon[32833]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:23.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:22 smithi153 ceph-mon[32896]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 402 B/s wr, 10 op/s 2024-04-25T00:15:23.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:22 smithi153 ceph-mon[32896]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.724+0000 7fed42b3c700 1 -- 172.21.15.27:0/3339831036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed3c103490 msgr2=0x7fed3c1038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.724+0000 7fed42b3c700 1 --2- 172.21.15.27:0/3339831036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed3c103490 0x7fed3c1038f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fed280099c0 tx=0x7fed28009cd0 comp rx=0 tx=0).stop 2024-04-25T00:15:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.724+0000 7fed42b3c700 1 -- 172.21.15.27:0/3339831036 shutdown_connections 2024-04-25T00:15:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.724+0000 7fed42b3c700 1 --2- 172.21.15.27:0/3339831036 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed3c103e30 0x7fed3c10a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.724+0000 7fed42b3c700 1 --2- 172.21.15.27:0/3339831036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed3c102290 0x7fed3c1026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.724+0000 7fed42b3c700 1 --2- 172.21.15.27:0/3339831036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed3c103490 0x7fed3c1038f0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.724+0000 7fed42b3c700 1 -- 172.21.15.27:0/3339831036 >> 172.21.15.27:0/3339831036 conn(0x7fed3c0fd830 msgr2=0x7fed3c0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.724+0000 7fed42b3c700 1 -- 172.21.15.27:0/3339831036 shutdown_connections 2024-04-25T00:15:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.724+0000 7fed42b3c700 1 -- 172.21.15.27:0/3339831036 wait complete. 2024-04-25T00:15:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.724+0000 7fed42b3c700 1 Processor -- start 2024-04-25T00:15:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.724+0000 7fed42b3c700 1 -- start start 2024-04-25T00:15:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.724+0000 7fed42b3c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed3c102290 0x7fed3c198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.724+0000 7fed42b3c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed3c103490 0x7fed3c198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.724+0000 7fed42b3c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed3c103e30 0x7fed3c19f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.724+0000 7fed42b3c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c10de50 con 0x7fed3c102290 2024-04-25T00:15:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.725+0000 7fed42b3c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c10dcd0 con 0x7fed3c103490 2024-04-25T00:15:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.725+0000 7fed42b3c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c10dfd0 con 0x7fed3c103e30 2024-04-25T00:15:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.725+0000 7fed408d8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed3c102290 0x7fed3c198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.725+0000 7fed3bfff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed3c103490 0x7fed3c198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.725+0000 7fed408d8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed3c102290 0x7fed3c198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54954/0 (socket says 172.21.15.27:54954) 2024-04-25T00:15:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.725+0000 7fed408d8700 1 -- 172.21.15.27:0/3269631851 learned_addr learned my addr 172.21.15.27:0/3269631851 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.725+0000 7fed410d9700 1 --2- 172.21.15.27:0/3269631851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed3c103e30 0x7fed3c19f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.725+0000 7fed408d8700 1 -- 172.21.15.27:0/3269631851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed3c103e30 msgr2=0x7fed3c19f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.725+0000 7fed408d8700 1 --2- 172.21.15.27:0/3269631851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed3c103e30 0x7fed3c19f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.725+0000 7fed408d8700 1 -- 172.21.15.27:0/3269631851 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed3c103490 msgr2=0x7fed3c198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.725+0000 7fed408d8700 1 --2- 172.21.15.27:0/3269631851 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed3c103490 0x7fed3c198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.725+0000 7fed408d8700 1 -- 172.21.15.27:0/3269631851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3c1a0060 con 0x7fed3c102290 2024-04-25T00:15:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.725+0000 7fed408d8700 1 --2- 172.21.15.27:0/3269631851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed3c102290 0x7fed3c198400 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fed3000c9c0 tx=0x7fed3000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.725+0000 7fed39ffb700 1 -- 172.21.15.27:0/3269631851 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3000ce60 con 0x7fed3c102290 2024-04-25T00:15:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.725+0000 7fed42b3c700 1 -- 172.21.15.27:0/3269631851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed3c0744c0 con 0x7fed3c102290 2024-04-25T00:15:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.725+0000 7fed39ffb700 1 -- 172.21.15.27:0/3269631851 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed3000f450 con 0x7fed3c102290 2024-04-25T00:15:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.725+0000 7fed39ffb700 1 -- 172.21.15.27:0/3269631851 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed30018660 con 0x7fed3c102290 2024-04-25T00:15:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.726+0000 7fed42b3c700 1 -- 172.21.15.27:0/3269631851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed3c074a00 con 0x7fed3c102290 2024-04-25T00:15:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.727+0000 7fed42b3c700 1 -- 172.21.15.27:0/3269631851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed3c068f20 con 0x7fed3c102290 2024-04-25T00:15:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.728+0000 7fed39ffb700 1 -- 172.21.15.27:0/3269631851 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fed30007ac0 con 0x7fed3c102290 2024-04-25T00:15:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.728+0000 7fed39ffb700 1 --2- 172.21.15.27:0/3269631851 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fed2c0a2c20 0x7fed2c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.729+0000 7fed3bfff700 1 --2- 172.21.15.27:0/3269631851 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fed2c0a2c20 0x7fed2c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.729+0000 7fed39ffb700 1 -- 172.21.15.27:0/3269631851 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fed300cf290 con 0x7fed3c102290 2024-04-25T00:15:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.729+0000 7fed3bfff700 1 --2- 172.21.15.27:0/3269631851 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fed2c0a2c20 0x7fed2c0a50e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fed28000c00 tx=0x7fed28005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.733+0000 7fed39ffb700 1 -- 172.21.15.27:0/3269631851 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fed30014030 con 0x7fed3c102290 2024-04-25T00:15:24.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:24 smithi027 ceph-mon[29208]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-25T00:15:24.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.949+0000 7fed42b3c700 1 -- 172.21.15.27:0/3269631851 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fed3c075eb0 con 0x7fed2c0a2c20 2024-04-25T00:15:24.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.951+0000 7fed39ffb700 1 -- 172.21.15.27:0/3269631851 <== mgr.14150 v2:172.21.15.27:6800/2161554778 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fed3c075eb0 con 0x7fed2c0a2c20 2024-04-25T00:15:24.952 INFO:teuthology.orchestra.run.smithi027.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-25T00:15:24.952 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331001C400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-25T00:15:24.952 INFO:teuthology.orchestra.run.smithi027.stdout:smithi139 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500N6400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-25T00:15:24.952 INFO:teuthology.orchestra.run.smithi027.stdout:smithi153 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300MV400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-25T00:15:24.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.953+0000 7fed42b3c700 1 -- 172.21.15.27:0/3269631851 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fed2c0a2c20 msgr2=0x7fed2c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:24.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.953+0000 7fed42b3c700 1 --2- 172.21.15.27:0/3269631851 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fed2c0a2c20 0x7fed2c0a50e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fed28000c00 tx=0x7fed28005fb0 comp rx=0 tx=0).stop 2024-04-25T00:15:24.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.953+0000 7fed42b3c700 1 -- 172.21.15.27:0/3269631851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed3c102290 msgr2=0x7fed3c198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:24.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.953+0000 7fed42b3c700 1 --2- 172.21.15.27:0/3269631851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed3c102290 0x7fed3c198400 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fed3000c9c0 tx=0x7fed3000ccd0 comp rx=0 tx=0).stop 2024-04-25T00:15:24.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.953+0000 7fed42b3c700 1 -- 172.21.15.27:0/3269631851 shutdown_connections 2024-04-25T00:15:24.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.953+0000 7fed42b3c700 1 --2- 172.21.15.27:0/3269631851 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fed2c0a2c20 0x7fed2c0a50e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:24.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.953+0000 7fed42b3c700 1 --2- 172.21.15.27:0/3269631851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed3c103e30 0x7fed3c19f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:24.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.953+0000 7fed42b3c700 1 --2- 172.21.15.27:0/3269631851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed3c102290 0x7fed3c198400 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:24.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.953+0000 7fed42b3c700 1 --2- 172.21.15.27:0/3269631851 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed3c103490 0x7fed3c198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:24.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.953+0000 7fed42b3c700 1 -- 172.21.15.27:0/3269631851 >> 172.21.15.27:0/3269631851 conn(0x7fed3c0fd830 msgr2=0x7fed3c106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:24.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.954+0000 7fed42b3c700 1 -- 172.21.15.27:0/3269631851 shutdown_connections 2024-04-25T00:15:24.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:24.954+0000 7fed42b3c700 1 -- 172.21.15.27:0/3269631851 wait complete. 2024-04-25T00:15:25.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:24 smithi139 ceph-mon[32833]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-25T00:15:25.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:24 smithi153 ceph-mon[32896]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-25T00:15:25.671 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T00:15:25.681 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T00:15:25.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-25T00:15:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:15:25.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:25 smithi027 ceph-mon[29208]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:26.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:25 smithi139 ceph-mon[32833]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:26.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:25 smithi153 ceph-mon[32896]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:26.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:26 smithi027 ceph-mon[29208]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:15:27.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:26 smithi139 ceph-mon[32833]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:15:27.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:26 smithi153 ceph-mon[32896]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:15:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.595+0000 7f0731382700 1 -- 172.21.15.27:0/1031486005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f072c1034b0 msgr2=0x7f072c103910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.595+0000 7f0731382700 1 --2- 172.21.15.27:0/1031486005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f072c1034b0 0x7f072c103910 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f07200099c0 tx=0x7f0720009cd0 comp rx=0 tx=0).stop 2024-04-25T00:15:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.595+0000 7f0731382700 1 -- 172.21.15.27:0/1031486005 shutdown_connections 2024-04-25T00:15:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.595+0000 7f0731382700 1 --2- 172.21.15.27:0/1031486005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f072c103e50 0x7f072c10a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.595+0000 7f0731382700 1 --2- 172.21.15.27:0/1031486005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f072c1022b0 0x7f072c1026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.595+0000 7f0731382700 1 --2- 172.21.15.27:0/1031486005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f072c1034b0 0x7f072c103910 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.595+0000 7f0731382700 1 -- 172.21.15.27:0/1031486005 >> 172.21.15.27:0/1031486005 conn(0x7f072c0fd850 msgr2=0x7f072c0ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.595+0000 7f0731382700 1 -- 172.21.15.27:0/1031486005 shutdown_connections 2024-04-25T00:15:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.595+0000 7f0731382700 1 -- 172.21.15.27:0/1031486005 wait complete. 2024-04-25T00:15:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.595+0000 7f0731382700 1 Processor -- start 2024-04-25T00:15:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f0731382700 1 -- start start 2024-04-25T00:15:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f0731382700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f072c1022b0 0x7f072c1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f0731382700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f072c1034b0 0x7f072c198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f0731382700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f072c103e50 0x7f072c19fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f0731382700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f072c10deb0 con 0x7f072c103e50 2024-04-25T00:15:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f0731382700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f072c10dd30 con 0x7f072c1022b0 2024-04-25T00:15:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f0731382700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f072c10e030 con 0x7f072c1034b0 2024-04-25T00:15:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f072b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f072c103e50 0x7f072c19fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f072b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f072c103e50 0x7f072c19fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56508/0 (socket says 172.21.15.27:56508) 2024-04-25T00:15:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f072b7fe700 1 -- 172.21.15.27:0/3942853235 learned_addr learned my addr 172.21.15.27:0/3942853235 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f072affd700 1 --2- 172.21.15.27:0/3942853235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f072c1022b0 0x7f072c1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f072a7fc700 1 --2- 172.21.15.27:0/3942853235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f072c1034b0 0x7f072c198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f072b7fe700 1 -- 172.21.15.27:0/3942853235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f072c1034b0 msgr2=0x7f072c198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f072b7fe700 1 --2- 172.21.15.27:0/3942853235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f072c1034b0 0x7f072c198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f072b7fe700 1 -- 172.21.15.27:0/3942853235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f072c1022b0 msgr2=0x7f072c1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f072b7fe700 1 --2- 172.21.15.27:0/3942853235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f072c1022b0 0x7f072c1984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:27.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f072b7fe700 1 -- 172.21.15.27:0/3942853235 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072c1a0120 con 0x7f072c103e50 2024-04-25T00:15:27.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f072a7fc700 1 --2- 172.21.15.27:0/3942853235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f072c1034b0 0x7f072c198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:27.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.596+0000 7f072affd700 1 --2- 172.21.15.27:0/3942853235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f072c1022b0 0x7f072c1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:27.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.597+0000 7f072b7fe700 1 --2- 172.21.15.27:0/3942853235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f072c103e50 0x7f072c19fa80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f072400add0 tx=0x7f072400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:27.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.597+0000 7f0713fff700 1 -- 172.21.15.27:0/3942853235 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724017020 con 0x7f072c103e50 2024-04-25T00:15:27.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.597+0000 7f0731382700 1 -- 172.21.15.27:0/3942853235 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f072c1a0410 con 0x7f072c103e50 2024-04-25T00:15:27.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.597+0000 7f0713fff700 1 -- 172.21.15.27:0/3942853235 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f072400ce20 con 0x7f072c103e50 2024-04-25T00:15:27.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.597+0000 7f0713fff700 1 -- 172.21.15.27:0/3942853235 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724005520 con 0x7f072c103e50 2024-04-25T00:15:27.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.597+0000 7f0731382700 1 -- 172.21.15.27:0/3942853235 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f072c1a0900 con 0x7f072c103e50 2024-04-25T00:15:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.598+0000 7f0731382700 1 -- 172.21.15.27:0/3942853235 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f072c068f20 con 0x7f072c103e50 2024-04-25T00:15:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.599+0000 7f0713fff700 1 -- 172.21.15.27:0/3942853235 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0724004ad0 con 0x7f072c103e50 2024-04-25T00:15:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.600+0000 7f0713fff700 1 --2- 172.21.15.27:0/3942853235 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f07140a2c20 0x7f07140a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.600+0000 7f072affd700 1 --2- 172.21.15.27:0/3942853235 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f07140a2c20 0x7f07140a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.600+0000 7f0713fff700 1 -- 172.21.15.27:0/3942853235 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f07240cebf0 con 0x7f072c103e50 2024-04-25T00:15:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.600+0000 7f072affd700 1 --2- 172.21.15.27:0/3942853235 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f07140a2c20 0x7f07140a50e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f071c005950 tx=0x7f071c0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:27.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.604+0000 7f0713fff700 1 -- 172.21.15.27:0/3942853235 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0724021020 con 0x7f072c103e50 2024-04-25T00:15:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.861+0000 7f0731382700 1 -- 172.21.15.27:0/3942853235 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f072c062e40 con 0x7f072c103e50 2024-04-25T00:15:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.861+0000 7f0713fff700 1 -- 172.21.15.27:0/3942853235 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3100 (secure 0 0 0) 0x7f072409b850 con 0x7f072c103e50 2024-04-25T00:15:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:e28 2024-04-25T00:15:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T00:15:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T00:15:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-25T00:15:27.863 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-25T00:15:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-25T00:15:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 28 2024-04-25T00:15:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-25T00:15:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-25T00:13:18.872210+0000 2024-04-25T00:15:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-25T00:13:47.575422+0000 2024-04-25T00:15:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-25T00:15:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-25T00:15:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 300 2024-04-25T00:15:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-25T00:15:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-25T00:15:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-25T00:15:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-25T00:15:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-25T00:15:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 0 2024-04-25T00:15:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T00:15:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 5 2024-04-25T00:15:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:in 0,1,2,3,4 2024-04-25T00:15:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=24521,1=24514,2=24496,3=24430,4=24428} 2024-04-25T00:15:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-25T00:15:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-25T00:15:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-25T00:15:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3,4] 2024-04-25T00:15:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-25T00:15:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-25T00:15:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-25T00:15:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-25T00:15:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 0 2024-04-25T00:15:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 24521 members: 24428,24496,24521,24430,24514 2024-04-25T00:15:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b{0:24521} state up:active seq 3 addr [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T00:15:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.h{0:24488} state up:standby-replay seq 1 addr [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T00:15:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.i{1:24514} state up:active seq 11 addr [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T00:15:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.f{1:24478} state up:standby-replay seq 1 addr [v2:172.21.15.153:6834/584718807,v1:172.21.15.153:6835/584718807] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T00:15:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c{2:24496} state up:active seq 13 addr [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T00:15:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.e{2:24461} state up:standby-replay seq 1 addr [v2:172.21.15.139:6834/3731284556,v1:172.21.15.139:6835/3731284556] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T00:15:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.l{3:24430} state up:active seq 18 addr [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T00:15:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a{3:14664} state up:standby-replay seq 1 addr [v2:172.21.15.27:6840/2271898034,v1:172.21.15.27:6841/2271898034] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T00:15:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.k{4:24428} state up:active seq 17 addr [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T00:15:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.g{4:14652} state up:standby-replay seq 1 addr [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T00:15:27.866 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:27.866 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-25T00:15:27.866 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.j{-1:14574} state up:standby seq 1 addr [v2:172.21.15.27:6834/2280519367,v1:172.21.15.27:6835/2280519367] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T00:15:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d{-1:14586} state up:standby seq 1 addr [v2:172.21.15.27:6836/590015015,v1:172.21.15.27:6837/590015015] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T00:15:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.863+0000 7f0731382700 1 -- 172.21.15.27:0/3942853235 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f07140a2c20 msgr2=0x7f07140a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.863+0000 7f0731382700 1 --2- 172.21.15.27:0/3942853235 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f07140a2c20 0x7f07140a50e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f071c005950 tx=0x7f071c0058e0 comp rx=0 tx=0).stop 2024-04-25T00:15:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.863+0000 7f0731382700 1 -- 172.21.15.27:0/3942853235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f072c103e50 msgr2=0x7f072c19fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.863+0000 7f0731382700 1 --2- 172.21.15.27:0/3942853235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f072c103e50 0x7f072c19fa80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f072400add0 tx=0x7f072400c330 comp rx=0 tx=0).stop 2024-04-25T00:15:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.864+0000 7f0731382700 1 -- 172.21.15.27:0/3942853235 shutdown_connections 2024-04-25T00:15:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.864+0000 7f0731382700 1 --2- 172.21.15.27:0/3942853235 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f07140a2c20 0x7f07140a50e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.864+0000 7f0731382700 1 --2- 172.21.15.27:0/3942853235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f072c103e50 0x7f072c19fa80 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.864+0000 7f0731382700 1 --2- 172.21.15.27:0/3942853235 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f072c1022b0 0x7f072c1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.864+0000 7f0731382700 1 --2- 172.21.15.27:0/3942853235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f072c1034b0 0x7f072c198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.864+0000 7f0731382700 1 -- 172.21.15.27:0/3942853235 >> 172.21.15.27:0/3942853235 conn(0x7f072c0fd850 msgr2=0x7f072c1067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.864+0000 7f0731382700 1 -- 172.21.15.27:0/3942853235 shutdown_connections 2024-04-25T00:15:27.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:27.864+0000 7f0731382700 1 -- 172.21.15.27:0/3942853235 wait complete. 2024-04-25T00:15:27.868 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 28 2024-04-25T00:15:28.531 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-25T00:15:28.785 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:15:28.906 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:28 smithi027 ceph-mon[29208]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:15:28.906 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:28 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3942853235' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T00:15:29.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:28 smithi139 ceph-mon[32833]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:15:29.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:28 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3942853235' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T00:15:29.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:28 smithi153 ceph-mon[32896]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:15:29.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:28 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3942853235' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T00:15:30.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.526+0000 7fc92d12f700 1 -- 172.21.15.27:0/3123421102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc928103e50 msgr2=0x7fc92810a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:30.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.526+0000 7fc92d12f700 1 --2- 172.21.15.27:0/3123421102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc928103e50 0x7fc92810a700 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc92000b0d0 tx=0x7fc92000b3e0 comp rx=0 tx=0).stop 2024-04-25T00:15:30.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.526+0000 7fc92d12f700 1 -- 172.21.15.27:0/3123421102 shutdown_connections 2024-04-25T00:15:30.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.526+0000 7fc92d12f700 1 --2- 172.21.15.27:0/3123421102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc928103e50 0x7fc92810a700 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:30.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.526+0000 7fc92d12f700 1 --2- 172.21.15.27:0/3123421102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9281022b0 0x7fc9281026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.526+0000 7fc92d12f700 1 --2- 172.21.15.27:0/3123421102 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9281034b0 0x7fc928103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.526+0000 7fc92d12f700 1 -- 172.21.15.27:0/3123421102 >> 172.21.15.27:0/3123421102 conn(0x7fc9280fd850 msgr2=0x7fc9280ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.526+0000 7fc92d12f700 1 -- 172.21.15.27:0/3123421102 shutdown_connections 2024-04-25T00:15:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.526+0000 7fc92d12f700 1 -- 172.21.15.27:0/3123421102 wait complete. 2024-04-25T00:15:30.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.527+0000 7fc92d12f700 1 Processor -- start 2024-04-25T00:15:30.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.527+0000 7fc92d12f700 1 -- start start 2024-04-25T00:15:30.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.527+0000 7fc92d12f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9281022b0 0x7fc928075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:30.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.527+0000 7fc92d12f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9281034b0 0x7fc928074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:30.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.527+0000 7fc92d12f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc928103e50 0x7fc928074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:30.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.527+0000 7fc92d12f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc92810df10 con 0x7fc928103e50 2024-04-25T00:15:30.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.527+0000 7fc92d12f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc92810dd90 con 0x7fc9281034b0 2024-04-25T00:15:30.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.527+0000 7fc92d12f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc92810e090 con 0x7fc9281022b0 2024-04-25T00:15:30.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.527+0000 7fc92759e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc928103e50 0x7fc928074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:30.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.527+0000 7fc92759e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc928103e50 0x7fc928074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56524/0 (socket says 172.21.15.27:56524) 2024-04-25T00:15:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.528+0000 7fc92759e700 1 -- 172.21.15.27:0/1800103455 learned_addr learned my addr 172.21.15.27:0/1800103455 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.528+0000 7fc92659c700 1 --2- 172.21.15.27:0/1800103455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9281034b0 0x7fc928074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.528+0000 7fc926d9d700 1 --2- 172.21.15.27:0/1800103455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9281022b0 0x7fc928075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.528+0000 7fc92759e700 1 -- 172.21.15.27:0/1800103455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9281022b0 msgr2=0x7fc928075fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.528+0000 7fc92759e700 1 --2- 172.21.15.27:0/1800103455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9281022b0 0x7fc928075fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.528+0000 7fc92759e700 1 -- 172.21.15.27:0/1800103455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9281034b0 msgr2=0x7fc928074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.528+0000 7fc92759e700 1 --2- 172.21.15.27:0/1800103455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9281034b0 0x7fc928074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.528+0000 7fc92759e700 1 -- 172.21.15.27:0/1800103455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc928075410 con 0x7fc928103e50 2024-04-25T00:15:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.528+0000 7fc92759e700 1 --2- 172.21.15.27:0/1800103455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc928103e50 0x7fc928074b60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc920006070 tx=0x7fc920007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.528+0000 7fc91ffff700 1 -- 172.21.15.27:0/1800103455 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc920021020 con 0x7fc928103e50 2024-04-25T00:15:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.528+0000 7fc92d12f700 1 -- 172.21.15.27:0/1800103455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9281a25d0 con 0x7fc928103e50 2024-04-25T00:15:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.528+0000 7fc91ffff700 1 -- 172.21.15.27:0/1800103455 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc92001d950 con 0x7fc928103e50 2024-04-25T00:15:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.528+0000 7fc91ffff700 1 -- 172.21.15.27:0/1800103455 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc920005370 con 0x7fc928103e50 2024-04-25T00:15:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.528+0000 7fc92d12f700 1 -- 172.21.15.27:0/1800103455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9281a2ae0 con 0x7fc928103e50 2024-04-25T00:15:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.529+0000 7fc92d12f700 1 -- 172.21.15.27:0/1800103455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc928068f20 con 0x7fc928103e50 2024-04-25T00:15:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.530+0000 7fc91ffff700 1 -- 172.21.15.27:0/1800103455 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc920024020 con 0x7fc928103e50 2024-04-25T00:15:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.531+0000 7fc91ffff700 1 --2- 172.21.15.27:0/1800103455 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc9140a2c20 0x7fc9140a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:30.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.531+0000 7fc926d9d700 1 --2- 172.21.15.27:0/1800103455 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc9140a2c20 0x7fc9140a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:30.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.531+0000 7fc91ffff700 1 -- 172.21.15.27:0/1800103455 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc92000f020 con 0x7fc928103e50 2024-04-25T00:15:30.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.532+0000 7fc926d9d700 1 --2- 172.21.15.27:0/1800103455 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc9140a2c20 0x7fc9140a50e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc928103310 tx=0x7fc918009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:30.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.535+0000 7fc91ffff700 1 -- 172.21.15.27:0/1800103455 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc92009d990 con 0x7fc928103e50 2024-04-25T00:15:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.742+0000 7fc92d12f700 1 -- 172.21.15.27:0/1800103455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fc92802db30 con 0x7fc928103e50 2024-04-25T00:15:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.742+0000 7fc91ffff700 1 -- 172.21.15.27:0/1800103455 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5649 (secure 0 0 0) 0x7fc92009d290 con 0x7fc928103e50 2024-04-25T00:15:30.743 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 93 2024-04-25T00:15:30.744 INFO:teuthology.orchestra.run.smithi027.stdout:fsid 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:15:30.744 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-25T00:05:10.376114+0000 2024-04-25T00:15:30.744 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-25T00:15:11.415317+0000 2024-04-25T00:15:30.744 INFO:teuthology.orchestra.run.smithi027.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-25T00:15:30.744 INFO:teuthology.orchestra.run.smithi027.stdout:crush_version 27 2024-04-25T00:15:30.744 INFO:teuthology.orchestra.run.smithi027.stdout:full_ratio 0.95 2024-04-25T00:15:30.744 INFO:teuthology.orchestra.run.smithi027.stdout:backfillfull_ratio 0.9 2024-04-25T00:15:30.744 INFO:teuthology.orchestra.run.smithi027.stdout:nearfull_ratio 0.85 2024-04-25T00:15:30.744 INFO:teuthology.orchestra.run.smithi027.stdout:require_min_compat_client luminous 2024-04-25T00:15:30.744 INFO:teuthology.orchestra.run.smithi027.stdout:min_compat_client luminous 2024-04-25T00:15:30.744 INFO:teuthology.orchestra.run.smithi027.stdout:require_osd_release squid 2024-04-25T00:15:30.744 INFO:teuthology.orchestra.run.smithi027.stdout:stretch_mode_enabled false 2024-04-25T00:15:30.744 INFO:teuthology.orchestra.run.smithi027.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-25T00:15:30.745 INFO:teuthology.orchestra.run.smithi027.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-25T00:15:30.745 INFO:teuthology.orchestra.run.smithi027.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-25T00:15:30.745 INFO:teuthology.orchestra.run.smithi027.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-25T00:15:30.745 INFO:teuthology.orchestra.run.smithi027.stdout:max_osd 12 2024-04-25T00:15:30.745 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6802/238818528,v1:172.21.15.27:6803/238818528] [v2:172.21.15.27:6804/238818528,v1:172.21.15.27:6805/238818528] exists,up b523ee1d-7722-4c0d-ba18-82aa50b46b3c 2024-04-25T00:15:30.745 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 up in weight 1 up_from 13 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6800/1346503461,v1:172.21.15.139:6801/1346503461] [v2:172.21.15.139:6802/1346503461,v1:172.21.15.139:6803/1346503461] exists,up 1317fe63-aa4c-4687-8cdb-47f9531f654f 2024-04-25T00:15:30.745 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6800/662156492,v1:172.21.15.153:6801/662156492] [v2:172.21.15.153:6802/662156492,v1:172.21.15.153:6803/662156492] exists,up fbc82ce9-2c51-4c2d-91d0-84132845c085 2024-04-25T00:15:30.745 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6810/2150456629,v1:172.21.15.27:6811/2150456629] [v2:172.21.15.27:6812/2150456629,v1:172.21.15.27:6813/2150456629] exists,up a5e90823-a44c-47e7-bef7-6274de4d59ee 2024-04-25T00:15:30.745 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6808/856291222,v1:172.21.15.139:6809/856291222] [v2:172.21.15.139:6810/856291222,v1:172.21.15.139:6811/856291222] exists,up be18fdcd-c36d-4176-8292-90b0aaa85cb8 2024-04-25T00:15:30.745 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6808/3798123772,v1:172.21.15.153:6809/3798123772] [v2:172.21.15.153:6810/3798123772,v1:172.21.15.153:6811/3798123772] exists,up 35ff5087-0d52-48f2-8417-a373b85a47e8 2024-04-25T00:15:30.745 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6818/769569119,v1:172.21.15.27:6819/769569119] [v2:172.21.15.27:6820/769569119,v1:172.21.15.27:6821/769569119] exists,up 923eafd7-9819-4ef8-9008-0c5ff2a611c2 2024-04-25T00:15:30.745 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7 up in weight 1 up_from 51 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6816/4023269486,v1:172.21.15.139:6817/4023269486] [v2:172.21.15.139:6818/4023269486,v1:172.21.15.139:6819/4023269486] exists,up a71b99a8-c31e-4ac9-91fc-61735cd196b2 2024-04-25T00:15:30.745 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8 up in weight 1 up_from 59 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6816/2251501067,v1:172.21.15.153:6817/2251501067] [v2:172.21.15.153:6818/2251501067,v1:172.21.15.153:6819/2251501067] exists,up b73934bc-9fd5-4da3-8c9b-047a2d116d19 2024-04-25T00:15:30.746 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9 up in weight 1 up_from 67 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6826/2816562594,v1:172.21.15.27:6827/2816562594] [v2:172.21.15.27:6828/2816562594,v1:172.21.15.27:6829/2816562594] exists,up 375ebd2b-ed51-4e03-b8bb-9725b30d81b5 2024-04-25T00:15:30.746 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6824/3154318186,v1:172.21.15.139:6825/3154318186] [v2:172.21.15.139:6826/3154318186,v1:172.21.15.139:6827/3154318186] exists,up ea5bca6d-3675-453d-a0f2-cae08531fe5f 2024-04-25T00:15:30.746 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6824/2357170710,v1:172.21.15.153:6825/2357170710] [v2:172.21.15.153:6826/2357170710,v1:172.21.15.153:6827/2357170710] exists,up 5a15ea59-44bf-4e78-b60a-8cc5944ec151 2024-04-25T00:15:30.746 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 2.a [2,11] 2024-04-25T00:15:30.746 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 2.b [8,11] 2024-04-25T00:15:30.746 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.2 [3,9] 2024-04-25T00:15:30.746 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.e [5,9] 2024-04-25T00:15:30.746 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.11 [4,6] 2024-04-25T00:15:30.746 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.1b [4,8] 2024-04-25T00:15:30.746 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.2a [3,8] 2024-04-25T00:15:30.746 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.3c [4,9] 2024-04-25T00:15:30.746 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 4.7 [5,8] 2024-04-25T00:15:30.746 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 4.b [9,8] 2024-04-25T00:15:30.746 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 4.1b [9,8] 2024-04-25T00:15:30.747 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 4.1c [9,10] 2024-04-25T00:15:30.747 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/3721121684 expires 2024-04-26T00:06:09.484973+0000 2024-04-25T00:15:30.747 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6800/1186428290 expires 2024-04-26T00:06:09.484973+0000 2024-04-25T00:15:30.747 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/2012441698 expires 2024-04-26T00:06:09.484973+0000 2024-04-25T00:15:30.747 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6800/3082151239 expires 2024-04-26T00:05:42.617192+0000 2024-04-25T00:15:30.747 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6801/3082151239 expires 2024-04-26T00:05:42.617192+0000 2024-04-25T00:15:30.747 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6801/1186428290 expires 2024-04-26T00:06:09.484973+0000 2024-04-25T00:15:30.747 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/2172615181 expires 2024-04-26T00:05:42.617192+0000 2024-04-25T00:15:30.747 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/316040314 expires 2024-04-26T00:05:42.617192+0000 2024-04-25T00:15:30.747 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/1671770035 expires 2024-04-26T00:05:42.617192+0000 2024-04-25T00:15:30.747 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/1035759944 expires 2024-04-26T00:06:09.484973+0000 2024-04-25T00:15:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.746+0000 7fc92d12f700 1 -- 172.21.15.27:0/1800103455 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc9140a2c20 msgr2=0x7fc9140a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.746+0000 7fc92d12f700 1 --2- 172.21.15.27:0/1800103455 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc9140a2c20 0x7fc9140a50e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc928103310 tx=0x7fc918009190 comp rx=0 tx=0).stop 2024-04-25T00:15:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.746+0000 7fc92d12f700 1 -- 172.21.15.27:0/1800103455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc928103e50 msgr2=0x7fc928074b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.746+0000 7fc92d12f700 1 --2- 172.21.15.27:0/1800103455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc928103e50 0x7fc928074b60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc920006070 tx=0x7fc920007d10 comp rx=0 tx=0).stop 2024-04-25T00:15:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.746+0000 7fc92d12f700 1 -- 172.21.15.27:0/1800103455 shutdown_connections 2024-04-25T00:15:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.746+0000 7fc92d12f700 1 --2- 172.21.15.27:0/1800103455 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc9140a2c20 0x7fc9140a50e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.746+0000 7fc92d12f700 1 --2- 172.21.15.27:0/1800103455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc928103e50 0x7fc928074b60 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:30.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.746+0000 7fc92d12f700 1 --2- 172.21.15.27:0/1800103455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9281022b0 0x7fc928075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:30.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.746+0000 7fc92d12f700 1 --2- 172.21.15.27:0/1800103455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9281034b0 0x7fc928074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:30.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.746+0000 7fc92d12f700 1 -- 172.21.15.27:0/1800103455 >> 172.21.15.27:0/1800103455 conn(0x7fc9280fd850 msgr2=0x7fc9281067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:30.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.746+0000 7fc92d12f700 1 -- 172.21.15.27:0/1800103455 shutdown_connections 2024-04-25T00:15:30.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:30.746+0000 7fc92d12f700 1 -- 172.21.15.27:0/1800103455 wait complete. 2024-04-25T00:15:30.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:30 smithi027 ceph-mon[29208]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:15:31.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:30 smithi139 ceph-mon[32833]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:15:31.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:30 smithi153 ceph-mon[32896]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T00:15:31.437 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-25T00:15:31.449 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:15:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.556+0000 7f68fbe64640 1 -- 172.21.15.27:0/2809372453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68f4107500 msgr2=0x7f68f41098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.556+0000 7f68fbe64640 1 --2- 172.21.15.27:0/2809372453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68f4107500 0x7f68f41098f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f68ec009be0 tx=0x7f68ec02a6c0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.556+0000 7f68fbe64640 1 -- 172.21.15.27:0/2809372453 shutdown_connections 2024-04-25T00:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.556+0000 7f68fbe64640 1 --2- 172.21.15.27:0/2809372453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68f4107500 0x7f68f41098f0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.556+0000 7f68fbe64640 1 --2- 172.21.15.27:0/2809372453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f4104bd0 0x7f68f4106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.556+0000 7f68fbe64640 1 --2- 172.21.15.27:0/2809372453 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68f41022a0 0x7f68f4104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.556+0000 7f68fbe64640 1 -- 172.21.15.27:0/2809372453 >> 172.21.15.27:0/2809372453 conn(0x7f68f40fbbf0 msgr2=0x7f68f40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.556+0000 7f68fbe64640 1 -- 172.21.15.27:0/2809372453 shutdown_connections 2024-04-25T00:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.556+0000 7f68fbe64640 1 -- 172.21.15.27:0/2809372453 wait complete. 2024-04-25T00:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68fbe64640 1 Processor -- start 2024-04-25T00:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68fbe64640 1 -- start start 2024-04-25T00:15:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68fbe64640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68f41022a0 0x7f68f4197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68fbe64640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f4104bd0 0x7f68f4197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68fbe64640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68f4107500 0x7f68f419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68fbe64640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f68f410d3d0 con 0x7f68f4107500 2024-04-25T00:15:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68fbe64640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f68f410d250 con 0x7f68f41022a0 2024-04-25T00:15:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68fbe64640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f68f410d550 con 0x7f68f4104bd0 2024-04-25T00:15:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68fa401640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68f4107500 0x7f68f419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68fa401640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68f4107500 0x7f68f419eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56540/0 (socket says 172.21.15.27:56540) 2024-04-25T00:15:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68fa401640 1 -- 172.21.15.27:0/3377276162 learned_addr learned my addr 172.21.15.27:0/3377276162 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68f9c00640 1 --2- 172.21.15.27:0/3377276162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68f41022a0 0x7f68f4197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68f93ff640 1 --2- 172.21.15.27:0/3377276162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f4104bd0 0x7f68f4197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68fa401640 1 -- 172.21.15.27:0/3377276162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f4104bd0 msgr2=0x7f68f4197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:31.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68fa401640 1 --2- 172.21.15.27:0/3377276162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f4104bd0 0x7f68f4197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68fa401640 1 -- 172.21.15.27:0/3377276162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68f41022a0 msgr2=0x7f68f4197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:31.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68fa401640 1 --2- 172.21.15.27:0/3377276162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68f41022a0 0x7f68f4197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68fa401640 1 -- 172.21.15.27:0/3377276162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f419f660 con 0x7f68f4107500 2024-04-25T00:15:31.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68f9c00640 1 --2- 172.21.15.27:0/3377276162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68f41022a0 0x7f68f4197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:31.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.557+0000 7f68fa401640 1 --2- 172.21.15.27:0/3377276162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68f4107500 0x7f68f419eff0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0305b0 tx=0x7f68ec0305e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:31.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.558+0000 7f68eaffd640 1 -- 172.21.15.27:0/3377276162 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec041020 con 0x7f68f4107500 2024-04-25T00:15:31.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.558+0000 7f68eaffd640 1 -- 172.21.15.27:0/3377276162 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68ec037040 con 0x7f68f4107500 2024-04-25T00:15:31.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.558+0000 7f68fbe64640 1 -- 172.21.15.27:0/3377276162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f419f8f0 con 0x7f68f4107500 2024-04-25T00:15:31.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.558+0000 7f68eaffd640 1 -- 172.21.15.27:0/3377276162 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec036070 con 0x7f68f4107500 2024-04-25T00:15:31.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.558+0000 7f68fbe64640 1 -- 172.21.15.27:0/3377276162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f419fde0 con 0x7f68f4107500 2024-04-25T00:15:31.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.559+0000 7f68fbe64640 1 -- 172.21.15.27:0/3377276162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f40fd6e0 con 0x7f68f4107500 2024-04-25T00:15:31.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.560+0000 7f68eaffd640 1 -- 172.21.15.27:0/3377276162 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f68ec044020 con 0x7f68f4107500 2024-04-25T00:15:31.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.560+0000 7f68eaffd640 1 --2- 172.21.15.27:0/3377276162 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f68e00a2b90 0x7f68e00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:31.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.561+0000 7f68f9c00640 1 --2- 172.21.15.27:0/3377276162 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f68e00a2b90 0x7f68e00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:31.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.561+0000 7f68eaffd640 1 -- 172.21.15.27:0/3377276162 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f68ec0f0110 con 0x7f68f4107500 2024-04-25T00:15:31.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.561+0000 7f68f9c00640 1 --2- 172.21.15.27:0/3377276162 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f68e00a2b90 0x7f68e00a5050 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f68f0005fd0 tx=0x7f68f0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.565+0000 7f68eaffd640 1 -- 172.21.15.27:0/3377276162 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f68ec04b020 con 0x7f68f4107500 2024-04-25T00:15:31.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.752+0000 7f68fbe64640 1 -- 172.21.15.27:0/3377276162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f68f4062340 con 0x7f68f4107500 2024-04-25T00:15:31.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.754+0000 7f68eaffd640 1 -- 172.21.15.27:0/3377276162 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24736 (secure 0 0 0) 0x7f68ec004670 con 0x7f68f4107500 2024-04-25T00:15:31.755 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:31.755 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":93,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:15:11.415317+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:15:31.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.756+0000 7f68fbe64640 1 -- 172.21.15.27:0/3377276162 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f68e00a2b90 msgr2=0x7f68e00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:31.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.756+0000 7f68fbe64640 1 --2- 172.21.15.27:0/3377276162 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f68e00a2b90 0x7f68e00a5050 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f68f0005fd0 tx=0x7f68f0009260 comp rx=0 tx=0).stop 2024-04-25T00:15:31.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.756+0000 7f68fbe64640 1 -- 172.21.15.27:0/3377276162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68f4107500 msgr2=0x7f68f419eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:31.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.756+0000 7f68fbe64640 1 --2- 172.21.15.27:0/3377276162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68f4107500 0x7f68f419eff0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0305b0 tx=0x7f68ec0305e0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.756+0000 7f68fbe64640 1 -- 172.21.15.27:0/3377276162 shutdown_connections 2024-04-25T00:15:31.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.756+0000 7f68fbe64640 1 --2- 172.21.15.27:0/3377276162 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f68e00a2b90 0x7f68e00a5050 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.756+0000 7f68fbe64640 1 --2- 172.21.15.27:0/3377276162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68f4107500 0x7f68f419eff0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.756+0000 7f68fbe64640 1 --2- 172.21.15.27:0/3377276162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f4104bd0 0x7f68f4197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.756+0000 7f68fbe64640 1 --2- 172.21.15.27:0/3377276162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68f41022a0 0x7f68f4197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.756+0000 7f68fbe64640 1 -- 172.21.15.27:0/3377276162 >> 172.21.15.27:0/3377276162 conn(0x7f68f40fbbf0 msgr2=0x7f68f4102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:31.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.756+0000 7f68fbe64640 1 -- 172.21.15.27:0/3377276162 shutdown_connections 2024-04-25T00:15:31.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.756+0000 7f68fbe64640 1 -- 172.21.15.27:0/3377276162 wait complete. 2024-04-25T00:15:31.769 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:08:34.006973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T00:13:17.337307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T00:13:18.253713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T00:13:21.995987+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T00:15:31.769 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:15:31.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.873+0000 7f597822c640 1 -- 172.21.15.27:0/3277652444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5970103350 msgr2=0x7f5970109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:31.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.873+0000 7f597822c640 1 --2- 172.21.15.27:0/3277652444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5970103350 0x7f5970109be0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5968009ca0 tx=0x7f596802a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.873+0000 7f597822c640 1 -- 172.21.15.27:0/3277652444 shutdown_connections 2024-04-25T00:15:31.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.873+0000 7f597822c640 1 --2- 172.21.15.27:0/3277652444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5970103350 0x7f5970109be0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.873+0000 7f597822c640 1 --2- 172.21.15.27:0/3277652444 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5970102990 0x7f5970102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.873+0000 7f597822c640 1 --2- 172.21.15.27:0/3277652444 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59701017a0 0x7f5970101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.873+0000 7f597822c640 1 -- 172.21.15.27:0/3277652444 >> 172.21.15.27:0/3277652444 conn(0x7f59700fcd10 msgr2=0x7f59700ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:31.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.873+0000 7f597822c640 1 -- 172.21.15.27:0/3277652444 shutdown_connections 2024-04-25T00:15:31.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.874+0000 7f597822c640 1 -- 172.21.15.27:0/3277652444 wait complete. 2024-04-25T00:15:31.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.874+0000 7f597822c640 1 Processor -- start 2024-04-25T00:15:31.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.874+0000 7f597822c640 1 -- start start 2024-04-25T00:15:31.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.874+0000 7f597822c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59701017a0 0x7f5970197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:31.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.874+0000 7f597822c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5970102990 0x7f5970198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:31.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.874+0000 7f597822c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5970103350 0x7f597019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:31.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.874+0000 7f597822c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f597010d420 con 0x7f5970102990 2024-04-25T00:15:31.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.874+0000 7f597822c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f597010d2a0 con 0x7f5970103350 2024-04-25T00:15:31.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.874+0000 7f597822c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f597010d5a0 con 0x7f59701017a0 2024-04-25T00:15:31.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.874+0000 7f59757c7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5970102990 0x7f5970198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:31.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.874+0000 7f59757c7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5970102990 0x7f5970198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56552/0 (socket says 172.21.15.27:56552) 2024-04-25T00:15:31.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.874+0000 7f59757c7640 1 -- 172.21.15.27:0/1412782830 learned_addr learned my addr 172.21.15.27:0/1412782830 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:31.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.874+0000 7f59767c9640 1 --2- 172.21.15.27:0/1412782830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5970103350 0x7f597019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:31.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.874+0000 7f5975fc8640 1 --2- 172.21.15.27:0/1412782830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59701017a0 0x7f5970197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:31.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.875+0000 7f59757c7640 1 -- 172.21.15.27:0/1412782830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59701017a0 msgr2=0x7f5970197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:31.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.875+0000 7f59757c7640 1 --2- 172.21.15.27:0/1412782830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59701017a0 0x7f5970197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.875+0000 7f59757c7640 1 -- 172.21.15.27:0/1412782830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5970103350 msgr2=0x7f597019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:31.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.875+0000 7f59757c7640 1 --2- 172.21.15.27:0/1412782830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5970103350 0x7f597019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.875+0000 7f59757c7640 1 -- 172.21.15.27:0/1412782830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597019f6e0 con 0x7f5970102990 2024-04-25T00:15:31.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.875+0000 7f59757c7640 1 --2- 172.21.15.27:0/1412782830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5970102990 0x7f5970198000 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f596000e9f0 tx=0x7f596000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:31.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.875+0000 7f5966ffd640 1 -- 172.21.15.27:0/1412782830 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596000cd80 con 0x7f5970102990 2024-04-25T00:15:31.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.875+0000 7f597822c640 1 -- 172.21.15.27:0/1412782830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f597019f9d0 con 0x7f5970102990 2024-04-25T00:15:31.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.875+0000 7f5966ffd640 1 -- 172.21.15.27:0/1412782830 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5960010910 con 0x7f5970102990 2024-04-25T00:15:31.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.875+0000 7f5966ffd640 1 -- 172.21.15.27:0/1412782830 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59600188f0 con 0x7f5970102990 2024-04-25T00:15:31.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.875+0000 7f597822c640 1 -- 172.21.15.27:0/1412782830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f597019ff20 con 0x7f5970102990 2024-04-25T00:15:31.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.876+0000 7f597822c640 1 -- 172.21.15.27:0/1412782830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5970108a60 con 0x7f5970102990 2024-04-25T00:15:31.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.877+0000 7f5966ffd640 1 -- 172.21.15.27:0/1412782830 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5960010ac0 con 0x7f5970102990 2024-04-25T00:15:31.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.878+0000 7f5966ffd640 1 --2- 172.21.15.27:0/1412782830 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f595c0a2b90 0x7f595c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:31.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.878+0000 7f5975fc8640 1 --2- 172.21.15.27:0/1412782830 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f595c0a2b90 0x7f595c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:31.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.878+0000 7f5966ffd640 1 -- 172.21.15.27:0/1412782830 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f59600cfc70 con 0x7f5970102990 2024-04-25T00:15:31.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.878+0000 7f5975fc8640 1 --2- 172.21.15.27:0/1412782830 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f595c0a2b90 0x7f595c0a5050 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f596c005fd0 tx=0x7f596c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:31.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:31.881+0000 7f5966ffd640 1 -- 172.21.15.27:0/1412782830 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f596009c840 con 0x7f5970102990 2024-04-25T00:15:32.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:31 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1800103455' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T00:15:32.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:31 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1800103455' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T00:15:32.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.065+0000 7f597822c640 1 -- 172.21.15.27:0/1412782830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f597002d300 con 0x7f5970102990 2024-04-25T00:15:32.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.066+0000 7f5966ffd640 1 -- 172.21.15.27:0/1412782830 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f596009c140 con 0x7f5970102990 2024-04-25T00:15:32.067 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T00:15:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.068+0000 7f597822c640 1 -- 172.21.15.27:0/1412782830 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f595c0a2b90 msgr2=0x7f595c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.068+0000 7f597822c640 1 --2- 172.21.15.27:0/1412782830 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f595c0a2b90 0x7f595c0a5050 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f596c005fd0 tx=0x7f596c005850 comp rx=0 tx=0).stop 2024-04-25T00:15:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.068+0000 7f597822c640 1 -- 172.21.15.27:0/1412782830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5970102990 msgr2=0x7f5970198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.068+0000 7f597822c640 1 --2- 172.21.15.27:0/1412782830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5970102990 0x7f5970198000 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f596000e9f0 tx=0x7f596000eeb0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.068+0000 7f597822c640 1 -- 172.21.15.27:0/1412782830 shutdown_connections 2024-04-25T00:15:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.068+0000 7f597822c640 1 --2- 172.21.15.27:0/1412782830 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f595c0a2b90 0x7f595c0a5050 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.068+0000 7f597822c640 1 --2- 172.21.15.27:0/1412782830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5970103350 0x7f597019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.068+0000 7f597822c640 1 --2- 172.21.15.27:0/1412782830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5970102990 0x7f5970198000 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.068+0000 7f597822c640 1 --2- 172.21.15.27:0/1412782830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59701017a0 0x7f5970197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.068+0000 7f597822c640 1 -- 172.21.15.27:0/1412782830 >> 172.21.15.27:0/1412782830 conn(0x7f59700fcd10 msgr2=0x7f5970105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:32.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.068+0000 7f597822c640 1 -- 172.21.15.27:0/1412782830 shutdown_connections 2024-04-25T00:15:32.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.068+0000 7f597822c640 1 -- 172.21.15.27:0/1412782830 wait complete. 2024-04-25T00:15:32.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:31 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1800103455' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T00:15:32.080 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T00:15:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.184+0000 7fa715691640 1 -- 172.21.15.27:0/3176428544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa710101890 msgr2=0x7fa710101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.184+0000 7fa715691640 1 --2- 172.21.15.27:0/3176428544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa710101890 0x7fa710101d10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8009900 tx=0x7fa6f802a410 comp rx=0 tx=0).stop 2024-04-25T00:15:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.184+0000 7fa715691640 1 -- 172.21.15.27:0/3176428544 shutdown_connections 2024-04-25T00:15:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.184+0000 7fa715691640 1 --2- 172.21.15.27:0/3176428544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa710102250 0x7fa710108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.184+0000 7fa715691640 1 --2- 172.21.15.27:0/3176428544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa710101890 0x7fa710101d10 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.184+0000 7fa715691640 1 --2- 172.21.15.27:0/3176428544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7101006a0 0x7fa710100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.184+0000 7fa715691640 1 -- 172.21.15.27:0/3176428544 >> 172.21.15.27:0/3176428544 conn(0x7fa7100fbc30 msgr2=0x7fa7100fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.184+0000 7fa715691640 1 -- 172.21.15.27:0/3176428544 shutdown_connections 2024-04-25T00:15:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.184+0000 7fa715691640 1 -- 172.21.15.27:0/3176428544 wait complete. 2024-04-25T00:15:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.185+0000 7fa715691640 1 Processor -- start 2024-04-25T00:15:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.185+0000 7fa715691640 1 -- start start 2024-04-25T00:15:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.185+0000 7fa715691640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7101006a0 0x7fa710199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.185+0000 7fa715691640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa710101890 0x7fa71019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.185+0000 7fa715691640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa710102250 0x7fa7101a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.185+0000 7fa715691640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa71010c350 con 0x7fa7101006a0 2024-04-25T00:15:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.185+0000 7fa715691640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa71010c1d0 con 0x7fa710102250 2024-04-25T00:15:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.185+0000 7fa715691640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa71010c4d0 con 0x7fa710101890 2024-04-25T00:15:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.185+0000 7fa70effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7101006a0 0x7fa710199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.185+0000 7fa70effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7101006a0 0x7fa710199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56572/0 (socket says 172.21.15.27:56572) 2024-04-25T00:15:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.185+0000 7fa70effd640 1 -- 172.21.15.27:0/2329032872 learned_addr learned my addr 172.21.15.27:0/2329032872 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.185+0000 7fa707fff640 1 --2- 172.21.15.27:0/2329032872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa710101890 0x7fa71019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.186+0000 7fa70effd640 1 -- 172.21.15.27:0/2329032872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa710101890 msgr2=0x7fa71019a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.186+0000 7fa70effd640 1 --2- 172.21.15.27:0/2329032872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa710101890 0x7fa71019a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.186+0000 7fa70effd640 1 -- 172.21.15.27:0/2329032872 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa710102250 msgr2=0x7fa7101a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.186+0000 7fa70effd640 1 --2- 172.21.15.27:0/2329032872 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa710102250 0x7fa7101a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.186+0000 7fa70effd640 1 -- 172.21.15.27:0/2329032872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7101a1910 con 0x7fa7101006a0 2024-04-25T00:15:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.186+0000 7fa70effd640 1 --2- 172.21.15.27:0/2329032872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7101006a0 0x7fa710199db0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa70000d950 tx=0x7fa70000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.186+0000 7fa70cff9640 1 -- 172.21.15.27:0/2329032872 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa700017020 con 0x7fa7101006a0 2024-04-25T00:15:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.186+0000 7fa70cff9640 1 -- 172.21.15.27:0/2329032872 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa700009c30 con 0x7fa7101006a0 2024-04-25T00:15:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.186+0000 7fa715691640 1 -- 172.21.15.27:0/2329032872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7101a1c00 con 0x7fa7101006a0 2024-04-25T00:15:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.186+0000 7fa70cff9640 1 -- 172.21.15.27:0/2329032872 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70000f740 con 0x7fa7101006a0 2024-04-25T00:15:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.186+0000 7fa715691640 1 -- 172.21.15.27:0/2329032872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7101a2170 con 0x7fa7101006a0 2024-04-25T00:15:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.187+0000 7fa715691640 1 -- 172.21.15.27:0/2329032872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7101a2340 con 0x7fa7101006a0 2024-04-25T00:15:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.188+0000 7fa70cff9640 1 -- 172.21.15.27:0/2329032872 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa70001a020 con 0x7fa7101006a0 2024-04-25T00:15:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.189+0000 7fa70cff9640 1 --2- 172.21.15.27:0/2329032872 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa6fc0a2c10 0x7fa6fc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.189+0000 7fa707fff640 1 --2- 172.21.15.27:0/2329032872 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa6fc0a2c10 0x7fa6fc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.189+0000 7fa70cff9640 1 -- 172.21.15.27:0/2329032872 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa7000cf190 con 0x7fa7101006a0 2024-04-25T00:15:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.189+0000 7fa707fff640 1 --2- 172.21.15.27:0/2329032872 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa6fc0a2c10 0x7fa6fc0a50d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8006010 tx=0x7fa6f8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:32.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.192+0000 7fa70cff9640 1 -- 172.21.15.27:0/2329032872 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa700021070 con 0x7fa7101006a0 2024-04-25T00:15:32.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.377+0000 7fa715691640 1 -- 172.21.15.27:0/2329032872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa7101a2510 con 0x7fa7101006a0 2024-04-25T00:15:32.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.377+0000 7fa70cff9640 1 -- 172.21.15.27:0/2329032872 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fa700009de0 con 0x7fa7101006a0 2024-04-25T00:15:32.378 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:15:32.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.379+0000 7fa715691640 1 -- 172.21.15.27:0/2329032872 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa6fc0a2c10 msgr2=0x7fa6fc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.379+0000 7fa715691640 1 --2- 172.21.15.27:0/2329032872 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa6fc0a2c10 0x7fa6fc0a50d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8006010 tx=0x7fa6f8000bc0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.379+0000 7fa715691640 1 -- 172.21.15.27:0/2329032872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7101006a0 msgr2=0x7fa710199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.379+0000 7fa715691640 1 --2- 172.21.15.27:0/2329032872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7101006a0 0x7fa710199db0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa70000d950 tx=0x7fa70000de10 comp rx=0 tx=0).stop 2024-04-25T00:15:32.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.379+0000 7fa715691640 1 -- 172.21.15.27:0/2329032872 shutdown_connections 2024-04-25T00:15:32.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.379+0000 7fa715691640 1 --2- 172.21.15.27:0/2329032872 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fa6fc0a2c10 0x7fa6fc0a50d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.379+0000 7fa715691640 1 --2- 172.21.15.27:0/2329032872 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa710102250 0x7fa7101a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.379+0000 7fa715691640 1 --2- 172.21.15.27:0/2329032872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa710101890 0x7fa71019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.379+0000 7fa715691640 1 --2- 172.21.15.27:0/2329032872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7101006a0 0x7fa710199db0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.379+0000 7fa715691640 1 -- 172.21.15.27:0/2329032872 >> 172.21.15.27:0/2329032872 conn(0x7fa7100fbc30 msgr2=0x7fa7100fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:32.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.380+0000 7fa715691640 1 -- 172.21.15.27:0/2329032872 shutdown_connections 2024-04-25T00:15:32.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.380+0000 7fa715691640 1 -- 172.21.15.27:0/2329032872 wait complete. 2024-04-25T00:15:32.392 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T00:15:32.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.496+0000 7effe1264640 1 -- 172.21.15.27:0/952367174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc102250 msgr2=0x7effdc108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.496+0000 7effe1264640 1 --2- 172.21.15.27:0/952367174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc102250 0x7effdc108ae0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7effd4009ca0 tx=0x7effd402a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.496+0000 7effe1264640 1 -- 172.21.15.27:0/952367174 shutdown_connections 2024-04-25T00:15:32.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.496+0000 7effe1264640 1 --2- 172.21.15.27:0/952367174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc102250 0x7effdc108ae0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.496+0000 7effe1264640 1 --2- 172.21.15.27:0/952367174 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7effdc101890 0x7effdc101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.496+0000 7effe1264640 1 --2- 172.21.15.27:0/952367174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7effdc1006a0 0x7effdc100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.496+0000 7effe1264640 1 -- 172.21.15.27:0/952367174 >> 172.21.15.27:0/952367174 conn(0x7effdc0fbc30 msgr2=0x7effdc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:32.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.496+0000 7effe1264640 1 -- 172.21.15.27:0/952367174 shutdown_connections 2024-04-25T00:15:32.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.496+0000 7effe1264640 1 -- 172.21.15.27:0/952367174 wait complete. 2024-04-25T00:15:32.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.496+0000 7effe1264640 1 Processor -- start 2024-04-25T00:15:32.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.497+0000 7effe1264640 1 -- start start 2024-04-25T00:15:32.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.497+0000 7effe1264640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7effdc1006a0 0x7effdc196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:32.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.497+0000 7effe1264640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7effdc101890 0x7effdc196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:32.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.497+0000 7effe1264640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc102250 0x7effdc19def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:32.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.497+0000 7effe1264640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7effdc10c320 con 0x7effdc102250 2024-04-25T00:15:32.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.497+0000 7effe1264640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7effdc10c1a0 con 0x7effdc1006a0 2024-04-25T00:15:32.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.497+0000 7effe1264640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7effdc10c4a0 con 0x7effdc101890 2024-04-25T00:15:32.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.497+0000 7effdb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc102250 0x7effdc19def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:32.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.497+0000 7effdaffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7effdc1006a0 0x7effdc196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:32.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.497+0000 7effdb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc102250 0x7effdc19def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56592/0 (socket says 172.21.15.27:56592) 2024-04-25T00:15:32.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.497+0000 7effdb7fe640 1 -- 172.21.15.27:0/3345340092 learned_addr learned my addr 172.21.15.27:0/3345340092 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:32.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.497+0000 7effda7fc640 1 --2- 172.21.15.27:0/3345340092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7effdc101890 0x7effdc196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:32.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.498+0000 7effdb7fe640 1 -- 172.21.15.27:0/3345340092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7effdc101890 msgr2=0x7effdc196e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.498+0000 7effdb7fe640 1 --2- 172.21.15.27:0/3345340092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7effdc101890 0x7effdc196e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.498+0000 7effdb7fe640 1 -- 172.21.15.27:0/3345340092 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7effdc1006a0 msgr2=0x7effdc196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.498+0000 7effdb7fe640 1 --2- 172.21.15.27:0/3345340092 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7effdc1006a0 0x7effdc196940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.498+0000 7effdb7fe640 1 -- 172.21.15.27:0/3345340092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effdc19e560 con 0x7effdc102250 2024-04-25T00:15:32.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.498+0000 7effdb7fe640 1 --2- 172.21.15.27:0/3345340092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc102250 0x7effdc19def0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7effd4005e30 tx=0x7effd4031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:32.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.498+0000 7effc3fff640 1 -- 172.21.15.27:0/3345340092 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd4049070 con 0x7effdc102250 2024-04-25T00:15:32.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.498+0000 7effc3fff640 1 -- 172.21.15.27:0/3345340092 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effd4003c40 con 0x7effdc102250 2024-04-25T00:15:32.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.498+0000 7effe1264640 1 -- 172.21.15.27:0/3345340092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effdc069d40 con 0x7effdc102250 2024-04-25T00:15:32.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.498+0000 7effc3fff640 1 -- 172.21.15.27:0/3345340092 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd402fb80 con 0x7effdc102250 2024-04-25T00:15:32.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.498+0000 7effe1264640 1 -- 172.21.15.27:0/3345340092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effdc06a290 con 0x7effdc102250 2024-04-25T00:15:32.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.499+0000 7effe1264640 1 -- 172.21.15.27:0/3345340092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effdc0689b0 con 0x7effdc102250 2024-04-25T00:15:32.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.500+0000 7effc3fff640 1 -- 172.21.15.27:0/3345340092 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7effd404c070 con 0x7effdc102250 2024-04-25T00:15:32.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.501+0000 7effc3fff640 1 --2- 172.21.15.27:0/3345340092 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7effc40a2b90 0x7effc40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:32.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.501+0000 7effc3fff640 1 -- 172.21.15.27:0/3345340092 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7effd40f1bb0 con 0x7effdc102250 2024-04-25T00:15:32.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.501+0000 7effdaffd640 1 --2- 172.21.15.27:0/3345340092 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7effc40a2b90 0x7effc40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:32.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.501+0000 7effdaffd640 1 --2- 172.21.15.27:0/3345340092 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7effc40a2b90 0x7effc40a5050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7effcc005fd0 tx=0x7effcc00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:32.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.504+0000 7effc3fff640 1 -- 172.21.15.27:0/3345340092 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7effd4080740 con 0x7effdc102250 2024-04-25T00:15:32.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.688+0000 7effe1264640 1 -- 172.21.15.27:0/3345340092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7effdc19e730 con 0x7effdc102250 2024-04-25T00:15:32.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.688+0000 7effc3fff640 1 -- 172.21.15.27:0/3345340092 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7effd40be130 con 0x7effdc102250 2024-04-25T00:15:32.689 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T00:15:32.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.690+0000 7effe1264640 1 -- 172.21.15.27:0/3345340092 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7effc40a2b90 msgr2=0x7effc40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.690+0000 7effe1264640 1 --2- 172.21.15.27:0/3345340092 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7effc40a2b90 0x7effc40a5050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7effcc005fd0 tx=0x7effcc00b390 comp rx=0 tx=0).stop 2024-04-25T00:15:32.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.690+0000 7effe1264640 1 -- 172.21.15.27:0/3345340092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc102250 msgr2=0x7effdc19def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.690+0000 7effe1264640 1 --2- 172.21.15.27:0/3345340092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc102250 0x7effdc19def0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7effd4005e30 tx=0x7effd4031ee0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.690+0000 7effe1264640 1 -- 172.21.15.27:0/3345340092 shutdown_connections 2024-04-25T00:15:32.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.690+0000 7effe1264640 1 --2- 172.21.15.27:0/3345340092 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7effc40a2b90 0x7effc40a5050 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.690+0000 7effe1264640 1 --2- 172.21.15.27:0/3345340092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc102250 0x7effdc19def0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.690+0000 7effe1264640 1 --2- 172.21.15.27:0/3345340092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7effdc101890 0x7effdc196e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.690+0000 7effe1264640 1 --2- 172.21.15.27:0/3345340092 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7effdc1006a0 0x7effdc196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.690+0000 7effe1264640 1 -- 172.21.15.27:0/3345340092 >> 172.21.15.27:0/3345340092 conn(0x7effdc0fbc30 msgr2=0x7effdc104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:32.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.690+0000 7effe1264640 1 -- 172.21.15.27:0/3345340092 shutdown_connections 2024-04-25T00:15:32.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.690+0000 7effe1264640 1 -- 172.21.15.27:0/3345340092 wait complete. 2024-04-25T00:15:32.704 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T00:15:32.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.808+0000 7fbd56a41640 1 -- 172.21.15.27:0/3480031824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd501017a0 msgr2=0x7fbd50101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.808+0000 7fbd56a41640 1 --2- 172.21.15.27:0/3480031824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd501017a0 0x7fbd50101bc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbd40009900 tx=0x7fbd4002a410 comp rx=0 tx=0).stop 2024-04-25T00:15:32.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.808+0000 7fbd56a41640 1 -- 172.21.15.27:0/3480031824 shutdown_connections 2024-04-25T00:15:32.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.808+0000 7fbd56a41640 1 --2- 172.21.15.27:0/3480031824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd50103350 0x7fbd50109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.808+0000 7fbd56a41640 1 --2- 172.21.15.27:0/3480031824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd50102990 0x7fbd50102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.808+0000 7fbd56a41640 1 --2- 172.21.15.27:0/3480031824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd501017a0 0x7fbd50101bc0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.808+0000 7fbd56a41640 1 -- 172.21.15.27:0/3480031824 >> 172.21.15.27:0/3480031824 conn(0x7fbd500fcd10 msgr2=0x7fbd500ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:32.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.808+0000 7fbd56a41640 1 -- 172.21.15.27:0/3480031824 shutdown_connections 2024-04-25T00:15:32.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.808+0000 7fbd56a41640 1 -- 172.21.15.27:0/3480031824 wait complete. 2024-04-25T00:15:32.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd56a41640 1 Processor -- start 2024-04-25T00:15:32.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd56a41640 1 -- start start 2024-04-25T00:15:32.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd56a41640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd501017a0 0x7fbd500754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:32.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd56a41640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd50102990 0x7fbd50073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:32.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd56a41640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd50103350 0x7fbd50074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:32.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd56a41640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5010d3f0 con 0x7fbd50102990 2024-04-25T00:15:32.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd56a41640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5010d270 con 0x7fbd501017a0 2024-04-25T00:15:32.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd56a41640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5010d570 con 0x7fbd50103350 2024-04-25T00:15:32.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd4f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd50102990 0x7fbd50073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:32.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd4f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd50102990 0x7fbd50073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56622/0 (socket says 172.21.15.27:56622) 2024-04-25T00:15:32.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd4f7fe640 1 -- 172.21.15.27:0/2242004412 learned_addr learned my addr 172.21.15.27:0/2242004412 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:32.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd4ffff640 1 --2- 172.21.15.27:0/2242004412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd501017a0 0x7fbd500754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:32.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd54fde640 1 --2- 172.21.15.27:0/2242004412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd50103350 0x7fbd50074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:32.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd4f7fe640 1 -- 172.21.15.27:0/2242004412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd50103350 msgr2=0x7fbd50074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd4f7fe640 1 --2- 172.21.15.27:0/2242004412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd50103350 0x7fbd50074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd4f7fe640 1 -- 172.21.15.27:0/2242004412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd501017a0 msgr2=0x7fbd500754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd4f7fe640 1 --2- 172.21.15.27:0/2242004412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd501017a0 0x7fbd500754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd4f7fe640 1 -- 172.21.15.27:0/2242004412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd50074900 con 0x7fbd50102990 2024-04-25T00:15:32.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.809+0000 7fbd4ffff640 1 --2- 172.21.15.27:0/2242004412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd501017a0 0x7fbd500754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:32.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.810+0000 7fbd4f7fe640 1 --2- 172.21.15.27:0/2242004412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd50102990 0x7fbd50073b30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800e9f0 tx=0x7fbd3800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:32.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.810+0000 7fbd4d7fa640 1 -- 172.21.15.27:0/2242004412 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3800cd80 con 0x7fbd50102990 2024-04-25T00:15:32.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.810+0000 7fbd56a41640 1 -- 172.21.15.27:0/2242004412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd501a1bf0 con 0x7fbd50102990 2024-04-25T00:15:32.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.810+0000 7fbd4d7fa640 1 -- 172.21.15.27:0/2242004412 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd38010910 con 0x7fbd50102990 2024-04-25T00:15:32.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.810+0000 7fbd4d7fa640 1 -- 172.21.15.27:0/2242004412 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38018950 con 0x7fbd50102990 2024-04-25T00:15:32.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.810+0000 7fbd56a41640 1 -- 172.21.15.27:0/2242004412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd501a2140 con 0x7fbd50102990 2024-04-25T00:15:32.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.811+0000 7fbd56a41640 1 -- 172.21.15.27:0/2242004412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd50108b30 con 0x7fbd50102990 2024-04-25T00:15:32.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.812+0000 7fbd4d7fa640 1 -- 172.21.15.27:0/2242004412 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbd38010ac0 con 0x7fbd50102990 2024-04-25T00:15:32.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.813+0000 7fbd4d7fa640 1 --2- 172.21.15.27:0/2242004412 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fbd3c0a2b90 0x7fbd3c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:32.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.813+0000 7fbd4d7fa640 1 -- 172.21.15.27:0/2242004412 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbd380cfc80 con 0x7fbd50102990 2024-04-25T00:15:32.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.813+0000 7fbd4ffff640 1 --2- 172.21.15.27:0/2242004412 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fbd3c0a2b90 0x7fbd3c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:32.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.813+0000 7fbd4ffff640 1 --2- 172.21.15.27:0/2242004412 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fbd3c0a2b90 0x7fbd3c0a5050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbd40006010 tx=0x7fbd40032cf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:32.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:32.816+0000 7fbd4d7fa640 1 -- 172.21.15.27:0/2242004412 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbd3809c850 con 0x7fbd50102990 2024-04-25T00:15:33.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.002+0000 7fbd56a41640 1 -- 172.21.15.27:0/2242004412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fbd5002d370 con 0x7fbd50102990 2024-04-25T00:15:33.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.002+0000 7fbd4d7fa640 1 -- 172.21.15.27:0/2242004412 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fbd3809c150 con 0x7fbd50102990 2024-04-25T00:15:33.003 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:15:33.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:32 smithi139 ceph-mon[32833]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:15:33.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:32 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3377276162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:33.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:32 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1412782830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:33.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:32 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2329032872' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:33.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:32 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3345340092' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:33.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.005+0000 7fbd56a41640 1 -- 172.21.15.27:0/2242004412 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fbd3c0a2b90 msgr2=0x7fbd3c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.005+0000 7fbd56a41640 1 --2- 172.21.15.27:0/2242004412 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fbd3c0a2b90 0x7fbd3c0a5050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbd40006010 tx=0x7fbd40032cf0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.005+0000 7fbd56a41640 1 -- 172.21.15.27:0/2242004412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd50102990 msgr2=0x7fbd50073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.005+0000 7fbd56a41640 1 --2- 172.21.15.27:0/2242004412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd50102990 0x7fbd50073b30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800e9f0 tx=0x7fbd3800eeb0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.005+0000 7fbd56a41640 1 -- 172.21.15.27:0/2242004412 shutdown_connections 2024-04-25T00:15:33.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.005+0000 7fbd56a41640 1 --2- 172.21.15.27:0/2242004412 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fbd3c0a2b90 0x7fbd3c0a5050 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.005+0000 7fbd56a41640 1 --2- 172.21.15.27:0/2242004412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd50103350 0x7fbd50074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.005+0000 7fbd56a41640 1 --2- 172.21.15.27:0/2242004412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd50102990 0x7fbd50073b30 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.005+0000 7fbd56a41640 1 --2- 172.21.15.27:0/2242004412 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd501017a0 0x7fbd500754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.005+0000 7fbd56a41640 1 -- 172.21.15.27:0/2242004412 >> 172.21.15.27:0/2242004412 conn(0x7fbd500fcd10 msgr2=0x7fbd50105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:33.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.005+0000 7fbd56a41640 1 -- 172.21.15.27:0/2242004412 shutdown_connections 2024-04-25T00:15:33.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.005+0000 7fbd56a41640 1 -- 172.21.15.27:0/2242004412 wait complete. 2024-04-25T00:15:33.016 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:32 smithi027 ceph-mon[29208]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:15:33.016 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:32 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3377276162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:33.016 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:32 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1412782830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:33.016 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:32 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2329032872' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:33.016 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:32 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3345340092' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:33.017 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:33.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:32 smithi153 ceph-mon[32896]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:15:33.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:32 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3377276162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:33.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:32 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1412782830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:33.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:32 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2329032872' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:33.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:32 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3345340092' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.118+0000 7f78d6e38640 1 -- 172.21.15.27:0/1380521024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d01006a0 msgr2=0x7f78d0100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.118+0000 7f78d6e38640 1 --2- 172.21.15.27:0/1380521024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d01006a0 0x7f78d0100ac0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009900 tx=0x7f78c402a410 comp rx=0 tx=0).stop 2024-04-25T00:15:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.118+0000 7f78d6e38640 1 -- 172.21.15.27:0/1380521024 shutdown_connections 2024-04-25T00:15:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.118+0000 7f78d6e38640 1 --2- 172.21.15.27:0/1380521024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d0102250 0x7f78d0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.118+0000 7f78d6e38640 1 --2- 172.21.15.27:0/1380521024 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78d0101890 0x7f78d0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.118+0000 7f78d6e38640 1 --2- 172.21.15.27:0/1380521024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d01006a0 0x7f78d0100ac0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.118+0000 7f78d6e38640 1 -- 172.21.15.27:0/1380521024 >> 172.21.15.27:0/1380521024 conn(0x7f78d00fbc30 msgr2=0x7f78d00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.118+0000 7f78d6e38640 1 -- 172.21.15.27:0/1380521024 shutdown_connections 2024-04-25T00:15:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.118+0000 7f78d6e38640 1 -- 172.21.15.27:0/1380521024 wait complete. 2024-04-25T00:15:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d6e38640 1 Processor -- start 2024-04-25T00:15:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d6e38640 1 -- start start 2024-04-25T00:15:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d6e38640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78d01006a0 0x7f78d0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d6e38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d0101890 0x7f78d0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d6e38640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d0102250 0x7f78d019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d6e38640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f78d010c320 con 0x7f78d0102250 2024-04-25T00:15:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d6e38640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f78d010c1a0 con 0x7f78d01006a0 2024-04-25T00:15:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d6e38640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f78d010c4a0 con 0x7f78d0101890 2024-04-25T00:15:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d53d5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d0102250 0x7f78d019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d53d5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d0102250 0x7f78d019f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56638/0 (socket says 172.21.15.27:56638) 2024-04-25T00:15:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d53d5640 1 -- 172.21.15.27:0/2901237150 learned_addr learned my addr 172.21.15.27:0/2901237150 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d4bd4640 1 --2- 172.21.15.27:0/2901237150 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78d01006a0 0x7f78d0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78cffff640 1 --2- 172.21.15.27:0/2901237150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d0101890 0x7f78d0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d53d5640 1 -- 172.21.15.27:0/2901237150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d0101890 msgr2=0x7f78d0198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d53d5640 1 --2- 172.21.15.27:0/2901237150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d0101890 0x7f78d0198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d53d5640 1 -- 172.21.15.27:0/2901237150 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78d01006a0 msgr2=0x7f78d0197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d53d5640 1 --2- 172.21.15.27:0/2901237150 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78d01006a0 0x7f78d0197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d53d5640 1 -- 172.21.15.27:0/2901237150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d019f6e0 con 0x7f78d0102250 2024-04-25T00:15:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78cffff640 1 --2- 172.21.15.27:0/2901237150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d0101890 0x7f78d0198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.119+0000 7f78d4bd4640 1 --2- 172.21.15.27:0/2901237150 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78d01006a0 0x7f78d0197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.120+0000 7f78d53d5640 1 --2- 172.21.15.27:0/2901237150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d0102250 0x7f78d019f070 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f78c800cad0 tx=0x7f78c800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.120+0000 7f78cdffb640 1 -- 172.21.15.27:0/2901237150 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c800ae20 con 0x7f78d0102250 2024-04-25T00:15:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.120+0000 7f78d6e38640 1 -- 172.21.15.27:0/2901237150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78d019f9d0 con 0x7f78d0102250 2024-04-25T00:15:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.120+0000 7f78cdffb640 1 -- 172.21.15.27:0/2901237150 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78c8005020 con 0x7f78d0102250 2024-04-25T00:15:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.120+0000 7f78cdffb640 1 -- 172.21.15.27:0/2901237150 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c800f6b0 con 0x7f78d0102250 2024-04-25T00:15:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.120+0000 7f78d6e38640 1 -- 172.21.15.27:0/2901237150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78d019ff70 con 0x7f78d0102250 2024-04-25T00:15:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.121+0000 7f78d6e38640 1 -- 172.21.15.27:0/2901237150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78d01a0200 con 0x7f78d0102250 2024-04-25T00:15:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.122+0000 7f78cdffb640 1 -- 172.21.15.27:0/2901237150 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f78c80051d0 con 0x7f78d0102250 2024-04-25T00:15:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.123+0000 7f78cdffb640 1 --2- 172.21.15.27:0/2901237150 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78c00a2b90 0x7f78c00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.123+0000 7f78d4bd4640 1 --2- 172.21.15.27:0/2901237150 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78c00a2b90 0x7f78c00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.123+0000 7f78cdffb640 1 -- 172.21.15.27:0/2901237150 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f78c80cf810 con 0x7f78d0102250 2024-04-25T00:15:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.123+0000 7f78d4bd4640 1 --2- 172.21.15.27:0/2901237150 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78c00a2b90 0x7f78c00a5050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f78c4006010 tx=0x7f78c4032dd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:33.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.127+0000 7f78cdffb640 1 -- 172.21.15.27:0/2901237150 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f78c8014030 con 0x7f78d0102250 2024-04-25T00:15:33.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.352+0000 7f78d6e38640 1 -- 172.21.15.27:0/2901237150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78d01a03d0 con 0x7f78d0102250 2024-04-25T00:15:33.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.352+0000 7f78cdffb640 1 -- 172.21.15.27:0/2901237150 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11944 (secure 0 0 0) 0x7f78c809c3e0 con 0x7f78d0102250 2024-04-25T00:15:33.353 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:33.353 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:13:47.575422+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24428,24496,24521,24430,24514]},"id":1}]} 2024-04-25T00:15:33.354 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 28 2024-04-25T00:15:33.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.354+0000 7f78d6e38640 1 -- 172.21.15.27:0/2901237150 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78c00a2b90 msgr2=0x7f78c00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.354+0000 7f78d6e38640 1 --2- 172.21.15.27:0/2901237150 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78c00a2b90 0x7f78c00a5050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f78c4006010 tx=0x7f78c4032dd0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.354+0000 7f78d6e38640 1 -- 172.21.15.27:0/2901237150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d0102250 msgr2=0x7f78d019f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.354+0000 7f78d6e38640 1 --2- 172.21.15.27:0/2901237150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d0102250 0x7f78d019f070 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f78c800cad0 tx=0x7f78c800cf90 comp rx=0 tx=0).stop 2024-04-25T00:15:33.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.354+0000 7f78d6e38640 1 -- 172.21.15.27:0/2901237150 shutdown_connections 2024-04-25T00:15:33.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.354+0000 7f78d6e38640 1 --2- 172.21.15.27:0/2901237150 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f78c00a2b90 0x7f78c00a5050 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.354+0000 7f78d6e38640 1 --2- 172.21.15.27:0/2901237150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d0102250 0x7f78d019f070 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.354+0000 7f78d6e38640 1 --2- 172.21.15.27:0/2901237150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78d0101890 0x7f78d0198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.354+0000 7f78d6e38640 1 --2- 172.21.15.27:0/2901237150 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78d01006a0 0x7f78d0197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.354+0000 7f78d6e38640 1 -- 172.21.15.27:0/2901237150 >> 172.21.15.27:0/2901237150 conn(0x7f78d00fbc30 msgr2=0x7f78d00fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:33.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.354+0000 7f78d6e38640 1 -- 172.21.15.27:0/2901237150 shutdown_connections 2024-04-25T00:15:33.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.355+0000 7f78d6e38640 1 -- 172.21.15.27:0/2901237150 wait complete. 2024-04-25T00:15:33.367 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:15:33.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.471+0000 7f6e779a2640 1 -- 172.21.15.27:0/3843962675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e701017b0 msgr2=0x7f6e70101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.471+0000 7f6e779a2640 1 --2- 172.21.15.27:0/3843962675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e701017b0 0x7f6e70101bd0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6e68009ca0 tx=0x7f6e6802a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.471+0000 7f6e779a2640 1 -- 172.21.15.27:0/3843962675 shutdown_connections 2024-04-25T00:15:33.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.471+0000 7f6e779a2640 1 --2- 172.21.15.27:0/3843962675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e70103360 0x7f6e70109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.471+0000 7f6e779a2640 1 --2- 172.21.15.27:0/3843962675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e701029a0 0x7f6e70102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.471+0000 7f6e779a2640 1 --2- 172.21.15.27:0/3843962675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e701017b0 0x7f6e70101bd0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.471+0000 7f6e779a2640 1 -- 172.21.15.27:0/3843962675 >> 172.21.15.27:0/3843962675 conn(0x7f6e700fcd40 msgr2=0x7f6e700ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:33.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.472+0000 7f6e779a2640 1 -- 172.21.15.27:0/3843962675 shutdown_connections 2024-04-25T00:15:33.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.472+0000 7f6e779a2640 1 -- 172.21.15.27:0/3843962675 wait complete. 2024-04-25T00:15:33.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.472+0000 7f6e779a2640 1 Processor -- start 2024-04-25T00:15:33.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.472+0000 7f6e779a2640 1 -- start start 2024-04-25T00:15:33.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.472+0000 7f6e779a2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e701017b0 0x7f6e70197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.472+0000 7f6e779a2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e701029a0 0x7f6e70197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.472+0000 7f6e779a2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e70103360 0x7f6e7019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.472+0000 7f6e779a2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7010d480 con 0x7f6e70103360 2024-04-25T00:15:33.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.472+0000 7f6e779a2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7010d300 con 0x7f6e701017b0 2024-04-25T00:15:33.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.472+0000 7f6e779a2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7010d600 con 0x7f6e701029a0 2024-04-25T00:15:33.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.472+0000 7f6e75f3f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e70103360 0x7f6e7019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.472+0000 7f6e75f3f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e70103360 0x7f6e7019f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56662/0 (socket says 172.21.15.27:56662) 2024-04-25T00:15:33.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.472+0000 7f6e75f3f640 1 -- 172.21.15.27:0/3182126100 learned_addr learned my addr 172.21.15.27:0/3182126100 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:33.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.473+0000 7f6e7573e640 1 --2- 172.21.15.27:0/3182126100 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e701017b0 0x7f6e70197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.473+0000 7f6e74f3d640 1 --2- 172.21.15.27:0/3182126100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e701029a0 0x7f6e70197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.473+0000 7f6e75f3f640 1 -- 172.21.15.27:0/3182126100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e701029a0 msgr2=0x7f6e70197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.473+0000 7f6e75f3f640 1 --2- 172.21.15.27:0/3182126100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e701029a0 0x7f6e70197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.473+0000 7f6e75f3f640 1 -- 172.21.15.27:0/3182126100 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e701017b0 msgr2=0x7f6e70197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.473+0000 7f6e75f3f640 1 --2- 172.21.15.27:0/3182126100 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e701017b0 0x7f6e70197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.473+0000 7f6e75f3f640 1 -- 172.21.15.27:0/3182126100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e7019f680 con 0x7f6e70103360 2024-04-25T00:15:33.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.473+0000 7f6e74f3d640 1 --2- 172.21.15.27:0/3182126100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e701029a0 0x7f6e70197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:33.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.473+0000 7f6e75f3f640 1 --2- 172.21.15.27:0/3182126100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e70103360 0x7f6e7019f010 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c009fd0 tx=0x7f6e6c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:33.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.473+0000 7f6e667fc640 1 -- 172.21.15.27:0/3182126100 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c011830 con 0x7f6e70103360 2024-04-25T00:15:33.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.473+0000 7f6e667fc640 1 -- 172.21.15.27:0/3182126100 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e6c016070 con 0x7f6e70103360 2024-04-25T00:15:33.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.473+0000 7f6e779a2640 1 -- 172.21.15.27:0/3182126100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e7019f970 con 0x7f6e70103360 2024-04-25T00:15:33.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.473+0000 7f6e667fc640 1 -- 172.21.15.27:0/3182126100 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c00f2e0 con 0x7f6e70103360 2024-04-25T00:15:33.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.473+0000 7f6e779a2640 1 -- 172.21.15.27:0/3182126100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e7019ff40 con 0x7f6e70103360 2024-04-25T00:15:33.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.474+0000 7f6e779a2640 1 -- 172.21.15.27:0/3182126100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e70108ad0 con 0x7f6e70103360 2024-04-25T00:15:33.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.475+0000 7f6e667fc640 1 -- 172.21.15.27:0/3182126100 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6e6c0119d0 con 0x7f6e70103360 2024-04-25T00:15:33.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.476+0000 7f6e667fc640 1 --2- 172.21.15.27:0/3182126100 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6e5c0a2c10 0x7f6e5c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.476+0000 7f6e7573e640 1 --2- 172.21.15.27:0/3182126100 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6e5c0a2c10 0x7f6e5c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.476+0000 7f6e667fc640 1 -- 172.21.15.27:0/3182126100 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6e6c0ce9e0 con 0x7f6e70103360 2024-04-25T00:15:33.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.476+0000 7f6e7573e640 1 --2- 172.21.15.27:0/3182126100 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6e5c0a2c10 0x7f6e5c0a50d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6e68006010 tx=0x7f6e68000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:33.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.481+0000 7f6e667fc640 1 -- 172.21.15.27:0/3182126100 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6e6c09b5b0 con 0x7f6e70103360 2024-04-25T00:15:33.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.667+0000 7f6e779a2640 1 -- 172.21.15.27:0/3182126100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6e70062340 con 0x7f6e70103360 2024-04-25T00:15:33.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.668+0000 7f6e667fc640 1 -- 172.21.15.27:0/3182126100 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24736 (secure 0 0 0) 0x7f6e6c09aeb0 con 0x7f6e70103360 2024-04-25T00:15:33.670 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:33.670 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":93,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:15:11.415317+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:15:33.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.670+0000 7f6e779a2640 1 -- 172.21.15.27:0/3182126100 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6e5c0a2c10 msgr2=0x7f6e5c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.670+0000 7f6e779a2640 1 --2- 172.21.15.27:0/3182126100 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6e5c0a2c10 0x7f6e5c0a50d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6e68006010 tx=0x7f6e68000f40 comp rx=0 tx=0).stop 2024-04-25T00:15:33.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.670+0000 7f6e779a2640 1 -- 172.21.15.27:0/3182126100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e70103360 msgr2=0x7f6e7019f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.670+0000 7f6e779a2640 1 --2- 172.21.15.27:0/3182126100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e70103360 0x7f6e7019f010 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c009fd0 tx=0x7f6e6c00b8e0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.670+0000 7f6e779a2640 1 -- 172.21.15.27:0/3182126100 shutdown_connections 2024-04-25T00:15:33.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.670+0000 7f6e779a2640 1 --2- 172.21.15.27:0/3182126100 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6e5c0a2c10 0x7f6e5c0a50d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.670+0000 7f6e779a2640 1 --2- 172.21.15.27:0/3182126100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e70103360 0x7f6e7019f010 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.670+0000 7f6e779a2640 1 --2- 172.21.15.27:0/3182126100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e701029a0 0x7f6e70197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.670+0000 7f6e779a2640 1 --2- 172.21.15.27:0/3182126100 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e701017b0 0x7f6e70197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.670+0000 7f6e779a2640 1 -- 172.21.15.27:0/3182126100 >> 172.21.15.27:0/3182126100 conn(0x7f6e700fcd40 msgr2=0x7f6e70105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:33.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.671+0000 7f6e779a2640 1 -- 172.21.15.27:0/3182126100 shutdown_connections 2024-04-25T00:15:33.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.671+0000 7f6e779a2640 1 -- 172.21.15.27:0/3182126100 wait complete. 2024-04-25T00:15:33.683 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:08:34.006973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T00:13:17.337307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T00:13:18.253713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T00:13:21.995987+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T00:15:33.684 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:15:33.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.789+0000 7f7b13688640 1 -- 172.21.15.27:0/4171450482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b0c1006a0 msgr2=0x7f7b0c100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.789+0000 7f7b13688640 1 --2- 172.21.15.27:0/4171450482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b0c1006a0 0x7f7b0c100ac0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009900 tx=0x7f7b0802a410 comp rx=0 tx=0).stop 2024-04-25T00:15:33.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.789+0000 7f7b13688640 1 -- 172.21.15.27:0/4171450482 shutdown_connections 2024-04-25T00:15:33.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.789+0000 7f7b13688640 1 --2- 172.21.15.27:0/4171450482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b0c102250 0x7f7b0c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.789+0000 7f7b13688640 1 --2- 172.21.15.27:0/4171450482 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b0c101890 0x7f7b0c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.789+0000 7f7b13688640 1 --2- 172.21.15.27:0/4171450482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b0c1006a0 0x7f7b0c100ac0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.789+0000 7f7b13688640 1 -- 172.21.15.27:0/4171450482 >> 172.21.15.27:0/4171450482 conn(0x7f7b0c0fbc30 msgr2=0x7f7b0c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:33.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.789+0000 7f7b13688640 1 -- 172.21.15.27:0/4171450482 shutdown_connections 2024-04-25T00:15:33.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.789+0000 7f7b13688640 1 -- 172.21.15.27:0/4171450482 wait complete. 2024-04-25T00:15:33.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.789+0000 7f7b13688640 1 Processor -- start 2024-04-25T00:15:33.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b13688640 1 -- start start 2024-04-25T00:15:33.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b13688640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b0c1006a0 0x7f7b0c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b13688640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b0c101890 0x7f7b0c198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b13688640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b0c102250 0x7f7b0c19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b13688640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0c10d440 con 0x7f7b0c1006a0 2024-04-25T00:15:33.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b13688640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0c10d2c0 con 0x7f7b0c102250 2024-04-25T00:15:33.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b13688640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0c10d5c0 con 0x7f7b0c101890 2024-04-25T00:15:33.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b11424640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b0c1006a0 0x7f7b0c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b11424640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b0c1006a0 0x7f7b0c197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56686/0 (socket says 172.21.15.27:56686) 2024-04-25T00:15:33.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b11424640 1 -- 172.21.15.27:0/3113454790 learned_addr learned my addr 172.21.15.27:0/3113454790 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:33.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b11c25640 1 --2- 172.21.15.27:0/3113454790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b0c102250 0x7f7b0c19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b10c23640 1 --2- 172.21.15.27:0/3113454790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b0c101890 0x7f7b0c198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b11424640 1 -- 172.21.15.27:0/3113454790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b0c101890 msgr2=0x7f7b0c198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b11424640 1 --2- 172.21.15.27:0/3113454790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b0c101890 0x7f7b0c198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b11424640 1 -- 172.21.15.27:0/3113454790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b0c102250 msgr2=0x7f7b0c19f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b11424640 1 --2- 172.21.15.27:0/3113454790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b0c102250 0x7f7b0c19f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b11424640 1 -- 172.21.15.27:0/3113454790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b0c19f6f0 con 0x7f7b0c1006a0 2024-04-25T00:15:33.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.790+0000 7f7b11424640 1 --2- 172.21.15.27:0/3113454790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b0c1006a0 0x7f7b0c197ad0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7b08031610 tx=0x7f7b080043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:33.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.791+0000 7f7b027fc640 1 -- 172.21.15.27:0/3113454790 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0803f030 con 0x7f7b0c1006a0 2024-04-25T00:15:33.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.791+0000 7f7b027fc640 1 -- 172.21.15.27:0/3113454790 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b08003ce0 con 0x7f7b0c1006a0 2024-04-25T00:15:33.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.791+0000 7f7b13688640 1 -- 172.21.15.27:0/3113454790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b0c19f980 con 0x7f7b0c1006a0 2024-04-25T00:15:33.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.791+0000 7f7b027fc640 1 -- 172.21.15.27:0/3113454790 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b08035070 con 0x7f7b0c1006a0 2024-04-25T00:15:33.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.791+0000 7f7b13688640 1 -- 172.21.15.27:0/3113454790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b0c19fe70 con 0x7f7b0c1006a0 2024-04-25T00:15:33.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.792+0000 7f7b13688640 1 -- 172.21.15.27:0/3113454790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b0c107a40 con 0x7f7b0c1006a0 2024-04-25T00:15:33.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.793+0000 7f7b027fc640 1 -- 172.21.15.27:0/3113454790 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7b08042030 con 0x7f7b0c1006a0 2024-04-25T00:15:33.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.793+0000 7f7b027fc640 1 --2- 172.21.15.27:0/3113454790 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7af80a2b90 0x7f7af80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.794+0000 7f7b10c23640 1 --2- 172.21.15.27:0/3113454790 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7af80a2b90 0x7f7af80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.794+0000 7f7b027fc640 1 -- 172.21.15.27:0/3113454790 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7b080ef590 con 0x7f7b0c1006a0 2024-04-25T00:15:33.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.794+0000 7f7b10c23640 1 --2- 172.21.15.27:0/3113454790 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7af80a2b90 0x7f7af80a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c198fe0 tx=0x7f7afc00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:33.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.798+0000 7f7b027fc640 1 -- 172.21.15.27:0/3113454790 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7b080bbb10 con 0x7f7b0c1006a0 2024-04-25T00:15:33.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:33 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2242004412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:33.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:33 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2901237150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:33.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:33 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3182126100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:33.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.982+0000 7f7b13688640 1 -- 172.21.15.27:0/3113454790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7b0c02d3d0 con 0x7f7b0c1006a0 2024-04-25T00:15:33.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.982+0000 7f7b027fc640 1 -- 172.21.15.27:0/3113454790 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f7b080bfe10 con 0x7f7b0c1006a0 2024-04-25T00:15:33.983 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T00:15:33.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.984+0000 7f7b13688640 1 -- 172.21.15.27:0/3113454790 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7af80a2b90 msgr2=0x7f7af80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.984+0000 7f7b13688640 1 --2- 172.21.15.27:0/3113454790 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7af80a2b90 0x7f7af80a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c198fe0 tx=0x7f7afc00a2d0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.984+0000 7f7b13688640 1 -- 172.21.15.27:0/3113454790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b0c1006a0 msgr2=0x7f7b0c197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.984+0000 7f7b13688640 1 --2- 172.21.15.27:0/3113454790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b0c1006a0 0x7f7b0c197ad0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7b08031610 tx=0x7f7b080043b0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.984+0000 7f7b13688640 1 -- 172.21.15.27:0/3113454790 shutdown_connections 2024-04-25T00:15:33.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.984+0000 7f7b13688640 1 --2- 172.21.15.27:0/3113454790 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7af80a2b90 0x7f7af80a5050 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.984+0000 7f7b13688640 1 --2- 172.21.15.27:0/3113454790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b0c102250 0x7f7b0c19f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.984+0000 7f7b13688640 1 --2- 172.21.15.27:0/3113454790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b0c101890 0x7f7b0c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.984+0000 7f7b13688640 1 --2- 172.21.15.27:0/3113454790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b0c1006a0 0x7f7b0c197ad0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.984+0000 7f7b13688640 1 -- 172.21.15.27:0/3113454790 >> 172.21.15.27:0/3113454790 conn(0x7f7b0c0fbc30 msgr2=0x7f7b0c104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:33.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.984+0000 7f7b13688640 1 -- 172.21.15.27:0/3113454790 shutdown_connections 2024-04-25T00:15:33.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:33.984+0000 7f7b13688640 1 -- 172.21.15.27:0/3113454790 wait complete. 2024-04-25T00:15:33.996 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T00:15:34.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:33 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2242004412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:34.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:33 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2901237150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:34.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:33 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3182126100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:34.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:33 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2242004412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:34.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:33 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2901237150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:34.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:33 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3182126100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.103+0000 7f524541c640 1 -- 172.21.15.27:0/2302499046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52401012f0 msgr2=0x7f5240101770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.103+0000 7f524541c640 1 --2- 172.21.15.27:0/2302499046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52401012f0 0x7f5240101770 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5238009ca0 tx=0x7f523802a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.103+0000 7f524541c640 1 -- 172.21.15.27:0/2302499046 shutdown_connections 2024-04-25T00:15:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.103+0000 7f524541c640 1 --2- 172.21.15.27:0/2302499046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52401012f0 0x7f5240101770 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.103+0000 7f524541c640 1 --2- 172.21.15.27:0/2302499046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5240100950 0x7f5240100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.103+0000 7f524541c640 1 --2- 172.21.15.27:0/2302499046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52400ffff0 0x7f5240100410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.103+0000 7f524541c640 1 -- 172.21.15.27:0/2302499046 >> 172.21.15.27:0/2302499046 conn(0x7f52400fbc30 msgr2=0x7f52400fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.103+0000 7f524541c640 1 -- 172.21.15.27:0/2302499046 shutdown_connections 2024-04-25T00:15:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.103+0000 7f524541c640 1 -- 172.21.15.27:0/2302499046 wait complete. 2024-04-25T00:15:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.103+0000 7f524541c640 1 Processor -- start 2024-04-25T00:15:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.103+0000 7f524541c640 1 -- start start 2024-04-25T00:15:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f524541c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52400ffff0 0x7f5240199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f524541c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5240100950 0x7f524019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f524541c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52401012f0 0x7f52401a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f524541c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f524010f5f0 con 0x7f5240100950 2024-04-25T00:15:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f524541c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f524010f470 con 0x7f52400ffff0 2024-04-25T00:15:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f524541c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f524010f770 con 0x7f52401012f0 2024-04-25T00:15:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f523e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5240100950 0x7f524019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f523e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5240100950 0x7f524019a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56698/0 (socket says 172.21.15.27:56698) 2024-04-25T00:15:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f523e7fc640 1 -- 172.21.15.27:0/2566088530 learned_addr learned my addr 172.21.15.27:0/2566088530 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f523f7fe640 1 --2- 172.21.15.27:0/2566088530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52401012f0 0x7f52401a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f523effd640 1 --2- 172.21.15.27:0/2566088530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52400ffff0 0x7f5240199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f523e7fc640 1 -- 172.21.15.27:0/2566088530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52401012f0 msgr2=0x7f52401a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f523e7fc640 1 --2- 172.21.15.27:0/2566088530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52401012f0 0x7f52401a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f523e7fc640 1 -- 172.21.15.27:0/2566088530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52400ffff0 msgr2=0x7f5240199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f523e7fc640 1 --2- 172.21.15.27:0/2566088530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52400ffff0 0x7f5240199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f523e7fc640 1 -- 172.21.15.27:0/2566088530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52401a1890 con 0x7f5240100950 2024-04-25T00:15:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f523f7fe640 1 --2- 172.21.15.27:0/2566088530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52401012f0 0x7f52401a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f523e7fc640 1 --2- 172.21.15.27:0/2566088530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5240100950 0x7f524019a1c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f523400d770 tx=0x7f523400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f5227fff640 1 -- 172.21.15.27:0/2566088530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5234017070 con 0x7f5240100950 2024-04-25T00:15:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f5227fff640 1 -- 172.21.15.27:0/2566088530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5234005500 con 0x7f5240100950 2024-04-25T00:15:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f524541c640 1 -- 172.21.15.27:0/2566088530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52401a1b80 con 0x7f5240100950 2024-04-25T00:15:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f5227fff640 1 -- 172.21.15.27:0/2566088530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5234004e00 con 0x7f5240100950 2024-04-25T00:15:34.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.104+0000 7f524541c640 1 -- 172.21.15.27:0/2566088530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52401a2080 con 0x7f5240100950 2024-04-25T00:15:34.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.105+0000 7f524541c640 1 -- 172.21.15.27:0/2566088530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f524010ae70 con 0x7f5240100950 2024-04-25T00:15:34.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.107+0000 7f5227fff640 1 -- 172.21.15.27:0/2566088530 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5234004fa0 con 0x7f5240100950 2024-04-25T00:15:34.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.107+0000 7f5227fff640 1 --2- 172.21.15.27:0/2566088530 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f52280a2b90 0x7f52280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.107+0000 7f523effd640 1 --2- 172.21.15.27:0/2566088530 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f52280a2b90 0x7f52280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.107+0000 7f5227fff640 1 -- 172.21.15.27:0/2566088530 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f52340cef90 con 0x7f5240100950 2024-04-25T00:15:34.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.108+0000 7f523effd640 1 --2- 172.21.15.27:0/2566088530 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f52280a2b90 0x7f52280a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f524019b150 tx=0x7f5230005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:34.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.112+0000 7f5227fff640 1 -- 172.21.15.27:0/2566088530 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f523409bb60 con 0x7f5240100950 2024-04-25T00:15:34.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.296+0000 7f524541c640 1 -- 172.21.15.27:0/2566088530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f52400623e0 con 0x7f5240100950 2024-04-25T00:15:34.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.297+0000 7f5227fff640 1 -- 172.21.15.27:0/2566088530 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f523409b460 con 0x7f5240100950 2024-04-25T00:15:34.298 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:15:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.299+0000 7f524541c640 1 -- 172.21.15.27:0/2566088530 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f52280a2b90 msgr2=0x7f52280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.299+0000 7f524541c640 1 --2- 172.21.15.27:0/2566088530 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f52280a2b90 0x7f52280a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f524019b150 tx=0x7f5230005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.299+0000 7f524541c640 1 -- 172.21.15.27:0/2566088530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5240100950 msgr2=0x7f524019a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.299+0000 7f524541c640 1 --2- 172.21.15.27:0/2566088530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5240100950 0x7f524019a1c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f523400d770 tx=0x7f523400dc30 comp rx=0 tx=0).stop 2024-04-25T00:15:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.299+0000 7f524541c640 1 -- 172.21.15.27:0/2566088530 shutdown_connections 2024-04-25T00:15:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.299+0000 7f524541c640 1 --2- 172.21.15.27:0/2566088530 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f52280a2b90 0x7f52280a5050 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.299+0000 7f524541c640 1 --2- 172.21.15.27:0/2566088530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52401012f0 0x7f52401a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.299+0000 7f524541c640 1 --2- 172.21.15.27:0/2566088530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5240100950 0x7f524019a1c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.299+0000 7f524541c640 1 --2- 172.21.15.27:0/2566088530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52400ffff0 0x7f5240199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.299+0000 7f524541c640 1 -- 172.21.15.27:0/2566088530 >> 172.21.15.27:0/2566088530 conn(0x7f52400fbc30 msgr2=0x7f52400fd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:34.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.299+0000 7f524541c640 1 -- 172.21.15.27:0/2566088530 shutdown_connections 2024-04-25T00:15:34.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.299+0000 7f524541c640 1 -- 172.21.15.27:0/2566088530 wait complete. 2024-04-25T00:15:34.311 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T00:15:34.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.416+0000 7f6e603a2640 1 -- 172.21.15.27:0/1458252946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e58100670 msgr2=0x7f6e58100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.416+0000 7f6e603a2640 1 --2- 172.21.15.27:0/1458252946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e58100670 0x7f6e58100a90 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009900 tx=0x7f6e5402a410 comp rx=0 tx=0).stop 2024-04-25T00:15:34.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.416+0000 7f6e603a2640 1 -- 172.21.15.27:0/1458252946 shutdown_connections 2024-04-25T00:15:34.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.416+0000 7f6e603a2640 1 --2- 172.21.15.27:0/1458252946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e58102220 0x7f6e58108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.416+0000 7f6e603a2640 1 --2- 172.21.15.27:0/1458252946 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e58101860 0x7f6e58101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.416+0000 7f6e603a2640 1 --2- 172.21.15.27:0/1458252946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e58100670 0x7f6e58100a90 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.416+0000 7f6e603a2640 1 -- 172.21.15.27:0/1458252946 >> 172.21.15.27:0/1458252946 conn(0x7f6e580fbbe0 msgr2=0x7f6e580fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:34.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.416+0000 7f6e603a2640 1 -- 172.21.15.27:0/1458252946 shutdown_connections 2024-04-25T00:15:34.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.416+0000 7f6e603a2640 1 -- 172.21.15.27:0/1458252946 wait complete. 2024-04-25T00:15:34.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.417+0000 7f6e603a2640 1 Processor -- start 2024-04-25T00:15:34.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.417+0000 7f6e603a2640 1 -- start start 2024-04-25T00:15:34.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.417+0000 7f6e603a2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e58100670 0x7f6e58197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.417+0000 7f6e603a2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e58101860 0x7f6e58198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.417+0000 7f6e603a2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e58102220 0x7f6e5819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.417+0000 7f6e603a2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5810c2f0 con 0x7f6e58101860 2024-04-25T00:15:34.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.417+0000 7f6e603a2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5810c170 con 0x7f6e58100670 2024-04-25T00:15:34.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.417+0000 7f6e603a2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5810c470 con 0x7f6e58102220 2024-04-25T00:15:34.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.417+0000 7f6e5d93d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e58101860 0x7f6e58198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.417+0000 7f6e5d93d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e58101860 0x7f6e58198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56722/0 (socket says 172.21.15.27:56722) 2024-04-25T00:15:34.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.417+0000 7f6e5d93d640 1 -- 172.21.15.27:0/565596642 learned_addr learned my addr 172.21.15.27:0/565596642 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:34.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.417+0000 7f6e5e13e640 1 --2- 172.21.15.27:0/565596642 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e58100670 0x7f6e58197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.417+0000 7f6e5d93d640 1 -- 172.21.15.27:0/565596642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e58102220 msgr2=0x7f6e5819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.417+0000 7f6e5d93d640 1 --2- 172.21.15.27:0/565596642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e58102220 0x7f6e5819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.417+0000 7f6e5d93d640 1 -- 172.21.15.27:0/565596642 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e58100670 msgr2=0x7f6e58197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.417+0000 7f6e5d93d640 1 --2- 172.21.15.27:0/565596642 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e58100670 0x7f6e58197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.417+0000 7f6e5d93d640 1 -- 172.21.15.27:0/565596642 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e5819f690 con 0x7f6e58101860 2024-04-25T00:15:34.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.418+0000 7f6e5d93d640 1 --2- 172.21.15.27:0/565596642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e58101860 0x7f6e58198020 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6e4800c920 tx=0x7f6e4800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:34.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.418+0000 7f6e4f7fe640 1 -- 172.21.15.27:0/565596642 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e48007c70 con 0x7f6e58101860 2024-04-25T00:15:34.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.418+0000 7f6e4f7fe640 1 -- 172.21.15.27:0/565596642 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e48007e10 con 0x7f6e58101860 2024-04-25T00:15:34.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.418+0000 7f6e603a2640 1 -- 172.21.15.27:0/565596642 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e5819f980 con 0x7f6e58101860 2024-04-25T00:15:34.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.418+0000 7f6e4f7fe640 1 -- 172.21.15.27:0/565596642 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e48018890 con 0x7f6e58101860 2024-04-25T00:15:34.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.418+0000 7f6e603a2640 1 -- 172.21.15.27:0/565596642 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e5819fed0 con 0x7f6e58101860 2024-04-25T00:15:34.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.419+0000 7f6e603a2640 1 -- 172.21.15.27:0/565596642 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e581a0160 con 0x7f6e58101860 2024-04-25T00:15:34.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.420+0000 7f6e4f7fe640 1 -- 172.21.15.27:0/565596642 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6e48018a70 con 0x7f6e58101860 2024-04-25T00:15:34.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.421+0000 7f6e4f7fe640 1 --2- 172.21.15.27:0/565596642 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6e440a2b90 0x7f6e440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.421+0000 7f6e5e13e640 1 --2- 172.21.15.27:0/565596642 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6e440a2b90 0x7f6e440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.421+0000 7f6e4f7fe640 1 -- 172.21.15.27:0/565596642 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6e480cf5d0 con 0x7f6e58101860 2024-04-25T00:15:34.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.421+0000 7f6e5e13e640 1 --2- 172.21.15.27:0/565596642 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6e440a2b90 0x7f6e440a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6e54006010 tx=0x7f6e54000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:34.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.425+0000 7f6e4f7fe640 1 -- 172.21.15.27:0/565596642 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6e48014030 con 0x7f6e58101860 2024-04-25T00:15:34.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.610+0000 7f6e603a2640 1 -- 172.21.15.27:0/565596642 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6e581a0330 con 0x7f6e58101860 2024-04-25T00:15:34.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.611+0000 7f6e4f7fe640 1 -- 172.21.15.27:0/565596642 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f6e4800de20 con 0x7f6e58101860 2024-04-25T00:15:34.612 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T00:15:34.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.613+0000 7f6e603a2640 1 -- 172.21.15.27:0/565596642 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6e440a2b90 msgr2=0x7f6e440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.613+0000 7f6e603a2640 1 --2- 172.21.15.27:0/565596642 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6e440a2b90 0x7f6e440a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6e54006010 tx=0x7f6e54000bc0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.613+0000 7f6e603a2640 1 -- 172.21.15.27:0/565596642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e58101860 msgr2=0x7f6e58198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.613+0000 7f6e603a2640 1 --2- 172.21.15.27:0/565596642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e58101860 0x7f6e58198020 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6e4800c920 tx=0x7f6e4800cde0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.613+0000 7f6e603a2640 1 -- 172.21.15.27:0/565596642 shutdown_connections 2024-04-25T00:15:34.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.613+0000 7f6e603a2640 1 --2- 172.21.15.27:0/565596642 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f6e440a2b90 0x7f6e440a5050 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.613+0000 7f6e603a2640 1 --2- 172.21.15.27:0/565596642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e58102220 0x7f6e5819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.613+0000 7f6e603a2640 1 --2- 172.21.15.27:0/565596642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e58101860 0x7f6e58198020 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.613+0000 7f6e603a2640 1 --2- 172.21.15.27:0/565596642 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e58100670 0x7f6e58197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.613+0000 7f6e603a2640 1 -- 172.21.15.27:0/565596642 >> 172.21.15.27:0/565596642 conn(0x7f6e580fbbe0 msgr2=0x7f6e580fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:34.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.613+0000 7f6e603a2640 1 -- 172.21.15.27:0/565596642 shutdown_connections 2024-04-25T00:15:34.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.613+0000 7f6e603a2640 1 -- 172.21.15.27:0/565596642 wait complete. 2024-04-25T00:15:34.625 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T00:15:34.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.732+0000 7f911505d640 1 -- 172.21.15.27:0/3055724627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9110101790 msgr2=0x7f9110101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.732+0000 7f911505d640 1 --2- 172.21.15.27:0/3055724627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9110101790 0x7f9110101bb0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9100009900 tx=0x7f910002a3f0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.732+0000 7f911505d640 1 -- 172.21.15.27:0/3055724627 shutdown_connections 2024-04-25T00:15:34.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.732+0000 7f911505d640 1 --2- 172.21.15.27:0/3055724627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9110103340 0x7f9110109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.732+0000 7f911505d640 1 --2- 172.21.15.27:0/3055724627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9110102980 0x7f9110102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.732+0000 7f911505d640 1 --2- 172.21.15.27:0/3055724627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9110101790 0x7f9110101bb0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.732+0000 7f911505d640 1 -- 172.21.15.27:0/3055724627 >> 172.21.15.27:0/3055724627 conn(0x7f91100fcd20 msgr2=0x7f91100ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:34.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.732+0000 7f911505d640 1 -- 172.21.15.27:0/3055724627 shutdown_connections 2024-04-25T00:15:34.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.732+0000 7f911505d640 1 -- 172.21.15.27:0/3055724627 wait complete. 2024-04-25T00:15:34.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.732+0000 7f911505d640 1 Processor -- start 2024-04-25T00:15:34.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f911505d640 1 -- start start 2024-04-25T00:15:34.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f911505d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9110101790 0x7f9110197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f911505d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9110102980 0x7f9110197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f911505d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9110103340 0x7f911019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f911505d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f911010d3e0 con 0x7f9110101790 2024-04-25T00:15:34.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f911505d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f911010d260 con 0x7f9110102980 2024-04-25T00:15:34.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f911505d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f911010d560 con 0x7f9110103340 2024-04-25T00:15:34.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f910ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9110101790 0x7f9110197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f910ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9110101790 0x7f9110197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56752/0 (socket says 172.21.15.27:56752) 2024-04-25T00:15:34.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f910ed9d640 1 -- 172.21.15.27:0/3747698751 learned_addr learned my addr 172.21.15.27:0/3747698751 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:34.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f910f59e640 1 --2- 172.21.15.27:0/3747698751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9110103340 0x7f911019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f910e59c640 1 --2- 172.21.15.27:0/3747698751 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9110102980 0x7f9110197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f910ed9d640 1 -- 172.21.15.27:0/3747698751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9110103340 msgr2=0x7f911019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f910ed9d640 1 --2- 172.21.15.27:0/3747698751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9110103340 0x7f911019f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f910ed9d640 1 -- 172.21.15.27:0/3747698751 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9110102980 msgr2=0x7f9110197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f910ed9d640 1 --2- 172.21.15.27:0/3747698751 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9110102980 0x7f9110197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f910ed9d640 1 -- 172.21.15.27:0/3747698751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f911019f6a0 con 0x7f9110101790 2024-04-25T00:15:34.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.733+0000 7f910e59c640 1 --2- 172.21.15.27:0/3747698751 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9110102980 0x7f9110197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:34.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.734+0000 7f910ed9d640 1 --2- 172.21.15.27:0/3747698751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9110101790 0x7f9110197a80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9100031610 tx=0x7f91000043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:34.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.734+0000 7f9107fff640 1 -- 172.21.15.27:0/3747698751 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910003f030 con 0x7f9110101790 2024-04-25T00:15:34.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.734+0000 7f911505d640 1 -- 172.21.15.27:0/3747698751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f911019f930 con 0x7f9110101790 2024-04-25T00:15:34.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.734+0000 7f9107fff640 1 -- 172.21.15.27:0/3747698751 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9100003ce0 con 0x7f9110101790 2024-04-25T00:15:34.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.734+0000 7f9107fff640 1 -- 172.21.15.27:0/3747698751 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9100035070 con 0x7f9110101790 2024-04-25T00:15:34.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.734+0000 7f911505d640 1 -- 172.21.15.27:0/3747698751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f911019fe70 con 0x7f9110101790 2024-04-25T00:15:34.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.735+0000 7f911505d640 1 -- 172.21.15.27:0/3747698751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9110108b30 con 0x7f9110101790 2024-04-25T00:15:34.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.736+0000 7f9107fff640 1 -- 172.21.15.27:0/3747698751 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9100042030 con 0x7f9110101790 2024-04-25T00:15:34.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.736+0000 7f9107fff640 1 --2- 172.21.15.27:0/3747698751 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f90fc0a2c10 0x7f90fc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.737+0000 7f910e59c640 1 --2- 172.21.15.27:0/3747698751 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f90fc0a2c10 0x7f90fc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.737+0000 7f9107fff640 1 -- 172.21.15.27:0/3747698751 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f91000efa00 con 0x7f9110101790 2024-04-25T00:15:34.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.737+0000 7f910e59c640 1 --2- 172.21.15.27:0/3747698751 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f90fc0a2c10 0x7f90fc0a50d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9110198f90 tx=0x7f90f800a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:34.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.741+0000 7f9107fff640 1 -- 172.21.15.27:0/3747698751 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f91000bc5d0 con 0x7f9110101790 2024-04-25T00:15:34.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.928+0000 7f911505d640 1 -- 172.21.15.27:0/3747698751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f91100623e0 con 0x7f9110101790 2024-04-25T00:15:34.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.929+0000 7f9107fff640 1 -- 172.21.15.27:0/3747698751 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f91000bbed0 con 0x7f9110101790 2024-04-25T00:15:34.930 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:15:34.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.930+0000 7f911505d640 1 -- 172.21.15.27:0/3747698751 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f90fc0a2c10 msgr2=0x7f90fc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.930+0000 7f911505d640 1 --2- 172.21.15.27:0/3747698751 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f90fc0a2c10 0x7f90fc0a50d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9110198f90 tx=0x7f90f800a5c0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.930+0000 7f911505d640 1 -- 172.21.15.27:0/3747698751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9110101790 msgr2=0x7f9110197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.930+0000 7f911505d640 1 --2- 172.21.15.27:0/3747698751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9110101790 0x7f9110197a80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9100031610 tx=0x7f91000043b0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.930+0000 7f911505d640 1 -- 172.21.15.27:0/3747698751 shutdown_connections 2024-04-25T00:15:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.930+0000 7f911505d640 1 --2- 172.21.15.27:0/3747698751 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f90fc0a2c10 0x7f90fc0a50d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.930+0000 7f911505d640 1 --2- 172.21.15.27:0/3747698751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9110103340 0x7f911019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.930+0000 7f911505d640 1 --2- 172.21.15.27:0/3747698751 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9110102980 0x7f9110197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.930+0000 7f911505d640 1 --2- 172.21.15.27:0/3747698751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9110101790 0x7f9110197a80 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.930+0000 7f911505d640 1 -- 172.21.15.27:0/3747698751 >> 172.21.15.27:0/3747698751 conn(0x7f91100fcd20 msgr2=0x7f9110105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.931+0000 7f911505d640 1 -- 172.21.15.27:0/3747698751 shutdown_connections 2024-04-25T00:15:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:34.931+0000 7f911505d640 1 -- 172.21.15.27:0/3747698751 wait complete. 2024-04-25T00:15:34.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:34 smithi027 ceph-mon[29208]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-25T00:15:34.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:34 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3113454790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:34.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:34 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2566088530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:34.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:34 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/565596642' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:34.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:35.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:34 smithi139 ceph-mon[32833]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-25T00:15:35.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:34 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3113454790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:35.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:34 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2566088530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:35.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:34 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/565596642' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:35.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:34 smithi153 ceph-mon[32896]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-25T00:15:35.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:34 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3113454790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:35.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:34 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2566088530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:35.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:34 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/565596642' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:35.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.043+0000 7f14f5076640 1 -- 172.21.15.27:0/896693190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14f01029a0 msgr2=0x7f14f0102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.043+0000 7f14f5076640 1 --2- 172.21.15.27:0/896693190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14f01029a0 0x7f14f0102e20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f14e0009900 tx=0x7f14e002a410 comp rx=0 tx=0).stop 2024-04-25T00:15:35.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.044+0000 7f14f5076640 1 -- 172.21.15.27:0/896693190 shutdown_connections 2024-04-25T00:15:35.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.044+0000 7f14f5076640 1 --2- 172.21.15.27:0/896693190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14f0103360 0x7f14f0109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.044+0000 7f14f5076640 1 --2- 172.21.15.27:0/896693190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14f01029a0 0x7f14f0102e20 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.044+0000 7f14f5076640 1 --2- 172.21.15.27:0/896693190 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14f01017b0 0x7f14f0101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.044+0000 7f14f5076640 1 -- 172.21.15.27:0/896693190 >> 172.21.15.27:0/896693190 conn(0x7f14f00fcd40 msgr2=0x7f14f00ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.044+0000 7f14f5076640 1 -- 172.21.15.27:0/896693190 shutdown_connections 2024-04-25T00:15:35.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.044+0000 7f14f5076640 1 -- 172.21.15.27:0/896693190 wait complete. 2024-04-25T00:15:35.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.044+0000 7f14f5076640 1 Processor -- start 2024-04-25T00:15:35.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.044+0000 7f14f5076640 1 -- start start 2024-04-25T00:15:35.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14f5076640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14f01017b0 0x7f14f0197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14f5076640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14f01029a0 0x7f14f0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14f5076640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14f0103360 0x7f14f019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14f5076640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f14f010d430 con 0x7f14f0103360 2024-04-25T00:15:35.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14f5076640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f14f010d2b0 con 0x7f14f01029a0 2024-04-25T00:15:35.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14f5076640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f14f010d5b0 con 0x7f14f01017b0 2024-04-25T00:15:35.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14ef59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14f0103360 0x7f14f019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14ef59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14f0103360 0x7f14f019f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56780/0 (socket says 172.21.15.27:56780) 2024-04-25T00:15:35.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14ef59e640 1 -- 172.21.15.27:0/2638895567 learned_addr learned my addr 172.21.15.27:0/2638895567 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:35.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14ee59c640 1 --2- 172.21.15.27:0/2638895567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14f01029a0 0x7f14f0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14eed9d640 1 --2- 172.21.15.27:0/2638895567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14f01017b0 0x7f14f0197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14ef59e640 1 -- 172.21.15.27:0/2638895567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14f01017b0 msgr2=0x7f14f0197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14ef59e640 1 --2- 172.21.15.27:0/2638895567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14f01017b0 0x7f14f0197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14ef59e640 1 -- 172.21.15.27:0/2638895567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14f01029a0 msgr2=0x7f14f0197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14ef59e640 1 --2- 172.21.15.27:0/2638895567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14f01029a0 0x7f14f0197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14ef59e640 1 -- 172.21.15.27:0/2638895567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f019f6a0 con 0x7f14f0103360 2024-04-25T00:15:35.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14ee59c640 1 --2- 172.21.15.27:0/2638895567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14f01029a0 0x7f14f0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:35.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14ef59e640 1 --2- 172.21.15.27:0/2638895567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14f0103360 0x7f14f019f030 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f14e800eb10 tx=0x7f14e800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14d7fff640 1 -- 172.21.15.27:0/2638895567 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e8004b20 con 0x7f14f0103360 2024-04-25T00:15:35.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14f5076640 1 -- 172.21.15.27:0/2638895567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14f019f990 con 0x7f14f0103360 2024-04-25T00:15:35.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14d7fff640 1 -- 172.21.15.27:0/2638895567 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14e8004cc0 con 0x7f14f0103360 2024-04-25T00:15:35.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14d7fff640 1 -- 172.21.15.27:0/2638895567 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e8010760 con 0x7f14f0103360 2024-04-25T00:15:35.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.045+0000 7f14f5076640 1 -- 172.21.15.27:0/2638895567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14f019feb0 con 0x7f14f0103360 2024-04-25T00:15:35.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.047+0000 7f14f5076640 1 -- 172.21.15.27:0/2638895567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14f0108ad0 con 0x7f14f0103360 2024-04-25T00:15:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.048+0000 7f14d7fff640 1 -- 172.21.15.27:0/2638895567 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f14e8005020 con 0x7f14f0103360 2024-04-25T00:15:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.048+0000 7f14d7fff640 1 --2- 172.21.15.27:0/2638895567 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f14d80a2b90 0x7f14d80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.048+0000 7f14eed9d640 1 --2- 172.21.15.27:0/2638895567 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f14d80a2b90 0x7f14d80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.048+0000 7f14d7fff640 1 -- 172.21.15.27:0/2638895567 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f14e80cfbc0 con 0x7f14f0103360 2024-04-25T00:15:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.049+0000 7f14eed9d640 1 --2- 172.21.15.27:0/2638895567 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f14d80a2b90 0x7f14d80a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f14f0102800 tx=0x7f14e4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.053+0000 7f14d7fff640 1 -- 172.21.15.27:0/2638895567 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f14e809c790 con 0x7f14f0103360 2024-04-25T00:15:35.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.278+0000 7f14f5076640 1 -- 172.21.15.27:0/2638895567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14f002d170 con 0x7f14f0103360 2024-04-25T00:15:35.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.279+0000 7f14d7fff640 1 -- 172.21.15.27:0/2638895567 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11944 (secure 0 0 0) 0x7f14e809c090 con 0x7f14f0103360 2024-04-25T00:15:35.280 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:35.280 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:13:47.575422+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24428,24496,24521,24430,24514]},"id":1}]} 2024-04-25T00:15:35.281 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 28 2024-04-25T00:15:35.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.281+0000 7f14f5076640 1 -- 172.21.15.27:0/2638895567 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f14d80a2b90 msgr2=0x7f14d80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.281+0000 7f14f5076640 1 --2- 172.21.15.27:0/2638895567 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f14d80a2b90 0x7f14d80a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f14f0102800 tx=0x7f14e4005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:35.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.281+0000 7f14f5076640 1 -- 172.21.15.27:0/2638895567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14f0103360 msgr2=0x7f14f019f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.281+0000 7f14f5076640 1 --2- 172.21.15.27:0/2638895567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14f0103360 0x7f14f019f030 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f14e800eb10 tx=0x7f14e800c2d0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.281+0000 7f14f5076640 1 -- 172.21.15.27:0/2638895567 shutdown_connections 2024-04-25T00:15:35.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.281+0000 7f14f5076640 1 --2- 172.21.15.27:0/2638895567 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f14d80a2b90 0x7f14d80a5050 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.281+0000 7f14f5076640 1 --2- 172.21.15.27:0/2638895567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14f0103360 0x7f14f019f030 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.281+0000 7f14f5076640 1 --2- 172.21.15.27:0/2638895567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14f01029a0 0x7f14f0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.281+0000 7f14f5076640 1 --2- 172.21.15.27:0/2638895567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14f01017b0 0x7f14f0197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.281+0000 7f14f5076640 1 -- 172.21.15.27:0/2638895567 >> 172.21.15.27:0/2638895567 conn(0x7f14f00fcd40 msgr2=0x7f14f0105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.282+0000 7f14f5076640 1 -- 172.21.15.27:0/2638895567 shutdown_connections 2024-04-25T00:15:35.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.282+0000 7f14f5076640 1 -- 172.21.15.27:0/2638895567 wait complete. 2024-04-25T00:15:35.295 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:15:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.398+0000 7fef61812640 1 -- 172.21.15.27:0/2603923914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c100680 msgr2=0x7fef5c100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.398+0000 7fef61812640 1 --2- 172.21.15.27:0/2603923914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c100680 0x7fef5c100aa0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fef4c009900 tx=0x7fef4c02a410 comp rx=0 tx=0).stop 2024-04-25T00:15:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.398+0000 7fef61812640 1 -- 172.21.15.27:0/2603923914 shutdown_connections 2024-04-25T00:15:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.398+0000 7fef61812640 1 --2- 172.21.15.27:0/2603923914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef5c102230 0x7fef5c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.399+0000 7fef61812640 1 --2- 172.21.15.27:0/2603923914 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef5c101870 0x7fef5c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.399+0000 7fef61812640 1 --2- 172.21.15.27:0/2603923914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c100680 0x7fef5c100aa0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.399+0000 7fef61812640 1 -- 172.21.15.27:0/2603923914 >> 172.21.15.27:0/2603923914 conn(0x7fef5c0fbbf0 msgr2=0x7fef5c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.399+0000 7fef61812640 1 -- 172.21.15.27:0/2603923914 shutdown_connections 2024-04-25T00:15:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.399+0000 7fef61812640 1 -- 172.21.15.27:0/2603923914 wait complete. 2024-04-25T00:15:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.399+0000 7fef61812640 1 Processor -- start 2024-04-25T00:15:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.399+0000 7fef61812640 1 -- start start 2024-04-25T00:15:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.399+0000 7fef61812640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef5c100680 0x7fef5c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef61812640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c101870 0x7fef5c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef61812640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef5c102230 0x7fef5c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef61812640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c10c300 con 0x7fef5c101870 2024-04-25T00:15:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef61812640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c10c180 con 0x7fef5c100680 2024-04-25T00:15:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef61812640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c10c480 con 0x7fef5c102230 2024-04-25T00:15:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef5a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c101870 0x7fef5c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef5a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c101870 0x7fef5c197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56800/0 (socket says 172.21.15.27:56800) 2024-04-25T00:15:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef5a7fc640 1 -- 172.21.15.27:0/1594519559 learned_addr learned my addr 172.21.15.27:0/1594519559 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef5affd640 1 --2- 172.21.15.27:0/1594519559 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef5c100680 0x7fef5c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef5b7fe640 1 --2- 172.21.15.27:0/1594519559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef5c102230 0x7fef5c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef5a7fc640 1 -- 172.21.15.27:0/1594519559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef5c102230 msgr2=0x7fef5c19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef5a7fc640 1 --2- 172.21.15.27:0/1594519559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef5c102230 0x7fef5c19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef5a7fc640 1 -- 172.21.15.27:0/1594519559 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef5c100680 msgr2=0x7fef5c197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef5a7fc640 1 --2- 172.21.15.27:0/1594519559 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef5c100680 0x7fef5c197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef5a7fc640 1 -- 172.21.15.27:0/1594519559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5c19f6c0 con 0x7fef5c101870 2024-04-25T00:15:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef5affd640 1 --2- 172.21.15.27:0/1594519559 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef5c100680 0x7fef5c197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef5a7fc640 1 --2- 172.21.15.27:0/1594519559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c101870 0x7fef5c197fe0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fef5000d950 tx=0x7fef5000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef60810640 1 -- 172.21.15.27:0/1594519559 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef50017030 con 0x7fef5c101870 2024-04-25T00:15:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef61812640 1 -- 172.21.15.27:0/1594519559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef5c19f9b0 con 0x7fef5c101870 2024-04-25T00:15:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef60810640 1 -- 172.21.15.27:0/1594519559 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef50004020 con 0x7fef5c101870 2024-04-25T00:15:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef60810640 1 -- 172.21.15.27:0/1594519559 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef500047a0 con 0x7fef5c101870 2024-04-25T00:15:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.400+0000 7fef61812640 1 -- 172.21.15.27:0/1594519559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef5c19ff50 con 0x7fef5c101870 2024-04-25T00:15:35.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.401+0000 7fef61812640 1 -- 172.21.15.27:0/1594519559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef5c1a01e0 con 0x7fef5c101870 2024-04-25T00:15:35.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.403+0000 7fef60810640 1 -- 172.21.15.27:0/1594519559 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fef500041d0 con 0x7fef5c101870 2024-04-25T00:15:35.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.403+0000 7fef60810640 1 --2- 172.21.15.27:0/1594519559 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fef440a2b90 0x7fef440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.404+0000 7fef5affd640 1 --2- 172.21.15.27:0/1594519559 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fef440a2b90 0x7fef440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.404+0000 7fef60810640 1 -- 172.21.15.27:0/1594519559 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fef500cf4b0 con 0x7fef5c101870 2024-04-25T00:15:35.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.404+0000 7fef5affd640 1 --2- 172.21.15.27:0/1594519559 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fef440a2b90 0x7fef440a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fef4c006010 tx=0x7fef4c032cf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.408+0000 7fef60810640 1 -- 172.21.15.27:0/1594519559 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fef5009c080 con 0x7fef5c101870 2024-04-25T00:15:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.601+0000 7fef61812640 1 -- 172.21.15.27:0/1594519559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fef5c1a03b0 con 0x7fef5c101870 2024-04-25T00:15:35.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.603+0000 7fef60810640 1 -- 172.21.15.27:0/1594519559 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24736 (secure 0 0 0) 0x7fef5009b980 con 0x7fef5c101870 2024-04-25T00:15:35.604 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:35.604 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":93,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:15:11.415317+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:15:35.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.605+0000 7fef61812640 1 -- 172.21.15.27:0/1594519559 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fef440a2b90 msgr2=0x7fef440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.605+0000 7fef61812640 1 --2- 172.21.15.27:0/1594519559 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fef440a2b90 0x7fef440a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fef4c006010 tx=0x7fef4c032cf0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.605+0000 7fef61812640 1 -- 172.21.15.27:0/1594519559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c101870 msgr2=0x7fef5c197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.605+0000 7fef61812640 1 --2- 172.21.15.27:0/1594519559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c101870 0x7fef5c197fe0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fef5000d950 tx=0x7fef5000de10 comp rx=0 tx=0).stop 2024-04-25T00:15:35.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.605+0000 7fef61812640 1 -- 172.21.15.27:0/1594519559 shutdown_connections 2024-04-25T00:15:35.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.605+0000 7fef61812640 1 --2- 172.21.15.27:0/1594519559 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fef440a2b90 0x7fef440a5050 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.605+0000 7fef61812640 1 --2- 172.21.15.27:0/1594519559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef5c102230 0x7fef5c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.605+0000 7fef61812640 1 --2- 172.21.15.27:0/1594519559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c101870 0x7fef5c197fe0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.605+0000 7fef61812640 1 --2- 172.21.15.27:0/1594519559 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef5c100680 0x7fef5c197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.605+0000 7fef61812640 1 -- 172.21.15.27:0/1594519559 >> 172.21.15.27:0/1594519559 conn(0x7fef5c0fbbf0 msgr2=0x7fef5c0fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.605+0000 7fef61812640 1 -- 172.21.15.27:0/1594519559 shutdown_connections 2024-04-25T00:15:35.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.605+0000 7fef61812640 1 -- 172.21.15.27:0/1594519559 wait complete. 2024-04-25T00:15:35.618 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T00:13:18.872210+0000', 'modified': '2024-04-25T00:13:47.575422+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24521, 'mds_1': 24514, 'mds_2': 24496, 'mds_3': 24430, 'mds_4': 24428}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14652': {'gid': 14652, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.27:6839/3286639641', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6838', 'nonce': 3286639641}, {'type': 'v1', 'addr': '172.21.15.27:6839', 'nonce': 3286639641}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14664': {'gid': 14664, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.27:6841/2271898034', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6840', 'nonce': 2271898034}, {'type': 'v1', 'addr': '172.21.15.27:6841', 'nonce': 2271898034}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24428': {'gid': 24428, 'name': 'k', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.139:6833/3964352148', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6832', 'nonce': 3964352148}, {'type': 'v1', 'addr': '172.21.15.139:6833', 'nonce': 3964352148}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24430': {'gid': 24430, 'name': 'l', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.153:6833/2700717364', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6832', 'nonce': 2700717364}, {'type': 'v1', 'addr': '172.21.15.153:6833', 'nonce': 2700717364}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24461': {'gid': 24461, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.139:6835/3731284556', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6834', 'nonce': 3731284556}, {'type': 'v1', 'addr': '172.21.15.139:6835', 'nonce': 3731284556}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6835/584718807', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6834', 'nonce': 584718807}, {'type': 'v1', 'addr': '172.21.15.153:6835', 'nonce': 584718807}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.139:6837/2477304919', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6836', 'nonce': 2477304919}, {'type': 'v1', 'addr': '172.21.15.139:6837', 'nonce': 2477304919}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24496': {'gid': 24496, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.153:6837/4066071541', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6836', 'nonce': 4066071541}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 4066071541}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24514': {'gid': 24514, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.153:6839/1144169732', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 1144169732}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 1144169732}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24521': {'gid': 24521, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.139:6839/2728523692', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6838', 'nonce': 2728523692}, {'type': 'v1', 'addr': '172.21.15.139:6839', 'nonce': 2728523692}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24521, 'qdb_cluster': [24428, 24496, 24521, 24430, 24514]} 2024-04-25T00:15:35.619 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-25T00:15:35.619 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-04-25T00:15:35.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.730+0000 7fb288c9e640 1 -- 172.21.15.27:0/2831845358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb284101870 msgr2=0x7fb284101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.730+0000 7fb288c9e640 1 --2- 172.21.15.27:0/2831845358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb284101870 0x7fb284101cf0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fb278009900 tx=0x7fb27802a410 comp rx=0 tx=0).stop 2024-04-25T00:15:35.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.731+0000 7fb288c9e640 1 -- 172.21.15.27:0/2831845358 shutdown_connections 2024-04-25T00:15:35.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.731+0000 7fb288c9e640 1 --2- 172.21.15.27:0/2831845358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb284102230 0x7fb284108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.731+0000 7fb288c9e640 1 --2- 172.21.15.27:0/2831845358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb284101870 0x7fb284101cf0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.731+0000 7fb288c9e640 1 --2- 172.21.15.27:0/2831845358 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb284100680 0x7fb284100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.731+0000 7fb288c9e640 1 -- 172.21.15.27:0/2831845358 >> 172.21.15.27:0/2831845358 conn(0x7fb2840fbbf0 msgr2=0x7fb2840fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.731+0000 7fb288c9e640 1 -- 172.21.15.27:0/2831845358 shutdown_connections 2024-04-25T00:15:35.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.731+0000 7fb288c9e640 1 -- 172.21.15.27:0/2831845358 wait complete. 2024-04-25T00:15:35.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb288c9e640 1 Processor -- start 2024-04-25T00:15:35.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb288c9e640 1 -- start start 2024-04-25T00:15:35.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb288c9e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb284100680 0x7fb284069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb288c9e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb284102230 0x7fb28419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb288c9e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb28419f370 0x7fb2841a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb288c9e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb28410c300 con 0x7fb284100680 2024-04-25T00:15:35.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb288c9e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb28410c180 con 0x7fb284102230 2024-04-25T00:15:35.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb288c9e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb28410c480 con 0x7fb28419f370 2024-04-25T00:15:35.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb28259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb284100680 0x7fb284069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb28259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb284100680 0x7fb284069440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56808/0 (socket says 172.21.15.27:56808) 2024-04-25T00:15:35.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb28259c640 1 -- 172.21.15.27:0/1201784912 learned_addr learned my addr 172.21.15.27:0/1201784912 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:35.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb282d9d640 1 --2- 172.21.15.27:0/1201784912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb28419f370 0x7fb2841a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb281d9b640 1 --2- 172.21.15.27:0/1201784912 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb284102230 0x7fb28419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb28259c640 1 -- 172.21.15.27:0/1201784912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb28419f370 msgr2=0x7fb2841a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb28259c640 1 --2- 172.21.15.27:0/1201784912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb28419f370 0x7fb2841a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb28259c640 1 -- 172.21.15.27:0/1201784912 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb284102230 msgr2=0x7fb28419ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb28259c640 1 --2- 172.21.15.27:0/1201784912 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb284102230 0x7fb28419ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb28259c640 1 -- 172.21.15.27:0/1201784912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2841a1f50 con 0x7fb284100680 2024-04-25T00:15:35.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.732+0000 7fb281d9b640 1 --2- 172.21.15.27:0/1201784912 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb284102230 0x7fb28419ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:35.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.733+0000 7fb28259c640 1 --2- 172.21.15.27:0/1201784912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb284100680 0x7fb284069440 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb27400d770 tx=0x7fb27400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.733+0000 7fb2737fe640 1 -- 172.21.15.27:0/1201784912 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb274017070 con 0x7fb284100680 2024-04-25T00:15:35.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.733+0000 7fb2737fe640 1 -- 172.21.15.27:0/1201784912 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb274009c30 con 0x7fb284100680 2024-04-25T00:15:35.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.733+0000 7fb288c9e640 1 -- 172.21.15.27:0/1201784912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2841a21e0 con 0x7fb284100680 2024-04-25T00:15:35.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.733+0000 7fb2737fe640 1 -- 172.21.15.27:0/1201784912 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb274003dd0 con 0x7fb284100680 2024-04-25T00:15:35.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.733+0000 7fb288c9e640 1 -- 172.21.15.27:0/1201784912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2841a2700 con 0x7fb284100680 2024-04-25T00:15:35.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.735+0000 7fb28b9ad740 1 Processor -- start 2024-04-25T00:15:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb28b9ad740 1 -- start start 2024-04-25T00:15:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb28b9ad740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80a440 0x5590ce80a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb28b9ad740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce80f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb28b9ad740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80ffa0 0x5590ce812430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb28b9ad740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5590ce76df00 con 0x5590ce80ae30 2024-04-25T00:15:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb28b9ad740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5590ce716750 con 0x5590ce80a440 2024-04-25T00:15:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb28b9ad740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5590ce769ec0 con 0x5590ce80ffa0 2024-04-25T00:15:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb282d9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80a440 0x5590ce80a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb282d9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80a440 0x5590ce80a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:60910/0 (socket says 172.21.15.27:60910) 2024-04-25T00:15:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb281d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80ffa0 0x5590ce812430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb282d9d640 1 -- 172.21.15.27:0/3350278894 learned_addr learned my addr 172.21.15.27:0/3350278894 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb2737fe640 1 -- 172.21.15.27:0/1201784912 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb274003f70 con 0x7fb284100680 2024-04-25T00:15:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb282d9d640 1 -- 172.21.15.27:0/3350278894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80ffa0 msgr2=0x5590ce812430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb282d9d640 1 --2- 172.21.15.27:0/3350278894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80ffa0 0x5590ce812430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb28259c640 1 --2- 172.21.15.27:0/3350278894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce80f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb282d9d640 1 -- 172.21.15.27:0/3350278894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 msgr2=0x5590ce80f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb282d9d640 1 --2- 172.21.15.27:0/3350278894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce80f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb282d9d640 1 -- 172.21.15.27:0/3350278894 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590ce4822d0 con 0x5590ce80a440 2024-04-25T00:15:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb281d9b640 1 --2- 172.21.15.27:0/3350278894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80ffa0 0x5590ce812430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.736+0000 7fb28259c640 1 --2- 172.21.15.27:0/3350278894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce80f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.737+0000 7fb282d9d640 1 --2- 172.21.15.27:0/3350278894 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80a440 0x5590ce80a860 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb27c010370 tx=0x7fb27c010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aceaf61237d915c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.737+0000 7fb270ff9640 1 -- 172.21.15.27:0/3350278894 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27c016070 con 0x5590ce80a440 2024-04-25T00:15:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.737+0000 7fb2737fe640 1 --2- 172.21.15.27:0/1201784912 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb26c0a2bc0 0x7fb26c0a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.737+0000 7fb270ff9640 1 -- 172.21.15.27:0/3350278894 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb27c004d20 con 0x5590ce80a440 2024-04-25T00:15:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.737+0000 7fb270ff9640 1 -- 172.21.15.27:0/3350278894 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27c005480 con 0x5590ce80a440 2024-04-25T00:15:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.737+0000 7fb28259c640 1 --2- 172.21.15.27:0/1201784912 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb26c0a2bc0 0x7fb26c0a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.737+0000 7fb2737fe640 1 -- 172.21.15.27:0/1201784912 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb2740ced30 con 0x7fb284100680 2024-04-25T00:15:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 1 -- 172.21.15.27:0/3350278894 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80a440 msgr2=0x5590ce80a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/3350278894 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80a440 0x5590ce80a860 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb27c010370 tx=0x7fb27c010830 comp rx=0 tx=0).stop 2024-04-25T00:15:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28259c640 1 --2- 172.21.15.27:0/1201784912 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb26c0a2bc0 0x7fb26c0a5080 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb2740cc810 tx=0x7fb274000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 1 -- 172.21.15.27:0/3350278894 shutdown_connections 2024-04-25T00:15:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/3350278894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80ffa0 0x5590ce812430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/3350278894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce80f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/3350278894 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80a440 0x5590ce80a860 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 1 -- 172.21.15.27:0/3350278894 >> 172.21.15.27:0/3350278894 conn(0x5590ce805ab0 msgr2=0x5590ce805eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 1 -- 172.21.15.27:0/3350278894 shutdown_connections 2024-04-25T00:15:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 1 -- 172.21.15.27:0/3350278894 wait complete. 2024-04-25T00:15:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 1 -- start start 2024-04-25T00:15:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 1 Processor -- start 2024-04-25T00:15:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80a440 0x5590ce814850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce814d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80ffa0 0x5590ce81bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5590ce812f50 con 0x5590ce80ae30 2024-04-25T00:15:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5590ce729cf0 con 0x5590ce80a440 2024-04-25T00:15:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5590ce8130d0 con 0x5590ce80ffa0 2024-04-25T00:15:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.738+0000 7fb28b9ad740 10 client.0 _ll_register_callbacks cb 0x5590ce7296e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb281d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce814d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb281d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce814d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56824/0 (socket says 172.21.15.27:56824) 2024-04-25T00:15:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb281d9b640 1 -- 172.21.15.27:0/402238855 learned_addr learned my addr 172.21.15.27:0/402238855 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb282d9d640 1 --2- 172.21.15.27:0/402238855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80a440 0x5590ce814850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb282d9d640 1 --2- 172.21.15.27:0/402238855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80ffa0 0x5590ce81bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb281d9b640 1 -- 172.21.15.27:0/402238855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80ffa0 msgr2=0x5590ce81bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb281d9b640 1 --2- 172.21.15.27:0/402238855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80ffa0 0x5590ce81bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb281d9b640 1 -- 172.21.15.27:0/402238855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80a440 msgr2=0x5590ce814850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:35.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb281d9b640 1 --2- 172.21.15.27:0/402238855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80a440 0x5590ce814850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb281d9b640 1 -- 172.21.15.27:0/402238855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590ce54c6b0 con 0x5590ce80ae30 2024-04-25T00:15:35.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb281d9b640 1 --2- 172.21.15.27:0/402238855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce814d90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb284101300 tx=0x7fb278004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff4c14413a5f3771 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb269ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:35.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb269ffb640 1 -- 172.21.15.27:0/402238855 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb278035040 con 0x5590ce80ae30 2024-04-25T00:15:35.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb28b9ad740 1 -- 172.21.15.27:0/402238855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb27c004d20 con 0x5590ce80ae30 2024-04-25T00:15:35.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb269ffb640 1 -- 172.21.15.27:0/402238855 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb278005020 con 0x5590ce80ae30 2024-04-25T00:15:35.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb269ffb640 1 -- 172.21.15.27:0/402238855 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb27804d050 con 0x5590ce80ae30 2024-04-25T00:15:35.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb28b9ad740 10 client.15462 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:35.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.739+0000 7fb28b9ad740 1 -- 172.21.15.27:0/402238855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5590ce81ed90 con 0x5590ce80ae30 2024-04-25T00:15:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.740+0000 7fb269ffb640 1 -- 172.21.15.27:0/402238855 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb27c004d20 con 0x5590ce80ae30 2024-04-25T00:15:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.740+0000 7fb269ffb640 1 -- 172.21.15.27:0/402238855 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb2780035f0 con 0x5590ce80ae30 2024-04-25T00:15:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.740+0000 7fb28b9ad740 10 client.15462 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.740+0000 7fb28b9ad740 10 client.15462 resolve_mds: resolved name 'k' to daemon mds.k 2024-04-25T00:15:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.740+0000 7fb28b9ad740 20 client.15462 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.740+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/402238855 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5590ce8213e0 0x5590ce822850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.740+0000 7fb28b9ad740 4 client.15462 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:15:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.740+0000 7fb28b9ad740 1 -- 172.21.15.27:0/402238855 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590ce77b440 con 0x5590ce8213e0 2024-04-25T00:15:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.740+0000 7fb282d9d640 1 --2- 172.21.15.27:0/402238855 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5590ce8213e0 0x5590ce822850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.741+0000 7fb282d9d640 1 --2- 172.21.15.27:0/402238855 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5590ce8213e0 0x5590ce822850 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7f7fa5932ca2222a server_cookie=f0cf402d29c35d27 in_seq=0 out_seq=0 2024-04-25T00:15:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.741+0000 7fb269ffb640 10 client.15462 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:15:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.741+0000 7fb269ffb640 1 -- 172.21.15.27:0/402238855 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5590ce77b440 con 0x5590ce8213e0 2024-04-25T00:15:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.741+0000 7fb269ffb640 10 client.15462 handle_command_reply: tid=0 2024-04-25T00:15:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.741+0000 7fb28b9ad740 1 client.15462 shutdown 2024-04-25T00:15:35.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.741+0000 7fb28b9ad740 1 -- 172.21.15.27:0/402238855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 msgr2=0x5590ce814d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.741+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/402238855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce814d90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb284101300 tx=0x7fb278004270 comp rx=0 tx=0).stop 2024-04-25T00:15:35.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.741+0000 7fb28b9ad740 1 -- 172.21.15.27:0/402238855 shutdown_connections 2024-04-25T00:15:35.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.741+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/402238855 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5590ce8213e0 0x5590ce822850 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.741+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/402238855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80ffa0 0x5590ce81bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.741+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/402238855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce814d90 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.741+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/402238855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80a440 0x5590ce814850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.741+0000 7fb28b9ad740 1 -- 172.21.15.27:0/402238855 >> 172.21.15.27:0/402238855 conn(0x5590ce805ab0 msgr2=0x5590ce804ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.741+0000 7fb269ffb640 0 client.15462 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:15:35.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.742+0000 7fb28b9ad740 1 -- 172.21.15.27:0/402238855 shutdown_connections 2024-04-25T00:15:35.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.742+0000 7fb28b9ad740 1 -- 172.21.15.27:0/402238855 wait complete. 2024-04-25T00:15:35.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.742+0000 7fb28b9ad740 20 client.15462 trim_cache size 0 max 16384 2024-04-25T00:15:35.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.760+0000 7fb28b9ad740 1 Processor -- start 2024-04-25T00:15:35.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.760+0000 7fb28b9ad740 1 -- start start 2024-04-25T00:15:35.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.760+0000 7fb28b9ad740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80a440 0x5590ce80b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.760+0000 7fb28b9ad740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce81bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.760+0000 7fb28b9ad740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80ffa0 0x5590ce812390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.760+0000 7fb28b9ad740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5590ce820b90 con 0x5590ce80ae30 2024-04-25T00:15:35.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.760+0000 7fb28b9ad740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5590ce821260 con 0x5590ce80ffa0 2024-04-25T00:15:35.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.760+0000 7fb28b9ad740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5590ce821930 con 0x5590ce80a440 2024-04-25T00:15:35.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.760+0000 7fb281d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce81bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.760+0000 7fb281d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce81bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56836/0 (socket says 172.21.15.27:56836) 2024-04-25T00:15:35.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.760+0000 7fb281d9b640 1 -- 172.21.15.27:0/114163975 learned_addr learned my addr 172.21.15.27:0/114163975 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:35.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.761+0000 7fb282d9d640 1 --2- 172.21.15.27:0/114163975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80a440 0x5590ce80b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.761+0000 7fb281d9b640 1 -- 172.21.15.27:0/114163975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80a440 msgr2=0x5590ce80b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:35.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.761+0000 7fb281d9b640 1 --2- 172.21.15.27:0/114163975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80a440 0x5590ce80b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.761+0000 7fb281d9b640 1 -- 172.21.15.27:0/114163975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80ffa0 msgr2=0x5590ce812390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:35.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.761+0000 7fb281d9b640 1 --2- 172.21.15.27:0/114163975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80ffa0 0x5590ce812390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.761+0000 7fb281d9b640 1 -- 172.21.15.27:0/114163975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590ce80b6c0 con 0x5590ce80ae30 2024-04-25T00:15:35.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.761+0000 7fb282d9d640 1 --2- 172.21.15.27:0/114163975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80a440 0x5590ce80b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:35.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.761+0000 7fb281d9b640 1 --2- 172.21.15.27:0/114163975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce81bd90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb278000c00 tx=0x7fb27802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32449dd42499d224 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.761+0000 7fb26bfff640 1 -- 172.21.15.27:0/114163975 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb278035050 con 0x5590ce80ae30 2024-04-25T00:15:35.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.761+0000 7fb26bfff640 1 -- 172.21.15.27:0/114163975 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb278056070 con 0x5590ce80ae30 2024-04-25T00:15:35.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.761+0000 7fb26bfff640 1 -- 172.21.15.27:0/114163975 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb278003800 con 0x5590ce80ae30 2024-04-25T00:15:35.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.762+0000 7fb28b9ad740 1 -- 172.21.15.27:0/114163975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 msgr2=0x5590ce81bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.762+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/114163975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce81bd90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb278000c00 tx=0x7fb27802a410 comp rx=0 tx=0).stop 2024-04-25T00:15:35.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.762+0000 7fb28b9ad740 1 -- 172.21.15.27:0/114163975 shutdown_connections 2024-04-25T00:15:35.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.762+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/114163975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80ffa0 0x5590ce812390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.762+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/114163975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce81bd90 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.762+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/114163975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80a440 0x5590ce80b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.762+0000 7fb28b9ad740 1 -- 172.21.15.27:0/114163975 >> 172.21.15.27:0/114163975 conn(0x5590ce805ab0 msgr2=0x5590ce805e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.762+0000 7fb28b9ad740 1 -- 172.21.15.27:0/114163975 shutdown_connections 2024-04-25T00:15:35.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.762+0000 7fb28b9ad740 1 -- 172.21.15.27:0/114163975 wait complete. 2024-04-25T00:15:35.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.762+0000 7fb28b9ad740 1 -- start start 2024-04-25T00:15:35.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.762+0000 7fb28b9ad740 1 Processor -- start 2024-04-25T00:15:35.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb28b9ad740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80a440 0x5590ce819ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb28b9ad740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce883370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb28b9ad740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80ffa0 0x5590ce8858c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb28b9ad740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5590ce8226d0 con 0x5590ce80ae30 2024-04-25T00:15:35.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb28b9ad740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5590ce822000 con 0x5590ce80ffa0 2024-04-25T00:15:35.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb28b9ad740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5590ce822da0 con 0x5590ce80a440 2024-04-25T00:15:35.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb28b9ad740 10 client.0 _ll_register_callbacks cb 0x7fb2780318a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:35.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb281d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce883370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb281d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce883370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56846/0 (socket says 172.21.15.27:56846) 2024-04-25T00:15:35.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb281d9b640 1 -- 172.21.15.27:0/719222571 learned_addr learned my addr 172.21.15.27:0/719222571 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:35.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb282d9d640 1 --2- 172.21.15.27:0/719222571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80ffa0 0x5590ce8858c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb282d9d640 1 --2- 172.21.15.27:0/719222571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80a440 0x5590ce819ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb281d9b640 1 -- 172.21.15.27:0/719222571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80a440 msgr2=0x5590ce819ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:35.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb281d9b640 1 --2- 172.21.15.27:0/719222571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80a440 0x5590ce819ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb281d9b640 1 -- 172.21.15.27:0/719222571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80ffa0 msgr2=0x5590ce8858c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:35.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb281d9b640 1 --2- 172.21.15.27:0/719222571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80ffa0 0x5590ce8858c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb281d9b640 1 -- 172.21.15.27:0/719222571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590ce885fb0 con 0x5590ce80ae30 2024-04-25T00:15:35.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb281d9b640 1 --2- 172.21.15.27:0/719222571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce883370 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb278009850 tx=0x7fb278004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=694a230cdf5f7d01 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb269ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:35.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb269ffb640 1 -- 172.21.15.27:0/719222571 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb278035070 con 0x5590ce80ae30 2024-04-25T00:15:35.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb269ffb640 1 -- 172.21.15.27:0/719222571 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb278003df0 con 0x5590ce80ae30 2024-04-25T00:15:35.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.763+0000 7fb28b9ad740 1 -- 172.21.15.27:0/719222571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb278056070 con 0x5590ce80ae30 2024-04-25T00:15:35.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.764+0000 7fb269ffb640 1 -- 172.21.15.27:0/719222571 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb278005040 con 0x5590ce80ae30 2024-04-25T00:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.764+0000 7fb28b9ad740 10 client.15468 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.764+0000 7fb28b9ad740 1 -- 172.21.15.27:0/719222571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5590ce887200 con 0x5590ce80ae30 2024-04-25T00:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.764+0000 7fb269ffb640 1 -- 172.21.15.27:0/719222571 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb278056070 con 0x5590ce80ae30 2024-04-25T00:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.764+0000 7fb269ffb640 1 -- 172.21.15.27:0/719222571 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb27805a450 con 0x5590ce80ae30 2024-04-25T00:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.764+0000 7fb28b9ad740 10 client.15468 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.764+0000 7fb28b9ad740 10 client.15468 resolve_mds: resolved name 'k' to daemon mds.k 2024-04-25T00:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.764+0000 7fb28b9ad740 20 client.15468 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.764+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/719222571 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5590ce88bad0 0x5590ce88a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.764+0000 7fb28b9ad740 4 client.15468 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T00:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.764+0000 7fb28b9ad740 1 -- 172.21.15.27:0/719222571 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "status"}) -- 0x5590ce775790 con 0x5590ce88bad0 2024-04-25T00:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.764+0000 7fb282d9d640 1 --2- 172.21.15.27:0/719222571 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5590ce88bad0 0x5590ce88a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.765+0000 7fb282d9d640 1 --2- 172.21.15.27:0/719222571 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5590ce88bad0 0x5590ce88a7e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3429d94ffe4d79ee server_cookie=68ddd66f54dc7ce5 in_seq=0 out_seq=0 2024-04-25T00:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.765+0000 7fb269ffb640 10 client.15468 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:15:35.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.765+0000 7fb269ffb640 1 -- 172.21.15.27:0/719222571 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5590ce775790 con 0x5590ce88bad0 2024-04-25T00:15:35.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.765+0000 7fb269ffb640 10 client.15468 handle_command_reply: tid=0 2024-04-25T00:15:35.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.765+0000 7fb28b9ad740 1 client.15468 shutdown 2024-04-25T00:15:35.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.765+0000 7fb28b9ad740 1 -- 172.21.15.27:0/719222571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 msgr2=0x5590ce883370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.765+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/719222571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce883370 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb278009850 tx=0x7fb278004250 comp rx=0 tx=0).stop 2024-04-25T00:15:35.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.765+0000 7fb28b9ad740 1 -- 172.21.15.27:0/719222571 shutdown_connections 2024-04-25T00:15:35.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.765+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/719222571 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5590ce88bad0 0x5590ce88a7e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.765+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/719222571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5590ce80ffa0 0x5590ce8858c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.765+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/719222571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5590ce80ae30 0x5590ce883370 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.765+0000 7fb28b9ad740 1 --2- 172.21.15.27:0/719222571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590ce80a440 0x5590ce819ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.765+0000 7fb28b9ad740 1 -- 172.21.15.27:0/719222571 >> 172.21.15.27:0/719222571 conn(0x5590ce805ab0 msgr2=0x5590ce800690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.765+0000 7fb269ffb640 0 client.15468 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:15:35.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.765+0000 7fb28b9ad740 1 -- 172.21.15.27:0/719222571 shutdown_connections 2024-04-25T00:15:35.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.765+0000 7fb28b9ad740 1 -- 172.21.15.27:0/719222571 wait complete. 2024-04-25T00:15:35.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.765+0000 7fb28b9ad740 20 client.15468 trim_cache size 0 max 16384 2024-04-25T00:15:35.774 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:15:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "518857f8-0297-11ef-bc93-c7b262605968", 2024-04-25T00:15:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 4, 2024-04-25T00:15:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24428, 2024-04-25T00:15:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T00:15:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T00:15:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T00:15:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 109.19445154100001, 2024-04-25T00:15:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 28, 2024-04-25T00:15:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 93, 2024-04-25T00:15:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T00:15:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 171.321034115 2024-04-25T00:15:35.775 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:15:35.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.766+0000 7fb288c9e640 1 -- 172.21.15.27:0/1201784912 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb26c0a2bc0 msgr2=0x7fb26c0a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.766+0000 7fb288c9e640 1 --2- 172.21.15.27:0/1201784912 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb26c0a2bc0 0x7fb26c0a5080 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb2740cc810 tx=0x7fb274000bc0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.766+0000 7fb288c9e640 1 -- 172.21.15.27:0/1201784912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb284100680 msgr2=0x7fb284069440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.766+0000 7fb288c9e640 1 --2- 172.21.15.27:0/1201784912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb284100680 0x7fb284069440 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb27400d770 tx=0x7fb27400dc30 comp rx=0 tx=0).stop 2024-04-25T00:15:35.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.766+0000 7fb288c9e640 1 -- 172.21.15.27:0/1201784912 shutdown_connections 2024-04-25T00:15:35.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.766+0000 7fb288c9e640 1 --2- 172.21.15.27:0/1201784912 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb26c0a2bc0 0x7fb26c0a5080 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.766+0000 7fb288c9e640 1 --2- 172.21.15.27:0/1201784912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb28419f370 0x7fb2841a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.766+0000 7fb288c9e640 1 --2- 172.21.15.27:0/1201784912 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb284102230 0x7fb28419ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.766+0000 7fb288c9e640 1 --2- 172.21.15.27:0/1201784912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb284100680 0x7fb284069440 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.766+0000 7fb288c9e640 1 -- 172.21.15.27:0/1201784912 >> 172.21.15.27:0/1201784912 conn(0x7fb2840fbbf0 msgr2=0x7fb284104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.766+0000 7fb288c9e640 1 -- 172.21.15.27:0/1201784912 shutdown_connections 2024-04-25T00:15:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.766+0000 7fb288c9e640 1 -- 172.21.15.27:0/1201784912 wait complete. 2024-04-25T00:15:35.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:35 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3747698751' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:35.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:35 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2638895567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:35.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:35 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1594519559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:35.778 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2024-04-25T00:15:35.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.881+0000 7f82f5322640 1 -- 172.21.15.27:0/3319673084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f0100680 msgr2=0x7f82f0100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.881+0000 7f82f5322640 1 --2- 172.21.15.27:0/3319673084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f0100680 0x7f82f0100aa0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009900 tx=0x7f82e002a410 comp rx=0 tx=0).stop 2024-04-25T00:15:35.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.881+0000 7f82f5322640 1 -- 172.21.15.27:0/3319673084 shutdown_connections 2024-04-25T00:15:35.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.881+0000 7f82f5322640 1 --2- 172.21.15.27:0/3319673084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82f0102230 0x7f82f0108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.881+0000 7f82f5322640 1 --2- 172.21.15.27:0/3319673084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82f0101870 0x7f82f0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.881+0000 7f82f5322640 1 --2- 172.21.15.27:0/3319673084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f0100680 0x7f82f0100aa0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.881+0000 7f82f5322640 1 -- 172.21.15.27:0/3319673084 >> 172.21.15.27:0/3319673084 conn(0x7f82f00fbbf0 msgr2=0x7f82f00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.881+0000 7f82f5322640 1 -- 172.21.15.27:0/3319673084 shutdown_connections 2024-04-25T00:15:35.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.881+0000 7f82f5322640 1 -- 172.21.15.27:0/3319673084 wait complete. 2024-04-25T00:15:35.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82f5322640 1 Processor -- start 2024-04-25T00:15:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82f5322640 1 -- start start 2024-04-25T00:15:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82f5322640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f0100680 0x7f82f01047b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82f5322640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82f0101870 0x7f82f0102df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82f5322640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82f0102230 0x7f82f0103330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82f5322640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f82f010c2d0 con 0x7f82f0100680 2024-04-25T00:15:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82f5322640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f82f010c150 con 0x7f82f0102230 2024-04-25T00:15:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82f5322640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f82f010c450 con 0x7f82f0101870 2024-04-25T00:15:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82eeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f0100680 0x7f82f01047b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82eeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f0100680 0x7f82f01047b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56858/0 (socket says 172.21.15.27:56858) 2024-04-25T00:15:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82eeffd640 1 -- 172.21.15.27:0/1373369688 learned_addr learned my addr 172.21.15.27:0/1373369688 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82e7fff640 1 --2- 172.21.15.27:0/1373369688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82f0101870 0x7f82f0102df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82ef7fe640 1 --2- 172.21.15.27:0/1373369688 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82f0102230 0x7f82f0103330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82eeffd640 1 -- 172.21.15.27:0/1373369688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82f0101870 msgr2=0x7f82f0102df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82eeffd640 1 --2- 172.21.15.27:0/1373369688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82f0101870 0x7f82f0102df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82eeffd640 1 -- 172.21.15.27:0/1373369688 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82f0102230 msgr2=0x7f82f0103330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82eeffd640 1 --2- 172.21.15.27:0/1373369688 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82f0102230 0x7f82f0103330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.882+0000 7f82eeffd640 1 -- 172.21.15.27:0/1373369688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f0103bc0 con 0x7f82f0100680 2024-04-25T00:15:35.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.883+0000 7f82eeffd640 1 --2- 172.21.15.27:0/1373369688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f0100680 0x7f82f01047b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f82e0031610 tx=0x7f82e00043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.883+0000 7f82ecff9640 1 -- 172.21.15.27:0/1373369688 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e003f030 con 0x7f82f0100680 2024-04-25T00:15:35.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.883+0000 7f82ecff9640 1 -- 172.21.15.27:0/1373369688 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82e0003ce0 con 0x7f82f0100680 2024-04-25T00:15:35.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.883+0000 7f82f5322640 1 -- 172.21.15.27:0/1373369688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82f01a1b90 con 0x7f82f0100680 2024-04-25T00:15:35.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.883+0000 7f82ecff9640 1 -- 172.21.15.27:0/1373369688 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e0035070 con 0x7f82f0100680 2024-04-25T00:15:35.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.883+0000 7f82f5322640 1 -- 172.21.15.27:0/1373369688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82f01a20e0 con 0x7f82f0100680 2024-04-25T00:15:35.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.885+0000 7f82ecff9640 1 -- 172.21.15.27:0/1373369688 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f82e0042030 con 0x7f82f0100680 2024-04-25T00:15:35.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.885+0000 7f82f8034740 1 Processor -- start 2024-04-25T00:15:35.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.885+0000 7f82f8034740 1 -- start start 2024-04-25T00:15:35.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82f8034740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23052440 0x55dc23052860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82f8034740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc23057930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82f8034740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23057fa0 0x55dc2305a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82f8034740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55dc22fb5f00 con 0x55dc23052e30 2024-04-25T00:15:35.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82f8034740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55dc22f5e750 con 0x55dc23052440 2024-04-25T00:15:35.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82f8034740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dc22fb1ec0 con 0x55dc23057fa0 2024-04-25T00:15:35.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82eeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc23057930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82eeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc23057930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56872/0 (socket says 172.21.15.27:56872) 2024-04-25T00:15:35.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82eeffd640 1 -- 172.21.15.27:0/3262335257 learned_addr learned my addr 172.21.15.27:0/3262335257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:35.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82ef7fe640 1 --2- 172.21.15.27:0/3262335257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23052440 0x55dc23052860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82e7fff640 1 --2- 172.21.15.27:0/3262335257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23057fa0 0x55dc2305a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82ecff9640 1 --2- 172.21.15.27:0/1373369688 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f82dc0a2b90 0x7f82dc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82e7fff640 1 -- 172.21.15.27:0/3262335257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23052440 msgr2=0x55dc23052860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:35.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82e7fff640 1 --2- 172.21.15.27:0/3262335257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23052440 0x55dc23052860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82ef7fe640 1 --2- 172.21.15.27:0/1373369688 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f82dc0a2b90 0x7f82dc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82e7fff640 1 -- 172.21.15.27:0/3262335257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 msgr2=0x55dc23057930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:35.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82e7fff640 1 --2- 172.21.15.27:0/3262335257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc23057930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82e7fff640 1 -- 172.21.15.27:0/3262335257 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc22cca2d0 con 0x55dc23057fa0 2024-04-25T00:15:35.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.886+0000 7f82eeffd640 1 --2- 172.21.15.27:0/3262335257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc23057930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:35.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.887+0000 7f82ecff9640 1 -- 172.21.15.27:0/1373369688 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f82e00ef750 con 0x7f82f0100680 2024-04-25T00:15:35.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.887+0000 7f82e7fff640 1 --2- 172.21.15.27:0/3262335257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23057fa0 0x55dc2305a430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f82d800bbb0 tx=0x7f82d800f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=784a3bc0420a8ac7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.887+0000 7f82ef7fe640 1 --2- 172.21.15.27:0/1373369688 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f82dc0a2b90 0x7f82dc0a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55dc23053690 tx=0x7f82e8004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.887+0000 7f82e57fa640 1 -- 172.21.15.27:0/3262335257 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d8015070 con 0x55dc23057fa0 2024-04-25T00:15:35.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.887+0000 7f82e57fa640 1 -- 172.21.15.27:0/3262335257 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82d80035f0 con 0x55dc23057fa0 2024-04-25T00:15:35.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.887+0000 7f82e57fa640 1 -- 172.21.15.27:0/3262335257 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d80116a0 con 0x55dc23057fa0 2024-04-25T00:15:35.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.888+0000 7f82f8034740 1 -- 172.21.15.27:0/3262335257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23057fa0 msgr2=0x55dc2305a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.888+0000 7f82f8034740 1 --2- 172.21.15.27:0/3262335257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23057fa0 0x55dc2305a430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f82d800bbb0 tx=0x7f82d800f730 comp rx=0 tx=0).stop 2024-04-25T00:15:35.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.888+0000 7f82f8034740 1 -- 172.21.15.27:0/3262335257 shutdown_connections 2024-04-25T00:15:35.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.888+0000 7f82f8034740 1 --2- 172.21.15.27:0/3262335257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23057fa0 0x55dc2305a430 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.888+0000 7f82f8034740 1 --2- 172.21.15.27:0/3262335257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc23057930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.888+0000 7f82f8034740 1 --2- 172.21.15.27:0/3262335257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23052440 0x55dc23052860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.888+0000 7f82f8034740 1 -- 172.21.15.27:0/3262335257 >> 172.21.15.27:0/3262335257 conn(0x55dc2304dab0 msgr2=0x55dc2304deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.888+0000 7f82f8034740 1 -- 172.21.15.27:0/3262335257 shutdown_connections 2024-04-25T00:15:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.888+0000 7f82f8034740 1 -- 172.21.15.27:0/3262335257 wait complete. 2024-04-25T00:15:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.888+0000 7f82f8034740 1 -- start start 2024-04-25T00:15:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.888+0000 7f82f8034740 1 Processor -- start 2024-04-25T00:15:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82f8034740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23052440 0x55dc2305c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82f8034740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc2305cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82f8034740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23057fa0 0x55dc23063dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82f8034740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55dc2305af50 con 0x55dc23052e30 2024-04-25T00:15:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82f8034740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55dc22f71cf0 con 0x55dc23052440 2024-04-25T00:15:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82f8034740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dc2305b0d0 con 0x55dc23057fa0 2024-04-25T00:15:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82f8034740 10 client.0 _ll_register_callbacks cb 0x55dc22f716e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82e7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc2305cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82e7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc2305cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56884/0 (socket says 172.21.15.27:56884) 2024-04-25T00:15:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82e7fff640 1 -- 172.21.15.27:0/1469875842 learned_addr learned my addr 172.21.15.27:0/1469875842 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82ef7fe640 1 --2- 172.21.15.27:0/1469875842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23057fa0 0x55dc23063dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82eeffd640 1 --2- 172.21.15.27:0/1469875842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23052440 0x55dc2305c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82e7fff640 1 -- 172.21.15.27:0/1469875842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23057fa0 msgr2=0x55dc23063dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82e7fff640 1 --2- 172.21.15.27:0/1469875842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23057fa0 0x55dc23063dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82e7fff640 1 -- 172.21.15.27:0/1469875842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23052440 msgr2=0x55dc2305c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82e7fff640 1 --2- 172.21.15.27:0/1469875842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23052440 0x55dc2305c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82e7fff640 1 -- 172.21.15.27:0/1469875842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc22d946b0 con 0x55dc23052e30 2024-04-25T00:15:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82eeffd640 1 --2- 172.21.15.27:0/1469875842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23052440 0x55dc2305c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82ef7fe640 1 --2- 172.21.15.27:0/1469875842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23057fa0 0x55dc23063dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82e7fff640 1 --2- 172.21.15.27:0/1469875842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc2305cd80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f82d8000c00 tx=0x7f82d800bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85cb6c7bc552e2fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82d67fc640 1 -- 172.21.15.27:0/1469875842 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d8015030 con 0x55dc23052e30 2024-04-25T00:15:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82f8034740 1 -- 172.21.15.27:0/1469875842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82d80035f0 con 0x55dc23052e30 2024-04-25T00:15:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.889+0000 7f82d67fc640 1 -- 172.21.15.27:0/1469875842 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f82d8003a70 con 0x55dc23052e30 2024-04-25T00:15:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.890+0000 7f82d67fc640 1 -- 172.21.15.27:0/1469875842 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f82d8004000 con 0x55dc23052e30 2024-04-25T00:15:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.890+0000 7f82f8034740 10 client.15480 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.890+0000 7f82f8034740 1 -- 172.21.15.27:0/1469875842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dc23066df0 con 0x55dc23052e30 2024-04-25T00:15:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.890+0000 7f82d67fc640 1 -- 172.21.15.27:0/1469875842 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82d80035f0 con 0x55dc23052e30 2024-04-25T00:15:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.890+0000 7f82d67fc640 1 -- 172.21.15.27:0/1469875842 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f82d8004890 con 0x55dc23052e30 2024-04-25T00:15:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.890+0000 7f82f8034740 10 client.15480 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.890+0000 7f82f8034740 10 client.15480 resolve_mds: resolved name 'l' to daemon mds.l 2024-04-25T00:15:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.890+0000 7f82f8034740 20 client.15480 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.890+0000 7f82f8034740 1 --2- 172.21.15.27:0/1469875842 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55dc23069200 0x55dc2306a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.890+0000 7f82f8034740 4 client.15480 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:15:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.890+0000 7f82f8034740 1 -- 172.21.15.27:0/1469875842 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc22fc3440 con 0x55dc23069200 2024-04-25T00:15:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.890+0000 7f82eeffd640 1 --2- 172.21.15.27:0/1469875842 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55dc23069200 0x55dc2306a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.891+0000 7f82eeffd640 1 --2- 172.21.15.27:0/1469875842 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55dc23069200 0x55dc2306a670 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fa7655abf40f9ac4 server_cookie=593d3ceca1e245a5 in_seq=0 out_seq=0 2024-04-25T00:15:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.891+0000 7f82d67fc640 10 client.15480 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:15:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.892+0000 7f82d67fc640 1 -- 172.21.15.27:0/1469875842 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dc22fc3440 con 0x55dc23069200 2024-04-25T00:15:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.892+0000 7f82d67fc640 10 client.15480 handle_command_reply: tid=0 2024-04-25T00:15:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.892+0000 7f82f8034740 1 client.15480 shutdown 2024-04-25T00:15:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.892+0000 7f82f8034740 1 -- 172.21.15.27:0/1469875842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 msgr2=0x55dc2305cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.892+0000 7f82f8034740 1 --2- 172.21.15.27:0/1469875842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc2305cd80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f82d8000c00 tx=0x7f82d800bed0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.892+0000 7f82f8034740 1 -- 172.21.15.27:0/1469875842 shutdown_connections 2024-04-25T00:15:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.892+0000 7f82f8034740 1 --2- 172.21.15.27:0/1469875842 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55dc23069200 0x55dc2306a670 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.892+0000 7f82f8034740 1 --2- 172.21.15.27:0/1469875842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23057fa0 0x55dc23063dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.892+0000 7f82f8034740 1 --2- 172.21.15.27:0/1469875842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc2305cd80 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.892+0000 7f82d67fc640 0 client.15480 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:15:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.892+0000 7f82f8034740 1 --2- 172.21.15.27:0/1469875842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23052440 0x55dc2305c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.892+0000 7f82f8034740 1 -- 172.21.15.27:0/1469875842 >> 172.21.15.27:0/1469875842 conn(0x55dc2304dab0 msgr2=0x55dc2304cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.892+0000 7f82f8034740 1 -- 172.21.15.27:0/1469875842 shutdown_connections 2024-04-25T00:15:35.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.892+0000 7f82f8034740 1 -- 172.21.15.27:0/1469875842 wait complete. 2024-04-25T00:15:35.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.892+0000 7f82f8034740 20 client.15480 trim_cache size 0 max 16384 2024-04-25T00:15:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.911+0000 7f82f8034740 1 Processor -- start 2024-04-25T00:15:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.911+0000 7f82f8034740 1 -- start start 2024-04-25T00:15:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.911+0000 7f82f8034740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052440 0x55dc2304de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.911+0000 7f82f8034740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23052e30 0x55dc23063d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.911+0000 7f82f8034740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23057fa0 0x55dc230681a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.911+0000 7f82f8034740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55dc230689b0 con 0x55dc23052440 2024-04-25T00:15:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.911+0000 7f82f8034740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55dc23069080 con 0x55dc23052e30 2024-04-25T00:15:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.911+0000 7f82f8034740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dc23069750 con 0x55dc23057fa0 2024-04-25T00:15:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.911+0000 7f82eeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052440 0x55dc2304de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.911+0000 7f82eeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052440 0x55dc2304de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56900/0 (socket says 172.21.15.27:56900) 2024-04-25T00:15:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.911+0000 7f82eeffd640 1 -- 172.21.15.27:0/3910109294 learned_addr learned my addr 172.21.15.27:0/3910109294 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.911+0000 7f82ef7fe640 1 --2- 172.21.15.27:0/3910109294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23057fa0 0x55dc230681a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.911+0000 7f82e7fff640 1 --2- 172.21.15.27:0/3910109294 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23052e30 0x55dc23063d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.912+0000 7f82eeffd640 1 -- 172.21.15.27:0/3910109294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23057fa0 msgr2=0x55dc230681a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.912+0000 7f82eeffd640 1 --2- 172.21.15.27:0/3910109294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23057fa0 0x55dc230681a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.912+0000 7f82eeffd640 1 -- 172.21.15.27:0/3910109294 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23052e30 msgr2=0x55dc23063d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.912+0000 7f82eeffd640 1 --2- 172.21.15.27:0/3910109294 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23052e30 0x55dc23063d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.912+0000 7f82eeffd640 1 -- 172.21.15.27:0/3910109294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc230536c0 con 0x55dc23052440 2024-04-25T00:15:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.912+0000 7f82eeffd640 1 --2- 172.21.15.27:0/3910109294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052440 0x55dc2304de90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f82e0031440 tx=0x7f82e003b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3b70715db9fb8a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.912+0000 7f82e4ff9640 1 -- 172.21.15.27:0/3910109294 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e0049040 con 0x55dc23052440 2024-04-25T00:15:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.912+0000 7f82e4ff9640 1 -- 172.21.15.27:0/3910109294 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82e00d2070 con 0x55dc23052440 2024-04-25T00:15:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.912+0000 7f82e4ff9640 1 -- 172.21.15.27:0/3910109294 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e00b9f50 con 0x55dc23052440 2024-04-25T00:15:35.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.913+0000 7f82f8034740 1 -- 172.21.15.27:0/3910109294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052440 msgr2=0x55dc2304de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.913+0000 7f82f8034740 1 --2- 172.21.15.27:0/3910109294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052440 0x55dc2304de90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f82e0031440 tx=0x7f82e003b870 comp rx=0 tx=0).stop 2024-04-25T00:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.913+0000 7f82f8034740 1 -- 172.21.15.27:0/3910109294 shutdown_connections 2024-04-25T00:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.913+0000 7f82f8034740 1 --2- 172.21.15.27:0/3910109294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23057fa0 0x55dc230681a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.913+0000 7f82f8034740 1 --2- 172.21.15.27:0/3910109294 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23052e30 0x55dc23063d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.913+0000 7f82f8034740 1 --2- 172.21.15.27:0/3910109294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052440 0x55dc2304de90 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.913+0000 7f82f8034740 1 -- 172.21.15.27:0/3910109294 >> 172.21.15.27:0/3910109294 conn(0x55dc2304dab0 msgr2=0x55dc23057860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.913+0000 7f82f8034740 1 -- 172.21.15.27:0/3910109294 shutdown_connections 2024-04-25T00:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.913+0000 7f82f8034740 1 -- 172.21.15.27:0/3910109294 wait complete. 2024-04-25T00:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.913+0000 7f82f8034740 1 -- start start 2024-04-25T00:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.913+0000 7f82f8034740 1 Processor -- start 2024-04-25T00:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.913+0000 7f82f8034740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23052440 0x55dc23056c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.913+0000 7f82f8034740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc23063d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.913+0000 7f82f8034740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23057fa0 0x55dc230cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.913+0000 7f82f8034740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55dc2306a4f0 con 0x55dc23052e30 2024-04-25T00:15:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82f8034740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55dc23069e20 con 0x55dc23057fa0 2024-04-25T00:15:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82f8034740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dc2306abc0 con 0x55dc23052440 2024-04-25T00:15:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82f8034740 10 client.0 _ll_register_callbacks cb 0x7f82d800ba30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82e7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc23063d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82e7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc23063d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56904/0 (socket says 172.21.15.27:56904) 2024-04-25T00:15:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82e7fff640 1 -- 172.21.15.27:0/1811187787 learned_addr learned my addr 172.21.15.27:0/1811187787 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82eeffd640 1 --2- 172.21.15.27:0/1811187787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23052440 0x55dc23056c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82ef7fe640 1 --2- 172.21.15.27:0/1811187787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23057fa0 0x55dc230cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82e7fff640 1 -- 172.21.15.27:0/1811187787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23052440 msgr2=0x55dc23056c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82e7fff640 1 --2- 172.21.15.27:0/1811187787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23052440 0x55dc23056c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82e7fff640 1 -- 172.21.15.27:0/1811187787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23057fa0 msgr2=0x55dc230cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82e7fff640 1 --2- 172.21.15.27:0/1811187787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23057fa0 0x55dc230cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82e7fff640 1 -- 172.21.15.27:0/1811187787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc230cd9f0 con 0x55dc23052e30 2024-04-25T00:15:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82e7fff640 1 --2- 172.21.15.27:0/1811187787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc23063d80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f82d8003df0 tx=0x7f82d8005f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8310276b2711a0f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82d67fc640 1 -- 172.21.15.27:0/1811187787 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d8015070 con 0x55dc23052e30 2024-04-25T00:15:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82f8034740 1 -- 172.21.15.27:0/1811187787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82e00d2070 con 0x55dc23052e30 2024-04-25T00:15:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82d67fc640 1 -- 172.21.15.27:0/1811187787 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f82d800b6f0 con 0x55dc23052e30 2024-04-25T00:15:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82d67fc640 1 -- 172.21.15.27:0/1811187787 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f82d80035f0 con 0x55dc23052e30 2024-04-25T00:15:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.914+0000 7f82f8034740 10 client.15486 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.915+0000 7f82f8034740 1 -- 172.21.15.27:0/1811187787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dc230cecf0 con 0x55dc23052e30 2024-04-25T00:15:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.915+0000 7f82d67fc640 1 -- 172.21.15.27:0/1811187787 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82e00d2070 con 0x55dc23052e30 2024-04-25T00:15:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.915+0000 7f82d67fc640 1 -- 172.21.15.27:0/1811187787 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f82d8011440 con 0x55dc23052e30 2024-04-25T00:15:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.915+0000 7f82f8034740 10 client.15486 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.915+0000 7f82f8034740 10 client.15486 resolve_mds: resolved name 'l' to daemon mds.l 2024-04-25T00:15:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.915+0000 7f82f8034740 20 client.15486 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.915+0000 7f82f8034740 1 --2- 172.21.15.27:0/1811187787 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55dc230d3730 0x55dc230d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.915+0000 7f82f8034740 4 client.15486 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T00:15:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.915+0000 7f82f8034740 1 -- 172.21.15.27:0/1811187787 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "status"}) -- 0x55dc22fbd790 con 0x55dc230d3730 2024-04-25T00:15:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.915+0000 7f82eeffd640 1 --2- 172.21.15.27:0/1811187787 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55dc230d3730 0x55dc230d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.916+0000 7f82d67fc640 10 client.15486 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:15:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.916+0000 7f82eeffd640 1 --2- 172.21.15.27:0/1811187787 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55dc230d3730 0x55dc230d2440 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7b89e541c62ed20b server_cookie=7c203655dd1f7e6d in_seq=0 out_seq=0 2024-04-25T00:15:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.916+0000 7f82d67fc640 1 -- 172.21.15.27:0/1811187787 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55dc22fbd790 con 0x55dc230d3730 2024-04-25T00:15:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.916+0000 7f82d67fc640 10 client.15486 handle_command_reply: tid=0 2024-04-25T00:15:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.916+0000 7f82f8034740 1 client.15486 shutdown 2024-04-25T00:15:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.916+0000 7f82f8034740 1 -- 172.21.15.27:0/1811187787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 msgr2=0x55dc23063d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.916+0000 7f82f8034740 1 --2- 172.21.15.27:0/1811187787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc23063d80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f82d8003df0 tx=0x7f82d8005f40 comp rx=0 tx=0).stop 2024-04-25T00:15:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.916+0000 7f82f8034740 1 -- 172.21.15.27:0/1811187787 shutdown_connections 2024-04-25T00:15:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.916+0000 7f82f8034740 1 --2- 172.21.15.27:0/1811187787 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55dc230d3730 0x55dc230d2440 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.916+0000 7f82d67fc640 0 client.15486 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:15:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.916+0000 7f82f8034740 1 --2- 172.21.15.27:0/1811187787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dc23057fa0 0x55dc230cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.916+0000 7f82f8034740 1 --2- 172.21.15.27:0/1811187787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dc23052e30 0x55dc23063d80 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.916+0000 7f82f8034740 1 --2- 172.21.15.27:0/1811187787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc23052440 0x55dc23056c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.916+0000 7f82f8034740 1 -- 172.21.15.27:0/1811187787 >> 172.21.15.27:0/1811187787 conn(0x55dc2304dab0 msgr2=0x55dc230681a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.916+0000 7f82f8034740 1 -- 172.21.15.27:0/1811187787 shutdown_connections 2024-04-25T00:15:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.916+0000 7f82f8034740 1 -- 172.21.15.27:0/1811187787 wait complete. 2024-04-25T00:15:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.916+0000 7f82f8034740 20 client.15486 trim_cache size 0 max 16384 2024-04-25T00:15:35.924 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:15:35.924 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "518857f8-0297-11ef-bc93-c7b262605968", 2024-04-25T00:15:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 3, 2024-04-25T00:15:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24430, 2024-04-25T00:15:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T00:15:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T00:15:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T00:15:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 110.354781891, 2024-04-25T00:15:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 28, 2024-04-25T00:15:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 93, 2024-04-25T00:15:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T00:15:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 174.36979974900001 2024-04-25T00:15:35.925 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:15:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.917+0000 7f82f5322640 1 -- 172.21.15.27:0/1373369688 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f82dc0a2b90 msgr2=0x7f82dc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.917+0000 7f82f5322640 1 --2- 172.21.15.27:0/1373369688 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f82dc0a2b90 0x7f82dc0a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55dc23053690 tx=0x7f82e8004ab0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.917+0000 7f82f5322640 1 -- 172.21.15.27:0/1373369688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f0100680 msgr2=0x7f82f01047b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.917+0000 7f82f5322640 1 --2- 172.21.15.27:0/1373369688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f0100680 0x7f82f01047b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f82e0031610 tx=0x7f82e00043b0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.917+0000 7f82f5322640 1 -- 172.21.15.27:0/1373369688 shutdown_connections 2024-04-25T00:15:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.917+0000 7f82f5322640 1 --2- 172.21.15.27:0/1373369688 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f82dc0a2b90 0x7f82dc0a5050 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.917+0000 7f82f5322640 1 --2- 172.21.15.27:0/1373369688 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82f0102230 0x7f82f0103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.917+0000 7f82f5322640 1 --2- 172.21.15.27:0/1373369688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82f0101870 0x7f82f0102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.917+0000 7f82f5322640 1 --2- 172.21.15.27:0/1373369688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f0100680 0x7f82f01047b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.917+0000 7f82f5322640 1 -- 172.21.15.27:0/1373369688 >> 172.21.15.27:0/1373369688 conn(0x7f82f00fbbf0 msgr2=0x7f82f00688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.917+0000 7f82f5322640 1 -- 172.21.15.27:0/1373369688 shutdown_connections 2024-04-25T00:15:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:35.917+0000 7f82f5322640 1 -- 172.21.15.27:0/1373369688 wait complete. 2024-04-25T00:15:35.929 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T00:15:36.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:35 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3747698751' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:36.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:35 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2638895567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:36.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:35 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1594519559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:36.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:35 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3747698751' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:36.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:35 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2638895567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:36.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:35 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1594519559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.031+0000 7fb53159e640 1 -- 172.21.15.27:0/2358831441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c101740 msgr2=0x7fb52c101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.031+0000 7fb53159e640 1 --2- 172.21.15.27:0/2358831441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c101740 0x7fb52c101bc0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb520009900 tx=0x7fb52002a410 comp rx=0 tx=0).stop 2024-04-25T00:15:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.031+0000 7fb53159e640 1 -- 172.21.15.27:0/2358831441 shutdown_connections 2024-04-25T00:15:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.031+0000 7fb53159e640 1 --2- 172.21.15.27:0/2358831441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c102100 0x7fb52c108990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.031+0000 7fb53159e640 1 --2- 172.21.15.27:0/2358831441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c101740 0x7fb52c101bc0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.031+0000 7fb53159e640 1 --2- 172.21.15.27:0/2358831441 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb52c100550 0x7fb52c100970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.031+0000 7fb53159e640 1 -- 172.21.15.27:0/2358831441 >> 172.21.15.27:0/2358831441 conn(0x7fb52c0fbae0 msgr2=0x7fb52c0fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.031+0000 7fb53159e640 1 -- 172.21.15.27:0/2358831441 shutdown_connections 2024-04-25T00:15:36.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.031+0000 7fb53159e640 1 -- 172.21.15.27:0/2358831441 wait complete. 2024-04-25T00:15:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.032+0000 7fb53159e640 1 Processor -- start 2024-04-25T00:15:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.032+0000 7fb53159e640 1 -- start start 2024-04-25T00:15:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.032+0000 7fb53159e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb52c100550 0x7fb52c197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.032+0000 7fb53159e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c101740 0x7fb52c197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.032+0000 7fb53159e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c102100 0x7fb52c19eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.032+0000 7fb53159e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c10c1e0 con 0x7fb52c102100 2024-04-25T00:15:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.032+0000 7fb53159e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c10c060 con 0x7fb52c100550 2024-04-25T00:15:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.032+0000 7fb53159e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c10c360 con 0x7fb52c101740 2024-04-25T00:15:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.032+0000 7fb530d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c102100 0x7fb52c19eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.032+0000 7fb530d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c102100 0x7fb52c19eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56918/0 (socket says 172.21.15.27:56918) 2024-04-25T00:15:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.032+0000 7fb530d9d640 1 -- 172.21.15.27:0/3107296271 learned_addr learned my addr 172.21.15.27:0/3107296271 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.032+0000 7fb52bfff640 1 --2- 172.21.15.27:0/3107296271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb52c100550 0x7fb52c197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.032+0000 7fb52b7fe640 1 --2- 172.21.15.27:0/3107296271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c101740 0x7fb52c197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.033+0000 7fb530d9d640 1 -- 172.21.15.27:0/3107296271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c101740 msgr2=0x7fb52c197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.033+0000 7fb530d9d640 1 --2- 172.21.15.27:0/3107296271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c101740 0x7fb52c197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.033+0000 7fb530d9d640 1 -- 172.21.15.27:0/3107296271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb52c100550 msgr2=0x7fb52c197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.033+0000 7fb530d9d640 1 --2- 172.21.15.27:0/3107296271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb52c100550 0x7fb52c197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.033+0000 7fb530d9d640 1 -- 172.21.15.27:0/3107296271 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb52c19f550 con 0x7fb52c102100 2024-04-25T00:15:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.033+0000 7fb530d9d640 1 --2- 172.21.15.27:0/3107296271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c102100 0x7fb52c19eee0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb52400eaa0 tx=0x7fb52400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.033+0000 7fb5297fa640 1 -- 172.21.15.27:0/3107296271 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52400cdb0 con 0x7fb52c102100 2024-04-25T00:15:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.033+0000 7fb53159e640 1 -- 172.21.15.27:0/3107296271 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb52c19f840 con 0x7fb52c102100 2024-04-25T00:15:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.033+0000 7fb5297fa640 1 -- 172.21.15.27:0/3107296271 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb524005020 con 0x7fb52c102100 2024-04-25T00:15:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.033+0000 7fb5297fa640 1 -- 172.21.15.27:0/3107296271 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5240116b0 con 0x7fb52c102100 2024-04-25T00:15:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.033+0000 7fb53159e640 1 -- 172.21.15.27:0/3107296271 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb52c19fd60 con 0x7fb52c102100 2024-04-25T00:15:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.035+0000 7fb5297fa640 1 -- 172.21.15.27:0/3107296271 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb524005560 con 0x7fb52c102100 2024-04-25T00:15:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.035+0000 7fb5355ee740 1 Processor -- start 2024-04-25T00:15:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.035+0000 7fb5355ee740 1 -- start start 2024-04-25T00:15:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb5355ee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d33e640 0x55aa4d33ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb5355ee740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d33f030 0x55aa4d343b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb5355ee740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d3441a0 0x55aa4d346630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb5355ee740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d2a1f00 con 0x55aa4d3441a0 2024-04-25T00:15:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb5355ee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d24a750 con 0x55aa4d33e640 2024-04-25T00:15:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb5355ee740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d29dec0 con 0x55aa4d33f030 2024-04-25T00:15:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb52b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d3441a0 0x55aa4d346630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb52b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d3441a0 0x55aa4d346630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56932/0 (socket says 172.21.15.27:56932) 2024-04-25T00:15:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb52b7fe640 1 -- 172.21.15.27:0/18424042 learned_addr learned my addr 172.21.15.27:0/18424042 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb52b7fe640 1 --2- 172.21.15.27:0/18424042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d33e640 0x55aa4d33ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb5297fa640 1 --2- 172.21.15.27:0/3107296271 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb5140a2b90 0x7fb5140a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb52bfff640 1 --2- 172.21.15.27:0/18424042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d33f030 0x55aa4d343b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb52b7fe640 1 -- 172.21.15.27:0/18424042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d33f030 msgr2=0x55aa4d343b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb52b7fe640 1 --2- 172.21.15.27:0/18424042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d33f030 0x55aa4d343b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb52b7fe640 1 -- 172.21.15.27:0/18424042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d33e640 msgr2=0x55aa4d33ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb52b7fe640 1 --2- 172.21.15.27:0/18424042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d33e640 0x55aa4d33ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb52b7fe640 1 -- 172.21.15.27:0/18424042 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa4cfb62d0 con 0x55aa4d3441a0 2024-04-25T00:15:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb530d9d640 1 --2- 172.21.15.27:0/3107296271 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb5140a2b90 0x7fb5140a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb5297fa640 1 -- 172.21.15.27:0/3107296271 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb5240cf5a0 con 0x7fb52c102100 2024-04-25T00:15:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.036+0000 7fb52b7fe640 1 --2- 172.21.15.27:0/18424042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d3441a0 0x55aa4d346630 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb5200096c0 tx=0x7fb520005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f62fc2e19a8b8858 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.037+0000 7fb5127fc640 1 -- 172.21.15.27:0/18424042 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520035070 con 0x55aa4d3441a0 2024-04-25T00:15:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.037+0000 7fb5127fc640 1 -- 172.21.15.27:0/18424042 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb520005510 con 0x55aa4d3441a0 2024-04-25T00:15:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.037+0000 7fb5127fc640 1 -- 172.21.15.27:0/18424042 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520004ba0 con 0x55aa4d3441a0 2024-04-25T00:15:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.037+0000 7fb5355ee740 1 -- 172.21.15.27:0/18424042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d3441a0 msgr2=0x55aa4d346630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.037+0000 7fb530d9d640 1 --2- 172.21.15.27:0/3107296271 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb5140a2b90 0x7fb5140a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb5240984f0 tx=0x7fb52406b9d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.037+0000 7fb5355ee740 1 --2- 172.21.15.27:0/18424042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d3441a0 0x55aa4d346630 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb5200096c0 tx=0x7fb520005fd0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.037+0000 7fb5355ee740 1 -- 172.21.15.27:0/18424042 shutdown_connections 2024-04-25T00:15:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.037+0000 7fb5355ee740 1 --2- 172.21.15.27:0/18424042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d3441a0 0x55aa4d346630 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.037+0000 7fb5355ee740 1 --2- 172.21.15.27:0/18424042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d33f030 0x55aa4d343b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.037+0000 7fb5355ee740 1 --2- 172.21.15.27:0/18424042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d33e640 0x55aa4d33ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.037+0000 7fb5355ee740 1 -- 172.21.15.27:0/18424042 >> 172.21.15.27:0/18424042 conn(0x55aa4d339cb0 msgr2=0x55aa4d33a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.037+0000 7fb5355ee740 1 -- 172.21.15.27:0/18424042 shutdown_connections 2024-04-25T00:15:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.038+0000 7fb5355ee740 1 -- 172.21.15.27:0/18424042 wait complete. 2024-04-25T00:15:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.038+0000 7fb5355ee740 1 -- start start 2024-04-25T00:15:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.038+0000 7fb5355ee740 1 Processor -- start 2024-04-25T00:15:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.038+0000 7fb5355ee740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d33e640 0x55aa4d348a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.038+0000 7fb5355ee740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d33f030 0x55aa4d348fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.038+0000 7fb5355ee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d3441a0 0x55aa4d34ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.038+0000 7fb5355ee740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d3470f0 con 0x55aa4d33e640 2024-04-25T00:15:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.038+0000 7fb5355ee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d25dcf0 con 0x55aa4d3441a0 2024-04-25T00:15:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.038+0000 7fb5355ee740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d347270 con 0x55aa4d33f030 2024-04-25T00:15:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.038+0000 7fb5355ee740 10 client.0 _ll_register_callbacks cb 0x55aa4d25d6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.038+0000 7fb52b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d33e640 0x55aa4d348a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.038+0000 7fb52b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d33e640 0x55aa4d348a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56942/0 (socket says 172.21.15.27:56942) 2024-04-25T00:15:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.038+0000 7fb52b7fe640 1 -- 172.21.15.27:0/3035507250 learned_addr learned my addr 172.21.15.27:0/3035507250 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.038+0000 7fb52bfff640 1 --2- 172.21.15.27:0/3035507250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d33f030 0x55aa4d348fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.039+0000 7fb52b7fe640 1 --2- 172.21.15.27:0/3035507250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d3441a0 0x55aa4d34ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.039+0000 7fb52b7fe640 1 -- 172.21.15.27:0/3035507250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d33f030 msgr2=0x55aa4d348fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.039+0000 7fb52b7fe640 1 --2- 172.21.15.27:0/3035507250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d33f030 0x55aa4d348fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.039+0000 7fb52b7fe640 1 -- 172.21.15.27:0/3035507250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d3441a0 msgr2=0x55aa4d34ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.039+0000 7fb52b7fe640 1 --2- 172.21.15.27:0/3035507250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d3441a0 0x55aa4d34ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.039+0000 7fb52b7fe640 1 -- 172.21.15.27:0/3035507250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa4d0806b0 con 0x55aa4d33e640 2024-04-25T00:15:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.039+0000 7fb52b7fe640 1 --2- 172.21.15.27:0/3035507250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d33e640 0x55aa4d348a60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb520031440 tx=0x7fb5200315b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c9a9c1fc29694c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.039+0000 7fb50b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.039+0000 7fb50b7fe640 1 -- 172.21.15.27:0/3035507250 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520035070 con 0x55aa4d33e640 2024-04-25T00:15:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.039+0000 7fb5355ee740 1 -- 172.21.15.27:0/3035507250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb520005510 con 0x55aa4d33e640 2024-04-25T00:15:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.039+0000 7fb50b7fe640 1 -- 172.21.15.27:0/3035507250 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb5200037f0 con 0x55aa4d33e640 2024-04-25T00:15:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.039+0000 7fb50b7fe640 1 -- 172.21.15.27:0/3035507250 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb520004400 con 0x55aa4d33e640 2024-04-25T00:15:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.039+0000 7fb5355ee740 10 client.15498 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.039+0000 7fb5355ee740 1 -- 172.21.15.27:0/3035507250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55aa4d352fc0 con 0x55aa4d33e640 2024-04-25T00:15:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.039+0000 7fb50b7fe640 1 -- 172.21.15.27:0/3035507250 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb520005510 con 0x55aa4d33e640 2024-04-25T00:15:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.040+0000 7fb50b7fe640 1 -- 172.21.15.27:0/3035507250 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb52003db50 con 0x55aa4d33e640 2024-04-25T00:15:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.040+0000 7fb5355ee740 10 client.15498 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.040+0000 7fb5355ee740 10 client.15498 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T00:15:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.040+0000 7fb5355ee740 20 client.15498 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.040+0000 7fb5355ee740 1 --2- 172.21.15.27:0/3035507250 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55aa4d3550f0 0x55aa4d356560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.040+0000 7fb5355ee740 4 client.15498 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:15:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.040+0000 7fb5355ee740 1 -- 172.21.15.27:0/3035507250 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa4d2af440 con 0x55aa4d3550f0 2024-04-25T00:15:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.040+0000 7fb52bfff640 1 --2- 172.21.15.27:0/3035507250 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55aa4d3550f0 0x55aa4d356560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.041+0000 7fb50b7fe640 10 client.15498 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:15:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.041+0000 7fb52bfff640 1 --2- 172.21.15.27:0/3035507250 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55aa4d3550f0 0x55aa4d356560 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1c2e493bd4dd098 server_cookie=b97815e60ff0e5c in_seq=0 out_seq=0 2024-04-25T00:15:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.041+0000 7fb50b7fe640 1 -- 172.21.15.27:0/3035507250 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aa4d2af440 con 0x55aa4d3550f0 2024-04-25T00:15:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.041+0000 7fb50b7fe640 10 client.15498 handle_command_reply: tid=0 2024-04-25T00:15:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.041+0000 7fb5355ee740 1 client.15498 shutdown 2024-04-25T00:15:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.042+0000 7fb5355ee740 1 -- 172.21.15.27:0/3035507250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d33e640 msgr2=0x55aa4d348a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.042+0000 7fb5355ee740 1 --2- 172.21.15.27:0/3035507250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d33e640 0x55aa4d348a60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb520031440 tx=0x7fb5200315b0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.042+0000 7fb5355ee740 1 -- 172.21.15.27:0/3035507250 shutdown_connections 2024-04-25T00:15:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.042+0000 7fb5355ee740 1 --2- 172.21.15.27:0/3035507250 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55aa4d3550f0 0x55aa4d356560 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.042+0000 7fb5355ee740 1 --2- 172.21.15.27:0/3035507250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d3441a0 0x55aa4d34ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.042+0000 7fb5355ee740 1 --2- 172.21.15.27:0/3035507250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d33f030 0x55aa4d348fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.042+0000 7fb5355ee740 1 --2- 172.21.15.27:0/3035507250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d33e640 0x55aa4d348a60 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.042+0000 7fb5355ee740 1 -- 172.21.15.27:0/3035507250 >> 172.21.15.27:0/3035507250 conn(0x55aa4d339cb0 msgr2=0x55aa4d338e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.042+0000 7fb50b7fe640 0 client.15498 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:15:36.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.042+0000 7fb5355ee740 1 -- 172.21.15.27:0/3035507250 shutdown_connections 2024-04-25T00:15:36.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.042+0000 7fb5355ee740 1 -- 172.21.15.27:0/3035507250 wait complete. 2024-04-25T00:15:36.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.042+0000 7fb5355ee740 20 client.15498 trim_cache size 0 max 16384 2024-04-25T00:15:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.060+0000 7fb5355ee740 1 Processor -- start 2024-04-25T00:15:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.060+0000 7fb5355ee740 1 -- start start 2024-04-25T00:15:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.060+0000 7fb5355ee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d33e640 0x55aa4d346590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.060+0000 7fb5355ee740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d33f030 0x55aa4d34ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.060+0000 7fb5355ee740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d3441a0 0x55aa4d348670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.060+0000 7fb5355ee740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d3548a0 con 0x55aa4d3441a0 2024-04-25T00:15:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.060+0000 7fb5355ee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d354f70 con 0x55aa4d33e640 2024-04-25T00:15:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.060+0000 7fb5355ee740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d355640 con 0x55aa4d33f030 2024-04-25T00:15:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.060+0000 7fb52b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d3441a0 0x55aa4d348670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.060+0000 7fb52b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d3441a0 0x55aa4d348670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56956/0 (socket says 172.21.15.27:56956) 2024-04-25T00:15:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.060+0000 7fb52b7fe640 1 -- 172.21.15.27:0/1420724573 learned_addr learned my addr 172.21.15.27:0/1420724573 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.060+0000 7fb52b7fe640 1 --2- 172.21.15.27:0/1420724573 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d33e640 0x55aa4d346590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.061+0000 7fb52b7fe640 1 -- 172.21.15.27:0/1420724573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d33f030 msgr2=0x55aa4d34ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.061+0000 7fb52b7fe640 1 --2- 172.21.15.27:0/1420724573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d33f030 0x55aa4d34ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.061+0000 7fb52b7fe640 1 -- 172.21.15.27:0/1420724573 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d33e640 msgr2=0x55aa4d346590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.061+0000 7fb52b7fe640 1 --2- 172.21.15.27:0/1420724573 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d33e640 0x55aa4d346590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.061+0000 7fb52b7fe640 1 -- 172.21.15.27:0/1420724573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa4d355af0 con 0x55aa4d3441a0 2024-04-25T00:15:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.061+0000 7fb52b7fe640 1 --2- 172.21.15.27:0/1420724573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d3441a0 0x55aa4d348670 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb520004650 tx=0x7fb52003efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a54d6d7ce76b123c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.061+0000 7fb511ffb640 1 -- 172.21.15.27:0/1420724573 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520065070 con 0x55aa4d3441a0 2024-04-25T00:15:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.061+0000 7fb511ffb640 1 -- 172.21.15.27:0/1420724573 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb52003d450 con 0x55aa4d3441a0 2024-04-25T00:15:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.061+0000 7fb511ffb640 1 -- 172.21.15.27:0/1420724573 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520035040 con 0x55aa4d3441a0 2024-04-25T00:15:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.062+0000 7fb5355ee740 1 -- 172.21.15.27:0/1420724573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d3441a0 msgr2=0x55aa4d348670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.062+0000 7fb5355ee740 1 --2- 172.21.15.27:0/1420724573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d3441a0 0x55aa4d348670 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb520004650 tx=0x7fb52003efb0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.062+0000 7fb5355ee740 1 -- 172.21.15.27:0/1420724573 shutdown_connections 2024-04-25T00:15:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.062+0000 7fb5355ee740 1 --2- 172.21.15.27:0/1420724573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d3441a0 0x55aa4d348670 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.062+0000 7fb5355ee740 1 --2- 172.21.15.27:0/1420724573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d33f030 0x55aa4d34ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.062+0000 7fb5355ee740 1 --2- 172.21.15.27:0/1420724573 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d33e640 0x55aa4d346590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.062+0000 7fb5355ee740 1 -- 172.21.15.27:0/1420724573 >> 172.21.15.27:0/1420724573 conn(0x55aa4d339cb0 msgr2=0x55aa4d340450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.062+0000 7fb5355ee740 1 -- 172.21.15.27:0/1420724573 shutdown_connections 2024-04-25T00:15:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.062+0000 7fb5355ee740 1 -- 172.21.15.27:0/1420724573 wait complete. 2024-04-25T00:15:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.062+0000 7fb5355ee740 1 -- start start 2024-04-25T00:15:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.062+0000 7fb5355ee740 1 Processor -- start 2024-04-25T00:15:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.062+0000 7fb5355ee740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d33e640 0x55aa4d34bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.062+0000 7fb5355ee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d33f030 0x55aa4d34c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.062+0000 7fb5355ee740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d3441a0 0x55aa4d34f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.062+0000 7fb5355ee740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d3563e0 con 0x55aa4d33e640 2024-04-25T00:15:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.062+0000 7fb5355ee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d355d10 con 0x55aa4d33f030 2024-04-25T00:15:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.062+0000 7fb5355ee740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d356ab0 con 0x55aa4d3441a0 2024-04-25T00:15:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb5355ee740 10 client.0 _ll_register_callbacks cb 0x7fb5200317d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb52b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d33e640 0x55aa4d34bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb52b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d33e640 0x55aa4d34bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56964/0 (socket says 172.21.15.27:56964) 2024-04-25T00:15:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb52b7fe640 1 -- 172.21.15.27:0/999116858 learned_addr learned my addr 172.21.15.27:0/999116858 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb52bfff640 1 --2- 172.21.15.27:0/999116858 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d33f030 0x55aa4d34c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb52b7fe640 1 --2- 172.21.15.27:0/999116858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d3441a0 0x55aa4d34f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb52b7fe640 1 -- 172.21.15.27:0/999116858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d3441a0 msgr2=0x55aa4d34f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb52b7fe640 1 --2- 172.21.15.27:0/999116858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d3441a0 0x55aa4d34f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb52b7fe640 1 -- 172.21.15.27:0/999116858 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d33f030 msgr2=0x55aa4d34c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb52b7fe640 1 --2- 172.21.15.27:0/999116858 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d33f030 0x55aa4d34c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb52b7fe640 1 -- 172.21.15.27:0/999116858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa4d34fa40 con 0x55aa4d33e640 2024-04-25T00:15:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb52bfff640 1 --2- 172.21.15.27:0/999116858 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d33f030 0x55aa4d34c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb52b7fe640 1 --2- 172.21.15.27:0/999116858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d33e640 0x55aa4d34bf60 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb520063040 tx=0x7fb520000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad678fae1e9c383 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb50b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb50b7fe640 1 -- 172.21.15.27:0/999116858 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520065030 con 0x55aa4d33e640 2024-04-25T00:15:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb50b7fe640 1 -- 172.21.15.27:0/999116858 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb52003d810 con 0x55aa4d33e640 2024-04-25T00:15:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb5355ee740 1 -- 172.21.15.27:0/999116858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb52003d450 con 0x55aa4d33e640 2024-04-25T00:15:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.063+0000 7fb50b7fe640 1 -- 172.21.15.27:0/999116858 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb520075050 con 0x55aa4d33e640 2024-04-25T00:15:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.064+0000 7fb5355ee740 10 client.15504 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.064+0000 7fb5355ee740 1 -- 172.21.15.27:0/999116858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55aa4d3bb8e0 con 0x55aa4d33e640 2024-04-25T00:15:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.064+0000 7fb50b7fe640 1 -- 172.21.15.27:0/999116858 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb52003d450 con 0x55aa4d33e640 2024-04-25T00:15:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.064+0000 7fb50b7fe640 1 -- 172.21.15.27:0/999116858 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb52005b440 con 0x55aa4d33e640 2024-04-25T00:15:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.064+0000 7fb5355ee740 10 client.15504 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.064+0000 7fb5355ee740 10 client.15504 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T00:15:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.064+0000 7fb5355ee740 20 client.15504 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.064+0000 7fb5355ee740 1 --2- 172.21.15.27:0/999116858 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55aa4d3bff30 0x55aa4d3bca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.064+0000 7fb5355ee740 4 client.15504 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T00:15:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.064+0000 7fb5355ee740 1 -- 172.21.15.27:0/999116858 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "status"}) -- 0x55aa4d2a9790 con 0x55aa4d3bff30 2024-04-25T00:15:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.064+0000 7fb52bfff640 1 --2- 172.21.15.27:0/999116858 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55aa4d3bff30 0x55aa4d3bca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.065+0000 7fb50b7fe640 10 client.15504 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:15:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.065+0000 7fb52bfff640 1 --2- 172.21.15.27:0/999116858 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55aa4d3bff30 0x55aa4d3bca40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=22c355c4815aff42 server_cookie=4203c0ce00c06fdc in_seq=0 out_seq=0 2024-04-25T00:15:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.065+0000 7fb50b7fe640 1 -- 172.21.15.27:0/999116858 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55aa4d2a9790 con 0x55aa4d3bff30 2024-04-25T00:15:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.065+0000 7fb50b7fe640 10 client.15504 handle_command_reply: tid=0 2024-04-25T00:15:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.065+0000 7fb5355ee740 1 client.15504 shutdown 2024-04-25T00:15:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.065+0000 7fb5355ee740 1 -- 172.21.15.27:0/999116858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d33e640 msgr2=0x55aa4d34bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.065+0000 7fb5355ee740 1 --2- 172.21.15.27:0/999116858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d33e640 0x55aa4d34bf60 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb520063040 tx=0x7fb520000bc0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.065+0000 7fb5355ee740 1 -- 172.21.15.27:0/999116858 shutdown_connections 2024-04-25T00:15:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.065+0000 7fb5355ee740 1 --2- 172.21.15.27:0/999116858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aa4d3441a0 0x55aa4d34f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.065+0000 7fb5355ee740 1 --2- 172.21.15.27:0/999116858 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55aa4d3bff30 0x55aa4d3bca40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.065+0000 7fb5355ee740 1 --2- 172.21.15.27:0/999116858 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aa4d33f030 0x55aa4d34c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.065+0000 7fb5355ee740 1 --2- 172.21.15.27:0/999116858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa4d33e640 0x55aa4d34bf60 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.065+0000 7fb50b7fe640 0 client.15504 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:15:36.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.065+0000 7fb5355ee740 1 -- 172.21.15.27:0/999116858 >> 172.21.15.27:0/999116858 conn(0x55aa4d339cb0 msgr2=0x55aa4d348670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.065+0000 7fb5355ee740 1 -- 172.21.15.27:0/999116858 shutdown_connections 2024-04-25T00:15:36.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.065+0000 7fb5355ee740 1 -- 172.21.15.27:0/999116858 wait complete. 2024-04-25T00:15:36.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.065+0000 7fb5355ee740 20 client.15504 trim_cache size 0 max 16384 2024-04-25T00:15:36.073 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:15:36.073 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "518857f8-0297-11ef-bc93-c7b262605968", 2024-04-25T00:15:36.074 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 2, 2024-04-25T00:15:36.074 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24496, 2024-04-25T00:15:36.074 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T00:15:36.074 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T00:15:36.074 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T00:15:36.074 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 111.416691386, 2024-04-25T00:15:36.074 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 28, 2024-04-25T00:15:36.074 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 93, 2024-04-25T00:15:36.074 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T00:15:36.074 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 157.17658788200001 2024-04-25T00:15:36.074 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:15:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.066+0000 7fb53159e640 1 -- 172.21.15.27:0/3107296271 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb5140a2b90 msgr2=0x7fb5140a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.066+0000 7fb53159e640 1 --2- 172.21.15.27:0/3107296271 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb5140a2b90 0x7fb5140a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb5240984f0 tx=0x7fb52406b9d0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.066+0000 7fb53159e640 1 -- 172.21.15.27:0/3107296271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c102100 msgr2=0x7fb52c19eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.066+0000 7fb53159e640 1 --2- 172.21.15.27:0/3107296271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c102100 0x7fb52c19eee0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb52400eaa0 tx=0x7fb52400ef60 comp rx=0 tx=0).stop 2024-04-25T00:15:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.066+0000 7fb53159e640 1 -- 172.21.15.27:0/3107296271 shutdown_connections 2024-04-25T00:15:36.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.066+0000 7fb53159e640 1 --2- 172.21.15.27:0/3107296271 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb5140a2b90 0x7fb5140a5050 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.066+0000 7fb53159e640 1 --2- 172.21.15.27:0/3107296271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb52c102100 0x7fb52c19eee0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.066+0000 7fb53159e640 1 --2- 172.21.15.27:0/3107296271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c101740 0x7fb52c197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.066+0000 7fb53159e640 1 --2- 172.21.15.27:0/3107296271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb52c100550 0x7fb52c197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.066+0000 7fb53159e640 1 -- 172.21.15.27:0/3107296271 >> 172.21.15.27:0/3107296271 conn(0x7fb52c0fbae0 msgr2=0x7fb52c0fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.066+0000 7fb53159e640 1 -- 172.21.15.27:0/3107296271 shutdown_connections 2024-04-25T00:15:36.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.066+0000 7fb53159e640 1 -- 172.21.15.27:0/3107296271 wait complete. 2024-04-25T00:15:36.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T00:15:36.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.180+0000 7fc6cc44d640 1 -- 172.21.15.27:0/3528270135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c4101790 msgr2=0x7fc6c4101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.180+0000 7fc6cc44d640 1 --2- 172.21.15.27:0/3528270135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c4101790 0x7fc6c4101bb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8009900 tx=0x7fc6b802a410 comp rx=0 tx=0).stop 2024-04-25T00:15:36.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.180+0000 7fc6cc44d640 1 -- 172.21.15.27:0/3528270135 shutdown_connections 2024-04-25T00:15:36.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.180+0000 7fc6cc44d640 1 --2- 172.21.15.27:0/3528270135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6c4103340 0x7fc6c4109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.180+0000 7fc6cc44d640 1 --2- 172.21.15.27:0/3528270135 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6c4102980 0x7fc6c4102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.180+0000 7fc6cc44d640 1 --2- 172.21.15.27:0/3528270135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c4101790 0x7fc6c4101bb0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.180+0000 7fc6cc44d640 1 -- 172.21.15.27:0/3528270135 >> 172.21.15.27:0/3528270135 conn(0x7fc6c40fcd20 msgr2=0x7fc6c40ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.181+0000 7fc6cc44d640 1 -- 172.21.15.27:0/3528270135 shutdown_connections 2024-04-25T00:15:36.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.181+0000 7fc6cc44d640 1 -- 172.21.15.27:0/3528270135 wait complete. 2024-04-25T00:15:36.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.181+0000 7fc6cc44d640 1 Processor -- start 2024-04-25T00:15:36.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.181+0000 7fc6cc44d640 1 -- start start 2024-04-25T00:15:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.181+0000 7fc6cc44d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6c4101790 0x7fc6c4197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.181+0000 7fc6cc44d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c4102980 0x7fc6c4198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.181+0000 7fc6cc44d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6c4103340 0x7fc6c419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.181+0000 7fc6cc44d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c410d3e0 con 0x7fc6c4102980 2024-04-25T00:15:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.181+0000 7fc6cc44d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c410d260 con 0x7fc6c4103340 2024-04-25T00:15:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.181+0000 7fc6cc44d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c410d560 con 0x7fc6c4101790 2024-04-25T00:15:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.181+0000 7fc6c99e8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c4102980 0x7fc6c4198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.181+0000 7fc6c99e8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c4102980 0x7fc6c4198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56990/0 (socket says 172.21.15.27:56990) 2024-04-25T00:15:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.181+0000 7fc6c99e8640 1 -- 172.21.15.27:0/1747425810 learned_addr learned my addr 172.21.15.27:0/1747425810 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.182+0000 7fc6ca1e9640 1 --2- 172.21.15.27:0/1747425810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6c4101790 0x7fc6c4197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.182+0000 7fc6ca9ea640 1 --2- 172.21.15.27:0/1747425810 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6c4103340 0x7fc6c419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.182+0000 7fc6c99e8640 1 -- 172.21.15.27:0/1747425810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6c4101790 msgr2=0x7fc6c4197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.182+0000 7fc6c99e8640 1 --2- 172.21.15.27:0/1747425810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6c4101790 0x7fc6c4197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.182+0000 7fc6c99e8640 1 -- 172.21.15.27:0/1747425810 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6c4103340 msgr2=0x7fc6c419f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.182+0000 7fc6c99e8640 1 --2- 172.21.15.27:0/1747425810 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6c4103340 0x7fc6c419f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.182+0000 7fc6c99e8640 1 -- 172.21.15.27:0/1747425810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c419f710 con 0x7fc6c4102980 2024-04-25T00:15:36.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.182+0000 7fc6c99e8640 1 --2- 172.21.15.27:0/1747425810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c4102980 0x7fc6c4198080 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000d950 tx=0x7fc6c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.182+0000 7fc6b77fe640 1 -- 172.21.15.27:0/1747425810 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c0017070 con 0x7fc6c4102980 2024-04-25T00:15:36.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.182+0000 7fc6cc44d640 1 -- 172.21.15.27:0/1747425810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6c419fa00 con 0x7fc6c4102980 2024-04-25T00:15:36.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.182+0000 7fc6b77fe640 1 -- 172.21.15.27:0/1747425810 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6c0009c30 con 0x7fc6c4102980 2024-04-25T00:15:36.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.182+0000 7fc6b77fe640 1 -- 172.21.15.27:0/1747425810 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c000f6b0 con 0x7fc6c4102980 2024-04-25T00:15:36.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.182+0000 7fc6cc44d640 1 -- 172.21.15.27:0/1747425810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6c419ffa0 con 0x7fc6c4102980 2024-04-25T00:15:36.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.185+0000 7fc6b77fe640 1 -- 172.21.15.27:0/1747425810 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc6c00109d0 con 0x7fc6c4102980 2024-04-25T00:15:36.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.185+0000 7fc6cf237740 1 Processor -- start 2024-04-25T00:15:36.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.185+0000 7fc6cf237740 1 -- start start 2024-04-25T00:15:36.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.185+0000 7fc6cf237740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4a850 0x56221da4ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.185+0000 7fc6cf237740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da4b240 0x56221da4fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.185+0000 7fc6cf237740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da503b0 0x56221da52840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.185+0000 7fc6cf237740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56221d9a9f00 con 0x56221da4b240 2024-04-25T00:15:36.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.185+0000 7fc6cf237740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56221d952750 con 0x56221da4a850 2024-04-25T00:15:36.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.185+0000 7fc6cf237740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56221d9a5ec0 con 0x56221da503b0 2024-04-25T00:15:36.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.185+0000 7fc6ca1e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da4b240 0x56221da4fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.186+0000 7fc6ca1e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da4b240 0x56221da4fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57002/0 (socket says 172.21.15.27:57002) 2024-04-25T00:15:36.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.186+0000 7fc6ca1e9640 1 -- 172.21.15.27:0/1771274496 learned_addr learned my addr 172.21.15.27:0/1771274496 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:36.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.186+0000 7fc6ca9ea640 1 --2- 172.21.15.27:0/1771274496 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4a850 0x56221da4ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.186+0000 7fc6c99e8640 1 --2- 172.21.15.27:0/1771274496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da503b0 0x56221da52840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.186+0000 7fc6ca1e9640 1 -- 172.21.15.27:0/1771274496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da503b0 msgr2=0x56221da52840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.186+0000 7fc6ca1e9640 1 --2- 172.21.15.27:0/1771274496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da503b0 0x56221da52840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.186+0000 7fc6ca1e9640 1 -- 172.21.15.27:0/1771274496 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4a850 msgr2=0x56221da4ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.186+0000 7fc6ca1e9640 1 --2- 172.21.15.27:0/1771274496 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4a850 0x56221da4ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.186+0000 7fc6b77fe640 1 --2- 172.21.15.27:0/1747425810 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc6b00a2b90 0x7fc6b00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.186+0000 7fc6ca1e9640 1 -- 172.21.15.27:0/1771274496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56221d6be2d0 con 0x56221da4b240 2024-04-25T00:15:36.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.186+0000 7fc6ca9ea640 1 --2- 172.21.15.27:0/1771274496 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4a850 0x56221da4ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:36.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.186+0000 7fc6c99e8640 1 --2- 172.21.15.27:0/1747425810 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc6b00a2b90 0x7fc6b00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.186+0000 7fc6b77fe640 1 -- 172.21.15.27:0/1747425810 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc6c00cefe0 con 0x7fc6c4102980 2024-04-25T00:15:36.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.186+0000 7fc6ca1e9640 1 --2- 172.21.15.27:0/1771274496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da4b240 0x56221da4fd40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8031610 tx=0x7fc6b8005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f92a0258855ccf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.186+0000 7fc6b4bfe640 1 -- 172.21.15.27:0/1771274496 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8040070 con 0x56221da4b240 2024-04-25T00:15:36.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.186+0000 7fc6b4bfe640 1 -- 172.21.15.27:0/1771274496 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6b80318b0 con 0x56221da4b240 2024-04-25T00:15:36.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.186+0000 7fc6b4bfe640 1 -- 172.21.15.27:0/1771274496 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8035040 con 0x56221da4b240 2024-04-25T00:15:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.187+0000 7fc6cf237740 1 -- 172.21.15.27:0/1771274496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da4b240 msgr2=0x56221da4fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.187+0000 7fc6c99e8640 1 --2- 172.21.15.27:0/1747425810 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc6b00a2b90 0x7fc6b00a5050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0097e80 tx=0x7fc6c00d2ac0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.187+0000 7fc6cf237740 1 --2- 172.21.15.27:0/1771274496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da4b240 0x56221da4fd40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8031610 tx=0x7fc6b8005c00 comp rx=0 tx=0).stop 2024-04-25T00:15:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.187+0000 7fc6cf237740 1 -- 172.21.15.27:0/1771274496 shutdown_connections 2024-04-25T00:15:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.187+0000 7fc6cf237740 1 --2- 172.21.15.27:0/1771274496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da503b0 0x56221da52840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.187+0000 7fc6cf237740 1 --2- 172.21.15.27:0/1771274496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da4b240 0x56221da4fd40 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.187+0000 7fc6cf237740 1 --2- 172.21.15.27:0/1771274496 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4a850 0x56221da4ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.187+0000 7fc6cf237740 1 -- 172.21.15.27:0/1771274496 >> 172.21.15.27:0/1771274496 conn(0x56221da45ec0 msgr2=0x56221da462c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.187+0000 7fc6cf237740 1 -- 172.21.15.27:0/1771274496 shutdown_connections 2024-04-25T00:15:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.187+0000 7fc6cf237740 1 -- 172.21.15.27:0/1771274496 wait complete. 2024-04-25T00:15:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6cf237740 1 -- start start 2024-04-25T00:15:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6cf237740 1 Processor -- start 2024-04-25T00:15:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6cf237740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da4a850 0x56221da54c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6cf237740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4b240 0x56221da55190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6cf237740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da503b0 0x56221da5c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6cf237740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56221da53300 con 0x56221da503b0 2024-04-25T00:15:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6cf237740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56221d965cf0 con 0x56221da4b240 2024-04-25T00:15:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6cf237740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56221da53480 con 0x56221da4a850 2024-04-25T00:15:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6cf237740 10 client.0 _ll_register_callbacks cb 0x56221d9656e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6ca9ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da503b0 0x56221da5c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6ca9ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da503b0 0x56221da5c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57010/0 (socket says 172.21.15.27:57010) 2024-04-25T00:15:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6ca9ea640 1 -- 172.21.15.27:0/1958172800 learned_addr learned my addr 172.21.15.27:0/1958172800 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6ca1e9640 1 --2- 172.21.15.27:0/1958172800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4b240 0x56221da55190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6ca9ea640 1 --2- 172.21.15.27:0/1958172800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da4a850 0x56221da54c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6ca9ea640 1 -- 172.21.15.27:0/1958172800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da4a850 msgr2=0x56221da54c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6ca9ea640 1 --2- 172.21.15.27:0/1958172800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da4a850 0x56221da54c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6ca9ea640 1 -- 172.21.15.27:0/1958172800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4b240 msgr2=0x56221da55190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6ca9ea640 1 --2- 172.21.15.27:0/1958172800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4b240 0x56221da55190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6ca9ea640 1 -- 172.21.15.27:0/1958172800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56221d7886b0 con 0x56221da503b0 2024-04-25T00:15:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6ca9ea640 1 --2- 172.21.15.27:0/1958172800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da503b0 0x56221da5c1d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc014c20 tx=0x7fc6bc009d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fece67e9c6da9f25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.188+0000 7fc6adffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.189+0000 7fc6adffb640 1 -- 172.21.15.27:0/1958172800 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc003970 con 0x56221da503b0 2024-04-25T00:15:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.189+0000 7fc6adffb640 1 -- 172.21.15.27:0/1958172800 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc6bc003b10 con 0x56221da503b0 2024-04-25T00:15:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.189+0000 7fc6cf237740 1 -- 172.21.15.27:0/1958172800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6b80318b0 con 0x56221da503b0 2024-04-25T00:15:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.189+0000 7fc6adffb640 1 -- 172.21.15.27:0/1958172800 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc6bc025050 con 0x56221da503b0 2024-04-25T00:15:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.189+0000 7fc6cf237740 10 client.15516 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.189+0000 7fc6cf237740 1 -- 172.21.15.27:0/1958172800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56221da5f2d0 con 0x56221da503b0 2024-04-25T00:15:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.189+0000 7fc6adffb640 1 -- 172.21.15.27:0/1958172800 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6b80318b0 con 0x56221da503b0 2024-04-25T00:15:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.189+0000 7fc6adffb640 1 -- 172.21.15.27:0/1958172800 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc6bc011740 con 0x56221da503b0 2024-04-25T00:15:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.189+0000 7fc6cf237740 10 client.15516 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.189+0000 7fc6cf237740 10 client.15516 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T00:15:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.189+0000 7fc6cf237740 20 client.15516 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.189+0000 7fc6cf237740 1 --2- 172.21.15.27:0/1958172800 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x56221da619d0 0x56221da62e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.189+0000 7fc6cf237740 4 client.15516 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:15:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.189+0000 7fc6cf237740 1 -- 172.21.15.27:0/1958172800 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56221d9b7440 con 0x56221da619d0 2024-04-25T00:15:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.190+0000 7fc6ca1e9640 1 --2- 172.21.15.27:0/1958172800 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x56221da619d0 0x56221da62e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.190+0000 7fc6ca1e9640 1 --2- 172.21.15.27:0/1958172800 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x56221da619d0 0x56221da62e40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bdd198f51fdea75d server_cookie=c082a4b880115cca in_seq=0 out_seq=0 2024-04-25T00:15:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.190+0000 7fc6adffb640 10 client.15516 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:15:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.191+0000 7fc6adffb640 1 -- 172.21.15.27:0/1958172800 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56221d9b7440 con 0x56221da619d0 2024-04-25T00:15:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.191+0000 7fc6adffb640 10 client.15516 handle_command_reply: tid=0 2024-04-25T00:15:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.191+0000 7fc6cf237740 1 client.15516 shutdown 2024-04-25T00:15:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.191+0000 7fc6cf237740 1 -- 172.21.15.27:0/1958172800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da503b0 msgr2=0x56221da5c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.191+0000 7fc6cf237740 1 --2- 172.21.15.27:0/1958172800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da503b0 0x56221da5c1d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc014c20 tx=0x7fc6bc009d60 comp rx=0 tx=0).stop 2024-04-25T00:15:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.191+0000 7fc6cf237740 1 -- 172.21.15.27:0/1958172800 shutdown_connections 2024-04-25T00:15:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.191+0000 7fc6cf237740 1 --2- 172.21.15.27:0/1958172800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da503b0 0x56221da5c1d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.191+0000 7fc6cf237740 1 --2- 172.21.15.27:0/1958172800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4b240 0x56221da55190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.191+0000 7fc6cf237740 1 --2- 172.21.15.27:0/1958172800 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x56221da619d0 0x56221da62e40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.191+0000 7fc6cf237740 1 --2- 172.21.15.27:0/1958172800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da4a850 0x56221da54c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.191+0000 7fc6cf237740 1 -- 172.21.15.27:0/1958172800 >> 172.21.15.27:0/1958172800 conn(0x56221da45ec0 msgr2=0x56221da40ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.191+0000 7fc6adffb640 0 client.15516 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:15:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.191+0000 7fc6cf237740 1 -- 172.21.15.27:0/1958172800 shutdown_connections 2024-04-25T00:15:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.191+0000 7fc6cf237740 1 -- 172.21.15.27:0/1958172800 wait complete. 2024-04-25T00:15:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.191+0000 7fc6cf237740 20 client.15516 trim_cache size 0 max 16384 2024-04-25T00:15:36.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.209+0000 7fc6cf237740 1 Processor -- start 2024-04-25T00:15:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.209+0000 7fc6cf237740 1 -- start start 2024-04-25T00:15:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.209+0000 7fc6cf237740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4a850 0x56221da4bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.209+0000 7fc6cf237740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da4b240 0x56221da5c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6cf237740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da503b0 0x56221da3c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6cf237740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56221da61180 con 0x56221da4b240 2024-04-25T00:15:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6cf237740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56221da61850 con 0x56221da4a850 2024-04-25T00:15:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6cf237740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56221da61f20 con 0x56221da503b0 2024-04-25T00:15:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6ca1e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da4b240 0x56221da5c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6ca1e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da4b240 0x56221da5c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57026/0 (socket says 172.21.15.27:57026) 2024-04-25T00:15:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6ca1e9640 1 -- 172.21.15.27:0/265150948 learned_addr learned my addr 172.21.15.27:0/265150948 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6ca9ea640 1 --2- 172.21.15.27:0/265150948 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4a850 0x56221da4bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6ca1e9640 1 -- 172.21.15.27:0/265150948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da503b0 msgr2=0x56221da3c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6ca1e9640 1 --2- 172.21.15.27:0/265150948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da503b0 0x56221da3c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6ca1e9640 1 -- 172.21.15.27:0/265150948 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4a850 msgr2=0x56221da4bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6ca1e9640 1 --2- 172.21.15.27:0/265150948 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4a850 0x56221da4bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6ca1e9640 1 -- 172.21.15.27:0/265150948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56221da46b90 con 0x56221da4b240 2024-04-25T00:15:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6ca9ea640 1 --2- 172.21.15.27:0/265150948 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4a850 0x56221da4bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6ca1e9640 1 --2- 172.21.15.27:0/265150948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da4b240 0x56221da5c0f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc6b802a930 tx=0x7fc6b802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d16cfda0cf4e1ad9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6affff640 1 -- 172.21.15.27:0/265150948 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8056070 con 0x56221da4b240 2024-04-25T00:15:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6affff640 1 -- 172.21.15.27:0/265150948 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6b80317f0 con 0x56221da4b240 2024-04-25T00:15:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.210+0000 7fc6affff640 1 -- 172.21.15.27:0/265150948 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8035040 con 0x56221da4b240 2024-04-25T00:15:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.211+0000 7fc6cf237740 1 -- 172.21.15.27:0/265150948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da4b240 msgr2=0x56221da5c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.211+0000 7fc6cf237740 1 --2- 172.21.15.27:0/265150948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da4b240 0x56221da5c0f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc6b802a930 tx=0x7fc6b802a410 comp rx=0 tx=0).stop 2024-04-25T00:15:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.211+0000 7fc6cf237740 1 -- 172.21.15.27:0/265150948 shutdown_connections 2024-04-25T00:15:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.211+0000 7fc6cf237740 1 --2- 172.21.15.27:0/265150948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da503b0 0x56221da3c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.211+0000 7fc6cf237740 1 --2- 172.21.15.27:0/265150948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da4b240 0x56221da5c0f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.211+0000 7fc6cf237740 1 --2- 172.21.15.27:0/265150948 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4a850 0x56221da4bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.211+0000 7fc6cf237740 1 -- 172.21.15.27:0/265150948 >> 172.21.15.27:0/265150948 conn(0x56221da45ec0 msgr2=0x56221da462a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.211+0000 7fc6cf237740 1 -- 172.21.15.27:0/265150948 shutdown_connections 2024-04-25T00:15:36.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.211+0000 7fc6cf237740 1 -- 172.21.15.27:0/265150948 wait complete. 2024-04-25T00:15:36.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.211+0000 7fc6cf237740 1 -- start start 2024-04-25T00:15:36.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.211+0000 7fc6cf237740 1 Processor -- start 2024-04-25T00:15:36.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6cf237740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4a850 0x56221da58320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6cf237740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da4b240 0x56221da58860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6cf237740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da503b0 0x56221da5b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6cf237740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56221da62cc0 con 0x56221da503b0 2024-04-25T00:15:36.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6cf237740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56221da625f0 con 0x56221da4a850 2024-04-25T00:15:36.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6cf237740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56221da63390 con 0x56221da4b240 2024-04-25T00:15:36.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6cf237740 10 client.0 _ll_register_callbacks cb 0x7fc6bc014830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:36.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6ca9ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da503b0 0x56221da5b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6ca9ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da503b0 0x56221da5b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57040/0 (socket says 172.21.15.27:57040) 2024-04-25T00:15:36.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6ca9ea640 1 -- 172.21.15.27:0/3636571584 learned_addr learned my addr 172.21.15.27:0/3636571584 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:36.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6ca1e9640 1 --2- 172.21.15.27:0/3636571584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da4b240 0x56221da58860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6ca9ea640 1 --2- 172.21.15.27:0/3636571584 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4a850 0x56221da58320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6ca9ea640 1 -- 172.21.15.27:0/3636571584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da4b240 msgr2=0x56221da58860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6ca9ea640 1 --2- 172.21.15.27:0/3636571584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da4b240 0x56221da58860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6ca9ea640 1 -- 172.21.15.27:0/3636571584 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4a850 msgr2=0x56221da58320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6ca9ea640 1 --2- 172.21.15.27:0/3636571584 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4a850 0x56221da58320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6ca9ea640 1 -- 172.21.15.27:0/3636571584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56221da5b770 con 0x56221da503b0 2024-04-25T00:15:36.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6ca9ea640 1 --2- 172.21.15.27:0/3636571584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da503b0 0x56221da5b130 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc002230 tx=0x7fc6bc009440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edaaaa1a08643f70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6adffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:36.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6adffb640 1 -- 172.21.15.27:0/3636571584 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc031020 con 0x56221da503b0 2024-04-25T00:15:36.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6cf237740 1 -- 172.21.15.27:0/3636571584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6b80317f0 con 0x56221da503b0 2024-04-25T00:15:36.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.212+0000 7fc6adffb640 1 -- 172.21.15.27:0/3636571584 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc6bc003970 con 0x56221da503b0 2024-04-25T00:15:36.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.213+0000 7fc6adffb640 1 -- 172.21.15.27:0/3636571584 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc6bc003b70 con 0x56221da503b0 2024-04-25T00:15:36.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.213+0000 7fc6cf237740 10 client.15522 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.213+0000 7fc6cf237740 1 -- 172.21.15.27:0/3636571584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56221dac7cd0 con 0x56221da503b0 2024-04-25T00:15:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.213+0000 7fc6adffb640 1 -- 172.21.15.27:0/3636571584 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6b80317f0 con 0x56221da503b0 2024-04-25T00:15:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.213+0000 7fc6adffb640 1 -- 172.21.15.27:0/3636571584 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc6bc004020 con 0x56221da503b0 2024-04-25T00:15:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.213+0000 7fc6cf237740 10 client.15522 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.213+0000 7fc6cf237740 10 client.15522 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T00:15:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.213+0000 7fc6cf237740 20 client.15522 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.213+0000 7fc6cf237740 1 --2- 172.21.15.27:0/3636571584 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x56221dacc7a0 0x56221dacb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.213+0000 7fc6cf237740 4 client.15522 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T00:15:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.213+0000 7fc6cf237740 1 -- 172.21.15.27:0/3636571584 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "status"}) -- 0x56221d9b1790 con 0x56221dacc7a0 2024-04-25T00:15:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.213+0000 7fc6ca1e9640 1 --2- 172.21.15.27:0/3636571584 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x56221dacc7a0 0x56221dacb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.214+0000 7fc6ca1e9640 1 --2- 172.21.15.27:0/3636571584 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x56221dacc7a0 0x56221dacb4b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=74e49439c1ee46ea server_cookie=d75e188667c357c8 in_seq=0 out_seq=0 2024-04-25T00:15:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.214+0000 7fc6adffb640 10 client.15522 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:15:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.214+0000 7fc6adffb640 1 -- 172.21.15.27:0/3636571584 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56221d9b1790 con 0x56221dacc7a0 2024-04-25T00:15:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.214+0000 7fc6adffb640 10 client.15522 handle_command_reply: tid=0 2024-04-25T00:15:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.214+0000 7fc6cf237740 1 client.15522 shutdown 2024-04-25T00:15:36.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.214+0000 7fc6cf237740 1 -- 172.21.15.27:0/3636571584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da503b0 msgr2=0x56221da5b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.214+0000 7fc6cf237740 1 --2- 172.21.15.27:0/3636571584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da503b0 0x56221da5b130 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc002230 tx=0x7fc6bc009440 comp rx=0 tx=0).stop 2024-04-25T00:15:36.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.214+0000 7fc6cf237740 1 -- 172.21.15.27:0/3636571584 shutdown_connections 2024-04-25T00:15:36.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.214+0000 7fc6cf237740 1 --2- 172.21.15.27:0/3636571584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56221da503b0 0x56221da5b130 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.214+0000 7fc6cf237740 1 --2- 172.21.15.27:0/3636571584 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x56221dacc7a0 0x56221dacb4b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.214+0000 7fc6cf237740 1 --2- 172.21.15.27:0/3636571584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56221da4b240 0x56221da58860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.214+0000 7fc6cf237740 1 --2- 172.21.15.27:0/3636571584 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56221da4a850 0x56221da58320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.214+0000 7fc6adffb640 0 client.15522 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:15:36.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.214+0000 7fc6cf237740 1 -- 172.21.15.27:0/3636571584 >> 172.21.15.27:0/3636571584 conn(0x56221da45ec0 msgr2=0x56221da3c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.214+0000 7fc6cf237740 1 -- 172.21.15.27:0/3636571584 shutdown_connections 2024-04-25T00:15:36.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.214+0000 7fc6cf237740 1 -- 172.21.15.27:0/3636571584 wait complete. 2024-04-25T00:15:36.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.214+0000 7fc6cf237740 20 client.15522 trim_cache size 0 max 16384 2024-04-25T00:15:36.222 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:15:36.222 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "518857f8-0297-11ef-bc93-c7b262605968", 2024-04-25T00:15:36.222 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 1, 2024-04-25T00:15:36.222 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24514, 2024-04-25T00:15:36.222 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T00:15:36.222 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T00:15:36.222 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T00:15:36.222 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 112.573736557, 2024-04-25T00:15:36.222 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 28, 2024-04-25T00:15:36.222 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 93, 2024-04-25T00:15:36.222 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T00:15:36.223 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 151.48642623800001 2024-04-25T00:15:36.223 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:15:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.215+0000 7fc6cc44d640 1 -- 172.21.15.27:0/1747425810 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc6b00a2b90 msgr2=0x7fc6b00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.215+0000 7fc6cc44d640 1 --2- 172.21.15.27:0/1747425810 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc6b00a2b90 0x7fc6b00a5050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0097e80 tx=0x7fc6c00d2ac0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.215+0000 7fc6cc44d640 1 -- 172.21.15.27:0/1747425810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c4102980 msgr2=0x7fc6c4198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.215+0000 7fc6cc44d640 1 --2- 172.21.15.27:0/1747425810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c4102980 0x7fc6c4198080 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000d950 tx=0x7fc6c000de10 comp rx=0 tx=0).stop 2024-04-25T00:15:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.215+0000 7fc6cc44d640 1 -- 172.21.15.27:0/1747425810 shutdown_connections 2024-04-25T00:15:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.215+0000 7fc6cc44d640 1 --2- 172.21.15.27:0/1747425810 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fc6b00a2b90 0x7fc6b00a5050 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.215+0000 7fc6cc44d640 1 --2- 172.21.15.27:0/1747425810 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6c4103340 0x7fc6c419f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.215+0000 7fc6cc44d640 1 --2- 172.21.15.27:0/1747425810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c4102980 0x7fc6c4198080 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.215+0000 7fc6cc44d640 1 --2- 172.21.15.27:0/1747425810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6c4101790 0x7fc6c4197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.215+0000 7fc6cc44d640 1 -- 172.21.15.27:0/1747425810 >> 172.21.15.27:0/1747425810 conn(0x7fc6c40fcd20 msgr2=0x7fc6c4105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.215+0000 7fc6cc44d640 1 -- 172.21.15.27:0/1747425810 shutdown_connections 2024-04-25T00:15:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.215+0000 7fc6cc44d640 1 -- 172.21.15.27:0/1747425810 wait complete. 2024-04-25T00:15:36.227 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T00:15:36.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.329+0000 7f574063f640 1 -- 172.21.15.27:0/2999937420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5738103350 msgr2=0x7f5738109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.329+0000 7f574063f640 1 --2- 172.21.15.27:0/2999937420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5738103350 0x7f5738109be0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f5730009ca0 tx=0x7f573002a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.329+0000 7f574063f640 1 -- 172.21.15.27:0/2999937420 shutdown_connections 2024-04-25T00:15:36.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.329+0000 7f574063f640 1 --2- 172.21.15.27:0/2999937420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5738103350 0x7f5738109be0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.329+0000 7f574063f640 1 --2- 172.21.15.27:0/2999937420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5738102990 0x7f5738102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.329+0000 7f574063f640 1 --2- 172.21.15.27:0/2999937420 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57381017a0 0x7f5738101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.329+0000 7f574063f640 1 -- 172.21.15.27:0/2999937420 >> 172.21.15.27:0/2999937420 conn(0x7f57380fcd10 msgr2=0x7f57380ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.329+0000 7f574063f640 1 -- 172.21.15.27:0/2999937420 shutdown_connections 2024-04-25T00:15:36.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.329+0000 7f574063f640 1 -- 172.21.15.27:0/2999937420 wait complete. 2024-04-25T00:15:36.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.329+0000 7f574063f640 1 Processor -- start 2024-04-25T00:15:36.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.329+0000 7f574063f640 1 -- start start 2024-04-25T00:15:36.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f574063f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57381017a0 0x7f57380754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f574063f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5738102990 0x7f5738073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f574063f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5738103350 0x7f5738074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f574063f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f573810d400 con 0x7f57381017a0 2024-04-25T00:15:36.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f574063f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f573810d280 con 0x7f5738103350 2024-04-25T00:15:36.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f574063f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f573810d580 con 0x7f5738102990 2024-04-25T00:15:36.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f573e3db640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57381017a0 0x7f57380754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f573e3db640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57381017a0 0x7f57380754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57056/0 (socket says 172.21.15.27:57056) 2024-04-25T00:15:36.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f573e3db640 1 -- 172.21.15.27:0/562776785 learned_addr learned my addr 172.21.15.27:0/562776785 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:36.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f573ebdc640 1 --2- 172.21.15.27:0/562776785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5738103350 0x7f5738074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f573dbda640 1 --2- 172.21.15.27:0/562776785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5738102990 0x7f5738073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f573e3db640 1 -- 172.21.15.27:0/562776785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5738102990 msgr2=0x7f5738073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f573e3db640 1 --2- 172.21.15.27:0/562776785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5738102990 0x7f5738073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f573e3db640 1 -- 172.21.15.27:0/562776785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5738103350 msgr2=0x7f5738074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f573e3db640 1 --2- 172.21.15.27:0/562776785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5738103350 0x7f5738074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f573e3db640 1 -- 172.21.15.27:0/562776785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5738074900 con 0x7f57381017a0 2024-04-25T00:15:36.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f573dbda640 1 --2- 172.21.15.27:0/562776785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5738102990 0x7f5738073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:36.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f573e3db640 1 --2- 172.21.15.27:0/562776785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57381017a0 0x7f57380754f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f572800d770 tx=0x7f572800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f572f7fe640 1 -- 172.21.15.27:0/562776785 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5728017070 con 0x7f57381017a0 2024-04-25T00:15:36.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f574063f640 1 -- 172.21.15.27:0/562776785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57381a1b40 con 0x7f57381017a0 2024-04-25T00:15:36.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f572f7fe640 1 -- 172.21.15.27:0/562776785 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5728009c30 con 0x7f57381017a0 2024-04-25T00:15:36.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f572f7fe640 1 -- 172.21.15.27:0/562776785 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5728003dd0 con 0x7f57381017a0 2024-04-25T00:15:36.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.330+0000 7f574063f640 1 -- 172.21.15.27:0/562776785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57381a2090 con 0x7f57381017a0 2024-04-25T00:15:36.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.333+0000 7f574342e740 1 Processor -- start 2024-04-25T00:15:36.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f574342e740 1 -- start start 2024-04-25T00:15:36.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f572f7fe640 1 -- 172.21.15.27:0/562776785 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5728003f70 con 0x7f57381017a0 2024-04-25T00:15:36.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f574342e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a4d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f574342e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 0x556366a52930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f574342e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a52fa0 0x556366a55430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f574342e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5563669b0f00 con 0x556366a4d440 2024-04-25T00:15:36.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f574342e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556366959750 con 0x556366a4de30 2024-04-25T00:15:36.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f574342e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5563669acec0 con 0x556366a52fa0 2024-04-25T00:15:36.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f573ebdc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a4d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f573ebdc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a4d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57064/0 (socket says 172.21.15.27:57064) 2024-04-25T00:15:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f573ebdc640 1 -- 172.21.15.27:0/1780913103 learned_addr learned my addr 172.21.15.27:0/1780913103 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f573e3db640 1 --2- 172.21.15.27:0/1780913103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 0x556366a52930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f573dbda640 1 --2- 172.21.15.27:0/1780913103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a52fa0 0x556366a55430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f573ebdc640 1 -- 172.21.15.27:0/1780913103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a52fa0 msgr2=0x556366a55430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f573ebdc640 1 --2- 172.21.15.27:0/1780913103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a52fa0 0x556366a55430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f573ebdc640 1 -- 172.21.15.27:0/1780913103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 msgr2=0x556366a52930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f573ebdc640 1 --2- 172.21.15.27:0/1780913103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 0x556366a52930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.334+0000 7f573ebdc640 1 -- 172.21.15.27:0/1780913103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563666c52d0 con 0x556366a4d440 2024-04-25T00:15:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.335+0000 7f572f7fe640 1 --2- 172.21.15.27:0/562776785 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f57240a2b90 0x7f57240a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.335+0000 7f573dbda640 1 --2- 172.21.15.27:0/1780913103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a52fa0 0x556366a55430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.335+0000 7f573e3db640 1 --2- 172.21.15.27:0/562776785 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f57240a2b90 0x7f57240a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.335+0000 7f573ebdc640 1 --2- 172.21.15.27:0/1780913103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a4d860 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5730035000 tx=0x7f5730005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79386d7bea57de6e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.335+0000 7f572f7fe640 1 -- 172.21.15.27:0/562776785 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f57280cee40 con 0x7f57381017a0 2024-04-25T00:15:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.335+0000 7f572cbfe640 1 -- 172.21.15.27:0/1780913103 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730031450 con 0x556366a4d440 2024-04-25T00:15:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.335+0000 7f572cbfe640 1 -- 172.21.15.27:0/1780913103 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5730036070 con 0x556366a4d440 2024-04-25T00:15:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.335+0000 7f572cbfe640 1 -- 172.21.15.27:0/1780913103 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57300044b0 con 0x556366a4d440 2024-04-25T00:15:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.336+0000 7f574342e740 1 -- 172.21.15.27:0/1780913103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 msgr2=0x556366a4d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.336+0000 7f574342e740 1 --2- 172.21.15.27:0/1780913103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a4d860 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5730035000 tx=0x7f5730005fa0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.336+0000 7f573e3db640 1 --2- 172.21.15.27:0/562776785 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f57240a2b90 0x7f57240a5050 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f57280cc920 tx=0x7f57280d4040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.336+0000 7f574342e740 1 -- 172.21.15.27:0/1780913103 shutdown_connections 2024-04-25T00:15:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.336+0000 7f574342e740 1 --2- 172.21.15.27:0/1780913103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a52fa0 0x556366a55430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.336+0000 7f574342e740 1 --2- 172.21.15.27:0/1780913103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 0x556366a52930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.336+0000 7f574342e740 1 --2- 172.21.15.27:0/1780913103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a4d860 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.336+0000 7f574342e740 1 -- 172.21.15.27:0/1780913103 >> 172.21.15.27:0/1780913103 conn(0x556366a48ab0 msgr2=0x556366a48eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.336+0000 7f574342e740 1 -- 172.21.15.27:0/1780913103 shutdown_connections 2024-04-25T00:15:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.336+0000 7f574342e740 1 -- 172.21.15.27:0/1780913103 wait complete. 2024-04-25T00:15:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.336+0000 7f574342e740 1 -- start start 2024-04-25T00:15:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.336+0000 7f574342e740 1 Processor -- start 2024-04-25T00:15:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.336+0000 7f574342e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a57690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.336+0000 7f574342e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 0x556366a57bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.336+0000 7f574342e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a52fa0 0x556366a5ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f574342e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556366a55d70 con 0x556366a4d440 2024-04-25T00:15:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f574342e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55636696ccf0 con 0x556366a4de30 2024-04-25T00:15:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f574342e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556366a55ef0 con 0x556366a52fa0 2024-04-25T00:15:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f574342e740 10 client.0 _ll_register_callbacks cb 0x55636696c6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f573ebdc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a57690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f573ebdc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a57690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57074/0 (socket says 172.21.15.27:57074) 2024-04-25T00:15:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f573ebdc640 1 -- 172.21.15.27:0/939777881 learned_addr learned my addr 172.21.15.27:0/939777881 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f573dbda640 1 --2- 172.21.15.27:0/939777881 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 0x556366a57bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f573ebdc640 1 --2- 172.21.15.27:0/939777881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a52fa0 0x556366a5ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f573ebdc640 1 -- 172.21.15.27:0/939777881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a52fa0 msgr2=0x556366a5ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f573ebdc640 1 --2- 172.21.15.27:0/939777881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a52fa0 0x556366a5ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f573ebdc640 1 -- 172.21.15.27:0/939777881 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 msgr2=0x556366a57bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f573ebdc640 1 --2- 172.21.15.27:0/939777881 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 0x556366a57bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f573ebdc640 1 -- 172.21.15.27:0/939777881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55636678f6b0 con 0x556366a4d440 2024-04-25T00:15:36.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f573ebdc640 1 --2- 172.21.15.27:0/939777881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a57690 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f573002ade0 tx=0x7f573002afe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b86b33422a571a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f5721ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:36.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f5721ffb640 1 -- 172.21.15.27:0/939777881 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730050060 con 0x556366a4d440 2024-04-25T00:15:36.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f5721ffb640 1 -- 172.21.15.27:0/939777881 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f57300046c0 con 0x556366a4d440 2024-04-25T00:15:36.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f574342e740 1 -- 172.21.15.27:0/939777881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5730036070 con 0x556366a4d440 2024-04-25T00:15:36.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.337+0000 7f5721ffb640 1 -- 172.21.15.27:0/939777881 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f573002f610 con 0x556366a4d440 2024-04-25T00:15:36.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.338+0000 7f574342e740 10 client.15534 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:36.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.338+0000 7f574342e740 1 -- 172.21.15.27:0/939777881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556366a55d70 con 0x556366a4d440 2024-04-25T00:15:36.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.338+0000 7f5721ffb640 1 -- 172.21.15.27:0/939777881 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5730036070 con 0x556366a4d440 2024-04-25T00:15:36.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.338+0000 7f5721ffb640 1 -- 172.21.15.27:0/939777881 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5730042020 con 0x556366a4d440 2024-04-25T00:15:36.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.338+0000 7f574342e740 10 client.15534 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:36.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.338+0000 7f574342e740 10 client.15534 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T00:15:36.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.338+0000 7f574342e740 20 client.15534 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:36.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.338+0000 7f574342e740 1 --2- 172.21.15.27:0/939777881 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556366a63db0 0x556366a65220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.338+0000 7f574342e740 4 client.15534 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:15:36.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.338+0000 7f574342e740 1 -- 172.21.15.27:0/939777881 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563669be440 con 0x556366a63db0 2024-04-25T00:15:36.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.338+0000 7f573dbda640 1 --2- 172.21.15.27:0/939777881 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556366a63db0 0x556366a65220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.339+0000 7f573dbda640 1 --2- 172.21.15.27:0/939777881 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556366a63db0 0x556366a65220 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=211398b196cd9d09 server_cookie=9ecf3e4774c9059c in_seq=0 out_seq=0 2024-04-25T00:15:36.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.339+0000 7f5721ffb640 10 client.15534 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:15:36.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.340+0000 7f5721ffb640 1 -- 172.21.15.27:0/939777881 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5563669be440 con 0x556366a63db0 2024-04-25T00:15:36.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.340+0000 7f5721ffb640 10 client.15534 handle_command_reply: tid=0 2024-04-25T00:15:36.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.340+0000 7f574342e740 1 client.15534 shutdown 2024-04-25T00:15:36.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.340+0000 7f574342e740 1 -- 172.21.15.27:0/939777881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 msgr2=0x556366a57690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.340+0000 7f574342e740 1 --2- 172.21.15.27:0/939777881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a57690 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f573002ade0 tx=0x7f573002afe0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.340+0000 7f574342e740 1 -- 172.21.15.27:0/939777881 shutdown_connections 2024-04-25T00:15:36.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.340+0000 7f574342e740 1 --2- 172.21.15.27:0/939777881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a52fa0 0x556366a5ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.340+0000 7f574342e740 1 --2- 172.21.15.27:0/939777881 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556366a63db0 0x556366a65220 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.340+0000 7f574342e740 1 --2- 172.21.15.27:0/939777881 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 0x556366a57bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.340+0000 7f574342e740 1 --2- 172.21.15.27:0/939777881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a57690 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.340+0000 7f574342e740 1 -- 172.21.15.27:0/939777881 >> 172.21.15.27:0/939777881 conn(0x556366a48ab0 msgr2=0x556366a47ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.340+0000 7f5721ffb640 0 client.15534 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:15:36.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.340+0000 7f574342e740 1 -- 172.21.15.27:0/939777881 shutdown_connections 2024-04-25T00:15:36.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.340+0000 7f574342e740 1 -- 172.21.15.27:0/939777881 wait complete. 2024-04-25T00:15:36.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.340+0000 7f574342e740 20 client.15534 trim_cache size 0 max 16384 2024-04-25T00:15:36.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.358+0000 7f574342e740 1 Processor -- start 2024-04-25T00:15:36.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.358+0000 7f574342e740 1 -- start start 2024-04-25T00:15:36.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f574342e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a43690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f574342e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 0x556366a5ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f574342e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a52fa0 0x556366a62ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f574342e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556366a63560 con 0x556366a4d440 2024-04-25T00:15:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f574342e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556366a63c30 con 0x556366a4de30 2024-04-25T00:15:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f574342e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556366a64300 con 0x556366a52fa0 2024-04-25T00:15:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f573ebdc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a43690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f573ebdc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a43690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57078/0 (socket says 172.21.15.27:57078) 2024-04-25T00:15:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f573ebdc640 1 -- 172.21.15.27:0/1641370494 learned_addr learned my addr 172.21.15.27:0/1641370494 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f573dbda640 1 --2- 172.21.15.27:0/1641370494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 0x556366a5ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f573ebdc640 1 -- 172.21.15.27:0/1641370494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a52fa0 msgr2=0x556366a62ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f573ebdc640 1 --2- 172.21.15.27:0/1641370494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a52fa0 0x556366a62ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f573ebdc640 1 -- 172.21.15.27:0/1641370494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 msgr2=0x556366a5ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f573ebdc640 1 --2- 172.21.15.27:0/1641370494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 0x556366a5ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f573ebdc640 1 -- 172.21.15.27:0/1641370494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556366a485b0 con 0x556366a4d440 2024-04-25T00:15:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f573ebdc640 1 --2- 172.21.15.27:0/1641370494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a43690 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f5730005d40 tx=0x7f57300021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82eed9861291569 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f5723fff640 1 -- 172.21.15.27:0/1641370494 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573003f070 con 0x556366a4d440 2024-04-25T00:15:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f5723fff640 1 -- 172.21.15.27:0/1641370494 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5730036040 con 0x556366a4d440 2024-04-25T00:15:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.359+0000 7f5723fff640 1 -- 172.21.15.27:0/1641370494 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730050070 con 0x556366a4d440 2024-04-25T00:15:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.360+0000 7f574342e740 1 -- 172.21.15.27:0/1641370494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 msgr2=0x556366a43690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.360+0000 7f574342e740 1 --2- 172.21.15.27:0/1641370494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a43690 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f5730005d40 tx=0x7f57300021d0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.360+0000 7f574342e740 1 -- 172.21.15.27:0/1641370494 shutdown_connections 2024-04-25T00:15:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.360+0000 7f574342e740 1 --2- 172.21.15.27:0/1641370494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a52fa0 0x556366a62ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.360+0000 7f574342e740 1 --2- 172.21.15.27:0/1641370494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 0x556366a5ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.360+0000 7f574342e740 1 --2- 172.21.15.27:0/1641370494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a4d440 0x556366a43690 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.360+0000 7f574342e740 1 -- 172.21.15.27:0/1641370494 >> 172.21.15.27:0/1641370494 conn(0x556366a48ab0 msgr2=0x556366a48e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.360+0000 7f574342e740 1 -- 172.21.15.27:0/1641370494 shutdown_connections 2024-04-25T00:15:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.360+0000 7f574342e740 1 -- 172.21.15.27:0/1641370494 wait complete. 2024-04-25T00:15:36.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.360+0000 7f574342e740 1 -- start start 2024-04-25T00:15:36.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.360+0000 7f574342e740 1 Processor -- start 2024-04-25T00:15:36.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f574342e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a4d440 0x556366a5a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f574342e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 0x556366a5abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f574342e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a52fa0 0x556366a5dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f574342e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556366a650a0 con 0x556366a52fa0 2024-04-25T00:15:36.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f574342e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556366a649d0 con 0x556366a4de30 2024-04-25T00:15:36.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f574342e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556366a65770 con 0x556366a4d440 2024-04-25T00:15:36.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f574342e740 10 client.0 _ll_register_callbacks cb 0x7f5730003b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:36.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f573ebdc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a52fa0 0x556366a5dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f573ebdc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a52fa0 0x556366a5dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57086/0 (socket says 172.21.15.27:57086) 2024-04-25T00:15:36.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f573ebdc640 1 -- 172.21.15.27:0/391409493 learned_addr learned my addr 172.21.15.27:0/391409493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:36.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f573dbda640 1 --2- 172.21.15.27:0/391409493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 0x556366a5abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f573ebdc640 1 --2- 172.21.15.27:0/391409493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a4d440 0x556366a5a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f573ebdc640 1 -- 172.21.15.27:0/391409493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a4d440 msgr2=0x556366a5a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f573ebdc640 1 --2- 172.21.15.27:0/391409493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a4d440 0x556366a5a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f573ebdc640 1 -- 172.21.15.27:0/391409493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 msgr2=0x556366a5abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:36.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f573ebdc640 1 --2- 172.21.15.27:0/391409493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 0x556366a5abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f573ebdc640 1 -- 172.21.15.27:0/391409493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556366a49780 con 0x556366a52fa0 2024-04-25T00:15:36.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f573ebdc640 1 --2- 172.21.15.27:0/391409493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a52fa0 0x556366a5dc00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5730003a50 tx=0x7f5730005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d287be5bd1a98309 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f5721ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:36.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f5721ffb640 1 -- 172.21.15.27:0/391409493 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730050030 con 0x556366a52fa0 2024-04-25T00:15:36.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f5721ffb640 1 -- 172.21.15.27:0/391409493 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f573002f7b0 con 0x556366a52fa0 2024-04-25T00:15:36.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.361+0000 7f574342e740 1 -- 172.21.15.27:0/391409493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5730036040 con 0x556366a52fa0 2024-04-25T00:15:36.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.362+0000 7f5721ffb640 1 -- 172.21.15.27:0/391409493 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f573003f040 con 0x556366a52fa0 2024-04-25T00:15:36.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.362+0000 7f574342e740 10 client.15540 fetch_fsmap learned FSMap version 28 2024-04-25T00:15:36.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.362+0000 7f574342e740 1 -- 172.21.15.27:0/391409493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556366ac9e90 con 0x556366a52fa0 2024-04-25T00:15:36.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.362+0000 7f5721ffb640 1 -- 172.21.15.27:0/391409493 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5730036040 con 0x556366a52fa0 2024-04-25T00:15:36.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.362+0000 7f5721ffb640 1 -- 172.21.15.27:0/391409493 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5730042070 con 0x556366a52fa0 2024-04-25T00:15:36.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.362+0000 7f574342e740 10 client.15540 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T00:15:36.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.362+0000 7f574342e740 10 client.15540 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T00:15:36.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.362+0000 7f574342e740 20 client.15540 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:36.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.362+0000 7f574342e740 1 --2- 172.21.15.27:0/391409493 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556366ace880 0x556366acd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.362+0000 7f574342e740 4 client.15540 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T00:15:36.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.362+0000 7f574342e740 1 -- 172.21.15.27:0/391409493 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "status"}) -- 0x5563669b8790 con 0x556366ace880 2024-04-25T00:15:36.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.362+0000 7f573dbda640 1 --2- 172.21.15.27:0/391409493 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556366ace880 0x556366acd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.363+0000 7f5721ffb640 10 client.15540 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:15:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.363+0000 7f573dbda640 1 --2- 172.21.15.27:0/391409493 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556366ace880 0x556366acd590 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28c45b4e437bfba9 server_cookie=3b22830941087d34 in_seq=0 out_seq=0 2024-04-25T00:15:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.363+0000 7f5721ffb640 1 -- 172.21.15.27:0/391409493 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5563669b8790 con 0x556366ace880 2024-04-25T00:15:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.363+0000 7f5721ffb640 10 client.15540 handle_command_reply: tid=0 2024-04-25T00:15:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.363+0000 7f574342e740 1 client.15540 shutdown 2024-04-25T00:15:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.363+0000 7f574342e740 1 -- 172.21.15.27:0/391409493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a52fa0 msgr2=0x556366a5dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.363+0000 7f574342e740 1 --2- 172.21.15.27:0/391409493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a52fa0 0x556366a5dc00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5730003a50 tx=0x7f5730005fa0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.363+0000 7f574342e740 1 -- 172.21.15.27:0/391409493 shutdown_connections 2024-04-25T00:15:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.363+0000 7f574342e740 1 --2- 172.21.15.27:0/391409493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556366a52fa0 0x556366a5dc00 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.363+0000 7f574342e740 1 --2- 172.21.15.27:0/391409493 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556366ace880 0x556366acd590 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.363+0000 7f574342e740 1 --2- 172.21.15.27:0/391409493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556366a4de30 0x556366a5abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.363+0000 7f574342e740 1 --2- 172.21.15.27:0/391409493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556366a4d440 0x556366a5a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.363+0000 7f574342e740 1 -- 172.21.15.27:0/391409493 >> 172.21.15.27:0/391409493 conn(0x556366a48ab0 msgr2=0x556366a43690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.363+0000 7f5721ffb640 0 client.15540 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:15:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.363+0000 7f574342e740 1 -- 172.21.15.27:0/391409493 shutdown_connections 2024-04-25T00:15:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.363+0000 7f574342e740 1 -- 172.21.15.27:0/391409493 wait complete. 2024-04-25T00:15:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.364+0000 7f574342e740 20 client.15540 trim_cache size 0 max 16384 2024-04-25T00:15:36.375 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:15:36.375 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "518857f8-0297-11ef-bc93-c7b262605968", 2024-04-25T00:15:36.375 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-04-25T00:15:36.375 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24521, 2024-04-25T00:15:36.375 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T00:15:36.376 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T00:15:36.376 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T00:15:36.376 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 136.85745635699999, 2024-04-25T00:15:36.376 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 28, 2024-04-25T00:15:36.376 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 93, 2024-04-25T00:15:36.376 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T00:15:36.376 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 142.98909185799999 2024-04-25T00:15:36.376 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:15:36.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.364+0000 7f574063f640 1 -- 172.21.15.27:0/562776785 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f57240a2b90 msgr2=0x7f57240a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.364+0000 7f574063f640 1 --2- 172.21.15.27:0/562776785 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f57240a2b90 0x7f57240a5050 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f57280cc920 tx=0x7f57280d4040 comp rx=0 tx=0).stop 2024-04-25T00:15:36.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.364+0000 7f574063f640 1 -- 172.21.15.27:0/562776785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57381017a0 msgr2=0x7f57380754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.364+0000 7f574063f640 1 --2- 172.21.15.27:0/562776785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57381017a0 0x7f57380754f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f572800d770 tx=0x7f572800dc30 comp rx=0 tx=0).stop 2024-04-25T00:15:36.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.364+0000 7f574063f640 1 -- 172.21.15.27:0/562776785 shutdown_connections 2024-04-25T00:15:36.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.364+0000 7f574063f640 1 --2- 172.21.15.27:0/562776785 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f57240a2b90 0x7f57240a5050 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.364+0000 7f574063f640 1 --2- 172.21.15.27:0/562776785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5738103350 0x7f5738074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.364+0000 7f574063f640 1 --2- 172.21.15.27:0/562776785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5738102990 0x7f5738073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.364+0000 7f574063f640 1 --2- 172.21.15.27:0/562776785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57381017a0 0x7f57380754f0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.364+0000 7f574063f640 1 -- 172.21.15.27:0/562776785 >> 172.21.15.27:0/562776785 conn(0x7f57380fcd10 msgr2=0x7f5738105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.364+0000 7f574063f640 1 -- 172.21.15.27:0/562776785 shutdown_connections 2024-04-25T00:15:36.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.364+0000 7f574063f640 1 -- 172.21.15.27:0/562776785 wait complete. 2024-04-25T00:15:36.379 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T00:15:36.388 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T00:15:36.388 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T00:15:36.389 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T00:15:36.389 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-25T00:15:36.390 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T00:15:36.390 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T00:15:36.390 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-25T00:15:36.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.491+0000 7f4068d1b640 1 -- 172.21.15.27:0/2941089192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40641029c0 msgr2=0x7f4064102e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.491+0000 7f4068d1b640 1 --2- 172.21.15.27:0/2941089192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40641029c0 0x7f4064102e40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4058009900 tx=0x7f405802a410 comp rx=0 tx=0).stop 2024-04-25T00:15:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.491+0000 7f4068d1b640 1 -- 172.21.15.27:0/2941089192 shutdown_connections 2024-04-25T00:15:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.491+0000 7f4068d1b640 1 --2- 172.21.15.27:0/2941089192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4064103380 0x7f4064109c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.491+0000 7f4068d1b640 1 --2- 172.21.15.27:0/2941089192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40641029c0 0x7f4064102e40 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.491+0000 7f4068d1b640 1 --2- 172.21.15.27:0/2941089192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f40641017d0 0x7f4064101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.491+0000 7f4068d1b640 1 -- 172.21.15.27:0/2941089192 >> 172.21.15.27:0/2941089192 conn(0x7f40640fcd60 msgr2=0x7f40640ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.491+0000 7f4068d1b640 1 -- 172.21.15.27:0/2941089192 shutdown_connections 2024-04-25T00:15:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.491+0000 7f4068d1b640 1 -- 172.21.15.27:0/2941089192 wait complete. 2024-04-25T00:15:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.492+0000 7f4068d1b640 1 Processor -- start 2024-04-25T00:15:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.492+0000 7f4068d1b640 1 -- start start 2024-04-25T00:15:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.492+0000 7f4068d1b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40641017d0 0x7f40641979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.492+0000 7f4068d1b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40641029c0 0x7f4064197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.492+0000 7f4068d1b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4064103380 0x7f406419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.492+0000 7f4068d1b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f406410d450 con 0x7f40641017d0 2024-04-25T00:15:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.492+0000 7f4068d1b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f406410d2d0 con 0x7f4064103380 2024-04-25T00:15:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.492+0000 7f4068d1b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f406410d5d0 con 0x7f40641029c0 2024-04-25T00:15:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.493+0000 7f406259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40641017d0 0x7f40641979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.493+0000 7f406259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40641017d0 0x7f40641979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57106/0 (socket says 172.21.15.27:57106) 2024-04-25T00:15:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.493+0000 7f406259c640 1 -- 172.21.15.27:0/2033700156 learned_addr learned my addr 172.21.15.27:0/2033700156 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.493+0000 7f4061d9b640 1 --2- 172.21.15.27:0/2033700156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40641029c0 0x7f4064197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.493+0000 7f4062d9d640 1 --2- 172.21.15.27:0/2033700156 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4064103380 0x7f406419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.493+0000 7f406259c640 1 -- 172.21.15.27:0/2033700156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40641029c0 msgr2=0x7f4064197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.493+0000 7f406259c640 1 --2- 172.21.15.27:0/2033700156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40641029c0 0x7f4064197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.493+0000 7f406259c640 1 -- 172.21.15.27:0/2033700156 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4064103380 msgr2=0x7f406419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.493+0000 7f406259c640 1 --2- 172.21.15.27:0/2033700156 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4064103380 0x7f406419ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.493+0000 7f406259c640 1 -- 172.21.15.27:0/2033700156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406419f5f0 con 0x7f40641017d0 2024-04-25T00:15:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.493+0000 7f406259c640 1 --2- 172.21.15.27:0/2033700156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40641017d0 0x7f40641979d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f405400d770 tx=0x7f405400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.493+0000 7f40537fe640 1 -- 172.21.15.27:0/2033700156 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4054017070 con 0x7f40641017d0 2024-04-25T00:15:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.493+0000 7f4068d1b640 1 -- 172.21.15.27:0/2033700156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f406419f8e0 con 0x7f40641017d0 2024-04-25T00:15:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.493+0000 7f40537fe640 1 -- 172.21.15.27:0/2033700156 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4054009c30 con 0x7f40641017d0 2024-04-25T00:15:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.493+0000 7f40537fe640 1 -- 172.21.15.27:0/2033700156 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4054003dd0 con 0x7f40641017d0 2024-04-25T00:15:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.493+0000 7f4068d1b640 1 -- 172.21.15.27:0/2033700156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f406419fe00 con 0x7f40641017d0 2024-04-25T00:15:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.494+0000 7f4068d1b640 1 -- 172.21.15.27:0/2033700156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4064108a20 con 0x7f40641017d0 2024-04-25T00:15:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.496+0000 7f40537fe640 1 -- 172.21.15.27:0/2033700156 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4054003f70 con 0x7f40641017d0 2024-04-25T00:15:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.496+0000 7f40537fe640 1 --2- 172.21.15.27:0/2033700156 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f404c0a2c30 0x7f404c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.497+0000 7f4061d9b640 1 --2- 172.21.15.27:0/2033700156 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f404c0a2c30 0x7f404c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.497+0000 7f40537fe640 1 -- 172.21.15.27:0/2033700156 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f40540cee90 con 0x7f40641017d0 2024-04-25T00:15:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.497+0000 7f4061d9b640 1 --2- 172.21.15.27:0/2033700156 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f404c0a2c30 0x7f404c0a50f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4058000c00 tx=0x7f40580315d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.501+0000 7f40537fe640 1 -- 172.21.15.27:0/2033700156 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f405409ba60 con 0x7f40641017d0 2024-04-25T00:15:36.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:36.738+0000 7f4068d1b640 1 -- 172.21.15.27:0/2033700156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f4064062340 con 0x7f40641017d0 2024-04-25T00:15:37.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:36 smithi139 ceph-mon[32833]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:15:37.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:36 smithi153 ceph-mon[32896]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:15:37.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:36 smithi027 ceph-mon[29208]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:15:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.715+0000 7f40537fe640 1 -- 172.21.15.27:0/2033700156 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f4054009de0 con 0x7f40641017d0 2024-04-25T00:15:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.719+0000 7f4068d1b640 1 -- 172.21.15.27:0/2033700156 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f404c0a2c30 msgr2=0x7f404c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.719+0000 7f4068d1b640 1 --2- 172.21.15.27:0/2033700156 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f404c0a2c30 0x7f404c0a50f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4058000c00 tx=0x7f40580315d0 comp rx=0 tx=0).stop 2024-04-25T00:15:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.719+0000 7f4068d1b640 1 -- 172.21.15.27:0/2033700156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40641017d0 msgr2=0x7f40641979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.719+0000 7f4068d1b640 1 --2- 172.21.15.27:0/2033700156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40641017d0 0x7f40641979d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f405400d770 tx=0x7f405400dc30 comp rx=0 tx=0).stop 2024-04-25T00:15:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.719+0000 7f4068d1b640 1 -- 172.21.15.27:0/2033700156 shutdown_connections 2024-04-25T00:15:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.719+0000 7f4068d1b640 1 --2- 172.21.15.27:0/2033700156 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f404c0a2c30 0x7f404c0a50f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.719+0000 7f4068d1b640 1 --2- 172.21.15.27:0/2033700156 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4064103380 0x7f406419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:37.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.719+0000 7f4068d1b640 1 --2- 172.21.15.27:0/2033700156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40641029c0 0x7f4064197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:37.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.719+0000 7f4068d1b640 1 --2- 172.21.15.27:0/2033700156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40641017d0 0x7f40641979d0 secure :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f405400d770 tx=0x7f405400dc30 comp rx=0 tx=0).stop 2024-04-25T00:15:37.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.719+0000 7f4068d1b640 1 -- 172.21.15.27:0/2033700156 >> 172.21.15.27:0/2033700156 conn(0x7f40640fcd60 msgr2=0x7f4064105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:37.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.719+0000 7f4068d1b640 1 -- 172.21.15.27:0/2033700156 shutdown_connections 2024-04-25T00:15:37.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.719+0000 7f4068d1b640 1 -- 172.21.15.27:0/2033700156 wait complete. 2024-04-25T00:15:37.741 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-25T00:15:37.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.859+0000 7f7a1eea5640 1 -- 172.21.15.27:0/1282166868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a18107480 msgr2=0x7f7a181098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:37.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.859+0000 7f7a1eea5640 1 --2- 172.21.15.27:0/1282166868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a18107480 0x7f7a181098d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7a10009ca0 tx=0x7f7a1002a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.859+0000 7f7a1eea5640 1 -- 172.21.15.27:0/1282166868 shutdown_connections 2024-04-25T00:15:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.859+0000 7f7a1eea5640 1 --2- 172.21.15.27:0/1282166868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a18107480 0x7f7a181098d0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.859+0000 7f7a1eea5640 1 --2- 172.21.15.27:0/1282166868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a18104b50 0x7f7a18106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.859+0000 7f7a1eea5640 1 --2- 172.21.15.27:0/1282166868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a18102220 0x7f7a18104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.859+0000 7f7a1eea5640 1 -- 172.21.15.27:0/1282166868 >> 172.21.15.27:0/1282166868 conn(0x7f7a180fbc10 msgr2=0x7f7a180fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.859+0000 7f7a1eea5640 1 -- 172.21.15.27:0/1282166868 shutdown_connections 2024-04-25T00:15:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.859+0000 7f7a1eea5640 1 -- 172.21.15.27:0/1282166868 wait complete. 2024-04-25T00:15:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a1eea5640 1 Processor -- start 2024-04-25T00:15:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a1eea5640 1 -- start start 2024-04-25T00:15:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a1eea5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a18102220 0x7f7a18197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a1eea5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a18104b50 0x7f7a18197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a1eea5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a18107480 0x7f7a1819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:37.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a1eea5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1810d440 con 0x7f7a18107480 2024-04-25T00:15:37.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a1eea5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1810d2c0 con 0x7f7a18104b50 2024-04-25T00:15:37.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a1eea5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1810d5c0 con 0x7f7a18102220 2024-04-25T00:15:37.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a1d442640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a18107480 0x7f7a1819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:37.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a1d442640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a18107480 0x7f7a1819efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35384/0 (socket says 172.21.15.27:35384) 2024-04-25T00:15:37.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a1cc41640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a18102220 0x7f7a18197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:37.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a1d442640 1 -- 172.21.15.27:0/1914966992 learned_addr learned my addr 172.21.15.27:0/1914966992 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:37.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a17fff640 1 --2- 172.21.15.27:0/1914966992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a18104b50 0x7f7a18197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:37.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a1d442640 1 -- 172.21.15.27:0/1914966992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a18102220 msgr2=0x7f7a18197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:37.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a1d442640 1 --2- 172.21.15.27:0/1914966992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a18102220 0x7f7a18197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:37.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a1d442640 1 -- 172.21.15.27:0/1914966992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a18104b50 msgr2=0x7f7a18197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:37.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a1d442640 1 --2- 172.21.15.27:0/1914966992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a18104b50 0x7f7a18197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:37.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.860+0000 7f7a1d442640 1 -- 172.21.15.27:0/1914966992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a1819f640 con 0x7f7a18107480 2024-04-25T00:15:37.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.861+0000 7f7a1d442640 1 --2- 172.21.15.27:0/1914966992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a18107480 0x7f7a1819efd0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7a10031ae0 tx=0x7f7a10031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:37.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.861+0000 7f7a15ffb640 1 -- 172.21.15.27:0/1914966992 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a10003b70 con 0x7f7a18107480 2024-04-25T00:15:37.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.861+0000 7f7a1eea5640 1 -- 172.21.15.27:0/1914966992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a1819f8d0 con 0x7f7a18107480 2024-04-25T00:15:37.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.861+0000 7f7a15ffb640 1 -- 172.21.15.27:0/1914966992 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a10003d10 con 0x7f7a18107480 2024-04-25T00:15:37.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.861+0000 7f7a15ffb640 1 -- 172.21.15.27:0/1914966992 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1002fc40 con 0x7f7a18107480 2024-04-25T00:15:37.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.861+0000 7f7a1eea5640 1 -- 172.21.15.27:0/1914966992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a1819fe70 con 0x7f7a18107480 2024-04-25T00:15:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.862+0000 7f7a1eea5640 1 -- 172.21.15.27:0/1914966992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a180fd700 con 0x7f7a18107480 2024-04-25T00:15:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.863+0000 7f7a15ffb640 1 -- 172.21.15.27:0/1914966992 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f7a10004050 con 0x7f7a18107480 2024-04-25T00:15:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.864+0000 7f7a15ffb640 1 --2- 172.21.15.27:0/1914966992 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7a080a2c80 0x7f7a080a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.864+0000 7f7a1cc41640 1 -- 172.21.15.27:0/1914966992 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7a080a2c80 msgr2=0x7f7a080a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2161554778 2024-04-25T00:15:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.864+0000 7f7a15ffb640 1 -- 172.21.15.27:0/1914966992 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7a100f0710 con 0x7f7a18107480 2024-04-25T00:15:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.864+0000 7f7a1cc41640 1 --2- 172.21.15.27:0/1914966992 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7a080a2c80 0x7f7a080a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T00:15:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:37.867+0000 7f7a15ffb640 1 -- 172.21.15.27:0/1914966992 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7a100bd2e0 con 0x7f7a18107480 2024-04-25T00:15:38.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:37 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2033700156' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T00:15:38.004 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:37 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: ignoring --setuser ceph since I am not root 2024-04-25T00:15:38.004 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:37 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: ignoring --setgroup ceph since I am not root 2024-04-25T00:15:38.004 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:37 smithi139 ceph-mgr[34218]: -- 172.21.15.139:0/1615691362 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5582b6e065a0 con 0x5582b6ddac00 2024-04-25T00:15:38.004 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:37 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:37.897+0000 7f2c46372200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T00:15:38.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:37 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2033700156' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T00:15:38.027 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:37 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: ignoring --setuser ceph since I am not root 2024-04-25T00:15:38.027 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:37 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: ignoring --setgroup ceph since I am not root 2024-04-25T00:15:38.027 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:37 smithi153 ceph-mgr[34782]: -- 172.21.15.153:0/1295259184 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55be3fb9e5a0 con 0x55be3fb72800 2024-04-25T00:15:38.027 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:37 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:37.896+0000 7f5d23980200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T00:15:38.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.057+0000 7f7a1eea5640 1 -- 172.21.15.27:0/1914966992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f7a1802d170 con 0x7f7a18107480 2024-04-25T00:15:38.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.061+0000 7f7a15ffb640 1 -- 172.21.15.27:0/1914966992 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v26)=0 v26) ==== 138+0+0 (secure 0 0 0) 0x7f7a100bcbe0 con 0x7f7a18107480 2024-04-25T00:15:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.064+0000 7f7a1eea5640 1 -- 172.21.15.27:0/1914966992 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7a080a2c80 msgr2=0x7f7a080a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:15:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.064+0000 7f7a1eea5640 1 --2- 172.21.15.27:0/1914966992 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7a080a2c80 0x7f7a080a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.064+0000 7f7a1eea5640 1 -- 172.21.15.27:0/1914966992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a18107480 msgr2=0x7f7a1819efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.064+0000 7f7a1eea5640 1 --2- 172.21.15.27:0/1914966992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a18107480 0x7f7a1819efd0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7a10031ae0 tx=0x7f7a10031ee0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.064+0000 7f7a1eea5640 1 -- 172.21.15.27:0/1914966992 shutdown_connections 2024-04-25T00:15:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.064+0000 7f7a1eea5640 1 --2- 172.21.15.27:0/1914966992 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f7a080a2c80 0x7f7a080a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.064+0000 7f7a1eea5640 1 --2- 172.21.15.27:0/1914966992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a18107480 0x7f7a1819efd0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.064+0000 7f7a1eea5640 1 --2- 172.21.15.27:0/1914966992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a18104b50 0x7f7a18197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.064+0000 7f7a1eea5640 1 --2- 172.21.15.27:0/1914966992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a18102220 0x7f7a18197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.064+0000 7f7a1eea5640 1 -- 172.21.15.27:0/1914966992 >> 172.21.15.27:0/1914966992 conn(0x7f7a180fbc10 msgr2=0x7f7a18102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:38.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.064+0000 7f7a1eea5640 1 -- 172.21.15.27:0/1914966992 shutdown_connections 2024-04-25T00:15:38.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.064+0000 7f7a1eea5640 1 -- 172.21.15.27:0/1914966992 wait complete. 2024-04-25T00:15:38.075 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:37 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2033700156' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T00:15:38.075 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:37 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: ignoring --setuser ceph since I am not root 2024-04-25T00:15:38.075 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:37 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: ignoring --setgroup ceph since I am not root 2024-04-25T00:15:38.075 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:37 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:37.919+0000 7fa62c066200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T00:15:38.076 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-25T00:15:38.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.179+0000 7f197df63640 1 -- 172.21.15.27:0/2018094997 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19781006a0 msgr2=0x7f1978100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:38.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.179+0000 7f197df63640 1 --2- 172.21.15.27:0/2018094997 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19781006a0 0x7f1978100ac0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1968009ca0 tx=0x7f196802a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.179+0000 7f197df63640 1 -- 172.21.15.27:0/2018094997 shutdown_connections 2024-04-25T00:15:38.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.179+0000 7f197df63640 1 --2- 172.21.15.27:0/2018094997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1978102250 0x7f1978108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.179+0000 7f197df63640 1 --2- 172.21.15.27:0/2018094997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1978101890 0x7f1978101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.179+0000 7f197df63640 1 --2- 172.21.15.27:0/2018094997 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19781006a0 0x7f1978100ac0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.179+0000 7f197df63640 1 -- 172.21.15.27:0/2018094997 >> 172.21.15.27:0/2018094997 conn(0x7f19780fbc10 msgr2=0x7f19780fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:38.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.179+0000 7f197df63640 1 -- 172.21.15.27:0/2018094997 shutdown_connections 2024-04-25T00:15:38.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.179+0000 7f197df63640 1 -- 172.21.15.27:0/2018094997 wait complete. 2024-04-25T00:15:38.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.180+0000 7f197df63640 1 Processor -- start 2024-04-25T00:15:38.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.180+0000 7f197df63640 1 -- start start 2024-04-25T00:15:38.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.180+0000 7f197df63640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19781006a0 0x7f1978197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:38.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.180+0000 7f197df63640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1978101890 0x7f1978197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.180+0000 7f197df63640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1978102250 0x7f197819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.180+0000 7f197df63640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f197810c340 con 0x7f1978101890 2024-04-25T00:15:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.180+0000 7f197df63640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f197810c1c0 con 0x7f19781006a0 2024-04-25T00:15:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.180+0000 7f197df63640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f197810c4c0 con 0x7f1978102250 2024-04-25T00:15:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f1976ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1978101890 0x7f1978197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f19777fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19781006a0 0x7f1978197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f1977fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1978102250 0x7f197819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f1976ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1978101890 0x7f1978197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35392/0 (socket says 172.21.15.27:35392) 2024-04-25T00:15:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f1976ffd640 1 -- 172.21.15.27:0/346933826 learned_addr learned my addr 172.21.15.27:0/346933826 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f1977fff640 1 -- 172.21.15.27:0/346933826 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19781006a0 msgr2=0x7f1978197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f1977fff640 1 --2- 172.21.15.27:0/346933826 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19781006a0 0x7f1978197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f1977fff640 1 -- 172.21.15.27:0/346933826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1978101890 msgr2=0x7f1978197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f1977fff640 1 --2- 172.21.15.27:0/346933826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1978101890 0x7f1978197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f1977fff640 1 -- 172.21.15.27:0/346933826 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197819f6c0 con 0x7f1978102250 2024-04-25T00:15:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f19777fe640 1 --2- 172.21.15.27:0/346933826 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19781006a0 0x7f1978197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:38.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f1976ffd640 1 --2- 172.21.15.27:0/346933826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1978101890 0x7f1978197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:38.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f1977fff640 1 --2- 172.21.15.27:0/346933826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1978102250 0x7f197819f050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f197000c710 tx=0x7f197000cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:38.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f1974ff9640 1 -- 172.21.15.27:0/346933826 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1970004220 con 0x7f1978102250 2024-04-25T00:15:38.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f197df63640 1 -- 172.21.15.27:0/346933826 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f197819f9b0 con 0x7f1978102250 2024-04-25T00:15:38.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f197df63640 1 -- 172.21.15.27:0/346933826 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f197819ff00 con 0x7f1978102250 2024-04-25T00:15:38.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f1974ff9640 1 -- 172.21.15.27:0/346933826 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f197000ce20 con 0x7f1978102250 2024-04-25T00:15:38.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.181+0000 7f1974ff9640 1 -- 172.21.15.27:0/346933826 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197001d060 con 0x7f1978102250 2024-04-25T00:15:38.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.183+0000 7f197df63640 1 -- 172.21.15.27:0/346933826 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1964005290 con 0x7f1978102250 2024-04-25T00:15:38.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.184+0000 7f1974ff9640 1 -- 172.21.15.27:0/346933826 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f19700079b0 con 0x7f1978102250 2024-04-25T00:15:38.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.184+0000 7f1974ff9640 1 --2- 172.21.15.27:0/346933826 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f19600a2c80 0x7f19600a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:38.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.184+0000 7f19777fe640 1 -- 172.21.15.27:0/346933826 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f19600a2c80 msgr2=0x7f19600a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2161554778 2024-04-25T00:15:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.184+0000 7f1974ff9640 1 -- 172.21.15.27:0/346933826 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f19700cec30 con 0x7f1978102250 2024-04-25T00:15:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.184+0000 7f19777fe640 1 --2- 172.21.15.27:0/346933826 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f19600a2c80 0x7f19600a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T00:15:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.193+0000 7f1974ff9640 1 -- 172.21.15.27:0/346933826 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1970014030 con 0x7f1978102250 2024-04-25T00:15:38.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.381+0000 7f197df63640 1 -- 172.21.15.27:0/346933826 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f1964005580 con 0x7f1978102250 2024-04-25T00:15:38.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.385+0000 7f19777fe640 1 -- 172.21.15.27:0/346933826 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f19600a2c80 msgr2=0x7f19600a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2161554778 2024-04-25T00:15:38.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.385+0000 7f19777fe640 1 --2- 172.21.15.27:0/346933826 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f19600a2c80 0x7f19600a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T00:15:38.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.388+0000 7f1974ff9640 1 -- 172.21.15.27:0/346933826 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v27)=0 v27) ==== 133+0+0 (secure 0 0 0) 0x7f197009b1b0 con 0x7f1978102250 2024-04-25T00:15:38.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.391+0000 7f197df63640 1 -- 172.21.15.27:0/346933826 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f19600a2c80 msgr2=0x7f19600a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:15:38.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.391+0000 7f197df63640 1 --2- 172.21.15.27:0/346933826 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f19600a2c80 0x7f19600a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.391+0000 7f197df63640 1 -- 172.21.15.27:0/346933826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1978102250 msgr2=0x7f197819f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:38.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.391+0000 7f197df63640 1 --2- 172.21.15.27:0/346933826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1978102250 0x7f197819f050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f197000c710 tx=0x7f197000cbd0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.391+0000 7f197df63640 1 -- 172.21.15.27:0/346933826 shutdown_connections 2024-04-25T00:15:38.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.391+0000 7f197df63640 1 --2- 172.21.15.27:0/346933826 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f19600a2c80 0x7f19600a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.391+0000 7f197df63640 1 --2- 172.21.15.27:0/346933826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1978102250 0x7f197819f050 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.391+0000 7f197df63640 1 --2- 172.21.15.27:0/346933826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1978101890 0x7f1978197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.391+0000 7f197df63640 1 --2- 172.21.15.27:0/346933826 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19781006a0 0x7f1978197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.391+0000 7f197df63640 1 -- 172.21.15.27:0/346933826 >> 172.21.15.27:0/346933826 conn(0x7f19780fbc10 msgr2=0x7f19780fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:38.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.391+0000 7f197df63640 1 -- 172.21.15.27:0/346933826 shutdown_connections 2024-04-25T00:15:38.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:38.391+0000 7f197df63640 1 -- 172.21.15.27:0/346933826 wait complete. 2024-04-25T00:15:38.402 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:38 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:38.114+0000 7fa62c066200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:15:38.402 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:38 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:38.224+0000 7fa62c066200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T00:15:38.403 INFO:teuthology.run_tasks:Running task kclient... 2024-04-25T00:15:38.419 INFO:tasks.kclient:Mounting kernel clients... 2024-04-25T00:15:38.419 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-25T00:15:38.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:15:38.420 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-04-25T00:15:38.435 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-25T00:15:38.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:15:38.436 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T00:15:38.492 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:38 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:38.082+0000 7f2c46372200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:15:38.492 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:38 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:38.183+0000 7f2c46372200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T00:15:38.498 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:38 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:38.081+0000 7f5d23980200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:15:38.498 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:38 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:38.184+0000 7f5d23980200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T00:15:38.498 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:38 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:38.497+0000 7f5d23980200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T00:15:38.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:15:38.502 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-04-25T00:15:38.607 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-25T00:15:38.607 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T00:15:38.607 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T00:15:38.607 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-25T00:15:38.692 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:38 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:38.555+0000 7fa62c066200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T00:15:38.692 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:38 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:38.648+0000 7fa62c066200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T00:15:38.753 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:38 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:38.491+0000 7f2c46372200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T00:15:38.753 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:38 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:38.576+0000 7f2c46372200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T00:15:38.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:38 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2033700156' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T00:15:38.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:38 smithi139 ceph-mon[32833]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T00:15:38.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:38 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1914966992' entity='client.admin' 2024-04-25T00:15:38.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:38 smithi139 ceph-mon[32833]: from='client.? ' entity='client.admin' 2024-04-25T00:15:38.777 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:38 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:38.581+0000 7f5d23980200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T00:15:38.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:38 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2033700156' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T00:15:38.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:38 smithi153 ceph-mon[32896]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T00:15:38.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:38 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1914966992' entity='client.admin' 2024-04-25T00:15:38.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:38 smithi153 ceph-mon[32896]: from='client.? ' entity='client.admin' 2024-04-25T00:15:38.866 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:15:38.988 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:38 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2033700156' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T00:15:38.988 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:38 smithi027 ceph-mon[29208]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T00:15:38.989 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:38 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1914966992' entity='client.admin' 2024-04-25T00:15:38.989 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:38 smithi027 ceph-mon[29208]: from='client.? ' entity='client.admin' 2024-04-25T00:15:40.027 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:39 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:39.759+0000 7f5d23980200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T00:15:40.027 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:39 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:39.928+0000 7f5d23980200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T00:15:40.067 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:39 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:39.916+0000 7fa62c066200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T00:15:40.096 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:39 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:39.756+0000 7f2c46372200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T00:15:40.096 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:39 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:39.925+0000 7f2c46372200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T00:15:40.384 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:40 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:40.095+0000 7f2c46372200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T00:15:40.389 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:40 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:40.100+0000 7f5d23980200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T00:15:40.392 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:40 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:40.097+0000 7fa62c066200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T00:15:40.392 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:40 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:40.279+0000 7fa62c066200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T00:15:40.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.625+0000 7f3ae1134700 1 -- 172.21.15.27:0/4279487295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3adc1071a0 msgr2=0x7f3adc107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:40.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.625+0000 7f3ae1134700 1 --2- 172.21.15.27:0/4279487295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3adc1071a0 0x7f3adc107580 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3acc009990 tx=0x7f3acc009ca0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.625+0000 7f3ae1134700 1 -- 172.21.15.27:0/4279487295 shutdown_connections 2024-04-25T00:15:40.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.625+0000 7f3ae1134700 1 --2- 172.21.15.27:0/4279487295 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3adc101af0 0x7f3adc10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.625+0000 7f3ae1134700 1 --2- 172.21.15.27:0/4279487295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3adc1071a0 0x7f3adc107580 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.625+0000 7f3ae1134700 1 --2- 172.21.15.27:0/4279487295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3adc101130 0x7f3adc1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.625+0000 7f3ae1134700 1 -- 172.21.15.27:0/4279487295 >> 172.21.15.27:0/4279487295 conn(0x7f3adc0780e0 msgr2=0x7f3adc0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:40.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.625+0000 7f3ae1134700 1 -- 172.21.15.27:0/4279487295 shutdown_connections 2024-04-25T00:15:40.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.625+0000 7f3ae1134700 1 -- 172.21.15.27:0/4279487295 wait complete. 2024-04-25T00:15:40.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.626+0000 7f3ae1134700 1 Processor -- start 2024-04-25T00:15:40.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.626+0000 7f3ae1134700 1 -- start start 2024-04-25T00:15:40.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3ae1134700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3adc101130 0x7f3adc1998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3ae1134700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3adc101af0 0x7f3adc199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3ae1134700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3adc1071a0 0x7f3adc1a4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3ae1134700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3adc111440 con 0x7f3adc101130 2024-04-25T00:15:40.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3ae1134700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3adc1112c0 con 0x7f3adc1071a0 2024-04-25T00:15:40.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3ae1134700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3adc1115c0 con 0x7f3adc101af0 2024-04-25T00:15:40.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3adad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3adc101130 0x7f3adc1998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:40.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3adad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3adc101130 0x7f3adc1998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35410/0 (socket says 172.21.15.27:35410) 2024-04-25T00:15:40.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3adad9d700 1 -- 172.21.15.27:0/2220382946 learned_addr learned my addr 172.21.15.27:0/2220382946 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:40.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3adb59e700 1 --2- 172.21.15.27:0/2220382946 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3adc1071a0 0x7f3adc1a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:40.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3ada59c700 1 --2- 172.21.15.27:0/2220382946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3adc101af0 0x7f3adc199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:40.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3adad9d700 1 -- 172.21.15.27:0/2220382946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3adc101af0 msgr2=0x7f3adc199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:40.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3adad9d700 1 --2- 172.21.15.27:0/2220382946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3adc101af0 0x7f3adc199de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3adad9d700 1 -- 172.21.15.27:0/2220382946 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3adc1071a0 msgr2=0x7f3adc1a4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:40.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3adad9d700 1 --2- 172.21.15.27:0/2220382946 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3adc1071a0 0x7f3adc1a4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3adad9d700 1 -- 172.21.15.27:0/2220382946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3adc1a4e60 con 0x7f3adc101130 2024-04-25T00:15:40.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3adb59e700 1 --2- 172.21.15.27:0/2220382946 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3adc1071a0 0x7f3adc1a4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:40.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.627+0000 7f3adad9d700 1 --2- 172.21.15.27:0/2220382946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3adc101130 0x7f3adc1998a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3acc000c00 tx=0x7f3acc00bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:40.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.628+0000 7f3ac3fff700 1 -- 172.21.15.27:0/2220382946 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3acc027070 con 0x7f3adc101130 2024-04-25T00:15:40.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.628+0000 7f3ae1134700 1 -- 172.21.15.27:0/2220382946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3adc1a5030 con 0x7f3adc101130 2024-04-25T00:15:40.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.628+0000 7f3ac3fff700 1 -- 172.21.15.27:0/2220382946 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3acc003d70 con 0x7f3adc101130 2024-04-25T00:15:40.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.628+0000 7f3ac3fff700 1 -- 172.21.15.27:0/2220382946 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3acc00f460 con 0x7f3adc101130 2024-04-25T00:15:40.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.628+0000 7f3ae1134700 1 -- 172.21.15.27:0/2220382946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3adc1a5480 con 0x7f3adc101130 2024-04-25T00:15:40.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.629+0000 7f3ae1134700 1 -- 172.21.15.27:0/2220382946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3adc068f20 con 0x7f3adc101130 2024-04-25T00:15:40.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.630+0000 7f3ac3fff700 1 -- 172.21.15.27:0/2220382946 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f3acc02a070 con 0x7f3adc101130 2024-04-25T00:15:40.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.631+0000 7f3ac3fff700 1 --2- 172.21.15.27:0/2220382946 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3ac40a2cf0 0x7f3ac40a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.631+0000 7f3ada59c700 1 -- 172.21.15.27:0/2220382946 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3ac40a2cf0 msgr2=0x7f3ac40a51b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2161554778 2024-04-25T00:15:40.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.631+0000 7f3ac3fff700 1 -- 172.21.15.27:0/2220382946 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3acc0d01f0 con 0x7f3adc101130 2024-04-25T00:15:40.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.631+0000 7f3ada59c700 1 --2- 172.21.15.27:0/2220382946 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3ac40a2cf0 0x7f3ac40a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T00:15:40.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.635+0000 7f3ac3fff700 1 -- 172.21.15.27:0/2220382946 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3acc013070 con 0x7f3adc101130 2024-04-25T00:15:40.680 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:40 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:40.383+0000 7f2c46372200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T00:15:40.681 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:40 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:40.475+0000 7f2c46372200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T00:15:40.685 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:40 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:40.388+0000 7f5d23980200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T00:15:40.686 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:40 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:40.481+0000 7f5d23980200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T00:15:40.686 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:40 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:40.685+0000 7f5d23980200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T00:15:40.688 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:40 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:40.587+0000 7fa62c066200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T00:15:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.832+0000 7f3ada59c700 1 -- 172.21.15.27:0/2220382946 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3ac40a2cf0 msgr2=0x7f3ac40a51b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2161554778 2024-04-25T00:15:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.832+0000 7f3ada59c700 1 --2- 172.21.15.27:0/2220382946 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3ac40a2cf0 0x7f3ac40a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T00:15:40.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.889+0000 7f3ae1134700 1 -- 172.21.15.27:0/2220382946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f3adc02dcf0 con 0x7f3adc101130 2024-04-25T00:15:40.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.890+0000 7f3ac3fff700 1 -- 172.21.15.27:0/2220382946 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+88 (secure 0 0 0) 0x7f3acc09cdf0 con 0x7f3adc101130 2024-04-25T00:15:40.890 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T00:15:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.892+0000 7f3ae1134700 1 -- 172.21.15.27:0/2220382946 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3ac40a2cf0 msgr2=0x7f3ac40a51b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:15:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.892+0000 7f3ae1134700 1 --2- 172.21.15.27:0/2220382946 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3ac40a2cf0 0x7f3ac40a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.892+0000 7f3ae1134700 1 -- 172.21.15.27:0/2220382946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3adc101130 msgr2=0x7f3adc1998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.892+0000 7f3ae1134700 1 --2- 172.21.15.27:0/2220382946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3adc101130 0x7f3adc1998a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3acc000c00 tx=0x7f3acc00bab0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.892+0000 7f3ae1134700 1 -- 172.21.15.27:0/2220382946 shutdown_connections 2024-04-25T00:15:40.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.892+0000 7f3ae1134700 1 --2- 172.21.15.27:0/2220382946 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f3ac40a2cf0 0x7f3ac40a51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.892+0000 7f3ae1134700 1 --2- 172.21.15.27:0/2220382946 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3adc1071a0 0x7f3adc1a4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.892+0000 7f3ae1134700 1 --2- 172.21.15.27:0/2220382946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3adc101130 0x7f3adc1998a0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.892+0000 7f3ae1134700 1 --2- 172.21.15.27:0/2220382946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3adc101af0 0x7f3adc199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.892+0000 7f3ae1134700 1 -- 172.21.15.27:0/2220382946 >> 172.21.15.27:0/2220382946 conn(0x7f3adc0780e0 msgr2=0x7f3adc0ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:40.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.892+0000 7f3ae1134700 1 -- 172.21.15.27:0/2220382946 shutdown_connections 2024-04-25T00:15:40.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:40.892+0000 7f3ae1134700 1 -- 172.21.15.27:0/2220382946 wait complete. 2024-04-25T00:15:40.941 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:40 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:40.688+0000 7fa62c066200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T00:15:40.942 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:40 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:40.902+0000 7fa62c066200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T00:15:40.942 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:40 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:40.878+0000 7f5d23980200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:15:40.942 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:40 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:40.679+0000 7f2c46372200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T00:15:40.942 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:40 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:40.873+0000 7f2c46372200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:15:41.192 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:41 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:41.108+0000 7fa62c066200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:15:41.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:40 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2220382946' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T00:15:41.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:40 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2220382946' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T00:15:41.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:40 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2220382946' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T00:15:41.712 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T00:15:41.712 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T00:15:41.712 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-04-25T00:15:41.712 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T00:15:41.712 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T00:15:41.712 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T00:15:41.712 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T00:15:41.712 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T00:15:41.712 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T00:15:41.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:15:41.713 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-04-25T00:15:41.730 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T00:15:41.730 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T00:15:41.730 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T00:15:41.730 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-25T00:15:41.730 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-04-25T00:15:41.730 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-25T00:15:41.730 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T00:15:41.730 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-04-25T00:15:41.730 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-04-25T00:15:41.730 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T00:15:41.731 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-04-25T00:15:41.731 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-04-25T00:15:41.731 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T00:15:41.731 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-04-25T00:15:41.731 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-04-25T00:15:41.731 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T00:15:41.731 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42044sec preferred_lft 42044sec 2024-04-25T00:15:41.731 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-04-25T00:15:41.731 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-25T00:15:41.731 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T00:15:41.731 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-04-25T00:15:41.731 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-04-25T00:15:41.732 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T00:15:41.732 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:15:41.732 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T00:15:41.732 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add name ceph-brx type bridge 2024-04-25T00:15:41.732 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr flush dev ceph-brx 2024-04-25T00:15:41.732 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set ceph-brx up 2024-04-25T00:15:41.732 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T00:15:41.732 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T00:15:41.753 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:41 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:41.474+0000 7f2c46372200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T00:15:41.776 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:41 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:41.478+0000 7f5d23980200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T00:15:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:15:41.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:15:41.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:15:41.874 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T00:15:41.898 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-04-25T00:15:41.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:15:41.900 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-04-25T00:15:41.942 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:41 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:41.741+0000 7fa62c066200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T00:15:41.957 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-04-25T00:15:41.957 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T00:15:41.957 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T00:15:41.957 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T00:15:41.957 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T00:15:41.957 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:15:41.957 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T00:15:41.957 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T00:15:41.957 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T00:15:41.958 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T00:15:41.958 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T00:15:42.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:15:42.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:15:42.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:15:42.085 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-04-25T00:15:42.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:15:42.101 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2024-04-25T00:15:42.156 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:15:42.156 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T00:15:42.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T00:15:42.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T00:15:42.157 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T00:15:42.192 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:41 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:41.895+0000 7f2c46372200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T00:15:42.192 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:41 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:41.991+0000 7f2c46372200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T00:15:42.203 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:41 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:41.903+0000 7f5d23980200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T00:15:42.203 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:42 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:42.000+0000 7f5d23980200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T00:15:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:15:42.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:15:42.276 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T00:15:42.276 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:15:42.277 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T00:15:42.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T00:15:42.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T00:15:42.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T00:15:42.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T00:15:42.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T00:15:42.277 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T00:15:42.312 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:42 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:42.206+0000 7fa62c066200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T00:15:42.312 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:42 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:42.312+0000 7fa62c066200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T00:15:42.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:15:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:15:42.392 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:15:42.393 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T00:15:42.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-04-25T00:15:42.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T00:15:42.393 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T00:15:42.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:15:42.503 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:42 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:42.191+0000 7f2c46372200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T00:15:42.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:15:42.512 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T00:15:42.512 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T00:15:42.526 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:42 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:42.202+0000 7f5d23980200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T00:15:42.537 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T00:15:42.537 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T00:15:42.537 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T00:15:42.593 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-25T00:15:42.593 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-25T00:15:42.593 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/c29414da-1300-48ce-9002-67b724424f7f 2024-04-25T00:15:42.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:15:42.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/c29414da-1300-48ce-9002-67b724424f7f /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-04-25T00:15:42.654 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:42 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:42.536+0000 7fa62c066200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T00:15:42.676 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-04-25T00:15:42.677 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-04-25T00:15:42.677 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-04-25T00:15:42.809 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-04-25T00:15:42.809 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-04-25T00:15:42.809 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-04-25T00:15:42.809 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.139:3300,172.21.15.153:3300" 2024-04-25T00:15:42.809 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:3300,172.21.15.139:3300,172.21.15.153:3300:/volumes/qa/sv_1/c29414da-1300-48ce-9002-67b724424f7f 2024-04-25T00:15:42.809 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,key=0,fsid=518857f8-0297-11ef-bc93-c7b262605968" will pass to kernel 2024-04-25T00:15:42.810 INFO:teuthology.orchestra.run.smithi027.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T00:15:42.810 INFO:teuthology.orchestra.run.smithi027.stdout: You just mounted a file system that supports labels which does not 2024-04-25T00:15:42.810 INFO:teuthology.orchestra.run.smithi027.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T00:15:42.810 INFO:teuthology.orchestra.run.smithi027.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T00:15:42.810 INFO:teuthology.orchestra.run.smithi027.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T00:15:42.812 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T00:15:42.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:15:42.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T00:15:42.881 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T00:15:42.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T00:15:42.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:15:42.988 INFO:teuthology.orchestra.run.smithi027.stdout:client15582 2024-04-25T00:15:42.989 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:15:42.989 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T00:15:43.003 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:42 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:42.716+0000 7f2c46372200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T00:15:43.003 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:42 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:42.809+0000 7f2c46372200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T00:15:43.005 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T00:15:43.026 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:42 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:42.727+0000 7f5d23980200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T00:15:43.026 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:42 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:42.819+0000 7f5d23980200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T00:15:43.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T00:15:43.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:15:43.103 INFO:teuthology.orchestra.run.smithi027.stdout:client15582 2024-04-25T00:15:43.103 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T00:15:43.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T00:15:43.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:15:43.161 INFO:teuthology.orchestra.run.smithi027.stdout:client15582 2024-04-25T00:15:43.161 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/518857f8-0297-11ef-bc93-c7b262605968.client15582/status') 2024-04-25T00:15:43.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:15:43.221 INFO:teuthology.orchestra.run.smithi027.stdout:instance: client.15582 (3)192.168.0.1:0/4137657918 2024-04-25T00:15:43.221 INFO:teuthology.orchestra.run.smithi027.stdout:blocklisted: false 2024-04-25T00:15:43.221 INFO:teuthology.orchestra.run.smithi027.stderr:0+1 records in 2024-04-25T00:15:43.221 INFO:teuthology.orchestra.run.smithi027.stderr:0+1 records out 2024-04-25T00:15:43.221 INFO:teuthology.orchestra.run.smithi027.stderr:70 bytes copied, 2.519e-05 s, 2.8 MB/s 2024-04-25T00:15:43.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:15:43.229 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T00:15:43.238 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T00:15:43.238 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T00:15:43.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-25T00:15:43.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.343+0000 7fb1cf479640 1 -- 172.21.15.27:0/617344534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1c81006a0 msgr2=0x7fb1c8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:43.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.343+0000 7fb1cf479640 1 --2- 172.21.15.27:0/617344534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1c81006a0 0x7fb1c8100ac0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4009900 tx=0x7fb1c402a410 comp rx=0 tx=0).stop 2024-04-25T00:15:43.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.343+0000 7fb1cf479640 1 -- 172.21.15.27:0/617344534 shutdown_connections 2024-04-25T00:15:43.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.343+0000 7fb1cf479640 1 --2- 172.21.15.27:0/617344534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1c8102250 0x7fb1c8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.343+0000 7fb1cf479640 1 --2- 172.21.15.27:0/617344534 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c8101890 0x7fb1c8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.343+0000 7fb1cf479640 1 --2- 172.21.15.27:0/617344534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1c81006a0 0x7fb1c8100ac0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.343+0000 7fb1cf479640 1 -- 172.21.15.27:0/617344534 >> 172.21.15.27:0/617344534 conn(0x7fb1c80fbc10 msgr2=0x7fb1c80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:43.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.343+0000 7fb1cf479640 1 -- 172.21.15.27:0/617344534 shutdown_connections 2024-04-25T00:15:43.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.343+0000 7fb1cf479640 1 -- 172.21.15.27:0/617344534 wait complete. 2024-04-25T00:15:43.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.343+0000 7fb1cf479640 1 Processor -- start 2024-04-25T00:15:43.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cf479640 1 -- start start 2024-04-25T00:15:43.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cf479640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1c81006a0 0x7fb1c8197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:43.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cf479640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1c8101890 0x7fb1c8197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:43.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cf479640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c8102250 0x7fb1c819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:43.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cf479640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c810d440 con 0x7fb1c81006a0 2024-04-25T00:15:43.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cf479640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c810d2c0 con 0x7fb1c8102250 2024-04-25T00:15:43.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cf479640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c810d5c0 con 0x7fb1c8101890 2024-04-25T00:15:43.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cd215640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1c81006a0 0x7fb1c8197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:43.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cd215640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1c81006a0 0x7fb1c8197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35420/0 (socket says 172.21.15.27:35420) 2024-04-25T00:15:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cd215640 1 -- 172.21.15.27:0/2066651133 learned_addr learned my addr 172.21.15.27:0/2066651133 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cca14640 1 --2- 172.21.15.27:0/2066651133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1c8101890 0x7fb1c8197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cda16640 1 --2- 172.21.15.27:0/2066651133 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c8102250 0x7fb1c819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cd215640 1 -- 172.21.15.27:0/2066651133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1c8101890 msgr2=0x7fb1c8197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cd215640 1 --2- 172.21.15.27:0/2066651133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1c8101890 0x7fb1c8197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cd215640 1 -- 172.21.15.27:0/2066651133 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c8102250 msgr2=0x7fb1c819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cd215640 1 --2- 172.21.15.27:0/2066651133 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c8102250 0x7fb1c819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cd215640 1 -- 172.21.15.27:0/2066651133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c819f660 con 0x7fb1c81006a0 2024-04-25T00:15:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.344+0000 7fb1cd215640 1 --2- 172.21.15.27:0/2066651133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1c81006a0 0x7fb1c8197a40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4031610 tx=0x7fb1c4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.345+0000 7fb1be7fc640 1 -- 172.21.15.27:0/2066651133 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c403f030 con 0x7fb1c81006a0 2024-04-25T00:15:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.345+0000 7fb1cf479640 1 -- 172.21.15.27:0/2066651133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1c819f8f0 con 0x7fb1c81006a0 2024-04-25T00:15:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.345+0000 7fb1be7fc640 1 -- 172.21.15.27:0/2066651133 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1c4003ad0 con 0x7fb1c81006a0 2024-04-25T00:15:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.345+0000 7fb1be7fc640 1 -- 172.21.15.27:0/2066651133 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4035070 con 0x7fb1c81006a0 2024-04-25T00:15:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.345+0000 7fb1cf479640 1 -- 172.21.15.27:0/2066651133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1c819fe30 con 0x7fb1c81006a0 2024-04-25T00:15:43.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.346+0000 7fb1cf479640 1 -- 172.21.15.27:0/2066651133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1c81079c0 con 0x7fb1c81006a0 2024-04-25T00:15:43.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.347+0000 7fb1be7fc640 1 -- 172.21.15.27:0/2066651133 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fb1c4042030 con 0x7fb1c81006a0 2024-04-25T00:15:43.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.347+0000 7fb1be7fc640 1 --2- 172.21.15.27:0/2066651133 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb1b40a2c80 0x7fb1b40a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:43.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.347+0000 7fb1cca14640 1 -- 172.21.15.27:0/2066651133 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb1b40a2c80 msgr2=0x7fb1b40a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2161554778 2024-04-25T00:15:43.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.347+0000 7fb1be7fc640 1 -- 172.21.15.27:0/2066651133 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb1c40ef9c0 con 0x7fb1c81006a0 2024-04-25T00:15:43.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.347+0000 7fb1cca14640 1 --2- 172.21.15.27:0/2066651133 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb1b40a2c80 0x7fb1b40a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T00:15:43.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.352+0000 7fb1be7fc640 1 -- 172.21.15.27:0/2066651133 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb1c40f1e20 con 0x7fb1c81006a0 2024-04-25T00:15:43.399 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:43 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:43.051+0000 7f2c46372200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T00:15:43.399 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:43 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:43.186+0000 7f2c46372200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T00:15:43.399 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:43 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:43.281+0000 7f2c46372200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T00:15:43.399 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:43 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:43.398+0000 7f2c46372200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T00:15:43.406 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:43 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:43.061+0000 7f5d23980200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T00:15:43.406 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:43 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:43.196+0000 7f5d23980200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T00:15:43.407 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:43 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:43.290+0000 7f5d23980200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T00:15:43.442 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:43 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:43.106+0000 7fa62c066200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T00:15:43.442 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:43 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:43.206+0000 7fa62c066200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T00:15:43.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.548+0000 7fb1cca14640 1 -- 172.21.15.27:0/2066651133 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb1b40a2c80 msgr2=0x7fb1b40a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2161554778 2024-04-25T00:15:43.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.548+0000 7fb1cca14640 1 --2- 172.21.15.27:0/2066651133 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb1b40a2c80 0x7fb1b40a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T00:15:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.586+0000 7fb1cf479640 1 -- 172.21.15.27:0/2066651133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fb1c802d170 con 0x7fb1c81006a0 2024-04-25T00:15:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.638+0000 7fb1be7fc640 1 -- 172.21.15.27:0/2066651133 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v29) ==== 118+0+0 (secure 0 0 0) 0x7fb1c40bbf40 con 0x7fb1c81006a0 2024-04-25T00:15:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.641+0000 7fb1cf479640 1 -- 172.21.15.27:0/2066651133 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb1b40a2c80 msgr2=0x7fb1b40a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:15:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.641+0000 7fb1cf479640 1 --2- 172.21.15.27:0/2066651133 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb1b40a2c80 0x7fb1b40a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.641+0000 7fb1cf479640 1 -- 172.21.15.27:0/2066651133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1c81006a0 msgr2=0x7fb1c8197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.641+0000 7fb1cf479640 1 --2- 172.21.15.27:0/2066651133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1c81006a0 0x7fb1c8197a40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4031610 tx=0x7fb1c4004270 comp rx=0 tx=0).stop 2024-04-25T00:15:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.642+0000 7fb1cf479640 1 -- 172.21.15.27:0/2066651133 shutdown_connections 2024-04-25T00:15:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.642+0000 7fb1cf479640 1 --2- 172.21.15.27:0/2066651133 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7fb1b40a2c80 0x7fb1b40a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.642+0000 7fb1cf479640 1 --2- 172.21.15.27:0/2066651133 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c8102250 0x7fb1c819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.642+0000 7fb1cf479640 1 --2- 172.21.15.27:0/2066651133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1c8101890 0x7fb1c8197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.642+0000 7fb1cf479640 1 --2- 172.21.15.27:0/2066651133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1c81006a0 0x7fb1c8197a40 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.642+0000 7fb1cf479640 1 -- 172.21.15.27:0/2066651133 >> 172.21.15.27:0/2066651133 conn(0x7fb1c80fbc10 msgr2=0x7fb1c8104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:43.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.642+0000 7fb1cf479640 1 -- 172.21.15.27:0/2066651133 shutdown_connections 2024-04-25T00:15:43.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.642+0000 7fb1cf479640 1 -- 172.21.15.27:0/2066651133 wait complete. 2024-04-25T00:15:43.653 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-25T00:15:43.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:43 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2066651133' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-25T00:15:43.753 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:43 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:43.492+0000 7f2c46372200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T00:15:43.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.758+0000 7f4fcb8ca640 1 -- 172.21.15.27:0/2247512116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc41018b0 msgr2=0x7f4fc4101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:43.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.758+0000 7f4fcb8ca640 1 --2- 172.21.15.27:0/2247512116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc41018b0 0x7f4fc4101d30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4009900 tx=0x7f4fb402a410 comp rx=0 tx=0).stop 2024-04-25T00:15:43.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.758+0000 7f4fcb8ca640 1 -- 172.21.15.27:0/2247512116 shutdown_connections 2024-04-25T00:15:43.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.758+0000 7f4fcb8ca640 1 --2- 172.21.15.27:0/2247512116 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4fc4102270 0x7f4fc4108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.758+0000 7f4fcb8ca640 1 --2- 172.21.15.27:0/2247512116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc41018b0 0x7f4fc4101d30 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.758+0000 7f4fcb8ca640 1 --2- 172.21.15.27:0/2247512116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fc41006c0 0x7f4fc4100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.758+0000 7f4fcb8ca640 1 -- 172.21.15.27:0/2247512116 >> 172.21.15.27:0/2247512116 conn(0x7f4fc40fbc50 msgr2=0x7f4fc40fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.758+0000 7f4fcb8ca640 1 -- 172.21.15.27:0/2247512116 shutdown_connections 2024-04-25T00:15:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.758+0000 7f4fcb8ca640 1 -- 172.21.15.27:0/2247512116 wait complete. 2024-04-25T00:15:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.759+0000 7f4fcb8ca640 1 Processor -- start 2024-04-25T00:15:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.759+0000 7f4fcb8ca640 1 -- start start 2024-04-25T00:15:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.759+0000 7f4fcb8ca640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4fc41006c0 0x7f4fc4196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.759+0000 7f4fcb8ca640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fc41018b0 0x7f4fc4196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.759+0000 7f4fcb8ca640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc4102270 0x7f4fc419df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.759+0000 7f4fcb8ca640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc410c340 con 0x7f4fc4102270 2024-04-25T00:15:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.759+0000 7f4fcb8ca640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc410c1c0 con 0x7f4fc41006c0 2024-04-25T00:15:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.759+0000 7f4fcb8ca640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc410c4c0 con 0x7f4fc41018b0 2024-04-25T00:15:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.759+0000 7f4fc9e67640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc4102270 0x7f4fc419df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.759+0000 7f4fc9666640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4fc41006c0 0x7f4fc4196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.759+0000 7f4fc8e65640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fc41018b0 0x7f4fc4196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.759+0000 7f4fc9666640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4fc41006c0 0x7f4fc4196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:55112/0 (socket says 172.21.15.27:55112) 2024-04-25T00:15:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.759+0000 7f4fc9e67640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc4102270 0x7f4fc419df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35444/0 (socket says 172.21.15.27:35444) 2024-04-25T00:15:43.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.759+0000 7f4fc9666640 1 -- 172.21.15.27:0/2080796518 learned_addr learned my addr 172.21.15.27:0/2080796518 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:43.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.760+0000 7f4fc8e65640 1 -- 172.21.15.27:0/2080796518 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4fc41006c0 msgr2=0x7f4fc4196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:43.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.760+0000 7f4fc8e65640 1 --2- 172.21.15.27:0/2080796518 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4fc41006c0 0x7f4fc4196980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.760+0000 7f4fc8e65640 1 -- 172.21.15.27:0/2080796518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc4102270 msgr2=0x7f4fc419df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:43.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.760+0000 7f4fc8e65640 1 --2- 172.21.15.27:0/2080796518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc4102270 0x7f4fc419df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.760+0000 7f4fc8e65640 1 -- 172.21.15.27:0/2080796518 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc419e5a0 con 0x7f4fc41018b0 2024-04-25T00:15:43.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.760+0000 7f4fc8e65640 1 --2- 172.21.15.27:0/2080796518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fc41018b0 0x7f4fc4196ec0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4031440 tx=0x7f4fb4031b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:43.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.760+0000 7f4fba7fc640 1 -- 172.21.15.27:0/2080796518 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4035050 con 0x7f4fc41018b0 2024-04-25T00:15:43.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.760+0000 7f4fcb8ca640 1 -- 172.21.15.27:0/2080796518 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc40699a0 con 0x7f4fc41018b0 2024-04-25T00:15:43.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.760+0000 7f4fba7fc640 1 -- 172.21.15.27:0/2080796518 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fb4003ad0 con 0x7f4fc41018b0 2024-04-25T00:15:43.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.760+0000 7f4fba7fc640 1 -- 172.21.15.27:0/2080796518 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4038b80 con 0x7f4fc41018b0 2024-04-25T00:15:43.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.760+0000 7f4fcb8ca640 1 -- 172.21.15.27:0/2080796518 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc4069e90 con 0x7f4fc41018b0 2024-04-25T00:15:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.761+0000 7f4fcb8ca640 1 -- 172.21.15.27:0/2080796518 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fc404f500 con 0x7f4fc41018b0 2024-04-25T00:15:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.762+0000 7f4fba7fc640 1 -- 172.21.15.27:0/2080796518 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f4fb40035f0 con 0x7f4fc41018b0 2024-04-25T00:15:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.763+0000 7f4fba7fc640 1 --2- 172.21.15.27:0/2080796518 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4fb00a2c80 0x7f4fb00a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.763+0000 7f4fc9666640 1 -- 172.21.15.27:0/2080796518 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4fb00a2c80 msgr2=0x7f4fb00a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2161554778 2024-04-25T00:15:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.763+0000 7f4fba7fc640 1 -- 172.21.15.27:0/2080796518 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f4fb40f14d0 con 0x7f4fc41018b0 2024-04-25T00:15:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.763+0000 7f4fc9666640 1 --2- 172.21.15.27:0/2080796518 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4fb00a2c80 0x7f4fb00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T00:15:43.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.766+0000 7f4fba7fc640 1 -- 172.21.15.27:0/2080796518 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4fb40be0a0 con 0x7f4fc41018b0 2024-04-25T00:15:43.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:43 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2066651133' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-25T00:15:43.776 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:43 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:43.406+0000 7f5d23980200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T00:15:43.776 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:43 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:43.501+0000 7f5d23980200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T00:15:43.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:43 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2066651133' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-25T00:15:43.847 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:43 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:43.473+0000 7fa62c066200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T00:15:43.847 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:43 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:43.615+0000 7fa62c066200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T00:15:43.847 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:43 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:43.720+0000 7fa62c066200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T00:15:43.847 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:43 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:43.846+0000 7fa62c066200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T00:15:43.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.964+0000 7f4fc9666640 1 -- 172.21.15.27:0/2080796518 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4fb00a2c80 msgr2=0x7f4fb00a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2161554778 2024-04-25T00:15:43.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.964+0000 7f4fc9666640 1 --2- 172.21.15.27:0/2080796518 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4fb00a2c80 0x7f4fb00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T00:15:43.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:43.982+0000 7f4fcb8ca640 1 -- 172.21.15.27:0/2080796518 --> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f4fc406a0f0 con 0x7f4fb00a2c80 2024-04-25T00:15:44.192 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:43 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:43.945+0000 7fa62c066200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T00:15:44.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:44.365+0000 7f4fc9666640 1 -- 172.21.15.27:0/2080796518 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4fb00a2c80 msgr2=0x7f4fb00a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2161554778 2024-04-25T00:15:44.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:44.365+0000 7f4fc9666640 1 --2- 172.21.15.27:0/2080796518 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4fb00a2c80 0x7f4fb00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T00:15:44.647 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:44 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:44.307+0000 7f2c46372200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T00:15:44.648 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:15:44 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y[34214]: 2024-04-25T00:15:44.523+0000 7f2c46372200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T00:15:44.648 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:44 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:44.317+0000 7f5d23980200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T00:15:44.648 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:15:44 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z[34778]: 2024-04-25T00:15:44.532+0000 7f5d23980200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T00:15:44.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:44.661+0000 7f4fba7fc640 1 -- 172.21.15.27:0/2080796518 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 20) ==== 137850+0+0 (secure 0 0 0) 0x7f4fb40ba010 con 0x7f4fc41018b0 2024-04-25T00:15:44.942 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:44 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:44.804+0000 7fa62c066200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T00:15:44.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:44 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2066651133' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-25T00:15:44.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:44 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:15:44.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:44 smithi027 ceph-mon[29208]: Standby manager daemon y restarted 2024-04-25T00:15:44.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:44 smithi027 ceph-mon[29208]: Standby manager daemon y started 2024-04-25T00:15:44.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:44 smithi027 ceph-mon[29208]: from='mgr.? 172.21.15.139:0/3523014399' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T00:15:44.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:44 smithi027 ceph-mon[29208]: from='mgr.? 172.21.15.139:0/3523014399' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T00:15:44.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:44 smithi027 ceph-mon[29208]: from='mgr.? 172.21.15.139:0/3523014399' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T00:15:44.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:44 smithi027 ceph-mon[29208]: from='mgr.? 172.21.15.139:0/3523014399' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T00:15:44.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:44 smithi027 ceph-mon[29208]: Standby manager daemon z restarted 2024-04-25T00:15:44.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:44 smithi027 ceph-mon[29208]: Standby manager daemon z started 2024-04-25T00:15:44.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:44 smithi027 ceph-mon[29208]: from='mgr.? 172.21.15.153:0/1162186335' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T00:15:44.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:44 smithi027 ceph-mon[29208]: from='mgr.? 172.21.15.153:0/1162186335' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T00:15:44.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:44 smithi027 ceph-mon[29208]: from='mgr.? 172.21.15.153:0/1162186335' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T00:15:44.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:44 smithi027 ceph-mon[29208]: from='mgr.? 172.21.15.153:0/1162186335' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T00:15:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:44 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2066651133' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-25T00:15:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:44 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:15:45.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:44 smithi139 ceph-mon[32833]: Standby manager daemon y restarted 2024-04-25T00:15:45.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:44 smithi139 ceph-mon[32833]: Standby manager daemon y started 2024-04-25T00:15:45.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:44 smithi139 ceph-mon[32833]: from='mgr.? 172.21.15.139:0/3523014399' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T00:15:45.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:44 smithi139 ceph-mon[32833]: from='mgr.? 172.21.15.139:0/3523014399' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T00:15:45.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:44 smithi139 ceph-mon[32833]: from='mgr.? 172.21.15.139:0/3523014399' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T00:15:45.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:44 smithi139 ceph-mon[32833]: from='mgr.? 172.21.15.139:0/3523014399' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T00:15:45.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:44 smithi139 ceph-mon[32833]: Standby manager daemon z restarted 2024-04-25T00:15:45.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:44 smithi139 ceph-mon[32833]: Standby manager daemon z started 2024-04-25T00:15:45.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:44 smithi139 ceph-mon[32833]: from='mgr.? 172.21.15.153:0/1162186335' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T00:15:45.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:44 smithi139 ceph-mon[32833]: from='mgr.? 172.21.15.153:0/1162186335' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T00:15:45.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:44 smithi139 ceph-mon[32833]: from='mgr.? 172.21.15.153:0/1162186335' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T00:15:45.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:44 smithi139 ceph-mon[32833]: from='mgr.? 172.21.15.153:0/1162186335' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T00:15:45.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:44 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2066651133' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-25T00:15:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:44 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:15:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:44 smithi153 ceph-mon[32896]: Standby manager daemon y restarted 2024-04-25T00:15:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:44 smithi153 ceph-mon[32896]: Standby manager daemon y started 2024-04-25T00:15:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:44 smithi153 ceph-mon[32896]: from='mgr.? 172.21.15.139:0/3523014399' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T00:15:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:44 smithi153 ceph-mon[32896]: from='mgr.? 172.21.15.139:0/3523014399' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T00:15:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:44 smithi153 ceph-mon[32896]: from='mgr.? 172.21.15.139:0/3523014399' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T00:15:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:44 smithi153 ceph-mon[32896]: from='mgr.? 172.21.15.139:0/3523014399' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T00:15:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:44 smithi153 ceph-mon[32896]: Standby manager daemon z restarted 2024-04-25T00:15:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:44 smithi153 ceph-mon[32896]: Standby manager daemon z started 2024-04-25T00:15:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:44 smithi153 ceph-mon[32896]: from='mgr.? 172.21.15.153:0/1162186335' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T00:15:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:44 smithi153 ceph-mon[32896]: from='mgr.? 172.21.15.153:0/1162186335' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T00:15:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:44 smithi153 ceph-mon[32896]: from='mgr.? 172.21.15.153:0/1162186335' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T00:15:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:44 smithi153 ceph-mon[32896]: from='mgr.? 172.21.15.153:0/1162186335' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T00:15:45.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:45.046+0000 7f4fba7fc640 1 -- 172.21.15.27:0/2080796518 <== mon.1 v2:172.21.15.153:3300/0 8 ==== mgrmap(e 21) ==== 137518+0+0 (secure 0 0 0) 0x7f4fb40eec90 con 0x7f4fc41018b0 2024-04-25T00:15:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:45.046+0000 7f4fba7fc640 1 -- 172.21.15.27:0/2080796518 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4fb00a2c80 msgr2=0x7f4fb00a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:15:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:45.046+0000 7f4fba7fc640 1 --2- 172.21.15.27:0/2080796518 >> [v2:172.21.15.27:6800/2161554778,v1:172.21.15.27:6801/2161554778] conn(0x7f4fb00a2c80 0x7f4fb00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:45.291 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:45.033+0000 7fa62c066200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T00:15:45.656 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:45.303+0000 7fa58e9f1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:15:45.656 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:45.303+0000 7fa58e9f1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:15:45.656 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:45.303+0000 7fa58e9f1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:15:45.656 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:45.303+0000 7fa58e9f1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:15:45.656 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:45.303+0000 7fa58e9f1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:15:45.656 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:45.303+0000 7fa58b9eb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:15:45.657 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:45.304+0000 7fa58b9eb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:15:45.657 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:45.304+0000 7fa58b9eb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:15:45.657 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:45.304+0000 7fa58b9eb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:15:45.657 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:45.304+0000 7fa58b9eb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:15:45.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-04-25T00:15:45.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: Active manager daemon x restarted 2024-04-25T00:15:45.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: Activating manager daemon x 2024-04-25T00:15:45.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T00:15:45.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: mgrmap e21: x(active, starting, since 0.0110945s), standbys: z, y 2024-04-25T00:15:45.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:15:45.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:15:45.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:15:45.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T00:15:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T00:15:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T00:15:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T00:15:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T00:15:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T00:15:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T00:15:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T00:15:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T00:15:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T00:15:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T00:15:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T00:15:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T00:15:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T00:15:45.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T00:15:45.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:15:45.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:15:45.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:15:45.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:15:45.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:15:45.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:15:45.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:15:45.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:15:45.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:15:45.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:15:45.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:15:45.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:15:45.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T00:15:45.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T00:15:45.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T00:15:45.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: Manager daemon x is now available 2024-04-25T00:15:45.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:15:45.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T00:15:45.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:45 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T00:15:46.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-04-25T00:15:46.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: Active manager daemon x restarted 2024-04-25T00:15:46.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: Activating manager daemon x 2024-04-25T00:15:46.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T00:15:46.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: mgrmap e21: x(active, starting, since 0.0110945s), standbys: z, y 2024-04-25T00:15:46.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:15:46.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:15:46.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:15:46.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T00:15:46.005 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T00:15:46.005 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T00:15:46.005 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T00:15:46.005 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T00:15:46.005 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T00:15:46.005 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T00:15:46.005 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T00:15:46.005 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T00:15:46.005 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T00:15:46.005 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T00:15:46.005 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T00:15:46.005 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T00:15:46.005 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T00:15:46.005 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T00:15:46.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:15:46.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:15:46.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:15:46.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:15:46.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:15:46.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:15:46.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:15:46.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:15:46.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:15:46.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:15:46.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:15:46.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:15:46.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T00:15:46.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T00:15:46.007 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T00:15:46.007 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: Manager daemon x is now available 2024-04-25T00:15:46.007 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:15:46.007 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T00:15:46.007 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:45 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T00:15:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-04-25T00:15:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: Active manager daemon x restarted 2024-04-25T00:15:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: Activating manager daemon x 2024-04-25T00:15:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T00:15:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: mgrmap e21: x(active, starting, since 0.0110945s), standbys: z, y 2024-04-25T00:15:46.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T00:15:46.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T00:15:46.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T00:15:46.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T00:15:46.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T00:15:46.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T00:15:46.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T00:15:46.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T00:15:46.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T00:15:46.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T00:15:46.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T00:15:46.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T00:15:46.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T00:15:46.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T00:15:46.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T00:15:46.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T00:15:46.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T00:15:46.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T00:15:46.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T00:15:46.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T00:15:46.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T00:15:46.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T00:15:46.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T00:15:46.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T00:15:46.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T00:15:46.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T00:15:46.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T00:15:46.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T00:15:46.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T00:15:46.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T00:15:46.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T00:15:46.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T00:15:46.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T00:15:46.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: Manager daemon x is now available 2024-04-25T00:15:46.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:15:46.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T00:15:46.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:45 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T00:15:46.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.052+0000 7f4fba7fc640 1 -- 172.21.15.27:0/2080796518 <== mon.1 v2:172.21.15.153:3300/0 9 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f4fb408b170 con 0x7f4fc41018b0 2024-04-25T00:15:46.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.054+0000 7f4fba7fc640 1 --2- 172.21.15.27:0/2080796518 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4fb0034f80 0x7f4fb0035360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:46.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.054+0000 7f4fba7fc640 1 -- 172.21.15.27:0/2080796518 --> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f4fb001db70 con 0x7f4fb0034f80 2024-04-25T00:15:46.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.056+0000 7f4fc9666640 1 --2- 172.21.15.27:0/2080796518 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4fb0034f80 0x7f4fb0035360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.058+0000 7f4fc9666640 1 --2- 172.21.15.27:0/2080796518 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4fb0034f80 0x7f4fb0035360 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4fc00058c0 tx=0x7f4fc0005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:46.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.081+0000 7f4fba7fc640 1 -- 172.21.15.27:0/2080796518 <== mgr.15564 v2:172.21.15.27:6800/1076858552 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f4fb001db70 con 0x7f4fb0034f80 2024-04-25T00:15:46.081 INFO:teuthology.orchestra.run.smithi027.stdout:{} 2024-04-25T00:15:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.083+0000 7f4fcb8ca640 1 -- 172.21.15.27:0/2080796518 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4fb0034f80 msgr2=0x7f4fb0035360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.083+0000 7f4fcb8ca640 1 --2- 172.21.15.27:0/2080796518 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4fb0034f80 0x7f4fb0035360 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4fc00058c0 tx=0x7f4fc0005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.083+0000 7f4fcb8ca640 1 -- 172.21.15.27:0/2080796518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fc41018b0 msgr2=0x7f4fc4196ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.083+0000 7f4fcb8ca640 1 --2- 172.21.15.27:0/2080796518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fc41018b0 0x7f4fc4196ec0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4031440 tx=0x7f4fb4031b60 comp rx=0 tx=0).stop 2024-04-25T00:15:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.083+0000 7f4fc9e67640 1 -- 172.21.15.27:0/2080796518 reap_dead start 2024-04-25T00:15:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.083+0000 7f4fcb8ca640 1 -- 172.21.15.27:0/2080796518 shutdown_connections 2024-04-25T00:15:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.083+0000 7f4fcb8ca640 1 -- 172.21.15.27:0/2080796518 >> 172.21.15.27:0/2080796518 conn(0x7f4fc40fbc50 msgr2=0x7f4fc4104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.083+0000 7f4fcb8ca640 1 -- 172.21.15.27:0/2080796518 shutdown_connections 2024-04-25T00:15:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.083+0000 7f4fcb8ca640 1 -- 172.21.15.27:0/2080796518 wait complete. 2024-04-25T00:15:46.096 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-25T00:15:46.108 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-25T00:15:46.118 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:15:46.192 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:46 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:46.064+0000 7fa588965700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:15:46.192 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:46 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:46.064+0000 7fa588965700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:15:46.193 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:46 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:46.064+0000 7fa588965700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:15:46.193 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:46 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:46.064+0000 7fa588965700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:15:46.193 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:15:46 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:15:46.064+0000 7fa588965700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:15:46.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.223+0000 7fa254d5a640 1 -- 172.21.15.27:0/4101335734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa250101890 msgr2=0x7fa250101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:46.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.223+0000 7fa254d5a640 1 --2- 172.21.15.27:0/4101335734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa250101890 0x7fa250101d10 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa244009ca0 tx=0x7fa24402a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.223+0000 7fa254d5a640 1 -- 172.21.15.27:0/4101335734 shutdown_connections 2024-04-25T00:15:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.223+0000 7fa254d5a640 1 --2- 172.21.15.27:0/4101335734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa250102250 0x7fa250108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.223+0000 7fa254d5a640 1 --2- 172.21.15.27:0/4101335734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa250101890 0x7fa250101d10 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.223+0000 7fa254d5a640 1 --2- 172.21.15.27:0/4101335734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa2501006a0 0x7fa250100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.223+0000 7fa254d5a640 1 -- 172.21.15.27:0/4101335734 >> 172.21.15.27:0/4101335734 conn(0x7fa2500fbc30 msgr2=0x7fa2500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.223+0000 7fa254d5a640 1 -- 172.21.15.27:0/4101335734 shutdown_connections 2024-04-25T00:15:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.223+0000 7fa254d5a640 1 -- 172.21.15.27:0/4101335734 wait complete. 2024-04-25T00:15:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa254d5a640 1 Processor -- start 2024-04-25T00:15:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa254d5a640 1 -- start start 2024-04-25T00:15:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa254d5a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2501006a0 0x7fa250197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa254d5a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa250101890 0x7fa250197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa254d5a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa250102250 0x7fa25019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa254d5a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa25010c320 con 0x7fa250102250 2024-04-25T00:15:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa254d5a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa25010c1a0 con 0x7fa250101890 2024-04-25T00:15:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa254d5a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa25010c4a0 con 0x7fa2501006a0 2024-04-25T00:15:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa24ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa250102250 0x7fa25019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa24ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa250102250 0x7fa25019f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35596/0 (socket says 172.21.15.27:35596) 2024-04-25T00:15:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa24ed9d640 1 -- 172.21.15.27:0/1351600316 learned_addr learned my addr 172.21.15.27:0/1351600316 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa24e59c640 1 --2- 172.21.15.27:0/1351600316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2501006a0 0x7fa250197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa24dd9b640 1 --2- 172.21.15.27:0/1351600316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa250101890 0x7fa250197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa24ed9d640 1 -- 172.21.15.27:0/1351600316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2501006a0 msgr2=0x7fa250197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa24ed9d640 1 --2- 172.21.15.27:0/1351600316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2501006a0 0x7fa250197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa24ed9d640 1 -- 172.21.15.27:0/1351600316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa250101890 msgr2=0x7fa250197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa24ed9d640 1 --2- 172.21.15.27:0/1351600316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa250101890 0x7fa250197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa24ed9d640 1 -- 172.21.15.27:0/1351600316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25019f6b0 con 0x7fa250102250 2024-04-25T00:15:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.224+0000 7fa24dd9b640 1 --2- 172.21.15.27:0/1351600316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa250101890 0x7fa250197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.225+0000 7fa24ed9d640 1 --2- 172.21.15.27:0/1351600316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa250102250 0x7fa25019f040 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa24000d750 tx=0x7fa24000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.225+0000 7fa23f7fe640 1 -- 172.21.15.27:0/1351600316 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa240017070 con 0x7fa250102250 2024-04-25T00:15:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.225+0000 7fa23f7fe640 1 -- 172.21.15.27:0/1351600316 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa240005500 con 0x7fa250102250 2024-04-25T00:15:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.225+0000 7fa254d5a640 1 -- 172.21.15.27:0/1351600316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa25019f9a0 con 0x7fa250102250 2024-04-25T00:15:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.225+0000 7fa23f7fe640 1 -- 172.21.15.27:0/1351600316 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa240004d90 con 0x7fa250102250 2024-04-25T00:15:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.225+0000 7fa254d5a640 1 -- 172.21.15.27:0/1351600316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa25019fef0 con 0x7fa250102250 2024-04-25T00:15:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.226+0000 7fa254d5a640 1 -- 172.21.15.27:0/1351600316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2501a0180 con 0x7fa250102250 2024-04-25T00:15:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.227+0000 7fa23f7fe640 1 -- 172.21.15.27:0/1351600316 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7fa240004f30 con 0x7fa250102250 2024-04-25T00:15:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.228+0000 7fa23f7fe640 1 --2- 172.21.15.27:0/1351600316 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa2380a2990 0x7fa2380a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.228+0000 7fa23f7fe640 1 -- 172.21.15.27:0/1351600316 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fa2400ceaa0 con 0x7fa250102250 2024-04-25T00:15:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.228+0000 7fa24e59c640 1 --2- 172.21.15.27:0/1351600316 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa2380a2990 0x7fa2380a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.228+0000 7fa24e59c640 1 --2- 172.21.15.27:0/1351600316 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa2380a2990 0x7fa2380a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa2501016f0 tx=0x7fa248005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.232+0000 7fa23f7fe640 1 -- 172.21.15.27:0/1351600316 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa24009b860 con 0x7fa250102250 2024-04-25T00:15:46.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.421+0000 7fa254d5a640 1 -- 172.21.15.27:0/1351600316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa2501a0470 con 0x7fa250102250 2024-04-25T00:15:46.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.423+0000 7fa23f7fe640 1 -- 172.21.15.27:0/1351600316 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25113 (secure 0 0 0) 0x7fa24009b160 con 0x7fa250102250 2024-04-25T00:15:46.423 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:46.424 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:15:45.034931+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1885111026":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/2528869729":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/4085720546":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/236412562":"2024-04-26T00:15:45.034899+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/2161554778":"2024-04-26T00:15:45.034899+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6800/2161554778":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:15:46.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.425+0000 7fa254d5a640 1 -- 172.21.15.27:0/1351600316 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa2380a2990 msgr2=0x7fa2380a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:46.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.425+0000 7fa254d5a640 1 --2- 172.21.15.27:0/1351600316 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa2380a2990 0x7fa2380a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa2501016f0 tx=0x7fa248005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:46.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.425+0000 7fa254d5a640 1 -- 172.21.15.27:0/1351600316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa250102250 msgr2=0x7fa25019f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:46.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.425+0000 7fa254d5a640 1 --2- 172.21.15.27:0/1351600316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa250102250 0x7fa25019f040 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa24000d750 tx=0x7fa24000dc10 comp rx=0 tx=0).stop 2024-04-25T00:15:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.425+0000 7fa254d5a640 1 -- 172.21.15.27:0/1351600316 shutdown_connections 2024-04-25T00:15:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.425+0000 7fa254d5a640 1 --2- 172.21.15.27:0/1351600316 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa2380a2990 0x7fa2380a4e50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.425+0000 7fa254d5a640 1 --2- 172.21.15.27:0/1351600316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa250102250 0x7fa25019f040 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.425+0000 7fa254d5a640 1 --2- 172.21.15.27:0/1351600316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa250101890 0x7fa250197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.425+0000 7fa254d5a640 1 --2- 172.21.15.27:0/1351600316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2501006a0 0x7fa250197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.425+0000 7fa254d5a640 1 -- 172.21.15.27:0/1351600316 >> 172.21.15.27:0/1351600316 conn(0x7fa2500fbc30 msgr2=0x7fa2500fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.425+0000 7fa254d5a640 1 -- 172.21.15.27:0/1351600316 shutdown_connections 2024-04-25T00:15:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.425+0000 7fa254d5a640 1 -- 172.21.15.27:0/1351600316 wait complete. 2024-04-25T00:15:46.438 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:08:34.006973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T00:13:17.337307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T00:13:18.253713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T00:13:21.995987+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T00:15:46.439 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:15:46.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.550+0000 7f216c129640 1 -- 172.21.15.27:0/4188652840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2164102230 msgr2=0x7f2164108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:46.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.550+0000 7f216c129640 1 --2- 172.21.15.27:0/4188652840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2164102230 0x7f2164108ac0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f215c009ca0 tx=0x7f215c02a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.551+0000 7f216c129640 1 -- 172.21.15.27:0/4188652840 shutdown_connections 2024-04-25T00:15:46.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.551+0000 7f216c129640 1 --2- 172.21.15.27:0/4188652840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2164102230 0x7f2164108ac0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.551+0000 7f216c129640 1 --2- 172.21.15.27:0/4188652840 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2164101870 0x7f2164101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.551+0000 7f216c129640 1 --2- 172.21.15.27:0/4188652840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2164100680 0x7f2164100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.551+0000 7f216c129640 1 -- 172.21.15.27:0/4188652840 >> 172.21.15.27:0/4188652840 conn(0x7f21640fbbf0 msgr2=0x7f21640fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:46.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.551+0000 7f216c129640 1 -- 172.21.15.27:0/4188652840 shutdown_connections 2024-04-25T00:15:46.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.551+0000 7f216c129640 1 -- 172.21.15.27:0/4188652840 wait complete. 2024-04-25T00:15:46.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.551+0000 7f216c129640 1 Processor -- start 2024-04-25T00:15:46.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.551+0000 7f216c129640 1 -- start start 2024-04-25T00:15:46.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.551+0000 7f216c129640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2164100680 0x7f2164197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:46.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.551+0000 7f216c129640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2164101870 0x7f2164198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:46.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.551+0000 7f216c129640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2164102230 0x7f216419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:46.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.551+0000 7f216c129640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f216410c300 con 0x7f2164102230 2024-04-25T00:15:46.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.551+0000 7f216c129640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f216410c180 con 0x7f2164100680 2024-04-25T00:15:46.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.551+0000 7f216c129640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f216410c480 con 0x7f2164101870 2024-04-25T00:15:46.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f216a6c6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2164102230 0x7f216419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:46.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f2169ec5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2164100680 0x7f2164197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:46.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f216a6c6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2164102230 0x7f216419f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35620/0 (socket says 172.21.15.27:35620) 2024-04-25T00:15:46.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f216a6c6640 1 -- 172.21.15.27:0/997559831 learned_addr learned my addr 172.21.15.27:0/997559831 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:46.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f21696c4640 1 --2- 172.21.15.27:0/997559831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2164101870 0x7f2164198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:46.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f216a6c6640 1 -- 172.21.15.27:0/997559831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2164101870 msgr2=0x7f2164198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:46.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f216a6c6640 1 --2- 172.21.15.27:0/997559831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2164101870 0x7f2164198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f216a6c6640 1 -- 172.21.15.27:0/997559831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2164100680 msgr2=0x7f2164197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:46.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f216a6c6640 1 --2- 172.21.15.27:0/997559831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2164100680 0x7f2164197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f216a6c6640 1 -- 172.21.15.27:0/997559831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f216419f680 con 0x7f2164102230 2024-04-25T00:15:46.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f2169ec5640 1 --2- 172.21.15.27:0/997559831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2164100680 0x7f2164197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:46.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f21696c4640 1 --2- 172.21.15.27:0/997559831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2164101870 0x7f2164198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:46.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f216a6c6640 1 --2- 172.21.15.27:0/997559831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2164102230 0x7f216419f010 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f215c031ae0 tx=0x7f215c031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:46.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f215affd640 1 -- 172.21.15.27:0/997559831 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c003b70 con 0x7f2164102230 2024-04-25T00:15:46.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f216c129640 1 -- 172.21.15.27:0/997559831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f216419f910 con 0x7f2164102230 2024-04-25T00:15:46.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f215affd640 1 -- 172.21.15.27:0/997559831 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f215c003d10 con 0x7f2164102230 2024-04-25T00:15:46.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f215affd640 1 -- 172.21.15.27:0/997559831 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c02fc40 con 0x7f2164102230 2024-04-25T00:15:46.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.552+0000 7f216c129640 1 -- 172.21.15.27:0/997559831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f216419fe60 con 0x7f2164102230 2024-04-25T00:15:46.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.553+0000 7f216c129640 1 -- 172.21.15.27:0/997559831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21641a0090 con 0x7f2164102230 2024-04-25T00:15:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.554+0000 7f215affd640 1 -- 172.21.15.27:0/997559831 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f215c004050 con 0x7f2164102230 2024-04-25T00:15:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.555+0000 7f215affd640 1 --2- 172.21.15.27:0/997559831 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f21500a2990 0x7f21500a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.555+0000 7f215affd640 1 -- 172.21.15.27:0/997559831 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f215c0f01a0 con 0x7f2164102230 2024-04-25T00:15:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.555+0000 7f2169ec5640 1 --2- 172.21.15.27:0/997559831 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f21500a2990 0x7f21500a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.555+0000 7f2169ec5640 1 --2- 172.21.15.27:0/997559831 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f21500a2990 0x7f21500a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f21641016d0 tx=0x7f2160005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.560+0000 7f215affd640 1 -- 172.21.15.27:0/997559831 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f215c0bd080 con 0x7f2164102230 2024-04-25T00:15:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.746+0000 7f216c129640 1 -- 172.21.15.27:0/997559831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f21641a0260 con 0x7f2164102230 2024-04-25T00:15:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.746+0000 7f215affd640 1 -- 172.21.15.27:0/997559831 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f215c0bc980 con 0x7f2164102230 2024-04-25T00:15:46.746 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T00:15:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.748+0000 7f216c129640 1 -- 172.21.15.27:0/997559831 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f21500a2990 msgr2=0x7f21500a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.748+0000 7f216c129640 1 --2- 172.21.15.27:0/997559831 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f21500a2990 0x7f21500a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f21641016d0 tx=0x7f2160005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.748+0000 7f216c129640 1 -- 172.21.15.27:0/997559831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2164102230 msgr2=0x7f216419f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.748+0000 7f216c129640 1 --2- 172.21.15.27:0/997559831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2164102230 0x7f216419f010 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f215c031ae0 tx=0x7f215c031ee0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.748+0000 7f216c129640 1 -- 172.21.15.27:0/997559831 shutdown_connections 2024-04-25T00:15:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.748+0000 7f216c129640 1 --2- 172.21.15.27:0/997559831 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f21500a2990 0x7f21500a4e50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.748+0000 7f216c129640 1 --2- 172.21.15.27:0/997559831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2164102230 0x7f216419f010 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.748+0000 7f216c129640 1 --2- 172.21.15.27:0/997559831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2164101870 0x7f2164198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.748+0000 7f216c129640 1 --2- 172.21.15.27:0/997559831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2164100680 0x7f2164197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.748+0000 7f216c129640 1 -- 172.21.15.27:0/997559831 >> 172.21.15.27:0/997559831 conn(0x7f21640fbbf0 msgr2=0x7f21640fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.748+0000 7f216c129640 1 -- 172.21.15.27:0/997559831 shutdown_connections 2024-04-25T00:15:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.748+0000 7f216c129640 1 -- 172.21.15.27:0/997559831 wait complete. 2024-04-25T00:15:46.760 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T00:15:46.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.871+0000 7f16890c3640 1 -- 172.21.15.27:0/3818011859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1684102230 msgr2=0x7f1684108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:46.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.871+0000 7f16890c3640 1 --2- 172.21.15.27:0/3818011859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1684102230 0x7f1684108ac0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f167c009ca0 tx=0x7f167c02a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.871+0000 7f16890c3640 1 -- 172.21.15.27:0/3818011859 shutdown_connections 2024-04-25T00:15:46.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.871+0000 7f16890c3640 1 --2- 172.21.15.27:0/3818011859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1684102230 0x7f1684108ac0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.871+0000 7f16890c3640 1 --2- 172.21.15.27:0/3818011859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1684101870 0x7f1684101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.871+0000 7f16890c3640 1 --2- 172.21.15.27:0/3818011859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1684100680 0x7f1684100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.871+0000 7f16890c3640 1 -- 172.21.15.27:0/3818011859 >> 172.21.15.27:0/3818011859 conn(0x7f16840fbbf0 msgr2=0x7f16840fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:46.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.871+0000 7f16890c3640 1 -- 172.21.15.27:0/3818011859 shutdown_connections 2024-04-25T00:15:46.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.871+0000 7f16890c3640 1 -- 172.21.15.27:0/3818011859 wait complete. 2024-04-25T00:15:46.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.872+0000 7f16890c3640 1 Processor -- start 2024-04-25T00:15:46.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.872+0000 7f16890c3640 1 -- start start 2024-04-25T00:15:46.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.872+0000 7f16890c3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1684100680 0x7f1684197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:46.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.872+0000 7f16890c3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1684101870 0x7f1684197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:46.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.872+0000 7f16890c3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1684102230 0x7f168419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:46.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.872+0000 7f16890c3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f168410c300 con 0x7f1684102230 2024-04-25T00:15:46.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.872+0000 7f16890c3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f168410c180 con 0x7f1684100680 2024-04-25T00:15:46.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.872+0000 7f16890c3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f168410c480 con 0x7f1684101870 2024-04-25T00:15:46.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.872+0000 7f168359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1684102230 0x7f168419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:46.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.872+0000 7f168359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1684102230 0x7f168419f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47486/0 (socket says 172.21.15.27:47486) 2024-04-25T00:15:46.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.872+0000 7f168359e640 1 -- 172.21.15.27:0/1663156552 learned_addr learned my addr 172.21.15.27:0/1663156552 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:46.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.872+0000 7f167bfff640 1 --2- 172.21.15.27:0/1663156552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1684101870 0x7f1684197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:46.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.872+0000 7f1682d9d640 1 --2- 172.21.15.27:0/1663156552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1684100680 0x7f1684197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:46.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.873+0000 7f167bfff640 1 -- 172.21.15.27:0/1663156552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1684100680 msgr2=0x7f1684197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:46.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.873+0000 7f167bfff640 1 --2- 172.21.15.27:0/1663156552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1684100680 0x7f1684197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.873+0000 7f167bfff640 1 -- 172.21.15.27:0/1663156552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1684102230 msgr2=0x7f168419f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:46.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.873+0000 7f167bfff640 1 --2- 172.21.15.27:0/1663156552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1684102230 0x7f168419f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:46.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.873+0000 7f167bfff640 1 -- 172.21.15.27:0/1663156552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f168419f6b0 con 0x7f1684101870 2024-04-25T00:15:46.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.873+0000 7f168359e640 1 --2- 172.21.15.27:0/1663156552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1684102230 0x7f168419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:46.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.873+0000 7f167bfff640 1 --2- 172.21.15.27:0/1663156552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1684101870 0x7f1684197fd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f166c00e9f0 tx=0x7f166c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:46.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.873+0000 7f1680d99640 1 -- 172.21.15.27:0/1663156552 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c00ccc0 con 0x7f1684101870 2024-04-25T00:15:46.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.873+0000 7f1680d99640 1 -- 172.21.15.27:0/1663156552 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f166c005020 con 0x7f1684101870 2024-04-25T00:15:46.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.873+0000 7f16890c3640 1 -- 172.21.15.27:0/1663156552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f168419f9a0 con 0x7f1684101870 2024-04-25T00:15:46.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.873+0000 7f1680d99640 1 -- 172.21.15.27:0/1663156552 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c0049e0 con 0x7f1684101870 2024-04-25T00:15:46.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.873+0000 7f16890c3640 1 -- 172.21.15.27:0/1663156552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f168419fea0 con 0x7f1684101870 2024-04-25T00:15:46.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.874+0000 7f16890c3640 1 -- 172.21.15.27:0/1663156552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16841a0130 con 0x7f1684101870 2024-04-25T00:15:46.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.876+0000 7f1680d99640 1 -- 172.21.15.27:0/1663156552 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f166c005560 con 0x7f1684101870 2024-04-25T00:15:46.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.876+0000 7f1680d99640 1 --2- 172.21.15.27:0/1663156552 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f16700a2990 0x7f16700a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:46.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.876+0000 7f1682d9d640 1 --2- 172.21.15.27:0/1663156552 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f16700a2990 0x7f16700a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:46.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.876+0000 7f1680d99640 1 -- 172.21.15.27:0/1663156552 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f166c0cf0e0 con 0x7f1684101870 2024-04-25T00:15:46.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.877+0000 7f1682d9d640 1 --2- 172.21.15.27:0/1663156552 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f16700a2990 0x7f16700a4e50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f16841016d0 tx=0x7f1674005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:46.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:46.883+0000 7f1680d99640 1 -- 172.21.15.27:0/1663156552 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f166c0051d0 con 0x7f1684101870 2024-04-25T00:15:47.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.060+0000 7f1680d99640 1 -- 172.21.15.27:0/1663156552 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f166c097e80 con 0x7f1684101870 2024-04-25T00:15:47.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.095+0000 7f16890c3640 1 -- 172.21.15.27:0/1663156552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f16841a0300 con 0x7f1684101870 2024-04-25T00:15:47.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.095+0000 7f1680d99640 1 -- 172.21.15.27:0/1663156552 <== mon.1 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f166c09bf10 con 0x7f1684101870 2024-04-25T00:15:47.095 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:15:47.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.097+0000 7f16890c3640 1 -- 172.21.15.27:0/1663156552 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f16700a2990 msgr2=0x7f16700a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.097+0000 7f16890c3640 1 --2- 172.21.15.27:0/1663156552 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f16700a2990 0x7f16700a4e50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f16841016d0 tx=0x7f1674005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.097+0000 7f16890c3640 1 -- 172.21.15.27:0/1663156552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1684101870 msgr2=0x7f1684197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.097+0000 7f16890c3640 1 --2- 172.21.15.27:0/1663156552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1684101870 0x7f1684197fd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f166c00e9f0 tx=0x7f166c00eeb0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.098+0000 7f16890c3640 1 -- 172.21.15.27:0/1663156552 shutdown_connections 2024-04-25T00:15:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.098+0000 7f16890c3640 1 --2- 172.21.15.27:0/1663156552 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f16700a2990 0x7f16700a4e50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.098+0000 7f16890c3640 1 --2- 172.21.15.27:0/1663156552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1684102230 0x7f168419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.098+0000 7f16890c3640 1 --2- 172.21.15.27:0/1663156552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1684101870 0x7f1684197fd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.098+0000 7f16890c3640 1 --2- 172.21.15.27:0/1663156552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1684100680 0x7f1684197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.098+0000 7f16890c3640 1 -- 172.21.15.27:0/1663156552 >> 172.21.15.27:0/1663156552 conn(0x7f16840fbbf0 msgr2=0x7f16840fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.098+0000 7f16890c3640 1 -- 172.21.15.27:0/1663156552 shutdown_connections 2024-04-25T00:15:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.098+0000 7f16890c3640 1 -- 172.21.15.27:0/1663156552 wait complete. 2024-04-25T00:15:47.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T00:15:47.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.236+0000 7f0dcb9c1640 1 -- 172.21.15.27:0/3625089271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dc4107480 msgr2=0x7f0dc41098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:47.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.236+0000 7f0dcb9c1640 1 --2- 172.21.15.27:0/3625089271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dc4107480 0x7f0dc41098d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009ca0 tx=0x7f0dbc02a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.236+0000 7f0dcb9c1640 1 -- 172.21.15.27:0/3625089271 shutdown_connections 2024-04-25T00:15:47.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.236+0000 7f0dcb9c1640 1 --2- 172.21.15.27:0/3625089271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dc4107480 0x7f0dc41098d0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.236+0000 7f0dcb9c1640 1 --2- 172.21.15.27:0/3625089271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc4104b50 0x7f0dc4106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.236+0000 7f0dcb9c1640 1 --2- 172.21.15.27:0/3625089271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0dc4102220 0x7f0dc4104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.236+0000 7f0dcb9c1640 1 -- 172.21.15.27:0/3625089271 >> 172.21.15.27:0/3625089271 conn(0x7f0dc40fbc30 msgr2=0x7f0dc40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.236+0000 7f0dcb9c1640 1 -- 172.21.15.27:0/3625089271 shutdown_connections 2024-04-25T00:15:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.236+0000 7f0dcb9c1640 1 -- 172.21.15.27:0/3625089271 wait complete. 2024-04-25T00:15:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dcb9c1640 1 Processor -- start 2024-04-25T00:15:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dcb9c1640 1 -- start start 2024-04-25T00:15:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dcb9c1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc4102220 0x7f0dc4199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dcb9c1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0dc4104b50 0x7f0dc419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dcb9c1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dc4107480 0x7f0dc41a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dcb9c1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc410d420 con 0x7f0dc4107480 2024-04-25T00:15:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dcb9c1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc410d2a0 con 0x7f0dc4104b50 2024-04-25T00:15:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dcb9c1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc410d5a0 con 0x7f0dc4102220 2024-04-25T00:15:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dc9f5e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dc4107480 0x7f0dc41a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dc9f5e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dc4107480 0x7f0dc41a1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47518/0 (socket says 172.21.15.27:47518) 2024-04-25T00:15:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dc9f5e640 1 -- 172.21.15.27:0/1189263107 learned_addr learned my addr 172.21.15.27:0/1189263107 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dc975d640 1 --2- 172.21.15.27:0/1189263107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc4102220 0x7f0dc4199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dc8f5c640 1 --2- 172.21.15.27:0/1189263107 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0dc4104b50 0x7f0dc419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dc9f5e640 1 -- 172.21.15.27:0/1189263107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc4102220 msgr2=0x7f0dc4199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dc9f5e640 1 --2- 172.21.15.27:0/1189263107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc4102220 0x7f0dc4199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dc9f5e640 1 -- 172.21.15.27:0/1189263107 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0dc4104b50 msgr2=0x7f0dc419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dc9f5e640 1 --2- 172.21.15.27:0/1189263107 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0dc4104b50 0x7f0dc419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dc9f5e640 1 -- 172.21.15.27:0/1189263107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc41a1920 con 0x7f0dc4107480 2024-04-25T00:15:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.237+0000 7f0dc9f5e640 1 --2- 172.21.15.27:0/1189263107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dc4107480 0x7f0dc41a1280 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc005e30 tx=0x7f0dbc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.238+0000 7f0dba7fc640 1 -- 172.21.15.27:0/1189263107 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc003b70 con 0x7f0dc4107480 2024-04-25T00:15:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.238+0000 7f0dba7fc640 1 -- 172.21.15.27:0/1189263107 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0dbc003d10 con 0x7f0dc4107480 2024-04-25T00:15:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.238+0000 7f0dcb9c1640 1 -- 172.21.15.27:0/1189263107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dc41a1bb0 con 0x7f0dc4107480 2024-04-25T00:15:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.238+0000 7f0dba7fc640 1 -- 172.21.15.27:0/1189263107 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc02fbd0 con 0x7f0dc4107480 2024-04-25T00:15:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.238+0000 7f0dcb9c1640 1 -- 172.21.15.27:0/1189263107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dc41a2150 con 0x7f0dc4107480 2024-04-25T00:15:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.239+0000 7f0dcb9c1640 1 -- 172.21.15.27:0/1189263107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dc4108f70 con 0x7f0dc4107480 2024-04-25T00:15:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.240+0000 7f0dba7fc640 1 -- 172.21.15.27:0/1189263107 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f0dbc02fd70 con 0x7f0dc4107480 2024-04-25T00:15:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.241+0000 7f0dba7fc640 1 --2- 172.21.15.27:0/1189263107 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0db00a2e10 0x7f0db00a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.241+0000 7f0dba7fc640 1 -- 172.21.15.27:0/1189263107 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f0dbc0f0ac0 con 0x7f0dc4107480 2024-04-25T00:15:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.241+0000 7f0dc975d640 1 --2- 172.21.15.27:0/1189263107 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0db00a2e10 0x7f0db00a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.242+0000 7f0dc975d640 1 --2- 172.21.15.27:0/1189263107 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0db00a2e10 0x7f0db00a52d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0dc419b1e0 tx=0x7f0dc0005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.245+0000 7f0dba7fc640 1 -- 172.21.15.27:0/1189263107 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0dbc0bd310 con 0x7f0dc4107480 2024-04-25T00:15:47.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.432+0000 7f0dcb9c1640 1 -- 172.21.15.27:0/1189263107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0dc402d300 con 0x7f0dc4107480 2024-04-25T00:15:47.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.432+0000 7f0dba7fc640 1 -- 172.21.15.27:0/1189263107 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f0dbc0bcc10 con 0x7f0dc4107480 2024-04-25T00:15:47.432 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T00:15:47.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.435+0000 7f0dcb9c1640 1 -- 172.21.15.27:0/1189263107 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0db00a2e10 msgr2=0x7f0db00a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:47.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.435+0000 7f0dcb9c1640 1 --2- 172.21.15.27:0/1189263107 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0db00a2e10 0x7f0db00a52d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0dc419b1e0 tx=0x7f0dc0005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:47.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.435+0000 7f0dcb9c1640 1 -- 172.21.15.27:0/1189263107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dc4107480 msgr2=0x7f0dc41a1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:47.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.435+0000 7f0dcb9c1640 1 --2- 172.21.15.27:0/1189263107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dc4107480 0x7f0dc41a1280 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc005e30 tx=0x7f0dbc004ab0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.435+0000 7f0dcb9c1640 1 -- 172.21.15.27:0/1189263107 shutdown_connections 2024-04-25T00:15:47.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.435+0000 7f0dcb9c1640 1 --2- 172.21.15.27:0/1189263107 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0db00a2e10 0x7f0db00a52d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.435+0000 7f0dcb9c1640 1 --2- 172.21.15.27:0/1189263107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dc4107480 0x7f0dc41a1280 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.435+0000 7f0dcb9c1640 1 --2- 172.21.15.27:0/1189263107 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0dc4104b50 0x7f0dc419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.435+0000 7f0dcb9c1640 1 --2- 172.21.15.27:0/1189263107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc4102220 0x7f0dc4199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.435+0000 7f0dcb9c1640 1 -- 172.21.15.27:0/1189263107 >> 172.21.15.27:0/1189263107 conn(0x7f0dc40fbc30 msgr2=0x7f0dc4101980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:47.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.435+0000 7f0dcb9c1640 1 -- 172.21.15.27:0/1189263107 shutdown_connections 2024-04-25T00:15:47.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.435+0000 7f0dcb9c1640 1 -- 172.21.15.27:0/1189263107 wait complete. 2024-04-25T00:15:47.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:47 smithi027 ceph-mon[29208]: [25/Apr/2024:00:15:45] ENGINE Bus STARTING 2024-04-25T00:15:47.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:47 smithi027 ceph-mon[29208]: [25/Apr/2024:00:15:45] ENGINE Serving on http://172.21.15.27:8765 2024-04-25T00:15:47.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:47 smithi027 ceph-mon[29208]: [25/Apr/2024:00:15:46] ENGINE Serving on https://172.21.15.27:7150 2024-04-25T00:15:47.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:47 smithi027 ceph-mon[29208]: [25/Apr/2024:00:15:46] ENGINE Bus STARTED 2024-04-25T00:15:47.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:47 smithi027 ceph-mon[29208]: mgrmap e22: x(active, since 1.01886s), standbys: z, y 2024-04-25T00:15:47.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:47 smithi027 ceph-mon[29208]: from='client.25207 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:47.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:47 smithi027 ceph-mon[29208]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:15:47.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:47 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1351600316' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:47.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:47 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/997559831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:47.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T00:15:47.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:47 smithi139 ceph-mon[32833]: [25/Apr/2024:00:15:45] ENGINE Bus STARTING 2024-04-25T00:15:47.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:47 smithi139 ceph-mon[32833]: [25/Apr/2024:00:15:45] ENGINE Serving on http://172.21.15.27:8765 2024-04-25T00:15:47.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:47 smithi139 ceph-mon[32833]: [25/Apr/2024:00:15:46] ENGINE Serving on https://172.21.15.27:7150 2024-04-25T00:15:47.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:47 smithi139 ceph-mon[32833]: [25/Apr/2024:00:15:46] ENGINE Bus STARTED 2024-04-25T00:15:47.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:47 smithi139 ceph-mon[32833]: mgrmap e22: x(active, since 1.01886s), standbys: z, y 2024-04-25T00:15:47.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:47 smithi139 ceph-mon[32833]: from='client.25207 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:47.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:47 smithi139 ceph-mon[32833]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:15:47.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:47 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1351600316' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:47.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:47 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/997559831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:47.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:47 smithi153 ceph-mon[32896]: [25/Apr/2024:00:15:45] ENGINE Bus STARTING 2024-04-25T00:15:47.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:47 smithi153 ceph-mon[32896]: [25/Apr/2024:00:15:45] ENGINE Serving on http://172.21.15.27:8765 2024-04-25T00:15:47.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:47 smithi153 ceph-mon[32896]: [25/Apr/2024:00:15:46] ENGINE Serving on https://172.21.15.27:7150 2024-04-25T00:15:47.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:47 smithi153 ceph-mon[32896]: [25/Apr/2024:00:15:46] ENGINE Bus STARTED 2024-04-25T00:15:47.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:47 smithi153 ceph-mon[32896]: mgrmap e22: x(active, since 1.01886s), standbys: z, y 2024-04-25T00:15:47.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:47 smithi153 ceph-mon[32896]: from='client.25207 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:47.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:47 smithi153 ceph-mon[32896]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:15:47.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:47 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1351600316' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:47.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:47 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/997559831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:47.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.552+0000 7f34bfa15640 1 -- 172.21.15.27:0/380031304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34b8102230 msgr2=0x7f34b8108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:47.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.552+0000 7f34bfa15640 1 --2- 172.21.15.27:0/380031304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34b8102230 0x7f34b8108ac0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009ca0 tx=0x7f34b002a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.552+0000 7f34bfa15640 1 -- 172.21.15.27:0/380031304 shutdown_connections 2024-04-25T00:15:47.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.552+0000 7f34bfa15640 1 --2- 172.21.15.27:0/380031304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34b8102230 0x7f34b8108ac0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.552+0000 7f34bfa15640 1 --2- 172.21.15.27:0/380031304 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34b8101870 0x7f34b8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.552+0000 7f34bfa15640 1 --2- 172.21.15.27:0/380031304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34b8100680 0x7f34b8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.552+0000 7f34bfa15640 1 -- 172.21.15.27:0/380031304 >> 172.21.15.27:0/380031304 conn(0x7f34b80fbbf0 msgr2=0x7f34b80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.552+0000 7f34bfa15640 1 -- 172.21.15.27:0/380031304 shutdown_connections 2024-04-25T00:15:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.552+0000 7f34bfa15640 1 -- 172.21.15.27:0/380031304 wait complete. 2024-04-25T00:15:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.553+0000 7f34bfa15640 1 Processor -- start 2024-04-25T00:15:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.553+0000 7f34bfa15640 1 -- start start 2024-04-25T00:15:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.553+0000 7f34bfa15640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34b8100680 0x7f34b8069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.553+0000 7f34bfa15640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34b8101870 0x7f34b819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.553+0000 7f34bfa15640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34b8102230 0x7f34b81a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.553+0000 7f34bfa15640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f34b810c2e0 con 0x7f34b8102230 2024-04-25T00:15:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.553+0000 7f34bfa15640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f34b810c160 con 0x7f34b8100680 2024-04-25T00:15:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.553+0000 7f34bfa15640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f34b810c460 con 0x7f34b8101870 2024-04-25T00:15:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.553+0000 7f34bdfb2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34b8102230 0x7f34b81a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.553+0000 7f34bcfb0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34b8101870 0x7f34b819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.553+0000 7f34bd7b1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34b8100680 0x7f34b8069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:47.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.553+0000 7f34bcfb0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34b8101870 0x7f34b819eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:59678/0 (socket says 172.21.15.27:59678) 2024-04-25T00:15:47.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.553+0000 7f34bcfb0640 1 -- 172.21.15.27:0/4073110288 learned_addr learned my addr 172.21.15.27:0/4073110288 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:47.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.553+0000 7f34bdfb2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34b8102230 0x7f34b81a10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47546/0 (socket says 172.21.15.27:47546) 2024-04-25T00:15:47.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.554+0000 7f34bcfb0640 1 -- 172.21.15.27:0/4073110288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34b8100680 msgr2=0x7f34b8069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:47.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.554+0000 7f34bcfb0640 1 --2- 172.21.15.27:0/4073110288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34b8100680 0x7f34b8069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.554+0000 7f34bcfb0640 1 -- 172.21.15.27:0/4073110288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34b8102230 msgr2=0x7f34b81a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:47.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.554+0000 7f34bcfb0640 1 --2- 172.21.15.27:0/4073110288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34b8102230 0x7f34b81a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.554+0000 7f34bcfb0640 1 -- 172.21.15.27:0/4073110288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b81a18e0 con 0x7f34b8101870 2024-04-25T00:15:47.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.554+0000 7f34bd7b1640 1 --2- 172.21.15.27:0/4073110288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34b8100680 0x7f34b8069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:47.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.554+0000 7f34bdfb2640 1 --2- 172.21.15.27:0/4073110288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34b8102230 0x7f34b81a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:47.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.554+0000 7f34bcfb0640 1 --2- 172.21.15.27:0/4073110288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34b8101870 0x7f34b819eba0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f34b400e9f0 tx=0x7f34b400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:47.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.554+0000 7f34ae7fc640 1 -- 172.21.15.27:0/4073110288 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b400cd80 con 0x7f34b8101870 2024-04-25T00:15:47.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.554+0000 7f34ae7fc640 1 -- 172.21.15.27:0/4073110288 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34b4010910 con 0x7f34b8101870 2024-04-25T00:15:47.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.554+0000 7f34bfa15640 1 -- 172.21.15.27:0/4073110288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b81a1b70 con 0x7f34b8101870 2024-04-25T00:15:47.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.554+0000 7f34ae7fc640 1 -- 172.21.15.27:0/4073110288 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b40188f0 con 0x7f34b8101870 2024-04-25T00:15:47.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.554+0000 7f34bfa15640 1 -- 172.21.15.27:0/4073110288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b81a20c0 con 0x7f34b8101870 2024-04-25T00:15:47.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.555+0000 7f34bfa15640 1 -- 172.21.15.27:0/4073110288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34b8107a10 con 0x7f34b8101870 2024-04-25T00:15:47.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.556+0000 7f34ae7fc640 1 -- 172.21.15.27:0/4073110288 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f34b4010ac0 con 0x7f34b8101870 2024-04-25T00:15:47.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.557+0000 7f34ae7fc640 1 --2- 172.21.15.27:0/4073110288 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f34a40a2dc0 0x7f34a40a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:47.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.557+0000 7f34bd7b1640 1 --2- 172.21.15.27:0/4073110288 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f34a40a2dc0 0x7f34a40a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:47.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.557+0000 7f34ae7fc640 1 -- 172.21.15.27:0/4073110288 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f34b40d0060 con 0x7f34b8101870 2024-04-25T00:15:47.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.557+0000 7f34bd7b1640 1 --2- 172.21.15.27:0/4073110288 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f34a40a2dc0 0x7f34a40a5280 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f34b81016d0 tx=0x7f34a8005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:47.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.560+0000 7f34ae7fc640 1 -- 172.21.15.27:0/4073110288 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f34b409c150 con 0x7f34b8101870 2024-04-25T00:15:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.751+0000 7f34bfa15640 1 -- 172.21.15.27:0/4073110288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f34b802d300 con 0x7f34b8101870 2024-04-25T00:15:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.751+0000 7f34ae7fc640 1 -- 172.21.15.27:0/4073110288 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f34b40a0450 con 0x7f34b8101870 2024-04-25T00:15:47.752 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:15:47.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.753+0000 7f34bfa15640 1 -- 172.21.15.27:0/4073110288 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f34a40a2dc0 msgr2=0x7f34a40a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:47.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.753+0000 7f34bfa15640 1 --2- 172.21.15.27:0/4073110288 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f34a40a2dc0 0x7f34a40a5280 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f34b81016d0 tx=0x7f34a8005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:47.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.753+0000 7f34bfa15640 1 -- 172.21.15.27:0/4073110288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34b8101870 msgr2=0x7f34b819eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:47.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.753+0000 7f34bfa15640 1 --2- 172.21.15.27:0/4073110288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34b8101870 0x7f34b819eba0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f34b400e9f0 tx=0x7f34b400eeb0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.753+0000 7f34bfa15640 1 -- 172.21.15.27:0/4073110288 shutdown_connections 2024-04-25T00:15:47.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.753+0000 7f34bfa15640 1 --2- 172.21.15.27:0/4073110288 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f34a40a2dc0 0x7f34a40a5280 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.753+0000 7f34bfa15640 1 --2- 172.21.15.27:0/4073110288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34b8102230 0x7f34b81a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.753+0000 7f34bfa15640 1 --2- 172.21.15.27:0/4073110288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34b8101870 0x7f34b819eba0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.753+0000 7f34bfa15640 1 --2- 172.21.15.27:0/4073110288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34b8100680 0x7f34b8069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.753+0000 7f34bfa15640 1 -- 172.21.15.27:0/4073110288 >> 172.21.15.27:0/4073110288 conn(0x7f34b80fbbf0 msgr2=0x7f34b8104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:47.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.753+0000 7f34bfa15640 1 -- 172.21.15.27:0/4073110288 shutdown_connections 2024-04-25T00:15:47.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.753+0000 7f34bfa15640 1 -- 172.21.15.27:0/4073110288 wait complete. 2024-04-25T00:15:47.765 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:15:47.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.870+0000 7f7e50763640 1 -- 172.21.15.27:0/3100506356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e48100060 msgr2=0x7f7e48100480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:47.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.870+0000 7f7e50763640 1 --2- 172.21.15.27:0/3100506356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e48100060 0x7f7e48100480 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7e44009900 tx=0x7f7e4402a410 comp rx=0 tx=0).stop 2024-04-25T00:15:47.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.870+0000 7f7e50763640 1 -- 172.21.15.27:0/3100506356 shutdown_connections 2024-04-25T00:15:47.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.870+0000 7f7e50763640 1 --2- 172.21.15.27:0/3100506356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48101360 0x7f7e481017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.870+0000 7f7e50763640 1 --2- 172.21.15.27:0/3100506356 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e481009c0 0x7f7e48100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.870+0000 7f7e50763640 1 --2- 172.21.15.27:0/3100506356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e48100060 0x7f7e48100480 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.870+0000 7f7e50763640 1 -- 172.21.15.27:0/3100506356 >> 172.21.15.27:0/3100506356 conn(0x7f7e480fbc20 msgr2=0x7f7e480fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.870+0000 7f7e50763640 1 -- 172.21.15.27:0/3100506356 shutdown_connections 2024-04-25T00:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.870+0000 7f7e50763640 1 -- 172.21.15.27:0/3100506356 wait complete. 2024-04-25T00:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.870+0000 7f7e50763640 1 Processor -- start 2024-04-25T00:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.870+0000 7f7e50763640 1 -- start start 2024-04-25T00:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.870+0000 7f7e50763640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48100060 0x7f7e48199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.870+0000 7f7e50763640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e481009c0 0x7f7e4819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e50763640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e48101360 0x7f7e481a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e50763640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4810f5d0 con 0x7f7e481009c0 2024-04-25T00:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e50763640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4810f450 con 0x7f7e48101360 2024-04-25T00:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e50763640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4810f750 con 0x7f7e48100060 2024-04-25T00:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e4dcfe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e481009c0 0x7f7e4819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e4e4ff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48100060 0x7f7e48199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e4dcfe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e481009c0 0x7f7e4819a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47570/0 (socket says 172.21.15.27:47570) 2024-04-25T00:15:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e4dcfe640 1 -- 172.21.15.27:0/3300843800 learned_addr learned my addr 172.21.15.27:0/3300843800 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e4ed00640 1 --2- 172.21.15.27:0/3300843800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e48101360 0x7f7e481a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e4e4ff640 1 -- 172.21.15.27:0/3300843800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e48101360 msgr2=0x7f7e481a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e4e4ff640 1 --2- 172.21.15.27:0/3300843800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e48101360 0x7f7e481a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e4e4ff640 1 -- 172.21.15.27:0/3300843800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e481009c0 msgr2=0x7f7e4819a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e4e4ff640 1 --2- 172.21.15.27:0/3300843800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e481009c0 0x7f7e4819a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e4e4ff640 1 -- 172.21.15.27:0/3300843800 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e481a18b0 con 0x7f7e48100060 2024-04-25T00:15:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e4dcfe640 1 --2- 172.21.15.27:0/3300843800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e481009c0 0x7f7e4819a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e4ed00640 1 --2- 172.21.15.27:0/3300843800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e48101360 0x7f7e481a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e4e4ff640 1 --2- 172.21.15.27:0/3300843800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48100060 0x7f7e48199c70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7e44031440 tx=0x7f7e44004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.871+0000 7f7e3f7fe640 1 -- 172.21.15.27:0/3300843800 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4403f070 con 0x7f7e48100060 2024-04-25T00:15:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.872+0000 7f7e3f7fe640 1 -- 172.21.15.27:0/3300843800 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e44003cd0 con 0x7f7e48100060 2024-04-25T00:15:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.872+0000 7f7e50763640 1 -- 172.21.15.27:0/3300843800 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e481a1b40 con 0x7f7e48100060 2024-04-25T00:15:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.872+0000 7f7e3f7fe640 1 -- 172.21.15.27:0/3300843800 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e44035070 con 0x7f7e48100060 2024-04-25T00:15:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.872+0000 7f7e50763640 1 -- 172.21.15.27:0/3300843800 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e481a2050 con 0x7f7e48100060 2024-04-25T00:15:47.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.873+0000 7f7e50763640 1 -- 172.21.15.27:0/3300843800 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e4810afa0 con 0x7f7e48100060 2024-04-25T00:15:47.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.874+0000 7f7e3f7fe640 1 -- 172.21.15.27:0/3300843800 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f7e4403b4f0 con 0x7f7e48100060 2024-04-25T00:15:47.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.875+0000 7f7e3f7fe640 1 --2- 172.21.15.27:0/3300843800 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7e340a2e10 0x7f7e340a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:47.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.875+0000 7f7e3f7fe640 1 -- 172.21.15.27:0/3300843800 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f7e440f0010 con 0x7f7e48100060 2024-04-25T00:15:47.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.875+0000 7f7e4dcfe640 1 --2- 172.21.15.27:0/3300843800 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7e340a2e10 0x7f7e340a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:47.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.876+0000 7f7e4dcfe640 1 --2- 172.21.15.27:0/3300843800 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7e340a2e10 0x7f7e340a52d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7e4819b170 tx=0x7f7e38005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:47.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:47.880+0000 7f7e3f7fe640 1 -- 172.21.15.27:0/3300843800 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7e440bc100 con 0x7f7e48100060 2024-04-25T00:15:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.091+0000 7f7e50763640 1 -- 172.21.15.27:0/3300843800 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7e48062340 con 0x7f7e48100060 2024-04-25T00:15:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.093+0000 7f7e3f7fe640 1 -- 172.21.15.27:0/3300843800 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25113 (secure 0 0 0) 0x7f7e440c0400 con 0x7f7e48100060 2024-04-25T00:15:48.093 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:48.094 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:15:45.034931+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1885111026":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/2528869729":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/4085720546":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/236412562":"2024-04-26T00:15:45.034899+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/2161554778":"2024-04-26T00:15:45.034899+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6800/2161554778":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:15:48.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.095+0000 7f7e50763640 1 -- 172.21.15.27:0/3300843800 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7e340a2e10 msgr2=0x7f7e340a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:48.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.095+0000 7f7e50763640 1 --2- 172.21.15.27:0/3300843800 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7e340a2e10 0x7f7e340a52d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7e4819b170 tx=0x7f7e38005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:48.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.095+0000 7f7e50763640 1 -- 172.21.15.27:0/3300843800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48100060 msgr2=0x7f7e48199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:48.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.095+0000 7f7e50763640 1 --2- 172.21.15.27:0/3300843800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48100060 0x7f7e48199c70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7e44031440 tx=0x7f7e44004270 comp rx=0 tx=0).stop 2024-04-25T00:15:48.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.095+0000 7f7e50763640 1 -- 172.21.15.27:0/3300843800 shutdown_connections 2024-04-25T00:15:48.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.095+0000 7f7e50763640 1 --2- 172.21.15.27:0/3300843800 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7e340a2e10 0x7f7e340a52d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.095+0000 7f7e50763640 1 --2- 172.21.15.27:0/3300843800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e48101360 0x7f7e481a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.095+0000 7f7e50763640 1 --2- 172.21.15.27:0/3300843800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e481009c0 0x7f7e4819a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.095+0000 7f7e50763640 1 --2- 172.21.15.27:0/3300843800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48100060 0x7f7e48199c70 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.095+0000 7f7e50763640 1 -- 172.21.15.27:0/3300843800 >> 172.21.15.27:0/3300843800 conn(0x7f7e480fbc20 msgr2=0x7f7e480fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:48.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.096+0000 7f7e50763640 1 -- 172.21.15.27:0/3300843800 shutdown_connections 2024-04-25T00:15:48.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.096+0000 7f7e50763640 1 -- 172.21.15.27:0/3300843800 wait complete. 2024-04-25T00:15:48.107 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:08:34.006973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T00:13:17.337307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T00:13:18.253713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T00:13:21.995987+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T00:15:48.108 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:15:48.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.218+0000 7f9677de7640 1 -- 172.21.15.27:0/2253446934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96701006a0 msgr2=0x7f9670100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:48.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.218+0000 7f9677de7640 1 --2- 172.21.15.27:0/2253446934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96701006a0 0x7f9670100ac0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f966c009900 tx=0x7f966c02a410 comp rx=0 tx=0).stop 2024-04-25T00:15:48.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.218+0000 7f9677de7640 1 -- 172.21.15.27:0/2253446934 shutdown_connections 2024-04-25T00:15:48.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.218+0000 7f9677de7640 1 --2- 172.21.15.27:0/2253446934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9670102250 0x7f9670108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.218+0000 7f9677de7640 1 --2- 172.21.15.27:0/2253446934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670101890 0x7f9670101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.218+0000 7f9677de7640 1 --2- 172.21.15.27:0/2253446934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96701006a0 0x7f9670100ac0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.218+0000 7f9677de7640 1 -- 172.21.15.27:0/2253446934 >> 172.21.15.27:0/2253446934 conn(0x7f96700fbc30 msgr2=0x7f96700fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.218+0000 7f9677de7640 1 -- 172.21.15.27:0/2253446934 shutdown_connections 2024-04-25T00:15:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.218+0000 7f9677de7640 1 -- 172.21.15.27:0/2253446934 wait complete. 2024-04-25T00:15:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.219+0000 7f9677de7640 1 Processor -- start 2024-04-25T00:15:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.219+0000 7f9677de7640 1 -- start start 2024-04-25T00:15:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.219+0000 7f9677de7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96701006a0 0x7f967006a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.219+0000 7f9677de7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9670101890 0x7f967019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.219+0000 7f9677de7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670102250 0x7f967019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.219+0000 7f9677de7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f967010c2f0 con 0x7f9670101890 2024-04-25T00:15:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.219+0000 7f9677de7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f967010c170 con 0x7f96701006a0 2024-04-25T00:15:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.219+0000 7f9677de7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f967010c470 con 0x7f9670102250 2024-04-25T00:15:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.220+0000 7f9675382640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9670101890 0x7f967019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.220+0000 7f9675b83640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96701006a0 0x7f967006a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.220+0000 7f9675382640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9670101890 0x7f967019c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47598/0 (socket says 172.21.15.27:47598) 2024-04-25T00:15:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.220+0000 7f9675382640 1 -- 172.21.15.27:0/221768785 learned_addr learned my addr 172.21.15.27:0/221768785 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.220+0000 7f9676384640 1 --2- 172.21.15.27:0/221768785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670102250 0x7f967019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.220+0000 7f9675382640 1 -- 172.21.15.27:0/221768785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670102250 msgr2=0x7f967019eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.220+0000 7f9675382640 1 --2- 172.21.15.27:0/221768785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670102250 0x7f967019eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.220+0000 7f9675382640 1 -- 172.21.15.27:0/221768785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96701006a0 msgr2=0x7f967006a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.220+0000 7f9675382640 1 --2- 172.21.15.27:0/221768785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96701006a0 0x7f967006a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.220+0000 7f9675382640 1 -- 172.21.15.27:0/221768785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f967019f7d0 con 0x7f9670101890 2024-04-25T00:15:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.220+0000 7f9675b83640 1 --2- 172.21.15.27:0/221768785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96701006a0 0x7f967006a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.220+0000 7f9675382640 1 --2- 172.21.15.27:0/221768785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9670101890 0x7f967019c8a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f966000e9f0 tx=0x7f966000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.220+0000 7f9666ffd640 1 -- 172.21.15.27:0/221768785 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966000cd80 con 0x7f9670101890 2024-04-25T00:15:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.220+0000 7f9677de7640 1 -- 172.21.15.27:0/221768785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f967019fa60 con 0x7f9670101890 2024-04-25T00:15:48.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.220+0000 7f9666ffd640 1 -- 172.21.15.27:0/221768785 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9660010910 con 0x7f9670101890 2024-04-25T00:15:48.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.221+0000 7f9666ffd640 1 -- 172.21.15.27:0/221768785 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96600188f0 con 0x7f9670101890 2024-04-25T00:15:48.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.221+0000 7f9677de7640 1 -- 172.21.15.27:0/221768785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96701a0000 con 0x7f9670101890 2024-04-25T00:15:48.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.222+0000 7f9677de7640 1 -- 172.21.15.27:0/221768785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9670107a60 con 0x7f9670101890 2024-04-25T00:15:48.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.223+0000 7f9666ffd640 1 -- 172.21.15.27:0/221768785 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f9660010ac0 con 0x7f9670101890 2024-04-25T00:15:48.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.223+0000 7f9666ffd640 1 --2- 172.21.15.27:0/221768785 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f965c0a2e10 0x7f965c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:48.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.224+0000 7f9675b83640 1 --2- 172.21.15.27:0/221768785 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f965c0a2e10 0x7f965c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:48.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.224+0000 7f9666ffd640 1 -- 172.21.15.27:0/221768785 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f96600cfc70 con 0x7f9670101890 2024-04-25T00:15:48.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.224+0000 7f9675b83640 1 --2- 172.21.15.27:0/221768785 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f965c0a2e10 0x7f965c0a52d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f966c009230 tx=0x7f966c0315b0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.228+0000 7f9666ffd640 1 -- 172.21.15.27:0/221768785 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f966009bd60 con 0x7f9670101890 2024-04-25T00:15:48.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.418+0000 7f9677de7640 1 -- 172.21.15.27:0/221768785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f967002d320 con 0x7f9670101890 2024-04-25T00:15:48.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.418+0000 7f9666ffd640 1 -- 172.21.15.27:0/221768785 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f96600a0060 con 0x7f9670101890 2024-04-25T00:15:48.419 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T00:15:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.421+0000 7f9677de7640 1 -- 172.21.15.27:0/221768785 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f965c0a2e10 msgr2=0x7f965c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.421+0000 7f9677de7640 1 --2- 172.21.15.27:0/221768785 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f965c0a2e10 0x7f965c0a52d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f966c009230 tx=0x7f966c0315b0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.421+0000 7f9677de7640 1 -- 172.21.15.27:0/221768785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9670101890 msgr2=0x7f967019c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.421+0000 7f9677de7640 1 --2- 172.21.15.27:0/221768785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9670101890 0x7f967019c8a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f966000e9f0 tx=0x7f966000eeb0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.421+0000 7f9677de7640 1 -- 172.21.15.27:0/221768785 shutdown_connections 2024-04-25T00:15:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.421+0000 7f9677de7640 1 --2- 172.21.15.27:0/221768785 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f965c0a2e10 0x7f965c0a52d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.421+0000 7f9677de7640 1 --2- 172.21.15.27:0/221768785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670102250 0x7f967019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.421+0000 7f9677de7640 1 --2- 172.21.15.27:0/221768785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9670101890 0x7f967019c8a0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.421+0000 7f9677de7640 1 --2- 172.21.15.27:0/221768785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96701006a0 0x7f967006a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.421+0000 7f9677de7640 1 -- 172.21.15.27:0/221768785 >> 172.21.15.27:0/221768785 conn(0x7f96700fbc30 msgr2=0x7f9670104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:48.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.421+0000 7f9677de7640 1 -- 172.21.15.27:0/221768785 shutdown_connections 2024-04-25T00:15:48.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.421+0000 7f9677de7640 1 -- 172.21.15.27:0/221768785 wait complete. 2024-04-25T00:15:48.432 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:48 smithi027 ceph-mon[29208]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:15:48.432 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:48 smithi027 ceph-mon[29208]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-25T00:15:48.432 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:48 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1663156552' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:48.433 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:48 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.433 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:48 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.433 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:48 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.433 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:48 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.433 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:48 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.433 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:48 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.433 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:48 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1189263107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:48.433 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:48 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4073110288' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:48.434 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T00:15:48.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:48 smithi139 ceph-mon[32833]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:15:48.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:48 smithi139 ceph-mon[32833]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-25T00:15:48.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:48 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1663156552' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:48.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:48 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:48 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:48 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:48 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:48 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:48 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:48 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1189263107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:48.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:48 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4073110288' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:48.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:48 smithi153 ceph-mon[32896]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:15:48.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:48 smithi153 ceph-mon[32896]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-25T00:15:48.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:48 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1663156552' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:48.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:48 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:48 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:48 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:48 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:48 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:48 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:48.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:48 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1189263107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:48.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:48 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4073110288' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.544+0000 7fdab97e7640 1 -- 172.21.15.27:0/1774788265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab4102230 msgr2=0x7fdab4108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.544+0000 7fdab97e7640 1 --2- 172.21.15.27:0/1774788265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab4102230 0x7fdab4108ac0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fdaac009ca0 tx=0x7fdaac02a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.545+0000 7fdab97e7640 1 -- 172.21.15.27:0/1774788265 shutdown_connections 2024-04-25T00:15:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.545+0000 7fdab97e7640 1 --2- 172.21.15.27:0/1774788265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab4102230 0x7fdab4108ac0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.545+0000 7fdab97e7640 1 --2- 172.21.15.27:0/1774788265 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdab4101870 0x7fdab4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.545+0000 7fdab97e7640 1 --2- 172.21.15.27:0/1774788265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdab4100680 0x7fdab4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.545+0000 7fdab97e7640 1 -- 172.21.15.27:0/1774788265 >> 172.21.15.27:0/1774788265 conn(0x7fdab40fbbf0 msgr2=0x7fdab40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.545+0000 7fdab97e7640 1 -- 172.21.15.27:0/1774788265 shutdown_connections 2024-04-25T00:15:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.545+0000 7fdab97e7640 1 -- 172.21.15.27:0/1774788265 wait complete. 2024-04-25T00:15:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.545+0000 7fdab97e7640 1 Processor -- start 2024-04-25T00:15:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.545+0000 7fdab97e7640 1 -- start start 2024-04-25T00:15:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.545+0000 7fdab97e7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdab4100680 0x7fdab4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.545+0000 7fdab97e7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab4101870 0x7fdab4198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fdab97e7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdab4102230 0x7fdab419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fdab97e7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdab410c350 con 0x7fdab4101870 2024-04-25T00:15:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fdab97e7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdab410c1d0 con 0x7fdab4100680 2024-04-25T00:15:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fdab97e7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdab410c4d0 con 0x7fdab4102230 2024-04-25T00:15:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fdab27fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab4101870 0x7fdab4198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fdab27fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab4101870 0x7fdab4198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47616/0 (socket says 172.21.15.27:47616) 2024-04-25T00:15:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fdab27fc640 1 -- 172.21.15.27:0/2630269116 learned_addr learned my addr 172.21.15.27:0/2630269116 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fdab37fe640 1 --2- 172.21.15.27:0/2630269116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdab4102230 0x7fdab419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fdab2ffd640 1 --2- 172.21.15.27:0/2630269116 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdab4100680 0x7fdab4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fdab27fc640 1 -- 172.21.15.27:0/2630269116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdab4102230 msgr2=0x7fdab419f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fdab27fc640 1 --2- 172.21.15.27:0/2630269116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdab4102230 0x7fdab419f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fdab27fc640 1 -- 172.21.15.27:0/2630269116 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdab4100680 msgr2=0x7fdab4197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fdab27fc640 1 --2- 172.21.15.27:0/2630269116 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdab4100680 0x7fdab4197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fdab27fc640 1 -- 172.21.15.27:0/2630269116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab419f710 con 0x7fdab4101870 2024-04-25T00:15:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fdab37fe640 1 --2- 172.21.15.27:0/2630269116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdab4102230 0x7fdab419f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fdab27fc640 1 --2- 172.21.15.27:0/2630269116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab4101870 0x7fdab4198030 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fdaa800d950 tx=0x7fdaa800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fda9bfff640 1 -- 172.21.15.27:0/2630269116 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8017070 con 0x7fdab4101870 2024-04-25T00:15:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fdab97e7640 1 -- 172.21.15.27:0/2630269116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdab419fa00 con 0x7fdab4101870 2024-04-25T00:15:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fda9bfff640 1 -- 172.21.15.27:0/2630269116 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdaa8009c30 con 0x7fdab4101870 2024-04-25T00:15:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.546+0000 7fda9bfff640 1 -- 172.21.15.27:0/2630269116 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa800f6b0 con 0x7fdab4101870 2024-04-25T00:15:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.547+0000 7fdab97e7640 1 -- 172.21.15.27:0/2630269116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdab419ff50 con 0x7fdab4101870 2024-04-25T00:15:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.548+0000 7fdab97e7640 1 -- 172.21.15.27:0/2630269116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdab41a0180 con 0x7fdab4101870 2024-04-25T00:15:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.549+0000 7fda9bfff640 1 -- 172.21.15.27:0/2630269116 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7fdaa80109d0 con 0x7fdab4101870 2024-04-25T00:15:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.549+0000 7fda9bfff640 1 --2- 172.21.15.27:0/2630269116 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fda9c0a2d90 0x7fda9c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.550+0000 7fdab2ffd640 1 --2- 172.21.15.27:0/2630269116 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fda9c0a2d90 0x7fda9c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.550+0000 7fda9bfff640 1 -- 172.21.15.27:0/2630269116 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fdaa80cf5b0 con 0x7fdab4101870 2024-04-25T00:15:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.550+0000 7fdab2ffd640 1 --2- 172.21.15.27:0/2630269116 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fda9c0a2d90 0x7fda9c0a5250 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdab41016d0 tx=0x7fdaa4005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:48.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.554+0000 7fda9bfff640 1 -- 172.21.15.27:0/2630269116 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdaa8009de0 con 0x7fdab4101870 2024-04-25T00:15:48.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.764+0000 7fdab97e7640 1 -- 172.21.15.27:0/2630269116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdab41a0350 con 0x7fdab4101870 2024-04-25T00:15:48.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.765+0000 7fda9bfff640 1 -- 172.21.15.27:0/2630269116 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fdaa809bcf0 con 0x7fdab4101870 2024-04-25T00:15:48.765 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:15:48.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.767+0000 7fdab97e7640 1 -- 172.21.15.27:0/2630269116 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fda9c0a2d90 msgr2=0x7fda9c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:48.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.767+0000 7fdab97e7640 1 --2- 172.21.15.27:0/2630269116 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fda9c0a2d90 0x7fda9c0a5250 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdab41016d0 tx=0x7fdaa4005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:48.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.767+0000 7fdab97e7640 1 -- 172.21.15.27:0/2630269116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab4101870 msgr2=0x7fdab4198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:48.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.767+0000 7fdab97e7640 1 --2- 172.21.15.27:0/2630269116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab4101870 0x7fdab4198030 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fdaa800d950 tx=0x7fdaa800de10 comp rx=0 tx=0).stop 2024-04-25T00:15:48.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.767+0000 7fdab97e7640 1 -- 172.21.15.27:0/2630269116 shutdown_connections 2024-04-25T00:15:48.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.767+0000 7fdab97e7640 1 --2- 172.21.15.27:0/2630269116 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fda9c0a2d90 0x7fda9c0a5250 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.767+0000 7fdab97e7640 1 --2- 172.21.15.27:0/2630269116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdab4102230 0x7fdab419f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.767+0000 7fdab97e7640 1 --2- 172.21.15.27:0/2630269116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab4101870 0x7fdab4198030 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.767+0000 7fdab97e7640 1 --2- 172.21.15.27:0/2630269116 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdab4100680 0x7fdab4197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.767+0000 7fdab97e7640 1 -- 172.21.15.27:0/2630269116 >> 172.21.15.27:0/2630269116 conn(0x7fdab40fbbf0 msgr2=0x7fdab40fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:48.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.767+0000 7fdab97e7640 1 -- 172.21.15.27:0/2630269116 shutdown_connections 2024-04-25T00:15:48.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.767+0000 7fdab97e7640 1 -- 172.21.15.27:0/2630269116 wait complete. 2024-04-25T00:15:48.779 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T00:15:48.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.885+0000 7f8ccb1a2640 1 -- 172.21.15.27:0/4139596902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cc4100680 msgr2=0x7f8cc4100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:48.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.885+0000 7f8ccb1a2640 1 --2- 172.21.15.27:0/4139596902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cc4100680 0x7f8cc4100aa0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8009900 tx=0x7f8cb802a410 comp rx=0 tx=0).stop 2024-04-25T00:15:48.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.885+0000 7f8ccb1a2640 1 -- 172.21.15.27:0/4139596902 shutdown_connections 2024-04-25T00:15:48.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.885+0000 7f8ccb1a2640 1 --2- 172.21.15.27:0/4139596902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cc4102230 0x7f8cc4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.885+0000 7f8ccb1a2640 1 --2- 172.21.15.27:0/4139596902 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8cc4101870 0x7f8cc4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.885+0000 7f8ccb1a2640 1 --2- 172.21.15.27:0/4139596902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cc4100680 0x7f8cc4100aa0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.885+0000 7f8ccb1a2640 1 -- 172.21.15.27:0/4139596902 >> 172.21.15.27:0/4139596902 conn(0x7f8cc40fbbf0 msgr2=0x7f8cc40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:48.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.885+0000 7f8ccb1a2640 1 -- 172.21.15.27:0/4139596902 shutdown_connections 2024-04-25T00:15:48.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.885+0000 7f8ccb1a2640 1 -- 172.21.15.27:0/4139596902 wait complete. 2024-04-25T00:15:48.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.886+0000 7f8ccb1a2640 1 Processor -- start 2024-04-25T00:15:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.886+0000 7f8ccb1a2640 1 -- start start 2024-04-25T00:15:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.886+0000 7f8ccb1a2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cc4100680 0x7f8cc4104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.886+0000 7f8ccb1a2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cc4101870 0x7f8cc4102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.886+0000 7f8ccb1a2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8cc4102230 0x7f8cc4103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.886+0000 7f8ccb1a2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc410c300 con 0x7f8cc4100680 2024-04-25T00:15:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.886+0000 7f8ccb1a2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc410c180 con 0x7f8cc4102230 2024-04-25T00:15:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.886+0000 7f8ccb1a2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc410c480 con 0x7f8cc4101870 2024-04-25T00:15:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.886+0000 7f8cc8f3e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cc4100680 0x7f8cc4104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.886+0000 7f8cc8f3e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cc4100680 0x7f8cc4104610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47630/0 (socket says 172.21.15.27:47630) 2024-04-25T00:15:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.886+0000 7f8cc8f3e640 1 -- 172.21.15.27:0/3873171947 learned_addr learned my addr 172.21.15.27:0/3873171947 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.886+0000 7f8cc973f640 1 --2- 172.21.15.27:0/3873171947 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8cc4102230 0x7f8cc4103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.887+0000 7f8cc3fff640 1 --2- 172.21.15.27:0/3873171947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cc4101870 0x7f8cc4102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.887+0000 7f8cc8f3e640 1 -- 172.21.15.27:0/3873171947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cc4101870 msgr2=0x7f8cc4102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.887+0000 7f8cc8f3e640 1 --2- 172.21.15.27:0/3873171947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cc4101870 0x7f8cc4102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.887+0000 7f8cc8f3e640 1 -- 172.21.15.27:0/3873171947 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8cc4102230 msgr2=0x7f8cc4103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.887+0000 7f8cc8f3e640 1 --2- 172.21.15.27:0/3873171947 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8cc4102230 0x7f8cc4103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.887+0000 7f8cc8f3e640 1 -- 172.21.15.27:0/3873171947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc4103a20 con 0x7f8cc4100680 2024-04-25T00:15:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.887+0000 7f8cc8f3e640 1 --2- 172.21.15.27:0/3873171947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cc4100680 0x7f8cc4104610 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8031610 tx=0x7f8cb80043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.887+0000 7f8cc1ffb640 1 -- 172.21.15.27:0/3873171947 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb803f030 con 0x7f8cc4100680 2024-04-25T00:15:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.887+0000 7f8cc1ffb640 1 -- 172.21.15.27:0/3873171947 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cb8003ce0 con 0x7f8cc4100680 2024-04-25T00:15:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.887+0000 7f8ccb1a2640 1 -- 172.21.15.27:0/3873171947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cc41a1bc0 con 0x7f8cc4100680 2024-04-25T00:15:48.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.887+0000 7f8cc1ffb640 1 -- 172.21.15.27:0/3873171947 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8035070 con 0x7f8cc4100680 2024-04-25T00:15:48.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.887+0000 7f8ccb1a2640 1 -- 172.21.15.27:0/3873171947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cc41a2160 con 0x7f8cc4100680 2024-04-25T00:15:48.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.889+0000 7f8ccb1a2640 1 -- 172.21.15.27:0/3873171947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cc41a2330 con 0x7f8cc4100680 2024-04-25T00:15:48.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.889+0000 7f8cc1ffb640 1 -- 172.21.15.27:0/3873171947 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f8cb8042030 con 0x7f8cc4100680 2024-04-25T00:15:48.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.890+0000 7f8cc1ffb640 1 --2- 172.21.15.27:0/3873171947 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8cb40a2e10 0x7f8cb40a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:48.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.891+0000 7f8cc3fff640 1 --2- 172.21.15.27:0/3873171947 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8cb40a2e10 0x7f8cb40a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:48.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.891+0000 7f8cc1ffb640 1 -- 172.21.15.27:0/3873171947 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8cb80efe00 con 0x7f8cc4100680 2024-04-25T00:15:48.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.891+0000 7f8cc3fff640 1 --2- 172.21.15.27:0/3873171947 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8cb40a2e10 0x7f8cb40a52d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4104380 tx=0x7f8cb000b410 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:48.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:48.895+0000 7f8cc1ffb640 1 -- 172.21.15.27:0/3873171947 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8cb80bbef0 con 0x7f8cc4100680 2024-04-25T00:15:49.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.095+0000 7f8ccb1a2640 1 -- 172.21.15.27:0/3873171947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8cc41a2500 con 0x7f8cc4100680 2024-04-25T00:15:49.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.096+0000 7f8cc1ffb640 1 -- 172.21.15.27:0/3873171947 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f8cb80c01f0 con 0x7f8cc4100680 2024-04-25T00:15:49.096 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T00:15:49.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.098+0000 7f8ccb1a2640 1 -- 172.21.15.27:0/3873171947 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8cb40a2e10 msgr2=0x7f8cb40a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.098+0000 7f8ccb1a2640 1 --2- 172.21.15.27:0/3873171947 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8cb40a2e10 0x7f8cb40a52d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4104380 tx=0x7f8cb000b410 comp rx=0 tx=0).stop 2024-04-25T00:15:49.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.098+0000 7f8ccb1a2640 1 -- 172.21.15.27:0/3873171947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cc4100680 msgr2=0x7f8cc4104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.098+0000 7f8ccb1a2640 1 --2- 172.21.15.27:0/3873171947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cc4100680 0x7f8cc4104610 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8031610 tx=0x7f8cb80043b0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.098+0000 7f8ccb1a2640 1 -- 172.21.15.27:0/3873171947 shutdown_connections 2024-04-25T00:15:49.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.098+0000 7f8ccb1a2640 1 --2- 172.21.15.27:0/3873171947 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8cb40a2e10 0x7f8cb40a52d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.098+0000 7f8ccb1a2640 1 --2- 172.21.15.27:0/3873171947 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8cc4102230 0x7f8cc4103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.098+0000 7f8ccb1a2640 1 --2- 172.21.15.27:0/3873171947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cc4101870 0x7f8cc4102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.098+0000 7f8ccb1a2640 1 --2- 172.21.15.27:0/3873171947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cc4100680 0x7f8cc4104610 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.098+0000 7f8ccb1a2640 1 -- 172.21.15.27:0/3873171947 >> 172.21.15.27:0/3873171947 conn(0x7f8cc40fbbf0 msgr2=0x7f8cc40fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:49.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.098+0000 7f8ccb1a2640 1 -- 172.21.15.27:0/3873171947 shutdown_connections 2024-04-25T00:15:49.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.098+0000 7f8ccb1a2640 1 -- 172.21.15.27:0/3873171947 wait complete. 2024-04-25T00:15:49.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T00:15:49.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.220+0000 7faba38d6640 1 -- 172.21.15.27:0/3317286646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab9c101870 msgr2=0x7fab9c101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.220+0000 7faba38d6640 1 --2- 172.21.15.27:0/3317286646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab9c101870 0x7fab9c101cf0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fab8c009900 tx=0x7fab8c02a410 comp rx=0 tx=0).stop 2024-04-25T00:15:49.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.220+0000 7faba38d6640 1 -- 172.21.15.27:0/3317286646 shutdown_connections 2024-04-25T00:15:49.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.220+0000 7faba38d6640 1 --2- 172.21.15.27:0/3317286646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab9c102230 0x7fab9c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.220+0000 7faba38d6640 1 --2- 172.21.15.27:0/3317286646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab9c101870 0x7fab9c101cf0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.220+0000 7faba38d6640 1 --2- 172.21.15.27:0/3317286646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fab9c100680 0x7fab9c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.220+0000 7faba38d6640 1 -- 172.21.15.27:0/3317286646 >> 172.21.15.27:0/3317286646 conn(0x7fab9c0fbbf0 msgr2=0x7fab9c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:49.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.220+0000 7faba38d6640 1 -- 172.21.15.27:0/3317286646 shutdown_connections 2024-04-25T00:15:49.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.220+0000 7faba38d6640 1 -- 172.21.15.27:0/3317286646 wait complete. 2024-04-25T00:15:49.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.221+0000 7faba38d6640 1 Processor -- start 2024-04-25T00:15:49.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.221+0000 7faba38d6640 1 -- start start 2024-04-25T00:15:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.221+0000 7faba38d6640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fab9c100680 0x7fab9c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.221+0000 7faba38d6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab9c101870 0x7fab9c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.221+0000 7faba38d6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab9c102230 0x7fab9c19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.221+0000 7faba38d6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c10c330 con 0x7fab9c101870 2024-04-25T00:15:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.221+0000 7faba38d6640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c10c1b0 con 0x7fab9c100680 2024-04-25T00:15:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.221+0000 7faba38d6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c10c4b0 con 0x7fab9c102230 2024-04-25T00:15:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.221+0000 7faba0e71640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab9c101870 0x7fab9c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.221+0000 7faba0e71640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab9c101870 0x7fab9c198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47642/0 (socket says 172.21.15.27:47642) 2024-04-25T00:15:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.221+0000 7faba1672640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fab9c100680 0x7fab9c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.221+0000 7faba0e71640 1 -- 172.21.15.27:0/3500465566 learned_addr learned my addr 172.21.15.27:0/3500465566 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.221+0000 7faba1e73640 1 --2- 172.21.15.27:0/3500465566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab9c102230 0x7fab9c19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.222+0000 7faba0e71640 1 -- 172.21.15.27:0/3500465566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab9c102230 msgr2=0x7fab9c19f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.222+0000 7faba0e71640 1 --2- 172.21.15.27:0/3500465566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab9c102230 0x7fab9c19f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.222+0000 7faba0e71640 1 -- 172.21.15.27:0/3500465566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fab9c100680 msgr2=0x7fab9c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.222+0000 7faba0e71640 1 --2- 172.21.15.27:0/3500465566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fab9c100680 0x7fab9c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.222+0000 7faba0e71640 1 -- 172.21.15.27:0/3500465566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9c19f740 con 0x7fab9c101870 2024-04-25T00:15:49.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.222+0000 7faba1672640 1 --2- 172.21.15.27:0/3500465566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fab9c100680 0x7fab9c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:49.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.222+0000 7faba0e71640 1 --2- 172.21.15.27:0/3500465566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab9c101870 0x7fab9c198060 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fab8c031440 tx=0x7fab8c032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:49.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.222+0000 7fab927fc640 1 -- 172.21.15.27:0/3500465566 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8c03f030 con 0x7fab9c101870 2024-04-25T00:15:49.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.222+0000 7fab927fc640 1 -- 172.21.15.27:0/3500465566 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab8c003ce0 con 0x7fab9c101870 2024-04-25T00:15:49.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.222+0000 7faba38d6640 1 -- 172.21.15.27:0/3500465566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9c19f9d0 con 0x7fab9c101870 2024-04-25T00:15:49.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.222+0000 7fab927fc640 1 -- 172.21.15.27:0/3500465566 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8c035070 con 0x7fab9c101870 2024-04-25T00:15:49.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.222+0000 7faba38d6640 1 -- 172.21.15.27:0/3500465566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab9c19ff10 con 0x7fab9c101870 2024-04-25T00:15:49.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.223+0000 7faba38d6640 1 -- 172.21.15.27:0/3500465566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab9c1a01a0 con 0x7fab9c101870 2024-04-25T00:15:49.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.225+0000 7fab927fc640 1 -- 172.21.15.27:0/3500465566 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7fab8c042030 con 0x7fab9c101870 2024-04-25T00:15:49.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.225+0000 7fab927fc640 1 --2- 172.21.15.27:0/3500465566 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fab880a2e10 0x7fab880a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.226+0000 7faba1672640 1 --2- 172.21.15.27:0/3500465566 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fab880a2e10 0x7fab880a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.226+0000 7fab927fc640 1 -- 172.21.15.27:0/3500465566 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fab8c0f0270 con 0x7fab9c101870 2024-04-25T00:15:49.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.226+0000 7faba1672640 1 --2- 172.21.15.27:0/3500465566 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fab880a2e10 0x7fab880a52d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fab9c1016d0 tx=0x7fab98005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:49.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.230+0000 7fab927fc640 1 -- 172.21.15.27:0/3500465566 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fab8c0bc320 con 0x7fab9c101870 2024-04-25T00:15:49.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.431+0000 7faba38d6640 1 -- 172.21.15.27:0/3500465566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fab9c1a0370 con 0x7fab9c101870 2024-04-25T00:15:49.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.431+0000 7fab927fc640 1 -- 172.21.15.27:0/3500465566 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fab8c0c0620 con 0x7fab9c101870 2024-04-25T00:15:49.431 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:15:49.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.433+0000 7faba38d6640 1 -- 172.21.15.27:0/3500465566 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fab880a2e10 msgr2=0x7fab880a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.433+0000 7faba38d6640 1 --2- 172.21.15.27:0/3500465566 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fab880a2e10 0x7fab880a52d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fab9c1016d0 tx=0x7fab98005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:49.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.433+0000 7faba38d6640 1 -- 172.21.15.27:0/3500465566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab9c101870 msgr2=0x7fab9c198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.433+0000 7faba38d6640 1 --2- 172.21.15.27:0/3500465566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab9c101870 0x7fab9c198060 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fab8c031440 tx=0x7fab8c032ce0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.433+0000 7faba38d6640 1 -- 172.21.15.27:0/3500465566 shutdown_connections 2024-04-25T00:15:49.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.433+0000 7faba38d6640 1 --2- 172.21.15.27:0/3500465566 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fab880a2e10 0x7fab880a52d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.433+0000 7faba38d6640 1 --2- 172.21.15.27:0/3500465566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab9c102230 0x7fab9c19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.433+0000 7faba38d6640 1 --2- 172.21.15.27:0/3500465566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab9c101870 0x7fab9c198060 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.433+0000 7faba38d6640 1 --2- 172.21.15.27:0/3500465566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fab9c100680 0x7fab9c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.433+0000 7faba38d6640 1 -- 172.21.15.27:0/3500465566 >> 172.21.15.27:0/3500465566 conn(0x7fab9c0fbbf0 msgr2=0x7fab9c0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:49.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.433+0000 7faba38d6640 1 -- 172.21.15.27:0/3500465566 shutdown_connections 2024-04-25T00:15:49.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.433+0000 7faba38d6640 1 -- 172.21.15.27:0/3500465566 wait complete. 2024-04-25T00:15:49.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3300843800' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:49.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-25T00:15:49.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: Unable to set osd_memory_target on smithi153 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-25T00:15:49.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi139 to 176.2M 2024-04-25T00:15:49.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: Unable to set osd_memory_target on smithi139 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T00:15:49.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/221768785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:49.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi027 to 176.2M 2024-04-25T00:15:49.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: Unable to set osd_memory_target on smithi027 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-04-25T00:15:49.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:15:49.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:15:49.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T00:15:49.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: Updating smithi139:/etc/ceph/ceph.conf 2024-04-25T00:15:49.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-25T00:15:49.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: Updating smithi139:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:15:49.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: Updating smithi027:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:15:49.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: Updating smithi153:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:15:49.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:49 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2630269116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:49.446 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-25T00:15:49.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:49.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3300843800' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:49.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-25T00:15:49.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: Unable to set osd_memory_target on smithi153 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-25T00:15:49.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi139 to 176.2M 2024-04-25T00:15:49.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: Unable to set osd_memory_target on smithi139 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T00:15:49.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/221768785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:49.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi027 to 176.2M 2024-04-25T00:15:49.506 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: Unable to set osd_memory_target on smithi027 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-04-25T00:15:49.506 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:15:49.506 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:15:49.506 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T00:15:49.506 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: Updating smithi139:/etc/ceph/ceph.conf 2024-04-25T00:15:49.506 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-25T00:15:49.506 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: Updating smithi139:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:15:49.506 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: Updating smithi027:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:15:49.506 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: Updating smithi153:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:15:49.507 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:49 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2630269116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:49.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3300843800' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:49.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-25T00:15:49.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: Unable to set osd_memory_target on smithi153 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-25T00:15:49.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi139 to 176.2M 2024-04-25T00:15:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: Unable to set osd_memory_target on smithi139 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T00:15:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/221768785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:15:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi027 to 176.2M 2024-04-25T00:15:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: Unable to set osd_memory_target on smithi027 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-04-25T00:15:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:15:49.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:15:49.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T00:15:49.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: Updating smithi139:/etc/ceph/ceph.conf 2024-04-25T00:15:49.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-25T00:15:49.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: Updating smithi139:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:15:49.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: Updating smithi027:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:15:49.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: Updating smithi153:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T00:15:49.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:49 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2630269116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:49.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.556+0000 7fefcd514640 1 -- 172.21.15.27:0/421554782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefc81032f0 msgr2=0x7fefc810b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.556+0000 7fefcd514640 1 --2- 172.21.15.27:0/421554782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefc81032f0 0x7fefc810b7d0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fefb0009900 tx=0x7fefb002a510 comp rx=0 tx=0).stop 2024-04-25T00:15:49.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.556+0000 7fefcd514640 1 -- 172.21.15.27:0/421554782 shutdown_connections 2024-04-25T00:15:49.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.556+0000 7fefcd514640 1 --2- 172.21.15.27:0/421554782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc810bd10 0x7fefc810e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.556+0000 7fefcd514640 1 --2- 172.21.15.27:0/421554782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefc81032f0 0x7fefc810b7d0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.556+0000 7fefcd514640 1 --2- 172.21.15.27:0/421554782 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fefc81029d0 0x7fefc8102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.556+0000 7fefcd514640 1 -- 172.21.15.27:0/421554782 >> 172.21.15.27:0/421554782 conn(0x7fefc8077740 msgr2=0x7fefc8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:49.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.556+0000 7fefcd514640 1 -- 172.21.15.27:0/421554782 shutdown_connections 2024-04-25T00:15:49.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.556+0000 7fefcd514640 1 -- 172.21.15.27:0/421554782 wait complete. 2024-04-25T00:15:49.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.556+0000 7fefcd514640 1 Processor -- start 2024-04-25T00:15:49.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.556+0000 7fefcd514640 1 -- start start 2024-04-25T00:15:49.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.557+0000 7fefcd514640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc81029d0 0x7fefc8100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.557+0000 7fefcd514640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefc81032f0 0x7fefc8100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.557+0000 7fefcd514640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fefc810bd10 0x7fefc81010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.557+0000 7fefcd514640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fefc8111b10 con 0x7fefc81032f0 2024-04-25T00:15:49.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.557+0000 7fefcd514640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fefc8111990 con 0x7fefc810bd10 2024-04-25T00:15:49.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.557+0000 7fefcd514640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fefc8111c90 con 0x7fefc81029d0 2024-04-25T00:15:49.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.557+0000 7fefbffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefc81032f0 0x7fefc8100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.557+0000 7fefbffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefc81032f0 0x7fefc8100b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47660/0 (socket says 172.21.15.27:47660) 2024-04-25T00:15:49.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.557+0000 7fefbffff640 1 -- 172.21.15.27:0/687009595 learned_addr learned my addr 172.21.15.27:0/687009595 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:49.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.557+0000 7fefc77fe640 1 --2- 172.21.15.27:0/687009595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fefc810bd10 0x7fefc81010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.557+0000 7fefc6ffd640 1 --2- 172.21.15.27:0/687009595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc81029d0 0x7fefc8100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.557+0000 7fefbffff640 1 -- 172.21.15.27:0/687009595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc81029d0 msgr2=0x7fefc8100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.557+0000 7fefbffff640 1 --2- 172.21.15.27:0/687009595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc81029d0 0x7fefc8100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.557+0000 7fefbffff640 1 -- 172.21.15.27:0/687009595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fefc810bd10 msgr2=0x7fefc81010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.557+0000 7fefbffff640 1 --2- 172.21.15.27:0/687009595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fefc810bd10 0x7fefc81010a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.557+0000 7fefbffff640 1 -- 172.21.15.27:0/687009595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc81a64f0 con 0x7fefc81032f0 2024-04-25T00:15:49.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.557+0000 7fefbffff640 1 --2- 172.21.15.27:0/687009595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefc81032f0 0x7fefc8100b60 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fefb0006010 tx=0x7fefb0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:49.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.559+0000 7fefc4ff9640 1 -- 172.21.15.27:0/687009595 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb003f030 con 0x7fefc81032f0 2024-04-25T00:15:49.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.559+0000 7fefcd514640 1 -- 172.21.15.27:0/687009595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefc81a6780 con 0x7fefc81032f0 2024-04-25T00:15:49.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.559+0000 7fefcd514640 1 -- 172.21.15.27:0/687009595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefc81a6cc0 con 0x7fefc81032f0 2024-04-25T00:15:49.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.560+0000 7fefc4ff9640 1 -- 172.21.15.27:0/687009595 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb0004440 con 0x7fefc81032f0 2024-04-25T00:15:49.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.560+0000 7fefc4ff9640 1 -- 172.21.15.27:0/687009595 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb0049060 con 0x7fefc81032f0 2024-04-25T00:15:49.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.560+0000 7fefcd514640 1 -- 172.21.15.27:0/687009595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefa8005290 con 0x7fefc81032f0 2024-04-25T00:15:49.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.562+0000 7fefc4ff9640 1 -- 172.21.15.27:0/687009595 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fefb0042030 con 0x7fefc81032f0 2024-04-25T00:15:49.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.562+0000 7fefc4ff9640 1 --2- 172.21.15.27:0/687009595 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fefb40a2ea0 0x7fefb40a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.562+0000 7fefc6ffd640 1 --2- 172.21.15.27:0/687009595 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fefb40a2ea0 0x7fefb40a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.562+0000 7fefc4ff9640 1 -- 172.21.15.27:0/687009595 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fefb00efe30 con 0x7fefc81032f0 2024-04-25T00:15:49.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.563+0000 7fefc6ffd640 1 --2- 172.21.15.27:0/687009595 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fefb40a2ea0 0x7fefb40a5360 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009bd0 tx=0x7fefb8005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:49.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.566+0000 7fefc4ff9640 1 -- 172.21.15.27:0/687009595 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fefb00bbf10 con 0x7fefc81032f0 2024-04-25T00:15:49.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.792+0000 7fefcd514640 1 -- 172.21.15.27:0/687009595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefa8005580 con 0x7fefc81032f0 2024-04-25T00:15:49.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.793+0000 7fefc4ff9640 1 -- 172.21.15.27:0/687009595 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11944 (secure 0 0 0) 0x7fefb00c0210 con 0x7fefc81032f0 2024-04-25T00:15:49.794 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:49.794 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:15:43.587614+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24514,24430,24521,24428,24496]},"id":1}]} 2024-04-25T00:15:49.795 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 29 2024-04-25T00:15:49.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.795+0000 7fefcd514640 1 -- 172.21.15.27:0/687009595 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fefb40a2ea0 msgr2=0x7fefb40a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.795+0000 7fefcd514640 1 --2- 172.21.15.27:0/687009595 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fefb40a2ea0 0x7fefb40a5360 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009bd0 tx=0x7fefb8005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:49.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.795+0000 7fefcd514640 1 -- 172.21.15.27:0/687009595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefc81032f0 msgr2=0x7fefc8100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.795+0000 7fefcd514640 1 --2- 172.21.15.27:0/687009595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefc81032f0 0x7fefc8100b60 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fefb0006010 tx=0x7fefb0004040 comp rx=0 tx=0).stop 2024-04-25T00:15:49.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.796+0000 7fefcd514640 1 -- 172.21.15.27:0/687009595 shutdown_connections 2024-04-25T00:15:49.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.796+0000 7fefcd514640 1 --2- 172.21.15.27:0/687009595 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fefb40a2ea0 0x7fefb40a5360 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.796+0000 7fefcd514640 1 --2- 172.21.15.27:0/687009595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fefc810bd10 0x7fefc81010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.796+0000 7fefcd514640 1 --2- 172.21.15.27:0/687009595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefc81032f0 0x7fefc8100b60 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.796+0000 7fefcd514640 1 --2- 172.21.15.27:0/687009595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc81029d0 0x7fefc8100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.796+0000 7fefcd514640 1 -- 172.21.15.27:0/687009595 >> 172.21.15.27:0/687009595 conn(0x7fefc8077740 msgr2=0x7fefc80fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:49.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.796+0000 7fefcd514640 1 -- 172.21.15.27:0/687009595 shutdown_connections 2024-04-25T00:15:49.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.796+0000 7fefcd514640 1 -- 172.21.15.27:0/687009595 wait complete. 2024-04-25T00:15:49.807 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-25T00:15:49.807 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-25T00:15:49.807 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:15:49.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.910+0000 7f9f97f63640 1 -- 172.21.15.27:0/508892265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f90106650 msgr2=0x7f9f90106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.910+0000 7f9f97f63640 1 --2- 172.21.15.27:0/508892265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f90106650 0x7f9f90106a30 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009930 tx=0x7f9f8002a420 comp rx=0 tx=0).stop 2024-04-25T00:15:49.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.910+0000 7f9f97f63640 1 -- 172.21.15.27:0/508892265 shutdown_connections 2024-04-25T00:15:49.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.910+0000 7f9f97f63640 1 --2- 172.21.15.27:0/508892265 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f90101050 0x7f9f9010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.910+0000 7f9f97f63640 1 --2- 172.21.15.27:0/508892265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f90100690 0x7f9f90100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.910+0000 7f9f97f63640 1 --2- 172.21.15.27:0/508892265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f90106650 0x7f9f90106a30 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.910+0000 7f9f97f63640 1 -- 172.21.15.27:0/508892265 >> 172.21.15.27:0/508892265 conn(0x7f9f900777a0 msgr2=0x7f9f90077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:49.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.910+0000 7f9f97f63640 1 -- 172.21.15.27:0/508892265 shutdown_connections 2024-04-25T00:15:49.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.910+0000 7f9f97f63640 1 -- 172.21.15.27:0/508892265 wait complete. 2024-04-25T00:15:49.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f97f63640 1 Processor -- start 2024-04-25T00:15:49.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f97f63640 1 -- start start 2024-04-25T00:15:49.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f97f63640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f90100690 0x7f9f90198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f97f63640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f90101050 0x7f9f90199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f97f63640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f90106650 0x7f9f901a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f97f63640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9f90110a60 con 0x7f9f90100690 2024-04-25T00:15:49.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f97f63640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9f901108e0 con 0x7f9f90106650 2024-04-25T00:15:49.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f97f63640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9f90110be0 con 0x7f9f90101050 2024-04-25T00:15:49.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f95cff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f90100690 0x7f9f90198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f95cff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f90100690 0x7f9f90198ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47690/0 (socket says 172.21.15.27:47690) 2024-04-25T00:15:49.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f95cff640 1 -- 172.21.15.27:0/2843812311 learned_addr learned my addr 172.21.15.27:0/2843812311 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:49.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f954fe640 1 --2- 172.21.15.27:0/2843812311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f90101050 0x7f9f90199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f96500640 1 --2- 172.21.15.27:0/2843812311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f90106650 0x7f9f901a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f95cff640 1 -- 172.21.15.27:0/2843812311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f90101050 msgr2=0x7f9f90199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f95cff640 1 --2- 172.21.15.27:0/2843812311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f90101050 0x7f9f90199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f95cff640 1 -- 172.21.15.27:0/2843812311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f90106650 msgr2=0x7f9f901a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f95cff640 1 --2- 172.21.15.27:0/2843812311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f90106650 0x7f9f901a3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f95cff640 1 -- 172.21.15.27:0/2843812311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f901a44a0 con 0x7f9f90100690 2024-04-25T00:15:49.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.911+0000 7f9f954fe640 1 --2- 172.21.15.27:0/2843812311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f90101050 0x7f9f90199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:49.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.912+0000 7f9f95cff640 1 --2- 172.21.15.27:0/2843812311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f90100690 0x7f9f90198ee0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009230 tx=0x7f9f80004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:49.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.912+0000 7f9f86ffd640 1 -- 172.21.15.27:0/2843812311 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8003f030 con 0x7f9f90100690 2024-04-25T00:15:49.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.912+0000 7f9f97f63640 1 -- 172.21.15.27:0/2843812311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f901a4670 con 0x7f9f90100690 2024-04-25T00:15:49.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.912+0000 7f9f86ffd640 1 -- 172.21.15.27:0/2843812311 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f80004440 con 0x7f9f90100690 2024-04-25T00:15:49.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.912+0000 7f9f86ffd640 1 -- 172.21.15.27:0/2843812311 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f80035070 con 0x7f9f90100690 2024-04-25T00:15:49.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.912+0000 7f9f97f63640 1 -- 172.21.15.27:0/2843812311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f901a4b30 con 0x7f9f90100690 2024-04-25T00:15:49.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.913+0000 7f9f97f63640 1 -- 172.21.15.27:0/2843812311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f900689b0 con 0x7f9f90100690 2024-04-25T00:15:49.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.914+0000 7f9f86ffd640 1 -- 172.21.15.27:0/2843812311 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f9f80042030 con 0x7f9f90100690 2024-04-25T00:15:49.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.914+0000 7f9f86ffd640 1 --2- 172.21.15.27:0/2843812311 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9f7c0a2e20 0x7f9f7c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.915+0000 7f9f954fe640 1 --2- 172.21.15.27:0/2843812311 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9f7c0a2e20 0x7f9f7c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.915+0000 7f9f86ffd640 1 -- 172.21.15.27:0/2843812311 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f9f800efd20 con 0x7f9f90100690 2024-04-25T00:15:49.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.915+0000 7f9f954fe640 1 --2- 172.21.15.27:0/2843812311 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9f7c0a2e20 0x7f9f7c0a52e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9f9006a8f0 tx=0x7f9f8c009500 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:49.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:49.919+0000 7f9f86ffd640 1 -- 172.21.15.27:0/2843812311 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9f800bc450 con 0x7f9f90100690 2024-04-25T00:15:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.104+0000 7f9f97f63640 1 -- 172.21.15.27:0/2843812311 --> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9f90061b80 con 0x7f9f7c0a2e20 2024-04-25T00:15:50.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.114+0000 7f9f86ffd640 1 -- 172.21.15.27:0/2843812311 <== mgr.15564 v2:172.21.15.27:6800/1076858552 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339735 (secure 0 0 0) 0x7f9f90061b80 con 0x7f9f7c0a2e20 2024-04-25T00:15:50.119 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:50.121 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T00:15:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.117+0000 7f9f97f63640 1 -- 172.21.15.27:0/2843812311 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9f7c0a2e20 msgr2=0x7f9f7c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.117+0000 7f9f97f63640 1 --2- 172.21.15.27:0/2843812311 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9f7c0a2e20 0x7f9f7c0a52e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9f9006a8f0 tx=0x7f9f8c009500 comp rx=0 tx=0).stop 2024-04-25T00:15:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.117+0000 7f9f97f63640 1 -- 172.21.15.27:0/2843812311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f90100690 msgr2=0x7f9f90198ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.117+0000 7f9f97f63640 1 --2- 172.21.15.27:0/2843812311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f90100690 0x7f9f90198ee0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009230 tx=0x7f9f80004040 comp rx=0 tx=0).stop 2024-04-25T00:15:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.117+0000 7f9f97f63640 1 -- 172.21.15.27:0/2843812311 shutdown_connections 2024-04-25T00:15:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.117+0000 7f9f97f63640 1 --2- 172.21.15.27:0/2843812311 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9f7c0a2e20 0x7f9f7c0a52e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.117+0000 7f9f97f63640 1 --2- 172.21.15.27:0/2843812311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f90106650 0x7f9f901a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.117+0000 7f9f97f63640 1 --2- 172.21.15.27:0/2843812311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f90101050 0x7f9f90199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.117+0000 7f9f97f63640 1 --2- 172.21.15.27:0/2843812311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f90100690 0x7f9f90198ee0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.117+0000 7f9f97f63640 1 -- 172.21.15.27:0/2843812311 >> 172.21.15.27:0/2843812311 conn(0x7f9f900777a0 msgr2=0x7f9f90103310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.117+0000 7f9f97f63640 1 -- 172.21.15.27:0/2843812311 shutdown_connections 2024-04-25T00:15:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.117+0000 7f9f97f63640 1 -- 172.21.15.27:0/2843812311 wait complete. 2024-04-25T00:15:50.128 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-25T00:15:49.051541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190657,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1473,"num_read_kb":902,"num_write":555,"num_write_kb":4406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":939,"ondisk_log_size":939,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":349548,"kb_used_data":12844,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124592276,"statfs":{"total":1151940427776,"available":1151582490624,"internally_reserved":0,"allocated":13152256,"data_stored":9879770,"data_compressed":536900,"data_compressed_allocated":729088,"data_compressed_original":4308992,"omap_allocated":19143,"internal_metadata":344634681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.987382"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.294151+0000","last_change":"2024-04-25T00:13:19.429658+0000","last_active":"2024-04-25T00:15:45.294151+0000","last_peered":"2024-04-25T00:15:45.294151+0000","last_clean":"2024-04-25T00:15:45.294151+0000","last_became_active":"2024-04-25T00:13:19.428426+0000","last_became_peered":"2024-04-25T00:13:19.428426+0000","last_unstale":"2024-04-25T00:15:45.294151+0000","last_undegraded":"2024-04-25T00:15:45.294151+0000","last_fullsized":"2024-04-25T00:15:45.294151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:10.814361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115194+0000","last_change":"2024-04-25T00:13:19.425504+0000","last_active":"2024-04-25T00:15:45.115194+0000","last_peered":"2024-04-25T00:15:45.115194+0000","last_clean":"2024-04-25T00:15:45.115194+0000","last_became_active":"2024-04-25T00:13:19.425170+0000","last_became_peered":"2024-04-25T00:13:19.425170+0000","last_unstale":"2024-04-25T00:15:45.115194+0000","last_undegraded":"2024-04-25T00:15:45.115194+0000","last_fullsized":"2024-04-25T00:15:45.115194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:31:13.826981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.114835+0000","last_change":"2024-04-25T00:13:19.420842+0000","last_active":"2024-04-25T00:15:45.114835+0000","last_peered":"2024-04-25T00:15:45.114835+0000","last_clean":"2024-04-25T00:15:45.114835+0000","last_became_active":"2024-04-25T00:13:19.420241+0000","last_became_peered":"2024-04-25T00:13:19.420241+0000","last_unstale":"2024-04-25T00:15:45.114835+0000","last_undegraded":"2024-04-25T00:15:45.114835+0000","last_fullsized":"2024-04-25T00:15:45.114835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:52:39.500432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149314+0000","last_change":"2024-04-25T00:14:11.267688+0000","last_active":"2024-04-25T00:15:45.149314+0000","last_peered":"2024-04-25T00:15:45.149314+0000","last_clean":"2024-04-25T00:15:45.149314+0000","last_became_active":"2024-04-25T00:14:11.267225+0000","last_became_peered":"2024-04-25T00:14:11.267225+0000","last_unstale":"2024-04-25T00:15:45.149314+0000","last_undegraded":"2024-04-25T00:15:45.149314+0000","last_fullsized":"2024-04-25T00:15:45.149314+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:15.867489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151754+0000","last_change":"2024-04-25T00:13:19.416519+0000","last_active":"2024-04-25T00:15:45.151754+0000","last_peered":"2024-04-25T00:15:45.151754+0000","last_clean":"2024-04-25T00:15:45.151754+0000","last_became_active":"2024-04-25T00:13:19.416080+0000","last_became_peered":"2024-04-25T00:13:19.416080+0000","last_unstale":"2024-04-25T00:15:45.151754+0000","last_undegraded":"2024-04-25T00:15:45.151754+0000","last_fullsized":"2024-04-25T00:15:45.151754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:04.450160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.162148+0000","last_change":"2024-04-25T00:13:19.424378+0000","last_active":"2024-04-25T00:15:45.162148+0000","last_peered":"2024-04-25T00:15:45.162148+0000","last_clean":"2024-04-25T00:15:45.162148+0000","last_became_active":"2024-04-25T00:13:19.423918+0000","last_became_peered":"2024-04-25T00:13:19.423918+0000","last_unstale":"2024-04-25T00:15:45.162148+0000","last_undegraded":"2024-04-25T00:15:45.162148+0000","last_fullsized":"2024-04-25T00:15:45.162148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:01:11.397328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.311886+0000","last_change":"2024-04-25T00:13:19.424729+0000","last_active":"2024-04-25T00:15:45.311886+0000","last_peered":"2024-04-25T00:15:45.311886+0000","last_clean":"2024-04-25T00:15:45.311886+0000","last_became_active":"2024-04-25T00:13:19.422874+0000","last_became_peered":"2024-04-25T00:13:19.422874+0000","last_unstale":"2024-04-25T00:15:45.311886+0000","last_undegraded":"2024-04-25T00:15:45.311886+0000","last_fullsized":"2024-04-25T00:15:45.311886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:52:39.483589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.039933+0000","last_change":"2024-04-25T00:13:19.418043+0000","last_active":"2024-04-25T00:15:45.039933+0000","last_peered":"2024-04-25T00:15:45.039933+0000","last_clean":"2024-04-25T00:15:45.039933+0000","last_became_active":"2024-04-25T00:13:19.417443+0000","last_became_peered":"2024-04-25T00:13:19.417443+0000","last_unstale":"2024-04-25T00:15:45.039933+0000","last_undegraded":"2024-04-25T00:15:45.039933+0000","last_fullsized":"2024-04-25T00:15:45.039933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:47:53.141744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309810+0000","last_change":"2024-04-25T00:13:19.420989+0000","last_active":"2024-04-25T00:15:45.309810+0000","last_peered":"2024-04-25T00:15:45.309810+0000","last_clean":"2024-04-25T00:15:45.309810+0000","last_became_active":"2024-04-25T00:13:19.420669+0000","last_became_peered":"2024-04-25T00:13:19.420669+0000","last_unstale":"2024-04-25T00:15:45.309810+0000","last_undegraded":"2024-04-25T00:15:45.309810+0000","last_fullsized":"2024-04-25T00:15:45.309810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:18.236132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.162583+0000","last_change":"2024-04-25T00:13:19.424176+0000","last_active":"2024-04-25T00:15:45.162583+0000","last_peered":"2024-04-25T00:15:45.162583+0000","last_clean":"2024-04-25T00:15:45.162583+0000","last_became_active":"2024-04-25T00:13:19.423656+0000","last_became_peered":"2024-04-25T00:13:19.423656+0000","last_unstale":"2024-04-25T00:15:45.162583+0000","last_undegraded":"2024-04-25T00:15:45.162583+0000","last_fullsized":"2024-04-25T00:15:45.162583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:24:41.697077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312946+0000","last_change":"2024-04-25T00:13:19.419624+0000","last_active":"2024-04-25T00:15:45.312946+0000","last_peered":"2024-04-25T00:15:45.312946+0000","last_clean":"2024-04-25T00:15:45.312946+0000","last_became_active":"2024-04-25T00:13:19.419263+0000","last_became_peered":"2024-04-25T00:13:19.419263+0000","last_unstale":"2024-04-25T00:15:45.312946+0000","last_undegraded":"2024-04-25T00:15:45.312946+0000","last_fullsized":"2024-04-25T00:15:45.312946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:29.130733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041558+0000","last_change":"2024-04-25T00:13:19.427865+0000","last_active":"2024-04-25T00:15:45.041558+0000","last_peered":"2024-04-25T00:15:45.041558+0000","last_clean":"2024-04-25T00:15:45.041558+0000","last_became_active":"2024-04-25T00:13:19.427538+0000","last_became_peered":"2024-04-25T00:13:19.427538+0000","last_unstale":"2024-04-25T00:15:45.041558+0000","last_undegraded":"2024-04-25T00:15:45.041558+0000","last_fullsized":"2024-04-25T00:15:45.041558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:05.136948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314986+0000","last_change":"2024-04-25T00:13:19.419261+0000","last_active":"2024-04-25T00:15:45.314986+0000","last_peered":"2024-04-25T00:15:45.314986+0000","last_clean":"2024-04-25T00:15:45.314986+0000","last_became_active":"2024-04-25T00:13:19.418938+0000","last_became_peered":"2024-04-25T00:13:19.418938+0000","last_unstale":"2024-04-25T00:15:45.314986+0000","last_undegraded":"2024-04-25T00:15:45.314986+0000","last_fullsized":"2024-04-25T00:15:45.314986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:44.971640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.114768+0000","last_change":"2024-04-25T00:13:19.417654+0000","last_active":"2024-04-25T00:15:45.114768+0000","last_peered":"2024-04-25T00:15:45.114768+0000","last_clean":"2024-04-25T00:15:45.114768+0000","last_became_active":"2024-04-25T00:13:19.417319+0000","last_became_peered":"2024-04-25T00:13:19.417319+0000","last_unstale":"2024-04-25T00:15:45.114768+0000","last_undegraded":"2024-04-25T00:15:45.114768+0000","last_fullsized":"2024-04-25T00:15:45.114768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:43:54.447937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115100+0000","last_change":"2024-04-25T00:13:19.420761+0000","last_active":"2024-04-25T00:15:45.115100+0000","last_peered":"2024-04-25T00:15:45.115100+0000","last_clean":"2024-04-25T00:15:45.115100+0000","last_became_active":"2024-04-25T00:13:19.420334+0000","last_became_peered":"2024-04-25T00:13:19.420334+0000","last_unstale":"2024-04-25T00:15:45.115100+0000","last_undegraded":"2024-04-25T00:15:45.115100+0000","last_fullsized":"2024-04-25T00:15:45.115100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:20.136470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309282+0000","last_change":"2024-04-25T00:13:19.420132+0000","last_active":"2024-04-25T00:15:45.309282+0000","last_peered":"2024-04-25T00:15:45.309282+0000","last_clean":"2024-04-25T00:15:45.309282+0000","last_became_active":"2024-04-25T00:13:19.418910+0000","last_became_peered":"2024-04-25T00:13:19.418910+0000","last_unstale":"2024-04-25T00:15:45.309282+0000","last_undegraded":"2024-04-25T00:15:45.309282+0000","last_fullsized":"2024-04-25T00:15:45.309282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:17.895412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.294056+0000","last_change":"2024-04-25T00:13:19.430242+0000","last_active":"2024-04-25T00:15:45.294056+0000","last_peered":"2024-04-25T00:15:45.294056+0000","last_clean":"2024-04-25T00:15:45.294056+0000","last_became_active":"2024-04-25T00:13:19.429077+0000","last_became_peered":"2024-04-25T00:13:19.429077+0000","last_unstale":"2024-04-25T00:15:45.294056+0000","last_undegraded":"2024-04-25T00:15:45.294056+0000","last_fullsized":"2024-04-25T00:15:45.294056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:34:14.351157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149646+0000","last_change":"2024-04-25T00:13:19.414375+0000","last_active":"2024-04-25T00:15:45.149646+0000","last_peered":"2024-04-25T00:15:45.149646+0000","last_clean":"2024-04-25T00:15:45.149646+0000","last_became_active":"2024-04-25T00:13:19.414003+0000","last_became_peered":"2024-04-25T00:13:19.414003+0000","last_unstale":"2024-04-25T00:15:45.149646+0000","last_undegraded":"2024-04-25T00:15:45.149646+0000","last_fullsized":"2024-04-25T00:15:45.149646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:43:50.612743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041639+0000","last_change":"2024-04-25T00:13:19.415322+0000","last_active":"2024-04-25T00:15:45.041639+0000","last_peered":"2024-04-25T00:15:45.041639+0000","last_clean":"2024-04-25T00:15:45.041639+0000","last_became_active":"2024-04-25T00:13:19.414925+0000","last_became_peered":"2024-04-25T00:13:19.414925+0000","last_unstale":"2024-04-25T00:15:45.041639+0000","last_undegraded":"2024-04-25T00:15:45.041639+0000","last_fullsized":"2024-04-25T00:15:45.041639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:58.140640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041305+0000","last_change":"2024-04-25T00:13:23.584763+0000","last_active":"2024-04-25T00:15:45.041305+0000","last_peered":"2024-04-25T00:15:45.041305+0000","last_clean":"2024-04-25T00:15:45.041305+0000","last_became_active":"2024-04-25T00:13:23.584303+0000","last_became_peered":"2024-04-25T00:13:23.584303+0000","last_unstale":"2024-04-25T00:15:45.041305+0000","last_undegraded":"2024-04-25T00:15:45.041305+0000","last_fullsized":"2024-04-25T00:15:45.041305+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:54:11.447867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'14","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149948+0000","last_change":"2024-04-25T00:13:18.519576+0000","last_active":"2024-04-25T00:15:45.149948+0000","last_peered":"2024-04-25T00:15:45.149948+0000","last_clean":"2024-04-25T00:15:45.149948+0000","last_became_active":"2024-04-25T00:13:18.519201+0000","last_became_peered":"2024-04-25T00:13:18.519201+0000","last_unstale":"2024-04-25T00:15:45.149948+0000","last_undegraded":"2024-04-25T00:15:45.149948+0000","last_fullsized":"2024-04-25T00:15:45.149948+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:19.681096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040313+0000","last_change":"2024-04-25T00:13:19.419098+0000","last_active":"2024-04-25T00:15:45.040313+0000","last_peered":"2024-04-25T00:15:45.040313+0000","last_clean":"2024-04-25T00:15:45.040313+0000","last_became_active":"2024-04-25T00:13:19.418649+0000","last_became_peered":"2024-04-25T00:13:19.418649+0000","last_unstale":"2024-04-25T00:15:45.040313+0000","last_undegraded":"2024-04-25T00:15:45.040313+0000","last_fullsized":"2024-04-25T00:15:45.040313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:04.561470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041042+0000","last_change":"2024-04-25T00:13:23.580354+0000","last_active":"2024-04-25T00:15:45.041042+0000","last_peered":"2024-04-25T00:15:45.041042+0000","last_clean":"2024-04-25T00:15:45.041042+0000","last_became_active":"2024-04-25T00:13:23.579944+0000","last_became_peered":"2024-04-25T00:13:23.579944+0000","last_unstale":"2024-04-25T00:15:45.041042+0000","last_undegraded":"2024-04-25T00:15:45.041042+0000","last_fullsized":"2024-04-25T00:15:45.041042+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:48:57.788974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040339+0000","last_change":"2024-04-25T00:13:18.527471+0000","last_active":"2024-04-25T00:15:45.040339+0000","last_peered":"2024-04-25T00:15:45.040339+0000","last_clean":"2024-04-25T00:15:45.040339+0000","last_became_active":"2024-04-25T00:13:18.527138+0000","last_became_peered":"2024-04-25T00:13:18.527138+0000","last_unstale":"2024-04-25T00:15:45.040339+0000","last_undegraded":"2024-04-25T00:15:45.040339+0000","last_fullsized":"2024-04-25T00:15:45.040339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:38.003719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115017+0000","last_change":"2024-04-25T00:13:19.420563+0000","last_active":"2024-04-25T00:15:45.115017+0000","last_peered":"2024-04-25T00:15:45.115017+0000","last_clean":"2024-04-25T00:15:45.115017+0000","last_became_active":"2024-04-25T00:13:19.420142+0000","last_became_peered":"2024-04-25T00:13:19.420142+0000","last_unstale":"2024-04-25T00:15:45.115017+0000","last_undegraded":"2024-04-25T00:15:45.115017+0000","last_fullsized":"2024-04-25T00:15:45.115017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:23:08.423795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041136+0000","last_change":"2024-04-25T00:14:11.271321+0000","last_active":"2024-04-25T00:15:45.041136+0000","last_peered":"2024-04-25T00:15:45.041136+0000","last_clean":"2024-04-25T00:15:45.041136+0000","last_became_active":"2024-04-25T00:14:11.270478+0000","last_became_peered":"2024-04-25T00:14:11.270478+0000","last_unstale":"2024-04-25T00:15:45.041136+0000","last_undegraded":"2024-04-25T00:15:45.041136+0000","last_fullsized":"2024-04-25T00:15:45.041136+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:41.796155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'13","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.152142+0000","last_change":"2024-04-25T00:13:18.520509+0000","last_active":"2024-04-25T00:15:45.152142+0000","last_peered":"2024-04-25T00:15:45.152142+0000","last_clean":"2024-04-25T00:15:45.152142+0000","last_became_active":"2024-04-25T00:13:18.520139+0000","last_became_peered":"2024-04-25T00:13:18.520139+0000","last_unstale":"2024-04-25T00:15:45.152142+0000","last_undegraded":"2024-04-25T00:15:45.152142+0000","last_fullsized":"2024-04-25T00:15:45.152142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:08.797670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309574+0000","last_change":"2024-04-25T00:13:19.419115+0000","last_active":"2024-04-25T00:15:45.309574+0000","last_peered":"2024-04-25T00:15:45.309574+0000","last_clean":"2024-04-25T00:15:45.309574+0000","last_became_active":"2024-04-25T00:13:19.417865+0000","last_became_peered":"2024-04-25T00:13:19.417865+0000","last_unstale":"2024-04-25T00:15:45.309574+0000","last_undegraded":"2024-04-25T00:15:45.309574+0000","last_fullsized":"2024-04-25T00:15:45.309574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:20:11.173803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041021+0000","last_change":"2024-04-25T00:13:23.585052+0000","last_active":"2024-04-25T00:15:45.041021+0000","last_peered":"2024-04-25T00:15:45.041021+0000","last_clean":"2024-04-25T00:15:45.041021+0000","last_became_active":"2024-04-25T00:13:23.584696+0000","last_became_peered":"2024-04-25T00:13:23.584696+0000","last_unstale":"2024-04-25T00:15:45.041021+0000","last_undegraded":"2024-04-25T00:15:45.041021+0000","last_fullsized":"2024-04-25T00:15:45.041021+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:26.894183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151965+0000","last_change":"2024-04-25T00:13:18.522793+0000","last_active":"2024-04-25T00:15:45.151965+0000","last_peered":"2024-04-25T00:15:45.151965+0000","last_clean":"2024-04-25T00:15:45.151965+0000","last_became_active":"2024-04-25T00:13:18.522453+0000","last_became_peered":"2024-04-25T00:13:18.522453+0000","last_unstale":"2024-04-25T00:15:45.151965+0000","last_undegraded":"2024-04-25T00:15:45.151965+0000","last_fullsized":"2024-04-25T00:15:45.151965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:08.368214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.159151+0000","last_change":"2024-04-25T00:13:19.418573+0000","last_active":"2024-04-25T00:15:45.159151+0000","last_peered":"2024-04-25T00:15:45.159151+0000","last_clean":"2024-04-25T00:15:45.159151+0000","last_became_active":"2024-04-25T00:13:19.418219+0000","last_became_peered":"2024-04-25T00:13:19.418219+0000","last_unstale":"2024-04-25T00:15:45.159151+0000","last_undegraded":"2024-04-25T00:15:45.159151+0000","last_fullsized":"2024-04-25T00:15:45.159151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:08:44.453017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041456+0000","last_change":"2024-04-25T00:13:23.578716+0000","last_active":"2024-04-25T00:15:45.041456+0000","last_peered":"2024-04-25T00:15:45.041456+0000","last_clean":"2024-04-25T00:15:45.041456+0000","last_became_active":"2024-04-25T00:13:23.578325+0000","last_became_peered":"2024-04-25T00:13:23.578325+0000","last_unstale":"2024-04-25T00:15:45.041456+0000","last_undegraded":"2024-04-25T00:15:45.041456+0000","last_fullsized":"2024-04-25T00:15:45.041456+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:39:57.145656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.731145+0000","last_change":"2024-04-25T00:13:18.527369+0000","last_active":"2024-04-25T00:15:45.731145+0000","last_peered":"2024-04-25T00:15:45.731145+0000","last_clean":"2024-04-25T00:15:45.731145+0000","last_became_active":"2024-04-25T00:13:18.526032+0000","last_became_peered":"2024-04-25T00:13:18.526032+0000","last_unstale":"2024-04-25T00:15:45.731145+0000","last_undegraded":"2024-04-25T00:15:45.731145+0000","last_fullsized":"2024-04-25T00:15:45.731145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:52.635184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151572+0000","last_change":"2024-04-25T00:13:19.413305+0000","last_active":"2024-04-25T00:15:45.151572+0000","last_peered":"2024-04-25T00:15:45.151572+0000","last_clean":"2024-04-25T00:15:45.151572+0000","last_became_active":"2024-04-25T00:13:19.412956+0000","last_became_peered":"2024-04-25T00:13:19.412956+0000","last_unstale":"2024-04-25T00:15:45.151572+0000","last_undegraded":"2024-04-25T00:15:45.151572+0000","last_fullsized":"2024-04-25T00:15:45.151572+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:54.436470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041546+0000","last_change":"2024-04-25T00:14:11.273057+0000","last_active":"2024-04-25T00:15:45.041546+0000","last_peered":"2024-04-25T00:15:45.041546+0000","last_clean":"2024-04-25T00:15:45.041546+0000","last_became_active":"2024-04-25T00:14:11.272230+0000","last_became_peered":"2024-04-25T00:14:11.272230+0000","last_unstale":"2024-04-25T00:15:45.041546+0000","last_undegraded":"2024-04-25T00:15:45.041546+0000","last_fullsized":"2024-04-25T00:15:45.041546+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:37.695139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.162194+0000","last_change":"2024-04-25T00:13:18.517507+0000","last_active":"2024-04-25T00:15:45.162194+0000","last_peered":"2024-04-25T00:15:45.162194+0000","last_clean":"2024-04-25T00:15:45.162194+0000","last_became_active":"2024-04-25T00:13:18.517099+0000","last_became_peered":"2024-04-25T00:13:18.517099+0000","last_unstale":"2024-04-25T00:15:45.162194+0000","last_undegraded":"2024-04-25T00:15:45.162194+0000","last_fullsized":"2024-04-25T00:15:45.162194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:42:17.357610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.159046+0000","last_change":"2024-04-25T00:14:11.271691+0000","last_active":"2024-04-25T00:15:45.159046+0000","last_peered":"2024-04-25T00:15:45.159046+0000","last_clean":"2024-04-25T00:15:45.159046+0000","last_became_active":"2024-04-25T00:14:11.271160+0000","last_became_peered":"2024-04-25T00:14:11.271160+0000","last_unstale":"2024-04-25T00:15:45.159046+0000","last_undegraded":"2024-04-25T00:15:45.159046+0000","last_fullsized":"2024-04-25T00:15:45.159046+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:24.416810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.294089+0000","last_change":"2024-04-25T00:13:23.582526+0000","last_active":"2024-04-25T00:15:45.294089+0000","last_peered":"2024-04-25T00:15:45.294089+0000","last_clean":"2024-04-25T00:15:45.294089+0000","last_became_active":"2024-04-25T00:13:23.581221+0000","last_became_peered":"2024-04-25T00:13:23.581221+0000","last_unstale":"2024-04-25T00:15:45.294089+0000","last_undegraded":"2024-04-25T00:15:45.294089+0000","last_fullsized":"2024-04-25T00:15:45.294089+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:46.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314551+0000","last_change":"2024-04-25T00:13:18.522245+0000","last_active":"2024-04-25T00:15:45.314551+0000","last_peered":"2024-04-25T00:15:45.314551+0000","last_clean":"2024-04-25T00:15:45.314551+0000","last_became_active":"2024-04-25T00:13:18.521440+0000","last_became_peered":"2024-04-25T00:13:18.521440+0000","last_unstale":"2024-04-25T00:15:45.314551+0000","last_undegraded":"2024-04-25T00:15:45.314551+0000","last_fullsized":"2024-04-25T00:15:45.314551+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:21:37.847964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040022+0000","last_change":"2024-04-25T00:13:19.422771+0000","last_active":"2024-04-25T00:15:45.040022+0000","last_peered":"2024-04-25T00:15:45.040022+0000","last_clean":"2024-04-25T00:15:45.040022+0000","last_became_active":"2024-04-25T00:13:19.422387+0000","last_became_peered":"2024-04-25T00:13:19.422387+0000","last_unstale":"2024-04-25T00:15:45.040022+0000","last_undegraded":"2024-04-25T00:15:45.040022+0000","last_fullsized":"2024-04-25T00:15:45.040022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:15.647186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115073+0000","last_change":"2024-04-25T00:13:23.586235+0000","last_active":"2024-04-25T00:15:45.115073+0000","last_peered":"2024-04-25T00:15:45.115073+0000","last_clean":"2024-04-25T00:15:45.115073+0000","last_became_active":"2024-04-25T00:13:23.585803+0000","last_became_peered":"2024-04-25T00:13:23.585803+0000","last_unstale":"2024-04-25T00:15:45.115073+0000","last_undegraded":"2024-04-25T00:15:45.115073+0000","last_fullsized":"2024-04-25T00:15:45.115073+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:46.235435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158742+0000","last_change":"2024-04-25T00:13:18.525283+0000","last_active":"2024-04-25T00:15:45.158742+0000","last_peered":"2024-04-25T00:15:45.158742+0000","last_clean":"2024-04-25T00:15:45.158742+0000","last_became_active":"2024-04-25T00:13:18.524939+0000","last_became_peered":"2024-04-25T00:13:18.524939+0000","last_unstale":"2024-04-25T00:15:45.158742+0000","last_undegraded":"2024-04-25T00:15:45.158742+0000","last_fullsized":"2024-04-25T00:15:45.158742+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:09.258498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.148867+0000","last_change":"2024-04-25T00:13:19.415746+0000","last_active":"2024-04-25T00:15:45.148867+0000","last_peered":"2024-04-25T00:15:45.148867+0000","last_clean":"2024-04-25T00:15:45.148867+0000","last_became_active":"2024-04-25T00:13:19.415325+0000","last_became_peered":"2024-04-25T00:13:19.415325+0000","last_unstale":"2024-04-25T00:15:45.148867+0000","last_undegraded":"2024-04-25T00:15:45.148867+0000","last_fullsized":"2024-04-25T00:15:45.148867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:17:48.268845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.039798+0000","last_change":"2024-04-25T00:13:23.583153+0000","last_active":"2024-04-25T00:15:45.039798+0000","last_peered":"2024-04-25T00:15:45.039798+0000","last_clean":"2024-04-25T00:15:45.039798+0000","last_became_active":"2024-04-25T00:13:23.580257+0000","last_became_peered":"2024-04-25T00:13:23.580257+0000","last_unstale":"2024-04-25T00:15:45.039798+0000","last_undegraded":"2024-04-25T00:15:45.039798+0000","last_fullsized":"2024-04-25T00:15:45.039798+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:09:32.990253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'5","reported_seq":279,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.753951+0000","last_change":"2024-04-25T00:13:18.520365+0000","last_active":"2024-04-25T00:15:45.753951+0000","last_peered":"2024-04-25T00:15:45.753951+0000","last_clean":"2024-04-25T00:15:45.753951+0000","last_became_active":"2024-04-25T00:13:18.519918+0000","last_became_peered":"2024-04-25T00:13:18.519918+0000","last_unstale":"2024-04-25T00:15:45.753951+0000","last_undegraded":"2024-04-25T00:15:45.753951+0000","last_fullsized":"2024-04-25T00:15:45.753951+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:46.415197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":240,"num_read_kb":240,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309728+0000","last_change":"2024-04-25T00:13:19.414341+0000","last_active":"2024-04-25T00:15:45.309728+0000","last_peered":"2024-04-25T00:15:45.309728+0000","last_clean":"2024-04-25T00:15:45.309728+0000","last_became_active":"2024-04-25T00:13:19.413793+0000","last_became_peered":"2024-04-25T00:13:19.413793+0000","last_unstale":"2024-04-25T00:15:45.309728+0000","last_undegraded":"2024-04-25T00:15:45.309728+0000","last_fullsized":"2024-04-25T00:15:45.309728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:55.927082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.148616+0000","last_change":"2024-04-25T00:13:23.590631+0000","last_active":"2024-04-25T00:15:45.148616+0000","last_peered":"2024-04-25T00:15:45.148616+0000","last_clean":"2024-04-25T00:15:45.148616+0000","last_became_active":"2024-04-25T00:13:23.589980+0000","last_became_peered":"2024-04-25T00:13:23.589980+0000","last_unstale":"2024-04-25T00:15:45.148616+0000","last_undegraded":"2024-04-25T00:15:45.148616+0000","last_fullsized":"2024-04-25T00:15:45.148616+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:18.234048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.313143+0000","last_change":"2024-04-25T00:13:18.526353+0000","last_active":"2024-04-25T00:15:45.313143+0000","last_peered":"2024-04-25T00:15:45.313143+0000","last_clean":"2024-04-25T00:15:45.313143+0000","last_became_active":"2024-04-25T00:13:18.525951+0000","last_became_peered":"2024-04-25T00:13:18.525951+0000","last_unstale":"2024-04-25T00:15:45.313143+0000","last_undegraded":"2024-04-25T00:15:45.313143+0000","last_fullsized":"2024-04-25T00:15:45.313143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:10:29.597797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115219+0000","last_change":"2024-04-25T00:13:19.425486+0000","last_active":"2024-04-25T00:15:45.115219+0000","last_peered":"2024-04-25T00:15:45.115219+0000","last_clean":"2024-04-25T00:15:45.115219+0000","last_became_active":"2024-04-25T00:13:19.425087+0000","last_became_peered":"2024-04-25T00:13:19.425087+0000","last_unstale":"2024-04-25T00:15:45.115219+0000","last_undegraded":"2024-04-25T00:15:45.115219+0000","last_fullsized":"2024-04-25T00:15:45.115219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:00.216683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041314+0000","last_change":"2024-04-25T00:13:23.579747+0000","last_active":"2024-04-25T00:15:45.041314+0000","last_peered":"2024-04-25T00:15:45.041314+0000","last_clean":"2024-04-25T00:15:45.041314+0000","last_became_active":"2024-04-25T00:13:23.579346+0000","last_became_peered":"2024-04-25T00:13:23.579346+0000","last_unstale":"2024-04-25T00:15:45.041314+0000","last_undegraded":"2024-04-25T00:15:45.041314+0000","last_fullsized":"2024-04-25T00:15:45.041314+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:10:18.969942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.159127+0000","last_change":"2024-04-25T00:13:18.524707+0000","last_active":"2024-04-25T00:15:45.159127+0000","last_peered":"2024-04-25T00:15:45.159127+0000","last_clean":"2024-04-25T00:15:45.159127+0000","last_became_active":"2024-04-25T00:13:18.524352+0000","last_became_peered":"2024-04-25T00:13:18.524352+0000","last_unstale":"2024-04-25T00:15:45.159127+0000","last_undegraded":"2024-04-25T00:15:45.159127+0000","last_fullsized":"2024-04-25T00:15:45.159127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:29.992843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309736+0000","last_change":"2024-04-25T00:13:19.421845+0000","last_active":"2024-04-25T00:15:45.309736+0000","last_peered":"2024-04-25T00:15:45.309736+0000","last_clean":"2024-04-25T00:15:45.309736+0000","last_became_active":"2024-04-25T00:13:19.421458+0000","last_became_peered":"2024-04-25T00:13:19.421458+0000","last_unstale":"2024-04-25T00:15:45.309736+0000","last_undegraded":"2024-04-25T00:15:45.309736+0000","last_fullsized":"2024-04-25T00:15:45.309736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:09.286604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.148941+0000","last_change":"2024-04-25T00:13:23.583721+0000","last_active":"2024-04-25T00:15:45.148941+0000","last_peered":"2024-04-25T00:15:45.148941+0000","last_clean":"2024-04-25T00:15:45.148941+0000","last_became_active":"2024-04-25T00:13:23.583340+0000","last_became_peered":"2024-04-25T00:13:23.583340+0000","last_unstale":"2024-04-25T00:15:45.148941+0000","last_undegraded":"2024-04-25T00:15:45.148941+0000","last_fullsized":"2024-04-25T00:15:45.148941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:26.446468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":326,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.731678+0000","last_change":"2024-04-25T00:13:18.520653+0000","last_active":"2024-04-25T00:15:45.731678+0000","last_peered":"2024-04-25T00:15:45.731678+0000","last_clean":"2024-04-25T00:15:45.731678+0000","last_became_active":"2024-04-25T00:13:18.520278+0000","last_became_peered":"2024-04-25T00:13:18.520278+0000","last_unstale":"2024-04-25T00:15:45.731678+0000","last_undegraded":"2024-04-25T00:15:45.731678+0000","last_fullsized":"2024-04-25T00:15:45.731678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:58.311179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66804,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":252,"num_read_kb":191,"num_write":42,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.152025+0000","last_change":"2024-04-25T00:13:19.421599+0000","last_active":"2024-04-25T00:15:45.152025+0000","last_peered":"2024-04-25T00:15:45.152025+0000","last_clean":"2024-04-25T00:15:45.152025+0000","last_became_active":"2024-04-25T00:13:19.421107+0000","last_became_peered":"2024-04-25T00:13:19.421107+0000","last_unstale":"2024-04-25T00:15:45.152025+0000","last_undegraded":"2024-04-25T00:15:45.152025+0000","last_fullsized":"2024-04-25T00:15:45.152025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:59.577987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.159398+0000","last_change":"2024-04-25T00:13:23.580538+0000","last_active":"2024-04-25T00:15:45.159398+0000","last_peered":"2024-04-25T00:15:45.159398+0000","last_clean":"2024-04-25T00:15:45.159398+0000","last_became_active":"2024-04-25T00:13:23.580095+0000","last_became_peered":"2024-04-25T00:13:23.580095+0000","last_unstale":"2024-04-25T00:15:45.159398+0000","last_undegraded":"2024-04-25T00:15:45.159398+0000","last_fullsized":"2024-04-25T00:15:45.159398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:04.189530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041608+0000","last_change":"2024-04-25T00:13:18.522501+0000","last_active":"2024-04-25T00:15:45.041608+0000","last_peered":"2024-04-25T00:15:45.041608+0000","last_clean":"2024-04-25T00:15:45.041608+0000","last_became_active":"2024-04-25T00:13:18.522160+0000","last_became_peered":"2024-04-25T00:13:18.522160+0000","last_unstale":"2024-04-25T00:15:45.041608+0000","last_undegraded":"2024-04-25T00:15:45.041608+0000","last_fullsized":"2024-04-25T00:15:45.041608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:31.876731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314208+0000","last_change":"2024-04-25T00:13:19.417957+0000","last_active":"2024-04-25T00:15:45.314208+0000","last_peered":"2024-04-25T00:15:45.314208+0000","last_clean":"2024-04-25T00:15:45.314208+0000","last_became_active":"2024-04-25T00:13:19.417600+0000","last_became_peered":"2024-04-25T00:13:19.417600+0000","last_unstale":"2024-04-25T00:15:45.314208+0000","last_undegraded":"2024-04-25T00:15:45.314208+0000","last_fullsized":"2024-04-25T00:15:45.314208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:24.600965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293826+0000","last_change":"2024-04-25T00:13:23.582279+0000","last_active":"2024-04-25T00:15:45.293826+0000","last_peered":"2024-04-25T00:15:45.293826+0000","last_clean":"2024-04-25T00:15:45.293826+0000","last_became_active":"2024-04-25T00:13:23.581766+0000","last_became_peered":"2024-04-25T00:13:23.581766+0000","last_unstale":"2024-04-25T00:15:45.293826+0000","last_undegraded":"2024-04-25T00:15:45.293826+0000","last_fullsized":"2024-04-25T00:15:45.293826+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:36.115870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158999+0000","last_change":"2024-04-25T00:13:18.527443+0000","last_active":"2024-04-25T00:15:45.158999+0000","last_peered":"2024-04-25T00:15:45.158999+0000","last_clean":"2024-04-25T00:15:45.158999+0000","last_became_active":"2024-04-25T00:13:18.526919+0000","last_became_peered":"2024-04-25T00:13:18.526919+0000","last_unstale":"2024-04-25T00:15:45.158999+0000","last_undegraded":"2024-04-25T00:15:45.158999+0000","last_fullsized":"2024-04-25T00:15:45.158999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:32.737580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312419+0000","last_change":"2024-04-25T00:13:19.425947+0000","last_active":"2024-04-25T00:15:45.312419+0000","last_peered":"2024-04-25T00:15:45.312419+0000","last_clean":"2024-04-25T00:15:45.312419+0000","last_became_active":"2024-04-25T00:13:19.425613+0000","last_became_peered":"2024-04-25T00:13:19.425613+0000","last_unstale":"2024-04-25T00:15:45.312419+0000","last_undegraded":"2024-04-25T00:15:45.312419+0000","last_fullsized":"2024-04-25T00:15:45.312419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:28:34.729375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149578+0000","last_change":"2024-04-25T00:13:23.587794+0000","last_active":"2024-04-25T00:15:45.149578+0000","last_peered":"2024-04-25T00:15:45.149578+0000","last_clean":"2024-04-25T00:15:45.149578+0000","last_became_active":"2024-04-25T00:13:23.587334+0000","last_became_peered":"2024-04-25T00:13:23.587334+0000","last_unstale":"2024-04-25T00:15:45.149578+0000","last_undegraded":"2024-04-25T00:15:45.149578+0000","last_fullsized":"2024-04-25T00:15:45.149578+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:58.460439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":166,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.731310+0000","last_change":"2024-04-25T00:13:18.521701+0000","last_active":"2024-04-25T00:15:45.731310+0000","last_peered":"2024-04-25T00:15:45.731310+0000","last_clean":"2024-04-25T00:15:45.731310+0000","last_became_active":"2024-04-25T00:13:18.521360+0000","last_became_peered":"2024-04-25T00:13:18.521360+0000","last_unstale":"2024-04-25T00:15:45.731310+0000","last_undegraded":"2024-04-25T00:15:45.731310+0000","last_fullsized":"2024-04-25T00:15:45.731310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:18:11.567426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":126,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.311990+0000","last_change":"2024-04-25T00:13:19.424162+0000","last_active":"2024-04-25T00:15:45.311990+0000","last_peered":"2024-04-25T00:15:45.311990+0000","last_clean":"2024-04-25T00:15:45.311990+0000","last_became_active":"2024-04-25T00:13:19.423845+0000","last_became_peered":"2024-04-25T00:13:19.423845+0000","last_unstale":"2024-04-25T00:15:45.311990+0000","last_undegraded":"2024-04-25T00:15:45.311990+0000","last_fullsized":"2024-04-25T00:15:45.311990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:33.887577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314672+0000","last_change":"2024-04-25T00:13:23.582102+0000","last_active":"2024-04-25T00:15:45.314672+0000","last_peered":"2024-04-25T00:15:45.314672+0000","last_clean":"2024-04-25T00:15:45.314672+0000","last_became_active":"2024-04-25T00:13:23.581679+0000","last_became_peered":"2024-04-25T00:13:23.581679+0000","last_unstale":"2024-04-25T00:15:45.314672+0000","last_undegraded":"2024-04-25T00:15:45.314672+0000","last_fullsized":"2024-04-25T00:15:45.314672+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:05:39.518634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151589+0000","last_change":"2024-04-25T00:13:18.518285+0000","last_active":"2024-04-25T00:15:45.151589+0000","last_peered":"2024-04-25T00:15:45.151589+0000","last_clean":"2024-04-25T00:15:45.151589+0000","last_became_active":"2024-04-25T00:13:18.517826+0000","last_became_peered":"2024-04-25T00:13:18.517826+0000","last_unstale":"2024-04-25T00:15:45.151589+0000","last_undegraded":"2024-04-25T00:15:45.151589+0000","last_fullsized":"2024-04-25T00:15:45.151589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:12.901225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158812+0000","last_change":"2024-04-25T00:14:11.268444+0000","last_active":"2024-04-25T00:15:45.158812+0000","last_peered":"2024-04-25T00:15:45.158812+0000","last_clean":"2024-04-25T00:15:45.158812+0000","last_became_active":"2024-04-25T00:14:11.267934+0000","last_became_peered":"2024-04-25T00:14:11.267934+0000","last_unstale":"2024-04-25T00:15:45.158812+0000","last_undegraded":"2024-04-25T00:15:45.158812+0000","last_fullsized":"2024-04-25T00:15:45.158812+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:12:16.209890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158730+0000","last_change":"2024-04-25T00:13:23.578445+0000","last_active":"2024-04-25T00:15:45.158730+0000","last_peered":"2024-04-25T00:15:45.158730+0000","last_clean":"2024-04-25T00:15:45.158730+0000","last_became_active":"2024-04-25T00:13:23.578048+0000","last_became_peered":"2024-04-25T00:13:23.578048+0000","last_unstale":"2024-04-25T00:15:45.158730+0000","last_undegraded":"2024-04-25T00:15:45.158730+0000","last_fullsized":"2024-04-25T00:15:45.158730+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:55.072460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151953+0000","last_change":"2024-04-25T00:13:18.515939+0000","last_active":"2024-04-25T00:15:45.151953+0000","last_peered":"2024-04-25T00:15:45.151953+0000","last_clean":"2024-04-25T00:15:45.151953+0000","last_became_active":"2024-04-25T00:13:18.515597+0000","last_became_peered":"2024-04-25T00:13:18.515597+0000","last_unstale":"2024-04-25T00:15:45.151953+0000","last_undegraded":"2024-04-25T00:15:45.151953+0000","last_fullsized":"2024-04-25T00:15:45.151953+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:03:30.266404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.294003+0000","last_change":"2024-04-25T00:14:11.269143+0000","last_active":"2024-04-25T00:15:45.294003+0000","last_peered":"2024-04-25T00:15:45.294003+0000","last_clean":"2024-04-25T00:15:45.294003+0000","last_became_active":"2024-04-25T00:14:11.268736+0000","last_became_peered":"2024-04-25T00:14:11.268736+0000","last_unstale":"2024-04-25T00:15:45.294003+0000","last_undegraded":"2024-04-25T00:15:45.294003+0000","last_fullsized":"2024-04-25T00:15:45.294003+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:19.559974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151733+0000","last_change":"2024-04-25T00:13:23.862576+0000","last_active":"2024-04-25T00:15:45.151733+0000","last_peered":"2024-04-25T00:15:45.151733+0000","last_clean":"2024-04-25T00:15:45.151733+0000","last_became_active":"2024-04-25T00:13:23.862024+0000","last_became_peered":"2024-04-25T00:13:23.862024+0000","last_unstale":"2024-04-25T00:15:45.151733+0000","last_undegraded":"2024-04-25T00:15:45.151733+0000","last_fullsized":"2024-04-25T00:15:45.151733+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:17.249680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149273+0000","last_change":"2024-04-25T00:13:18.522503+0000","last_active":"2024-04-25T00:15:45.149273+0000","last_peered":"2024-04-25T00:15:45.149273+0000","last_clean":"2024-04-25T00:15:45.149273+0000","last_became_active":"2024-04-25T00:13:18.522126+0000","last_became_peered":"2024-04-25T00:13:18.522126+0000","last_unstale":"2024-04-25T00:15:45.149273+0000","last_undegraded":"2024-04-25T00:15:45.149273+0000","last_fullsized":"2024-04-25T00:15:45.149273+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:47:57.530098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.162498+0000","last_change":"2024-04-25T00:13:19.416862+0000","last_active":"2024-04-25T00:15:45.162498+0000","last_peered":"2024-04-25T00:15:45.162498+0000","last_clean":"2024-04-25T00:15:45.162498+0000","last_became_active":"2024-04-25T00:13:19.416375+0000","last_became_peered":"2024-04-25T00:13:19.416375+0000","last_unstale":"2024-04-25T00:15:45.162498+0000","last_undegraded":"2024-04-25T00:15:45.162498+0000","last_fullsized":"2024-04-25T00:15:45.162498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:43:32.031623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314336+0000","last_change":"2024-04-25T00:13:23.582387+0000","last_active":"2024-04-25T00:15:45.314336+0000","last_peered":"2024-04-25T00:15:45.314336+0000","last_clean":"2024-04-25T00:15:45.314336+0000","last_became_active":"2024-04-25T00:13:23.581984+0000","last_became_peered":"2024-04-25T00:13:23.581984+0000","last_unstale":"2024-04-25T00:15:45.314336+0000","last_undegraded":"2024-04-25T00:15:45.314336+0000","last_fullsized":"2024-04-25T00:15:45.314336+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:07:16.388293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"94'7","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.748717+0000","last_change":"2024-04-25T00:13:18.524121+0000","last_active":"2024-04-25T00:15:45.748717+0000","last_peered":"2024-04-25T00:15:45.748717+0000","last_clean":"2024-04-25T00:15:45.748717+0000","last_became_active":"2024-04-25T00:13:18.523731+0000","last_became_peered":"2024-04-25T00:13:18.523731+0000","last_unstale":"2024-04-25T00:15:45.748717+0000","last_undegraded":"2024-04-25T00:15:45.748717+0000","last_fullsized":"2024-04-25T00:15:45.748717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:22.105952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1287,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.162127+0000","last_change":"2024-04-25T00:13:19.424753+0000","last_active":"2024-04-25T00:15:45.162127+0000","last_peered":"2024-04-25T00:15:45.162127+0000","last_clean":"2024-04-25T00:15:45.162127+0000","last_became_active":"2024-04-25T00:13:19.423214+0000","last_became_peered":"2024-04-25T00:13:19.423214+0000","last_unstale":"2024-04-25T00:15:45.162127+0000","last_undegraded":"2024-04-25T00:15:45.162127+0000","last_fullsized":"2024-04-25T00:15:45.162127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:41.627442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314459+0000","last_change":"2024-04-25T00:14:11.272431+0000","last_active":"2024-04-25T00:15:45.314459+0000","last_peered":"2024-04-25T00:15:45.314459+0000","last_clean":"2024-04-25T00:15:45.314459+0000","last_became_active":"2024-04-25T00:14:11.271997+0000","last_became_peered":"2024-04-25T00:14:11.271997+0000","last_unstale":"2024-04-25T00:15:45.314459+0000","last_undegraded":"2024-04-25T00:15:45.314459+0000","last_fullsized":"2024-04-25T00:15:45.314459+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:18.768009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158863+0000","last_change":"2024-04-25T00:13:19.420617+0000","last_active":"2024-04-25T00:15:45.158863+0000","last_peered":"2024-04-25T00:15:45.158863+0000","last_clean":"2024-04-25T00:15:45.158863+0000","last_became_active":"2024-04-25T00:13:19.420274+0000","last_became_peered":"2024-04-25T00:13:19.420274+0000","last_unstale":"2024-04-25T00:15:45.158863+0000","last_undegraded":"2024-04-25T00:15:45.158863+0000","last_fullsized":"2024-04-25T00:15:45.158863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:07.515196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314314+0000","last_change":"2024-04-25T00:13:18.529753+0000","last_active":"2024-04-25T00:15:45.314314+0000","last_peered":"2024-04-25T00:15:45.314314+0000","last_clean":"2024-04-25T00:15:45.314314+0000","last_became_active":"2024-04-25T00:13:18.525521+0000","last_became_peered":"2024-04-25T00:13:18.525521+0000","last_unstale":"2024-04-25T00:15:45.314314+0000","last_undegraded":"2024-04-25T00:15:45.314314+0000","last_fullsized":"2024-04-25T00:15:45.314314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:17.044367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312649+0000","last_change":"2024-04-25T00:13:23.589613+0000","last_active":"2024-04-25T00:15:45.312649+0000","last_peered":"2024-04-25T00:15:45.312649+0000","last_clean":"2024-04-25T00:15:45.312649+0000","last_became_active":"2024-04-25T00:13:23.589323+0000","last_became_peered":"2024-04-25T00:13:23.589323+0000","last_unstale":"2024-04-25T00:15:45.312649+0000","last_undegraded":"2024-04-25T00:15:45.312649+0000","last_fullsized":"2024-04-25T00:15:45.312649+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:34:42.272706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149072+0000","last_change":"2024-04-25T00:15:11.431091+0000","last_active":"2024-04-25T00:15:45.149072+0000","last_peered":"2024-04-25T00:15:45.149072+0000","last_clean":"2024-04-25T00:15:45.149072+0000","last_became_active":"2024-04-25T00:15:11.430797+0000","last_became_peered":"2024-04-25T00:15:11.430797+0000","last_unstale":"2024-04-25T00:15:45.149072+0000","last_undegraded":"2024-04-25T00:15:45.149072+0000","last_fullsized":"2024-04-25T00:15:45.149072+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:26:30.121912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041204+0000","last_change":"2024-04-25T00:13:18.514560+0000","last_active":"2024-04-25T00:15:45.041204+0000","last_peered":"2024-04-25T00:15:45.041204+0000","last_clean":"2024-04-25T00:15:45.041204+0000","last_became_active":"2024-04-25T00:13:18.514034+0000","last_became_peered":"2024-04-25T00:13:18.514034+0000","last_unstale":"2024-04-25T00:15:45.041204+0000","last_undegraded":"2024-04-25T00:15:45.041204+0000","last_fullsized":"2024-04-25T00:15:45.041204+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:35:18.110206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":365,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.311817+0000","last_change":"2024-04-25T00:12:01.540807+0000","last_active":"2024-04-25T00:15:45.311817+0000","last_peered":"2024-04-25T00:15:45.311817+0000","last_clean":"2024-04-25T00:15:45.311817+0000","last_became_active":"2024-04-25T00:12:01.540272+0000","last_became_peered":"2024-04-25T00:12:01.540272+0000","last_unstale":"2024-04-25T00:15:45.311817+0000","last_undegraded":"2024-04-25T00:15:45.311817+0000","last_fullsized":"2024-04-25T00:15:45.311817+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:08:34.007429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:08:34.007429+0000","last_clean_scrub_stamp":"2024-04-25T00:08:34.007429+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:50:14.642604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.159144+0000","last_change":"2024-04-25T00:13:23.862493+0000","last_active":"2024-04-25T00:15:45.159144+0000","last_peered":"2024-04-25T00:15:45.159144+0000","last_clean":"2024-04-25T00:15:45.159144+0000","last_became_active":"2024-04-25T00:13:23.861972+0000","last_became_peered":"2024-04-25T00:13:23.861972+0000","last_unstale":"2024-04-25T00:15:45.159144+0000","last_undegraded":"2024-04-25T00:15:45.159144+0000","last_fullsized":"2024-04-25T00:15:45.159144+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:49:13.507357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314067+0000","last_change":"2024-04-25T00:13:19.421920+0000","last_active":"2024-04-25T00:15:45.314067+0000","last_peered":"2024-04-25T00:15:45.314067+0000","last_clean":"2024-04-25T00:15:45.314067+0000","last_became_active":"2024-04-25T00:13:19.421585+0000","last_became_peered":"2024-04-25T00:13:19.421585+0000","last_unstale":"2024-04-25T00:15:45.314067+0000","last_undegraded":"2024-04-25T00:15:45.314067+0000","last_fullsized":"2024-04-25T00:15:45.314067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:53:24.937520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.311886+0000","last_change":"2024-04-25T00:13:18.527062+0000","last_active":"2024-04-25T00:15:45.311886+0000","last_peered":"2024-04-25T00:15:45.311886+0000","last_clean":"2024-04-25T00:15:45.311886+0000","last_became_active":"2024-04-25T00:13:18.526338+0000","last_became_peered":"2024-04-25T00:13:18.526338+0000","last_unstale":"2024-04-25T00:15:45.311886+0000","last_undegraded":"2024-04-25T00:15:45.311886+0000","last_fullsized":"2024-04-25T00:15:45.311886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:21.756728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293792+0000","last_change":"2024-04-25T00:13:23.575162+0000","last_active":"2024-04-25T00:15:45.293792+0000","last_peered":"2024-04-25T00:15:45.293792+0000","last_clean":"2024-04-25T00:15:45.293792+0000","last_became_active":"2024-04-25T00:13:23.574722+0000","last_became_peered":"2024-04-25T00:13:23.574722+0000","last_unstale":"2024-04-25T00:15:45.293792+0000","last_undegraded":"2024-04-25T00:15:45.293792+0000","last_fullsized":"2024-04-25T00:15:45.293792+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:52.958539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.152034+0000","last_change":"2024-04-25T00:13:18.523019+0000","last_active":"2024-04-25T00:15:45.152034+0000","last_peered":"2024-04-25T00:15:45.152034+0000","last_clean":"2024-04-25T00:15:45.152034+0000","last_became_active":"2024-04-25T00:13:18.522677+0000","last_became_peered":"2024-04-25T00:13:18.522677+0000","last_unstale":"2024-04-25T00:15:45.152034+0000","last_undegraded":"2024-04-25T00:15:45.152034+0000","last_fullsized":"2024-04-25T00:15:45.152034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:14.470154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314556+0000","last_change":"2024-04-25T00:13:19.417672+0000","last_active":"2024-04-25T00:15:45.314556+0000","last_peered":"2024-04-25T00:15:45.314556+0000","last_clean":"2024-04-25T00:15:45.314556+0000","last_became_active":"2024-04-25T00:13:19.417323+0000","last_became_peered":"2024-04-25T00:13:19.417323+0000","last_unstale":"2024-04-25T00:15:45.314556+0000","last_undegraded":"2024-04-25T00:15:45.314556+0000","last_fullsized":"2024-04-25T00:15:45.314556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:53.734784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.039721+0000","last_change":"2024-04-25T00:13:23.862320+0000","last_active":"2024-04-25T00:15:45.039721+0000","last_peered":"2024-04-25T00:15:45.039721+0000","last_clean":"2024-04-25T00:15:45.039721+0000","last_became_active":"2024-04-25T00:13:23.861895+0000","last_became_peered":"2024-04-25T00:13:23.861895+0000","last_unstale":"2024-04-25T00:15:45.039721+0000","last_undegraded":"2024-04-25T00:15:45.039721+0000","last_fullsized":"2024-04-25T00:15:45.039721+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:16:21.324727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'49","reported_seq":106,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.267428+0000","last_change":"2024-04-25T00:13:18.520989+0000","last_active":"2024-04-25T00:15:45.267428+0000","last_peered":"2024-04-25T00:15:45.267428+0000","last_clean":"2024-04-25T00:15:45.267428+0000","last_became_active":"2024-04-25T00:13:18.520566+0000","last_became_peered":"2024-04-25T00:13:18.520566+0000","last_unstale":"2024-04-25T00:15:45.267428+0000","last_undegraded":"2024-04-25T00:15:45.267428+0000","last_fullsized":"2024-04-25T00:15:45.267428+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:22:32.532352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":86,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040218+0000","last_change":"2024-04-25T00:13:19.422741+0000","last_active":"2024-04-25T00:15:45.040218+0000","last_peered":"2024-04-25T00:15:45.040218+0000","last_clean":"2024-04-25T00:15:45.040218+0000","last_became_active":"2024-04-25T00:13:19.422379+0000","last_became_peered":"2024-04-25T00:13:19.422379+0000","last_unstale":"2024-04-25T00:15:45.040218+0000","last_undegraded":"2024-04-25T00:15:45.040218+0000","last_fullsized":"2024-04-25T00:15:45.040218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:41.337641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041168+0000","last_change":"2024-04-25T00:13:23.577888+0000","last_active":"2024-04-25T00:15:45.041168+0000","last_peered":"2024-04-25T00:15:45.041168+0000","last_clean":"2024-04-25T00:15:45.041168+0000","last_became_active":"2024-04-25T00:13:23.577440+0000","last_became_peered":"2024-04-25T00:13:23.577440+0000","last_unstale":"2024-04-25T00:15:45.041168+0000","last_undegraded":"2024-04-25T00:15:45.041168+0000","last_fullsized":"2024-04-25T00:15:45.041168+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:03.760277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041651+0000","last_change":"2024-04-25T00:13:18.527996+0000","last_active":"2024-04-25T00:15:45.041651+0000","last_peered":"2024-04-25T00:15:45.041651+0000","last_clean":"2024-04-25T00:15:45.041651+0000","last_became_active":"2024-04-25T00:13:18.527636+0000","last_became_peered":"2024-04-25T00:13:18.527636+0000","last_unstale":"2024-04-25T00:15:45.041651+0000","last_undegraded":"2024-04-25T00:15:45.041651+0000","last_fullsized":"2024-04-25T00:15:45.041651+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:52.137108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115077+0000","last_change":"2024-04-25T00:13:19.425538+0000","last_active":"2024-04-25T00:15:45.115077+0000","last_peered":"2024-04-25T00:15:45.115077+0000","last_clean":"2024-04-25T00:15:45.115077+0000","last_became_active":"2024-04-25T00:13:19.420790+0000","last_became_peered":"2024-04-25T00:13:19.420790+0000","last_unstale":"2024-04-25T00:15:45.115077+0000","last_undegraded":"2024-04-25T00:15:45.115077+0000","last_fullsized":"2024-04-25T00:15:45.115077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:04.319759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041457+0000","last_change":"2024-04-25T00:13:23.863187+0000","last_active":"2024-04-25T00:15:45.041457+0000","last_peered":"2024-04-25T00:15:45.041457+0000","last_clean":"2024-04-25T00:15:45.041457+0000","last_became_active":"2024-04-25T00:13:23.862206+0000","last_became_peered":"2024-04-25T00:13:23.862206+0000","last_unstale":"2024-04-25T00:15:45.041457+0000","last_undegraded":"2024-04-25T00:15:45.041457+0000","last_fullsized":"2024-04-25T00:15:45.041457+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:57:17.395007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309338+0000","last_change":"2024-04-25T00:13:18.529224+0000","last_active":"2024-04-25T00:15:45.309338+0000","last_peered":"2024-04-25T00:15:45.309338+0000","last_clean":"2024-04-25T00:15:45.309338+0000","last_became_active":"2024-04-25T00:13:18.528908+0000","last_became_peered":"2024-04-25T00:13:18.528908+0000","last_unstale":"2024-04-25T00:15:45.309338+0000","last_undegraded":"2024-04-25T00:15:45.309338+0000","last_fullsized":"2024-04-25T00:15:45.309338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:27.035726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151539+0000","last_change":"2024-04-25T00:13:19.422223+0000","last_active":"2024-04-25T00:15:45.151539+0000","last_peered":"2024-04-25T00:15:45.151539+0000","last_clean":"2024-04-25T00:15:45.151539+0000","last_became_active":"2024-04-25T00:13:19.421524+0000","last_became_peered":"2024-04-25T00:13:19.421524+0000","last_unstale":"2024-04-25T00:15:45.151539+0000","last_undegraded":"2024-04-25T00:15:45.151539+0000","last_fullsized":"2024-04-25T00:15:45.151539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:17:01.849503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149842+0000","last_change":"2024-04-25T00:13:23.583015+0000","last_active":"2024-04-25T00:15:45.149842+0000","last_peered":"2024-04-25T00:15:45.149842+0000","last_clean":"2024-04-25T00:15:45.149842+0000","last_became_active":"2024-04-25T00:13:23.582612+0000","last_became_peered":"2024-04-25T00:13:23.582612+0000","last_unstale":"2024-04-25T00:15:45.149842+0000","last_undegraded":"2024-04-25T00:15:45.149842+0000","last_fullsized":"2024-04-25T00:15:45.149842+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:36.872281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.039860+0000","last_change":"2024-04-25T00:13:18.530027+0000","last_active":"2024-04-25T00:15:45.039860+0000","last_peered":"2024-04-25T00:15:45.039860+0000","last_clean":"2024-04-25T00:15:45.039860+0000","last_became_active":"2024-04-25T00:13:18.529717+0000","last_became_peered":"2024-04-25T00:13:18.529717+0000","last_unstale":"2024-04-25T00:15:45.039860+0000","last_undegraded":"2024-04-25T00:15:45.039860+0000","last_fullsized":"2024-04-25T00:15:45.039860+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:23:31.020637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293701+0000","last_change":"2024-04-25T00:13:19.428650+0000","last_active":"2024-04-25T00:15:45.293701+0000","last_peered":"2024-04-25T00:15:45.293701+0000","last_clean":"2024-04-25T00:15:45.293701+0000","last_became_active":"2024-04-25T00:13:19.428156+0000","last_became_peered":"2024-04-25T00:13:19.428156+0000","last_unstale":"2024-04-25T00:15:45.293701+0000","last_undegraded":"2024-04-25T00:15:45.293701+0000","last_fullsized":"2024-04-25T00:15:45.293701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:43.094529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041458+0000","last_change":"2024-04-25T00:13:23.579125+0000","last_active":"2024-04-25T00:15:45.041458+0000","last_peered":"2024-04-25T00:15:45.041458+0000","last_clean":"2024-04-25T00:15:45.041458+0000","last_became_active":"2024-04-25T00:13:23.578728+0000","last_became_peered":"2024-04-25T00:13:23.578728+0000","last_unstale":"2024-04-25T00:15:45.041458+0000","last_undegraded":"2024-04-25T00:15:45.041458+0000","last_fullsized":"2024-04-25T00:15:45.041458+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:16:16.834399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'4","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293944+0000","last_change":"2024-04-25T00:14:11.278099+0000","last_active":"2024-04-25T00:15:45.293944+0000","last_peered":"2024-04-25T00:15:45.293944+0000","last_clean":"2024-04-25T00:15:45.293944+0000","last_became_active":"2024-04-25T00:14:11.269977+0000","last_became_peered":"2024-04-25T00:14:11.269977+0000","last_unstale":"2024-04-25T00:15:45.293944+0000","last_undegraded":"2024-04-25T00:15:45.293944+0000","last_fullsized":"2024-04-25T00:15:45.293944+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:48:59.261854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040032+0000","last_change":"2024-04-25T00:13:19.414848+0000","last_active":"2024-04-25T00:15:45.040032+0000","last_peered":"2024-04-25T00:15:45.040032+0000","last_clean":"2024-04-25T00:15:45.040032+0000","last_became_active":"2024-04-25T00:13:19.414370+0000","last_became_peered":"2024-04-25T00:13:19.414370+0000","last_unstale":"2024-04-25T00:15:45.040032+0000","last_undegraded":"2024-04-25T00:15:45.040032+0000","last_fullsized":"2024-04-25T00:15:45.040032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:41:08.779171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041407+0000","last_change":"2024-04-25T00:13:23.862577+0000","last_active":"2024-04-25T00:15:45.041407+0000","last_peered":"2024-04-25T00:15:45.041407+0000","last_clean":"2024-04-25T00:15:45.041407+0000","last_became_active":"2024-04-25T00:13:23.862098+0000","last_became_peered":"2024-04-25T00:13:23.862098+0000","last_unstale":"2024-04-25T00:15:45.041407+0000","last_undegraded":"2024-04-25T00:15:45.041407+0000","last_fullsized":"2024-04-25T00:15:45.041407+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:08:00.344041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293746+0000","last_change":"2024-04-25T00:14:11.274790+0000","last_active":"2024-04-25T00:15:45.293746+0000","last_peered":"2024-04-25T00:15:45.293746+0000","last_clean":"2024-04-25T00:15:45.293746+0000","last_became_active":"2024-04-25T00:14:11.267132+0000","last_became_peered":"2024-04-25T00:14:11.267132+0000","last_unstale":"2024-04-25T00:15:45.293746+0000","last_undegraded":"2024-04-25T00:15:45.293746+0000","last_fullsized":"2024-04-25T00:15:45.293746+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:58.302788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312161+0000","last_change":"2024-04-25T00:13:19.427992+0000","last_active":"2024-04-25T00:15:45.312161+0000","last_peered":"2024-04-25T00:15:45.312161+0000","last_clean":"2024-04-25T00:15:45.312161+0000","last_became_active":"2024-04-25T00:13:19.427665+0000","last_became_peered":"2024-04-25T00:13:19.427665+0000","last_unstale":"2024-04-25T00:15:45.312161+0000","last_undegraded":"2024-04-25T00:15:45.312161+0000","last_fullsized":"2024-04-25T00:15:45.312161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:12:31.582185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309231+0000","last_change":"2024-04-25T00:13:23.582430+0000","last_active":"2024-04-25T00:15:45.309231+0000","last_peered":"2024-04-25T00:15:45.309231+0000","last_clean":"2024-04-25T00:15:45.309231+0000","last_became_active":"2024-04-25T00:13:23.582008+0000","last_became_peered":"2024-04-25T00:13:23.582008+0000","last_unstale":"2024-04-25T00:15:45.309231+0000","last_undegraded":"2024-04-25T00:15:45.309231+0000","last_fullsized":"2024-04-25T00:15:45.309231+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:11.068720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"94'7","reported_seq":163,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.754413+0000","last_change":"2024-04-25T00:13:18.528030+0000","last_active":"2024-04-25T00:15:45.754413+0000","last_peered":"2024-04-25T00:15:45.754413+0000","last_clean":"2024-04-25T00:15:45.754413+0000","last_became_active":"2024-04-25T00:13:18.527636+0000","last_became_peered":"2024-04-25T00:13:18.527636+0000","last_unstale":"2024-04-25T00:15:45.754413+0000","last_undegraded":"2024-04-25T00:15:45.754413+0000","last_fullsized":"2024-04-25T00:15:45.754413+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:36.311810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1952,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312965+0000","last_change":"2024-04-25T00:13:19.428181+0000","last_active":"2024-04-25T00:15:45.312965+0000","last_peered":"2024-04-25T00:15:45.312965+0000","last_clean":"2024-04-25T00:15:45.312965+0000","last_became_active":"2024-04-25T00:13:19.427785+0000","last_became_peered":"2024-04-25T00:13:19.427785+0000","last_unstale":"2024-04-25T00:15:45.312965+0000","last_undegraded":"2024-04-25T00:15:45.312965+0000","last_fullsized":"2024-04-25T00:15:45.312965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:31.745679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.114716+0000","last_change":"2024-04-25T00:14:11.271507+0000","last_active":"2024-04-25T00:15:45.114716+0000","last_peered":"2024-04-25T00:15:45.114716+0000","last_clean":"2024-04-25T00:15:45.114716+0000","last_became_active":"2024-04-25T00:14:11.270452+0000","last_became_peered":"2024-04-25T00:14:11.270452+0000","last_unstale":"2024-04-25T00:15:45.114716+0000","last_undegraded":"2024-04-25T00:15:45.114716+0000","last_fullsized":"2024-04-25T00:15:45.114716+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:19.955242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'41","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.262583+0000","last_change":"2024-04-25T00:13:18.529622+0000","last_active":"2024-04-25T00:15:45.262583+0000","last_peered":"2024-04-25T00:15:45.262583+0000","last_clean":"2024-04-25T00:15:45.262583+0000","last_became_active":"2024-04-25T00:13:18.529305+0000","last_became_peered":"2024-04-25T00:13:18.529305+0000","last_unstale":"2024-04-25T00:15:45.262583+0000","last_undegraded":"2024-04-25T00:15:45.262583+0000","last_fullsized":"2024-04-25T00:15:45.262583+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:10:43.334712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312206+0000","last_change":"2024-04-25T00:13:19.421783+0000","last_active":"2024-04-25T00:15:45.312206+0000","last_peered":"2024-04-25T00:15:45.312206+0000","last_clean":"2024-04-25T00:15:45.312206+0000","last_became_active":"2024-04-25T00:13:19.421447+0000","last_became_peered":"2024-04-25T00:13:19.421447+0000","last_unstale":"2024-04-25T00:15:45.312206+0000","last_undegraded":"2024-04-25T00:15:45.312206+0000","last_fullsized":"2024-04-25T00:15:45.312206+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:16.107220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158542+0000","last_change":"2024-04-25T00:13:23.578252+0000","last_active":"2024-04-25T00:15:45.158542+0000","last_peered":"2024-04-25T00:15:45.158542+0000","last_clean":"2024-04-25T00:15:45.158542+0000","last_became_active":"2024-04-25T00:13:23.577779+0000","last_became_peered":"2024-04-25T00:13:23.577779+0000","last_unstale":"2024-04-25T00:15:45.158542+0000","last_undegraded":"2024-04-25T00:15:45.158542+0000","last_fullsized":"2024-04-25T00:15:45.158542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:43.822447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314487+0000","last_change":"2024-04-25T00:13:18.523583+0000","last_active":"2024-04-25T00:15:45.314487+0000","last_peered":"2024-04-25T00:15:45.314487+0000","last_clean":"2024-04-25T00:15:45.314487+0000","last_became_active":"2024-04-25T00:13:18.523238+0000","last_became_peered":"2024-04-25T00:13:18.523238+0000","last_unstale":"2024-04-25T00:15:45.314487+0000","last_undegraded":"2024-04-25T00:15:45.314487+0000","last_fullsized":"2024-04-25T00:15:45.314487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:03.869276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.313212+0000","last_change":"2024-04-25T00:13:19.426021+0000","last_active":"2024-04-25T00:15:45.313212+0000","last_peered":"2024-04-25T00:15:45.313212+0000","last_clean":"2024-04-25T00:15:45.313212+0000","last_became_active":"2024-04-25T00:13:19.425676+0000","last_became_peered":"2024-04-25T00:13:19.425676+0000","last_unstale":"2024-04-25T00:15:45.313212+0000","last_undegraded":"2024-04-25T00:15:45.313212+0000","last_fullsized":"2024-04-25T00:15:45.313212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:10.044202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293774+0000","last_change":"2024-04-25T00:13:19.422402+0000","last_active":"2024-04-25T00:15:45.293774+0000","last_peered":"2024-04-25T00:15:45.293774+0000","last_clean":"2024-04-25T00:15:45.293774+0000","last_became_active":"2024-04-25T00:13:19.422057+0000","last_became_peered":"2024-04-25T00:13:19.422057+0000","last_unstale":"2024-04-25T00:15:45.293774+0000","last_undegraded":"2024-04-25T00:15:45.293774+0000","last_fullsized":"2024-04-25T00:15:45.293774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:31:43.703030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041874+0000","last_change":"2024-04-25T00:13:19.425434+0000","last_active":"2024-04-25T00:15:45.041874+0000","last_peered":"2024-04-25T00:15:45.041874+0000","last_clean":"2024-04-25T00:15:45.041874+0000","last_became_active":"2024-04-25T00:13:19.425051+0000","last_became_peered":"2024-04-25T00:13:19.425051+0000","last_unstale":"2024-04-25T00:15:45.041874+0000","last_undegraded":"2024-04-25T00:15:45.041874+0000","last_fullsized":"2024-04-25T00:15:45.041874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:17.568415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041226+0000","last_change":"2024-04-25T00:13:19.413835+0000","last_active":"2024-04-25T00:15:45.041226+0000","last_peered":"2024-04-25T00:15:45.041226+0000","last_clean":"2024-04-25T00:15:45.041226+0000","last_became_active":"2024-04-25T00:13:19.413406+0000","last_became_peered":"2024-04-25T00:13:19.413406+0000","last_unstale":"2024-04-25T00:15:45.041226+0000","last_undegraded":"2024-04-25T00:15:45.041226+0000","last_fullsized":"2024-04-25T00:15:45.041226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:33.852391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.114923+0000","last_change":"2024-04-25T00:13:19.426875+0000","last_active":"2024-04-25T00:15:45.114923+0000","last_peered":"2024-04-25T00:15:45.114923+0000","last_clean":"2024-04-25T00:15:45.114923+0000","last_became_active":"2024-04-25T00:13:19.425828+0000","last_became_peered":"2024-04-25T00:13:19.425828+0000","last_unstale":"2024-04-25T00:15:45.114923+0000","last_undegraded":"2024-04-25T00:15:45.114923+0000","last_fullsized":"2024-04-25T00:15:45.114923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:21.119551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314721+0000","last_change":"2024-04-25T00:13:19.423637+0000","last_active":"2024-04-25T00:15:45.314721+0000","last_peered":"2024-04-25T00:15:45.314721+0000","last_clean":"2024-04-25T00:15:45.314721+0000","last_became_active":"2024-04-25T00:13:19.423272+0000","last_became_peered":"2024-04-25T00:13:19.423272+0000","last_unstale":"2024-04-25T00:15:45.314721+0000","last_undegraded":"2024-04-25T00:15:45.314721+0000","last_fullsized":"2024-04-25T00:15:45.314721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:32:38.794966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293982+0000","last_change":"2024-04-25T00:13:19.423538+0000","last_active":"2024-04-25T00:15:45.293982+0000","last_peered":"2024-04-25T00:15:45.293982+0000","last_clean":"2024-04-25T00:15:45.293982+0000","last_became_active":"2024-04-25T00:13:19.423028+0000","last_became_peered":"2024-04-25T00:13:19.423028+0000","last_unstale":"2024-04-25T00:15:45.293982+0000","last_undegraded":"2024-04-25T00:15:45.293982+0000","last_fullsized":"2024-04-25T00:15:45.293982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:29:41.975044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041851+0000","last_change":"2024-04-25T00:13:19.416899+0000","last_active":"2024-04-25T00:15:45.041851+0000","last_peered":"2024-04-25T00:15:45.041851+0000","last_clean":"2024-04-25T00:15:45.041851+0000","last_became_active":"2024-04-25T00:13:19.416505+0000","last_became_peered":"2024-04-25T00:13:19.416505+0000","last_unstale":"2024-04-25T00:15:45.041851+0000","last_undegraded":"2024-04-25T00:15:45.041851+0000","last_fullsized":"2024-04-25T00:15:45.041851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:37:40.204119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040400+0000","last_change":"2024-04-25T00:13:19.426215+0000","last_active":"2024-04-25T00:15:45.040400+0000","last_peered":"2024-04-25T00:15:45.040400+0000","last_clean":"2024-04-25T00:15:45.040400+0000","last_became_active":"2024-04-25T00:13:19.422804+0000","last_became_peered":"2024-04-25T00:13:19.422804+0000","last_unstale":"2024-04-25T00:15:45.040400+0000","last_undegraded":"2024-04-25T00:15:45.040400+0000","last_fullsized":"2024-04-25T00:15:45.040400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:24:44.630891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158710+0000","last_change":"2024-04-25T00:13:19.413564+0000","last_active":"2024-04-25T00:15:45.158710+0000","last_peered":"2024-04-25T00:15:45.158710+0000","last_clean":"2024-04-25T00:15:45.158710+0000","last_became_active":"2024-04-25T00:13:19.413223+0000","last_became_peered":"2024-04-25T00:13:19.413223+0000","last_unstale":"2024-04-25T00:15:45.158710+0000","last_undegraded":"2024-04-25T00:15:45.158710+0000","last_fullsized":"2024-04-25T00:15:45.158710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:45.629018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.039748+0000","last_change":"2024-04-25T00:13:19.425945+0000","last_active":"2024-04-25T00:15:45.039748+0000","last_peered":"2024-04-25T00:15:45.039748+0000","last_clean":"2024-04-25T00:15:45.039748+0000","last_became_active":"2024-04-25T00:13:19.422499+0000","last_became_peered":"2024-04-25T00:13:19.422499+0000","last_unstale":"2024-04-25T00:15:45.039748+0000","last_undegraded":"2024-04-25T00:15:45.039748+0000","last_fullsized":"2024-04-25T00:15:45.039748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:45.299300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309420+0000","last_change":"2024-04-25T00:15:11.584671+0000","last_active":"2024-04-25T00:15:45.309420+0000","last_peered":"2024-04-25T00:15:45.309420+0000","last_clean":"2024-04-25T00:15:45.309420+0000","last_became_active":"2024-04-25T00:15:11.584144+0000","last_became_peered":"2024-04-25T00:15:11.584144+0000","last_unstale":"2024-04-25T00:15:45.309420+0000","last_undegraded":"2024-04-25T00:15:45.309420+0000","last_fullsized":"2024-04-25T00:15:45.309420+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:37:52.522586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314719+0000","last_change":"2024-04-25T00:13:19.420550+0000","last_active":"2024-04-25T00:15:45.314719+0000","last_peered":"2024-04-25T00:15:45.314719+0000","last_clean":"2024-04-25T00:15:45.314719+0000","last_became_active":"2024-04-25T00:13:19.420235+0000","last_became_peered":"2024-04-25T00:13:19.420235+0000","last_unstale":"2024-04-25T00:15:45.314719+0000","last_undegraded":"2024-04-25T00:15:45.314719+0000","last_fullsized":"2024-04-25T00:15:45.314719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:59:36.543776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293898+0000","last_change":"2024-04-25T00:13:19.429995+0000","last_active":"2024-04-25T00:15:45.293898+0000","last_peered":"2024-04-25T00:15:45.293898+0000","last_clean":"2024-04-25T00:15:45.293898+0000","last_became_active":"2024-04-25T00:13:19.429185+0000","last_became_peered":"2024-04-25T00:13:19.429185+0000","last_unstale":"2024-04-25T00:15:45.293898+0000","last_undegraded":"2024-04-25T00:15:45.293898+0000","last_fullsized":"2024-04-25T00:15:45.293898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:29:27.974479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107352,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600055,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1337,"num_read_kb":783,"num_write":372,"num_write_kb":1466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":1800165,"data_compressed":80712,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":300,"ondisk_log_size":300,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383723,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":520207,"data_compressed":26578,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645358,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1212416,"data_stored":1315825,"data_compressed":73110,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":927315,"data_compressed":48714,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":923597,"data_compressed":48714,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1114112,"data_stored":1127718,"data_compressed":65352,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":526206,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":583001,"data_compressed":28180,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":527745,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1130496,"data_stored":594343,"data_compressed":31170,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":1050031,"data_compressed":54870,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":530113,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32292,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712860,"statfs":{"total":95995035648,"available":95961968640,"internally_reserved":0,"allocated":1142784,"data_stored":1253669,"data_compressed":71508,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":69255,"data_compressed":1602,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":69558,"data_compressed":1602,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":134420,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":200672,"data_compressed":7758,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:15:50.134 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:15:50.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.271+0000 7f12c7161640 1 -- 172.21.15.27:0/2569456765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c0103910 msgr2=0x7f12c0103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:50.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.271+0000 7f12c7161640 1 --2- 172.21.15.27:0/2569456765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c0103910 0x7f12c0103d90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009930 tx=0x7f12a802a420 comp rx=0 tx=0).stop 2024-04-25T00:15:50.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.271+0000 7f12c7161640 1 -- 172.21.15.27:0/2569456765 shutdown_connections 2024-04-25T00:15:50.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.271+0000 7f12c7161640 1 --2- 172.21.15.27:0/2569456765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12c01042d0 0x7f12c01107e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.271+0000 7f12c7161640 1 --2- 172.21.15.27:0/2569456765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c0103910 0x7f12c0103d90 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.271+0000 7f12c7161640 1 --2- 172.21.15.27:0/2569456765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12c0109920 0x7f12c0109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.271+0000 7f12c7161640 1 -- 172.21.15.27:0/2569456765 >> 172.21.15.27:0/2569456765 conn(0x7f12c00ff440 msgr2=0x7f12c0101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:50.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.271+0000 7f12c7161640 1 -- 172.21.15.27:0/2569456765 shutdown_connections 2024-04-25T00:15:50.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.271+0000 7f12c7161640 1 -- 172.21.15.27:0/2569456765 wait complete. 2024-04-25T00:15:50.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.271+0000 7f12c7161640 1 Processor -- start 2024-04-25T00:15:50.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.271+0000 7f12c7161640 1 -- start start 2024-04-25T00:15:50.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.271+0000 7f12c7161640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12c0103910 0x7f12c019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:50.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.271+0000 7f12c7161640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c01042d0 0x7f12c019a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:50.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.271+0000 7f12c7161640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12c0109920 0x7f12c019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12c7161640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f12c0113d80 con 0x7f12c01042d0 2024-04-25T00:15:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12c7161640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f12c0113c00 con 0x7f12c0109920 2024-04-25T00:15:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12c7161640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12c0113f00 con 0x7f12c0103910 2024-04-25T00:15:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12bffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c01042d0 0x7f12c019a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12bffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c01042d0 0x7f12c019a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47708/0 (socket says 172.21.15.27:47708) 2024-04-25T00:15:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12bffff640 1 -- 172.21.15.27:0/2637066012 learned_addr learned my addr 172.21.15.27:0/2637066012 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12c4efd640 1 --2- 172.21.15.27:0/2637066012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12c0103910 0x7f12c019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12c56fe640 1 --2- 172.21.15.27:0/2637066012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12c0109920 0x7f12c019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12bffff640 1 -- 172.21.15.27:0/2637066012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12c0103910 msgr2=0x7f12c019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12bffff640 1 --2- 172.21.15.27:0/2637066012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12c0103910 0x7f12c019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12bffff640 1 -- 172.21.15.27:0/2637066012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12c0109920 msgr2=0x7f12c019eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12bffff640 1 --2- 172.21.15.27:0/2637066012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12c0109920 0x7f12c019eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12bffff640 1 -- 172.21.15.27:0/2637066012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c019f270 con 0x7f12c01042d0 2024-04-25T00:15:50.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12c56fe640 1 --2- 172.21.15.27:0/2637066012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12c0109920 0x7f12c019eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:50.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12c4efd640 1 --2- 172.21.15.27:0/2637066012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12c0103910 0x7f12c019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:50.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12bffff640 1 --2- 172.21.15.27:0/2637066012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c01042d0 0x7f12c019a7c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f12a802a940 tx=0x7f12a8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:50.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12bdffb640 1 -- 172.21.15.27:0/2637066012 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a803f070 con 0x7f12c01042d0 2024-04-25T00:15:50.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12c7161640 1 -- 172.21.15.27:0/2637066012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12c019f500 con 0x7f12c01042d0 2024-04-25T00:15:50.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12bdffb640 1 -- 172.21.15.27:0/2637066012 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12a8031c90 con 0x7f12c01042d0 2024-04-25T00:15:50.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.272+0000 7f12bdffb640 1 -- 172.21.15.27:0/2637066012 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a8035030 con 0x7f12c01042d0 2024-04-25T00:15:50.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.273+0000 7f12c7161640 1 -- 172.21.15.27:0/2637066012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12c01a6df0 con 0x7f12c01042d0 2024-04-25T00:15:50.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.273+0000 7f12c7161640 1 -- 172.21.15.27:0/2637066012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12c00689b0 con 0x7f12c01042d0 2024-04-25T00:15:50.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.275+0000 7f12bdffb640 1 -- 172.21.15.27:0/2637066012 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f12a8042070 con 0x7f12c01042d0 2024-04-25T00:15:50.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.275+0000 7f12bdffb640 1 --2- 172.21.15.27:0/2637066012 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f12ac0a2e20 0x7f12ac0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:50.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.276+0000 7f12c4efd640 1 --2- 172.21.15.27:0/2637066012 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f12ac0a2e20 0x7f12ac0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:50.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.276+0000 7f12bdffb640 1 -- 172.21.15.27:0/2637066012 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f12a80f00e0 con 0x7f12c01042d0 2024-04-25T00:15:50.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.276+0000 7f12c4efd640 1 --2- 172.21.15.27:0/2637066012 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f12ac0a2e20 0x7f12ac0a52e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009f40 tx=0x7f12b0005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.280+0000 7f12bdffb640 1 -- 172.21.15.27:0/2637066012 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f12a80bc810 con 0x7f12c01042d0 2024-04-25T00:15:50.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:15:50.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:15:50.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:15:50.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:15:50.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3873171947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:50.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: Updating smithi139:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T00:15:50.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: Updating smithi027:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T00:15:50.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: Updating smithi153:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T00:15:50.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3500465566' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:50.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-25T00:15:50.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:15:50.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:15:50.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:15:50.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:50 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/687009595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:50.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.462+0000 7f12c7161640 1 -- 172.21.15.27:0/2637066012 --> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f12c010bee0 con 0x7f12ac0a2e20 2024-04-25T00:15:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.471+0000 7f12bdffb640 1 -- 172.21.15.27:0/2637066012 <== mgr.15564 v2:172.21.15.27:6800/1076858552 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339735 (secure 0 0 0) 0x7f12c010bee0 con 0x7f12ac0a2e20 2024-04-25T00:15:50.475 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:50.477 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T00:15:50.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.475+0000 7f12c7161640 1 -- 172.21.15.27:0/2637066012 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f12ac0a2e20 msgr2=0x7f12ac0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:50.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.475+0000 7f12c7161640 1 --2- 172.21.15.27:0/2637066012 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f12ac0a2e20 0x7f12ac0a52e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009f40 tx=0x7f12b0005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:50.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.475+0000 7f12c7161640 1 -- 172.21.15.27:0/2637066012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c01042d0 msgr2=0x7f12c019a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:50.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.475+0000 7f12c7161640 1 --2- 172.21.15.27:0/2637066012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c01042d0 0x7f12c019a7c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f12a802a940 tx=0x7f12a8003b90 comp rx=0 tx=0).stop 2024-04-25T00:15:50.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.475+0000 7f12c7161640 1 -- 172.21.15.27:0/2637066012 shutdown_connections 2024-04-25T00:15:50.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.475+0000 7f12c7161640 1 --2- 172.21.15.27:0/2637066012 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f12ac0a2e20 0x7f12ac0a52e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.475+0000 7f12c7161640 1 --2- 172.21.15.27:0/2637066012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12c0109920 0x7f12c019eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.475+0000 7f12c7161640 1 --2- 172.21.15.27:0/2637066012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c01042d0 0x7f12c019a7c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.475+0000 7f12c7161640 1 --2- 172.21.15.27:0/2637066012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12c0103910 0x7f12c019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.475+0000 7f12c7161640 1 -- 172.21.15.27:0/2637066012 >> 172.21.15.27:0/2637066012 conn(0x7f12c00ff440 msgr2=0x7f12c00ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:50.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.475+0000 7f12c7161640 1 -- 172.21.15.27:0/2637066012 shutdown_connections 2024-04-25T00:15:50.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.475+0000 7f12c7161640 1 -- 172.21.15.27:0/2637066012 wait complete. 2024-04-25T00:15:50.486 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-25T00:15:49.051541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190657,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1473,"num_read_kb":902,"num_write":555,"num_write_kb":4406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":939,"ondisk_log_size":939,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":349548,"kb_used_data":12844,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124592276,"statfs":{"total":1151940427776,"available":1151582490624,"internally_reserved":0,"allocated":13152256,"data_stored":9879770,"data_compressed":536900,"data_compressed_allocated":729088,"data_compressed_original":4308992,"omap_allocated":19143,"internal_metadata":344634681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.987382"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.294151+0000","last_change":"2024-04-25T00:13:19.429658+0000","last_active":"2024-04-25T00:15:45.294151+0000","last_peered":"2024-04-25T00:15:45.294151+0000","last_clean":"2024-04-25T00:15:45.294151+0000","last_became_active":"2024-04-25T00:13:19.428426+0000","last_became_peered":"2024-04-25T00:13:19.428426+0000","last_unstale":"2024-04-25T00:15:45.294151+0000","last_undegraded":"2024-04-25T00:15:45.294151+0000","last_fullsized":"2024-04-25T00:15:45.294151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:10.814361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115194+0000","last_change":"2024-04-25T00:13:19.425504+0000","last_active":"2024-04-25T00:15:45.115194+0000","last_peered":"2024-04-25T00:15:45.115194+0000","last_clean":"2024-04-25T00:15:45.115194+0000","last_became_active":"2024-04-25T00:13:19.425170+0000","last_became_peered":"2024-04-25T00:13:19.425170+0000","last_unstale":"2024-04-25T00:15:45.115194+0000","last_undegraded":"2024-04-25T00:15:45.115194+0000","last_fullsized":"2024-04-25T00:15:45.115194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:31:13.826981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.114835+0000","last_change":"2024-04-25T00:13:19.420842+0000","last_active":"2024-04-25T00:15:45.114835+0000","last_peered":"2024-04-25T00:15:45.114835+0000","last_clean":"2024-04-25T00:15:45.114835+0000","last_became_active":"2024-04-25T00:13:19.420241+0000","last_became_peered":"2024-04-25T00:13:19.420241+0000","last_unstale":"2024-04-25T00:15:45.114835+0000","last_undegraded":"2024-04-25T00:15:45.114835+0000","last_fullsized":"2024-04-25T00:15:45.114835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:52:39.500432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149314+0000","last_change":"2024-04-25T00:14:11.267688+0000","last_active":"2024-04-25T00:15:45.149314+0000","last_peered":"2024-04-25T00:15:45.149314+0000","last_clean":"2024-04-25T00:15:45.149314+0000","last_became_active":"2024-04-25T00:14:11.267225+0000","last_became_peered":"2024-04-25T00:14:11.267225+0000","last_unstale":"2024-04-25T00:15:45.149314+0000","last_undegraded":"2024-04-25T00:15:45.149314+0000","last_fullsized":"2024-04-25T00:15:45.149314+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:15.867489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151754+0000","last_change":"2024-04-25T00:13:19.416519+0000","last_active":"2024-04-25T00:15:45.151754+0000","last_peered":"2024-04-25T00:15:45.151754+0000","last_clean":"2024-04-25T00:15:45.151754+0000","last_became_active":"2024-04-25T00:13:19.416080+0000","last_became_peered":"2024-04-25T00:13:19.416080+0000","last_unstale":"2024-04-25T00:15:45.151754+0000","last_undegraded":"2024-04-25T00:15:45.151754+0000","last_fullsized":"2024-04-25T00:15:45.151754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:04.450160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.162148+0000","last_change":"2024-04-25T00:13:19.424378+0000","last_active":"2024-04-25T00:15:45.162148+0000","last_peered":"2024-04-25T00:15:45.162148+0000","last_clean":"2024-04-25T00:15:45.162148+0000","last_became_active":"2024-04-25T00:13:19.423918+0000","last_became_peered":"2024-04-25T00:13:19.423918+0000","last_unstale":"2024-04-25T00:15:45.162148+0000","last_undegraded":"2024-04-25T00:15:45.162148+0000","last_fullsized":"2024-04-25T00:15:45.162148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:01:11.397328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.311886+0000","last_change":"2024-04-25T00:13:19.424729+0000","last_active":"2024-04-25T00:15:45.311886+0000","last_peered":"2024-04-25T00:15:45.311886+0000","last_clean":"2024-04-25T00:15:45.311886+0000","last_became_active":"2024-04-25T00:13:19.422874+0000","last_became_peered":"2024-04-25T00:13:19.422874+0000","last_unstale":"2024-04-25T00:15:45.311886+0000","last_undegraded":"2024-04-25T00:15:45.311886+0000","last_fullsized":"2024-04-25T00:15:45.311886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:52:39.483589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.039933+0000","last_change":"2024-04-25T00:13:19.418043+0000","last_active":"2024-04-25T00:15:45.039933+0000","last_peered":"2024-04-25T00:15:45.039933+0000","last_clean":"2024-04-25T00:15:45.039933+0000","last_became_active":"2024-04-25T00:13:19.417443+0000","last_became_peered":"2024-04-25T00:13:19.417443+0000","last_unstale":"2024-04-25T00:15:45.039933+0000","last_undegraded":"2024-04-25T00:15:45.039933+0000","last_fullsized":"2024-04-25T00:15:45.039933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:47:53.141744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309810+0000","last_change":"2024-04-25T00:13:19.420989+0000","last_active":"2024-04-25T00:15:45.309810+0000","last_peered":"2024-04-25T00:15:45.309810+0000","last_clean":"2024-04-25T00:15:45.309810+0000","last_became_active":"2024-04-25T00:13:19.420669+0000","last_became_peered":"2024-04-25T00:13:19.420669+0000","last_unstale":"2024-04-25T00:15:45.309810+0000","last_undegraded":"2024-04-25T00:15:45.309810+0000","last_fullsized":"2024-04-25T00:15:45.309810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:18.236132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.162583+0000","last_change":"2024-04-25T00:13:19.424176+0000","last_active":"2024-04-25T00:15:45.162583+0000","last_peered":"2024-04-25T00:15:45.162583+0000","last_clean":"2024-04-25T00:15:45.162583+0000","last_became_active":"2024-04-25T00:13:19.423656+0000","last_became_peered":"2024-04-25T00:13:19.423656+0000","last_unstale":"2024-04-25T00:15:45.162583+0000","last_undegraded":"2024-04-25T00:15:45.162583+0000","last_fullsized":"2024-04-25T00:15:45.162583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:24:41.697077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312946+0000","last_change":"2024-04-25T00:13:19.419624+0000","last_active":"2024-04-25T00:15:45.312946+0000","last_peered":"2024-04-25T00:15:45.312946+0000","last_clean":"2024-04-25T00:15:45.312946+0000","last_became_active":"2024-04-25T00:13:19.419263+0000","last_became_peered":"2024-04-25T00:13:19.419263+0000","last_unstale":"2024-04-25T00:15:45.312946+0000","last_undegraded":"2024-04-25T00:15:45.312946+0000","last_fullsized":"2024-04-25T00:15:45.312946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:29.130733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041558+0000","last_change":"2024-04-25T00:13:19.427865+0000","last_active":"2024-04-25T00:15:45.041558+0000","last_peered":"2024-04-25T00:15:45.041558+0000","last_clean":"2024-04-25T00:15:45.041558+0000","last_became_active":"2024-04-25T00:13:19.427538+0000","last_became_peered":"2024-04-25T00:13:19.427538+0000","last_unstale":"2024-04-25T00:15:45.041558+0000","last_undegraded":"2024-04-25T00:15:45.041558+0000","last_fullsized":"2024-04-25T00:15:45.041558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:05.136948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314986+0000","last_change":"2024-04-25T00:13:19.419261+0000","last_active":"2024-04-25T00:15:45.314986+0000","last_peered":"2024-04-25T00:15:45.314986+0000","last_clean":"2024-04-25T00:15:45.314986+0000","last_became_active":"2024-04-25T00:13:19.418938+0000","last_became_peered":"2024-04-25T00:13:19.418938+0000","last_unstale":"2024-04-25T00:15:45.314986+0000","last_undegraded":"2024-04-25T00:15:45.314986+0000","last_fullsized":"2024-04-25T00:15:45.314986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:44.971640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.114768+0000","last_change":"2024-04-25T00:13:19.417654+0000","last_active":"2024-04-25T00:15:45.114768+0000","last_peered":"2024-04-25T00:15:45.114768+0000","last_clean":"2024-04-25T00:15:45.114768+0000","last_became_active":"2024-04-25T00:13:19.417319+0000","last_became_peered":"2024-04-25T00:13:19.417319+0000","last_unstale":"2024-04-25T00:15:45.114768+0000","last_undegraded":"2024-04-25T00:15:45.114768+0000","last_fullsized":"2024-04-25T00:15:45.114768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:43:54.447937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115100+0000","last_change":"2024-04-25T00:13:19.420761+0000","last_active":"2024-04-25T00:15:45.115100+0000","last_peered":"2024-04-25T00:15:45.115100+0000","last_clean":"2024-04-25T00:15:45.115100+0000","last_became_active":"2024-04-25T00:13:19.420334+0000","last_became_peered":"2024-04-25T00:13:19.420334+0000","last_unstale":"2024-04-25T00:15:45.115100+0000","last_undegraded":"2024-04-25T00:15:45.115100+0000","last_fullsized":"2024-04-25T00:15:45.115100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:20.136470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309282+0000","last_change":"2024-04-25T00:13:19.420132+0000","last_active":"2024-04-25T00:15:45.309282+0000","last_peered":"2024-04-25T00:15:45.309282+0000","last_clean":"2024-04-25T00:15:45.309282+0000","last_became_active":"2024-04-25T00:13:19.418910+0000","last_became_peered":"2024-04-25T00:13:19.418910+0000","last_unstale":"2024-04-25T00:15:45.309282+0000","last_undegraded":"2024-04-25T00:15:45.309282+0000","last_fullsized":"2024-04-25T00:15:45.309282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:17.895412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.294056+0000","last_change":"2024-04-25T00:13:19.430242+0000","last_active":"2024-04-25T00:15:45.294056+0000","last_peered":"2024-04-25T00:15:45.294056+0000","last_clean":"2024-04-25T00:15:45.294056+0000","last_became_active":"2024-04-25T00:13:19.429077+0000","last_became_peered":"2024-04-25T00:13:19.429077+0000","last_unstale":"2024-04-25T00:15:45.294056+0000","last_undegraded":"2024-04-25T00:15:45.294056+0000","last_fullsized":"2024-04-25T00:15:45.294056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:34:14.351157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149646+0000","last_change":"2024-04-25T00:13:19.414375+0000","last_active":"2024-04-25T00:15:45.149646+0000","last_peered":"2024-04-25T00:15:45.149646+0000","last_clean":"2024-04-25T00:15:45.149646+0000","last_became_active":"2024-04-25T00:13:19.414003+0000","last_became_peered":"2024-04-25T00:13:19.414003+0000","last_unstale":"2024-04-25T00:15:45.149646+0000","last_undegraded":"2024-04-25T00:15:45.149646+0000","last_fullsized":"2024-04-25T00:15:45.149646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:43:50.612743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041639+0000","last_change":"2024-04-25T00:13:19.415322+0000","last_active":"2024-04-25T00:15:45.041639+0000","last_peered":"2024-04-25T00:15:45.041639+0000","last_clean":"2024-04-25T00:15:45.041639+0000","last_became_active":"2024-04-25T00:13:19.414925+0000","last_became_peered":"2024-04-25T00:13:19.414925+0000","last_unstale":"2024-04-25T00:15:45.041639+0000","last_undegraded":"2024-04-25T00:15:45.041639+0000","last_fullsized":"2024-04-25T00:15:45.041639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:58.140640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041305+0000","last_change":"2024-04-25T00:13:23.584763+0000","last_active":"2024-04-25T00:15:45.041305+0000","last_peered":"2024-04-25T00:15:45.041305+0000","last_clean":"2024-04-25T00:15:45.041305+0000","last_became_active":"2024-04-25T00:13:23.584303+0000","last_became_peered":"2024-04-25T00:13:23.584303+0000","last_unstale":"2024-04-25T00:15:45.041305+0000","last_undegraded":"2024-04-25T00:15:45.041305+0000","last_fullsized":"2024-04-25T00:15:45.041305+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:54:11.447867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'14","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149948+0000","last_change":"2024-04-25T00:13:18.519576+0000","last_active":"2024-04-25T00:15:45.149948+0000","last_peered":"2024-04-25T00:15:45.149948+0000","last_clean":"2024-04-25T00:15:45.149948+0000","last_became_active":"2024-04-25T00:13:18.519201+0000","last_became_peered":"2024-04-25T00:13:18.519201+0000","last_unstale":"2024-04-25T00:15:45.149948+0000","last_undegraded":"2024-04-25T00:15:45.149948+0000","last_fullsized":"2024-04-25T00:15:45.149948+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:19.681096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040313+0000","last_change":"2024-04-25T00:13:19.419098+0000","last_active":"2024-04-25T00:15:45.040313+0000","last_peered":"2024-04-25T00:15:45.040313+0000","last_clean":"2024-04-25T00:15:45.040313+0000","last_became_active":"2024-04-25T00:13:19.418649+0000","last_became_peered":"2024-04-25T00:13:19.418649+0000","last_unstale":"2024-04-25T00:15:45.040313+0000","last_undegraded":"2024-04-25T00:15:45.040313+0000","last_fullsized":"2024-04-25T00:15:45.040313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:04.561470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041042+0000","last_change":"2024-04-25T00:13:23.580354+0000","last_active":"2024-04-25T00:15:45.041042+0000","last_peered":"2024-04-25T00:15:45.041042+0000","last_clean":"2024-04-25T00:15:45.041042+0000","last_became_active":"2024-04-25T00:13:23.579944+0000","last_became_peered":"2024-04-25T00:13:23.579944+0000","last_unstale":"2024-04-25T00:15:45.041042+0000","last_undegraded":"2024-04-25T00:15:45.041042+0000","last_fullsized":"2024-04-25T00:15:45.041042+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:48:57.788974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040339+0000","last_change":"2024-04-25T00:13:18.527471+0000","last_active":"2024-04-25T00:15:45.040339+0000","last_peered":"2024-04-25T00:15:45.040339+0000","last_clean":"2024-04-25T00:15:45.040339+0000","last_became_active":"2024-04-25T00:13:18.527138+0000","last_became_peered":"2024-04-25T00:13:18.527138+0000","last_unstale":"2024-04-25T00:15:45.040339+0000","last_undegraded":"2024-04-25T00:15:45.040339+0000","last_fullsized":"2024-04-25T00:15:45.040339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:38.003719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115017+0000","last_change":"2024-04-25T00:13:19.420563+0000","last_active":"2024-04-25T00:15:45.115017+0000","last_peered":"2024-04-25T00:15:45.115017+0000","last_clean":"2024-04-25T00:15:45.115017+0000","last_became_active":"2024-04-25T00:13:19.420142+0000","last_became_peered":"2024-04-25T00:13:19.420142+0000","last_unstale":"2024-04-25T00:15:45.115017+0000","last_undegraded":"2024-04-25T00:15:45.115017+0000","last_fullsized":"2024-04-25T00:15:45.115017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:23:08.423795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041136+0000","last_change":"2024-04-25T00:14:11.271321+0000","last_active":"2024-04-25T00:15:45.041136+0000","last_peered":"2024-04-25T00:15:45.041136+0000","last_clean":"2024-04-25T00:15:45.041136+0000","last_became_active":"2024-04-25T00:14:11.270478+0000","last_became_peered":"2024-04-25T00:14:11.270478+0000","last_unstale":"2024-04-25T00:15:45.041136+0000","last_undegraded":"2024-04-25T00:15:45.041136+0000","last_fullsized":"2024-04-25T00:15:45.041136+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:41.796155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'13","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.152142+0000","last_change":"2024-04-25T00:13:18.520509+0000","last_active":"2024-04-25T00:15:45.152142+0000","last_peered":"2024-04-25T00:15:45.152142+0000","last_clean":"2024-04-25T00:15:45.152142+0000","last_became_active":"2024-04-25T00:13:18.520139+0000","last_became_peered":"2024-04-25T00:13:18.520139+0000","last_unstale":"2024-04-25T00:15:45.152142+0000","last_undegraded":"2024-04-25T00:15:45.152142+0000","last_fullsized":"2024-04-25T00:15:45.152142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:08.797670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309574+0000","last_change":"2024-04-25T00:13:19.419115+0000","last_active":"2024-04-25T00:15:45.309574+0000","last_peered":"2024-04-25T00:15:45.309574+0000","last_clean":"2024-04-25T00:15:45.309574+0000","last_became_active":"2024-04-25T00:13:19.417865+0000","last_became_peered":"2024-04-25T00:13:19.417865+0000","last_unstale":"2024-04-25T00:15:45.309574+0000","last_undegraded":"2024-04-25T00:15:45.309574+0000","last_fullsized":"2024-04-25T00:15:45.309574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:20:11.173803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041021+0000","last_change":"2024-04-25T00:13:23.585052+0000","last_active":"2024-04-25T00:15:45.041021+0000","last_peered":"2024-04-25T00:15:45.041021+0000","last_clean":"2024-04-25T00:15:45.041021+0000","last_became_active":"2024-04-25T00:13:23.584696+0000","last_became_peered":"2024-04-25T00:13:23.584696+0000","last_unstale":"2024-04-25T00:15:45.041021+0000","last_undegraded":"2024-04-25T00:15:45.041021+0000","last_fullsized":"2024-04-25T00:15:45.041021+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:26.894183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151965+0000","last_change":"2024-04-25T00:13:18.522793+0000","last_active":"2024-04-25T00:15:45.151965+0000","last_peered":"2024-04-25T00:15:45.151965+0000","last_clean":"2024-04-25T00:15:45.151965+0000","last_became_active":"2024-04-25T00:13:18.522453+0000","last_became_peered":"2024-04-25T00:13:18.522453+0000","last_unstale":"2024-04-25T00:15:45.151965+0000","last_undegraded":"2024-04-25T00:15:45.151965+0000","last_fullsized":"2024-04-25T00:15:45.151965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:08.368214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.159151+0000","last_change":"2024-04-25T00:13:19.418573+0000","last_active":"2024-04-25T00:15:45.159151+0000","last_peered":"2024-04-25T00:15:45.159151+0000","last_clean":"2024-04-25T00:15:45.159151+0000","last_became_active":"2024-04-25T00:13:19.418219+0000","last_became_peered":"2024-04-25T00:13:19.418219+0000","last_unstale":"2024-04-25T00:15:45.159151+0000","last_undegraded":"2024-04-25T00:15:45.159151+0000","last_fullsized":"2024-04-25T00:15:45.159151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:08:44.453017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041456+0000","last_change":"2024-04-25T00:13:23.578716+0000","last_active":"2024-04-25T00:15:45.041456+0000","last_peered":"2024-04-25T00:15:45.041456+0000","last_clean":"2024-04-25T00:15:45.041456+0000","last_became_active":"2024-04-25T00:13:23.578325+0000","last_became_peered":"2024-04-25T00:13:23.578325+0000","last_unstale":"2024-04-25T00:15:45.041456+0000","last_undegraded":"2024-04-25T00:15:45.041456+0000","last_fullsized":"2024-04-25T00:15:45.041456+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:39:57.145656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.731145+0000","last_change":"2024-04-25T00:13:18.527369+0000","last_active":"2024-04-25T00:15:45.731145+0000","last_peered":"2024-04-25T00:15:45.731145+0000","last_clean":"2024-04-25T00:15:45.731145+0000","last_became_active":"2024-04-25T00:13:18.526032+0000","last_became_peered":"2024-04-25T00:13:18.526032+0000","last_unstale":"2024-04-25T00:15:45.731145+0000","last_undegraded":"2024-04-25T00:15:45.731145+0000","last_fullsized":"2024-04-25T00:15:45.731145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:52.635184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151572+0000","last_change":"2024-04-25T00:13:19.413305+0000","last_active":"2024-04-25T00:15:45.151572+0000","last_peered":"2024-04-25T00:15:45.151572+0000","last_clean":"2024-04-25T00:15:45.151572+0000","last_became_active":"2024-04-25T00:13:19.412956+0000","last_became_peered":"2024-04-25T00:13:19.412956+0000","last_unstale":"2024-04-25T00:15:45.151572+0000","last_undegraded":"2024-04-25T00:15:45.151572+0000","last_fullsized":"2024-04-25T00:15:45.151572+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:54.436470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041546+0000","last_change":"2024-04-25T00:14:11.273057+0000","last_active":"2024-04-25T00:15:45.041546+0000","last_peered":"2024-04-25T00:15:45.041546+0000","last_clean":"2024-04-25T00:15:45.041546+0000","last_became_active":"2024-04-25T00:14:11.272230+0000","last_became_peered":"2024-04-25T00:14:11.272230+0000","last_unstale":"2024-04-25T00:15:45.041546+0000","last_undegraded":"2024-04-25T00:15:45.041546+0000","last_fullsized":"2024-04-25T00:15:45.041546+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:37.695139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.162194+0000","last_change":"2024-04-25T00:13:18.517507+0000","last_active":"2024-04-25T00:15:45.162194+0000","last_peered":"2024-04-25T00:15:45.162194+0000","last_clean":"2024-04-25T00:15:45.162194+0000","last_became_active":"2024-04-25T00:13:18.517099+0000","last_became_peered":"2024-04-25T00:13:18.517099+0000","last_unstale":"2024-04-25T00:15:45.162194+0000","last_undegraded":"2024-04-25T00:15:45.162194+0000","last_fullsized":"2024-04-25T00:15:45.162194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:42:17.357610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.159046+0000","last_change":"2024-04-25T00:14:11.271691+0000","last_active":"2024-04-25T00:15:45.159046+0000","last_peered":"2024-04-25T00:15:45.159046+0000","last_clean":"2024-04-25T00:15:45.159046+0000","last_became_active":"2024-04-25T00:14:11.271160+0000","last_became_peered":"2024-04-25T00:14:11.271160+0000","last_unstale":"2024-04-25T00:15:45.159046+0000","last_undegraded":"2024-04-25T00:15:45.159046+0000","last_fullsized":"2024-04-25T00:15:45.159046+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:24.416810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.294089+0000","last_change":"2024-04-25T00:13:23.582526+0000","last_active":"2024-04-25T00:15:45.294089+0000","last_peered":"2024-04-25T00:15:45.294089+0000","last_clean":"2024-04-25T00:15:45.294089+0000","last_became_active":"2024-04-25T00:13:23.581221+0000","last_became_peered":"2024-04-25T00:13:23.581221+0000","last_unstale":"2024-04-25T00:15:45.294089+0000","last_undegraded":"2024-04-25T00:15:45.294089+0000","last_fullsized":"2024-04-25T00:15:45.294089+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:46.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314551+0000","last_change":"2024-04-25T00:13:18.522245+0000","last_active":"2024-04-25T00:15:45.314551+0000","last_peered":"2024-04-25T00:15:45.314551+0000","last_clean":"2024-04-25T00:15:45.314551+0000","last_became_active":"2024-04-25T00:13:18.521440+0000","last_became_peered":"2024-04-25T00:13:18.521440+0000","last_unstale":"2024-04-25T00:15:45.314551+0000","last_undegraded":"2024-04-25T00:15:45.314551+0000","last_fullsized":"2024-04-25T00:15:45.314551+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:21:37.847964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040022+0000","last_change":"2024-04-25T00:13:19.422771+0000","last_active":"2024-04-25T00:15:45.040022+0000","last_peered":"2024-04-25T00:15:45.040022+0000","last_clean":"2024-04-25T00:15:45.040022+0000","last_became_active":"2024-04-25T00:13:19.422387+0000","last_became_peered":"2024-04-25T00:13:19.422387+0000","last_unstale":"2024-04-25T00:15:45.040022+0000","last_undegraded":"2024-04-25T00:15:45.040022+0000","last_fullsized":"2024-04-25T00:15:45.040022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:15.647186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115073+0000","last_change":"2024-04-25T00:13:23.586235+0000","last_active":"2024-04-25T00:15:45.115073+0000","last_peered":"2024-04-25T00:15:45.115073+0000","last_clean":"2024-04-25T00:15:45.115073+0000","last_became_active":"2024-04-25T00:13:23.585803+0000","last_became_peered":"2024-04-25T00:13:23.585803+0000","last_unstale":"2024-04-25T00:15:45.115073+0000","last_undegraded":"2024-04-25T00:15:45.115073+0000","last_fullsized":"2024-04-25T00:15:45.115073+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:46.235435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158742+0000","last_change":"2024-04-25T00:13:18.525283+0000","last_active":"2024-04-25T00:15:45.158742+0000","last_peered":"2024-04-25T00:15:45.158742+0000","last_clean":"2024-04-25T00:15:45.158742+0000","last_became_active":"2024-04-25T00:13:18.524939+0000","last_became_peered":"2024-04-25T00:13:18.524939+0000","last_unstale":"2024-04-25T00:15:45.158742+0000","last_undegraded":"2024-04-25T00:15:45.158742+0000","last_fullsized":"2024-04-25T00:15:45.158742+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:09.258498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.148867+0000","last_change":"2024-04-25T00:13:19.415746+0000","last_active":"2024-04-25T00:15:45.148867+0000","last_peered":"2024-04-25T00:15:45.148867+0000","last_clean":"2024-04-25T00:15:45.148867+0000","last_became_active":"2024-04-25T00:13:19.415325+0000","last_became_peered":"2024-04-25T00:13:19.415325+0000","last_unstale":"2024-04-25T00:15:45.148867+0000","last_undegraded":"2024-04-25T00:15:45.148867+0000","last_fullsized":"2024-04-25T00:15:45.148867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:17:48.268845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.039798+0000","last_change":"2024-04-25T00:13:23.583153+0000","last_active":"2024-04-25T00:15:45.039798+0000","last_peered":"2024-04-25T00:15:45.039798+0000","last_clean":"2024-04-25T00:15:45.039798+0000","last_became_active":"2024-04-25T00:13:23.580257+0000","last_became_peered":"2024-04-25T00:13:23.580257+0000","last_unstale":"2024-04-25T00:15:45.039798+0000","last_undegraded":"2024-04-25T00:15:45.039798+0000","last_fullsized":"2024-04-25T00:15:45.039798+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:09:32.990253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'5","reported_seq":279,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.753951+0000","last_change":"2024-04-25T00:13:18.520365+0000","last_active":"2024-04-25T00:15:45.753951+0000","last_peered":"2024-04-25T00:15:45.753951+0000","last_clean":"2024-04-25T00:15:45.753951+0000","last_became_active":"2024-04-25T00:13:18.519918+0000","last_became_peered":"2024-04-25T00:13:18.519918+0000","last_unstale":"2024-04-25T00:15:45.753951+0000","last_undegraded":"2024-04-25T00:15:45.753951+0000","last_fullsized":"2024-04-25T00:15:45.753951+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:46.415197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":240,"num_read_kb":240,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309728+0000","last_change":"2024-04-25T00:13:19.414341+0000","last_active":"2024-04-25T00:15:45.309728+0000","last_peered":"2024-04-25T00:15:45.309728+0000","last_clean":"2024-04-25T00:15:45.309728+0000","last_became_active":"2024-04-25T00:13:19.413793+0000","last_became_peered":"2024-04-25T00:13:19.413793+0000","last_unstale":"2024-04-25T00:15:45.309728+0000","last_undegraded":"2024-04-25T00:15:45.309728+0000","last_fullsized":"2024-04-25T00:15:45.309728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:55.927082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.148616+0000","last_change":"2024-04-25T00:13:23.590631+0000","last_active":"2024-04-25T00:15:45.148616+0000","last_peered":"2024-04-25T00:15:45.148616+0000","last_clean":"2024-04-25T00:15:45.148616+0000","last_became_active":"2024-04-25T00:13:23.589980+0000","last_became_peered":"2024-04-25T00:13:23.589980+0000","last_unstale":"2024-04-25T00:15:45.148616+0000","last_undegraded":"2024-04-25T00:15:45.148616+0000","last_fullsized":"2024-04-25T00:15:45.148616+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:18.234048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.313143+0000","last_change":"2024-04-25T00:13:18.526353+0000","last_active":"2024-04-25T00:15:45.313143+0000","last_peered":"2024-04-25T00:15:45.313143+0000","last_clean":"2024-04-25T00:15:45.313143+0000","last_became_active":"2024-04-25T00:13:18.525951+0000","last_became_peered":"2024-04-25T00:13:18.525951+0000","last_unstale":"2024-04-25T00:15:45.313143+0000","last_undegraded":"2024-04-25T00:15:45.313143+0000","last_fullsized":"2024-04-25T00:15:45.313143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:10:29.597797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115219+0000","last_change":"2024-04-25T00:13:19.425486+0000","last_active":"2024-04-25T00:15:45.115219+0000","last_peered":"2024-04-25T00:15:45.115219+0000","last_clean":"2024-04-25T00:15:45.115219+0000","last_became_active":"2024-04-25T00:13:19.425087+0000","last_became_peered":"2024-04-25T00:13:19.425087+0000","last_unstale":"2024-04-25T00:15:45.115219+0000","last_undegraded":"2024-04-25T00:15:45.115219+0000","last_fullsized":"2024-04-25T00:15:45.115219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:00.216683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041314+0000","last_change":"2024-04-25T00:13:23.579747+0000","last_active":"2024-04-25T00:15:45.041314+0000","last_peered":"2024-04-25T00:15:45.041314+0000","last_clean":"2024-04-25T00:15:45.041314+0000","last_became_active":"2024-04-25T00:13:23.579346+0000","last_became_peered":"2024-04-25T00:13:23.579346+0000","last_unstale":"2024-04-25T00:15:45.041314+0000","last_undegraded":"2024-04-25T00:15:45.041314+0000","last_fullsized":"2024-04-25T00:15:45.041314+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:10:18.969942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.159127+0000","last_change":"2024-04-25T00:13:18.524707+0000","last_active":"2024-04-25T00:15:45.159127+0000","last_peered":"2024-04-25T00:15:45.159127+0000","last_clean":"2024-04-25T00:15:45.159127+0000","last_became_active":"2024-04-25T00:13:18.524352+0000","last_became_peered":"2024-04-25T00:13:18.524352+0000","last_unstale":"2024-04-25T00:15:45.159127+0000","last_undegraded":"2024-04-25T00:15:45.159127+0000","last_fullsized":"2024-04-25T00:15:45.159127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:29.992843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309736+0000","last_change":"2024-04-25T00:13:19.421845+0000","last_active":"2024-04-25T00:15:45.309736+0000","last_peered":"2024-04-25T00:15:45.309736+0000","last_clean":"2024-04-25T00:15:45.309736+0000","last_became_active":"2024-04-25T00:13:19.421458+0000","last_became_peered":"2024-04-25T00:13:19.421458+0000","last_unstale":"2024-04-25T00:15:45.309736+0000","last_undegraded":"2024-04-25T00:15:45.309736+0000","last_fullsized":"2024-04-25T00:15:45.309736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:09.286604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.148941+0000","last_change":"2024-04-25T00:13:23.583721+0000","last_active":"2024-04-25T00:15:45.148941+0000","last_peered":"2024-04-25T00:15:45.148941+0000","last_clean":"2024-04-25T00:15:45.148941+0000","last_became_active":"2024-04-25T00:13:23.583340+0000","last_became_peered":"2024-04-25T00:13:23.583340+0000","last_unstale":"2024-04-25T00:15:45.148941+0000","last_undegraded":"2024-04-25T00:15:45.148941+0000","last_fullsized":"2024-04-25T00:15:45.148941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:26.446468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":326,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.731678+0000","last_change":"2024-04-25T00:13:18.520653+0000","last_active":"2024-04-25T00:15:45.731678+0000","last_peered":"2024-04-25T00:15:45.731678+0000","last_clean":"2024-04-25T00:15:45.731678+0000","last_became_active":"2024-04-25T00:13:18.520278+0000","last_became_peered":"2024-04-25T00:13:18.520278+0000","last_unstale":"2024-04-25T00:15:45.731678+0000","last_undegraded":"2024-04-25T00:15:45.731678+0000","last_fullsized":"2024-04-25T00:15:45.731678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:58.311179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66804,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":252,"num_read_kb":191,"num_write":42,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.152025+0000","last_change":"2024-04-25T00:13:19.421599+0000","last_active":"2024-04-25T00:15:45.152025+0000","last_peered":"2024-04-25T00:15:45.152025+0000","last_clean":"2024-04-25T00:15:45.152025+0000","last_became_active":"2024-04-25T00:13:19.421107+0000","last_became_peered":"2024-04-25T00:13:19.421107+0000","last_unstale":"2024-04-25T00:15:45.152025+0000","last_undegraded":"2024-04-25T00:15:45.152025+0000","last_fullsized":"2024-04-25T00:15:45.152025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:59.577987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.159398+0000","last_change":"2024-04-25T00:13:23.580538+0000","last_active":"2024-04-25T00:15:45.159398+0000","last_peered":"2024-04-25T00:15:45.159398+0000","last_clean":"2024-04-25T00:15:45.159398+0000","last_became_active":"2024-04-25T00:13:23.580095+0000","last_became_peered":"2024-04-25T00:13:23.580095+0000","last_unstale":"2024-04-25T00:15:45.159398+0000","last_undegraded":"2024-04-25T00:15:45.159398+0000","last_fullsized":"2024-04-25T00:15:45.159398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:04.189530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041608+0000","last_change":"2024-04-25T00:13:18.522501+0000","last_active":"2024-04-25T00:15:45.041608+0000","last_peered":"2024-04-25T00:15:45.041608+0000","last_clean":"2024-04-25T00:15:45.041608+0000","last_became_active":"2024-04-25T00:13:18.522160+0000","last_became_peered":"2024-04-25T00:13:18.522160+0000","last_unstale":"2024-04-25T00:15:45.041608+0000","last_undegraded":"2024-04-25T00:15:45.041608+0000","last_fullsized":"2024-04-25T00:15:45.041608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:31.876731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314208+0000","last_change":"2024-04-25T00:13:19.417957+0000","last_active":"2024-04-25T00:15:45.314208+0000","last_peered":"2024-04-25T00:15:45.314208+0000","last_clean":"2024-04-25T00:15:45.314208+0000","last_became_active":"2024-04-25T00:13:19.417600+0000","last_became_peered":"2024-04-25T00:13:19.417600+0000","last_unstale":"2024-04-25T00:15:45.314208+0000","last_undegraded":"2024-04-25T00:15:45.314208+0000","last_fullsized":"2024-04-25T00:15:45.314208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:24.600965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293826+0000","last_change":"2024-04-25T00:13:23.582279+0000","last_active":"2024-04-25T00:15:45.293826+0000","last_peered":"2024-04-25T00:15:45.293826+0000","last_clean":"2024-04-25T00:15:45.293826+0000","last_became_active":"2024-04-25T00:13:23.581766+0000","last_became_peered":"2024-04-25T00:13:23.581766+0000","last_unstale":"2024-04-25T00:15:45.293826+0000","last_undegraded":"2024-04-25T00:15:45.293826+0000","last_fullsized":"2024-04-25T00:15:45.293826+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:36.115870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158999+0000","last_change":"2024-04-25T00:13:18.527443+0000","last_active":"2024-04-25T00:15:45.158999+0000","last_peered":"2024-04-25T00:15:45.158999+0000","last_clean":"2024-04-25T00:15:45.158999+0000","last_became_active":"2024-04-25T00:13:18.526919+0000","last_became_peered":"2024-04-25T00:13:18.526919+0000","last_unstale":"2024-04-25T00:15:45.158999+0000","last_undegraded":"2024-04-25T00:15:45.158999+0000","last_fullsized":"2024-04-25T00:15:45.158999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:32.737580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312419+0000","last_change":"2024-04-25T00:13:19.425947+0000","last_active":"2024-04-25T00:15:45.312419+0000","last_peered":"2024-04-25T00:15:45.312419+0000","last_clean":"2024-04-25T00:15:45.312419+0000","last_became_active":"2024-04-25T00:13:19.425613+0000","last_became_peered":"2024-04-25T00:13:19.425613+0000","last_unstale":"2024-04-25T00:15:45.312419+0000","last_undegraded":"2024-04-25T00:15:45.312419+0000","last_fullsized":"2024-04-25T00:15:45.312419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:28:34.729375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149578+0000","last_change":"2024-04-25T00:13:23.587794+0000","last_active":"2024-04-25T00:15:45.149578+0000","last_peered":"2024-04-25T00:15:45.149578+0000","last_clean":"2024-04-25T00:15:45.149578+0000","last_became_active":"2024-04-25T00:13:23.587334+0000","last_became_peered":"2024-04-25T00:13:23.587334+0000","last_unstale":"2024-04-25T00:15:45.149578+0000","last_undegraded":"2024-04-25T00:15:45.149578+0000","last_fullsized":"2024-04-25T00:15:45.149578+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:58.460439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":166,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.731310+0000","last_change":"2024-04-25T00:13:18.521701+0000","last_active":"2024-04-25T00:15:45.731310+0000","last_peered":"2024-04-25T00:15:45.731310+0000","last_clean":"2024-04-25T00:15:45.731310+0000","last_became_active":"2024-04-25T00:13:18.521360+0000","last_became_peered":"2024-04-25T00:13:18.521360+0000","last_unstale":"2024-04-25T00:15:45.731310+0000","last_undegraded":"2024-04-25T00:15:45.731310+0000","last_fullsized":"2024-04-25T00:15:45.731310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:18:11.567426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":126,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.311990+0000","last_change":"2024-04-25T00:13:19.424162+0000","last_active":"2024-04-25T00:15:45.311990+0000","last_peered":"2024-04-25T00:15:45.311990+0000","last_clean":"2024-04-25T00:15:45.311990+0000","last_became_active":"2024-04-25T00:13:19.423845+0000","last_became_peered":"2024-04-25T00:13:19.423845+0000","last_unstale":"2024-04-25T00:15:45.311990+0000","last_undegraded":"2024-04-25T00:15:45.311990+0000","last_fullsized":"2024-04-25T00:15:45.311990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:33.887577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314672+0000","last_change":"2024-04-25T00:13:23.582102+0000","last_active":"2024-04-25T00:15:45.314672+0000","last_peered":"2024-04-25T00:15:45.314672+0000","last_clean":"2024-04-25T00:15:45.314672+0000","last_became_active":"2024-04-25T00:13:23.581679+0000","last_became_peered":"2024-04-25T00:13:23.581679+0000","last_unstale":"2024-04-25T00:15:45.314672+0000","last_undegraded":"2024-04-25T00:15:45.314672+0000","last_fullsized":"2024-04-25T00:15:45.314672+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:05:39.518634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151589+0000","last_change":"2024-04-25T00:13:18.518285+0000","last_active":"2024-04-25T00:15:45.151589+0000","last_peered":"2024-04-25T00:15:45.151589+0000","last_clean":"2024-04-25T00:15:45.151589+0000","last_became_active":"2024-04-25T00:13:18.517826+0000","last_became_peered":"2024-04-25T00:13:18.517826+0000","last_unstale":"2024-04-25T00:15:45.151589+0000","last_undegraded":"2024-04-25T00:15:45.151589+0000","last_fullsized":"2024-04-25T00:15:45.151589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:12.901225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158812+0000","last_change":"2024-04-25T00:14:11.268444+0000","last_active":"2024-04-25T00:15:45.158812+0000","last_peered":"2024-04-25T00:15:45.158812+0000","last_clean":"2024-04-25T00:15:45.158812+0000","last_became_active":"2024-04-25T00:14:11.267934+0000","last_became_peered":"2024-04-25T00:14:11.267934+0000","last_unstale":"2024-04-25T00:15:45.158812+0000","last_undegraded":"2024-04-25T00:15:45.158812+0000","last_fullsized":"2024-04-25T00:15:45.158812+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:12:16.209890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158730+0000","last_change":"2024-04-25T00:13:23.578445+0000","last_active":"2024-04-25T00:15:45.158730+0000","last_peered":"2024-04-25T00:15:45.158730+0000","last_clean":"2024-04-25T00:15:45.158730+0000","last_became_active":"2024-04-25T00:13:23.578048+0000","last_became_peered":"2024-04-25T00:13:23.578048+0000","last_unstale":"2024-04-25T00:15:45.158730+0000","last_undegraded":"2024-04-25T00:15:45.158730+0000","last_fullsized":"2024-04-25T00:15:45.158730+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:55.072460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151953+0000","last_change":"2024-04-25T00:13:18.515939+0000","last_active":"2024-04-25T00:15:45.151953+0000","last_peered":"2024-04-25T00:15:45.151953+0000","last_clean":"2024-04-25T00:15:45.151953+0000","last_became_active":"2024-04-25T00:13:18.515597+0000","last_became_peered":"2024-04-25T00:13:18.515597+0000","last_unstale":"2024-04-25T00:15:45.151953+0000","last_undegraded":"2024-04-25T00:15:45.151953+0000","last_fullsized":"2024-04-25T00:15:45.151953+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:03:30.266404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.294003+0000","last_change":"2024-04-25T00:14:11.269143+0000","last_active":"2024-04-25T00:15:45.294003+0000","last_peered":"2024-04-25T00:15:45.294003+0000","last_clean":"2024-04-25T00:15:45.294003+0000","last_became_active":"2024-04-25T00:14:11.268736+0000","last_became_peered":"2024-04-25T00:14:11.268736+0000","last_unstale":"2024-04-25T00:15:45.294003+0000","last_undegraded":"2024-04-25T00:15:45.294003+0000","last_fullsized":"2024-04-25T00:15:45.294003+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:19.559974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151733+0000","last_change":"2024-04-25T00:13:23.862576+0000","last_active":"2024-04-25T00:15:45.151733+0000","last_peered":"2024-04-25T00:15:45.151733+0000","last_clean":"2024-04-25T00:15:45.151733+0000","last_became_active":"2024-04-25T00:13:23.862024+0000","last_became_peered":"2024-04-25T00:13:23.862024+0000","last_unstale":"2024-04-25T00:15:45.151733+0000","last_undegraded":"2024-04-25T00:15:45.151733+0000","last_fullsized":"2024-04-25T00:15:45.151733+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:17.249680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149273+0000","last_change":"2024-04-25T00:13:18.522503+0000","last_active":"2024-04-25T00:15:45.149273+0000","last_peered":"2024-04-25T00:15:45.149273+0000","last_clean":"2024-04-25T00:15:45.149273+0000","last_became_active":"2024-04-25T00:13:18.522126+0000","last_became_peered":"2024-04-25T00:13:18.522126+0000","last_unstale":"2024-04-25T00:15:45.149273+0000","last_undegraded":"2024-04-25T00:15:45.149273+0000","last_fullsized":"2024-04-25T00:15:45.149273+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:47:57.530098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.162498+0000","last_change":"2024-04-25T00:13:19.416862+0000","last_active":"2024-04-25T00:15:45.162498+0000","last_peered":"2024-04-25T00:15:45.162498+0000","last_clean":"2024-04-25T00:15:45.162498+0000","last_became_active":"2024-04-25T00:13:19.416375+0000","last_became_peered":"2024-04-25T00:13:19.416375+0000","last_unstale":"2024-04-25T00:15:45.162498+0000","last_undegraded":"2024-04-25T00:15:45.162498+0000","last_fullsized":"2024-04-25T00:15:45.162498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:43:32.031623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314336+0000","last_change":"2024-04-25T00:13:23.582387+0000","last_active":"2024-04-25T00:15:45.314336+0000","last_peered":"2024-04-25T00:15:45.314336+0000","last_clean":"2024-04-25T00:15:45.314336+0000","last_became_active":"2024-04-25T00:13:23.581984+0000","last_became_peered":"2024-04-25T00:13:23.581984+0000","last_unstale":"2024-04-25T00:15:45.314336+0000","last_undegraded":"2024-04-25T00:15:45.314336+0000","last_fullsized":"2024-04-25T00:15:45.314336+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:07:16.388293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"94'7","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.748717+0000","last_change":"2024-04-25T00:13:18.524121+0000","last_active":"2024-04-25T00:15:45.748717+0000","last_peered":"2024-04-25T00:15:45.748717+0000","last_clean":"2024-04-25T00:15:45.748717+0000","last_became_active":"2024-04-25T00:13:18.523731+0000","last_became_peered":"2024-04-25T00:13:18.523731+0000","last_unstale":"2024-04-25T00:15:45.748717+0000","last_undegraded":"2024-04-25T00:15:45.748717+0000","last_fullsized":"2024-04-25T00:15:45.748717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:22.105952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1287,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.162127+0000","last_change":"2024-04-25T00:13:19.424753+0000","last_active":"2024-04-25T00:15:45.162127+0000","last_peered":"2024-04-25T00:15:45.162127+0000","last_clean":"2024-04-25T00:15:45.162127+0000","last_became_active":"2024-04-25T00:13:19.423214+0000","last_became_peered":"2024-04-25T00:13:19.423214+0000","last_unstale":"2024-04-25T00:15:45.162127+0000","last_undegraded":"2024-04-25T00:15:45.162127+0000","last_fullsized":"2024-04-25T00:15:45.162127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:41.627442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314459+0000","last_change":"2024-04-25T00:14:11.272431+0000","last_active":"2024-04-25T00:15:45.314459+0000","last_peered":"2024-04-25T00:15:45.314459+0000","last_clean":"2024-04-25T00:15:45.314459+0000","last_became_active":"2024-04-25T00:14:11.271997+0000","last_became_peered":"2024-04-25T00:14:11.271997+0000","last_unstale":"2024-04-25T00:15:45.314459+0000","last_undegraded":"2024-04-25T00:15:45.314459+0000","last_fullsized":"2024-04-25T00:15:45.314459+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:18.768009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158863+0000","last_change":"2024-04-25T00:13:19.420617+0000","last_active":"2024-04-25T00:15:45.158863+0000","last_peered":"2024-04-25T00:15:45.158863+0000","last_clean":"2024-04-25T00:15:45.158863+0000","last_became_active":"2024-04-25T00:13:19.420274+0000","last_became_peered":"2024-04-25T00:13:19.420274+0000","last_unstale":"2024-04-25T00:15:45.158863+0000","last_undegraded":"2024-04-25T00:15:45.158863+0000","last_fullsized":"2024-04-25T00:15:45.158863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:07.515196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314314+0000","last_change":"2024-04-25T00:13:18.529753+0000","last_active":"2024-04-25T00:15:45.314314+0000","last_peered":"2024-04-25T00:15:45.314314+0000","last_clean":"2024-04-25T00:15:45.314314+0000","last_became_active":"2024-04-25T00:13:18.525521+0000","last_became_peered":"2024-04-25T00:13:18.525521+0000","last_unstale":"2024-04-25T00:15:45.314314+0000","last_undegraded":"2024-04-25T00:15:45.314314+0000","last_fullsized":"2024-04-25T00:15:45.314314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:17.044367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312649+0000","last_change":"2024-04-25T00:13:23.589613+0000","last_active":"2024-04-25T00:15:45.312649+0000","last_peered":"2024-04-25T00:15:45.312649+0000","last_clean":"2024-04-25T00:15:45.312649+0000","last_became_active":"2024-04-25T00:13:23.589323+0000","last_became_peered":"2024-04-25T00:13:23.589323+0000","last_unstale":"2024-04-25T00:15:45.312649+0000","last_undegraded":"2024-04-25T00:15:45.312649+0000","last_fullsized":"2024-04-25T00:15:45.312649+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:34:42.272706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149072+0000","last_change":"2024-04-25T00:15:11.431091+0000","last_active":"2024-04-25T00:15:45.149072+0000","last_peered":"2024-04-25T00:15:45.149072+0000","last_clean":"2024-04-25T00:15:45.149072+0000","last_became_active":"2024-04-25T00:15:11.430797+0000","last_became_peered":"2024-04-25T00:15:11.430797+0000","last_unstale":"2024-04-25T00:15:45.149072+0000","last_undegraded":"2024-04-25T00:15:45.149072+0000","last_fullsized":"2024-04-25T00:15:45.149072+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:26:30.121912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041204+0000","last_change":"2024-04-25T00:13:18.514560+0000","last_active":"2024-04-25T00:15:45.041204+0000","last_peered":"2024-04-25T00:15:45.041204+0000","last_clean":"2024-04-25T00:15:45.041204+0000","last_became_active":"2024-04-25T00:13:18.514034+0000","last_became_peered":"2024-04-25T00:13:18.514034+0000","last_unstale":"2024-04-25T00:15:45.041204+0000","last_undegraded":"2024-04-25T00:15:45.041204+0000","last_fullsized":"2024-04-25T00:15:45.041204+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:35:18.110206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":365,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.311817+0000","last_change":"2024-04-25T00:12:01.540807+0000","last_active":"2024-04-25T00:15:45.311817+0000","last_peered":"2024-04-25T00:15:45.311817+0000","last_clean":"2024-04-25T00:15:45.311817+0000","last_became_active":"2024-04-25T00:12:01.540272+0000","last_became_peered":"2024-04-25T00:12:01.540272+0000","last_unstale":"2024-04-25T00:15:45.311817+0000","last_undegraded":"2024-04-25T00:15:45.311817+0000","last_fullsized":"2024-04-25T00:15:45.311817+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:08:34.007429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:08:34.007429+0000","last_clean_scrub_stamp":"2024-04-25T00:08:34.007429+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:50:14.642604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.159144+0000","last_change":"2024-04-25T00:13:23.862493+0000","last_active":"2024-04-25T00:15:45.159144+0000","last_peered":"2024-04-25T00:15:45.159144+0000","last_clean":"2024-04-25T00:15:45.159144+0000","last_became_active":"2024-04-25T00:13:23.861972+0000","last_became_peered":"2024-04-25T00:13:23.861972+0000","last_unstale":"2024-04-25T00:15:45.159144+0000","last_undegraded":"2024-04-25T00:15:45.159144+0000","last_fullsized":"2024-04-25T00:15:45.159144+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:49:13.507357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314067+0000","last_change":"2024-04-25T00:13:19.421920+0000","last_active":"2024-04-25T00:15:45.314067+0000","last_peered":"2024-04-25T00:15:45.314067+0000","last_clean":"2024-04-25T00:15:45.314067+0000","last_became_active":"2024-04-25T00:13:19.421585+0000","last_became_peered":"2024-04-25T00:13:19.421585+0000","last_unstale":"2024-04-25T00:15:45.314067+0000","last_undegraded":"2024-04-25T00:15:45.314067+0000","last_fullsized":"2024-04-25T00:15:45.314067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:53:24.937520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.311886+0000","last_change":"2024-04-25T00:13:18.527062+0000","last_active":"2024-04-25T00:15:45.311886+0000","last_peered":"2024-04-25T00:15:45.311886+0000","last_clean":"2024-04-25T00:15:45.311886+0000","last_became_active":"2024-04-25T00:13:18.526338+0000","last_became_peered":"2024-04-25T00:13:18.526338+0000","last_unstale":"2024-04-25T00:15:45.311886+0000","last_undegraded":"2024-04-25T00:15:45.311886+0000","last_fullsized":"2024-04-25T00:15:45.311886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:21.756728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293792+0000","last_change":"2024-04-25T00:13:23.575162+0000","last_active":"2024-04-25T00:15:45.293792+0000","last_peered":"2024-04-25T00:15:45.293792+0000","last_clean":"2024-04-25T00:15:45.293792+0000","last_became_active":"2024-04-25T00:13:23.574722+0000","last_became_peered":"2024-04-25T00:13:23.574722+0000","last_unstale":"2024-04-25T00:15:45.293792+0000","last_undegraded":"2024-04-25T00:15:45.293792+0000","last_fullsized":"2024-04-25T00:15:45.293792+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:52.958539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.152034+0000","last_change":"2024-04-25T00:13:18.523019+0000","last_active":"2024-04-25T00:15:45.152034+0000","last_peered":"2024-04-25T00:15:45.152034+0000","last_clean":"2024-04-25T00:15:45.152034+0000","last_became_active":"2024-04-25T00:13:18.522677+0000","last_became_peered":"2024-04-25T00:13:18.522677+0000","last_unstale":"2024-04-25T00:15:45.152034+0000","last_undegraded":"2024-04-25T00:15:45.152034+0000","last_fullsized":"2024-04-25T00:15:45.152034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:14.470154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314556+0000","last_change":"2024-04-25T00:13:19.417672+0000","last_active":"2024-04-25T00:15:45.314556+0000","last_peered":"2024-04-25T00:15:45.314556+0000","last_clean":"2024-04-25T00:15:45.314556+0000","last_became_active":"2024-04-25T00:13:19.417323+0000","last_became_peered":"2024-04-25T00:13:19.417323+0000","last_unstale":"2024-04-25T00:15:45.314556+0000","last_undegraded":"2024-04-25T00:15:45.314556+0000","last_fullsized":"2024-04-25T00:15:45.314556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:53.734784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.039721+0000","last_change":"2024-04-25T00:13:23.862320+0000","last_active":"2024-04-25T00:15:45.039721+0000","last_peered":"2024-04-25T00:15:45.039721+0000","last_clean":"2024-04-25T00:15:45.039721+0000","last_became_active":"2024-04-25T00:13:23.861895+0000","last_became_peered":"2024-04-25T00:13:23.861895+0000","last_unstale":"2024-04-25T00:15:45.039721+0000","last_undegraded":"2024-04-25T00:15:45.039721+0000","last_fullsized":"2024-04-25T00:15:45.039721+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:16:21.324727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'49","reported_seq":106,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.267428+0000","last_change":"2024-04-25T00:13:18.520989+0000","last_active":"2024-04-25T00:15:45.267428+0000","last_peered":"2024-04-25T00:15:45.267428+0000","last_clean":"2024-04-25T00:15:45.267428+0000","last_became_active":"2024-04-25T00:13:18.520566+0000","last_became_peered":"2024-04-25T00:13:18.520566+0000","last_unstale":"2024-04-25T00:15:45.267428+0000","last_undegraded":"2024-04-25T00:15:45.267428+0000","last_fullsized":"2024-04-25T00:15:45.267428+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:22:32.532352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":86,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040218+0000","last_change":"2024-04-25T00:13:19.422741+0000","last_active":"2024-04-25T00:15:45.040218+0000","last_peered":"2024-04-25T00:15:45.040218+0000","last_clean":"2024-04-25T00:15:45.040218+0000","last_became_active":"2024-04-25T00:13:19.422379+0000","last_became_peered":"2024-04-25T00:13:19.422379+0000","last_unstale":"2024-04-25T00:15:45.040218+0000","last_undegraded":"2024-04-25T00:15:45.040218+0000","last_fullsized":"2024-04-25T00:15:45.040218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:41.337641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041168+0000","last_change":"2024-04-25T00:13:23.577888+0000","last_active":"2024-04-25T00:15:45.041168+0000","last_peered":"2024-04-25T00:15:45.041168+0000","last_clean":"2024-04-25T00:15:45.041168+0000","last_became_active":"2024-04-25T00:13:23.577440+0000","last_became_peered":"2024-04-25T00:13:23.577440+0000","last_unstale":"2024-04-25T00:15:45.041168+0000","last_undegraded":"2024-04-25T00:15:45.041168+0000","last_fullsized":"2024-04-25T00:15:45.041168+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:03.760277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041651+0000","last_change":"2024-04-25T00:13:18.527996+0000","last_active":"2024-04-25T00:15:45.041651+0000","last_peered":"2024-04-25T00:15:45.041651+0000","last_clean":"2024-04-25T00:15:45.041651+0000","last_became_active":"2024-04-25T00:13:18.527636+0000","last_became_peered":"2024-04-25T00:13:18.527636+0000","last_unstale":"2024-04-25T00:15:45.041651+0000","last_undegraded":"2024-04-25T00:15:45.041651+0000","last_fullsized":"2024-04-25T00:15:45.041651+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:52.137108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115077+0000","last_change":"2024-04-25T00:13:19.425538+0000","last_active":"2024-04-25T00:15:45.115077+0000","last_peered":"2024-04-25T00:15:45.115077+0000","last_clean":"2024-04-25T00:15:45.115077+0000","last_became_active":"2024-04-25T00:13:19.420790+0000","last_became_peered":"2024-04-25T00:13:19.420790+0000","last_unstale":"2024-04-25T00:15:45.115077+0000","last_undegraded":"2024-04-25T00:15:45.115077+0000","last_fullsized":"2024-04-25T00:15:45.115077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:04.319759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041457+0000","last_change":"2024-04-25T00:13:23.863187+0000","last_active":"2024-04-25T00:15:45.041457+0000","last_peered":"2024-04-25T00:15:45.041457+0000","last_clean":"2024-04-25T00:15:45.041457+0000","last_became_active":"2024-04-25T00:13:23.862206+0000","last_became_peered":"2024-04-25T00:13:23.862206+0000","last_unstale":"2024-04-25T00:15:45.041457+0000","last_undegraded":"2024-04-25T00:15:45.041457+0000","last_fullsized":"2024-04-25T00:15:45.041457+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:57:17.395007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309338+0000","last_change":"2024-04-25T00:13:18.529224+0000","last_active":"2024-04-25T00:15:45.309338+0000","last_peered":"2024-04-25T00:15:45.309338+0000","last_clean":"2024-04-25T00:15:45.309338+0000","last_became_active":"2024-04-25T00:13:18.528908+0000","last_became_peered":"2024-04-25T00:13:18.528908+0000","last_unstale":"2024-04-25T00:15:45.309338+0000","last_undegraded":"2024-04-25T00:15:45.309338+0000","last_fullsized":"2024-04-25T00:15:45.309338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:27.035726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151539+0000","last_change":"2024-04-25T00:13:19.422223+0000","last_active":"2024-04-25T00:15:45.151539+0000","last_peered":"2024-04-25T00:15:45.151539+0000","last_clean":"2024-04-25T00:15:45.151539+0000","last_became_active":"2024-04-25T00:13:19.421524+0000","last_became_peered":"2024-04-25T00:13:19.421524+0000","last_unstale":"2024-04-25T00:15:45.151539+0000","last_undegraded":"2024-04-25T00:15:45.151539+0000","last_fullsized":"2024-04-25T00:15:45.151539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:17:01.849503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149842+0000","last_change":"2024-04-25T00:13:23.583015+0000","last_active":"2024-04-25T00:15:45.149842+0000","last_peered":"2024-04-25T00:15:45.149842+0000","last_clean":"2024-04-25T00:15:45.149842+0000","last_became_active":"2024-04-25T00:13:23.582612+0000","last_became_peered":"2024-04-25T00:13:23.582612+0000","last_unstale":"2024-04-25T00:15:45.149842+0000","last_undegraded":"2024-04-25T00:15:45.149842+0000","last_fullsized":"2024-04-25T00:15:45.149842+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:36.872281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.039860+0000","last_change":"2024-04-25T00:13:18.530027+0000","last_active":"2024-04-25T00:15:45.039860+0000","last_peered":"2024-04-25T00:15:45.039860+0000","last_clean":"2024-04-25T00:15:45.039860+0000","last_became_active":"2024-04-25T00:13:18.529717+0000","last_became_peered":"2024-04-25T00:13:18.529717+0000","last_unstale":"2024-04-25T00:15:45.039860+0000","last_undegraded":"2024-04-25T00:15:45.039860+0000","last_fullsized":"2024-04-25T00:15:45.039860+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:23:31.020637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293701+0000","last_change":"2024-04-25T00:13:19.428650+0000","last_active":"2024-04-25T00:15:45.293701+0000","last_peered":"2024-04-25T00:15:45.293701+0000","last_clean":"2024-04-25T00:15:45.293701+0000","last_became_active":"2024-04-25T00:13:19.428156+0000","last_became_peered":"2024-04-25T00:13:19.428156+0000","last_unstale":"2024-04-25T00:15:45.293701+0000","last_undegraded":"2024-04-25T00:15:45.293701+0000","last_fullsized":"2024-04-25T00:15:45.293701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:43.094529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041458+0000","last_change":"2024-04-25T00:13:23.579125+0000","last_active":"2024-04-25T00:15:45.041458+0000","last_peered":"2024-04-25T00:15:45.041458+0000","last_clean":"2024-04-25T00:15:45.041458+0000","last_became_active":"2024-04-25T00:13:23.578728+0000","last_became_peered":"2024-04-25T00:13:23.578728+0000","last_unstale":"2024-04-25T00:15:45.041458+0000","last_undegraded":"2024-04-25T00:15:45.041458+0000","last_fullsized":"2024-04-25T00:15:45.041458+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:16:16.834399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'4","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293944+0000","last_change":"2024-04-25T00:14:11.278099+0000","last_active":"2024-04-25T00:15:45.293944+0000","last_peered":"2024-04-25T00:15:45.293944+0000","last_clean":"2024-04-25T00:15:45.293944+0000","last_became_active":"2024-04-25T00:14:11.269977+0000","last_became_peered":"2024-04-25T00:14:11.269977+0000","last_unstale":"2024-04-25T00:15:45.293944+0000","last_undegraded":"2024-04-25T00:15:45.293944+0000","last_fullsized":"2024-04-25T00:15:45.293944+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:48:59.261854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040032+0000","last_change":"2024-04-25T00:13:19.414848+0000","last_active":"2024-04-25T00:15:45.040032+0000","last_peered":"2024-04-25T00:15:45.040032+0000","last_clean":"2024-04-25T00:15:45.040032+0000","last_became_active":"2024-04-25T00:13:19.414370+0000","last_became_peered":"2024-04-25T00:13:19.414370+0000","last_unstale":"2024-04-25T00:15:45.040032+0000","last_undegraded":"2024-04-25T00:15:45.040032+0000","last_fullsized":"2024-04-25T00:15:45.040032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:41:08.779171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041407+0000","last_change":"2024-04-25T00:13:23.862577+0000","last_active":"2024-04-25T00:15:45.041407+0000","last_peered":"2024-04-25T00:15:45.041407+0000","last_clean":"2024-04-25T00:15:45.041407+0000","last_became_active":"2024-04-25T00:13:23.862098+0000","last_became_peered":"2024-04-25T00:13:23.862098+0000","last_unstale":"2024-04-25T00:15:45.041407+0000","last_undegraded":"2024-04-25T00:15:45.041407+0000","last_fullsized":"2024-04-25T00:15:45.041407+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:08:00.344041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293746+0000","last_change":"2024-04-25T00:14:11.274790+0000","last_active":"2024-04-25T00:15:45.293746+0000","last_peered":"2024-04-25T00:15:45.293746+0000","last_clean":"2024-04-25T00:15:45.293746+0000","last_became_active":"2024-04-25T00:14:11.267132+0000","last_became_peered":"2024-04-25T00:14:11.267132+0000","last_unstale":"2024-04-25T00:15:45.293746+0000","last_undegraded":"2024-04-25T00:15:45.293746+0000","last_fullsized":"2024-04-25T00:15:45.293746+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:58.302788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312161+0000","last_change":"2024-04-25T00:13:19.427992+0000","last_active":"2024-04-25T00:15:45.312161+0000","last_peered":"2024-04-25T00:15:45.312161+0000","last_clean":"2024-04-25T00:15:45.312161+0000","last_became_active":"2024-04-25T00:13:19.427665+0000","last_became_peered":"2024-04-25T00:13:19.427665+0000","last_unstale":"2024-04-25T00:15:45.312161+0000","last_undegraded":"2024-04-25T00:15:45.312161+0000","last_fullsized":"2024-04-25T00:15:45.312161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:12:31.582185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309231+0000","last_change":"2024-04-25T00:13:23.582430+0000","last_active":"2024-04-25T00:15:45.309231+0000","last_peered":"2024-04-25T00:15:45.309231+0000","last_clean":"2024-04-25T00:15:45.309231+0000","last_became_active":"2024-04-25T00:13:23.582008+0000","last_became_peered":"2024-04-25T00:13:23.582008+0000","last_unstale":"2024-04-25T00:15:45.309231+0000","last_undegraded":"2024-04-25T00:15:45.309231+0000","last_fullsized":"2024-04-25T00:15:45.309231+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:11.068720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"94'7","reported_seq":163,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.754413+0000","last_change":"2024-04-25T00:13:18.528030+0000","last_active":"2024-04-25T00:15:45.754413+0000","last_peered":"2024-04-25T00:15:45.754413+0000","last_clean":"2024-04-25T00:15:45.754413+0000","last_became_active":"2024-04-25T00:13:18.527636+0000","last_became_peered":"2024-04-25T00:13:18.527636+0000","last_unstale":"2024-04-25T00:15:45.754413+0000","last_undegraded":"2024-04-25T00:15:45.754413+0000","last_fullsized":"2024-04-25T00:15:45.754413+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:36.311810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1952,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312965+0000","last_change":"2024-04-25T00:13:19.428181+0000","last_active":"2024-04-25T00:15:45.312965+0000","last_peered":"2024-04-25T00:15:45.312965+0000","last_clean":"2024-04-25T00:15:45.312965+0000","last_became_active":"2024-04-25T00:13:19.427785+0000","last_became_peered":"2024-04-25T00:13:19.427785+0000","last_unstale":"2024-04-25T00:15:45.312965+0000","last_undegraded":"2024-04-25T00:15:45.312965+0000","last_fullsized":"2024-04-25T00:15:45.312965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:31.745679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.114716+0000","last_change":"2024-04-25T00:14:11.271507+0000","last_active":"2024-04-25T00:15:45.114716+0000","last_peered":"2024-04-25T00:15:45.114716+0000","last_clean":"2024-04-25T00:15:45.114716+0000","last_became_active":"2024-04-25T00:14:11.270452+0000","last_became_peered":"2024-04-25T00:14:11.270452+0000","last_unstale":"2024-04-25T00:15:45.114716+0000","last_undegraded":"2024-04-25T00:15:45.114716+0000","last_fullsized":"2024-04-25T00:15:45.114716+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:19.955242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'41","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.262583+0000","last_change":"2024-04-25T00:13:18.529622+0000","last_active":"2024-04-25T00:15:45.262583+0000","last_peered":"2024-04-25T00:15:45.262583+0000","last_clean":"2024-04-25T00:15:45.262583+0000","last_became_active":"2024-04-25T00:13:18.529305+0000","last_became_peered":"2024-04-25T00:13:18.529305+0000","last_unstale":"2024-04-25T00:15:45.262583+0000","last_undegraded":"2024-04-25T00:15:45.262583+0000","last_fullsized":"2024-04-25T00:15:45.262583+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:10:43.334712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312206+0000","last_change":"2024-04-25T00:13:19.421783+0000","last_active":"2024-04-25T00:15:45.312206+0000","last_peered":"2024-04-25T00:15:45.312206+0000","last_clean":"2024-04-25T00:15:45.312206+0000","last_became_active":"2024-04-25T00:13:19.421447+0000","last_became_peered":"2024-04-25T00:13:19.421447+0000","last_unstale":"2024-04-25T00:15:45.312206+0000","last_undegraded":"2024-04-25T00:15:45.312206+0000","last_fullsized":"2024-04-25T00:15:45.312206+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:16.107220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158542+0000","last_change":"2024-04-25T00:13:23.578252+0000","last_active":"2024-04-25T00:15:45.158542+0000","last_peered":"2024-04-25T00:15:45.158542+0000","last_clean":"2024-04-25T00:15:45.158542+0000","last_became_active":"2024-04-25T00:13:23.577779+0000","last_became_peered":"2024-04-25T00:13:23.577779+0000","last_unstale":"2024-04-25T00:15:45.158542+0000","last_undegraded":"2024-04-25T00:15:45.158542+0000","last_fullsized":"2024-04-25T00:15:45.158542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:43.822447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314487+0000","last_change":"2024-04-25T00:13:18.523583+0000","last_active":"2024-04-25T00:15:45.314487+0000","last_peered":"2024-04-25T00:15:45.314487+0000","last_clean":"2024-04-25T00:15:45.314487+0000","last_became_active":"2024-04-25T00:13:18.523238+0000","last_became_peered":"2024-04-25T00:13:18.523238+0000","last_unstale":"2024-04-25T00:15:45.314487+0000","last_undegraded":"2024-04-25T00:15:45.314487+0000","last_fullsized":"2024-04-25T00:15:45.314487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:03.869276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.313212+0000","last_change":"2024-04-25T00:13:19.426021+0000","last_active":"2024-04-25T00:15:45.313212+0000","last_peered":"2024-04-25T00:15:45.313212+0000","last_clean":"2024-04-25T00:15:45.313212+0000","last_became_active":"2024-04-25T00:13:19.425676+0000","last_became_peered":"2024-04-25T00:13:19.425676+0000","last_unstale":"2024-04-25T00:15:45.313212+0000","last_undegraded":"2024-04-25T00:15:45.313212+0000","last_fullsized":"2024-04-25T00:15:45.313212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:10.044202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293774+0000","last_change":"2024-04-25T00:13:19.422402+0000","last_active":"2024-04-25T00:15:45.293774+0000","last_peered":"2024-04-25T00:15:45.293774+0000","last_clean":"2024-04-25T00:15:45.293774+0000","last_became_active":"2024-04-25T00:13:19.422057+0000","last_became_peered":"2024-04-25T00:13:19.422057+0000","last_unstale":"2024-04-25T00:15:45.293774+0000","last_undegraded":"2024-04-25T00:15:45.293774+0000","last_fullsized":"2024-04-25T00:15:45.293774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:31:43.703030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041874+0000","last_change":"2024-04-25T00:13:19.425434+0000","last_active":"2024-04-25T00:15:45.041874+0000","last_peered":"2024-04-25T00:15:45.041874+0000","last_clean":"2024-04-25T00:15:45.041874+0000","last_became_active":"2024-04-25T00:13:19.425051+0000","last_became_peered":"2024-04-25T00:13:19.425051+0000","last_unstale":"2024-04-25T00:15:45.041874+0000","last_undegraded":"2024-04-25T00:15:45.041874+0000","last_fullsized":"2024-04-25T00:15:45.041874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:17.568415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041226+0000","last_change":"2024-04-25T00:13:19.413835+0000","last_active":"2024-04-25T00:15:45.041226+0000","last_peered":"2024-04-25T00:15:45.041226+0000","last_clean":"2024-04-25T00:15:45.041226+0000","last_became_active":"2024-04-25T00:13:19.413406+0000","last_became_peered":"2024-04-25T00:13:19.413406+0000","last_unstale":"2024-04-25T00:15:45.041226+0000","last_undegraded":"2024-04-25T00:15:45.041226+0000","last_fullsized":"2024-04-25T00:15:45.041226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:33.852391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.114923+0000","last_change":"2024-04-25T00:13:19.426875+0000","last_active":"2024-04-25T00:15:45.114923+0000","last_peered":"2024-04-25T00:15:45.114923+0000","last_clean":"2024-04-25T00:15:45.114923+0000","last_became_active":"2024-04-25T00:13:19.425828+0000","last_became_peered":"2024-04-25T00:13:19.425828+0000","last_unstale":"2024-04-25T00:15:45.114923+0000","last_undegraded":"2024-04-25T00:15:45.114923+0000","last_fullsized":"2024-04-25T00:15:45.114923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:21.119551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314721+0000","last_change":"2024-04-25T00:13:19.423637+0000","last_active":"2024-04-25T00:15:45.314721+0000","last_peered":"2024-04-25T00:15:45.314721+0000","last_clean":"2024-04-25T00:15:45.314721+0000","last_became_active":"2024-04-25T00:13:19.423272+0000","last_became_peered":"2024-04-25T00:13:19.423272+0000","last_unstale":"2024-04-25T00:15:45.314721+0000","last_undegraded":"2024-04-25T00:15:45.314721+0000","last_fullsized":"2024-04-25T00:15:45.314721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:32:38.794966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293982+0000","last_change":"2024-04-25T00:13:19.423538+0000","last_active":"2024-04-25T00:15:45.293982+0000","last_peered":"2024-04-25T00:15:45.293982+0000","last_clean":"2024-04-25T00:15:45.293982+0000","last_became_active":"2024-04-25T00:13:19.423028+0000","last_became_peered":"2024-04-25T00:13:19.423028+0000","last_unstale":"2024-04-25T00:15:45.293982+0000","last_undegraded":"2024-04-25T00:15:45.293982+0000","last_fullsized":"2024-04-25T00:15:45.293982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:29:41.975044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041851+0000","last_change":"2024-04-25T00:13:19.416899+0000","last_active":"2024-04-25T00:15:45.041851+0000","last_peered":"2024-04-25T00:15:45.041851+0000","last_clean":"2024-04-25T00:15:45.041851+0000","last_became_active":"2024-04-25T00:13:19.416505+0000","last_became_peered":"2024-04-25T00:13:19.416505+0000","last_unstale":"2024-04-25T00:15:45.041851+0000","last_undegraded":"2024-04-25T00:15:45.041851+0000","last_fullsized":"2024-04-25T00:15:45.041851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:37:40.204119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040400+0000","last_change":"2024-04-25T00:13:19.426215+0000","last_active":"2024-04-25T00:15:45.040400+0000","last_peered":"2024-04-25T00:15:45.040400+0000","last_clean":"2024-04-25T00:15:45.040400+0000","last_became_active":"2024-04-25T00:13:19.422804+0000","last_became_peered":"2024-04-25T00:13:19.422804+0000","last_unstale":"2024-04-25T00:15:45.040400+0000","last_undegraded":"2024-04-25T00:15:45.040400+0000","last_fullsized":"2024-04-25T00:15:45.040400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:24:44.630891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158710+0000","last_change":"2024-04-25T00:13:19.413564+0000","last_active":"2024-04-25T00:15:45.158710+0000","last_peered":"2024-04-25T00:15:45.158710+0000","last_clean":"2024-04-25T00:15:45.158710+0000","last_became_active":"2024-04-25T00:13:19.413223+0000","last_became_peered":"2024-04-25T00:13:19.413223+0000","last_unstale":"2024-04-25T00:15:45.158710+0000","last_undegraded":"2024-04-25T00:15:45.158710+0000","last_fullsized":"2024-04-25T00:15:45.158710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:45.629018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.039748+0000","last_change":"2024-04-25T00:13:19.425945+0000","last_active":"2024-04-25T00:15:45.039748+0000","last_peered":"2024-04-25T00:15:45.039748+0000","last_clean":"2024-04-25T00:15:45.039748+0000","last_became_active":"2024-04-25T00:13:19.422499+0000","last_became_peered":"2024-04-25T00:13:19.422499+0000","last_unstale":"2024-04-25T00:15:45.039748+0000","last_undegraded":"2024-04-25T00:15:45.039748+0000","last_fullsized":"2024-04-25T00:15:45.039748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:45.299300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309420+0000","last_change":"2024-04-25T00:15:11.584671+0000","last_active":"2024-04-25T00:15:45.309420+0000","last_peered":"2024-04-25T00:15:45.309420+0000","last_clean":"2024-04-25T00:15:45.309420+0000","last_became_active":"2024-04-25T00:15:11.584144+0000","last_became_peered":"2024-04-25T00:15:11.584144+0000","last_unstale":"2024-04-25T00:15:45.309420+0000","last_undegraded":"2024-04-25T00:15:45.309420+0000","last_fullsized":"2024-04-25T00:15:45.309420+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:37:52.522586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314719+0000","last_change":"2024-04-25T00:13:19.420550+0000","last_active":"2024-04-25T00:15:45.314719+0000","last_peered":"2024-04-25T00:15:45.314719+0000","last_clean":"2024-04-25T00:15:45.314719+0000","last_became_active":"2024-04-25T00:13:19.420235+0000","last_became_peered":"2024-04-25T00:13:19.420235+0000","last_unstale":"2024-04-25T00:15:45.314719+0000","last_undegraded":"2024-04-25T00:15:45.314719+0000","last_fullsized":"2024-04-25T00:15:45.314719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:59:36.543776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293898+0000","last_change":"2024-04-25T00:13:19.429995+0000","last_active":"2024-04-25T00:15:45.293898+0000","last_peered":"2024-04-25T00:15:45.293898+0000","last_clean":"2024-04-25T00:15:45.293898+0000","last_became_active":"2024-04-25T00:13:19.429185+0000","last_became_peered":"2024-04-25T00:13:19.429185+0000","last_unstale":"2024-04-25T00:15:45.293898+0000","last_undegraded":"2024-04-25T00:15:45.293898+0000","last_fullsized":"2024-04-25T00:15:45.293898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:29:27.974479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107352,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600055,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1337,"num_read_kb":783,"num_write":372,"num_write_kb":1466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":1800165,"data_compressed":80712,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":300,"ondisk_log_size":300,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383723,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":520207,"data_compressed":26578,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645358,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1212416,"data_stored":1315825,"data_compressed":73110,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":927315,"data_compressed":48714,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":923597,"data_compressed":48714,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1114112,"data_stored":1127718,"data_compressed":65352,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":526206,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":583001,"data_compressed":28180,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":527745,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1130496,"data_stored":594343,"data_compressed":31170,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":1050031,"data_compressed":54870,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":530113,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32292,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712860,"statfs":{"total":95995035648,"available":95961968640,"internally_reserved":0,"allocated":1142784,"data_stored":1253669,"data_compressed":71508,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":69255,"data_compressed":1602,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":69558,"data_compressed":1602,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":134420,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":200672,"data_compressed":7758,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:15:50.492 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-25T00:15:50.492 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:15:50.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:15:50.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:15:50.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:15:50.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:15:50.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3873171947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:50.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: Updating smithi139:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T00:15:50.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: Updating smithi027:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T00:15:50.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: Updating smithi153:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T00:15:50.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3500465566' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:50.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-25T00:15:50.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:15:50.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:15:50.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:15:50.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.505 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:50 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/687009595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:50.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:15:50.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:15:50.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-25T00:15:50.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:15:50.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3873171947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:50.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: Updating smithi139:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T00:15:50.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: Updating smithi027:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T00:15:50.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: Updating smithi153:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T00:15:50.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3500465566' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:50.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-25T00:15:50.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:15:50.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:15:50.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:15:50.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:15:50.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:50 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/687009595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:50.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.630+0000 7ff7cc8cd640 1 -- 172.21.15.27:0/1128535844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c81017c0 msgr2=0x7ff7c8101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:50.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.630+0000 7ff7cc8cd640 1 --2- 172.21.15.27:0/1128535844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c81017c0 0x7ff7c8101c20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8009900 tx=0x7ff7b802a480 comp rx=0 tx=0).stop 2024-04-25T00:15:50.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.630+0000 7ff7cc8cd640 1 -- 172.21.15.27:0/1128535844 shutdown_connections 2024-04-25T00:15:50.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.630+0000 7ff7cc8cd640 1 --2- 172.21.15.27:0/1128535844 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7c8102160 0x7ff7c810e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.630+0000 7ff7cc8cd640 1 --2- 172.21.15.27:0/1128535844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c81017c0 0x7ff7c8101c20 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.630+0000 7ff7cc8cd640 1 --2- 172.21.15.27:0/1128535844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7c8107750 0x7ff7c8107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.630+0000 7ff7cc8cd640 1 -- 172.21.15.27:0/1128535844 >> 172.21.15.27:0/1128535844 conn(0x7ff7c80788d0 msgr2=0x7ff7c8078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:50.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.630+0000 7ff7cc8cd640 1 -- 172.21.15.27:0/1128535844 shutdown_connections 2024-04-25T00:15:50.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.630+0000 7ff7cc8cd640 1 -- 172.21.15.27:0/1128535844 wait complete. 2024-04-25T00:15:50.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.631+0000 7ff7cc8cd640 1 Processor -- start 2024-04-25T00:15:50.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.631+0000 7ff7cc8cd640 1 -- start start 2024-04-25T00:15:50.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.631+0000 7ff7cc8cd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7c81017c0 0x7ff7c8197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:50.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.631+0000 7ff7cc8cd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7c8102160 0x7ff7c8198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:50.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.631+0000 7ff7cc8cd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c8107750 0x7ff7c819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:50.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.631+0000 7ff7cc8cd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c8111b10 con 0x7ff7c8107750 2024-04-25T00:15:50.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.631+0000 7ff7cc8cd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c8111990 con 0x7ff7c81017c0 2024-04-25T00:15:50.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.631+0000 7ff7cc8cd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c8111c90 con 0x7ff7c8102160 2024-04-25T00:15:50.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.631+0000 7ff7c6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c8107750 0x7ff7c819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:50.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.631+0000 7ff7c6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c8107750 0x7ff7c819c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47720/0 (socket says 172.21.15.27:47720) 2024-04-25T00:15:50.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.631+0000 7ff7c6d9d640 1 -- 172.21.15.27:0/3131483281 learned_addr learned my addr 172.21.15.27:0/3131483281 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:50.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.631+0000 7ff7c659c640 1 --2- 172.21.15.27:0/3131483281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7c81017c0 0x7ff7c8197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:50.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.631+0000 7ff7c5d9b640 1 --2- 172.21.15.27:0/3131483281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7c8102160 0x7ff7c8198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:50.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.632+0000 7ff7c6d9d640 1 -- 172.21.15.27:0/3131483281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7c8102160 msgr2=0x7ff7c8198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:50.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.632+0000 7ff7c6d9d640 1 --2- 172.21.15.27:0/3131483281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7c8102160 0x7ff7c8198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.632+0000 7ff7c6d9d640 1 -- 172.21.15.27:0/3131483281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7c81017c0 msgr2=0x7ff7c8197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:50.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.632+0000 7ff7c6d9d640 1 --2- 172.21.15.27:0/3131483281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7c81017c0 0x7ff7c8197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.632+0000 7ff7c6d9d640 1 -- 172.21.15.27:0/3131483281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c819cfe0 con 0x7ff7c8107750 2024-04-25T00:15:50.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.632+0000 7ff7c5d9b640 1 --2- 172.21.15.27:0/3131483281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7c8102160 0x7ff7c8198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:50.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.632+0000 7ff7c6d9d640 1 --2- 172.21.15.27:0/3131483281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c8107750 0x7ff7c819c8b0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000eb10 tx=0x7ff7c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:50.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.632+0000 7ff7bf7fe640 1 -- 172.21.15.27:0/3131483281 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0004b20 con 0x7ff7c8107750 2024-04-25T00:15:50.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.632+0000 7ff7cc8cd640 1 -- 172.21.15.27:0/3131483281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7c819d2d0 con 0x7ff7c8107750 2024-04-25T00:15:50.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.632+0000 7ff7bf7fe640 1 -- 172.21.15.27:0/3131483281 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7c0004cc0 con 0x7ff7c8107750 2024-04-25T00:15:50.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.632+0000 7ff7bf7fe640 1 -- 172.21.15.27:0/3131483281 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0010760 con 0x7ff7c8107750 2024-04-25T00:15:50.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.632+0000 7ff7cc8cd640 1 -- 172.21.15.27:0/3131483281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7c81a4b60 con 0x7ff7c8107750 2024-04-25T00:15:50.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.633+0000 7ff7cc8cd640 1 -- 172.21.15.27:0/3131483281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7c80689b0 con 0x7ff7c8107750 2024-04-25T00:15:50.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.634+0000 7ff7bf7fe640 1 -- 172.21.15.27:0/3131483281 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7ff7c0005020 con 0x7ff7c8107750 2024-04-25T00:15:50.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.635+0000 7ff7bf7fe640 1 --2- 172.21.15.27:0/3131483281 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff7b40a2ea0 0x7ff7b40a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:50.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.635+0000 7ff7c659c640 1 --2- 172.21.15.27:0/3131483281 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff7b40a2ea0 0x7ff7b40a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:50.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.635+0000 7ff7bf7fe640 1 -- 172.21.15.27:0/3131483281 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7ff7c00cffa0 con 0x7ff7c8107750 2024-04-25T00:15:50.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.635+0000 7ff7c659c640 1 --2- 172.21.15.27:0/3131483281 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff7b40a2ea0 0x7ff7b40a5360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0005fd0 tx=0x7ff7b000b390 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:50.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.639+0000 7ff7bf7fe640 1 -- 172.21.15.27:0/3131483281 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff7c009c600 con 0x7ff7c8107750 2024-04-25T00:15:50.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.821+0000 7ff7cc8cd640 1 -- 172.21.15.27:0/3131483281 --> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff7c8061b80 con 0x7ff7b40a2ea0 2024-04-25T00:15:50.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.830+0000 7ff7bf7fe640 1 -- 172.21.15.27:0/3131483281 <== mgr.15564 v2:172.21.15.27:6800/1076858552 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339735 (secure 0 0 0) 0x7ff7c8061b80 con 0x7ff7b40a2ea0 2024-04-25T00:15:50.833 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:50.835 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T00:15:50.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.833+0000 7ff7cc8cd640 1 -- 172.21.15.27:0/3131483281 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff7b40a2ea0 msgr2=0x7ff7b40a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:50.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.833+0000 7ff7cc8cd640 1 --2- 172.21.15.27:0/3131483281 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff7b40a2ea0 0x7ff7b40a5360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0005fd0 tx=0x7ff7b000b390 comp rx=0 tx=0).stop 2024-04-25T00:15:50.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.834+0000 7ff7cc8cd640 1 -- 172.21.15.27:0/3131483281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c8107750 msgr2=0x7ff7c819c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:50.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.834+0000 7ff7cc8cd640 1 --2- 172.21.15.27:0/3131483281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c8107750 0x7ff7c819c8b0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000eb10 tx=0x7ff7c000c2d0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.834+0000 7ff7cc8cd640 1 -- 172.21.15.27:0/3131483281 shutdown_connections 2024-04-25T00:15:50.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.834+0000 7ff7cc8cd640 1 --2- 172.21.15.27:0/3131483281 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff7b40a2ea0 0x7ff7b40a5360 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.834+0000 7ff7cc8cd640 1 --2- 172.21.15.27:0/3131483281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7c8107750 0x7ff7c819c8b0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.834+0000 7ff7cc8cd640 1 --2- 172.21.15.27:0/3131483281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7c8102160 0x7ff7c8198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.834+0000 7ff7cc8cd640 1 --2- 172.21.15.27:0/3131483281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7c81017c0 0x7ff7c8197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.834+0000 7ff7cc8cd640 1 -- 172.21.15.27:0/3131483281 >> 172.21.15.27:0/3131483281 conn(0x7ff7c80788d0 msgr2=0x7ff7c8100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:50.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.834+0000 7ff7cc8cd640 1 -- 172.21.15.27:0/3131483281 shutdown_connections 2024-04-25T00:15:50.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.834+0000 7ff7cc8cd640 1 -- 172.21.15.27:0/3131483281 wait complete. 2024-04-25T00:15:50.845 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-25T00:15:49.051541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190657,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1473,"num_read_kb":902,"num_write":555,"num_write_kb":4406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":939,"ondisk_log_size":939,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":349548,"kb_used_data":12844,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124592276,"statfs":{"total":1151940427776,"available":1151582490624,"internally_reserved":0,"allocated":13152256,"data_stored":9879770,"data_compressed":536900,"data_compressed_allocated":729088,"data_compressed_original":4308992,"omap_allocated":19143,"internal_metadata":344634681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.987382"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.294151+0000","last_change":"2024-04-25T00:13:19.429658+0000","last_active":"2024-04-25T00:15:45.294151+0000","last_peered":"2024-04-25T00:15:45.294151+0000","last_clean":"2024-04-25T00:15:45.294151+0000","last_became_active":"2024-04-25T00:13:19.428426+0000","last_became_peered":"2024-04-25T00:13:19.428426+0000","last_unstale":"2024-04-25T00:15:45.294151+0000","last_undegraded":"2024-04-25T00:15:45.294151+0000","last_fullsized":"2024-04-25T00:15:45.294151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:10.814361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115194+0000","last_change":"2024-04-25T00:13:19.425504+0000","last_active":"2024-04-25T00:15:45.115194+0000","last_peered":"2024-04-25T00:15:45.115194+0000","last_clean":"2024-04-25T00:15:45.115194+0000","last_became_active":"2024-04-25T00:13:19.425170+0000","last_became_peered":"2024-04-25T00:13:19.425170+0000","last_unstale":"2024-04-25T00:15:45.115194+0000","last_undegraded":"2024-04-25T00:15:45.115194+0000","last_fullsized":"2024-04-25T00:15:45.115194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:31:13.826981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.114835+0000","last_change":"2024-04-25T00:13:19.420842+0000","last_active":"2024-04-25T00:15:45.114835+0000","last_peered":"2024-04-25T00:15:45.114835+0000","last_clean":"2024-04-25T00:15:45.114835+0000","last_became_active":"2024-04-25T00:13:19.420241+0000","last_became_peered":"2024-04-25T00:13:19.420241+0000","last_unstale":"2024-04-25T00:15:45.114835+0000","last_undegraded":"2024-04-25T00:15:45.114835+0000","last_fullsized":"2024-04-25T00:15:45.114835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:52:39.500432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149314+0000","last_change":"2024-04-25T00:14:11.267688+0000","last_active":"2024-04-25T00:15:45.149314+0000","last_peered":"2024-04-25T00:15:45.149314+0000","last_clean":"2024-04-25T00:15:45.149314+0000","last_became_active":"2024-04-25T00:14:11.267225+0000","last_became_peered":"2024-04-25T00:14:11.267225+0000","last_unstale":"2024-04-25T00:15:45.149314+0000","last_undegraded":"2024-04-25T00:15:45.149314+0000","last_fullsized":"2024-04-25T00:15:45.149314+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:15.867489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151754+0000","last_change":"2024-04-25T00:13:19.416519+0000","last_active":"2024-04-25T00:15:45.151754+0000","last_peered":"2024-04-25T00:15:45.151754+0000","last_clean":"2024-04-25T00:15:45.151754+0000","last_became_active":"2024-04-25T00:13:19.416080+0000","last_became_peered":"2024-04-25T00:13:19.416080+0000","last_unstale":"2024-04-25T00:15:45.151754+0000","last_undegraded":"2024-04-25T00:15:45.151754+0000","last_fullsized":"2024-04-25T00:15:45.151754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:04.450160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.162148+0000","last_change":"2024-04-25T00:13:19.424378+0000","last_active":"2024-04-25T00:15:45.162148+0000","last_peered":"2024-04-25T00:15:45.162148+0000","last_clean":"2024-04-25T00:15:45.162148+0000","last_became_active":"2024-04-25T00:13:19.423918+0000","last_became_peered":"2024-04-25T00:13:19.423918+0000","last_unstale":"2024-04-25T00:15:45.162148+0000","last_undegraded":"2024-04-25T00:15:45.162148+0000","last_fullsized":"2024-04-25T00:15:45.162148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:01:11.397328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.311886+0000","last_change":"2024-04-25T00:13:19.424729+0000","last_active":"2024-04-25T00:15:45.311886+0000","last_peered":"2024-04-25T00:15:45.311886+0000","last_clean":"2024-04-25T00:15:45.311886+0000","last_became_active":"2024-04-25T00:13:19.422874+0000","last_became_peered":"2024-04-25T00:13:19.422874+0000","last_unstale":"2024-04-25T00:15:45.311886+0000","last_undegraded":"2024-04-25T00:15:45.311886+0000","last_fullsized":"2024-04-25T00:15:45.311886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:52:39.483589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.039933+0000","last_change":"2024-04-25T00:13:19.418043+0000","last_active":"2024-04-25T00:15:45.039933+0000","last_peered":"2024-04-25T00:15:45.039933+0000","last_clean":"2024-04-25T00:15:45.039933+0000","last_became_active":"2024-04-25T00:13:19.417443+0000","last_became_peered":"2024-04-25T00:13:19.417443+0000","last_unstale":"2024-04-25T00:15:45.039933+0000","last_undegraded":"2024-04-25T00:15:45.039933+0000","last_fullsized":"2024-04-25T00:15:45.039933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:47:53.141744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309810+0000","last_change":"2024-04-25T00:13:19.420989+0000","last_active":"2024-04-25T00:15:45.309810+0000","last_peered":"2024-04-25T00:15:45.309810+0000","last_clean":"2024-04-25T00:15:45.309810+0000","last_became_active":"2024-04-25T00:13:19.420669+0000","last_became_peered":"2024-04-25T00:13:19.420669+0000","last_unstale":"2024-04-25T00:15:45.309810+0000","last_undegraded":"2024-04-25T00:15:45.309810+0000","last_fullsized":"2024-04-25T00:15:45.309810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:18.236132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.162583+0000","last_change":"2024-04-25T00:13:19.424176+0000","last_active":"2024-04-25T00:15:45.162583+0000","last_peered":"2024-04-25T00:15:45.162583+0000","last_clean":"2024-04-25T00:15:45.162583+0000","last_became_active":"2024-04-25T00:13:19.423656+0000","last_became_peered":"2024-04-25T00:13:19.423656+0000","last_unstale":"2024-04-25T00:15:45.162583+0000","last_undegraded":"2024-04-25T00:15:45.162583+0000","last_fullsized":"2024-04-25T00:15:45.162583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:24:41.697077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312946+0000","last_change":"2024-04-25T00:13:19.419624+0000","last_active":"2024-04-25T00:15:45.312946+0000","last_peered":"2024-04-25T00:15:45.312946+0000","last_clean":"2024-04-25T00:15:45.312946+0000","last_became_active":"2024-04-25T00:13:19.419263+0000","last_became_peered":"2024-04-25T00:13:19.419263+0000","last_unstale":"2024-04-25T00:15:45.312946+0000","last_undegraded":"2024-04-25T00:15:45.312946+0000","last_fullsized":"2024-04-25T00:15:45.312946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:29.130733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041558+0000","last_change":"2024-04-25T00:13:19.427865+0000","last_active":"2024-04-25T00:15:45.041558+0000","last_peered":"2024-04-25T00:15:45.041558+0000","last_clean":"2024-04-25T00:15:45.041558+0000","last_became_active":"2024-04-25T00:13:19.427538+0000","last_became_peered":"2024-04-25T00:13:19.427538+0000","last_unstale":"2024-04-25T00:15:45.041558+0000","last_undegraded":"2024-04-25T00:15:45.041558+0000","last_fullsized":"2024-04-25T00:15:45.041558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:05.136948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314986+0000","last_change":"2024-04-25T00:13:19.419261+0000","last_active":"2024-04-25T00:15:45.314986+0000","last_peered":"2024-04-25T00:15:45.314986+0000","last_clean":"2024-04-25T00:15:45.314986+0000","last_became_active":"2024-04-25T00:13:19.418938+0000","last_became_peered":"2024-04-25T00:13:19.418938+0000","last_unstale":"2024-04-25T00:15:45.314986+0000","last_undegraded":"2024-04-25T00:15:45.314986+0000","last_fullsized":"2024-04-25T00:15:45.314986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:44.971640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.114768+0000","last_change":"2024-04-25T00:13:19.417654+0000","last_active":"2024-04-25T00:15:45.114768+0000","last_peered":"2024-04-25T00:15:45.114768+0000","last_clean":"2024-04-25T00:15:45.114768+0000","last_became_active":"2024-04-25T00:13:19.417319+0000","last_became_peered":"2024-04-25T00:13:19.417319+0000","last_unstale":"2024-04-25T00:15:45.114768+0000","last_undegraded":"2024-04-25T00:15:45.114768+0000","last_fullsized":"2024-04-25T00:15:45.114768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:43:54.447937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115100+0000","last_change":"2024-04-25T00:13:19.420761+0000","last_active":"2024-04-25T00:15:45.115100+0000","last_peered":"2024-04-25T00:15:45.115100+0000","last_clean":"2024-04-25T00:15:45.115100+0000","last_became_active":"2024-04-25T00:13:19.420334+0000","last_became_peered":"2024-04-25T00:13:19.420334+0000","last_unstale":"2024-04-25T00:15:45.115100+0000","last_undegraded":"2024-04-25T00:15:45.115100+0000","last_fullsized":"2024-04-25T00:15:45.115100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:20.136470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309282+0000","last_change":"2024-04-25T00:13:19.420132+0000","last_active":"2024-04-25T00:15:45.309282+0000","last_peered":"2024-04-25T00:15:45.309282+0000","last_clean":"2024-04-25T00:15:45.309282+0000","last_became_active":"2024-04-25T00:13:19.418910+0000","last_became_peered":"2024-04-25T00:13:19.418910+0000","last_unstale":"2024-04-25T00:15:45.309282+0000","last_undegraded":"2024-04-25T00:15:45.309282+0000","last_fullsized":"2024-04-25T00:15:45.309282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:17.895412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.294056+0000","last_change":"2024-04-25T00:13:19.430242+0000","last_active":"2024-04-25T00:15:45.294056+0000","last_peered":"2024-04-25T00:15:45.294056+0000","last_clean":"2024-04-25T00:15:45.294056+0000","last_became_active":"2024-04-25T00:13:19.429077+0000","last_became_peered":"2024-04-25T00:13:19.429077+0000","last_unstale":"2024-04-25T00:15:45.294056+0000","last_undegraded":"2024-04-25T00:15:45.294056+0000","last_fullsized":"2024-04-25T00:15:45.294056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:34:14.351157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149646+0000","last_change":"2024-04-25T00:13:19.414375+0000","last_active":"2024-04-25T00:15:45.149646+0000","last_peered":"2024-04-25T00:15:45.149646+0000","last_clean":"2024-04-25T00:15:45.149646+0000","last_became_active":"2024-04-25T00:13:19.414003+0000","last_became_peered":"2024-04-25T00:13:19.414003+0000","last_unstale":"2024-04-25T00:15:45.149646+0000","last_undegraded":"2024-04-25T00:15:45.149646+0000","last_fullsized":"2024-04-25T00:15:45.149646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:43:50.612743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041639+0000","last_change":"2024-04-25T00:13:19.415322+0000","last_active":"2024-04-25T00:15:45.041639+0000","last_peered":"2024-04-25T00:15:45.041639+0000","last_clean":"2024-04-25T00:15:45.041639+0000","last_became_active":"2024-04-25T00:13:19.414925+0000","last_became_peered":"2024-04-25T00:13:19.414925+0000","last_unstale":"2024-04-25T00:15:45.041639+0000","last_undegraded":"2024-04-25T00:15:45.041639+0000","last_fullsized":"2024-04-25T00:15:45.041639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:58.140640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041305+0000","last_change":"2024-04-25T00:13:23.584763+0000","last_active":"2024-04-25T00:15:45.041305+0000","last_peered":"2024-04-25T00:15:45.041305+0000","last_clean":"2024-04-25T00:15:45.041305+0000","last_became_active":"2024-04-25T00:13:23.584303+0000","last_became_peered":"2024-04-25T00:13:23.584303+0000","last_unstale":"2024-04-25T00:15:45.041305+0000","last_undegraded":"2024-04-25T00:15:45.041305+0000","last_fullsized":"2024-04-25T00:15:45.041305+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:54:11.447867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'14","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149948+0000","last_change":"2024-04-25T00:13:18.519576+0000","last_active":"2024-04-25T00:15:45.149948+0000","last_peered":"2024-04-25T00:15:45.149948+0000","last_clean":"2024-04-25T00:15:45.149948+0000","last_became_active":"2024-04-25T00:13:18.519201+0000","last_became_peered":"2024-04-25T00:13:18.519201+0000","last_unstale":"2024-04-25T00:15:45.149948+0000","last_undegraded":"2024-04-25T00:15:45.149948+0000","last_fullsized":"2024-04-25T00:15:45.149948+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:19.681096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040313+0000","last_change":"2024-04-25T00:13:19.419098+0000","last_active":"2024-04-25T00:15:45.040313+0000","last_peered":"2024-04-25T00:15:45.040313+0000","last_clean":"2024-04-25T00:15:45.040313+0000","last_became_active":"2024-04-25T00:13:19.418649+0000","last_became_peered":"2024-04-25T00:13:19.418649+0000","last_unstale":"2024-04-25T00:15:45.040313+0000","last_undegraded":"2024-04-25T00:15:45.040313+0000","last_fullsized":"2024-04-25T00:15:45.040313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:04.561470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041042+0000","last_change":"2024-04-25T00:13:23.580354+0000","last_active":"2024-04-25T00:15:45.041042+0000","last_peered":"2024-04-25T00:15:45.041042+0000","last_clean":"2024-04-25T00:15:45.041042+0000","last_became_active":"2024-04-25T00:13:23.579944+0000","last_became_peered":"2024-04-25T00:13:23.579944+0000","last_unstale":"2024-04-25T00:15:45.041042+0000","last_undegraded":"2024-04-25T00:15:45.041042+0000","last_fullsized":"2024-04-25T00:15:45.041042+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:48:57.788974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040339+0000","last_change":"2024-04-25T00:13:18.527471+0000","last_active":"2024-04-25T00:15:45.040339+0000","last_peered":"2024-04-25T00:15:45.040339+0000","last_clean":"2024-04-25T00:15:45.040339+0000","last_became_active":"2024-04-25T00:13:18.527138+0000","last_became_peered":"2024-04-25T00:13:18.527138+0000","last_unstale":"2024-04-25T00:15:45.040339+0000","last_undegraded":"2024-04-25T00:15:45.040339+0000","last_fullsized":"2024-04-25T00:15:45.040339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:38.003719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115017+0000","last_change":"2024-04-25T00:13:19.420563+0000","last_active":"2024-04-25T00:15:45.115017+0000","last_peered":"2024-04-25T00:15:45.115017+0000","last_clean":"2024-04-25T00:15:45.115017+0000","last_became_active":"2024-04-25T00:13:19.420142+0000","last_became_peered":"2024-04-25T00:13:19.420142+0000","last_unstale":"2024-04-25T00:15:45.115017+0000","last_undegraded":"2024-04-25T00:15:45.115017+0000","last_fullsized":"2024-04-25T00:15:45.115017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:23:08.423795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041136+0000","last_change":"2024-04-25T00:14:11.271321+0000","last_active":"2024-04-25T00:15:45.041136+0000","last_peered":"2024-04-25T00:15:45.041136+0000","last_clean":"2024-04-25T00:15:45.041136+0000","last_became_active":"2024-04-25T00:14:11.270478+0000","last_became_peered":"2024-04-25T00:14:11.270478+0000","last_unstale":"2024-04-25T00:15:45.041136+0000","last_undegraded":"2024-04-25T00:15:45.041136+0000","last_fullsized":"2024-04-25T00:15:45.041136+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:41.796155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'13","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.152142+0000","last_change":"2024-04-25T00:13:18.520509+0000","last_active":"2024-04-25T00:15:45.152142+0000","last_peered":"2024-04-25T00:15:45.152142+0000","last_clean":"2024-04-25T00:15:45.152142+0000","last_became_active":"2024-04-25T00:13:18.520139+0000","last_became_peered":"2024-04-25T00:13:18.520139+0000","last_unstale":"2024-04-25T00:15:45.152142+0000","last_undegraded":"2024-04-25T00:15:45.152142+0000","last_fullsized":"2024-04-25T00:15:45.152142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:08.797670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309574+0000","last_change":"2024-04-25T00:13:19.419115+0000","last_active":"2024-04-25T00:15:45.309574+0000","last_peered":"2024-04-25T00:15:45.309574+0000","last_clean":"2024-04-25T00:15:45.309574+0000","last_became_active":"2024-04-25T00:13:19.417865+0000","last_became_peered":"2024-04-25T00:13:19.417865+0000","last_unstale":"2024-04-25T00:15:45.309574+0000","last_undegraded":"2024-04-25T00:15:45.309574+0000","last_fullsized":"2024-04-25T00:15:45.309574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:20:11.173803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041021+0000","last_change":"2024-04-25T00:13:23.585052+0000","last_active":"2024-04-25T00:15:45.041021+0000","last_peered":"2024-04-25T00:15:45.041021+0000","last_clean":"2024-04-25T00:15:45.041021+0000","last_became_active":"2024-04-25T00:13:23.584696+0000","last_became_peered":"2024-04-25T00:13:23.584696+0000","last_unstale":"2024-04-25T00:15:45.041021+0000","last_undegraded":"2024-04-25T00:15:45.041021+0000","last_fullsized":"2024-04-25T00:15:45.041021+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:26.894183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151965+0000","last_change":"2024-04-25T00:13:18.522793+0000","last_active":"2024-04-25T00:15:45.151965+0000","last_peered":"2024-04-25T00:15:45.151965+0000","last_clean":"2024-04-25T00:15:45.151965+0000","last_became_active":"2024-04-25T00:13:18.522453+0000","last_became_peered":"2024-04-25T00:13:18.522453+0000","last_unstale":"2024-04-25T00:15:45.151965+0000","last_undegraded":"2024-04-25T00:15:45.151965+0000","last_fullsized":"2024-04-25T00:15:45.151965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:08.368214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.159151+0000","last_change":"2024-04-25T00:13:19.418573+0000","last_active":"2024-04-25T00:15:45.159151+0000","last_peered":"2024-04-25T00:15:45.159151+0000","last_clean":"2024-04-25T00:15:45.159151+0000","last_became_active":"2024-04-25T00:13:19.418219+0000","last_became_peered":"2024-04-25T00:13:19.418219+0000","last_unstale":"2024-04-25T00:15:45.159151+0000","last_undegraded":"2024-04-25T00:15:45.159151+0000","last_fullsized":"2024-04-25T00:15:45.159151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:08:44.453017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041456+0000","last_change":"2024-04-25T00:13:23.578716+0000","last_active":"2024-04-25T00:15:45.041456+0000","last_peered":"2024-04-25T00:15:45.041456+0000","last_clean":"2024-04-25T00:15:45.041456+0000","last_became_active":"2024-04-25T00:13:23.578325+0000","last_became_peered":"2024-04-25T00:13:23.578325+0000","last_unstale":"2024-04-25T00:15:45.041456+0000","last_undegraded":"2024-04-25T00:15:45.041456+0000","last_fullsized":"2024-04-25T00:15:45.041456+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:39:57.145656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.731145+0000","last_change":"2024-04-25T00:13:18.527369+0000","last_active":"2024-04-25T00:15:45.731145+0000","last_peered":"2024-04-25T00:15:45.731145+0000","last_clean":"2024-04-25T00:15:45.731145+0000","last_became_active":"2024-04-25T00:13:18.526032+0000","last_became_peered":"2024-04-25T00:13:18.526032+0000","last_unstale":"2024-04-25T00:15:45.731145+0000","last_undegraded":"2024-04-25T00:15:45.731145+0000","last_fullsized":"2024-04-25T00:15:45.731145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:52.635184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151572+0000","last_change":"2024-04-25T00:13:19.413305+0000","last_active":"2024-04-25T00:15:45.151572+0000","last_peered":"2024-04-25T00:15:45.151572+0000","last_clean":"2024-04-25T00:15:45.151572+0000","last_became_active":"2024-04-25T00:13:19.412956+0000","last_became_peered":"2024-04-25T00:13:19.412956+0000","last_unstale":"2024-04-25T00:15:45.151572+0000","last_undegraded":"2024-04-25T00:15:45.151572+0000","last_fullsized":"2024-04-25T00:15:45.151572+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:54.436470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041546+0000","last_change":"2024-04-25T00:14:11.273057+0000","last_active":"2024-04-25T00:15:45.041546+0000","last_peered":"2024-04-25T00:15:45.041546+0000","last_clean":"2024-04-25T00:15:45.041546+0000","last_became_active":"2024-04-25T00:14:11.272230+0000","last_became_peered":"2024-04-25T00:14:11.272230+0000","last_unstale":"2024-04-25T00:15:45.041546+0000","last_undegraded":"2024-04-25T00:15:45.041546+0000","last_fullsized":"2024-04-25T00:15:45.041546+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:37.695139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.162194+0000","last_change":"2024-04-25T00:13:18.517507+0000","last_active":"2024-04-25T00:15:45.162194+0000","last_peered":"2024-04-25T00:15:45.162194+0000","last_clean":"2024-04-25T00:15:45.162194+0000","last_became_active":"2024-04-25T00:13:18.517099+0000","last_became_peered":"2024-04-25T00:13:18.517099+0000","last_unstale":"2024-04-25T00:15:45.162194+0000","last_undegraded":"2024-04-25T00:15:45.162194+0000","last_fullsized":"2024-04-25T00:15:45.162194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:42:17.357610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.159046+0000","last_change":"2024-04-25T00:14:11.271691+0000","last_active":"2024-04-25T00:15:45.159046+0000","last_peered":"2024-04-25T00:15:45.159046+0000","last_clean":"2024-04-25T00:15:45.159046+0000","last_became_active":"2024-04-25T00:14:11.271160+0000","last_became_peered":"2024-04-25T00:14:11.271160+0000","last_unstale":"2024-04-25T00:15:45.159046+0000","last_undegraded":"2024-04-25T00:15:45.159046+0000","last_fullsized":"2024-04-25T00:15:45.159046+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:24.416810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.294089+0000","last_change":"2024-04-25T00:13:23.582526+0000","last_active":"2024-04-25T00:15:45.294089+0000","last_peered":"2024-04-25T00:15:45.294089+0000","last_clean":"2024-04-25T00:15:45.294089+0000","last_became_active":"2024-04-25T00:13:23.581221+0000","last_became_peered":"2024-04-25T00:13:23.581221+0000","last_unstale":"2024-04-25T00:15:45.294089+0000","last_undegraded":"2024-04-25T00:15:45.294089+0000","last_fullsized":"2024-04-25T00:15:45.294089+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:46.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314551+0000","last_change":"2024-04-25T00:13:18.522245+0000","last_active":"2024-04-25T00:15:45.314551+0000","last_peered":"2024-04-25T00:15:45.314551+0000","last_clean":"2024-04-25T00:15:45.314551+0000","last_became_active":"2024-04-25T00:13:18.521440+0000","last_became_peered":"2024-04-25T00:13:18.521440+0000","last_unstale":"2024-04-25T00:15:45.314551+0000","last_undegraded":"2024-04-25T00:15:45.314551+0000","last_fullsized":"2024-04-25T00:15:45.314551+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:21:37.847964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040022+0000","last_change":"2024-04-25T00:13:19.422771+0000","last_active":"2024-04-25T00:15:45.040022+0000","last_peered":"2024-04-25T00:15:45.040022+0000","last_clean":"2024-04-25T00:15:45.040022+0000","last_became_active":"2024-04-25T00:13:19.422387+0000","last_became_peered":"2024-04-25T00:13:19.422387+0000","last_unstale":"2024-04-25T00:15:45.040022+0000","last_undegraded":"2024-04-25T00:15:45.040022+0000","last_fullsized":"2024-04-25T00:15:45.040022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:15.647186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115073+0000","last_change":"2024-04-25T00:13:23.586235+0000","last_active":"2024-04-25T00:15:45.115073+0000","last_peered":"2024-04-25T00:15:45.115073+0000","last_clean":"2024-04-25T00:15:45.115073+0000","last_became_active":"2024-04-25T00:13:23.585803+0000","last_became_peered":"2024-04-25T00:13:23.585803+0000","last_unstale":"2024-04-25T00:15:45.115073+0000","last_undegraded":"2024-04-25T00:15:45.115073+0000","last_fullsized":"2024-04-25T00:15:45.115073+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:46.235435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158742+0000","last_change":"2024-04-25T00:13:18.525283+0000","last_active":"2024-04-25T00:15:45.158742+0000","last_peered":"2024-04-25T00:15:45.158742+0000","last_clean":"2024-04-25T00:15:45.158742+0000","last_became_active":"2024-04-25T00:13:18.524939+0000","last_became_peered":"2024-04-25T00:13:18.524939+0000","last_unstale":"2024-04-25T00:15:45.158742+0000","last_undegraded":"2024-04-25T00:15:45.158742+0000","last_fullsized":"2024-04-25T00:15:45.158742+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:09.258498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.148867+0000","last_change":"2024-04-25T00:13:19.415746+0000","last_active":"2024-04-25T00:15:45.148867+0000","last_peered":"2024-04-25T00:15:45.148867+0000","last_clean":"2024-04-25T00:15:45.148867+0000","last_became_active":"2024-04-25T00:13:19.415325+0000","last_became_peered":"2024-04-25T00:13:19.415325+0000","last_unstale":"2024-04-25T00:15:45.148867+0000","last_undegraded":"2024-04-25T00:15:45.148867+0000","last_fullsized":"2024-04-25T00:15:45.148867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:17:48.268845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.039798+0000","last_change":"2024-04-25T00:13:23.583153+0000","last_active":"2024-04-25T00:15:45.039798+0000","last_peered":"2024-04-25T00:15:45.039798+0000","last_clean":"2024-04-25T00:15:45.039798+0000","last_became_active":"2024-04-25T00:13:23.580257+0000","last_became_peered":"2024-04-25T00:13:23.580257+0000","last_unstale":"2024-04-25T00:15:45.039798+0000","last_undegraded":"2024-04-25T00:15:45.039798+0000","last_fullsized":"2024-04-25T00:15:45.039798+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:09:32.990253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'5","reported_seq":279,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.753951+0000","last_change":"2024-04-25T00:13:18.520365+0000","last_active":"2024-04-25T00:15:45.753951+0000","last_peered":"2024-04-25T00:15:45.753951+0000","last_clean":"2024-04-25T00:15:45.753951+0000","last_became_active":"2024-04-25T00:13:18.519918+0000","last_became_peered":"2024-04-25T00:13:18.519918+0000","last_unstale":"2024-04-25T00:15:45.753951+0000","last_undegraded":"2024-04-25T00:15:45.753951+0000","last_fullsized":"2024-04-25T00:15:45.753951+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:46.415197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":240,"num_read_kb":240,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309728+0000","last_change":"2024-04-25T00:13:19.414341+0000","last_active":"2024-04-25T00:15:45.309728+0000","last_peered":"2024-04-25T00:15:45.309728+0000","last_clean":"2024-04-25T00:15:45.309728+0000","last_became_active":"2024-04-25T00:13:19.413793+0000","last_became_peered":"2024-04-25T00:13:19.413793+0000","last_unstale":"2024-04-25T00:15:45.309728+0000","last_undegraded":"2024-04-25T00:15:45.309728+0000","last_fullsized":"2024-04-25T00:15:45.309728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:55.927082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.148616+0000","last_change":"2024-04-25T00:13:23.590631+0000","last_active":"2024-04-25T00:15:45.148616+0000","last_peered":"2024-04-25T00:15:45.148616+0000","last_clean":"2024-04-25T00:15:45.148616+0000","last_became_active":"2024-04-25T00:13:23.589980+0000","last_became_peered":"2024-04-25T00:13:23.589980+0000","last_unstale":"2024-04-25T00:15:45.148616+0000","last_undegraded":"2024-04-25T00:15:45.148616+0000","last_fullsized":"2024-04-25T00:15:45.148616+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:18.234048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.313143+0000","last_change":"2024-04-25T00:13:18.526353+0000","last_active":"2024-04-25T00:15:45.313143+0000","last_peered":"2024-04-25T00:15:45.313143+0000","last_clean":"2024-04-25T00:15:45.313143+0000","last_became_active":"2024-04-25T00:13:18.525951+0000","last_became_peered":"2024-04-25T00:13:18.525951+0000","last_unstale":"2024-04-25T00:15:45.313143+0000","last_undegraded":"2024-04-25T00:15:45.313143+0000","last_fullsized":"2024-04-25T00:15:45.313143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:10:29.597797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115219+0000","last_change":"2024-04-25T00:13:19.425486+0000","last_active":"2024-04-25T00:15:45.115219+0000","last_peered":"2024-04-25T00:15:45.115219+0000","last_clean":"2024-04-25T00:15:45.115219+0000","last_became_active":"2024-04-25T00:13:19.425087+0000","last_became_peered":"2024-04-25T00:13:19.425087+0000","last_unstale":"2024-04-25T00:15:45.115219+0000","last_undegraded":"2024-04-25T00:15:45.115219+0000","last_fullsized":"2024-04-25T00:15:45.115219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:00.216683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041314+0000","last_change":"2024-04-25T00:13:23.579747+0000","last_active":"2024-04-25T00:15:45.041314+0000","last_peered":"2024-04-25T00:15:45.041314+0000","last_clean":"2024-04-25T00:15:45.041314+0000","last_became_active":"2024-04-25T00:13:23.579346+0000","last_became_peered":"2024-04-25T00:13:23.579346+0000","last_unstale":"2024-04-25T00:15:45.041314+0000","last_undegraded":"2024-04-25T00:15:45.041314+0000","last_fullsized":"2024-04-25T00:15:45.041314+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:10:18.969942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.159127+0000","last_change":"2024-04-25T00:13:18.524707+0000","last_active":"2024-04-25T00:15:45.159127+0000","last_peered":"2024-04-25T00:15:45.159127+0000","last_clean":"2024-04-25T00:15:45.159127+0000","last_became_active":"2024-04-25T00:13:18.524352+0000","last_became_peered":"2024-04-25T00:13:18.524352+0000","last_unstale":"2024-04-25T00:15:45.159127+0000","last_undegraded":"2024-04-25T00:15:45.159127+0000","last_fullsized":"2024-04-25T00:15:45.159127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:29.992843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309736+0000","last_change":"2024-04-25T00:13:19.421845+0000","last_active":"2024-04-25T00:15:45.309736+0000","last_peered":"2024-04-25T00:15:45.309736+0000","last_clean":"2024-04-25T00:15:45.309736+0000","last_became_active":"2024-04-25T00:13:19.421458+0000","last_became_peered":"2024-04-25T00:13:19.421458+0000","last_unstale":"2024-04-25T00:15:45.309736+0000","last_undegraded":"2024-04-25T00:15:45.309736+0000","last_fullsized":"2024-04-25T00:15:45.309736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:09.286604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.148941+0000","last_change":"2024-04-25T00:13:23.583721+0000","last_active":"2024-04-25T00:15:45.148941+0000","last_peered":"2024-04-25T00:15:45.148941+0000","last_clean":"2024-04-25T00:15:45.148941+0000","last_became_active":"2024-04-25T00:13:23.583340+0000","last_became_peered":"2024-04-25T00:13:23.583340+0000","last_unstale":"2024-04-25T00:15:45.148941+0000","last_undegraded":"2024-04-25T00:15:45.148941+0000","last_fullsized":"2024-04-25T00:15:45.148941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:26.446468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":326,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.731678+0000","last_change":"2024-04-25T00:13:18.520653+0000","last_active":"2024-04-25T00:15:45.731678+0000","last_peered":"2024-04-25T00:15:45.731678+0000","last_clean":"2024-04-25T00:15:45.731678+0000","last_became_active":"2024-04-25T00:13:18.520278+0000","last_became_peered":"2024-04-25T00:13:18.520278+0000","last_unstale":"2024-04-25T00:15:45.731678+0000","last_undegraded":"2024-04-25T00:15:45.731678+0000","last_fullsized":"2024-04-25T00:15:45.731678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:58.311179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66804,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":252,"num_read_kb":191,"num_write":42,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.152025+0000","last_change":"2024-04-25T00:13:19.421599+0000","last_active":"2024-04-25T00:15:45.152025+0000","last_peered":"2024-04-25T00:15:45.152025+0000","last_clean":"2024-04-25T00:15:45.152025+0000","last_became_active":"2024-04-25T00:13:19.421107+0000","last_became_peered":"2024-04-25T00:13:19.421107+0000","last_unstale":"2024-04-25T00:15:45.152025+0000","last_undegraded":"2024-04-25T00:15:45.152025+0000","last_fullsized":"2024-04-25T00:15:45.152025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:59.577987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.159398+0000","last_change":"2024-04-25T00:13:23.580538+0000","last_active":"2024-04-25T00:15:45.159398+0000","last_peered":"2024-04-25T00:15:45.159398+0000","last_clean":"2024-04-25T00:15:45.159398+0000","last_became_active":"2024-04-25T00:13:23.580095+0000","last_became_peered":"2024-04-25T00:13:23.580095+0000","last_unstale":"2024-04-25T00:15:45.159398+0000","last_undegraded":"2024-04-25T00:15:45.159398+0000","last_fullsized":"2024-04-25T00:15:45.159398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:04.189530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041608+0000","last_change":"2024-04-25T00:13:18.522501+0000","last_active":"2024-04-25T00:15:45.041608+0000","last_peered":"2024-04-25T00:15:45.041608+0000","last_clean":"2024-04-25T00:15:45.041608+0000","last_became_active":"2024-04-25T00:13:18.522160+0000","last_became_peered":"2024-04-25T00:13:18.522160+0000","last_unstale":"2024-04-25T00:15:45.041608+0000","last_undegraded":"2024-04-25T00:15:45.041608+0000","last_fullsized":"2024-04-25T00:15:45.041608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:31.876731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314208+0000","last_change":"2024-04-25T00:13:19.417957+0000","last_active":"2024-04-25T00:15:45.314208+0000","last_peered":"2024-04-25T00:15:45.314208+0000","last_clean":"2024-04-25T00:15:45.314208+0000","last_became_active":"2024-04-25T00:13:19.417600+0000","last_became_peered":"2024-04-25T00:13:19.417600+0000","last_unstale":"2024-04-25T00:15:45.314208+0000","last_undegraded":"2024-04-25T00:15:45.314208+0000","last_fullsized":"2024-04-25T00:15:45.314208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:24.600965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293826+0000","last_change":"2024-04-25T00:13:23.582279+0000","last_active":"2024-04-25T00:15:45.293826+0000","last_peered":"2024-04-25T00:15:45.293826+0000","last_clean":"2024-04-25T00:15:45.293826+0000","last_became_active":"2024-04-25T00:13:23.581766+0000","last_became_peered":"2024-04-25T00:13:23.581766+0000","last_unstale":"2024-04-25T00:15:45.293826+0000","last_undegraded":"2024-04-25T00:15:45.293826+0000","last_fullsized":"2024-04-25T00:15:45.293826+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:36.115870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158999+0000","last_change":"2024-04-25T00:13:18.527443+0000","last_active":"2024-04-25T00:15:45.158999+0000","last_peered":"2024-04-25T00:15:45.158999+0000","last_clean":"2024-04-25T00:15:45.158999+0000","last_became_active":"2024-04-25T00:13:18.526919+0000","last_became_peered":"2024-04-25T00:13:18.526919+0000","last_unstale":"2024-04-25T00:15:45.158999+0000","last_undegraded":"2024-04-25T00:15:45.158999+0000","last_fullsized":"2024-04-25T00:15:45.158999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:32.737580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312419+0000","last_change":"2024-04-25T00:13:19.425947+0000","last_active":"2024-04-25T00:15:45.312419+0000","last_peered":"2024-04-25T00:15:45.312419+0000","last_clean":"2024-04-25T00:15:45.312419+0000","last_became_active":"2024-04-25T00:13:19.425613+0000","last_became_peered":"2024-04-25T00:13:19.425613+0000","last_unstale":"2024-04-25T00:15:45.312419+0000","last_undegraded":"2024-04-25T00:15:45.312419+0000","last_fullsized":"2024-04-25T00:15:45.312419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:28:34.729375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149578+0000","last_change":"2024-04-25T00:13:23.587794+0000","last_active":"2024-04-25T00:15:45.149578+0000","last_peered":"2024-04-25T00:15:45.149578+0000","last_clean":"2024-04-25T00:15:45.149578+0000","last_became_active":"2024-04-25T00:13:23.587334+0000","last_became_peered":"2024-04-25T00:13:23.587334+0000","last_unstale":"2024-04-25T00:15:45.149578+0000","last_undegraded":"2024-04-25T00:15:45.149578+0000","last_fullsized":"2024-04-25T00:15:45.149578+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:58.460439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":166,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.731310+0000","last_change":"2024-04-25T00:13:18.521701+0000","last_active":"2024-04-25T00:15:45.731310+0000","last_peered":"2024-04-25T00:15:45.731310+0000","last_clean":"2024-04-25T00:15:45.731310+0000","last_became_active":"2024-04-25T00:13:18.521360+0000","last_became_peered":"2024-04-25T00:13:18.521360+0000","last_unstale":"2024-04-25T00:15:45.731310+0000","last_undegraded":"2024-04-25T00:15:45.731310+0000","last_fullsized":"2024-04-25T00:15:45.731310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:18:11.567426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":126,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.311990+0000","last_change":"2024-04-25T00:13:19.424162+0000","last_active":"2024-04-25T00:15:45.311990+0000","last_peered":"2024-04-25T00:15:45.311990+0000","last_clean":"2024-04-25T00:15:45.311990+0000","last_became_active":"2024-04-25T00:13:19.423845+0000","last_became_peered":"2024-04-25T00:13:19.423845+0000","last_unstale":"2024-04-25T00:15:45.311990+0000","last_undegraded":"2024-04-25T00:15:45.311990+0000","last_fullsized":"2024-04-25T00:15:45.311990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:33.887577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314672+0000","last_change":"2024-04-25T00:13:23.582102+0000","last_active":"2024-04-25T00:15:45.314672+0000","last_peered":"2024-04-25T00:15:45.314672+0000","last_clean":"2024-04-25T00:15:45.314672+0000","last_became_active":"2024-04-25T00:13:23.581679+0000","last_became_peered":"2024-04-25T00:13:23.581679+0000","last_unstale":"2024-04-25T00:15:45.314672+0000","last_undegraded":"2024-04-25T00:15:45.314672+0000","last_fullsized":"2024-04-25T00:15:45.314672+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:05:39.518634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151589+0000","last_change":"2024-04-25T00:13:18.518285+0000","last_active":"2024-04-25T00:15:45.151589+0000","last_peered":"2024-04-25T00:15:45.151589+0000","last_clean":"2024-04-25T00:15:45.151589+0000","last_became_active":"2024-04-25T00:13:18.517826+0000","last_became_peered":"2024-04-25T00:13:18.517826+0000","last_unstale":"2024-04-25T00:15:45.151589+0000","last_undegraded":"2024-04-25T00:15:45.151589+0000","last_fullsized":"2024-04-25T00:15:45.151589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:12.901225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158812+0000","last_change":"2024-04-25T00:14:11.268444+0000","last_active":"2024-04-25T00:15:45.158812+0000","last_peered":"2024-04-25T00:15:45.158812+0000","last_clean":"2024-04-25T00:15:45.158812+0000","last_became_active":"2024-04-25T00:14:11.267934+0000","last_became_peered":"2024-04-25T00:14:11.267934+0000","last_unstale":"2024-04-25T00:15:45.158812+0000","last_undegraded":"2024-04-25T00:15:45.158812+0000","last_fullsized":"2024-04-25T00:15:45.158812+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:12:16.209890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158730+0000","last_change":"2024-04-25T00:13:23.578445+0000","last_active":"2024-04-25T00:15:45.158730+0000","last_peered":"2024-04-25T00:15:45.158730+0000","last_clean":"2024-04-25T00:15:45.158730+0000","last_became_active":"2024-04-25T00:13:23.578048+0000","last_became_peered":"2024-04-25T00:13:23.578048+0000","last_unstale":"2024-04-25T00:15:45.158730+0000","last_undegraded":"2024-04-25T00:15:45.158730+0000","last_fullsized":"2024-04-25T00:15:45.158730+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:55.072460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151953+0000","last_change":"2024-04-25T00:13:18.515939+0000","last_active":"2024-04-25T00:15:45.151953+0000","last_peered":"2024-04-25T00:15:45.151953+0000","last_clean":"2024-04-25T00:15:45.151953+0000","last_became_active":"2024-04-25T00:13:18.515597+0000","last_became_peered":"2024-04-25T00:13:18.515597+0000","last_unstale":"2024-04-25T00:15:45.151953+0000","last_undegraded":"2024-04-25T00:15:45.151953+0000","last_fullsized":"2024-04-25T00:15:45.151953+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:03:30.266404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.294003+0000","last_change":"2024-04-25T00:14:11.269143+0000","last_active":"2024-04-25T00:15:45.294003+0000","last_peered":"2024-04-25T00:15:45.294003+0000","last_clean":"2024-04-25T00:15:45.294003+0000","last_became_active":"2024-04-25T00:14:11.268736+0000","last_became_peered":"2024-04-25T00:14:11.268736+0000","last_unstale":"2024-04-25T00:15:45.294003+0000","last_undegraded":"2024-04-25T00:15:45.294003+0000","last_fullsized":"2024-04-25T00:15:45.294003+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:19.559974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151733+0000","last_change":"2024-04-25T00:13:23.862576+0000","last_active":"2024-04-25T00:15:45.151733+0000","last_peered":"2024-04-25T00:15:45.151733+0000","last_clean":"2024-04-25T00:15:45.151733+0000","last_became_active":"2024-04-25T00:13:23.862024+0000","last_became_peered":"2024-04-25T00:13:23.862024+0000","last_unstale":"2024-04-25T00:15:45.151733+0000","last_undegraded":"2024-04-25T00:15:45.151733+0000","last_fullsized":"2024-04-25T00:15:45.151733+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:17.249680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149273+0000","last_change":"2024-04-25T00:13:18.522503+0000","last_active":"2024-04-25T00:15:45.149273+0000","last_peered":"2024-04-25T00:15:45.149273+0000","last_clean":"2024-04-25T00:15:45.149273+0000","last_became_active":"2024-04-25T00:13:18.522126+0000","last_became_peered":"2024-04-25T00:13:18.522126+0000","last_unstale":"2024-04-25T00:15:45.149273+0000","last_undegraded":"2024-04-25T00:15:45.149273+0000","last_fullsized":"2024-04-25T00:15:45.149273+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:47:57.530098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.162498+0000","last_change":"2024-04-25T00:13:19.416862+0000","last_active":"2024-04-25T00:15:45.162498+0000","last_peered":"2024-04-25T00:15:45.162498+0000","last_clean":"2024-04-25T00:15:45.162498+0000","last_became_active":"2024-04-25T00:13:19.416375+0000","last_became_peered":"2024-04-25T00:13:19.416375+0000","last_unstale":"2024-04-25T00:15:45.162498+0000","last_undegraded":"2024-04-25T00:15:45.162498+0000","last_fullsized":"2024-04-25T00:15:45.162498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:43:32.031623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314336+0000","last_change":"2024-04-25T00:13:23.582387+0000","last_active":"2024-04-25T00:15:45.314336+0000","last_peered":"2024-04-25T00:15:45.314336+0000","last_clean":"2024-04-25T00:15:45.314336+0000","last_became_active":"2024-04-25T00:13:23.581984+0000","last_became_peered":"2024-04-25T00:13:23.581984+0000","last_unstale":"2024-04-25T00:15:45.314336+0000","last_undegraded":"2024-04-25T00:15:45.314336+0000","last_fullsized":"2024-04-25T00:15:45.314336+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:07:16.388293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"94'7","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.748717+0000","last_change":"2024-04-25T00:13:18.524121+0000","last_active":"2024-04-25T00:15:45.748717+0000","last_peered":"2024-04-25T00:15:45.748717+0000","last_clean":"2024-04-25T00:15:45.748717+0000","last_became_active":"2024-04-25T00:13:18.523731+0000","last_became_peered":"2024-04-25T00:13:18.523731+0000","last_unstale":"2024-04-25T00:15:45.748717+0000","last_undegraded":"2024-04-25T00:15:45.748717+0000","last_fullsized":"2024-04-25T00:15:45.748717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:22.105952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1287,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.162127+0000","last_change":"2024-04-25T00:13:19.424753+0000","last_active":"2024-04-25T00:15:45.162127+0000","last_peered":"2024-04-25T00:15:45.162127+0000","last_clean":"2024-04-25T00:15:45.162127+0000","last_became_active":"2024-04-25T00:13:19.423214+0000","last_became_peered":"2024-04-25T00:13:19.423214+0000","last_unstale":"2024-04-25T00:15:45.162127+0000","last_undegraded":"2024-04-25T00:15:45.162127+0000","last_fullsized":"2024-04-25T00:15:45.162127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:41.627442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314459+0000","last_change":"2024-04-25T00:14:11.272431+0000","last_active":"2024-04-25T00:15:45.314459+0000","last_peered":"2024-04-25T00:15:45.314459+0000","last_clean":"2024-04-25T00:15:45.314459+0000","last_became_active":"2024-04-25T00:14:11.271997+0000","last_became_peered":"2024-04-25T00:14:11.271997+0000","last_unstale":"2024-04-25T00:15:45.314459+0000","last_undegraded":"2024-04-25T00:15:45.314459+0000","last_fullsized":"2024-04-25T00:15:45.314459+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:18.768009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158863+0000","last_change":"2024-04-25T00:13:19.420617+0000","last_active":"2024-04-25T00:15:45.158863+0000","last_peered":"2024-04-25T00:15:45.158863+0000","last_clean":"2024-04-25T00:15:45.158863+0000","last_became_active":"2024-04-25T00:13:19.420274+0000","last_became_peered":"2024-04-25T00:13:19.420274+0000","last_unstale":"2024-04-25T00:15:45.158863+0000","last_undegraded":"2024-04-25T00:15:45.158863+0000","last_fullsized":"2024-04-25T00:15:45.158863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:07.515196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314314+0000","last_change":"2024-04-25T00:13:18.529753+0000","last_active":"2024-04-25T00:15:45.314314+0000","last_peered":"2024-04-25T00:15:45.314314+0000","last_clean":"2024-04-25T00:15:45.314314+0000","last_became_active":"2024-04-25T00:13:18.525521+0000","last_became_peered":"2024-04-25T00:13:18.525521+0000","last_unstale":"2024-04-25T00:15:45.314314+0000","last_undegraded":"2024-04-25T00:15:45.314314+0000","last_fullsized":"2024-04-25T00:15:45.314314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:17.044367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312649+0000","last_change":"2024-04-25T00:13:23.589613+0000","last_active":"2024-04-25T00:15:45.312649+0000","last_peered":"2024-04-25T00:15:45.312649+0000","last_clean":"2024-04-25T00:15:45.312649+0000","last_became_active":"2024-04-25T00:13:23.589323+0000","last_became_peered":"2024-04-25T00:13:23.589323+0000","last_unstale":"2024-04-25T00:15:45.312649+0000","last_undegraded":"2024-04-25T00:15:45.312649+0000","last_fullsized":"2024-04-25T00:15:45.312649+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:34:42.272706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149072+0000","last_change":"2024-04-25T00:15:11.431091+0000","last_active":"2024-04-25T00:15:45.149072+0000","last_peered":"2024-04-25T00:15:45.149072+0000","last_clean":"2024-04-25T00:15:45.149072+0000","last_became_active":"2024-04-25T00:15:11.430797+0000","last_became_peered":"2024-04-25T00:15:11.430797+0000","last_unstale":"2024-04-25T00:15:45.149072+0000","last_undegraded":"2024-04-25T00:15:45.149072+0000","last_fullsized":"2024-04-25T00:15:45.149072+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:26:30.121912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041204+0000","last_change":"2024-04-25T00:13:18.514560+0000","last_active":"2024-04-25T00:15:45.041204+0000","last_peered":"2024-04-25T00:15:45.041204+0000","last_clean":"2024-04-25T00:15:45.041204+0000","last_became_active":"2024-04-25T00:13:18.514034+0000","last_became_peered":"2024-04-25T00:13:18.514034+0000","last_unstale":"2024-04-25T00:15:45.041204+0000","last_undegraded":"2024-04-25T00:15:45.041204+0000","last_fullsized":"2024-04-25T00:15:45.041204+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:35:18.110206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":365,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.311817+0000","last_change":"2024-04-25T00:12:01.540807+0000","last_active":"2024-04-25T00:15:45.311817+0000","last_peered":"2024-04-25T00:15:45.311817+0000","last_clean":"2024-04-25T00:15:45.311817+0000","last_became_active":"2024-04-25T00:12:01.540272+0000","last_became_peered":"2024-04-25T00:12:01.540272+0000","last_unstale":"2024-04-25T00:15:45.311817+0000","last_undegraded":"2024-04-25T00:15:45.311817+0000","last_fullsized":"2024-04-25T00:15:45.311817+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:08:34.007429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:08:34.007429+0000","last_clean_scrub_stamp":"2024-04-25T00:08:34.007429+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:50:14.642604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.159144+0000","last_change":"2024-04-25T00:13:23.862493+0000","last_active":"2024-04-25T00:15:45.159144+0000","last_peered":"2024-04-25T00:15:45.159144+0000","last_clean":"2024-04-25T00:15:45.159144+0000","last_became_active":"2024-04-25T00:13:23.861972+0000","last_became_peered":"2024-04-25T00:13:23.861972+0000","last_unstale":"2024-04-25T00:15:45.159144+0000","last_undegraded":"2024-04-25T00:15:45.159144+0000","last_fullsized":"2024-04-25T00:15:45.159144+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:49:13.507357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314067+0000","last_change":"2024-04-25T00:13:19.421920+0000","last_active":"2024-04-25T00:15:45.314067+0000","last_peered":"2024-04-25T00:15:45.314067+0000","last_clean":"2024-04-25T00:15:45.314067+0000","last_became_active":"2024-04-25T00:13:19.421585+0000","last_became_peered":"2024-04-25T00:13:19.421585+0000","last_unstale":"2024-04-25T00:15:45.314067+0000","last_undegraded":"2024-04-25T00:15:45.314067+0000","last_fullsized":"2024-04-25T00:15:45.314067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:53:24.937520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.311886+0000","last_change":"2024-04-25T00:13:18.527062+0000","last_active":"2024-04-25T00:15:45.311886+0000","last_peered":"2024-04-25T00:15:45.311886+0000","last_clean":"2024-04-25T00:15:45.311886+0000","last_became_active":"2024-04-25T00:13:18.526338+0000","last_became_peered":"2024-04-25T00:13:18.526338+0000","last_unstale":"2024-04-25T00:15:45.311886+0000","last_undegraded":"2024-04-25T00:15:45.311886+0000","last_fullsized":"2024-04-25T00:15:45.311886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:21.756728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293792+0000","last_change":"2024-04-25T00:13:23.575162+0000","last_active":"2024-04-25T00:15:45.293792+0000","last_peered":"2024-04-25T00:15:45.293792+0000","last_clean":"2024-04-25T00:15:45.293792+0000","last_became_active":"2024-04-25T00:13:23.574722+0000","last_became_peered":"2024-04-25T00:13:23.574722+0000","last_unstale":"2024-04-25T00:15:45.293792+0000","last_undegraded":"2024-04-25T00:15:45.293792+0000","last_fullsized":"2024-04-25T00:15:45.293792+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:52.958539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.152034+0000","last_change":"2024-04-25T00:13:18.523019+0000","last_active":"2024-04-25T00:15:45.152034+0000","last_peered":"2024-04-25T00:15:45.152034+0000","last_clean":"2024-04-25T00:15:45.152034+0000","last_became_active":"2024-04-25T00:13:18.522677+0000","last_became_peered":"2024-04-25T00:13:18.522677+0000","last_unstale":"2024-04-25T00:15:45.152034+0000","last_undegraded":"2024-04-25T00:15:45.152034+0000","last_fullsized":"2024-04-25T00:15:45.152034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:14.470154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314556+0000","last_change":"2024-04-25T00:13:19.417672+0000","last_active":"2024-04-25T00:15:45.314556+0000","last_peered":"2024-04-25T00:15:45.314556+0000","last_clean":"2024-04-25T00:15:45.314556+0000","last_became_active":"2024-04-25T00:13:19.417323+0000","last_became_peered":"2024-04-25T00:13:19.417323+0000","last_unstale":"2024-04-25T00:15:45.314556+0000","last_undegraded":"2024-04-25T00:15:45.314556+0000","last_fullsized":"2024-04-25T00:15:45.314556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:53.734784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.039721+0000","last_change":"2024-04-25T00:13:23.862320+0000","last_active":"2024-04-25T00:15:45.039721+0000","last_peered":"2024-04-25T00:15:45.039721+0000","last_clean":"2024-04-25T00:15:45.039721+0000","last_became_active":"2024-04-25T00:13:23.861895+0000","last_became_peered":"2024-04-25T00:13:23.861895+0000","last_unstale":"2024-04-25T00:15:45.039721+0000","last_undegraded":"2024-04-25T00:15:45.039721+0000","last_fullsized":"2024-04-25T00:15:45.039721+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:16:21.324727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'49","reported_seq":106,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.267428+0000","last_change":"2024-04-25T00:13:18.520989+0000","last_active":"2024-04-25T00:15:45.267428+0000","last_peered":"2024-04-25T00:15:45.267428+0000","last_clean":"2024-04-25T00:15:45.267428+0000","last_became_active":"2024-04-25T00:13:18.520566+0000","last_became_peered":"2024-04-25T00:13:18.520566+0000","last_unstale":"2024-04-25T00:15:45.267428+0000","last_undegraded":"2024-04-25T00:15:45.267428+0000","last_fullsized":"2024-04-25T00:15:45.267428+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:22:32.532352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":86,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040218+0000","last_change":"2024-04-25T00:13:19.422741+0000","last_active":"2024-04-25T00:15:45.040218+0000","last_peered":"2024-04-25T00:15:45.040218+0000","last_clean":"2024-04-25T00:15:45.040218+0000","last_became_active":"2024-04-25T00:13:19.422379+0000","last_became_peered":"2024-04-25T00:13:19.422379+0000","last_unstale":"2024-04-25T00:15:45.040218+0000","last_undegraded":"2024-04-25T00:15:45.040218+0000","last_fullsized":"2024-04-25T00:15:45.040218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:41.337641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041168+0000","last_change":"2024-04-25T00:13:23.577888+0000","last_active":"2024-04-25T00:15:45.041168+0000","last_peered":"2024-04-25T00:15:45.041168+0000","last_clean":"2024-04-25T00:15:45.041168+0000","last_became_active":"2024-04-25T00:13:23.577440+0000","last_became_peered":"2024-04-25T00:13:23.577440+0000","last_unstale":"2024-04-25T00:15:45.041168+0000","last_undegraded":"2024-04-25T00:15:45.041168+0000","last_fullsized":"2024-04-25T00:15:45.041168+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:03.760277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041651+0000","last_change":"2024-04-25T00:13:18.527996+0000","last_active":"2024-04-25T00:15:45.041651+0000","last_peered":"2024-04-25T00:15:45.041651+0000","last_clean":"2024-04-25T00:15:45.041651+0000","last_became_active":"2024-04-25T00:13:18.527636+0000","last_became_peered":"2024-04-25T00:13:18.527636+0000","last_unstale":"2024-04-25T00:15:45.041651+0000","last_undegraded":"2024-04-25T00:15:45.041651+0000","last_fullsized":"2024-04-25T00:15:45.041651+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:52.137108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.115077+0000","last_change":"2024-04-25T00:13:19.425538+0000","last_active":"2024-04-25T00:15:45.115077+0000","last_peered":"2024-04-25T00:15:45.115077+0000","last_clean":"2024-04-25T00:15:45.115077+0000","last_became_active":"2024-04-25T00:13:19.420790+0000","last_became_peered":"2024-04-25T00:13:19.420790+0000","last_unstale":"2024-04-25T00:15:45.115077+0000","last_undegraded":"2024-04-25T00:15:45.115077+0000","last_fullsized":"2024-04-25T00:15:45.115077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:04.319759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041457+0000","last_change":"2024-04-25T00:13:23.863187+0000","last_active":"2024-04-25T00:15:45.041457+0000","last_peered":"2024-04-25T00:15:45.041457+0000","last_clean":"2024-04-25T00:15:45.041457+0000","last_became_active":"2024-04-25T00:13:23.862206+0000","last_became_peered":"2024-04-25T00:13:23.862206+0000","last_unstale":"2024-04-25T00:15:45.041457+0000","last_undegraded":"2024-04-25T00:15:45.041457+0000","last_fullsized":"2024-04-25T00:15:45.041457+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:57:17.395007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309338+0000","last_change":"2024-04-25T00:13:18.529224+0000","last_active":"2024-04-25T00:15:45.309338+0000","last_peered":"2024-04-25T00:15:45.309338+0000","last_clean":"2024-04-25T00:15:45.309338+0000","last_became_active":"2024-04-25T00:13:18.528908+0000","last_became_peered":"2024-04-25T00:13:18.528908+0000","last_unstale":"2024-04-25T00:15:45.309338+0000","last_undegraded":"2024-04-25T00:15:45.309338+0000","last_fullsized":"2024-04-25T00:15:45.309338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:27.035726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.151539+0000","last_change":"2024-04-25T00:13:19.422223+0000","last_active":"2024-04-25T00:15:45.151539+0000","last_peered":"2024-04-25T00:15:45.151539+0000","last_clean":"2024-04-25T00:15:45.151539+0000","last_became_active":"2024-04-25T00:13:19.421524+0000","last_became_peered":"2024-04-25T00:13:19.421524+0000","last_unstale":"2024-04-25T00:15:45.151539+0000","last_undegraded":"2024-04-25T00:15:45.151539+0000","last_fullsized":"2024-04-25T00:15:45.151539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:17:01.849503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.149842+0000","last_change":"2024-04-25T00:13:23.583015+0000","last_active":"2024-04-25T00:15:45.149842+0000","last_peered":"2024-04-25T00:15:45.149842+0000","last_clean":"2024-04-25T00:15:45.149842+0000","last_became_active":"2024-04-25T00:13:23.582612+0000","last_became_peered":"2024-04-25T00:13:23.582612+0000","last_unstale":"2024-04-25T00:15:45.149842+0000","last_undegraded":"2024-04-25T00:15:45.149842+0000","last_fullsized":"2024-04-25T00:15:45.149842+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:36.872281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.039860+0000","last_change":"2024-04-25T00:13:18.530027+0000","last_active":"2024-04-25T00:15:45.039860+0000","last_peered":"2024-04-25T00:15:45.039860+0000","last_clean":"2024-04-25T00:15:45.039860+0000","last_became_active":"2024-04-25T00:13:18.529717+0000","last_became_peered":"2024-04-25T00:13:18.529717+0000","last_unstale":"2024-04-25T00:15:45.039860+0000","last_undegraded":"2024-04-25T00:15:45.039860+0000","last_fullsized":"2024-04-25T00:15:45.039860+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:23:31.020637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293701+0000","last_change":"2024-04-25T00:13:19.428650+0000","last_active":"2024-04-25T00:15:45.293701+0000","last_peered":"2024-04-25T00:15:45.293701+0000","last_clean":"2024-04-25T00:15:45.293701+0000","last_became_active":"2024-04-25T00:13:19.428156+0000","last_became_peered":"2024-04-25T00:13:19.428156+0000","last_unstale":"2024-04-25T00:15:45.293701+0000","last_undegraded":"2024-04-25T00:15:45.293701+0000","last_fullsized":"2024-04-25T00:15:45.293701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:43.094529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041458+0000","last_change":"2024-04-25T00:13:23.579125+0000","last_active":"2024-04-25T00:15:45.041458+0000","last_peered":"2024-04-25T00:15:45.041458+0000","last_clean":"2024-04-25T00:15:45.041458+0000","last_became_active":"2024-04-25T00:13:23.578728+0000","last_became_peered":"2024-04-25T00:13:23.578728+0000","last_unstale":"2024-04-25T00:15:45.041458+0000","last_undegraded":"2024-04-25T00:15:45.041458+0000","last_fullsized":"2024-04-25T00:15:45.041458+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:16:16.834399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'4","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293944+0000","last_change":"2024-04-25T00:14:11.278099+0000","last_active":"2024-04-25T00:15:45.293944+0000","last_peered":"2024-04-25T00:15:45.293944+0000","last_clean":"2024-04-25T00:15:45.293944+0000","last_became_active":"2024-04-25T00:14:11.269977+0000","last_became_peered":"2024-04-25T00:14:11.269977+0000","last_unstale":"2024-04-25T00:15:45.293944+0000","last_undegraded":"2024-04-25T00:15:45.293944+0000","last_fullsized":"2024-04-25T00:15:45.293944+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:48:59.261854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040032+0000","last_change":"2024-04-25T00:13:19.414848+0000","last_active":"2024-04-25T00:15:45.040032+0000","last_peered":"2024-04-25T00:15:45.040032+0000","last_clean":"2024-04-25T00:15:45.040032+0000","last_became_active":"2024-04-25T00:13:19.414370+0000","last_became_peered":"2024-04-25T00:13:19.414370+0000","last_unstale":"2024-04-25T00:15:45.040032+0000","last_undegraded":"2024-04-25T00:15:45.040032+0000","last_fullsized":"2024-04-25T00:15:45.040032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:41:08.779171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041407+0000","last_change":"2024-04-25T00:13:23.862577+0000","last_active":"2024-04-25T00:15:45.041407+0000","last_peered":"2024-04-25T00:15:45.041407+0000","last_clean":"2024-04-25T00:15:45.041407+0000","last_became_active":"2024-04-25T00:13:23.862098+0000","last_became_peered":"2024-04-25T00:13:23.862098+0000","last_unstale":"2024-04-25T00:15:45.041407+0000","last_undegraded":"2024-04-25T00:15:45.041407+0000","last_fullsized":"2024-04-25T00:15:45.041407+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:08:00.344041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293746+0000","last_change":"2024-04-25T00:14:11.274790+0000","last_active":"2024-04-25T00:15:45.293746+0000","last_peered":"2024-04-25T00:15:45.293746+0000","last_clean":"2024-04-25T00:15:45.293746+0000","last_became_active":"2024-04-25T00:14:11.267132+0000","last_became_peered":"2024-04-25T00:14:11.267132+0000","last_unstale":"2024-04-25T00:15:45.293746+0000","last_undegraded":"2024-04-25T00:15:45.293746+0000","last_fullsized":"2024-04-25T00:15:45.293746+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:58.302788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312161+0000","last_change":"2024-04-25T00:13:19.427992+0000","last_active":"2024-04-25T00:15:45.312161+0000","last_peered":"2024-04-25T00:15:45.312161+0000","last_clean":"2024-04-25T00:15:45.312161+0000","last_became_active":"2024-04-25T00:13:19.427665+0000","last_became_peered":"2024-04-25T00:13:19.427665+0000","last_unstale":"2024-04-25T00:15:45.312161+0000","last_undegraded":"2024-04-25T00:15:45.312161+0000","last_fullsized":"2024-04-25T00:15:45.312161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:12:31.582185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309231+0000","last_change":"2024-04-25T00:13:23.582430+0000","last_active":"2024-04-25T00:15:45.309231+0000","last_peered":"2024-04-25T00:15:45.309231+0000","last_clean":"2024-04-25T00:15:45.309231+0000","last_became_active":"2024-04-25T00:13:23.582008+0000","last_became_peered":"2024-04-25T00:13:23.582008+0000","last_unstale":"2024-04-25T00:15:45.309231+0000","last_undegraded":"2024-04-25T00:15:45.309231+0000","last_fullsized":"2024-04-25T00:15:45.309231+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:11.068720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"94'7","reported_seq":163,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.754413+0000","last_change":"2024-04-25T00:13:18.528030+0000","last_active":"2024-04-25T00:15:45.754413+0000","last_peered":"2024-04-25T00:15:45.754413+0000","last_clean":"2024-04-25T00:15:45.754413+0000","last_became_active":"2024-04-25T00:13:18.527636+0000","last_became_peered":"2024-04-25T00:13:18.527636+0000","last_unstale":"2024-04-25T00:15:45.754413+0000","last_undegraded":"2024-04-25T00:15:45.754413+0000","last_fullsized":"2024-04-25T00:15:45.754413+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:36.311810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1952,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312965+0000","last_change":"2024-04-25T00:13:19.428181+0000","last_active":"2024-04-25T00:15:45.312965+0000","last_peered":"2024-04-25T00:15:45.312965+0000","last_clean":"2024-04-25T00:15:45.312965+0000","last_became_active":"2024-04-25T00:13:19.427785+0000","last_became_peered":"2024-04-25T00:13:19.427785+0000","last_unstale":"2024-04-25T00:15:45.312965+0000","last_undegraded":"2024-04-25T00:15:45.312965+0000","last_fullsized":"2024-04-25T00:15:45.312965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:31.745679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.114716+0000","last_change":"2024-04-25T00:14:11.271507+0000","last_active":"2024-04-25T00:15:45.114716+0000","last_peered":"2024-04-25T00:15:45.114716+0000","last_clean":"2024-04-25T00:15:45.114716+0000","last_became_active":"2024-04-25T00:14:11.270452+0000","last_became_peered":"2024-04-25T00:14:11.270452+0000","last_unstale":"2024-04-25T00:15:45.114716+0000","last_undegraded":"2024-04-25T00:15:45.114716+0000","last_fullsized":"2024-04-25T00:15:45.114716+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:19.955242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'41","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.262583+0000","last_change":"2024-04-25T00:13:18.529622+0000","last_active":"2024-04-25T00:15:45.262583+0000","last_peered":"2024-04-25T00:15:45.262583+0000","last_clean":"2024-04-25T00:15:45.262583+0000","last_became_active":"2024-04-25T00:13:18.529305+0000","last_became_peered":"2024-04-25T00:13:18.529305+0000","last_unstale":"2024-04-25T00:15:45.262583+0000","last_undegraded":"2024-04-25T00:15:45.262583+0000","last_fullsized":"2024-04-25T00:15:45.262583+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:10:43.334712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.312206+0000","last_change":"2024-04-25T00:13:19.421783+0000","last_active":"2024-04-25T00:15:45.312206+0000","last_peered":"2024-04-25T00:15:45.312206+0000","last_clean":"2024-04-25T00:15:45.312206+0000","last_became_active":"2024-04-25T00:13:19.421447+0000","last_became_peered":"2024-04-25T00:13:19.421447+0000","last_unstale":"2024-04-25T00:15:45.312206+0000","last_undegraded":"2024-04-25T00:15:45.312206+0000","last_fullsized":"2024-04-25T00:15:45.312206+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:16.107220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158542+0000","last_change":"2024-04-25T00:13:23.578252+0000","last_active":"2024-04-25T00:15:45.158542+0000","last_peered":"2024-04-25T00:15:45.158542+0000","last_clean":"2024-04-25T00:15:45.158542+0000","last_became_active":"2024-04-25T00:13:23.577779+0000","last_became_peered":"2024-04-25T00:13:23.577779+0000","last_unstale":"2024-04-25T00:15:45.158542+0000","last_undegraded":"2024-04-25T00:15:45.158542+0000","last_fullsized":"2024-04-25T00:15:45.158542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:22.553597+0000","last_clean_scrub_stamp":"2024-04-25T00:13:22.553597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:43.822447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314487+0000","last_change":"2024-04-25T00:13:18.523583+0000","last_active":"2024-04-25T00:15:45.314487+0000","last_peered":"2024-04-25T00:15:45.314487+0000","last_clean":"2024-04-25T00:15:45.314487+0000","last_became_active":"2024-04-25T00:13:18.523238+0000","last_became_peered":"2024-04-25T00:13:18.523238+0000","last_unstale":"2024-04-25T00:15:45.314487+0000","last_undegraded":"2024-04-25T00:15:45.314487+0000","last_fullsized":"2024-04-25T00:15:45.314487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:17.497825+0000","last_clean_scrub_stamp":"2024-04-25T00:13:17.497825+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:03.869276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.313212+0000","last_change":"2024-04-25T00:13:19.426021+0000","last_active":"2024-04-25T00:15:45.313212+0000","last_peered":"2024-04-25T00:15:45.313212+0000","last_clean":"2024-04-25T00:15:45.313212+0000","last_became_active":"2024-04-25T00:13:19.425676+0000","last_became_peered":"2024-04-25T00:13:19.425676+0000","last_unstale":"2024-04-25T00:15:45.313212+0000","last_undegraded":"2024-04-25T00:15:45.313212+0000","last_fullsized":"2024-04-25T00:15:45.313212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:10.044202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293774+0000","last_change":"2024-04-25T00:13:19.422402+0000","last_active":"2024-04-25T00:15:45.293774+0000","last_peered":"2024-04-25T00:15:45.293774+0000","last_clean":"2024-04-25T00:15:45.293774+0000","last_became_active":"2024-04-25T00:13:19.422057+0000","last_became_peered":"2024-04-25T00:13:19.422057+0000","last_unstale":"2024-04-25T00:15:45.293774+0000","last_undegraded":"2024-04-25T00:15:45.293774+0000","last_fullsized":"2024-04-25T00:15:45.293774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:31:43.703030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041874+0000","last_change":"2024-04-25T00:13:19.425434+0000","last_active":"2024-04-25T00:15:45.041874+0000","last_peered":"2024-04-25T00:15:45.041874+0000","last_clean":"2024-04-25T00:15:45.041874+0000","last_became_active":"2024-04-25T00:13:19.425051+0000","last_became_peered":"2024-04-25T00:13:19.425051+0000","last_unstale":"2024-04-25T00:15:45.041874+0000","last_undegraded":"2024-04-25T00:15:45.041874+0000","last_fullsized":"2024-04-25T00:15:45.041874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:17.568415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041226+0000","last_change":"2024-04-25T00:13:19.413835+0000","last_active":"2024-04-25T00:15:45.041226+0000","last_peered":"2024-04-25T00:15:45.041226+0000","last_clean":"2024-04-25T00:15:45.041226+0000","last_became_active":"2024-04-25T00:13:19.413406+0000","last_became_peered":"2024-04-25T00:13:19.413406+0000","last_unstale":"2024-04-25T00:15:45.041226+0000","last_undegraded":"2024-04-25T00:15:45.041226+0000","last_fullsized":"2024-04-25T00:15:45.041226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:33.852391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.114923+0000","last_change":"2024-04-25T00:13:19.426875+0000","last_active":"2024-04-25T00:15:45.114923+0000","last_peered":"2024-04-25T00:15:45.114923+0000","last_clean":"2024-04-25T00:15:45.114923+0000","last_became_active":"2024-04-25T00:13:19.425828+0000","last_became_peered":"2024-04-25T00:13:19.425828+0000","last_unstale":"2024-04-25T00:15:45.114923+0000","last_undegraded":"2024-04-25T00:15:45.114923+0000","last_fullsized":"2024-04-25T00:15:45.114923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:21.119551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314721+0000","last_change":"2024-04-25T00:13:19.423637+0000","last_active":"2024-04-25T00:15:45.314721+0000","last_peered":"2024-04-25T00:15:45.314721+0000","last_clean":"2024-04-25T00:15:45.314721+0000","last_became_active":"2024-04-25T00:13:19.423272+0000","last_became_peered":"2024-04-25T00:13:19.423272+0000","last_unstale":"2024-04-25T00:15:45.314721+0000","last_undegraded":"2024-04-25T00:15:45.314721+0000","last_fullsized":"2024-04-25T00:15:45.314721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:32:38.794966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293982+0000","last_change":"2024-04-25T00:13:19.423538+0000","last_active":"2024-04-25T00:15:45.293982+0000","last_peered":"2024-04-25T00:15:45.293982+0000","last_clean":"2024-04-25T00:15:45.293982+0000","last_became_active":"2024-04-25T00:13:19.423028+0000","last_became_peered":"2024-04-25T00:13:19.423028+0000","last_unstale":"2024-04-25T00:15:45.293982+0000","last_undegraded":"2024-04-25T00:15:45.293982+0000","last_fullsized":"2024-04-25T00:15:45.293982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:29:41.975044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.041851+0000","last_change":"2024-04-25T00:13:19.416899+0000","last_active":"2024-04-25T00:15:45.041851+0000","last_peered":"2024-04-25T00:15:45.041851+0000","last_clean":"2024-04-25T00:15:45.041851+0000","last_became_active":"2024-04-25T00:13:19.416505+0000","last_became_peered":"2024-04-25T00:13:19.416505+0000","last_unstale":"2024-04-25T00:15:45.041851+0000","last_undegraded":"2024-04-25T00:15:45.041851+0000","last_fullsized":"2024-04-25T00:15:45.041851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:37:40.204119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.040400+0000","last_change":"2024-04-25T00:13:19.426215+0000","last_active":"2024-04-25T00:15:45.040400+0000","last_peered":"2024-04-25T00:15:45.040400+0000","last_clean":"2024-04-25T00:15:45.040400+0000","last_became_active":"2024-04-25T00:13:19.422804+0000","last_became_peered":"2024-04-25T00:13:19.422804+0000","last_unstale":"2024-04-25T00:15:45.040400+0000","last_undegraded":"2024-04-25T00:15:45.040400+0000","last_fullsized":"2024-04-25T00:15:45.040400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:24:44.630891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.158710+0000","last_change":"2024-04-25T00:13:19.413564+0000","last_active":"2024-04-25T00:15:45.158710+0000","last_peered":"2024-04-25T00:15:45.158710+0000","last_clean":"2024-04-25T00:15:45.158710+0000","last_became_active":"2024-04-25T00:13:19.413223+0000","last_became_peered":"2024-04-25T00:13:19.413223+0000","last_unstale":"2024-04-25T00:15:45.158710+0000","last_undegraded":"2024-04-25T00:15:45.158710+0000","last_fullsized":"2024-04-25T00:15:45.158710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:45.629018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.039748+0000","last_change":"2024-04-25T00:13:19.425945+0000","last_active":"2024-04-25T00:15:45.039748+0000","last_peered":"2024-04-25T00:15:45.039748+0000","last_clean":"2024-04-25T00:15:45.039748+0000","last_became_active":"2024-04-25T00:13:19.422499+0000","last_became_peered":"2024-04-25T00:13:19.422499+0000","last_unstale":"2024-04-25T00:15:45.039748+0000","last_undegraded":"2024-04-25T00:15:45.039748+0000","last_fullsized":"2024-04-25T00:15:45.039748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:45.299300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.309420+0000","last_change":"2024-04-25T00:15:11.584671+0000","last_active":"2024-04-25T00:15:45.309420+0000","last_peered":"2024-04-25T00:15:45.309420+0000","last_clean":"2024-04-25T00:15:45.309420+0000","last_became_active":"2024-04-25T00:15:11.584144+0000","last_became_peered":"2024-04-25T00:15:11.584144+0000","last_unstale":"2024-04-25T00:15:45.309420+0000","last_undegraded":"2024-04-25T00:15:45.309420+0000","last_fullsized":"2024-04-25T00:15:45.309420+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:37:52.522586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.314719+0000","last_change":"2024-04-25T00:13:19.420550+0000","last_active":"2024-04-25T00:15:45.314719+0000","last_peered":"2024-04-25T00:15:45.314719+0000","last_clean":"2024-04-25T00:15:45.314719+0000","last_became_active":"2024-04-25T00:13:19.420235+0000","last_became_peered":"2024-04-25T00:13:19.420235+0000","last_unstale":"2024-04-25T00:15:45.314719+0000","last_undegraded":"2024-04-25T00:15:45.314719+0000","last_fullsized":"2024-04-25T00:15:45.314719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:59:36.543776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T00:15:45.293898+0000","last_change":"2024-04-25T00:13:19.429995+0000","last_active":"2024-04-25T00:15:45.293898+0000","last_peered":"2024-04-25T00:15:45.293898+0000","last_clean":"2024-04-25T00:15:45.293898+0000","last_became_active":"2024-04-25T00:13:19.429185+0000","last_became_peered":"2024-04-25T00:13:19.429185+0000","last_unstale":"2024-04-25T00:15:45.293898+0000","last_undegraded":"2024-04-25T00:15:45.293898+0000","last_fullsized":"2024-04-25T00:15:45.293898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:13:18.499911+0000","last_clean_scrub_stamp":"2024-04-25T00:13:18.499911+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:29:27.974479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107352,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600055,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1337,"num_read_kb":783,"num_write":372,"num_write_kb":1466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":1800165,"data_compressed":80712,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":300,"ondisk_log_size":300,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383723,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":520207,"data_compressed":26578,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645358,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1212416,"data_stored":1315825,"data_compressed":73110,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":927315,"data_compressed":48714,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":923597,"data_compressed":48714,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1114112,"data_stored":1127718,"data_compressed":65352,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":526206,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":583001,"data_compressed":28180,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":527745,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1130496,"data_stored":594343,"data_compressed":31170,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":1050031,"data_compressed":54870,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":530113,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32292,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712860,"statfs":{"total":95995035648,"available":95961968640,"internally_reserved":0,"allocated":1142784,"data_stored":1253669,"data_compressed":71508,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":69255,"data_compressed":1602,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":69558,"data_compressed":1602,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":134420,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":200672,"data_compressed":7758,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:15:50.851 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:15:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.987+0000 7f09d0458640 1 -- 172.21.15.27:0/4011925450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09c8100300 msgr2=0x7f09c81006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.987+0000 7f09d0458640 1 --2- 172.21.15.27:0/4011925450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09c8100300 0x7f09c81006e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009900 tx=0x7f09c002a410 comp rx=0 tx=0).stop 2024-04-25T00:15:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.987+0000 7f09d0458640 1 -- 172.21.15.27:0/4011925450 shutdown_connections 2024-04-25T00:15:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.987+0000 7f09d0458640 1 --2- 172.21.15.27:0/4011925450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09c8106ac0 0x7f09c810e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.987+0000 7f09d0458640 1 --2- 172.21.15.27:0/4011925450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c8100cb0 0x7f09c81063e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.987+0000 7f09d0458640 1 --2- 172.21.15.27:0/4011925450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09c8100300 0x7f09c81006e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.988+0000 7f09d0458640 1 -- 172.21.15.27:0/4011925450 >> 172.21.15.27:0/4011925450 conn(0x7f09c8075920 msgr2=0x7f09c8075d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.988+0000 7f09d0458640 1 -- 172.21.15.27:0/4011925450 shutdown_connections 2024-04-25T00:15:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.988+0000 7f09d0458640 1 -- 172.21.15.27:0/4011925450 wait complete. 2024-04-25T00:15:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.988+0000 7f09d0458640 1 Processor -- start 2024-04-25T00:15:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.988+0000 7f09d0458640 1 -- start start 2024-04-25T00:15:50.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.988+0000 7f09d0458640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c8100300 0x7f09c81980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:50.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.988+0000 7f09d0458640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09c8106ac0 0x7f09c81985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:50.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.988+0000 7f09d0458640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09c819c970 0x7f09c819ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:50.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.988+0000 7f09d0458640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f09c8111b90 con 0x7f09c8106ac0 2024-04-25T00:15:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.988+0000 7f09d0458640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f09c8111a10 con 0x7f09c819c970 2024-04-25T00:15:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.988+0000 7f09d0458640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f09c8111d10 con 0x7f09c8100300 2024-04-25T00:15:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.988+0000 7f09cd9f3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09c8106ac0 0x7f09c81985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.989+0000 7f09cd9f3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09c8106ac0 0x7f09c81985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47736/0 (socket says 172.21.15.27:47736) 2024-04-25T00:15:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.989+0000 7f09cd9f3640 1 -- 172.21.15.27:0/4253019242 learned_addr learned my addr 172.21.15.27:0/4253019242 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.989+0000 7f09ce9f5640 1 --2- 172.21.15.27:0/4253019242 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09c819c970 0x7f09c819ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.989+0000 7f09ce1f4640 1 --2- 172.21.15.27:0/4253019242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c8100300 0x7f09c81980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.989+0000 7f09cd9f3640 1 -- 172.21.15.27:0/4253019242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c8100300 msgr2=0x7f09c81980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.989+0000 7f09cd9f3640 1 --2- 172.21.15.27:0/4253019242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c8100300 0x7f09c81980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.989+0000 7f09cd9f3640 1 -- 172.21.15.27:0/4253019242 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09c819c970 msgr2=0x7f09c819ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.989+0000 7f09cd9f3640 1 --2- 172.21.15.27:0/4253019242 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09c819c970 0x7f09c819ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.989+0000 7f09cd9f3640 1 -- 172.21.15.27:0/4253019242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c819d460 con 0x7f09c8106ac0 2024-04-25T00:15:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.989+0000 7f09cd9f3640 1 --2- 172.21.15.27:0/4253019242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09c8106ac0 0x7f09c81985f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00e9f0 tx=0x7f09bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.989+0000 7f09c77fe640 1 -- 172.21.15.27:0/4253019242 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc00ccc0 con 0x7f09c8106ac0 2024-04-25T00:15:50.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.989+0000 7f09c77fe640 1 -- 172.21.15.27:0/4253019242 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09bc005500 con 0x7f09c8106ac0 2024-04-25T00:15:50.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.989+0000 7f09d0458640 1 -- 172.21.15.27:0/4253019242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c819d750 con 0x7f09c8106ac0 2024-04-25T00:15:50.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.989+0000 7f09c77fe640 1 -- 172.21.15.27:0/4253019242 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc010690 con 0x7f09c8106ac0 2024-04-25T00:15:50.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.989+0000 7f09d0458640 1 -- 172.21.15.27:0/4253019242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c81a4ef0 con 0x7f09c8106ac0 2024-04-25T00:15:50.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.990+0000 7f09d0458640 1 -- 172.21.15.27:0/4253019242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c80689b0 con 0x7f09c8106ac0 2024-04-25T00:15:50.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.991+0000 7f09c77fe640 1 -- 172.21.15.27:0/4253019242 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f09bc0046c0 con 0x7f09c8106ac0 2024-04-25T00:15:50.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.992+0000 7f09c77fe640 1 --2- 172.21.15.27:0/4253019242 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f09b00a2e20 0x7f09b00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:50.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.992+0000 7f09ce1f4640 1 --2- 172.21.15.27:0/4253019242 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f09b00a2e20 0x7f09b00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:50.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.992+0000 7f09c77fe640 1 -- 172.21.15.27:0/4253019242 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f09bc0cfe00 con 0x7f09c8106ac0 2024-04-25T00:15:50.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.992+0000 7f09ce1f4640 1 --2- 172.21.15.27:0/4253019242 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f09b00a2e20 0x7f09b00a52e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009230 tx=0x7f09c00058d0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:50.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:50.996+0000 7f09c77fe640 1 -- 172.21.15.27:0/4253019242 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f09bc09c460 con 0x7f09c8106ac0 2024-04-25T00:15:51.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.186+0000 7f09d0458640 1 -- 172.21.15.27:0/4253019242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09c8109cd0 con 0x7f09c8106ac0 2024-04-25T00:15:51.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.188+0000 7f09c77fe640 1 -- 172.21.15.27:0/4253019242 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25113 (secure 0 0 0) 0x7f09bc09bd60 con 0x7f09c8106ac0 2024-04-25T00:15:51.188 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:51.188 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:15:45.034931+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1885111026":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/2528869729":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/4085720546":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/236412562":"2024-04-26T00:15:45.034899+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/2161554778":"2024-04-26T00:15:45.034899+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6800/2161554778":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:15:51.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.190+0000 7f09d0458640 1 -- 172.21.15.27:0/4253019242 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f09b00a2e20 msgr2=0x7f09b00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.190+0000 7f09d0458640 1 --2- 172.21.15.27:0/4253019242 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f09b00a2e20 0x7f09b00a52e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009230 tx=0x7f09c00058d0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.190+0000 7f09d0458640 1 -- 172.21.15.27:0/4253019242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09c8106ac0 msgr2=0x7f09c81985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.190+0000 7f09d0458640 1 --2- 172.21.15.27:0/4253019242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09c8106ac0 0x7f09c81985f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00e9f0 tx=0x7f09bc00eeb0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.190+0000 7f09d0458640 1 -- 172.21.15.27:0/4253019242 shutdown_connections 2024-04-25T00:15:51.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.190+0000 7f09d0458640 1 --2- 172.21.15.27:0/4253019242 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f09b00a2e20 0x7f09b00a52e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.190+0000 7f09d0458640 1 --2- 172.21.15.27:0/4253019242 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09c819c970 0x7f09c819ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.190+0000 7f09d0458640 1 --2- 172.21.15.27:0/4253019242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09c8106ac0 0x7f09c81985f0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.190+0000 7f09d0458640 1 --2- 172.21.15.27:0/4253019242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09c8100300 0x7f09c81980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.190+0000 7f09d0458640 1 -- 172.21.15.27:0/4253019242 >> 172.21.15.27:0/4253019242 conn(0x7f09c8075920 msgr2=0x7f09c80fe400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:51.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.190+0000 7f09d0458640 1 -- 172.21.15.27:0/4253019242 shutdown_connections 2024-04-25T00:15:51.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.190+0000 7f09d0458640 1 -- 172.21.15.27:0/4253019242 wait complete. 2024-04-25T00:15:51.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:51 smithi027 ceph-mon[29208]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:15:51.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:51 smithi027 ceph-mon[29208]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:15:51.201 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:08:34.006973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T00:13:17.337307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T00:13:18.253713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T00:13:21.995987+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T00:15:51.202 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:15:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.307+0000 7f6d064f7640 1 -- 172.21.15.27:0/90931451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d0006a780 msgr2=0x7f6d0010e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.307+0000 7f6d064f7640 1 --2- 172.21.15.27:0/90931451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d0006a780 0x7f6d0010e5f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009cd0 tx=0x7f6cf802a8e0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.307+0000 7f6d064f7640 1 -- 172.21.15.27:0/90931451 shutdown_connections 2024-04-25T00:15:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.307+0000 7f6d064f7640 1 --2- 172.21.15.27:0/90931451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d0006a780 0x7f6d0010e5f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.307+0000 7f6d064f7640 1 --2- 172.21.15.27:0/90931451 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d00069dc0 0x7f6d0006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.308+0000 7f6d064f7640 1 --2- 172.21.15.27:0/90931451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d00107730 0x7f6d00107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.308+0000 7f6d064f7640 1 -- 172.21.15.27:0/90931451 >> 172.21.15.27:0/90931451 conn(0x7f6d000776a0 msgr2=0x7f6d00077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.308+0000 7f6d064f7640 1 -- 172.21.15.27:0/90931451 shutdown_connections 2024-04-25T00:15:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.308+0000 7f6d064f7640 1 -- 172.21.15.27:0/90931451 wait complete. 2024-04-25T00:15:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.308+0000 7f6d064f7640 1 Processor -- start 2024-04-25T00:15:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.308+0000 7f6d064f7640 1 -- start start 2024-04-25T00:15:51.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6d064f7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d00069dc0 0x7f6d00100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:51.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6d064f7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d0006a780 0x7f6d00100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:51.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6d064f7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d00107730 0x7f6d001010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:51.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6d064f7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6d00111b40 con 0x7f6d0006a780 2024-04-25T00:15:51.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6d064f7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6d001119c0 con 0x7f6d00107730 2024-04-25T00:15:51.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6d064f7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d00111cc0 con 0x7f6d00069dc0 2024-04-25T00:15:51.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6cff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d0006a780 0x7f6d00100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:51.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6cff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d0006a780 0x7f6d00100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47750/0 (socket says 172.21.15.27:47750) 2024-04-25T00:15:51.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6cff7fe640 1 -- 172.21.15.27:0/3285798110 learned_addr learned my addr 172.21.15.27:0/3285798110 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:51.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6cfffff640 1 --2- 172.21.15.27:0/3285798110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d00069dc0 0x7f6d00100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:51.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6d04a94640 1 --2- 172.21.15.27:0/3285798110 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d00107730 0x7f6d001010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:51.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6cff7fe640 1 -- 172.21.15.27:0/3285798110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d00069dc0 msgr2=0x7f6d00100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6cff7fe640 1 --2- 172.21.15.27:0/3285798110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d00069dc0 0x7f6d00100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6cff7fe640 1 -- 172.21.15.27:0/3285798110 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d00107730 msgr2=0x7f6d001010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6cff7fe640 1 --2- 172.21.15.27:0/3285798110 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d00107730 0x7f6d001010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6cff7fe640 1 -- 172.21.15.27:0/3285798110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d001a6460 con 0x7f6d0006a780 2024-04-25T00:15:51.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6cfffff640 1 --2- 172.21.15.27:0/3285798110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d00069dc0 0x7f6d00100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:51.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6cff7fe640 1 --2- 172.21.15.27:0/3285798110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d0006a780 0x7f6d00100bb0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000d950 tx=0x7f6cf000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:51.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6cfd7fa640 1 -- 172.21.15.27:0/3285798110 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0017070 con 0x7f6d0006a780 2024-04-25T00:15:51.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6cfd7fa640 1 -- 172.21.15.27:0/3285798110 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cf0009c30 con 0x7f6d0006a780 2024-04-25T00:15:51.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6d064f7640 1 -- 172.21.15.27:0/3285798110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d001a6750 con 0x7f6d0006a780 2024-04-25T00:15:51.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6cfd7fa640 1 -- 172.21.15.27:0/3285798110 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf000f6b0 con 0x7f6d0006a780 2024-04-25T00:15:51.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.309+0000 7f6d064f7640 1 -- 172.21.15.27:0/3285798110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d001a6ca0 con 0x7f6d0006a780 2024-04-25T00:15:51.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.310+0000 7f6d064f7640 1 -- 172.21.15.27:0/3285798110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d000689b0 con 0x7f6d0006a780 2024-04-25T00:15:51.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.312+0000 7f6cfd7fa640 1 -- 172.21.15.27:0/3285798110 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f6cf00109d0 con 0x7f6d0006a780 2024-04-25T00:15:51.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.312+0000 7f6cfd7fa640 1 --2- 172.21.15.27:0/3285798110 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6ce80a2e20 0x7f6ce80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:51.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.312+0000 7f6cfffff640 1 --2- 172.21.15.27:0/3285798110 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6ce80a2e20 0x7f6ce80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:51.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.313+0000 7f6cfd7fa640 1 -- 172.21.15.27:0/3285798110 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f6cf00cf760 con 0x7f6d0006a780 2024-04-25T00:15:51.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.313+0000 7f6cfffff640 1 --2- 172.21.15.27:0/3285798110 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6ce80a2e20 0x7f6ce80a52e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009ad0 tx=0x7f6cf4005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:51.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.316+0000 7f6cfd7fa640 1 -- 172.21.15.27:0/3285798110 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6cf0009de0 con 0x7f6d0006a780 2024-04-25T00:15:51.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.501+0000 7f6d064f7640 1 -- 172.21.15.27:0/3285798110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6d001a6f00 con 0x7f6d0006a780 2024-04-25T00:15:51.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.502+0000 7f6cfd7fa640 1 -- 172.21.15.27:0/3285798110 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f6cf009bdc0 con 0x7f6d0006a780 2024-04-25T00:15:51.502 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T00:15:51.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:51 smithi139 ceph-mon[32833]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:15:51.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:51 smithi139 ceph-mon[32833]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:15:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.504+0000 7f6d064f7640 1 -- 172.21.15.27:0/3285798110 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6ce80a2e20 msgr2=0x7f6ce80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.504+0000 7f6d064f7640 1 --2- 172.21.15.27:0/3285798110 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6ce80a2e20 0x7f6ce80a52e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009ad0 tx=0x7f6cf4005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.504+0000 7f6d064f7640 1 -- 172.21.15.27:0/3285798110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d0006a780 msgr2=0x7f6d00100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.504+0000 7f6d064f7640 1 --2- 172.21.15.27:0/3285798110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d0006a780 0x7f6d00100bb0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000d950 tx=0x7f6cf000de10 comp rx=0 tx=0).stop 2024-04-25T00:15:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.504+0000 7f6d064f7640 1 -- 172.21.15.27:0/3285798110 shutdown_connections 2024-04-25T00:15:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.504+0000 7f6d064f7640 1 --2- 172.21.15.27:0/3285798110 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6ce80a2e20 0x7f6ce80a52e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.504+0000 7f6d064f7640 1 --2- 172.21.15.27:0/3285798110 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d00107730 0x7f6d001010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.504+0000 7f6d064f7640 1 --2- 172.21.15.27:0/3285798110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d0006a780 0x7f6d00100bb0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.504+0000 7f6d064f7640 1 --2- 172.21.15.27:0/3285798110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d00069dc0 0x7f6d00100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.504+0000 7f6d064f7640 1 -- 172.21.15.27:0/3285798110 >> 172.21.15.27:0/3285798110 conn(0x7f6d000776a0 msgr2=0x7f6d000fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.504+0000 7f6d064f7640 1 -- 172.21.15.27:0/3285798110 shutdown_connections 2024-04-25T00:15:51.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.504+0000 7f6d064f7640 1 -- 172.21.15.27:0/3285798110 wait complete. 2024-04-25T00:15:51.515 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T00:15:51.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:51 smithi153 ceph-mon[32896]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:15:51.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:51 smithi153 ceph-mon[32896]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:15:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.620+0000 7f5395658640 1 -- 172.21.15.27:0/2775576473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5390106aa0 msgr2=0x7f539010e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.620+0000 7f5395658640 1 --2- 172.21.15.27:0/2775576473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5390106aa0 0x7f539010e6b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f5388009ca0 tx=0x7f538802a8c0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.620+0000 7f5395658640 1 -- 172.21.15.27:0/2775576473 shutdown_connections 2024-04-25T00:15:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.620+0000 7f5395658640 1 --2- 172.21.15.27:0/2775576473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5390106aa0 0x7f539010e6b0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.620+0000 7f5395658640 1 --2- 172.21.15.27:0/2775576473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5390100c90 0x7f53901063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.620+0000 7f5395658640 1 --2- 172.21.15.27:0/2775576473 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53901002e0 0x7f53901006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.620+0000 7f5395658640 1 -- 172.21.15.27:0/2775576473 >> 172.21.15.27:0/2775576473 conn(0x7f53900758d0 msgr2=0x7f5390075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.620+0000 7f5395658640 1 -- 172.21.15.27:0/2775576473 shutdown_connections 2024-04-25T00:15:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.620+0000 7f5395658640 1 -- 172.21.15.27:0/2775576473 wait complete. 2024-04-25T00:15:51.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.621+0000 7f5395658640 1 Processor -- start 2024-04-25T00:15:51.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.621+0000 7f5395658640 1 -- start start 2024-04-25T00:15:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.621+0000 7f5395658640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53901002e0 0x7f5390101250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.621+0000 7f5395658640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5390106aa0 0x7f5390101790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.621+0000 7f5395658640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5390101cd0 0x7f53901a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.621+0000 7f5395658640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5390111b90 con 0x7f5390106aa0 2024-04-25T00:15:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.621+0000 7f5395658640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5390111a10 con 0x7f53901002e0 2024-04-25T00:15:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.621+0000 7f5395658640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5390111d10 con 0x7f5390101cd0 2024-04-25T00:15:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.621+0000 7f538e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5390106aa0 0x7f5390101790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.621+0000 7f538e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5390106aa0 0x7f5390101790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47772/0 (socket says 172.21.15.27:47772) 2024-04-25T00:15:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.622+0000 7f538e7fc640 1 -- 172.21.15.27:0/4061542228 learned_addr learned my addr 172.21.15.27:0/4061542228 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.622+0000 7f538f7fe640 1 --2- 172.21.15.27:0/4061542228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5390101cd0 0x7f53901a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.622+0000 7f538effd640 1 --2- 172.21.15.27:0/4061542228 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53901002e0 0x7f5390101250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.622+0000 7f538e7fc640 1 -- 172.21.15.27:0/4061542228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5390101cd0 msgr2=0x7f53901a6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.622+0000 7f538e7fc640 1 --2- 172.21.15.27:0/4061542228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5390101cd0 0x7f53901a6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.622+0000 7f538e7fc640 1 -- 172.21.15.27:0/4061542228 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53901002e0 msgr2=0x7f5390101250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.622+0000 7f538e7fc640 1 --2- 172.21.15.27:0/4061542228 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53901002e0 0x7f5390101250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.622+0000 7f538e7fc640 1 -- 172.21.15.27:0/4061542228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53901a6920 con 0x7f5390106aa0 2024-04-25T00:15:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.622+0000 7f538effd640 1 --2- 172.21.15.27:0/4061542228 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53901002e0 0x7f5390101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.622+0000 7f538e7fc640 1 --2- 172.21.15.27:0/4061542228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5390106aa0 0x7f5390101790 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f538400d740 tx=0x7f538400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.622+0000 7f5377fff640 1 -- 172.21.15.27:0/4061542228 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5384017070 con 0x7f5390106aa0 2024-04-25T00:15:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.622+0000 7f5395658640 1 -- 172.21.15.27:0/4061542228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53901a6c10 con 0x7f5390106aa0 2024-04-25T00:15:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.622+0000 7f5377fff640 1 -- 172.21.15.27:0/4061542228 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f538400de50 con 0x7f5390106aa0 2024-04-25T00:15:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.622+0000 7f5377fff640 1 -- 172.21.15.27:0/4061542228 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5384004e00 con 0x7f5390106aa0 2024-04-25T00:15:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.622+0000 7f5395658640 1 -- 172.21.15.27:0/4061542228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53901a7110 con 0x7f5390106aa0 2024-04-25T00:15:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.623+0000 7f5395658640 1 -- 172.21.15.27:0/4061542228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53900689b0 con 0x7f5390106aa0 2024-04-25T00:15:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.624+0000 7f5377fff640 1 -- 172.21.15.27:0/4061542228 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f5384004fa0 con 0x7f5390106aa0 2024-04-25T00:15:51.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.625+0000 7f5377fff640 1 --2- 172.21.15.27:0/4061542228 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f53780a2e50 0x7f53780a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:51.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.625+0000 7f538effd640 1 --2- 172.21.15.27:0/4061542228 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f53780a2e50 0x7f53780a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:51.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.625+0000 7f5377fff640 1 -- 172.21.15.27:0/4061542228 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f53840cf370 con 0x7f5390106aa0 2024-04-25T00:15:51.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.625+0000 7f538effd640 1 --2- 172.21.15.27:0/4061542228 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f53780a2e50 0x7f53780a5310 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5380005fd0 tx=0x7f538000b5f0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:51.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.629+0000 7f5377fff640 1 -- 172.21.15.27:0/4061542228 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f538409b9d0 con 0x7f5390106aa0 2024-04-25T00:15:51.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.816+0000 7f5395658640 1 -- 172.21.15.27:0/4061542228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5390109da0 con 0x7f5390106aa0 2024-04-25T00:15:51.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.816+0000 7f5377fff640 1 -- 172.21.15.27:0/4061542228 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f538409b2d0 con 0x7f5390106aa0 2024-04-25T00:15:51.816 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:15:51.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.818+0000 7f5395658640 1 -- 172.21.15.27:0/4061542228 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f53780a2e50 msgr2=0x7f53780a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.818+0000 7f5395658640 1 --2- 172.21.15.27:0/4061542228 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f53780a2e50 0x7f53780a5310 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5380005fd0 tx=0x7f538000b5f0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.818+0000 7f5395658640 1 -- 172.21.15.27:0/4061542228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5390106aa0 msgr2=0x7f5390101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.818+0000 7f5395658640 1 --2- 172.21.15.27:0/4061542228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5390106aa0 0x7f5390101790 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f538400d740 tx=0x7f538400dc00 comp rx=0 tx=0).stop 2024-04-25T00:15:51.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.818+0000 7f5395658640 1 -- 172.21.15.27:0/4061542228 shutdown_connections 2024-04-25T00:15:51.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.818+0000 7f5395658640 1 --2- 172.21.15.27:0/4061542228 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f53780a2e50 0x7f53780a5310 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.818+0000 7f5395658640 1 --2- 172.21.15.27:0/4061542228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5390101cd0 0x7f53901a6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.818+0000 7f5395658640 1 --2- 172.21.15.27:0/4061542228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5390106aa0 0x7f5390101790 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.818+0000 7f5395658640 1 --2- 172.21.15.27:0/4061542228 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53901002e0 0x7f5390101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.818+0000 7f5395658640 1 -- 172.21.15.27:0/4061542228 >> 172.21.15.27:0/4061542228 conn(0x7f53900758d0 msgr2=0x7f53900fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:51.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.819+0000 7f5395658640 1 -- 172.21.15.27:0/4061542228 shutdown_connections 2024-04-25T00:15:51.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.819+0000 7f5395658640 1 -- 172.21.15.27:0/4061542228 wait complete. 2024-04-25T00:15:51.830 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T00:15:51.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.935+0000 7fd6d0b4c640 1 -- 172.21.15.27:0/1753868134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6cc101710 msgr2=0x7fd6cc101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.935+0000 7fd6d0b4c640 1 --2- 172.21.15.27:0/1753868134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6cc101710 0x7fd6cc101b90 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009930 tx=0x7fd6bc02a420 comp rx=0 tx=0).stop 2024-04-25T00:15:51.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.935+0000 7fd6d0b4c640 1 -- 172.21.15.27:0/1753868134 shutdown_connections 2024-04-25T00:15:51.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.935+0000 7fd6d0b4c640 1 --2- 172.21.15.27:0/1753868134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6cc1020d0 0x7fd6cc10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.935+0000 7fd6d0b4c640 1 --2- 172.21.15.27:0/1753868134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6cc101710 0x7fd6cc101b90 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.935+0000 7fd6d0b4c640 1 --2- 172.21.15.27:0/1753868134 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6cc107720 0x7fd6cc107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.935+0000 7fd6d0b4c640 1 -- 172.21.15.27:0/1753868134 >> 172.21.15.27:0/1753868134 conn(0x7fd6cc078b90 msgr2=0x7fd6cc078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:51.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.935+0000 7fd6d0b4c640 1 -- 172.21.15.27:0/1753868134 shutdown_connections 2024-04-25T00:15:51.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.935+0000 7fd6d0b4c640 1 -- 172.21.15.27:0/1753868134 wait complete. 2024-04-25T00:15:51.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.936+0000 7fd6d0b4c640 1 Processor -- start 2024-04-25T00:15:51.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.936+0000 7fd6d0b4c640 1 -- start start 2024-04-25T00:15:51.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.936+0000 7fd6d0b4c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6cc101710 0x7fd6cc1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:51.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.936+0000 7fd6d0b4c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6cc1020d0 0x7fd6cc1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:51.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.936+0000 7fd6d0b4c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6cc107720 0x7fd6cc19c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.936+0000 7fd6d0b4c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc111b30 con 0x7fd6cc101710 2024-04-25T00:15:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.936+0000 7fd6d0b4c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc1119b0 con 0x7fd6cc107720 2024-04-25T00:15:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.936+0000 7fd6d0b4c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc111cb0 con 0x7fd6cc1020d0 2024-04-25T00:15:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.936+0000 7fd6ca59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6cc101710 0x7fd6cc1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.936+0000 7fd6ca59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6cc101710 0x7fd6cc1980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47796/0 (socket says 172.21.15.27:47796) 2024-04-25T00:15:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.936+0000 7fd6ca59c640 1 -- 172.21.15.27:0/406807408 learned_addr learned my addr 172.21.15.27:0/406807408 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.936+0000 7fd6c9d9b640 1 --2- 172.21.15.27:0/406807408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6cc1020d0 0x7fd6cc1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.937+0000 7fd6cad9d640 1 --2- 172.21.15.27:0/406807408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6cc107720 0x7fd6cc19c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.937+0000 7fd6ca59c640 1 -- 172.21.15.27:0/406807408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6cc1020d0 msgr2=0x7fd6cc1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.937+0000 7fd6ca59c640 1 --2- 172.21.15.27:0/406807408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6cc1020d0 0x7fd6cc1985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.937+0000 7fd6ca59c640 1 -- 172.21.15.27:0/406807408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6cc107720 msgr2=0x7fd6cc19c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.937+0000 7fd6ca59c640 1 --2- 172.21.15.27:0/406807408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6cc107720 0x7fd6cc19c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.937+0000 7fd6ca59c640 1 -- 172.21.15.27:0/406807408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6cc19d050 con 0x7fd6cc101710 2024-04-25T00:15:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.937+0000 7fd6c9d9b640 1 --2- 172.21.15.27:0/406807408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6cc1020d0 0x7fd6cc1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:51.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.937+0000 7fd6ca59c640 1 --2- 172.21.15.27:0/406807408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6cc101710 0x7fd6cc1980b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400d950 tx=0x7fd6b400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:51.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.937+0000 7fd6c37fe640 1 -- 172.21.15.27:0/406807408 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4017030 con 0x7fd6cc101710 2024-04-25T00:15:51.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.937+0000 7fd6c37fe640 1 -- 172.21.15.27:0/406807408 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6b4004020 con 0x7fd6cc101710 2024-04-25T00:15:51.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.937+0000 7fd6d0b4c640 1 -- 172.21.15.27:0/406807408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6cc19d340 con 0x7fd6cc101710 2024-04-25T00:15:51.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.937+0000 7fd6c37fe640 1 -- 172.21.15.27:0/406807408 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b40047a0 con 0x7fd6cc101710 2024-04-25T00:15:51.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.937+0000 7fd6d0b4c640 1 -- 172.21.15.27:0/406807408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6cc1a4bd0 con 0x7fd6cc101710 2024-04-25T00:15:51.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.938+0000 7fd6d0b4c640 1 -- 172.21.15.27:0/406807408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6cc0689b0 con 0x7fd6cc101710 2024-04-25T00:15:51.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.939+0000 7fd6c37fe640 1 -- 172.21.15.27:0/406807408 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fd6b40041d0 con 0x7fd6cc101710 2024-04-25T00:15:51.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.940+0000 7fd6c37fe640 1 --2- 172.21.15.27:0/406807408 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd6b80a2e20 0x7fd6b80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:51.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.940+0000 7fd6c9d9b640 1 --2- 172.21.15.27:0/406807408 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd6b80a2e20 0x7fd6b80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:51.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.940+0000 7fd6c37fe640 1 -- 172.21.15.27:0/406807408 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fd6b40cf480 con 0x7fd6cc101710 2024-04-25T00:15:51.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.940+0000 7fd6c9d9b640 1 --2- 172.21.15.27:0/406807408 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd6b80a2e20 0x7fd6b80a52e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc000c00 tx=0x7fd6bc005a20 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:51.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:51.944+0000 7fd6c37fe640 1 -- 172.21.15.27:0/406807408 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd6b409bae0 con 0x7fd6cc101710 2024-04-25T00:15:52.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.130+0000 7fd6d0b4c640 1 -- 172.21.15.27:0/406807408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd6cc02d170 con 0x7fd6cc101710 2024-04-25T00:15:52.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.130+0000 7fd6c37fe640 1 -- 172.21.15.27:0/406807408 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fd6b409b3e0 con 0x7fd6cc101710 2024-04-25T00:15:52.130 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T00:15:52.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.132+0000 7fd6d0b4c640 1 -- 172.21.15.27:0/406807408 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd6b80a2e20 msgr2=0x7fd6b80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:52.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.132+0000 7fd6d0b4c640 1 --2- 172.21.15.27:0/406807408 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd6b80a2e20 0x7fd6b80a52e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc000c00 tx=0x7fd6bc005a20 comp rx=0 tx=0).stop 2024-04-25T00:15:52.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.132+0000 7fd6d0b4c640 1 -- 172.21.15.27:0/406807408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6cc101710 msgr2=0x7fd6cc1980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:52.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.132+0000 7fd6d0b4c640 1 --2- 172.21.15.27:0/406807408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6cc101710 0x7fd6cc1980b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400d950 tx=0x7fd6b400de10 comp rx=0 tx=0).stop 2024-04-25T00:15:52.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.132+0000 7fd6d0b4c640 1 -- 172.21.15.27:0/406807408 shutdown_connections 2024-04-25T00:15:52.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.132+0000 7fd6d0b4c640 1 --2- 172.21.15.27:0/406807408 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd6b80a2e20 0x7fd6b80a52e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.132+0000 7fd6d0b4c640 1 --2- 172.21.15.27:0/406807408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6cc107720 0x7fd6cc19c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.132+0000 7fd6d0b4c640 1 --2- 172.21.15.27:0/406807408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6cc1020d0 0x7fd6cc1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.132+0000 7fd6d0b4c640 1 --2- 172.21.15.27:0/406807408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6cc101710 0x7fd6cc1980b0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.132+0000 7fd6d0b4c640 1 -- 172.21.15.27:0/406807408 >> 172.21.15.27:0/406807408 conn(0x7fd6cc078b90 msgr2=0x7fd6cc0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:52.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.132+0000 7fd6d0b4c640 1 -- 172.21.15.27:0/406807408 shutdown_connections 2024-04-25T00:15:52.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.132+0000 7fd6d0b4c640 1 -- 172.21.15.27:0/406807408 wait complete. 2024-04-25T00:15:52.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:52 smithi027 ceph-mon[29208]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:15:52.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:52 smithi027 ceph-mon[29208]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:15:52.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:52 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4253019242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:52.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:52 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3285798110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:52.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:52 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4061542228' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:52.144 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T00:15:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.249+0000 7f1e631a9640 1 -- 172.21.15.27:0/2410780946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e5c1020d0 msgr2=0x7f1e5c10e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.249+0000 7f1e631a9640 1 --2- 172.21.15.27:0/2410780946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e5c1020d0 0x7f1e5c10e5e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f1e54009cd0 tx=0x7f1e5402a8e0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.249+0000 7f1e631a9640 1 -- 172.21.15.27:0/2410780946 shutdown_connections 2024-04-25T00:15:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.249+0000 7f1e631a9640 1 --2- 172.21.15.27:0/2410780946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e5c1020d0 0x7f1e5c10e5e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.249+0000 7f1e631a9640 1 --2- 172.21.15.27:0/2410780946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e5c101710 0x7f1e5c101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.249+0000 7f1e631a9640 1 --2- 172.21.15.27:0/2410780946 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e5c107720 0x7f1e5c107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.249+0000 7f1e631a9640 1 -- 172.21.15.27:0/2410780946 >> 172.21.15.27:0/2410780946 conn(0x7f1e5c0fd180 msgr2=0x7f1e5c0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.249+0000 7f1e631a9640 1 -- 172.21.15.27:0/2410780946 shutdown_connections 2024-04-25T00:15:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.249+0000 7f1e631a9640 1 -- 172.21.15.27:0/2410780946 wait complete. 2024-04-25T00:15:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.249+0000 7f1e631a9640 1 Processor -- start 2024-04-25T00:15:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.249+0000 7f1e631a9640 1 -- start start 2024-04-25T00:15:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e631a9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e5c101710 0x7f1e5c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:52.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e631a9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e5c1020d0 0x7f1e5c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:52.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e631a9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e5c107720 0x7f1e5c19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:52.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e631a9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c111b30 con 0x7f1e5c107720 2024-04-25T00:15:52.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e631a9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c1119b0 con 0x7f1e5c1020d0 2024-04-25T00:15:52.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e631a9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c111cb0 con 0x7f1e5c101710 2024-04-25T00:15:52.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e61746640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e5c107720 0x7f1e5c19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:52.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e61746640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e5c107720 0x7f1e5c19c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47810/0 (socket says 172.21.15.27:47810) 2024-04-25T00:15:52.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e61746640 1 -- 172.21.15.27:0/284151721 learned_addr learned my addr 172.21.15.27:0/284151721 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:52.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e60f45640 1 --2- 172.21.15.27:0/284151721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e5c101710 0x7f1e5c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:52.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e5bfff640 1 --2- 172.21.15.27:0/284151721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e5c1020d0 0x7f1e5c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:52.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e61746640 1 -- 172.21.15.27:0/284151721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e5c101710 msgr2=0x7f1e5c198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:52.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e61746640 1 --2- 172.21.15.27:0/284151721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e5c101710 0x7f1e5c198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e61746640 1 -- 172.21.15.27:0/284151721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e5c1020d0 msgr2=0x7f1e5c198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:52.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e61746640 1 --2- 172.21.15.27:0/284151721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e5c1020d0 0x7f1e5c198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e61746640 1 -- 172.21.15.27:0/284151721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5c19d020 con 0x7f1e5c107720 2024-04-25T00:15:52.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e61746640 1 --2- 172.21.15.27:0/284151721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e5c107720 0x7f1e5c19c8f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1e54005cc0 tx=0x7f1e54032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:52.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e59ffb640 1 -- 172.21.15.27:0/284151721 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54004040 con 0x7f1e5c107720 2024-04-25T00:15:52.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e631a9640 1 -- 172.21.15.27:0/284151721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e5c19d2b0 con 0x7f1e5c107720 2024-04-25T00:15:52.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e59ffb640 1 -- 172.21.15.27:0/284151721 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e540046c0 con 0x7f1e5c107720 2024-04-25T00:15:52.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.250+0000 7f1e59ffb640 1 -- 172.21.15.27:0/284151721 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54030e00 con 0x7f1e5c107720 2024-04-25T00:15:52.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.251+0000 7f1e631a9640 1 -- 172.21.15.27:0/284151721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e5c1a4ba0 con 0x7f1e5c107720 2024-04-25T00:15:52.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.252+0000 7f1e631a9640 1 -- 172.21.15.27:0/284151721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e5c0689b0 con 0x7f1e5c107720 2024-04-25T00:15:52.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.253+0000 7f1e59ffb640 1 -- 172.21.15.27:0/284151721 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f1e540305a0 con 0x7f1e5c107720 2024-04-25T00:15:52.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.253+0000 7f1e59ffb640 1 --2- 172.21.15.27:0/284151721 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1e440a2e20 0x7f1e440a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:52.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.253+0000 7f1e60f45640 1 --2- 172.21.15.27:0/284151721 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1e440a2e20 0x7f1e440a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:52.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.254+0000 7f1e59ffb640 1 -- 172.21.15.27:0/284151721 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f1e5402e030 con 0x7f1e5c107720 2024-04-25T00:15:52.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.254+0000 7f1e60f45640 1 --2- 172.21.15.27:0/284151721 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1e440a2e20 0x7f1e440a52e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c00bb60 tx=0x7f1e4c00b390 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:52.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.257+0000 7f1e59ffb640 1 -- 172.21.15.27:0/284151721 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1e5404b060 con 0x7f1e5c107720 2024-04-25T00:15:52.405 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:52 smithi139 ceph-mon[32833]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:15:52.405 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:52 smithi139 ceph-mon[32833]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:15:52.405 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:52 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4253019242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:52.406 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:52 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3285798110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:52.406 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:52 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4061542228' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:52.428 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:52 smithi153 ceph-mon[32896]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T00:15:52.428 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:52 smithi153 ceph-mon[32896]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T00:15:52.428 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:52 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4253019242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:52.428 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:52 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3285798110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:15:52.428 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:52 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4061542228' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:15:52.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.441+0000 7f1e631a9640 1 -- 172.21.15.27:0/284151721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1e5c02d170 con 0x7f1e5c107720 2024-04-25T00:15:52.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.442+0000 7f1e59ffb640 1 -- 172.21.15.27:0/284151721 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f1e540bd990 con 0x7f1e5c107720 2024-04-25T00:15:52.442 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:15:52.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.444+0000 7f1e631a9640 1 -- 172.21.15.27:0/284151721 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1e440a2e20 msgr2=0x7f1e440a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:52.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.444+0000 7f1e631a9640 1 --2- 172.21.15.27:0/284151721 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1e440a2e20 0x7f1e440a52e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c00bb60 tx=0x7f1e4c00b390 comp rx=0 tx=0).stop 2024-04-25T00:15:52.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.444+0000 7f1e631a9640 1 -- 172.21.15.27:0/284151721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e5c107720 msgr2=0x7f1e5c19c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:52.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.444+0000 7f1e631a9640 1 --2- 172.21.15.27:0/284151721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e5c107720 0x7f1e5c19c8f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1e54005cc0 tx=0x7f1e54032f40 comp rx=0 tx=0).stop 2024-04-25T00:15:52.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.444+0000 7f1e631a9640 1 -- 172.21.15.27:0/284151721 shutdown_connections 2024-04-25T00:15:52.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.444+0000 7f1e631a9640 1 --2- 172.21.15.27:0/284151721 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1e440a2e20 0x7f1e440a52e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.444+0000 7f1e631a9640 1 --2- 172.21.15.27:0/284151721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e5c107720 0x7f1e5c19c8f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.444+0000 7f1e631a9640 1 --2- 172.21.15.27:0/284151721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e5c1020d0 0x7f1e5c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.444+0000 7f1e631a9640 1 --2- 172.21.15.27:0/284151721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e5c101710 0x7f1e5c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.444+0000 7f1e631a9640 1 -- 172.21.15.27:0/284151721 >> 172.21.15.27:0/284151721 conn(0x7f1e5c0fd180 msgr2=0x7f1e5c0fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:52.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.444+0000 7f1e631a9640 1 -- 172.21.15.27:0/284151721 shutdown_connections 2024-04-25T00:15:52.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.444+0000 7f1e631a9640 1 -- 172.21.15.27:0/284151721 wait complete. 2024-04-25T00:15:52.456 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:52.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.559+0000 7f8150cc3640 1 -- 172.21.15.27:0/323919817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f814c103340 msgr2=0x7f814c10b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:52.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.559+0000 7f8150cc3640 1 --2- 172.21.15.27:0/323919817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f814c103340 0x7f814c10b820 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8134009900 tx=0x7f813402a480 comp rx=0 tx=0).stop 2024-04-25T00:15:52.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.559+0000 7f8150cc3640 1 -- 172.21.15.27:0/323919817 shutdown_connections 2024-04-25T00:15:52.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.559+0000 7f8150cc3640 1 --2- 172.21.15.27:0/323919817 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f814c10bd60 0x7f814c10e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.559+0000 7f8150cc3640 1 --2- 172.21.15.27:0/323919817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f814c103340 0x7f814c10b820 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.559+0000 7f8150cc3640 1 --2- 172.21.15.27:0/323919817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f814c102a20 0x7f814c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.559+0000 7f8150cc3640 1 -- 172.21.15.27:0/323919817 >> 172.21.15.27:0/323919817 conn(0x7f814c0777a0 msgr2=0x7f814c077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:52.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.559+0000 7f8150cc3640 1 -- 172.21.15.27:0/323919817 shutdown_connections 2024-04-25T00:15:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.559+0000 7f8150cc3640 1 -- 172.21.15.27:0/323919817 wait complete. 2024-04-25T00:15:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f8150cc3640 1 Processor -- start 2024-04-25T00:15:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f8150cc3640 1 -- start start 2024-04-25T00:15:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f8150cc3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f814c102a20 0x7f814c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f8150cc3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f814c103340 0x7f814c1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f8150cc3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f814c10bd60 0x7f814c19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f8150cc3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f814c111b30 con 0x7f814c102a20 2024-04-25T00:15:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f8150cc3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f814c1119b0 con 0x7f814c10bd60 2024-04-25T00:15:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f8150cc3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f814c111cb0 con 0x7f814c103340 2024-04-25T00:15:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f814a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f814c102a20 0x7f814c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f814a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f814c102a20 0x7f814c197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47828/0 (socket says 172.21.15.27:47828) 2024-04-25T00:15:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f814a59c640 1 -- 172.21.15.27:0/4153310434 learned_addr learned my addr 172.21.15.27:0/4153310434 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f814ad9d640 1 --2- 172.21.15.27:0/4153310434 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f814c10bd60 0x7f814c19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f8149d9b640 1 --2- 172.21.15.27:0/4153310434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f814c103340 0x7f814c1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:52.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f814a59c640 1 -- 172.21.15.27:0/4153310434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f814c103340 msgr2=0x7f814c1984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:52.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f814a59c640 1 --2- 172.21.15.27:0/4153310434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f814c103340 0x7f814c1984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f814a59c640 1 -- 172.21.15.27:0/4153310434 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f814c10bd60 msgr2=0x7f814c19c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:52.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f814a59c640 1 --2- 172.21.15.27:0/4153310434 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f814c10bd60 0x7f814c19c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f814a59c640 1 -- 172.21.15.27:0/4153310434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f814c19cf80 con 0x7f814c102a20 2024-04-25T00:15:52.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.560+0000 7f814ad9d640 1 --2- 172.21.15.27:0/4153310434 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f814c10bd60 0x7f814c19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:52.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.561+0000 7f814a59c640 1 --2- 172.21.15.27:0/4153310434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f814c102a20 0x7f814c197f90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f813c00c920 tx=0x7f813c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:52.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.561+0000 7f81437fe640 1 -- 172.21.15.27:0/4153310434 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813c007c70 con 0x7f814c102a20 2024-04-25T00:15:52.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.561+0000 7f81437fe640 1 -- 172.21.15.27:0/4153310434 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f813c007e10 con 0x7f814c102a20 2024-04-25T00:15:52.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.561+0000 7f8150cc3640 1 -- 172.21.15.27:0/4153310434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f814c19d270 con 0x7f814c102a20 2024-04-25T00:15:52.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.561+0000 7f81437fe640 1 -- 172.21.15.27:0/4153310434 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813c018890 con 0x7f814c102a20 2024-04-25T00:15:52.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.561+0000 7f8150cc3640 1 -- 172.21.15.27:0/4153310434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f814c1a4b00 con 0x7f814c102a20 2024-04-25T00:15:52.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.562+0000 7f8150cc3640 1 -- 172.21.15.27:0/4153310434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f814c0689b0 con 0x7f814c102a20 2024-04-25T00:15:52.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.563+0000 7f81437fe640 1 -- 172.21.15.27:0/4153310434 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f813c018a70 con 0x7f814c102a20 2024-04-25T00:15:52.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.564+0000 7f81437fe640 1 --2- 172.21.15.27:0/4153310434 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f81380a2e20 0x7f81380a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:52.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.564+0000 7f8149d9b640 1 --2- 172.21.15.27:0/4153310434 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f81380a2e20 0x7f81380a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:52.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.564+0000 7f81437fe640 1 -- 172.21.15.27:0/4153310434 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f813c0cfc70 con 0x7f814c102a20 2024-04-25T00:15:52.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.564+0000 7f8149d9b640 1 --2- 172.21.15.27:0/4153310434 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f81380a2e20 0x7f81380a52e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f814c10d930 tx=0x7f8134005a20 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:52.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.568+0000 7f81437fe640 1 -- 172.21.15.27:0/4153310434 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f813c014030 con 0x7f814c102a20 2024-04-25T00:15:52.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.793+0000 7f8150cc3640 1 -- 172.21.15.27:0/4153310434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f814c1992e0 con 0x7f814c102a20 2024-04-25T00:15:52.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.794+0000 7f81437fe640 1 -- 172.21.15.27:0/4153310434 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11944 (secure 0 0 0) 0x7f813c00de20 con 0x7f814c102a20 2024-04-25T00:15:52.794 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:52.794 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:15:43.587614+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24514,24430,24521,24428,24496]},"id":1}]} 2024-04-25T00:15:52.795 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 29 2024-04-25T00:15:52.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.796+0000 7f8150cc3640 1 -- 172.21.15.27:0/4153310434 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f81380a2e20 msgr2=0x7f81380a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:52.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.796+0000 7f8150cc3640 1 --2- 172.21.15.27:0/4153310434 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f81380a2e20 0x7f81380a52e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f814c10d930 tx=0x7f8134005a20 comp rx=0 tx=0).stop 2024-04-25T00:15:52.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.796+0000 7f8150cc3640 1 -- 172.21.15.27:0/4153310434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f814c102a20 msgr2=0x7f814c197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:52.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.796+0000 7f8150cc3640 1 --2- 172.21.15.27:0/4153310434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f814c102a20 0x7f814c197f90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f813c00c920 tx=0x7f813c00cde0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.796+0000 7f8150cc3640 1 -- 172.21.15.27:0/4153310434 shutdown_connections 2024-04-25T00:15:52.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.796+0000 7f8150cc3640 1 --2- 172.21.15.27:0/4153310434 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f81380a2e20 0x7f81380a52e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.796+0000 7f8150cc3640 1 --2- 172.21.15.27:0/4153310434 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f814c10bd60 0x7f814c19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.796+0000 7f8150cc3640 1 --2- 172.21.15.27:0/4153310434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f814c103340 0x7f814c1984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.796+0000 7f8150cc3640 1 --2- 172.21.15.27:0/4153310434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f814c102a20 0x7f814c197f90 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.796+0000 7f8150cc3640 1 -- 172.21.15.27:0/4153310434 >> 172.21.15.27:0/4153310434 conn(0x7f814c0777a0 msgr2=0x7f814c0fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:52.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.796+0000 7f8150cc3640 1 -- 172.21.15.27:0/4153310434 shutdown_connections 2024-04-25T00:15:52.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.796+0000 7f8150cc3640 1 -- 172.21.15.27:0/4153310434 wait complete. 2024-04-25T00:15:52.808 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:15:52.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.910+0000 7f201fa5d640 1 -- 172.21.15.27:0/2095643646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2018107720 msgr2=0x7f2018107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:52.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.910+0000 7f201fa5d640 1 --2- 172.21.15.27:0/2095643646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2018107720 0x7f2018107b00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f2014009930 tx=0x7f201402a420 comp rx=0 tx=0).stop 2024-04-25T00:15:52.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.910+0000 7f201fa5d640 1 -- 172.21.15.27:0/2095643646 shutdown_connections 2024-04-25T00:15:52.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.910+0000 7f201fa5d640 1 --2- 172.21.15.27:0/2095643646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20181020d0 0x7f201810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.910+0000 7f201fa5d640 1 --2- 172.21.15.27:0/2095643646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2018101710 0x7f2018101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.910+0000 7f201fa5d640 1 --2- 172.21.15.27:0/2095643646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2018107720 0x7f2018107b00 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.910+0000 7f201fa5d640 1 -- 172.21.15.27:0/2095643646 >> 172.21.15.27:0/2095643646 conn(0x7f2018078b00 msgr2=0x7f2018078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.910+0000 7f201fa5d640 1 -- 172.21.15.27:0/2095643646 shutdown_connections 2024-04-25T00:15:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.910+0000 7f201fa5d640 1 -- 172.21.15.27:0/2095643646 wait complete. 2024-04-25T00:15:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.911+0000 7f201fa5d640 1 Processor -- start 2024-04-25T00:15:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.911+0000 7f201fa5d640 1 -- start start 2024-04-25T00:15:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.911+0000 7f201fa5d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2018101710 0x7f2018198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.911+0000 7f201fa5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20181020d0 0x7f20181985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.911+0000 7f201fa5d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2018107720 0x7f201819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.911+0000 7f201fa5d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2018111b30 con 0x7f2018101710 2024-04-25T00:15:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.911+0000 7f201fa5d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f20181119b0 con 0x7f2018107720 2024-04-25T00:15:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.911+0000 7f201fa5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2018111cb0 con 0x7f20181020d0 2024-04-25T00:15:52.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.911+0000 7f201d7f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2018101710 0x7f2018198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:52.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.911+0000 7f201d7f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2018101710 0x7f2018198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47846/0 (socket says 172.21.15.27:47846) 2024-04-25T00:15:52.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.911+0000 7f201d7f9640 1 -- 172.21.15.27:0/3883020294 learned_addr learned my addr 172.21.15.27:0/3883020294 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:52.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.911+0000 7f201cff8640 1 --2- 172.21.15.27:0/3883020294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20181020d0 0x7f20181985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:52.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.911+0000 7f201dffa640 1 --2- 172.21.15.27:0/3883020294 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2018107720 0x7f201819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:52.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.911+0000 7f201d7f9640 1 -- 172.21.15.27:0/3883020294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20181020d0 msgr2=0x7f20181985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:52.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.911+0000 7f201d7f9640 1 --2- 172.21.15.27:0/3883020294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20181020d0 0x7f20181985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.911+0000 7f201d7f9640 1 -- 172.21.15.27:0/3883020294 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2018107720 msgr2=0x7f201819c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:52.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.911+0000 7f201d7f9640 1 --2- 172.21.15.27:0/3883020294 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2018107720 0x7f201819c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:52.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.912+0000 7f201d7f9640 1 -- 172.21.15.27:0/3883020294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201819d060 con 0x7f2018101710 2024-04-25T00:15:52.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.912+0000 7f201cff8640 1 --2- 172.21.15.27:0/3883020294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20181020d0 0x7f20181985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:52.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.912+0000 7f201dffa640 1 --2- 172.21.15.27:0/3883020294 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2018107720 0x7f201819c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:52.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.912+0000 7f201d7f9640 1 --2- 172.21.15.27:0/3883020294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2018101710 0x7f2018198070 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2014009230 tx=0x7f2014004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:52.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.912+0000 7f200e7fc640 1 -- 172.21.15.27:0/3883020294 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201403f030 con 0x7f2018101710 2024-04-25T00:15:52.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.912+0000 7f200e7fc640 1 -- 172.21.15.27:0/3883020294 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2014003ad0 con 0x7f2018101710 2024-04-25T00:15:52.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.912+0000 7f201fa5d640 1 -- 172.21.15.27:0/3883020294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f201819d2f0 con 0x7f2018101710 2024-04-25T00:15:52.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.912+0000 7f200e7fc640 1 -- 172.21.15.27:0/3883020294 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014035070 con 0x7f2018101710 2024-04-25T00:15:52.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.912+0000 7f201fa5d640 1 -- 172.21.15.27:0/3883020294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20181a4be0 con 0x7f2018101710 2024-04-25T00:15:52.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.913+0000 7f201fa5d640 1 -- 172.21.15.27:0/3883020294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20180689b0 con 0x7f2018101710 2024-04-25T00:15:52.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.914+0000 7f200e7fc640 1 -- 172.21.15.27:0/3883020294 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f2014042030 con 0x7f2018101710 2024-04-25T00:15:52.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.915+0000 7f200e7fc640 1 --2- 172.21.15.27:0/3883020294 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f20040a2ea0 0x7f20040a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:52.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.915+0000 7f201cff8640 1 --2- 172.21.15.27:0/3883020294 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f20040a2ea0 0x7f20040a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:52.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.915+0000 7f200e7fc640 1 -- 172.21.15.27:0/3883020294 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f20140f02c0 con 0x7f2018101710 2024-04-25T00:15:52.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.915+0000 7f201cff8640 1 --2- 172.21.15.27:0/3883020294 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f20040a2ea0 0x7f20040a5360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2008005d00 tx=0x7f2008009550 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:52.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:52.919+0000 7f200e7fc640 1 -- 172.21.15.27:0/3883020294 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f20140bc9b0 con 0x7f2018101710 2024-04-25T00:15:53.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.107+0000 7f201fa5d640 1 -- 172.21.15.27:0/3883020294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2018109c80 con 0x7f2018101710 2024-04-25T00:15:53.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.109+0000 7f200e7fc640 1 -- 172.21.15.27:0/3883020294 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25113 (secure 0 0 0) 0x7f20140b6e20 con 0x7f2018101710 2024-04-25T00:15:53.109 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:15:53.109 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:15:45.034931+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1885111026":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/2528869729":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/4085720546":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/236412562":"2024-04-26T00:15:45.034899+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/2161554778":"2024-04-26T00:15:45.034899+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6800/2161554778":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:15:53.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.111+0000 7f201fa5d640 1 -- 172.21.15.27:0/3883020294 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f20040a2ea0 msgr2=0x7f20040a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.111+0000 7f201fa5d640 1 --2- 172.21.15.27:0/3883020294 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f20040a2ea0 0x7f20040a5360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2008005d00 tx=0x7f2008009550 comp rx=0 tx=0).stop 2024-04-25T00:15:53.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.111+0000 7f201fa5d640 1 -- 172.21.15.27:0/3883020294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2018101710 msgr2=0x7f2018198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.111+0000 7f201fa5d640 1 --2- 172.21.15.27:0/3883020294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2018101710 0x7f2018198070 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2014009230 tx=0x7f2014004270 comp rx=0 tx=0).stop 2024-04-25T00:15:53.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.111+0000 7f201fa5d640 1 -- 172.21.15.27:0/3883020294 shutdown_connections 2024-04-25T00:15:53.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.111+0000 7f201fa5d640 1 --2- 172.21.15.27:0/3883020294 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f20040a2ea0 0x7f20040a5360 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.111+0000 7f201fa5d640 1 --2- 172.21.15.27:0/3883020294 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2018107720 0x7f201819c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.111+0000 7f201fa5d640 1 --2- 172.21.15.27:0/3883020294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20181020d0 0x7f20181985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.111+0000 7f201fa5d640 1 --2- 172.21.15.27:0/3883020294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2018101710 0x7f2018198070 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.111+0000 7f201fa5d640 1 -- 172.21.15.27:0/3883020294 >> 172.21.15.27:0/3883020294 conn(0x7f2018078b00 msgr2=0x7f2018104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:53.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.111+0000 7f201fa5d640 1 -- 172.21.15.27:0/3883020294 shutdown_connections 2024-04-25T00:15:53.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.111+0000 7f201fa5d640 1 -- 172.21.15.27:0/3883020294 wait complete. 2024-04-25T00:15:53.124 DEBUG:tasks.fwd_scrub:Yielding 2024-04-25T00:15:53.124 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T00:15:53.134 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T00:15:53.134 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T00:15:53.134 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-25T00:15:53.135 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-25T00:15:53.135 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T00:15:53.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:15:53.135 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T00:15:53.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f99083a9640 1 -- 172.21.15.27:0/2737064313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9900106650 msgr2=0x7f9900106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f99083a9640 1 --2- 172.21.15.27:0/2737064313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9900106650 0x7f9900106a30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f98fc009930 tx=0x7f98fc02a420 comp rx=0 tx=0).stop 2024-04-25T00:15:53.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f7639b29640 1 -- 172.21.15.27:0/880062441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76341066b0 msgr2=0x7f7634106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f7639b29640 1 --2- 172.21.15.27:0/880062441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76341066b0 0x7f7634106a90 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7624009930 tx=0x7f762402a420 comp rx=0 tx=0).stop 2024-04-25T00:15:53.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f7639b29640 1 -- 172.21.15.27:0/880062441 shutdown_connections 2024-04-25T00:15:53.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f7639b29640 1 --2- 172.21.15.27:0/880062441 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7634100fd0 0x7f763410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f7639b29640 1 --2- 172.21.15.27:0/880062441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7634100610 0x7f7634100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f7639b29640 1 --2- 172.21.15.27:0/880062441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76341066b0 0x7f7634106a90 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f7639b29640 1 -- 172.21.15.27:0/880062441 >> 172.21.15.27:0/880062441 conn(0x7f7634077740 msgr2=0x7f7634077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:53.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f99083a9640 1 -- 172.21.15.27:0/2737064313 shutdown_connections 2024-04-25T00:15:53.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f99083a9640 1 --2- 172.21.15.27:0/2737064313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9900101000 0x7f990010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f99083a9640 1 --2- 172.21.15.27:0/2737064313 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9900100640 0x7f9900100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f99083a9640 1 --2- 172.21.15.27:0/2737064313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9900106650 0x7f9900106a30 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f99083a9640 1 -- 172.21.15.27:0/2737064313 >> 172.21.15.27:0/2737064313 conn(0x7f9900077750 msgr2=0x7f9900077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:53.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f7639b29640 1 -- 172.21.15.27:0/880062441 shutdown_connections 2024-04-25T00:15:53.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f7639b29640 1 -- 172.21.15.27:0/880062441 wait complete. 2024-04-25T00:15:53.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f99083a9640 1 -- 172.21.15.27:0/2737064313 shutdown_connections 2024-04-25T00:15:53.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f99083a9640 1 -- 172.21.15.27:0/2737064313 wait complete. 2024-04-25T00:15:53.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f7639b29640 1 Processor -- start 2024-04-25T00:15:53.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f99083a9640 1 Processor -- start 2024-04-25T00:15:53.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.241+0000 7f99083a9640 1 -- start start 2024-04-25T00:15:53.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f7639b29640 1 -- start start 2024-04-25T00:15:53.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f99083a9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9900100640 0x7f9900198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f99083a9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9900101000 0x7f9900199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f99083a9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9900106650 0x7f99001a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f99083a9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9900110a60 con 0x7f9900101000 2024-04-25T00:15:53.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f99083a9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f99001108e0 con 0x7f9900106650 2024-04-25T00:15:53.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f99083a9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9900110be0 con 0x7f9900100640 2024-04-25T00:15:53.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f9905944640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9900101000 0x7f9900199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f9905944640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9900101000 0x7f9900199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47864/0 (socket says 172.21.15.27:47864) 2024-04-25T00:15:53.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f9905944640 1 -- 172.21.15.27:0/1629537310 learned_addr learned my addr 172.21.15.27:0/1629537310 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:53.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f7639b29640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7634100610 0x7f763410df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f7639b29640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7634100fd0 0x7f763410e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f7639b29640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76341066b0 0x7f763410e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f7639b29640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7634111b80 con 0x7f7634100fd0 2024-04-25T00:15:53.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f9906145640 1 --2- 172.21.15.27:0/1629537310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9900100640 0x7f9900198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f9906946640 1 --2- 172.21.15.27:0/1629537310 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9900106650 0x7f99001a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f7632ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7634100fd0 0x7f763410e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f7639b29640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7634111a00 con 0x7f76341066b0 2024-04-25T00:15:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f7639b29640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7634111d00 con 0x7f7634100610 2024-04-25T00:15:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f7632ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7634100fd0 0x7f763410e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47868/0 (socket says 172.21.15.27:47868) 2024-04-25T00:15:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f7632ffd640 1 -- 172.21.15.27:0/3709776289 learned_addr learned my addr 172.21.15.27:0/3709776289 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f7633fff640 1 --2- 172.21.15.27:0/3709776289 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76341066b0 0x7f763410e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f76337fe640 1 --2- 172.21.15.27:0/3709776289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7634100610 0x7f763410df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f9905944640 1 -- 172.21.15.27:0/1629537310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9900100640 msgr2=0x7f9900198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f9905944640 1 --2- 172.21.15.27:0/1629537310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9900100640 0x7f9900198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f9905944640 1 -- 172.21.15.27:0/1629537310 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9900106650 msgr2=0x7f99001a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f9905944640 1 --2- 172.21.15.27:0/1629537310 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9900106650 0x7f99001a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f9905944640 1 -- 172.21.15.27:0/1629537310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99001a43c0 con 0x7f9900101000 2024-04-25T00:15:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f9906145640 1 --2- 172.21.15.27:0/1629537310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9900100640 0x7f9900198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f7632ffd640 1 -- 172.21.15.27:0/3709776289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7634100610 msgr2=0x7f763410df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f7632ffd640 1 --2- 172.21.15.27:0/3709776289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7634100610 0x7f763410df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f7632ffd640 1 -- 172.21.15.27:0/3709776289 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76341066b0 msgr2=0x7f763410e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f7632ffd640 1 --2- 172.21.15.27:0/3709776289 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76341066b0 0x7f763410e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f7632ffd640 1 -- 172.21.15.27:0/3709776289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76341a64e0 con 0x7f7634100fd0 2024-04-25T00:15:53.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.242+0000 7f9905944640 1 --2- 172.21.15.27:0/1629537310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9900101000 0x7f9900199340 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f98f000d950 tx=0x7f98f000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:53.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.243+0000 7f7632ffd640 1 --2- 172.21.15.27:0/3709776289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7634100fd0 0x7f763410e4b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f762800e9f0 tx=0x7f762800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:53.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.243+0000 7f98f77fe640 1 -- 172.21.15.27:0/1629537310 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f0017030 con 0x7f9900101000 2024-04-25T00:15:53.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.243+0000 7f98f77fe640 1 -- 172.21.15.27:0/1629537310 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98f0004020 con 0x7f9900101000 2024-04-25T00:15:53.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.243+0000 7f99083a9640 1 -- 172.21.15.27:0/1629537310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99001a45f0 con 0x7f9900101000 2024-04-25T00:15:53.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.243+0000 7f98f77fe640 1 -- 172.21.15.27:0/1629537310 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f00047a0 con 0x7f9900101000 2024-04-25T00:15:53.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.243+0000 7f99083a9640 1 -- 172.21.15.27:0/1629537310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99001a4b60 con 0x7f9900101000 2024-04-25T00:15:53.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.243+0000 7f7630ff9640 1 -- 172.21.15.27:0/3709776289 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762800cd80 con 0x7f7634100fd0 2024-04-25T00:15:53.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.243+0000 7f7630ff9640 1 -- 172.21.15.27:0/3709776289 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7628005500 con 0x7f7634100fd0 2024-04-25T00:15:53.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.243+0000 7f7630ff9640 1 -- 172.21.15.27:0/3709776289 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7628010690 con 0x7f7634100fd0 2024-04-25T00:15:53.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.243+0000 7f7639b29640 1 -- 172.21.15.27:0/3709776289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76341a67d0 con 0x7f7634100fd0 2024-04-25T00:15:53.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.243+0000 7f7639b29640 1 -- 172.21.15.27:0/3709776289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76341a6d20 con 0x7f7634100fd0 2024-04-25T00:15:53.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.244+0000 7f99083a9640 1 -- 172.21.15.27:0/1629537310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f990004f500 con 0x7f9900101000 2024-04-25T00:15:53.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.245+0000 7f98f77fe640 1 -- 172.21.15.27:0/1629537310 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f98f00041d0 con 0x7f9900101000 2024-04-25T00:15:53.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.246+0000 7f98f77fe640 1 --2- 172.21.15.27:0/1629537310 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f98ec0a2f40 0x7f98ec0a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.246+0000 7f763c83a740 1 Processor -- start 2024-04-25T00:15:53.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.246+0000 7f7630ff9640 1 -- 172.21.15.27:0/3709776289 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f76280046c0 con 0x7f7634100fd0 2024-04-25T00:15:53.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.246+0000 7f763c83a740 1 -- start start 2024-04-25T00:15:53.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.246+0000 7f9906145640 1 --2- 172.21.15.27:0/1629537310 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f98ec0a2f40 0x7f98ec0a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.246+0000 7f98f77fe640 1 -- 172.21.15.27:0/1629537310 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f98f00cf3f0 con 0x7f9900101000 2024-04-25T00:15:53.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.246+0000 7f763c83a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618dacd0980 0x5618daccecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.246+0000 7f763c83a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618daccf290 0x5618daccf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.246+0000 7f763c83a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacda010 0x5618dacdc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.246+0000 7f763c83a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5618dac332c0 con 0x5618dacd0980 2024-04-25T00:15:53.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.246+0000 7f763c83a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5618dabdb910 con 0x5618daccf290 2024-04-25T00:15:53.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.246+0000 7f763c83a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5618dac2f280 con 0x5618dacda010 2024-04-25T00:15:53.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.246+0000 7f7633fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618dacd0980 0x5618daccecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.246+0000 7f7633fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618dacd0980 0x5618daccecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47870/0 (socket says 172.21.15.27:47870) 2024-04-25T00:15:53.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.246+0000 7f7633fff640 1 -- 172.21.15.27:0/2540028802 learned_addr learned my addr 172.21.15.27:0/2540028802 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:53.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.246+0000 7f7632ffd640 1 --2- 172.21.15.27:0/2540028802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacda010 0x5618dacdc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.246+0000 7f76337fe640 1 --2- 172.21.15.27:0/2540028802 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618daccf290 0x5618daccf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.247+0000 7f9906145640 1 --2- 172.21.15.27:0/1629537310 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f98ec0a2f40 0x7f98ec0a5400 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f98fc000c00 tx=0x7f98fc005a20 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:53.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.247+0000 7f7633fff640 1 -- 172.21.15.27:0/2540028802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacda010 msgr2=0x5618dacdc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:53.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.247+0000 7f7633fff640 1 --2- 172.21.15.27:0/2540028802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacda010 0x5618dacdc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.247+0000 7f7630ff9640 1 --2- 172.21.15.27:0/3709776289 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f761c0a2ea0 0x7f761c0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.247+0000 7f7633fff640 1 -- 172.21.15.27:0/2540028802 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618daccf290 msgr2=0x5618daccf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:53.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.247+0000 7f7633fff640 1 --2- 172.21.15.27:0/2540028802 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618daccf290 0x5618daccf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.247+0000 7f7633fff640 1 -- 172.21.15.27:0/2540028802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618dabecfb0 con 0x5618dacd0980 2024-04-25T00:15:53.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.247+0000 7f76337fe640 1 --2- 172.21.15.27:0/2540028802 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618daccf290 0x5618daccf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:53.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.247+0000 7f7633fff640 1 --2- 172.21.15.27:0/3709776289 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f761c0a2ea0 0x7f761c0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.247+0000 7f7630ff9640 1 -- 172.21.15.27:0/3709776289 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f76280cfec0 con 0x7f7634100fd0 2024-04-25T00:15:53.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.247+0000 7f7633fff640 1 --2- 172.21.15.27:0/2540028802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618dacd0980 0x5618daccecc0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f762c009f50 tx=0x7f762c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4b338cdd1e350a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:53.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.247+0000 7f7633fff640 1 --2- 172.21.15.27:0/3709776289 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f761c0a2ea0 0x7f761c0a5360 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f762c012fd0 tx=0x7f762c021000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:53.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.251+0000 7f7619ffb640 1 -- 172.21.15.27:0/2540028802 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c00f680 con 0x5618dacd0980 2024-04-25T00:15:53.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.251+0000 7f7619ffb640 1 -- 172.21.15.27:0/2540028802 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f762c003cb0 con 0x5618dacd0980 2024-04-25T00:15:53.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.251+0000 7f7619ffb640 1 -- 172.21.15.27:0/2540028802 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c01bc20 con 0x5618dacd0980 2024-04-25T00:15:53.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.251+0000 7f98f77fe640 1 -- 172.21.15.27:0/1629537310 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f98f009ba50 con 0x7f9900101000 2024-04-25T00:15:53.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.251+0000 7f763c83a740 1 -- 172.21.15.27:0/2540028802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618dacd0980 msgr2=0x5618daccecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.251+0000 7f763c83a740 1 --2- 172.21.15.27:0/2540028802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618dacd0980 0x5618daccecc0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f762c009f50 tx=0x7f762c0118d0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.252+0000 7f763c83a740 1 -- 172.21.15.27:0/2540028802 shutdown_connections 2024-04-25T00:15:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.252+0000 7f763c83a740 1 --2- 172.21.15.27:0/2540028802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacda010 0x5618dacdc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.252+0000 7f763c83a740 1 --2- 172.21.15.27:0/2540028802 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618daccf290 0x5618daccf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.252+0000 7f763c83a740 1 --2- 172.21.15.27:0/2540028802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618dacd0980 0x5618daccecc0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.252+0000 7f763c83a740 1 -- 172.21.15.27:0/2540028802 >> 172.21.15.27:0/2540028802 conn(0x5618dac97950 msgr2=0x5618daccb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.252+0000 7f763c83a740 1 -- 172.21.15.27:0/2540028802 shutdown_connections 2024-04-25T00:15:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.252+0000 7f763c83a740 1 -- 172.21.15.27:0/2540028802 wait complete. 2024-04-25T00:15:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.252+0000 7f763c83a740 1 -- start start 2024-04-25T00:15:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.252+0000 7f763c83a740 1 Processor -- start 2024-04-25T00:15:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f763c83a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618daccf290 0x5618dacd6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f763c83a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacd0980 0x5618dacd74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f763c83a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618dacda010 0x5618dacd79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f763c83a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5618dacdcea0 con 0x5618daccf290 2024-04-25T00:15:53.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f763c83a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5618dabeee90 con 0x5618dacda010 2024-04-25T00:15:53.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f763c83a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5618dacdd020 con 0x5618dacd0980 2024-04-25T00:15:53.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f763c83a740 10 client.0 _ll_register_callbacks cb 0x5618dac2a970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:53.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f76337fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618daccf290 0x5618dacd6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f76337fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618daccf290 0x5618dacd6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47882/0 (socket says 172.21.15.27:47882) 2024-04-25T00:15:53.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f76337fe640 1 -- 172.21.15.27:0/1640293195 learned_addr learned my addr 172.21.15.27:0/1640293195 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:53.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f7633fff640 1 --2- 172.21.15.27:0/1640293195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618dacda010 0x5618dacd79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f7632ffd640 1 --2- 172.21.15.27:0/1640293195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacd0980 0x5618dacd74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f76337fe640 1 -- 172.21.15.27:0/1640293195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacd0980 msgr2=0x5618dacd74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:53.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f76337fe640 1 --2- 172.21.15.27:0/1640293195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacd0980 0x5618dacd74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f76337fe640 1 -- 172.21.15.27:0/1640293195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618dacda010 msgr2=0x5618dacd79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:53.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f76337fe640 1 --2- 172.21.15.27:0/1640293195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618dacda010 0x5618dacd79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f76337fe640 1 -- 172.21.15.27:0/1640293195 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618daa0f7e0 con 0x5618daccf290 2024-04-25T00:15:53.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f7632ffd640 1 --2- 172.21.15.27:0/1640293195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacd0980 0x5618dacd74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:53.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f76337fe640 1 --2- 172.21.15.27:0/1640293195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618daccf290 0x5618dacd6f70 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7624000c00 tx=0x7f76240052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11bc6f168d8dfc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:53.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.253+0000 7f7612ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:53.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.254+0000 7f7612ffd640 1 -- 172.21.15.27:0/1640293195 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7624035030 con 0x5618daccf290 2024-04-25T00:15:53.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.254+0000 7f7612ffd640 1 -- 172.21.15.27:0/1640293195 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f76240048c0 con 0x5618daccf290 2024-04-25T00:15:53.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.254+0000 7f763c83a740 1 -- 172.21.15.27:0/1640293195 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5618dac075a0 con 0x5618daccf290 2024-04-25T00:15:53.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.254+0000 7f7612ffd640 1 -- 172.21.15.27:0/1640293195 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7624005d50 con 0x5618daccf290 2024-04-25T00:15:53.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.254+0000 7f763c83a740 10 client.15786 fetch_fsmap learned FSMap version 29 2024-04-25T00:15:53.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.254+0000 7f763c83a740 1 -- 172.21.15.27:0/1640293195 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5618daa6ff70 con 0x5618daccf290 2024-04-25T00:15:53.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.254+0000 7f7612ffd640 1 -- 172.21.15.27:0/1640293195 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5618dac075a0 con 0x5618daccf290 2024-04-25T00:15:53.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.254+0000 7f7612ffd640 1 -- 172.21.15.27:0/1640293195 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7624003b40 con 0x5618daccf290 2024-04-25T00:15:53.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.254+0000 7f763c83a740 10 client.15786 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:15:53.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.254+0000 7f763c83a740 10 client.15786 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:15:53.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.254+0000 7f763c83a740 20 client.15786 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:53.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.254+0000 7f763c83a740 1 --2- 172.21.15.27:0/1640293195 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5618dacea190 0x5618dacec600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.254+0000 7f763c83a740 4 client.15786 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:15:53.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.254+0000 7f763c83a740 1 -- 172.21.15.27:0/1640293195 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618dac40800 con 0x5618dacea190 2024-04-25T00:15:53.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.255+0000 7f7632ffd640 1 --2- 172.21.15.27:0/1640293195 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5618dacea190 0x5618dacec600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.255+0000 7f7632ffd640 1 --2- 172.21.15.27:0/1640293195 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5618dacea190 0x5618dacec600 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=615b52ad5fb6191d server_cookie=8ec1036c888de0d6 in_seq=0 out_seq=0 2024-04-25T00:15:53.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.255+0000 7f7612ffd640 10 client.15786 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:15:53.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.256+0000 7f7612ffd640 1 -- 172.21.15.27:0/1640293195 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5618dac40800 con 0x5618dacea190 2024-04-25T00:15:53.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.256+0000 7f7612ffd640 10 client.15786 handle_command_reply: tid=0 2024-04-25T00:15:53.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.256+0000 7f763c83a740 1 client.15786 shutdown 2024-04-25T00:15:53.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.256+0000 7f763c83a740 1 -- 172.21.15.27:0/1640293195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618daccf290 msgr2=0x5618dacd6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.256+0000 7f763c83a740 1 --2- 172.21.15.27:0/1640293195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618daccf290 0x5618dacd6f70 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7624000c00 tx=0x7f76240052f0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.256+0000 7f763c83a740 1 -- 172.21.15.27:0/1640293195 shutdown_connections 2024-04-25T00:15:53.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.256+0000 7f763c83a740 1 --2- 172.21.15.27:0/1640293195 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5618dacea190 0x5618dacec600 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.256+0000 7f763c83a740 1 --2- 172.21.15.27:0/1640293195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618dacda010 0x5618dacd79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.256+0000 7f763c83a740 1 --2- 172.21.15.27:0/1640293195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacd0980 0x5618dacd74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.256+0000 7f7612ffd640 0 client.15786 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:15:53.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.256+0000 7f763c83a740 1 --2- 172.21.15.27:0/1640293195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618daccf290 0x5618dacd6f70 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.256+0000 7f763c83a740 1 -- 172.21.15.27:0/1640293195 >> 172.21.15.27:0/1640293195 conn(0x5618dac97950 msgr2=0x5618dacd15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:53.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.256+0000 7f763c83a740 1 -- 172.21.15.27:0/1640293195 shutdown_connections 2024-04-25T00:15:53.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.256+0000 7f763c83a740 1 -- 172.21.15.27:0/1640293195 wait complete. 2024-04-25T00:15:53.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.256+0000 7f763c83a740 20 client.15786 trim_cache size 0 max 16384 2024-04-25T00:15:53.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.294+0000 7f763c83a740 1 Processor -- start 2024-04-25T00:15:53.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.294+0000 7f763c83a740 1 -- start start 2024-04-25T00:15:53.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.294+0000 7f763c83a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618daccf290 0x5618dace5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.294+0000 7f763c83a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacd0980 0x5618dacf1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.294+0000 7f763c83a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618dacda010 0x5618dacf2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.294+0000 7f763c83a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f76341b0de0 con 0x5618daccf290 2024-04-25T00:15:53.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.294+0000 7f763c83a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f76341b2920 con 0x5618dacda010 2024-04-25T00:15:53.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.294+0000 7f763c83a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f76341b14b0 con 0x5618dacd0980 2024-04-25T00:15:53.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.295+0000 7f76337fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618daccf290 0x5618dace5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.295+0000 7f76337fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618daccf290 0x5618dace5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47886/0 (socket says 172.21.15.27:47886) 2024-04-25T00:15:53.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.295+0000 7f76337fe640 1 -- 172.21.15.27:0/3320529936 learned_addr learned my addr 172.21.15.27:0/3320529936 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:53.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.295+0000 7f7632ffd640 1 --2- 172.21.15.27:0/3320529936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacd0980 0x5618dacf1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.295+0000 7f76337fe640 1 -- 172.21.15.27:0/3320529936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacd0980 msgr2=0x5618dacf1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.295+0000 7f76337fe640 1 --2- 172.21.15.27:0/3320529936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacd0980 0x5618dacf1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.295+0000 7f7633fff640 1 --2- 172.21.15.27:0/3320529936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618dacda010 0x5618dacf2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.295+0000 7f76337fe640 1 -- 172.21.15.27:0/3320529936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618dacda010 msgr2=0x5618dacf2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.295+0000 7f76337fe640 1 --2- 172.21.15.27:0/3320529936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618dacda010 0x5618dacf2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.295+0000 7f76337fe640 1 -- 172.21.15.27:0/3320529936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618da907170 con 0x5618daccf290 2024-04-25T00:15:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.295+0000 7f7633fff640 1 --2- 172.21.15.27:0/3320529936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618dacda010 0x5618dacf2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.295+0000 7f76337fe640 1 --2- 172.21.15.27:0/3320529936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618daccf290 0x5618dace5df0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7624006010 tx=0x7f7624009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c2f30432b57911c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.295+0000 7f76197fa640 1 -- 172.21.15.27:0/3320529936 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7624035070 con 0x5618daccf290 2024-04-25T00:15:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.295+0000 7f76197fa640 1 -- 172.21.15.27:0/3320529936 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f762403f070 con 0x5618daccf290 2024-04-25T00:15:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.295+0000 7f76197fa640 1 -- 172.21.15.27:0/3320529936 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7624031ba0 con 0x5618daccf290 2024-04-25T00:15:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.296+0000 7f763c83a740 1 -- 172.21.15.27:0/3320529936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618daccf290 msgr2=0x5618dace5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.296+0000 7f763c83a740 1 --2- 172.21.15.27:0/3320529936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618daccf290 0x5618dace5df0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7624006010 tx=0x7f7624009780 comp rx=0 tx=0).stop 2024-04-25T00:15:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.296+0000 7f763c83a740 1 -- 172.21.15.27:0/3320529936 shutdown_connections 2024-04-25T00:15:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.296+0000 7f763c83a740 1 --2- 172.21.15.27:0/3320529936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618dacda010 0x5618dacf2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.296+0000 7f763c83a740 1 --2- 172.21.15.27:0/3320529936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacd0980 0x5618dacf1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.296+0000 7f763c83a740 1 --2- 172.21.15.27:0/3320529936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618daccf290 0x5618dace5df0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.296+0000 7f763c83a740 1 -- 172.21.15.27:0/3320529936 >> 172.21.15.27:0/3320529936 conn(0x5618dac97950 msgr2=0x5618dacca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.296+0000 7f763c83a740 1 -- 172.21.15.27:0/3320529936 shutdown_connections 2024-04-25T00:15:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.296+0000 7f763c83a740 1 -- 172.21.15.27:0/3320529936 wait complete. 2024-04-25T00:15:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.297+0000 7f763c83a740 1 -- start start 2024-04-25T00:15:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.297+0000 7f763c83a740 1 Processor -- start 2024-04-25T00:15:53.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.297+0000 7f763c83a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618daccf290 0x5618dacdf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.297+0000 7f763c83a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618dacd0980 0x5618dacdf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.297+0000 7f763c83a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacda010 0x5618dacdfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.297+0000 7f763c83a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f76341b2250 con 0x5618dacd0980 2024-04-25T00:15:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.297+0000 7f763c83a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f76341b1b80 con 0x5618daccf290 2024-04-25T00:15:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.297+0000 7f763c83a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f76341aeda0 con 0x5618dacda010 2024-04-25T00:15:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.297+0000 7f763c83a740 10 client.0 _ll_register_callbacks cb 0x7f762c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.297+0000 7f7632ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618dacd0980 0x5618dacdf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.297+0000 7f7632ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618dacd0980 0x5618dacdf810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47900/0 (socket says 172.21.15.27:47900) 2024-04-25T00:15:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.297+0000 7f7632ffd640 1 -- 172.21.15.27:0/272970455 learned_addr learned my addr 172.21.15.27:0/272970455 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.297+0000 7f7633fff640 1 --2- 172.21.15.27:0/272970455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacda010 0x5618dacdfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.298+0000 7f76337fe640 1 --2- 172.21.15.27:0/272970455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618daccf290 0x5618dacdf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.298+0000 7f7632ffd640 1 -- 172.21.15.27:0/272970455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacda010 msgr2=0x5618dacdfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:53.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.298+0000 7f7632ffd640 1 --2- 172.21.15.27:0/272970455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacda010 0x5618dacdfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.298+0000 7f7632ffd640 1 -- 172.21.15.27:0/272970455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618daccf290 msgr2=0x5618dacdf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:53.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.298+0000 7f7632ffd640 1 --2- 172.21.15.27:0/272970455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618daccf290 0x5618dacdf2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.298+0000 7f7632ffd640 1 -- 172.21.15.27:0/272970455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618dad50900 con 0x5618dacd0980 2024-04-25T00:15:53.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.298+0000 7f7632ffd640 1 --2- 172.21.15.27:0/272970455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618dacd0980 0x5618dacdf810 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f76280987f0 tx=0x7f76280d3910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c994629733645f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:53.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.298+0000 7f7612ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:53.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.298+0000 7f7612ffd640 1 -- 172.21.15.27:0/272970455 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762809c520 con 0x5618dacd0980 2024-04-25T00:15:53.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.298+0000 7f763c83a740 1 -- 172.21.15.27:0/272970455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f762403f070 con 0x5618dacd0980 2024-04-25T00:15:53.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.298+0000 7f7612ffd640 1 -- 172.21.15.27:0/272970455 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f76280a4600 con 0x5618dacd0980 2024-04-25T00:15:53.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.298+0000 7f7612ffd640 1 -- 172.21.15.27:0/272970455 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f762809be20 con 0x5618dacd0980 2024-04-25T00:15:53.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.298+0000 7f763c83a740 10 client.15792 fetch_fsmap learned FSMap version 29 2024-04-25T00:15:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.298+0000 7f763c83a740 1 -- 172.21.15.27:0/272970455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5618dad51bd0 con 0x5618dacd0980 2024-04-25T00:15:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.298+0000 7f7612ffd640 1 -- 172.21.15.27:0/272970455 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f762403f070 con 0x5618dacd0980 2024-04-25T00:15:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.299+0000 7f7612ffd640 1 -- 172.21.15.27:0/272970455 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f76280a4ce0 con 0x5618dacd0980 2024-04-25T00:15:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.299+0000 7f763c83a740 10 client.15792 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:15:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.299+0000 7f763c83a740 10 client.15792 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:15:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.299+0000 7f763c83a740 20 client.15792 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.299+0000 7f763c83a740 1 --2- 172.21.15.27:0/272970455 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5618dad52280 0x5618dad546a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.299+0000 7f763c83a740 4 client.15792 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T00:15:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.299+0000 7f763c83a740 1 -- 172.21.15.27:0/272970455 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5618dac3ab50 con 0x5618dad52280 2024-04-25T00:15:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.299+0000 7f76337fe640 1 --2- 172.21.15.27:0/272970455 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5618dad52280 0x5618dad546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.299+0000 7f76337fe640 1 --2- 172.21.15.27:0/272970455 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5618dad52280 0x5618dad546a0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a351fd03aa2ea23a server_cookie=50d8c53a4b974bf0 in_seq=0 out_seq=0 2024-04-25T00:15:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.299+0000 7f7612ffd640 10 client.15792 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:15:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.300+0000 7f7612ffd640 1 -- 172.21.15.27:0/272970455 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5618dac3ab50 con 0x5618dad52280 2024-04-25T00:15:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.301+0000 7f7612ffd640 10 client.15792 handle_command_reply: tid=0 2024-04-25T00:15:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.301+0000 7f763c83a740 1 client.15792 shutdown 2024-04-25T00:15:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.301+0000 7f763c83a740 1 -- 172.21.15.27:0/272970455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618dacd0980 msgr2=0x5618dacdf810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.301+0000 7f763c83a740 1 --2- 172.21.15.27:0/272970455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618dacd0980 0x5618dacdf810 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f76280987f0 tx=0x7f76280d3910 comp rx=0 tx=0).stop 2024-04-25T00:15:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.301+0000 7f763c83a740 1 -- 172.21.15.27:0/272970455 shutdown_connections 2024-04-25T00:15:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.301+0000 7f763c83a740 1 --2- 172.21.15.27:0/272970455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618dacda010 0x5618dacdfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.301+0000 7f763c83a740 1 --2- 172.21.15.27:0/272970455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5618dacd0980 0x5618dacdf810 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.301+0000 7f763c83a740 1 --2- 172.21.15.27:0/272970455 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5618dad52280 0x5618dad546a0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.301+0000 7f763c83a740 1 --2- 172.21.15.27:0/272970455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5618daccf290 0x5618dacdf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.301+0000 7f7612ffd640 0 client.15792 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:15:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.301+0000 7f763c83a740 1 -- 172.21.15.27:0/272970455 >> 172.21.15.27:0/272970455 conn(0x5618dac97950 msgr2=0x5618dacd0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:53.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.301+0000 7f763c83a740 1 -- 172.21.15.27:0/272970455 shutdown_connections 2024-04-25T00:15:53.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.301+0000 7f763c83a740 1 -- 172.21.15.27:0/272970455 wait complete. 2024-04-25T00:15:53.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.301+0000 7f763c83a740 20 client.15792 trim_cache size 0 max 16384 2024-04-25T00:15:53.310 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:15:53.314 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T00:15:53.314 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "b6f50739-b6ab-49da-9f05-cff1bc7fd76e", 2024-04-25T00:15:53.315 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T00:15:53.315 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:15:53.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.302+0000 7f7639b29640 1 -- 172.21.15.27:0/3709776289 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f761c0a2ea0 msgr2=0x7f761c0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.302+0000 7f7639b29640 1 --2- 172.21.15.27:0/3709776289 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f761c0a2ea0 0x7f761c0a5360 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f762c012fd0 tx=0x7f762c021000 comp rx=0 tx=0).stop 2024-04-25T00:15:53.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.302+0000 7f7639b29640 1 -- 172.21.15.27:0/3709776289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7634100fd0 msgr2=0x7f763410e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.302+0000 7f7639b29640 1 --2- 172.21.15.27:0/3709776289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7634100fd0 0x7f763410e4b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f762800e9f0 tx=0x7f762800edf0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.302+0000 7f7639b29640 1 -- 172.21.15.27:0/3709776289 shutdown_connections 2024-04-25T00:15:53.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.302+0000 7f7639b29640 1 --2- 172.21.15.27:0/3709776289 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f761c0a2ea0 0x7f761c0a5360 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.302+0000 7f7639b29640 1 --2- 172.21.15.27:0/3709776289 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76341066b0 0x7f763410e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.302+0000 7f7639b29640 1 --2- 172.21.15.27:0/3709776289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7634100fd0 0x7f763410e4b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.302+0000 7f7639b29640 1 --2- 172.21.15.27:0/3709776289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7634100610 0x7f763410df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.302+0000 7f7639b29640 1 -- 172.21.15.27:0/3709776289 >> 172.21.15.27:0/3709776289 conn(0x7f7634077740 msgr2=0x7f76340ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:53.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.302+0000 7f7639b29640 1 -- 172.21.15.27:0/3709776289 shutdown_connections 2024-04-25T00:15:53.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.302+0000 7f7639b29640 1 -- 172.21.15.27:0/3709776289 wait complete. 2024-04-25T00:15:53.318 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:15:53.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:15:53.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:53 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/406807408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:53.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:53 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/284151721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:53.340 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:53 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4153310434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:53.418 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:53 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/406807408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:53.418 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:53 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/284151721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:53.418 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:53 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4153310434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:53.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.423+0000 7f9b30872640 1 -- 172.21.15.27:0/1458302154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b2c101720 msgr2=0x7f9b2c101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.423+0000 7f9b30872640 1 --2- 172.21.15.27:0/1458302154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b2c101720 0x7f9b2c101ba0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f9b14009930 tx=0x7f9b1402a420 comp rx=0 tx=0).stop 2024-04-25T00:15:53.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.423+0000 7f9b30872640 1 -- 172.21.15.27:0/1458302154 shutdown_connections 2024-04-25T00:15:53.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.423+0000 7f9b30872640 1 --2- 172.21.15.27:0/1458302154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b2c1020e0 0x7f9b2c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.423+0000 7f9b30872640 1 --2- 172.21.15.27:0/1458302154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b2c101720 0x7f9b2c101ba0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.423+0000 7f9b30872640 1 --2- 172.21.15.27:0/1458302154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b2c107730 0x7f9b2c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.423+0000 7f9b30872640 1 -- 172.21.15.27:0/1458302154 >> 172.21.15.27:0/1458302154 conn(0x7f9b2c078b40 msgr2=0x7f9b2c078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:53.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.423+0000 7f9b30872640 1 -- 172.21.15.27:0/1458302154 shutdown_connections 2024-04-25T00:15:53.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.423+0000 7f9b30872640 1 -- 172.21.15.27:0/1458302154 wait complete. 2024-04-25T00:15:53.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.424+0000 7f9b30872640 1 Processor -- start 2024-04-25T00:15:53.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.424+0000 7f9b30872640 1 -- start start 2024-04-25T00:15:53.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.424+0000 7f9b30872640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b2c101720 0x7f9b2c073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.424+0000 7f9b30872640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b2c1020e0 0x7f9b2c073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.424+0000 7f9b30872640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b2c107730 0x7f9b2c0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.424+0000 7f9b30872640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c111b00 con 0x7f9b2c107730 2024-04-25T00:15:53.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.424+0000 7f9b30872640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c111980 con 0x7f9b2c1020e0 2024-04-25T00:15:53.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.424+0000 7f9b30872640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c111c80 con 0x7f9b2c101720 2024-04-25T00:15:53.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.425+0000 7f9b2ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b2c107730 0x7f9b2c0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.425+0000 7f9b2ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b2c107730 0x7f9b2c0744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47930/0 (socket says 172.21.15.27:47930) 2024-04-25T00:15:53.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.425+0000 7f9b2ad9d640 1 -- 172.21.15.27:0/901235154 learned_addr learned my addr 172.21.15.27:0/901235154 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:53.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.425+0000 7f9b21d9b640 1 --2- 172.21.15.27:0/901235154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b2c1020e0 0x7f9b2c073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.425+0000 7f9b2a59c640 1 --2- 172.21.15.27:0/901235154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b2c101720 0x7f9b2c073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.425+0000 7f9b2ad9d640 1 -- 172.21.15.27:0/901235154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b2c101720 msgr2=0x7f9b2c073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.425+0000 7f9b2ad9d640 1 --2- 172.21.15.27:0/901235154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b2c101720 0x7f9b2c073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.425+0000 7f9b2ad9d640 1 -- 172.21.15.27:0/901235154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b2c1020e0 msgr2=0x7f9b2c073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.425+0000 7f9b2ad9d640 1 --2- 172.21.15.27:0/901235154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b2c1020e0 0x7f9b2c073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.425+0000 7f9b2ad9d640 1 -- 172.21.15.27:0/901235154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b2c1a6540 con 0x7f9b2c107730 2024-04-25T00:15:53.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.425+0000 7f9b2ad9d640 1 --2- 172.21.15.27:0/901235154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b2c107730 0x7f9b2c0744c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9b2400eb10 tx=0x7f9b2400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:53.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.425+0000 7f9b23fff640 1 -- 172.21.15.27:0/901235154 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b24004b20 con 0x7f9b2c107730 2024-04-25T00:15:53.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.425+0000 7f9b30872640 1 -- 172.21.15.27:0/901235154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b2c1a6830 con 0x7f9b2c107730 2024-04-25T00:15:53.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.425+0000 7f9b23fff640 1 -- 172.21.15.27:0/901235154 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b24004cc0 con 0x7f9b2c107730 2024-04-25T00:15:53.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.425+0000 7f9b23fff640 1 -- 172.21.15.27:0/901235154 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b24010760 con 0x7f9b2c107730 2024-04-25T00:15:53.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.425+0000 7f9b30872640 1 -- 172.21.15.27:0/901235154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b2c1a6d00 con 0x7f9b2c107730 2024-04-25T00:15:53.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.428+0000 7f9b3357e740 1 Processor -- start 2024-04-25T00:15:53.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.428+0000 7f9b23fff640 1 -- 172.21.15.27:0/901235154 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f9b24005020 con 0x7f9b2c107730 2024-04-25T00:15:53.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.428+0000 7f9b3357e740 1 -- start start 2024-04-25T00:15:53.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.428+0000 7f9b3357e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266966630 0x55c266966a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.428+0000 7f9b3357e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266966f50 0x55c2669710e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.428+0000 7f9b3357e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266971620 0x55c266973a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.428+0000 7f9b3357e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c2668cb6b0 con 0x55c266971620 2024-04-25T00:15:53.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.428+0000 7f9b3357e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c266873880 con 0x55c266966f50 2024-04-25T00:15:53.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.428+0000 7f9b3357e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c2668c7670 con 0x55c266966630 2024-04-25T00:15:53.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.428+0000 7f9b21d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266971620 0x55c266973a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.428+0000 7f9b21d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266971620 0x55c266973a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47940/0 (socket says 172.21.15.27:47940) 2024-04-25T00:15:53.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.428+0000 7f9b21d9b640 1 -- 172.21.15.27:0/404339517 learned_addr learned my addr 172.21.15.27:0/404339517 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:53.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.428+0000 7f9b2a59c640 1 --2- 172.21.15.27:0/404339517 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266966f50 0x55c2669710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.429+0000 7f9b21d9b640 1 --2- 172.21.15.27:0/404339517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266966630 0x55c266966a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.429+0000 7f9b23fff640 1 --2- 172.21.15.27:0/901235154 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b180a2e20 0x7f9b180a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.429+0000 7f9b2a59c640 1 -- 172.21.15.27:0/404339517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266966630 msgr2=0x55c266966a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:53.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.429+0000 7f9b2a59c640 1 --2- 172.21.15.27:0/404339517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266966630 0x55c266966a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.429+0000 7f9b2a59c640 1 -- 172.21.15.27:0/404339517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266971620 msgr2=0x55c266973a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:53.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.429+0000 7f9b2ad9d640 1 --2- 172.21.15.27:0/901235154 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b180a2e20 0x7f9b180a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.429+0000 7f9b2a59c640 1 --2- 172.21.15.27:0/404339517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266971620 0x55c266973a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.429+0000 7f9b2a59c640 1 -- 172.21.15.27:0/404339517 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2666a97d0 con 0x55c266966f50 2024-04-25T00:15:53.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.429+0000 7f9b23fff640 1 -- 172.21.15.27:0/901235154 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f9b240d0000 con 0x7f9b2c107730 2024-04-25T00:15:53.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.429+0000 7f9b2a59c640 1 --2- 172.21.15.27:0/404339517 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266966f50 0x55c2669710e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c005fd0 tx=0x7f9b1c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=602176bc69703131 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:53.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.429+0000 7f9b20d99640 1 -- 172.21.15.27:0/404339517 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c01c480 con 0x55c266966f50 2024-04-25T00:15:53.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.429+0000 7f9b2ad9d640 1 --2- 172.21.15.27:0/901235154 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b180a2e20 0x7f9b180a52e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9b240cd2c0 tx=0x7f9b2406df20 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:53.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.429+0000 7f9b20d99640 1 -- 172.21.15.27:0/404339517 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b1c01cb00 con 0x55c266966f50 2024-04-25T00:15:53.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.429+0000 7f9b20d99640 1 -- 172.21.15.27:0/404339517 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c011a80 con 0x55c266966f50 2024-04-25T00:15:53.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.430+0000 7f9b3357e740 1 -- 172.21.15.27:0/404339517 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266966f50 msgr2=0x55c2669710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.430+0000 7f9b3357e740 1 --2- 172.21.15.27:0/404339517 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266966f50 0x55c2669710e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c005fd0 tx=0x7f9b1c0107a0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.430+0000 7f9b3357e740 1 -- 172.21.15.27:0/404339517 shutdown_connections 2024-04-25T00:15:53.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.430+0000 7f9b3357e740 1 --2- 172.21.15.27:0/404339517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266971620 0x55c266973a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.430+0000 7f9b3357e740 1 --2- 172.21.15.27:0/404339517 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266966f50 0x55c2669710e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.430+0000 7f9b3357e740 1 --2- 172.21.15.27:0/404339517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266966630 0x55c266966a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.430+0000 7f9b3357e740 1 -- 172.21.15.27:0/404339517 >> 172.21.15.27:0/404339517 conn(0x55c266962950 msgr2=0x55c266962d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:53.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.430+0000 7f9b3357e740 1 -- 172.21.15.27:0/404339517 shutdown_connections 2024-04-25T00:15:53.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.430+0000 7f9b3357e740 1 -- 172.21.15.27:0/404339517 wait complete. 2024-04-25T00:15:53.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.430+0000 7f9b3357e740 1 -- start start 2024-04-25T00:15:53.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b3357e740 1 Processor -- start 2024-04-25T00:15:53.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b3357e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266966630 0x55c26696e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b3357e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266966f50 0x55c26696ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b3357e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266971620 0x55c26696efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b3357e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c266974410 con 0x55c266971620 2024-04-25T00:15:53.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b3357e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c266886f50 con 0x55c266966f50 2024-04-25T00:15:53.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b3357e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c266974590 con 0x55c266966630 2024-04-25T00:15:53.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b3357e740 10 client.0 _ll_register_callbacks cb 0x55c2668c2000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:53.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b21d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266971620 0x55c26696efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b21d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266971620 0x55c26696efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47952/0 (socket says 172.21.15.27:47952) 2024-04-25T00:15:53.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b21d9b640 1 -- 172.21.15.27:0/469270738 learned_addr learned my addr 172.21.15.27:0/469270738 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:53.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b21d9b640 1 --2- 172.21.15.27:0/469270738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266966630 0x55c26696e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b2a59c640 1 --2- 172.21.15.27:0/469270738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266966f50 0x55c26696ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b21d9b640 1 -- 172.21.15.27:0/469270738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266966630 msgr2=0x55c26696e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:53.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b21d9b640 1 --2- 172.21.15.27:0/469270738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266966630 0x55c26696e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b21d9b640 1 -- 172.21.15.27:0/469270738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266966f50 msgr2=0x55c26696ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:53.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b21d9b640 1 --2- 172.21.15.27:0/469270738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266966f50 0x55c26696ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b21d9b640 1 -- 172.21.15.27:0/469270738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2665df3f0 con 0x55c266971620 2024-04-25T00:15:53.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b21d9b640 1 --2- 172.21.15.27:0/469270738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266971620 0x55c26696efb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9b1402a940 tx=0x7f9b14009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ac3ec105e9b168 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:53.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.431+0000 7f9b11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:53.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.432+0000 7f9b11ffb640 1 -- 172.21.15.27:0/469270738 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b14035040 con 0x55c266971620 2024-04-25T00:15:53.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.432+0000 7f9b11ffb640 1 -- 172.21.15.27:0/469270738 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f9b14004030 con 0x55c266971620 2024-04-25T00:15:53.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.432+0000 7f9b3357e740 1 -- 172.21.15.27:0/469270738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b1c01cb00 con 0x55c266971620 2024-04-25T00:15:53.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.432+0000 7f9b11ffb640 1 -- 172.21.15.27:0/469270738 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9b14004230 con 0x55c266971620 2024-04-25T00:15:53.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.432+0000 7f9b3357e740 10 client.15804 fetch_fsmap learned FSMap version 29 2024-04-25T00:15:53.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.432+0000 7f9b3357e740 1 -- 172.21.15.27:0/469270738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c26659f0f0 con 0x55c266971620 2024-04-25T00:15:53.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.432+0000 7f9b11ffb640 1 -- 172.21.15.27:0/469270738 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b1c01cb00 con 0x55c266971620 2024-04-25T00:15:53.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.432+0000 7f9b11ffb640 1 -- 172.21.15.27:0/469270738 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9b140049e0 con 0x55c266971620 2024-04-25T00:15:53.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.432+0000 7f9b3357e740 10 client.15804 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:15:53.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.432+0000 7f9b3357e740 10 client.15804 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:15:53.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.432+0000 7f9b3357e740 20 client.15804 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:53.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.432+0000 7f9b3357e740 1 --2- 172.21.15.27:0/469270738 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55c2669818f0 0x55c266983ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.432+0000 7f9b3357e740 4 client.15804 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:15:53.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.432+0000 7f9b3357e740 1 -- 172.21.15.27:0/469270738 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2668d8bf0 con 0x55c2669818f0 2024-04-25T00:15:53.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.433+0000 7f9b2a59c640 1 --2- 172.21.15.27:0/469270738 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55c2669818f0 0x55c266983ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.433+0000 7f9b2a59c640 1 --2- 172.21.15.27:0/469270738 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55c2669818f0 0x55c266983ce0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc675ae3f965d0d server_cookie=d799ed4616ecd5e0 in_seq=0 out_seq=0 2024-04-25T00:15:53.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.433+0000 7f9b11ffb640 10 client.15804 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:15:53.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.434+0000 7f9b11ffb640 1 -- 172.21.15.27:0/469270738 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c2668d8bf0 con 0x55c2669818f0 2024-04-25T00:15:53.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.434+0000 7f9b11ffb640 10 client.15804 handle_command_reply: tid=0 2024-04-25T00:15:53.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.434+0000 7f9b3357e740 1 client.15804 shutdown 2024-04-25T00:15:53.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.434+0000 7f9b3357e740 1 -- 172.21.15.27:0/469270738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266971620 msgr2=0x55c26696efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.434+0000 7f9b3357e740 1 --2- 172.21.15.27:0/469270738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266971620 0x55c26696efb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9b1402a940 tx=0x7f9b14009640 comp rx=0 tx=0).stop 2024-04-25T00:15:53.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.434+0000 7f9b3357e740 1 -- 172.21.15.27:0/469270738 shutdown_connections 2024-04-25T00:15:53.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.434+0000 7f9b3357e740 1 --2- 172.21.15.27:0/469270738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266971620 0x55c26696efb0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.434+0000 7f9b3357e740 1 --2- 172.21.15.27:0/469270738 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55c2669818f0 0x55c266983ce0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.434+0000 7f9b3357e740 1 --2- 172.21.15.27:0/469270738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266966f50 0x55c26696ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.434+0000 7f9b3357e740 1 --2- 172.21.15.27:0/469270738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266966630 0x55c26696e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.434+0000 7f9b11ffb640 0 client.15804 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:15:53.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.434+0000 7f9b3357e740 1 -- 172.21.15.27:0/469270738 >> 172.21.15.27:0/469270738 conn(0x55c266962950 msgr2=0x55c266968750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:53.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.434+0000 7f9b3357e740 1 -- 172.21.15.27:0/469270738 shutdown_connections 2024-04-25T00:15:53.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.434+0000 7f9b3357e740 1 -- 172.21.15.27:0/469270738 wait complete. 2024-04-25T00:15:53.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.434+0000 7f9b3357e740 20 client.15804 trim_cache size 0 max 16384 2024-04-25T00:15:53.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:53 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/406807408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:15:53.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:53 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/284151721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:15:53.445 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:53 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4153310434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:15:53.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b3357e740 1 Processor -- start 2024-04-25T00:15:53.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b3357e740 1 -- start start 2024-04-25T00:15:53.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b3357e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266966630 0x55c26697d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b3357e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266966f50 0x55c266962d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b3357e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266971620 0x55c266968dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b3357e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c1b0ee0 con 0x55c266966630 2024-04-25T00:15:53.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b3357e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c1b2a20 con 0x55c266971620 2024-04-25T00:15:53.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b3357e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c1b15b0 con 0x55c266966f50 2024-04-25T00:15:53.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b21d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266966630 0x55c26697d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b21d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266966630 0x55c26697d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47958/0 (socket says 172.21.15.27:47958) 2024-04-25T00:15:53.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b21d9b640 1 -- 172.21.15.27:0/3590550531 learned_addr learned my addr 172.21.15.27:0/3590550531 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:53.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b2a59c640 1 --2- 172.21.15.27:0/3590550531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266966f50 0x55c266962d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b21d9b640 1 --2- 172.21.15.27:0/3590550531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266971620 0x55c266968dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b21d9b640 1 -- 172.21.15.27:0/3590550531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266966f50 msgr2=0x55c266962d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:53.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b21d9b640 1 --2- 172.21.15.27:0/3590550531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266966f50 0x55c266962d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b21d9b640 1 -- 172.21.15.27:0/3590550531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266971620 msgr2=0x55c266968dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:53.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b21d9b640 1 --2- 172.21.15.27:0/3590550531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266971620 0x55c266968dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.452+0000 7f9b21d9b640 1 -- 172.21.15.27:0/3590550531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c266966d80 con 0x55c266966630 2024-04-25T00:15:53.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.453+0000 7f9b21d9b640 1 --2- 172.21.15.27:0/3590550531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266966630 0x55c26697d330 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9b14009850 tx=0x7f9b14041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1125d577515328e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:53.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.453+0000 7f9b13fff640 1 -- 172.21.15.27:0/3590550531 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b14035070 con 0x55c266966630 2024-04-25T00:15:53.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.453+0000 7f9b13fff640 1 -- 172.21.15.27:0/3590550531 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b14043070 con 0x55c266966630 2024-04-25T00:15:53.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.453+0000 7f9b13fff640 1 -- 172.21.15.27:0/3590550531 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b14004fa0 con 0x55c266966630 2024-04-25T00:15:53.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.453+0000 7f9b3357e740 1 -- 172.21.15.27:0/3590550531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266966630 msgr2=0x55c26697d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.453+0000 7f9b3357e740 1 --2- 172.21.15.27:0/3590550531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266966630 0x55c26697d330 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9b14009850 tx=0x7f9b14041040 comp rx=0 tx=0).stop 2024-04-25T00:15:53.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.453+0000 7f9b3357e740 1 -- 172.21.15.27:0/3590550531 shutdown_connections 2024-04-25T00:15:53.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.453+0000 7f9b3357e740 1 --2- 172.21.15.27:0/3590550531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266971620 0x55c266968dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.453+0000 7f9b3357e740 1 --2- 172.21.15.27:0/3590550531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266966f50 0x55c266962d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.453+0000 7f9b3357e740 1 --2- 172.21.15.27:0/3590550531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266966630 0x55c26697d330 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.453+0000 7f9b3357e740 1 -- 172.21.15.27:0/3590550531 >> 172.21.15.27:0/3590550531 conn(0x55c266962950 msgr2=0x55c266961260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:53.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.453+0000 7f9b3357e740 1 -- 172.21.15.27:0/3590550531 shutdown_connections 2024-04-25T00:15:53.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.453+0000 7f9b3357e740 1 -- 172.21.15.27:0/3590550531 wait complete. 2024-04-25T00:15:53.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b3357e740 1 -- start start 2024-04-25T00:15:53.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b3357e740 1 Processor -- start 2024-04-25T00:15:53.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b3357e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266966630 0x55c266979840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b3357e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266966f50 0x55c266979d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b3357e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266971620 0x55c26696a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b3357e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c1b2350 con 0x55c266966f50 2024-04-25T00:15:53.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b3357e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c1b1c80 con 0x55c266966630 2024-04-25T00:15:53.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b3357e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c1aeea0 con 0x55c266971620 2024-04-25T00:15:53.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b3357e740 10 client.0 _ll_register_callbacks cb 0x7f9b14032f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:15:53.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b2a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266966f50 0x55c266979d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b2a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266966f50 0x55c266979d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47964/0 (socket says 172.21.15.27:47964) 2024-04-25T00:15:53.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b2a59c640 1 -- 172.21.15.27:0/707010996 learned_addr learned my addr 172.21.15.27:0/707010996 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:53.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b21d9b640 1 --2- 172.21.15.27:0/707010996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266966630 0x55c266979840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b21d9b640 1 --2- 172.21.15.27:0/707010996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266971620 0x55c26696a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b2a59c640 1 -- 172.21.15.27:0/707010996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266971620 msgr2=0x55c26696a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:53.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b2a59c640 1 --2- 172.21.15.27:0/707010996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266971620 0x55c26696a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b2a59c640 1 -- 172.21.15.27:0/707010996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266966630 msgr2=0x55c266979840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:53.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b2a59c640 1 --2- 172.21.15.27:0/707010996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266966630 0x55c266979840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b2a59c640 1 -- 172.21.15.27:0/707010996 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c26696b100 con 0x55c266966f50 2024-04-25T00:15:53.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.454+0000 7f9b21d9b640 1 --2- 172.21.15.27:0/707010996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266966630 0x55c266979840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:53.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.455+0000 7f9b2a59c640 1 --2- 172.21.15.27:0/707010996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266966f50 0x55c266979d80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c011630 tx=0x7f9b1c0115c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1625a234a8e25ab1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:53.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.455+0000 7f9b11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:15:53.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.455+0000 7f9b11ffb640 1 -- 172.21.15.27:0/707010996 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c00b710 con 0x55c266966f50 2024-04-25T00:15:53.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.455+0000 7f9b11ffb640 1 -- 172.21.15.27:0/707010996 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f9b1c004020 con 0x55c266966f50 2024-04-25T00:15:53.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.455+0000 7f9b3357e740 1 -- 172.21.15.27:0/707010996 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b14043070 con 0x55c266966f50 2024-04-25T00:15:53.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.455+0000 7f9b11ffb640 1 -- 172.21.15.27:0/707010996 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9b1c036050 con 0x55c266966f50 2024-04-25T00:15:53.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.455+0000 7f9b3357e740 10 client.15810 fetch_fsmap learned FSMap version 29 2024-04-25T00:15:53.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.455+0000 7f9b3357e740 1 -- 172.21.15.27:0/707010996 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c2669e77f0 con 0x55c266966f50 2024-04-25T00:15:53.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.455+0000 7f9b11ffb640 1 -- 172.21.15.27:0/707010996 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b14043070 con 0x55c266966f50 2024-04-25T00:15:53.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.455+0000 7f9b11ffb640 1 -- 172.21.15.27:0/707010996 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9b1c01c640 con 0x55c266966f50 2024-04-25T00:15:53.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.455+0000 7f9b3357e740 10 client.15810 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:15:53.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.455+0000 7f9b3357e740 10 client.15810 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:15:53.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.455+0000 7f9b3357e740 20 client.15810 populate_metadata read hostname 'smithi027' 2024-04-25T00:15:53.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.455+0000 7f9b3357e740 1 --2- 172.21.15.27:0/707010996 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55c2669e7ce0 0x55c2669ea100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.455+0000 7f9b3357e740 4 client.15810 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:15:53.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.455+0000 7f9b3357e740 1 -- 172.21.15.27:0/707010996 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c2668d2f40 con 0x55c2669e7ce0 2024-04-25T00:15:53.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.455+0000 7f9b21d9b640 1 --2- 172.21.15.27:0/707010996 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55c2669e7ce0 0x55c2669ea100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.456+0000 7f9b21d9b640 1 --2- 172.21.15.27:0/707010996 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55c2669e7ce0 0x55c2669ea100 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b50c2de61104bd62 server_cookie=106e5bc1b8dadcf0 in_seq=0 out_seq=0 2024-04-25T00:15:53.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.456+0000 7f9b11ffb640 10 client.15810 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:15:53.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.456+0000 7f9b11ffb640 1 -- 172.21.15.27:0/707010996 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c2668d2f40 con 0x55c2669e7ce0 2024-04-25T00:15:53.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.456+0000 7f9b11ffb640 10 client.15810 handle_command_reply: tid=0 2024-04-25T00:15:53.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.456+0000 7f9b3357e740 1 client.15810 shutdown 2024-04-25T00:15:53.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.456+0000 7f9b3357e740 1 -- 172.21.15.27:0/707010996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266966f50 msgr2=0x55c266979d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.456+0000 7f9b3357e740 1 --2- 172.21.15.27:0/707010996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266966f50 0x55c266979d80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c011630 tx=0x7f9b1c0115c0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.456+0000 7f9b3357e740 1 -- 172.21.15.27:0/707010996 shutdown_connections 2024-04-25T00:15:53.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.456+0000 7f9b3357e740 1 --2- 172.21.15.27:0/707010996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c266971620 0x55c26696a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.456+0000 7f9b3357e740 1 --2- 172.21.15.27:0/707010996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c266966f50 0x55c266979d80 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.456+0000 7f9b3357e740 1 --2- 172.21.15.27:0/707010996 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55c2669e7ce0 0x55c2669ea100 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.456+0000 7f9b3357e740 1 --2- 172.21.15.27:0/707010996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c266966630 0x55c266979840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.456+0000 7f9b11ffb640 0 client.15810 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:15:53.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.456+0000 7f9b3357e740 1 -- 172.21.15.27:0/707010996 >> 172.21.15.27:0/707010996 conn(0x55c266962950 msgr2=0x55c26696ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:53.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.456+0000 7f9b3357e740 1 -- 172.21.15.27:0/707010996 shutdown_connections 2024-04-25T00:15:53.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.456+0000 7f9b3357e740 1 -- 172.21.15.27:0/707010996 wait complete. 2024-04-25T00:15:53.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.456+0000 7f9b3357e740 20 client.15810 trim_cache size 0 max 16384 2024-04-25T00:15:53.463 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:15:53.463 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T00:15:53.463 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:15:53.463 INFO:teuthology.orchestra.run.smithi027.stdout: "b6f50739-b6ab-49da-9f05-cff1bc7fd76e": { 2024-04-25T00:15:53.463 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:15:53.464 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "b6f50739-b6ab-49da-9f05-cff1bc7fd76e", 2024-04-25T00:15:53.464 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:15:53.464 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:15:53.464 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:15:53.464 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:15:53.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.457+0000 7f9b30872640 1 -- 172.21.15.27:0/901235154 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b180a2e20 msgr2=0x7f9b180a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.457+0000 7f9b30872640 1 --2- 172.21.15.27:0/901235154 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b180a2e20 0x7f9b180a52e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9b240cd2c0 tx=0x7f9b2406df20 comp rx=0 tx=0).stop 2024-04-25T00:15:53.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.457+0000 7f9b30872640 1 -- 172.21.15.27:0/901235154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b2c107730 msgr2=0x7f9b2c0744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.457+0000 7f9b30872640 1 --2- 172.21.15.27:0/901235154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b2c107730 0x7f9b2c0744c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9b2400eb10 tx=0x7f9b2400c2d0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.457+0000 7f9b30872640 1 -- 172.21.15.27:0/901235154 shutdown_connections 2024-04-25T00:15:53.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.457+0000 7f9b30872640 1 --2- 172.21.15.27:0/901235154 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b180a2e20 0x7f9b180a52e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.457+0000 7f9b30872640 1 --2- 172.21.15.27:0/901235154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b2c107730 0x7f9b2c0744c0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.457+0000 7f9b30872640 1 --2- 172.21.15.27:0/901235154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b2c1020e0 0x7f9b2c073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.457+0000 7f9b30872640 1 --2- 172.21.15.27:0/901235154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b2c101720 0x7f9b2c073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.457+0000 7f9b30872640 1 -- 172.21.15.27:0/901235154 >> 172.21.15.27:0/901235154 conn(0x7f9b2c078b40 msgr2=0x7f9b2c1040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:53.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.457+0000 7f9b30872640 1 -- 172.21.15.27:0/901235154 shutdown_connections 2024-04-25T00:15:53.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.457+0000 7f9b30872640 1 -- 172.21.15.27:0/901235154 wait complete. 2024-04-25T00:15:53.468 INFO:tasks.cephfs.filesystem:scrub status for tag:b6f50739-b6ab-49da-9f05-cff1bc7fd76e - {'path': '/', 'tag': 'b6f50739-b6ab-49da-9f05-cff1bc7fd76e', 'options': 'recursive,force'} 2024-04-25T00:15:53.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.485+0000 7f99083a9640 1 -- 172.21.15.27:0/1629537310 --> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f9900108c20 con 0x7f98ec0a2f40 2024-04-25T00:15:53.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.551+0000 7f98f77fe640 1 -- 172.21.15.27:0/1629537310 <== mgr.15564 v2:172.21.15.27:6800/1076858552 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f98fc0315b0 con 0x7f98ec0a2f40 2024-04-25T00:15:53.551 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path / 2024-04-25T00:15:53.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.553+0000 7f99083a9640 1 -- 172.21.15.27:0/1629537310 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f98ec0a2f40 msgr2=0x7f98ec0a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.553+0000 7f99083a9640 1 --2- 172.21.15.27:0/1629537310 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f98ec0a2f40 0x7f98ec0a5400 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f98fc000c00 tx=0x7f98fc005a20 comp rx=0 tx=0).stop 2024-04-25T00:15:53.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.553+0000 7f99083a9640 1 -- 172.21.15.27:0/1629537310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9900101000 msgr2=0x7f9900199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.553+0000 7f99083a9640 1 --2- 172.21.15.27:0/1629537310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9900101000 0x7f9900199340 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f98f000d950 tx=0x7f98f000de10 comp rx=0 tx=0).stop 2024-04-25T00:15:53.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.553+0000 7f99083a9640 1 -- 172.21.15.27:0/1629537310 shutdown_connections 2024-04-25T00:15:53.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.553+0000 7f99083a9640 1 --2- 172.21.15.27:0/1629537310 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f98ec0a2f40 0x7f98ec0a5400 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.553+0000 7f99083a9640 1 --2- 172.21.15.27:0/1629537310 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9900106650 0x7f99001a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.553+0000 7f99083a9640 1 --2- 172.21.15.27:0/1629537310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9900101000 0x7f9900199340 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.553+0000 7f99083a9640 1 --2- 172.21.15.27:0/1629537310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9900100640 0x7f9900198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.553+0000 7f99083a9640 1 -- 172.21.15.27:0/1629537310 >> 172.21.15.27:0/1629537310 conn(0x7f9900077750 msgr2=0x7f99001031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:53.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.553+0000 7f99083a9640 1 -- 172.21.15.27:0/1629537310 shutdown_connections 2024-04-25T00:15:53.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.553+0000 7f99083a9640 1 -- 172.21.15.27:0/1629537310 wait complete. 2024-04-25T00:15:53.564 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-25T00:15:53.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.709+0000 7f9a1f064640 1 -- 172.21.15.27:0/1237420643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a18101020 msgr2=0x7f9a1810d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.709+0000 7f9a1f064640 1 --2- 172.21.15.27:0/1237420643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a18101020 0x7f9a1810d530 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9a10009cd0 tx=0x7f9a1002a8e0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.709+0000 7f9a1f064640 1 -- 172.21.15.27:0/1237420643 shutdown_connections 2024-04-25T00:15:53.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.709+0000 7f9a1f064640 1 --2- 172.21.15.27:0/1237420643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a18101020 0x7f9a1810d530 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.709+0000 7f9a1f064640 1 --2- 172.21.15.27:0/1237420643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a18100660 0x7f9a18100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.709+0000 7f9a1f064640 1 --2- 172.21.15.27:0/1237420643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a18106670 0x7f9a18106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.709+0000 7f9a1f064640 1 -- 172.21.15.27:0/1237420643 >> 172.21.15.27:0/1237420643 conn(0x7f9a180769c0 msgr2=0x7f9a18076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:53.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.709+0000 7f9a1f064640 1 -- 172.21.15.27:0/1237420643 shutdown_connections 2024-04-25T00:15:53.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.709+0000 7f9a1f064640 1 -- 172.21.15.27:0/1237420643 wait complete. 2024-04-25T00:15:53.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.710+0000 7f9a1f064640 1 Processor -- start 2024-04-25T00:15:53.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.710+0000 7f9a1f064640 1 -- start start 2024-04-25T00:15:53.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.710+0000 7f9a1f064640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a18100660 0x7f9a18198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.710+0000 7f9a1f064640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a18101020 0x7f9a18199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.710+0000 7f9a1f064640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a18106670 0x7f9a181a3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.710+0000 7f9a1f064640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9a18110ad0 con 0x7f9a18106670 2024-04-25T00:15:53.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.710+0000 7f9a1f064640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9a18110950 con 0x7f9a18100660 2024-04-25T00:15:53.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.710+0000 7f9a1f064640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9a18110c50 con 0x7f9a18101020 2024-04-25T00:15:53.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.710+0000 7f9a1d601640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a18106670 0x7f9a181a3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.710+0000 7f9a1d601640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a18106670 0x7f9a181a3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47992/0 (socket says 172.21.15.27:47992) 2024-04-25T00:15:53.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.710+0000 7f9a1d601640 1 -- 172.21.15.27:0/190582670 learned_addr learned my addr 172.21.15.27:0/190582670 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:53.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.710+0000 7f9a17fff640 1 --2- 172.21.15.27:0/190582670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a18101020 0x7f9a18199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.710+0000 7f9a1ce00640 1 --2- 172.21.15.27:0/190582670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a18100660 0x7f9a18198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.711+0000 7f9a1d601640 1 -- 172.21.15.27:0/190582670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a18101020 msgr2=0x7f9a18199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.711+0000 7f9a1d601640 1 --2- 172.21.15.27:0/190582670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a18101020 0x7f9a18199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.711+0000 7f9a1d601640 1 -- 172.21.15.27:0/190582670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a18100660 msgr2=0x7f9a18198dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.711+0000 7f9a1d601640 1 --2- 172.21.15.27:0/190582670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a18100660 0x7f9a18198dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.711+0000 7f9a1d601640 1 -- 172.21.15.27:0/190582670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a181a4390 con 0x7f9a18106670 2024-04-25T00:15:53.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.711+0000 7f9a1d601640 1 --2- 172.21.15.27:0/190582670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a18106670 0x7f9a181a3e50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9a10033040 tx=0x7f9a10031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:53.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.711+0000 7f9a15ffb640 1 -- 172.21.15.27:0/190582670 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10004040 con 0x7f9a18106670 2024-04-25T00:15:53.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.711+0000 7f9a1f064640 1 -- 172.21.15.27:0/190582670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a181a4560 con 0x7f9a18106670 2024-04-25T00:15:53.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.711+0000 7f9a15ffb640 1 -- 172.21.15.27:0/190582670 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a100046c0 con 0x7f9a18106670 2024-04-25T00:15:53.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.711+0000 7f9a15ffb640 1 -- 172.21.15.27:0/190582670 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10030ed0 con 0x7f9a18106670 2024-04-25T00:15:53.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.711+0000 7f9a1f064640 1 -- 172.21.15.27:0/190582670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a181a4a20 con 0x7f9a18106670 2024-04-25T00:15:53.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.712+0000 7f9a1f064640 1 -- 172.21.15.27:0/190582670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a1804f500 con 0x7f9a18106670 2024-04-25T00:15:53.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.714+0000 7f9a15ffb640 1 -- 172.21.15.27:0/190582670 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f9a100041e0 con 0x7f9a18106670 2024-04-25T00:15:53.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.715+0000 7f9a15ffb640 1 --2- 172.21.15.27:0/190582670 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9a080a2ec0 0x7f9a080a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:53.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.715+0000 7f9a15ffb640 1 -- 172.21.15.27:0/190582670 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f9a1002e030 con 0x7f9a18106670 2024-04-25T00:15:53.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.715+0000 7f9a1ce00640 1 --2- 172.21.15.27:0/190582670 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9a080a2ec0 0x7f9a080a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:53.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.715+0000 7f9a1ce00640 1 --2- 172.21.15.27:0/190582670 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9a080a2ec0 0x7f9a080a5380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a04009b50 tx=0x7f9a04005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:53.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.718+0000 7f9a15ffb640 1 -- 172.21.15.27:0/190582670 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9a1004b060 con 0x7f9a18106670 2024-04-25T00:15:53.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.929+0000 7f9a1f064640 1 -- 172.21.15.27:0/190582670 --> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f9a180fea50 con 0x7f9a080a2ec0 2024-04-25T00:15:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.984+0000 7f9a15ffb640 1 -- 172.21.15.27:0/190582670 <== mgr.15564 v2:172.21.15.27:6800/1076858552 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f9a180fea50 con 0x7f9a080a2ec0 2024-04-25T00:15:53.984 INFO:teuthology.orchestra.run.smithi027.stdout:Retention added to path / 2024-04-25T00:15:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.986+0000 7f9a1f064640 1 -- 172.21.15.27:0/190582670 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9a080a2ec0 msgr2=0x7f9a080a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.986+0000 7f9a1f064640 1 --2- 172.21.15.27:0/190582670 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9a080a2ec0 0x7f9a080a5380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a04009b50 tx=0x7f9a04005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.986+0000 7f9a1f064640 1 -- 172.21.15.27:0/190582670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a18106670 msgr2=0x7f9a181a3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.986+0000 7f9a1f064640 1 --2- 172.21.15.27:0/190582670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a18106670 0x7f9a181a3e50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9a10033040 tx=0x7f9a10031e60 comp rx=0 tx=0).stop 2024-04-25T00:15:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.986+0000 7f9a1f064640 1 -- 172.21.15.27:0/190582670 shutdown_connections 2024-04-25T00:15:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.986+0000 7f9a1f064640 1 --2- 172.21.15.27:0/190582670 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9a080a2ec0 0x7f9a080a5380 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.986+0000 7f9a1f064640 1 --2- 172.21.15.27:0/190582670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a18106670 0x7f9a181a3e50 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.986+0000 7f9a1f064640 1 --2- 172.21.15.27:0/190582670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a18101020 0x7f9a18199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.986+0000 7f9a1f064640 1 --2- 172.21.15.27:0/190582670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a18100660 0x7f9a18198dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:53.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.986+0000 7f9a1f064640 1 -- 172.21.15.27:0/190582670 >> 172.21.15.27:0/190582670 conn(0x7f9a180769c0 msgr2=0x7f9a180fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:53.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.986+0000 7f9a1f064640 1 -- 172.21.15.27:0/190582670 shutdown_connections 2024-04-25T00:15:53.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:53.986+0000 7f9a1f064640 1 -- 172.21.15.27:0/190582670 wait complete. 2024-04-25T00:15:53.997 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T00:15:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.102+0000 7fc86aa9a640 1 -- 172.21.15.27:0/1772598024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc864101030 msgr2=0x7fc86410d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.102+0000 7fc86aa9a640 1 --2- 172.21.15.27:0/1772598024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc864101030 0x7fc86410d4f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009cd0 tx=0x7fc85c02a8e0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.102+0000 7fc86aa9a640 1 -- 172.21.15.27:0/1772598024 shutdown_connections 2024-04-25T00:15:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.102+0000 7fc86aa9a640 1 --2- 172.21.15.27:0/1772598024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc864101030 0x7fc86410d4f0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.102+0000 7fc86aa9a640 1 --2- 172.21.15.27:0/1772598024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc864100670 0x7fc864100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.102+0000 7fc86aa9a640 1 --2- 172.21.15.27:0/1772598024 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc864106630 0x7fc864106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.102+0000 7fc86aa9a640 1 -- 172.21.15.27:0/1772598024 >> 172.21.15.27:0/1772598024 conn(0x7fc8640777c0 msgr2=0x7fc864077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.102+0000 7fc86aa9a640 1 -- 172.21.15.27:0/1772598024 shutdown_connections 2024-04-25T00:15:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.102+0000 7fc86aa9a640 1 -- 172.21.15.27:0/1772598024 wait complete. 2024-04-25T00:15:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.102+0000 7fc86aa9a640 1 Processor -- start 2024-04-25T00:15:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.102+0000 7fc86aa9a640 1 -- start start 2024-04-25T00:15:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.102+0000 7fc86aa9a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc864100670 0x7fc864196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.102+0000 7fc86aa9a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc864101030 0x7fc864197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.102+0000 7fc86aa9a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc864106630 0x7fc86419b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.102+0000 7fc86aa9a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc864110a40 con 0x7fc864101030 2024-04-25T00:15:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.102+0000 7fc86aa9a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc8641108c0 con 0x7fc864100670 2024-04-25T00:15:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.102+0000 7fc86aa9a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc864110bc0 con 0x7fc864106630 2024-04-25T00:15:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc85bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc864101030 0x7fc864197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc869037640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc864106630 0x7fc86419b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc868836640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc864100670 0x7fc864196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc85bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc864101030 0x7fc864197360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48012/0 (socket says 172.21.15.27:48012) 2024-04-25T00:15:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc869037640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc864106630 0x7fc86419b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:60172/0 (socket says 172.21.15.27:60172) 2024-04-25T00:15:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc868836640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc864100670 0x7fc864196e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:59634/0 (socket says 172.21.15.27:59634) 2024-04-25T00:15:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc85bfff640 1 -- 172.21.15.27:0/3249502060 learned_addr learned my addr 172.21.15.27:0/3249502060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc85bfff640 1 -- 172.21.15.27:0/3249502060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc864106630 msgr2=0x7fc86419b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc85bfff640 1 --2- 172.21.15.27:0/3249502060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc864106630 0x7fc86419b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc85bfff640 1 -- 172.21.15.27:0/3249502060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc864100670 msgr2=0x7fc864196e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc85bfff640 1 --2- 172.21.15.27:0/3249502060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc864100670 0x7fc864196e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc85bfff640 1 -- 172.21.15.27:0/3249502060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86419be10 con 0x7fc864101030 2024-04-25T00:15:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc868836640 1 --2- 172.21.15.27:0/3249502060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc864100670 0x7fc864196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc85bfff640 1 --2- 172.21.15.27:0/3249502060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc864101030 0x7fc864197360 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc85400e9f0 tx=0x7fc85400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc859ffb640 1 -- 172.21.15.27:0/3249502060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85400cd80 con 0x7fc864101030 2024-04-25T00:15:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc86aa9a640 1 -- 172.21.15.27:0/3249502060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc86419c100 con 0x7fc864101030 2024-04-25T00:15:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc859ffb640 1 -- 172.21.15.27:0/3249502060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc854005500 con 0x7fc864101030 2024-04-25T00:15:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.103+0000 7fc859ffb640 1 -- 172.21.15.27:0/3249502060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc854010690 con 0x7fc864101030 2024-04-25T00:15:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.104+0000 7fc86aa9a640 1 -- 172.21.15.27:0/3249502060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8641a3990 con 0x7fc864101030 2024-04-25T00:15:54.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.105+0000 7fc86aa9a640 1 -- 172.21.15.27:0/3249502060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc86404f500 con 0x7fc864101030 2024-04-25T00:15:54.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.105+0000 7fc859ffb640 1 -- 172.21.15.27:0/3249502060 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fc8540046c0 con 0x7fc864101030 2024-04-25T00:15:54.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.106+0000 7fc859ffb640 1 --2- 172.21.15.27:0/3249502060 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc84c0a2ec0 0x7fc84c0a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:54.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.106+0000 7fc868836640 1 --2- 172.21.15.27:0/3249502060 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc84c0a2ec0 0x7fc84c0a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:54.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.106+0000 7fc859ffb640 1 -- 172.21.15.27:0/3249502060 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fc8540cfe90 con 0x7fc864101030 2024-04-25T00:15:54.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.107+0000 7fc868836640 1 --2- 172.21.15.27:0/3249502060 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc84c0a2ec0 0x7fc84c0a5380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc860009f40 tx=0x7fc860005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:54.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.111+0000 7fc859ffb640 1 -- 172.21.15.27:0/3249502060 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc85409c4f0 con 0x7fc864101030 2024-04-25T00:15:54.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.315+0000 7fc86aa9a640 1 -- 172.21.15.27:0/3249502060 --> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fc864103760 con 0x7fc84c0a2ec0 2024-04-25T00:15:54.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.323+0000 7fc859ffb640 1 -- 172.21.15.27:0/3249502060 <== mgr.15564 v2:172.21.15.27:6800/1076858552 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fc864103760 con 0x7fc84c0a2ec0 2024-04-25T00:15:54.323 INFO:teuthology.orchestra.run.smithi027.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-25T00:00:00", "created": "2024-04-25T00:15:53", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-25T00:15:54.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.325+0000 7fc86aa9a640 1 -- 172.21.15.27:0/3249502060 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc84c0a2ec0 msgr2=0x7fc84c0a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:54.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.325+0000 7fc86aa9a640 1 --2- 172.21.15.27:0/3249502060 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc84c0a2ec0 0x7fc84c0a5380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc860009f40 tx=0x7fc860005f90 comp rx=0 tx=0).stop 2024-04-25T00:15:54.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.325+0000 7fc86aa9a640 1 -- 172.21.15.27:0/3249502060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc864101030 msgr2=0x7fc864197360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:54.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.325+0000 7fc86aa9a640 1 --2- 172.21.15.27:0/3249502060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc864101030 0x7fc864197360 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc85400e9f0 tx=0x7fc85400edf0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.325+0000 7fc86aa9a640 1 -- 172.21.15.27:0/3249502060 shutdown_connections 2024-04-25T00:15:54.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.325+0000 7fc86aa9a640 1 --2- 172.21.15.27:0/3249502060 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc84c0a2ec0 0x7fc84c0a5380 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.325+0000 7fc86aa9a640 1 --2- 172.21.15.27:0/3249502060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc864106630 0x7fc86419b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.325+0000 7fc86aa9a640 1 --2- 172.21.15.27:0/3249502060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc864101030 0x7fc864197360 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.325+0000 7fc86aa9a640 1 --2- 172.21.15.27:0/3249502060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc864100670 0x7fc864196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.325+0000 7fc86aa9a640 1 -- 172.21.15.27:0/3249502060 >> 172.21.15.27:0/3249502060 conn(0x7fc8640777c0 msgr2=0x7fc864103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:54.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.325+0000 7fc86aa9a640 1 -- 172.21.15.27:0/3249502060 shutdown_connections 2024-04-25T00:15:54.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.325+0000 7fc86aa9a640 1 -- 172.21.15.27:0/3249502060 wait complete. 2024-04-25T00:15:54.335 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:54 smithi027 ceph-mon[29208]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.4 KiB/s wr, 26 op/s 2024-04-25T00:15:54.335 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:54 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3883020294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:54.335 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:54 smithi027 ceph-mon[29208]: scrub queued for path: / 2024-04-25T00:15:54.335 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:54 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:15:54.335 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:54 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:15:54.335 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:54 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:15:54.335 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:54 smithi027 ceph-mon[29208]: from='client.15777 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:54.336 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-25T00:15:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.440+0000 7f93b3bce640 1 -- 172.21.15.27:0/3291774790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ac073a40 msgr2=0x7f93ac073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.440+0000 7f93b3bce640 1 --2- 172.21.15.27:0/3291774790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ac073a40 0x7f93ac073e20 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f93a8009900 tx=0x7f93a802a480 comp rx=0 tx=0).stop 2024-04-25T00:15:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.440+0000 7f93b3bce640 1 -- 172.21.15.27:0/3291774790 shutdown_connections 2024-04-25T00:15:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.440+0000 7f93b3bce640 1 --2- 172.21.15.27:0/3291774790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f93ac10e030 0x7f93ac110420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.440+0000 7f93b3bce640 1 --2- 172.21.15.27:0/3291774790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93ac074360 0x7f93ac10daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.440+0000 7f93b3bce640 1 --2- 172.21.15.27:0/3291774790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ac073a40 0x7f93ac073e20 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.440+0000 7f93b3bce640 1 -- 172.21.15.27:0/3291774790 >> 172.21.15.27:0/3291774790 conn(0x7f93ac078bd0 msgr2=0x7f93ac078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.440+0000 7f93b3bce640 1 -- 172.21.15.27:0/3291774790 shutdown_connections 2024-04-25T00:15:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.440+0000 7f93b3bce640 1 -- 172.21.15.27:0/3291774790 wait complete. 2024-04-25T00:15:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.441+0000 7f93b3bce640 1 Processor -- start 2024-04-25T00:15:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.441+0000 7f93b3bce640 1 -- start start 2024-04-25T00:15:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.441+0000 7f93b3bce640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f93ac073a40 0x7f93ac19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.441+0000 7f93b3bce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ac074360 0x7f93ac19a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.441+0000 7f93b3bce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93ac10e030 0x7f93ac19e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.441+0000 7f93b3bce640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f93ac113d80 con 0x7f93ac074360 2024-04-25T00:15:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.441+0000 7f93b3bce640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f93ac113c00 con 0x7f93ac073a40 2024-04-25T00:15:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.441+0000 7f93b3bce640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f93ac113f00 con 0x7f93ac10e030 2024-04-25T00:15:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.441+0000 7f93b1169640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ac074360 0x7f93ac19a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.441+0000 7f93b1169640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ac074360 0x7f93ac19a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48032/0 (socket says 172.21.15.27:48032) 2024-04-25T00:15:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.441+0000 7f93b196a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f93ac073a40 0x7f93ac19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.441+0000 7f93b1169640 1 -- 172.21.15.27:0/2701117632 learned_addr learned my addr 172.21.15.27:0/2701117632 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:15:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.441+0000 7f93b216b640 1 --2- 172.21.15.27:0/2701117632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93ac10e030 0x7f93ac19e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.441+0000 7f93b1169640 1 -- 172.21.15.27:0/2701117632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93ac10e030 msgr2=0x7f93ac19e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.442+0000 7f93b1169640 1 --2- 172.21.15.27:0/2701117632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93ac10e030 0x7f93ac19e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.442+0000 7f93b1169640 1 -- 172.21.15.27:0/2701117632 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f93ac073a40 msgr2=0x7f93ac19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.442+0000 7f93b1169640 1 --2- 172.21.15.27:0/2701117632 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f93ac073a40 0x7f93ac19a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.442+0000 7f93b1169640 1 -- 172.21.15.27:0/2701117632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93ac19f110 con 0x7f93ac074360 2024-04-25T00:15:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.442+0000 7f93b1169640 1 --2- 172.21.15.27:0/2701117632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ac074360 0x7f93ac19a660 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f939c00d950 tx=0x7f939c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.442+0000 7f93a2ffd640 1 -- 172.21.15.27:0/2701117632 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939c017030 con 0x7f93ac074360 2024-04-25T00:15:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.442+0000 7f93b3bce640 1 -- 172.21.15.27:0/2701117632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93ac19f400 con 0x7f93ac074360 2024-04-25T00:15:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.442+0000 7f93a2ffd640 1 -- 172.21.15.27:0/2701117632 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f939c004020 con 0x7f93ac074360 2024-04-25T00:15:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.442+0000 7f93a2ffd640 1 -- 172.21.15.27:0/2701117632 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939c0047a0 con 0x7f93ac074360 2024-04-25T00:15:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.442+0000 7f93b3bce640 1 -- 172.21.15.27:0/2701117632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93ac1a6c40 con 0x7f93ac074360 2024-04-25T00:15:54.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.443+0000 7f93b3bce640 1 -- 172.21.15.27:0/2701117632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93ac04f500 con 0x7f93ac074360 2024-04-25T00:15:54.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.444+0000 7f93a2ffd640 1 -- 172.21.15.27:0/2701117632 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f939c0041d0 con 0x7f93ac074360 2024-04-25T00:15:54.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.445+0000 7f93a2ffd640 1 --2- 172.21.15.27:0/2701117632 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f93980a2ec0 0x7f93980a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:54.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.445+0000 7f93b196a640 1 --2- 172.21.15.27:0/2701117632 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f93980a2ec0 0x7f93980a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:54.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.445+0000 7f93a2ffd640 1 -- 172.21.15.27:0/2701117632 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f939c0cfa40 con 0x7f93ac074360 2024-04-25T00:15:54.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.445+0000 7f93b196a640 1 --2- 172.21.15.27:0/2701117632 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f93980a2ec0 0x7f93980a5380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f93a8000c00 tx=0x7f93a8005b50 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.448+0000 7f93a2ffd640 1 -- 172.21.15.27:0/2701117632 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f939c09c0a0 con 0x7f93ac074360 2024-04-25T00:15:54.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:54 smithi139 ceph-mon[32833]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.4 KiB/s wr, 26 op/s 2024-04-25T00:15:54.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:54 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3883020294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:54.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:54 smithi139 ceph-mon[32833]: scrub queued for path: / 2024-04-25T00:15:54.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:54 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:15:54.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:54 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:15:54.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:54 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:15:54.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:54 smithi139 ceph-mon[32833]: from='client.15777 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:54.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:54 smithi153 ceph-mon[32896]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.4 KiB/s wr, 26 op/s 2024-04-25T00:15:54.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:54 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3883020294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:15:54.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:54 smithi153 ceph-mon[32896]: scrub queued for path: / 2024-04-25T00:15:54.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:54 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:15:54.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:54 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:15:54.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:54 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:15:54.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:54 smithi153 ceph-mon[32896]: from='client.15777 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:54.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.656+0000 7f93b3bce640 1 -- 172.21.15.27:0/2701117632 --> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f93ac0fec70 con 0x7f93980a2ec0 2024-04-25T00:15:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.662+0000 7f93a2ffd640 1 -- 172.21.15.27:0/2701117632 <== mgr.15564 v2:172.21.15.27:6800/1076858552 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f93ac0fec70 con 0x7f93980a2ec0 2024-04-25T00:15:54.662 INFO:teuthology.orchestra.run.smithi027.stdout:/ 1m 6m3h 2024-04-25T00:15:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.664+0000 7f93b3bce640 1 -- 172.21.15.27:0/2701117632 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f93980a2ec0 msgr2=0x7f93980a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.664+0000 7f93b3bce640 1 --2- 172.21.15.27:0/2701117632 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f93980a2ec0 0x7f93980a5380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f93a8000c00 tx=0x7f93a8005b50 comp rx=0 tx=0).stop 2024-04-25T00:15:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.664+0000 7f93b3bce640 1 -- 172.21.15.27:0/2701117632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ac074360 msgr2=0x7f93ac19a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.664+0000 7f93b3bce640 1 --2- 172.21.15.27:0/2701117632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ac074360 0x7f93ac19a660 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f939c00d950 tx=0x7f939c00de10 comp rx=0 tx=0).stop 2024-04-25T00:15:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.664+0000 7f93b3bce640 1 -- 172.21.15.27:0/2701117632 shutdown_connections 2024-04-25T00:15:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.664+0000 7f93b3bce640 1 --2- 172.21.15.27:0/2701117632 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f93980a2ec0 0x7f93980a5380 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.664+0000 7f93b3bce640 1 --2- 172.21.15.27:0/2701117632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93ac10e030 0x7f93ac19e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.664+0000 7f93b3bce640 1 --2- 172.21.15.27:0/2701117632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ac074360 0x7f93ac19a660 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.664+0000 7f93b3bce640 1 --2- 172.21.15.27:0/2701117632 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f93ac073a40 0x7f93ac19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.664+0000 7f93b3bce640 1 -- 172.21.15.27:0/2701117632 >> 172.21.15.27:0/2701117632 conn(0x7f93ac078bd0 msgr2=0x7f93ac0fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.665+0000 7f93b3bce640 1 -- 172.21.15.27:0/2701117632 shutdown_connections 2024-04-25T00:15:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:15:54.665+0000 7f93b3bce640 1 -- 172.21.15.27:0/2701117632 wait complete. 2024-04-25T00:15:54.675 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-25T00:15:54.701 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-25T00:15:54.713 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T00:15:54.731 INFO:tasks.workunit:Pulling workunits from ref d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:15:54.731 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T00:15:54.731 INFO:tasks.workunit:timeout=3h 2024-04-25T00:15:54.731 INFO:tasks.workunit:cleanup=True 2024-04-25T00:15:54.732 DEBUG:teuthology.orchestra.run.smithi027:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T00:15:54.758 INFO:teuthology.orchestra.run.smithi027.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-25T00:15:54.758 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-25T00:15:54.758 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-25T00:15:54.758 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:15:54.758 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-25T00:15:54.759 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-25 00:13:47.018356772 +0000 2024-04-25T00:15:54.759 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-25 00:13:47.018356772 +0000 2024-04-25T00:15:54.759 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-25 00:15:42.875296072 +0000 2024-04-25T00:15:54.759 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-25 00:13:47.018356772 +0000 2024-04-25T00:15:54.759 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-25T00:15:54.760 DEBUG:teuthology.orchestra.run.smithi027:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-25T00:15:54.832 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:15:54.887 INFO:tasks.workunit.client.0.smithi027.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T00:15:55.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:55 smithi027 ceph-mon[29208]: from='client.15816 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:55.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:55 smithi027 ceph-mon[29208]: from='client.15822 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:55.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:55 smithi027 ceph-mon[29208]: from='client.15828 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:55.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:55 smithi139 ceph-mon[32833]: from='client.15816 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:55.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:55 smithi139 ceph-mon[32833]: from='client.15822 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:55.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:55 smithi139 ceph-mon[32833]: from='client.15828 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:55.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:55 smithi153 ceph-mon[32896]: from='client.15816 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:55.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:55 smithi153 ceph-mon[32896]: from='client.15822 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:55.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:55 smithi153 ceph-mon[32896]: from='client.15828 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:15:56.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:56 smithi027 ceph-mon[29208]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.1 KiB/s wr, 20 op/s 2024-04-25T00:15:56.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:56 smithi139 ceph-mon[32833]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.1 KiB/s wr, 20 op/s 2024-04-25T00:15:56.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:56 smithi153 ceph-mon[32896]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.1 KiB/s wr, 20 op/s 2024-04-25T00:15:58.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:15:58 smithi027 ceph-mon[29208]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 85 KiB/s wr, 57 op/s 2024-04-25T00:15:58.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:15:58 smithi139 ceph-mon[32833]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 85 KiB/s wr, 57 op/s 2024-04-25T00:15:58.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:15:58 smithi153 ceph-mon[32896]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 85 KiB/s wr, 57 op/s 2024-04-25T00:16:00.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:00 smithi027 ceph-mon[29208]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 52 op/s 2024-04-25T00:16:00.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:00 smithi139 ceph-mon[32833]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 52 op/s 2024-04-25T00:16:00.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:00 smithi153 ceph-mon[32896]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 52 op/s 2024-04-25T00:16:01.442 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:16:00 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:16:00.994+0000 7fa563c9c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:16:01.442 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:16:00 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:16:00.994+0000 7fa563c9c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:16:01.443 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:16:00 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:16:00.994+0000 7fa563c9c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:16:01.443 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:16:00 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:16:00.994+0000 7fa563c9c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:16:01.443 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:16:00 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:16:00.994+0000 7fa563c9c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T00:16:02.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:02 smithi027 ceph-mon[29208]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 52 op/s 2024-04-25T00:16:02.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:02 smithi027 ceph-mon[29208]: mgrmap e25: x(active, since 16s), standbys: z, y 2024-04-25T00:16:02.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:02 smithi139 ceph-mon[32833]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 52 op/s 2024-04-25T00:16:02.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:02 smithi139 ceph-mon[32833]: mgrmap e25: x(active, since 16s), standbys: z, y 2024-04-25T00:16:02.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:02 smithi153 ceph-mon[32896]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 52 op/s 2024-04-25T00:16:02.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:02 smithi153 ceph-mon[32896]: mgrmap e25: x(active, since 16s), standbys: z, y 2024-04-25T00:16:04.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:04 smithi027 ceph-mon[29208]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 100 KiB/s wr, 66 op/s 2024-04-25T00:16:04.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:04 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:16:04.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:04 smithi139 ceph-mon[32833]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 100 KiB/s wr, 66 op/s 2024-04-25T00:16:04.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:04 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:16:04.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:04 smithi153 ceph-mon[32896]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 100 KiB/s wr, 66 op/s 2024-04-25T00:16:04.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:04 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:16:06.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:06 smithi027 ceph-mon[29208]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 99 KiB/s wr, 50 op/s 2024-04-25T00:16:06.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:06 smithi139 ceph-mon[32833]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 99 KiB/s wr, 50 op/s 2024-04-25T00:16:06.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:06 smithi153 ceph-mon[32896]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 99 KiB/s wr, 50 op/s 2024-04-25T00:16:08.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:08 smithi027 ceph-mon[29208]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 100 KiB/s wr, 60 op/s 2024-04-25T00:16:08.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:08 smithi139 ceph-mon[32833]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 100 KiB/s wr, 60 op/s 2024-04-25T00:16:08.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:08 smithi153 ceph-mon[32896]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 100 KiB/s wr, 60 op/s 2024-04-25T00:16:10.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:10 smithi027 ceph-mon[29208]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-25T00:16:10.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:10 smithi139 ceph-mon[32833]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-25T00:16:10.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:10 smithi153 ceph-mon[32896]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-25T00:16:12.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:12 smithi027 ceph-mon[29208]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-25T00:16:12.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:12 smithi139 ceph-mon[32833]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-25T00:16:12.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:12 smithi153 ceph-mon[32896]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-25T00:16:14.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:14 smithi027 ceph-mon[29208]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-04-25T00:16:14.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:14 smithi139 ceph-mon[32833]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-04-25T00:16:14.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:14 smithi153 ceph-mon[32896]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-04-25T00:16:16.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:16 smithi027 ceph-mon[29208]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 767 B/s wr, 14 op/s 2024-04-25T00:16:16.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:16 smithi139 ceph-mon[32833]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 767 B/s wr, 14 op/s 2024-04-25T00:16:16.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:16 smithi153 ceph-mon[32896]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 767 B/s wr, 14 op/s 2024-04-25T00:16:18.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:18 smithi027 ceph-mon[29208]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 767 B/s wr, 18 op/s 2024-04-25T00:16:18.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:18 smithi139 ceph-mon[32833]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 767 B/s wr, 18 op/s 2024-04-25T00:16:18.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:18 smithi153 ceph-mon[32896]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 767 B/s wr, 18 op/s 2024-04-25T00:16:20.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:20 smithi027 ceph-mon[29208]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T00:16:20.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:20 smithi139 ceph-mon[32833]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T00:16:20.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:20 smithi153 ceph-mon[32896]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T00:16:22.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:22 smithi027 ceph-mon[29208]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-25T00:16:22.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:22 smithi139 ceph-mon[32833]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-25T00:16:22.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:22 smithi153 ceph-mon[32896]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-25T00:16:23.469 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:16:23.469 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:16:23.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.580+0000 7f5b4d858640 1 -- 172.21.15.27:0/785663742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b48101790 msgr2=0x7f5b48101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:23.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.580+0000 7f5b4d858640 1 --2- 172.21.15.27:0/785663742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b48101790 0x7f5b48101c10 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009930 tx=0x7f5b3c02a420 comp rx=0 tx=0).stop 2024-04-25T00:16:23.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.580+0000 7f5b4d858640 1 -- 172.21.15.27:0/785663742 shutdown_connections 2024-04-25T00:16:23.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.580+0000 7f5b4d858640 1 --2- 172.21.15.27:0/785663742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5b48102150 0x7f5b4810e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.580+0000 7f5b4d858640 1 --2- 172.21.15.27:0/785663742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b48101790 0x7f5b48101c10 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.580+0000 7f5b4d858640 1 --2- 172.21.15.27:0/785663742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b481077a0 0x7f5b48107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.580+0000 7f5b4d858640 1 -- 172.21.15.27:0/785663742 >> 172.21.15.27:0/785663742 conn(0x7f5b480fd2a0 msgr2=0x7f5b480ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:23.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.580+0000 7f5b4d858640 1 -- 172.21.15.27:0/785663742 shutdown_connections 2024-04-25T00:16:23.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.580+0000 7f5b4d858640 1 -- 172.21.15.27:0/785663742 wait complete. 2024-04-25T00:16:23.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.581+0000 7f5b4d858640 1 Processor -- start 2024-04-25T00:16:23.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.581+0000 7f5b4d858640 1 -- start start 2024-04-25T00:16:23.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b4d858640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5b48101790 0x7f5b48198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b4d858640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b48102150 0x7f5b48198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b4d858640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b481077a0 0x7f5b4819c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b4d858640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5b48111bb0 con 0x7f5b48102150 2024-04-25T00:16:23.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b4d858640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5b48111a30 con 0x7f5b48101790 2024-04-25T00:16:23.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b4d858640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5b48111d30 con 0x7f5b481077a0 2024-04-25T00:16:23.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b467fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b48102150 0x7f5b48198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b467fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b48102150 0x7f5b48198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35164/0 (socket says 172.21.15.27:35164) 2024-04-25T00:16:23.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b467fc640 1 -- 172.21.15.27:0/4083452920 learned_addr learned my addr 172.21.15.27:0/4083452920 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:23.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b477fe640 1 --2- 172.21.15.27:0/4083452920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b481077a0 0x7f5b4819c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b46ffd640 1 --2- 172.21.15.27:0/4083452920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5b48101790 0x7f5b48198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b467fc640 1 -- 172.21.15.27:0/4083452920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b481077a0 msgr2=0x7f5b4819c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:23.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b467fc640 1 --2- 172.21.15.27:0/4083452920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b481077a0 0x7f5b4819c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b467fc640 1 -- 172.21.15.27:0/4083452920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5b48101790 msgr2=0x7f5b48198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:23.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b467fc640 1 --2- 172.21.15.27:0/4083452920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5b48101790 0x7f5b48198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b467fc640 1 -- 172.21.15.27:0/4083452920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b4819d120 con 0x7f5b48102150 2024-04-25T00:16:23.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b46ffd640 1 --2- 172.21.15.27:0/4083452920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5b48101790 0x7f5b48198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:23.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.582+0000 7f5b467fc640 1 --2- 172.21.15.27:0/4083452920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b48102150 0x7f5b48198670 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009230 tx=0x7f5b3c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:23.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.583+0000 7f5b4c856640 1 -- 172.21.15.27:0/4083452920 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c03f030 con 0x7f5b48102150 2024-04-25T00:16:23.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.583+0000 7f5b4d858640 1 -- 172.21.15.27:0/4083452920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b4819d3b0 con 0x7f5b48102150 2024-04-25T00:16:23.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.583+0000 7f5b4c856640 1 -- 172.21.15.27:0/4083452920 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b3c003ad0 con 0x7f5b48102150 2024-04-25T00:16:23.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.583+0000 7f5b4c856640 1 -- 172.21.15.27:0/4083452920 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c035070 con 0x7f5b48102150 2024-04-25T00:16:23.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.583+0000 7f5b4d858640 1 -- 172.21.15.27:0/4083452920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b481a4ca0 con 0x7f5b48102150 2024-04-25T00:16:23.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.585+0000 7f5b4c856640 1 -- 172.21.15.27:0/4083452920 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f5b3c042030 con 0x7f5b48102150 2024-04-25T00:16:23.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.585+0000 7f5b50567740 1 Processor -- start 2024-04-25T00:16:23.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.585+0000 7f5b50567740 1 -- start start 2024-04-25T00:16:23.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b50567740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9522360 0x5644b9522740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b50567740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9522d10 0x5644b95314f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b50567740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9531a30 0x5644b9533e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b50567740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5644b94876b0 con 0x5644b9531a30 2024-04-25T00:16:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b50567740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5644b942f880 con 0x5644b9522360 2024-04-25T00:16:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b50567740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5644b9483670 con 0x5644b9522d10 2024-04-25T00:16:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b467fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9531a30 0x5644b9533e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b477fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9522360 0x5644b9522740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b477fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9522360 0x5644b9522740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:44818/0 (socket says 172.21.15.27:44818) 2024-04-25T00:16:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b467fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9531a30 0x5644b9533e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35172/0 (socket says 172.21.15.27:35172) 2024-04-25T00:16:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b477fe640 1 -- 172.21.15.27:0/2481679963 learned_addr learned my addr 172.21.15.27:0/2481679963 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b4c856640 1 --2- 172.21.15.27:0/4083452920 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5b300a2eb0 0x7f5b300a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b46ffd640 1 --2- 172.21.15.27:0/2481679963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9522d10 0x5644b95314f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b467fc640 1 -- 172.21.15.27:0/2481679963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9522d10 msgr2=0x5644b95314f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b467fc640 1 --2- 172.21.15.27:0/2481679963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9522d10 0x5644b95314f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b467fc640 1 -- 172.21.15.27:0/2481679963 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9522360 msgr2=0x5644b9522740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b467fc640 1 --2- 172.21.15.27:0/2481679963 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9522360 0x5644b9522740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b467fc640 1 -- 172.21.15.27:0/2481679963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644b92657d0 con 0x5644b9531a30 2024-04-25T00:16:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b46ffd640 1 --2- 172.21.15.27:0/2481679963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9522d10 0x5644b95314f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b477fe640 1 --2- 172.21.15.27:0/2481679963 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9522360 0x5644b9522740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b477fe640 1 --2- 172.21.15.27:0/4083452920 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5b300a2eb0 0x7f5b300a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.586+0000 7f5b4c856640 1 -- 172.21.15.27:0/4083452920 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f5b3c0f00e0 con 0x7f5b48102150 2024-04-25T00:16:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.587+0000 7f5b467fc640 1 --2- 172.21.15.27:0/2481679963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9531a30 0x5644b9533e20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c0ed280 tx=0x7f5b3c04b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96bc96a2139ad6fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.587+0000 7f5b2dffb640 1 -- 172.21.15.27:0/2481679963 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c049070 con 0x5644b9531a30 2024-04-25T00:16:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.587+0000 7f5b477fe640 1 --2- 172.21.15.27:0/4083452920 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5b300a2eb0 0x7f5b300a5370 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5b40012690 tx=0x7f5b40004a40 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.587+0000 7f5b2dffb640 1 -- 172.21.15.27:0/2481679963 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b3c0bc750 con 0x5644b9531a30 2024-04-25T00:16:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.587+0000 7f5b2dffb640 1 -- 172.21.15.27:0/2481679963 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c0c4cc0 con 0x5644b9531a30 2024-04-25T00:16:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.587+0000 7f5b50567740 1 -- 172.21.15.27:0/2481679963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9531a30 msgr2=0x5644b9533e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.587+0000 7f5b50567740 1 --2- 172.21.15.27:0/2481679963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9531a30 0x5644b9533e20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c0ed280 tx=0x7f5b3c04b000 comp rx=0 tx=0).stop 2024-04-25T00:16:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.587+0000 7f5b50567740 1 -- 172.21.15.27:0/2481679963 shutdown_connections 2024-04-25T00:16:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.588+0000 7f5b50567740 1 --2- 172.21.15.27:0/2481679963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9531a30 0x5644b9533e20 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.588+0000 7f5b50567740 1 --2- 172.21.15.27:0/2481679963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9522d10 0x5644b95314f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.588+0000 7f5b50567740 1 --2- 172.21.15.27:0/2481679963 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9522360 0x5644b9522740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.588+0000 7f5b50567740 1 -- 172.21.15.27:0/2481679963 >> 172.21.15.27:0/2481679963 conn(0x5644b951e950 msgr2=0x5644b951ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.588+0000 7f5b50567740 1 -- 172.21.15.27:0/2481679963 shutdown_connections 2024-04-25T00:16:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.588+0000 7f5b50567740 1 -- 172.21.15.27:0/2481679963 wait complete. 2024-04-25T00:16:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.588+0000 7f5b50567740 1 -- start start 2024-04-25T00:16:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.588+0000 7f5b50567740 1 Processor -- start 2024-04-25T00:16:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b50567740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522360 0x5644b952e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b50567740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9522d10 0x5644b952ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b50567740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9531a30 0x5644b952f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b50567740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5644b94876b0 con 0x5644b9522360 2024-04-25T00:16:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b50567740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5644b9442f50 con 0x5644b9531a30 2024-04-25T00:16:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b50567740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5644b95345b0 con 0x5644b9522d10 2024-04-25T00:16:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b50567740 10 client.0 _ll_register_callbacks cb 0x5644b947e000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:23.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b46ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522360 0x5644b952e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b46ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522360 0x5644b952e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35174/0 (socket says 172.21.15.27:35174) 2024-04-25T00:16:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b46ffd640 1 -- 172.21.15.27:0/1585564239 learned_addr learned my addr 172.21.15.27:0/1585564239 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b477fe640 1 --2- 172.21.15.27:0/1585564239 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9531a30 0x5644b952f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b467fc640 1 --2- 172.21.15.27:0/1585564239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9522d10 0x5644b952ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b46ffd640 1 -- 172.21.15.27:0/1585564239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9522d10 msgr2=0x5644b952ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b46ffd640 1 --2- 172.21.15.27:0/1585564239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9522d10 0x5644b952ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b46ffd640 1 -- 172.21.15.27:0/1585564239 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9531a30 msgr2=0x5644b952f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b46ffd640 1 --2- 172.21.15.27:0/1585564239 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9531a30 0x5644b952f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b46ffd640 1 -- 172.21.15.27:0/1585564239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644b919b3f0 con 0x5644b9522360 2024-04-25T00:16:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b46ffd640 1 --2- 172.21.15.27:0/1585564239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522360 0x5644b952e950 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5b380126f0 tx=0x7f5b38012bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d356bdba7a3d5be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.589+0000 7f5b26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.590+0000 7f5b26ffd640 1 -- 172.21.15.27:0/1585564239 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b38004460 con 0x5644b9522360 2024-04-25T00:16:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.590+0000 7f5b26ffd640 1 -- 172.21.15.27:0/1585564239 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f5b38010950 con 0x5644b9522360 2024-04-25T00:16:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.590+0000 7f5b50567740 1 -- 172.21.15.27:0/1585564239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b3c0bc750 con 0x5644b9522360 2024-04-25T00:16:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.590+0000 7f5b26ffd640 1 -- 172.21.15.27:0/1585564239 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5b38010b50 con 0x5644b9522360 2024-04-25T00:16:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.590+0000 7f5b50567740 10 client.15846 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.590+0000 7f5b50567740 1 -- 172.21.15.27:0/1585564239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5644b915b0f0 con 0x5644b9522360 2024-04-25T00:16:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.590+0000 7f5b26ffd640 1 -- 172.21.15.27:0/1585564239 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b3c0bc750 con 0x5644b9522360 2024-04-25T00:16:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.590+0000 7f5b26ffd640 1 -- 172.21.15.27:0/1585564239 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5b38019070 con 0x5644b9522360 2024-04-25T00:16:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.590+0000 7f5b50567740 10 client.15846 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.590+0000 7f5b50567740 10 client.15846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:16:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.590+0000 7f5b50567740 20 client.15846 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.590+0000 7f5b50567740 1 --2- 172.21.15.27:0/1585564239 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5644b95419a0 0x5644b9543d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.590+0000 7f5b50567740 4 client.15846 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:16:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.590+0000 7f5b50567740 1 -- 172.21.15.27:0/1585564239 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644b9494bf0 con 0x5644b95419a0 2024-04-25T00:16:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.591+0000 7f5b467fc640 1 --2- 172.21.15.27:0/1585564239 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5644b95419a0 0x5644b9543d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.591+0000 7f5b26ffd640 10 client.15846 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.591+0000 7f5b467fc640 1 --2- 172.21.15.27:0/1585564239 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5644b95419a0 0x5644b9543d90 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1c938115bdded1d server_cookie=6eab5e98aef31b58 in_seq=0 out_seq=0 2024-04-25T00:16:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.592+0000 7f5b26ffd640 1 -- 172.21.15.27:0/1585564239 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5644b9494bf0 con 0x5644b95419a0 2024-04-25T00:16:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.592+0000 7f5b26ffd640 10 client.15846 handle_command_reply: tid=0 2024-04-25T00:16:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.592+0000 7f5b50567740 1 client.15846 shutdown 2024-04-25T00:16:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.592+0000 7f5b50567740 1 -- 172.21.15.27:0/1585564239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522360 msgr2=0x5644b952e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.592+0000 7f5b50567740 1 --2- 172.21.15.27:0/1585564239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522360 0x5644b952e950 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5b380126f0 tx=0x7f5b38012bb0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.592+0000 7f5b50567740 1 -- 172.21.15.27:0/1585564239 shutdown_connections 2024-04-25T00:16:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.592+0000 7f5b50567740 1 --2- 172.21.15.27:0/1585564239 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5644b95419a0 0x5644b9543d90 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.592+0000 7f5b50567740 1 --2- 172.21.15.27:0/1585564239 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9531a30 0x5644b952f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.592+0000 7f5b50567740 1 --2- 172.21.15.27:0/1585564239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9522d10 0x5644b952ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.592+0000 7f5b50567740 1 --2- 172.21.15.27:0/1585564239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522360 0x5644b952e950 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.592+0000 7f5b50567740 1 -- 172.21.15.27:0/1585564239 >> 172.21.15.27:0/1585564239 conn(0x5644b951e950 msgr2=0x5644b95247a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:23.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.592+0000 7f5b26ffd640 0 client.15846 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:23.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.592+0000 7f5b50567740 1 -- 172.21.15.27:0/1585564239 shutdown_connections 2024-04-25T00:16:23.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.592+0000 7f5b50567740 1 -- 172.21.15.27:0/1585564239 wait complete. 2024-04-25T00:16:23.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.592+0000 7f5b50567740 20 client.15846 trim_cache size 0 max 16384 2024-04-25T00:16:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b50567740 1 Processor -- start 2024-04-25T00:16:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b50567740 1 -- start start 2024-04-25T00:16:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b50567740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522360 0x5644b953d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b50567740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9522d10 0x5644b952e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b50567740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9531a30 0x5644b952f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b50567740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5b481ac990 con 0x5644b9522360 2024-04-25T00:16:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b50567740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5b481ae4d0 con 0x5644b9531a30 2024-04-25T00:16:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b50567740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5b481ad060 con 0x5644b9522d10 2024-04-25T00:16:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b46ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522360 0x5644b953d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b46ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522360 0x5644b953d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35176/0 (socket says 172.21.15.27:35176) 2024-04-25T00:16:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b46ffd640 1 -- 172.21.15.27:0/1476872259 learned_addr learned my addr 172.21.15.27:0/1476872259 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b477fe640 1 --2- 172.21.15.27:0/1476872259 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9531a30 0x5644b952f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b467fc640 1 --2- 172.21.15.27:0/1476872259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9522d10 0x5644b952e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b46ffd640 1 -- 172.21.15.27:0/1476872259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9522d10 msgr2=0x5644b952e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b46ffd640 1 --2- 172.21.15.27:0/1476872259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9522d10 0x5644b952e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b46ffd640 1 -- 172.21.15.27:0/1476872259 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9531a30 msgr2=0x5644b952f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b46ffd640 1 --2- 172.21.15.27:0/1476872259 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9531a30 0x5644b952f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b46ffd640 1 -- 172.21.15.27:0/1476872259 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644b951e780 con 0x5644b9522360 2024-04-25T00:16:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.621+0000 7f5b477fe640 1 --2- 172.21.15.27:0/1476872259 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9531a30 0x5644b952f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.622+0000 7f5b46ffd640 1 --2- 172.21.15.27:0/1476872259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522360 0x5644b953d330 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5b3801b6b0 tx=0x7f5b380096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21afb61c26fb9df2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.622+0000 7f5b2d7fa640 1 -- 172.21.15.27:0/1476872259 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3802e070 con 0x5644b9522360 2024-04-25T00:16:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.622+0000 7f5b2d7fa640 1 -- 172.21.15.27:0/1476872259 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b380057a0 con 0x5644b9522360 2024-04-25T00:16:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.622+0000 7f5b2d7fa640 1 -- 172.21.15.27:0/1476872259 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b38000e60 con 0x5644b9522360 2024-04-25T00:16:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.622+0000 7f5b50567740 1 -- 172.21.15.27:0/1476872259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522360 msgr2=0x5644b953d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.622+0000 7f5b50567740 1 --2- 172.21.15.27:0/1476872259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522360 0x5644b953d330 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5b3801b6b0 tx=0x7f5b380096a0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.622+0000 7f5b50567740 1 -- 172.21.15.27:0/1476872259 shutdown_connections 2024-04-25T00:16:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.622+0000 7f5b50567740 1 --2- 172.21.15.27:0/1476872259 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9531a30 0x5644b952f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.622+0000 7f5b50567740 1 --2- 172.21.15.27:0/1476872259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9522d10 0x5644b952e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.622+0000 7f5b50567740 1 --2- 172.21.15.27:0/1476872259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522360 0x5644b953d330 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.622+0000 7f5b50567740 1 -- 172.21.15.27:0/1476872259 >> 172.21.15.27:0/1476872259 conn(0x5644b951e950 msgr2=0x5644b952e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.623+0000 7f5b50567740 1 -- 172.21.15.27:0/1476872259 shutdown_connections 2024-04-25T00:16:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.623+0000 7f5b50567740 1 -- 172.21.15.27:0/1476872259 wait complete. 2024-04-25T00:16:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.623+0000 7f5b50567740 1 -- start start 2024-04-25T00:16:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.623+0000 7f5b50567740 1 Processor -- start 2024-04-25T00:16:23.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.623+0000 7f5b50567740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9522360 0x5644b9539840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.623+0000 7f5b50567740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522d10 0x5644b9539d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.623+0000 7f5b50567740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9531a30 0x5644b952b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.623+0000 7f5b50567740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5b481ade00 con 0x5644b9522d10 2024-04-25T00:16:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.623+0000 7f5b50567740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5b481ad730 con 0x5644b9522360 2024-04-25T00:16:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.623+0000 7f5b50567740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5b481aa950 con 0x5644b9531a30 2024-04-25T00:16:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.623+0000 7f5b50567740 10 client.0 _ll_register_callbacks cb 0x7f5b38009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.623+0000 7f5b467fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522d10 0x5644b9539d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.623+0000 7f5b467fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522d10 0x5644b9539d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35180/0 (socket says 172.21.15.27:35180) 2024-04-25T00:16:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.624+0000 7f5b467fc640 1 -- 172.21.15.27:0/3387999194 learned_addr learned my addr 172.21.15.27:0/3387999194 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.624+0000 7f5b46ffd640 1 --2- 172.21.15.27:0/3387999194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9522360 0x5644b9539840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.624+0000 7f5b477fe640 1 --2- 172.21.15.27:0/3387999194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9531a30 0x5644b952b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.624+0000 7f5b467fc640 1 -- 172.21.15.27:0/3387999194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9531a30 msgr2=0x5644b952b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.624+0000 7f5b467fc640 1 --2- 172.21.15.27:0/3387999194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9531a30 0x5644b952b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.624+0000 7f5b467fc640 1 -- 172.21.15.27:0/3387999194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9522360 msgr2=0x5644b9539840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:23.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.624+0000 7f5b467fc640 1 --2- 172.21.15.27:0/3387999194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9522360 0x5644b9539840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.624+0000 7f5b467fc640 1 -- 172.21.15.27:0/3387999194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644b952b980 con 0x5644b9522d10 2024-04-25T00:16:23.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.624+0000 7f5b467fc640 1 --2- 172.21.15.27:0/3387999194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522d10 0x5644b9539d80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009900 tx=0x7f5b3c02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d064a9593fc5269f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:23.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.624+0000 7f5b26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:23.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.624+0000 7f5b26ffd640 1 -- 172.21.15.27:0/3387999194 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c049040 con 0x5644b9522d10 2024-04-25T00:16:23.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.624+0000 7f5b26ffd640 1 -- 172.21.15.27:0/3387999194 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f5b3c0ba350 con 0x5644b9522d10 2024-04-25T00:16:23.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.624+0000 7f5b50567740 1 -- 172.21.15.27:0/3387999194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b380057a0 con 0x5644b9522d10 2024-04-25T00:16:23.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.624+0000 7f5b26ffd640 1 -- 172.21.15.27:0/3387999194 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5b3c0bc750 con 0x5644b9522d10 2024-04-25T00:16:23.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.624+0000 7f5b50567740 10 client.15852 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:23.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.624+0000 7f5b50567740 1 -- 172.21.15.27:0/3387999194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5644b95a7a30 con 0x5644b9522d10 2024-04-25T00:16:23.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.624+0000 7f5b26ffd640 1 -- 172.21.15.27:0/3387999194 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b380057a0 con 0x5644b9522d10 2024-04-25T00:16:23.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.625+0000 7f5b26ffd640 1 -- 172.21.15.27:0/3387999194 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5b3c0b86c0 con 0x5644b9522d10 2024-04-25T00:16:23.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.625+0000 7f5b50567740 10 client.15852 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.625+0000 7f5b50567740 10 client.15852 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:16:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.625+0000 7f5b50567740 20 client.15852 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.625+0000 7f5b50567740 1 --2- 172.21.15.27:0/3387999194 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5644b95a80c0 0x5644b95aa4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.625+0000 7f5b50567740 4 client.15852 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:16:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.625+0000 7f5b50567740 1 -- 172.21.15.27:0/3387999194 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5644b948ef40 con 0x5644b95a80c0 2024-04-25T00:16:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.625+0000 7f5b46ffd640 1 --2- 172.21.15.27:0/3387999194 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5644b95a80c0 0x5644b95aa4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.626+0000 7f5b46ffd640 1 --2- 172.21.15.27:0/3387999194 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5644b95a80c0 0x5644b95aa4e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=447bf9dc3cb31a70 server_cookie=473faac8accd4723 in_seq=0 out_seq=0 2024-04-25T00:16:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.626+0000 7f5b26ffd640 10 client.15852 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.626+0000 7f5b26ffd640 1 -- 172.21.15.27:0/3387999194 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5644b948ef40 con 0x5644b95a80c0 2024-04-25T00:16:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.626+0000 7f5b26ffd640 10 client.15852 handle_command_reply: tid=0 2024-04-25T00:16:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.626+0000 7f5b50567740 1 client.15852 shutdown 2024-04-25T00:16:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.626+0000 7f5b50567740 1 -- 172.21.15.27:0/3387999194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522d10 msgr2=0x5644b9539d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.626+0000 7f5b50567740 1 --2- 172.21.15.27:0/3387999194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522d10 0x5644b9539d80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009900 tx=0x7f5b3c02a420 comp rx=0 tx=0).stop 2024-04-25T00:16:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.626+0000 7f5b50567740 1 -- 172.21.15.27:0/3387999194 shutdown_connections 2024-04-25T00:16:23.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.626+0000 7f5b50567740 1 --2- 172.21.15.27:0/3387999194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644b9531a30 0x5644b952b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.626+0000 7f5b50567740 1 --2- 172.21.15.27:0/3387999194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644b9522d10 0x5644b9539d80 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.626+0000 7f5b50567740 1 --2- 172.21.15.27:0/3387999194 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5644b95a80c0 0x5644b95aa4e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.626+0000 7f5b50567740 1 --2- 172.21.15.27:0/3387999194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5644b9522360 0x5644b9539840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.626+0000 7f5b26ffd640 0 client.15852 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:23.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.626+0000 7f5b50567740 1 -- 172.21.15.27:0/3387999194 >> 172.21.15.27:0/3387999194 conn(0x5644b951e950 msgr2=0x5644b952f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:23.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.626+0000 7f5b50567740 1 -- 172.21.15.27:0/3387999194 shutdown_connections 2024-04-25T00:16:23.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.626+0000 7f5b50567740 1 -- 172.21.15.27:0/3387999194 wait complete. 2024-04-25T00:16:23.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.626+0000 7f5b50567740 20 client.15852 trim_cache size 0 max 16384 2024-04-25T00:16:23.632 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:16:23.632 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T00:16:23.633 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T00:16:23.633 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:16:23.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.627+0000 7f5b4d858640 1 -- 172.21.15.27:0/4083452920 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5b300a2eb0 msgr2=0x7f5b300a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:23.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.627+0000 7f5b4d858640 1 --2- 172.21.15.27:0/4083452920 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5b300a2eb0 0x7f5b300a5370 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5b40012690 tx=0x7f5b40004a40 comp rx=0 tx=0).stop 2024-04-25T00:16:23.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.627+0000 7f5b4d858640 1 -- 172.21.15.27:0/4083452920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b48102150 msgr2=0x7f5b48198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.627+0000 7f5b4d858640 1 --2- 172.21.15.27:0/4083452920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b48102150 0x7f5b48198670 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009230 tx=0x7f5b3c004270 comp rx=0 tx=0).stop 2024-04-25T00:16:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.627+0000 7f5b4d858640 1 -- 172.21.15.27:0/4083452920 shutdown_connections 2024-04-25T00:16:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.627+0000 7f5b4d858640 1 --2- 172.21.15.27:0/4083452920 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5b300a2eb0 0x7f5b300a5370 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.627+0000 7f5b4d858640 1 --2- 172.21.15.27:0/4083452920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b481077a0 0x7f5b4819c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.627+0000 7f5b4d858640 1 --2- 172.21.15.27:0/4083452920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b48102150 0x7f5b48198670 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.627+0000 7f5b4d858640 1 --2- 172.21.15.27:0/4083452920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5b48101790 0x7f5b48198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.627+0000 7f5b4d858640 1 -- 172.21.15.27:0/4083452920 >> 172.21.15.27:0/4083452920 conn(0x7f5b480fd2a0 msgr2=0x7f5b480fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.627+0000 7f5b4d858640 1 -- 172.21.15.27:0/4083452920 shutdown_connections 2024-04-25T00:16:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.627+0000 7f5b4d858640 1 -- 172.21.15.27:0/4083452920 wait complete. 2024-04-25T00:16:23.638 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T00:16:23.638 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.743+0000 7f82df0da640 1 -- 172.21.15.27:0/427176286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82d8100fe0 msgr2=0x7f82d810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.743+0000 7f82df0da640 1 --2- 172.21.15.27:0/427176286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82d8100fe0 0x7f82d810d4f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009cd0 tx=0x7f82d002a8e0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.744+0000 7f82df0da640 1 -- 172.21.15.27:0/427176286 shutdown_connections 2024-04-25T00:16:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.744+0000 7f82df0da640 1 --2- 172.21.15.27:0/427176286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82d8100fe0 0x7f82d810d4f0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.744+0000 7f82df0da640 1 --2- 172.21.15.27:0/427176286 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82d8100620 0x7f82d8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.744+0000 7f82df0da640 1 --2- 172.21.15.27:0/427176286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d8106630 0x7f82d8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.744+0000 7f82df0da640 1 -- 172.21.15.27:0/427176286 >> 172.21.15.27:0/427176286 conn(0x7f82d8077740 msgr2=0x7f82d8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.744+0000 7f82df0da640 1 -- 172.21.15.27:0/427176286 shutdown_connections 2024-04-25T00:16:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.744+0000 7f82df0da640 1 -- 172.21.15.27:0/427176286 wait complete. 2024-04-25T00:16:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.744+0000 7f82df0da640 1 Processor -- start 2024-04-25T00:16:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.744+0000 7f82df0da640 1 -- start start 2024-04-25T00:16:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.744+0000 7f82df0da640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82d8100620 0x7f82d8197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.744+0000 7f82df0da640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d8100fe0 0x7f82d81984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.744+0000 7f82df0da640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82d8106630 0x7f82d819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82df0da640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f82d8110a40 con 0x7f82d8106630 2024-04-25T00:16:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82df0da640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f82d81108c0 con 0x7f82d8100620 2024-04-25T00:16:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82df0da640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f82d8110bc0 con 0x7f82d8100fe0 2024-04-25T00:16:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82dd677640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82d8106630 0x7f82d819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82dd677640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82d8106630 0x7f82d819c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35198/0 (socket says 172.21.15.27:35198) 2024-04-25T00:16:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82dd677640 1 -- 172.21.15.27:0/4188219530 learned_addr learned my addr 172.21.15.27:0/4188219530 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82d7fff640 1 --2- 172.21.15.27:0/4188219530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d8100fe0 0x7f82d81984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82dce76640 1 --2- 172.21.15.27:0/4188219530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82d8100620 0x7f82d8197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82dd677640 1 -- 172.21.15.27:0/4188219530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d8100fe0 msgr2=0x7f82d81984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82dd677640 1 --2- 172.21.15.27:0/4188219530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d8100fe0 0x7f82d81984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82dd677640 1 -- 172.21.15.27:0/4188219530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82d8100620 msgr2=0x7f82d8197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82dd677640 1 --2- 172.21.15.27:0/4188219530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82d8100620 0x7f82d8197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82dd677640 1 -- 172.21.15.27:0/4188219530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d819cf50 con 0x7f82d8106630 2024-04-25T00:16:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82dce76640 1 --2- 172.21.15.27:0/4188219530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82d8100620 0x7f82d8197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82dd677640 1 --2- 172.21.15.27:0/4188219530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82d8106630 0x7f82d819c820 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f82d0005cc0 tx=0x7f82d0032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82d5ffb640 1 -- 172.21.15.27:0/4188219530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d0004040 con 0x7f82d8106630 2024-04-25T00:16:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82df0da640 1 -- 172.21.15.27:0/4188219530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82d819d1e0 con 0x7f82d8106630 2024-04-25T00:16:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82d5ffb640 1 -- 172.21.15.27:0/4188219530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82d00046c0 con 0x7f82d8106630 2024-04-25T00:16:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.745+0000 7f82d5ffb640 1 -- 172.21.15.27:0/4188219530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d0030e00 con 0x7f82d8106630 2024-04-25T00:16:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.746+0000 7f82df0da640 1 -- 172.21.15.27:0/4188219530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82d81a4ad0 con 0x7f82d8106630 2024-04-25T00:16:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.747+0000 7f82df0da640 1 -- 172.21.15.27:0/4188219530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82d8068560 con 0x7f82d8106630 2024-04-25T00:16:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.749+0000 7f82d5ffb640 1 -- 172.21.15.27:0/4188219530 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f82d00305a0 con 0x7f82d8106630 2024-04-25T00:16:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.749+0000 7f82d5ffb640 1 --2- 172.21.15.27:0/4188219530 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f82c80a2eb0 0x7f82c80a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.749+0000 7f82dce76640 1 --2- 172.21.15.27:0/4188219530 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f82c80a2eb0 0x7f82c80a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.749+0000 7f82d5ffb640 1 -- 172.21.15.27:0/4188219530 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f82d002e030 con 0x7f82d8106630 2024-04-25T00:16:23.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.750+0000 7f82dce76640 1 --2- 172.21.15.27:0/4188219530 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f82c80a2eb0 0x7f82c80a5370 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009ba0 tx=0x7f82cc005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.753+0000 7f82d5ffb640 1 -- 172.21.15.27:0/4188219530 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f82d004b060 con 0x7f82d8106630 2024-04-25T00:16:23.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.985+0000 7f82df0da640 1 -- 172.21.15.27:0/4188219530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82d81992b0 con 0x7f82d8106630 2024-04-25T00:16:23.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.986+0000 7f82d5ffb640 1 -- 172.21.15.27:0/4188219530 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11944 (secure 0 0 0) 0x7f82d81992b0 con 0x7f82d8106630 2024-04-25T00:16:23.986 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:16:23.986 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:15:43.587614+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24514,24430,24521,24428,24496]},"id":1}]} 2024-04-25T00:16:23.987 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 29 2024-04-25T00:16:23.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.988+0000 7f82df0da640 1 -- 172.21.15.27:0/4188219530 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f82c80a2eb0 msgr2=0x7f82c80a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:23.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.988+0000 7f82df0da640 1 --2- 172.21.15.27:0/4188219530 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f82c80a2eb0 0x7f82c80a5370 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009ba0 tx=0x7f82cc005f90 comp rx=0 tx=0).stop 2024-04-25T00:16:23.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.988+0000 7f82df0da640 1 -- 172.21.15.27:0/4188219530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82d8106630 msgr2=0x7f82d819c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:23.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.988+0000 7f82df0da640 1 --2- 172.21.15.27:0/4188219530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82d8106630 0x7f82d819c820 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f82d0005cc0 tx=0x7f82d0032f40 comp rx=0 tx=0).stop 2024-04-25T00:16:23.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.988+0000 7f82df0da640 1 -- 172.21.15.27:0/4188219530 shutdown_connections 2024-04-25T00:16:23.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.988+0000 7f82df0da640 1 --2- 172.21.15.27:0/4188219530 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f82c80a2eb0 0x7f82c80a5370 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.988+0000 7f82df0da640 1 --2- 172.21.15.27:0/4188219530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82d8106630 0x7f82d819c820 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.988+0000 7f82df0da640 1 --2- 172.21.15.27:0/4188219530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82d8100fe0 0x7f82d81984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.988+0000 7f82df0da640 1 --2- 172.21.15.27:0/4188219530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82d8100620 0x7f82d8197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:23.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.988+0000 7f82df0da640 1 -- 172.21.15.27:0/4188219530 >> 172.21.15.27:0/4188219530 conn(0x7f82d8077740 msgr2=0x7f82d8068b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:23.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.989+0000 7f82df0da640 1 -- 172.21.15.27:0/4188219530 shutdown_connections 2024-04-25T00:16:23.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:23.989+0000 7f82df0da640 1 -- 172.21.15.27:0/4188219530 wait complete. 2024-04-25T00:16:24.000 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T00:16:24.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.108+0000 7fb47a129640 1 -- 172.21.15.27:0/3356657075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4741066b0 msgr2=0x7fb474106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.108+0000 7fb47a129640 1 --2- 172.21.15.27:0/3356657075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4741066b0 0x7fb474106a90 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb464009930 tx=0x7fb46402a420 comp rx=0 tx=0).stop 2024-04-25T00:16:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.108+0000 7fb47a129640 1 -- 172.21.15.27:0/3356657075 shutdown_connections 2024-04-25T00:16:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.108+0000 7fb47a129640 1 --2- 172.21.15.27:0/3356657075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb474100fd0 0x7fb47410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.108+0000 7fb47a129640 1 --2- 172.21.15.27:0/3356657075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb474100610 0x7fb474100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.108+0000 7fb47a129640 1 --2- 172.21.15.27:0/3356657075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4741066b0 0x7fb474106a90 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.108+0000 7fb47a129640 1 -- 172.21.15.27:0/3356657075 >> 172.21.15.27:0/3356657075 conn(0x7fb474077740 msgr2=0x7fb474077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.108+0000 7fb47a129640 1 -- 172.21.15.27:0/3356657075 shutdown_connections 2024-04-25T00:16:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.108+0000 7fb47a129640 1 -- 172.21.15.27:0/3356657075 wait complete. 2024-04-25T00:16:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.109+0000 7fb47a129640 1 Processor -- start 2024-04-25T00:16:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.109+0000 7fb47a129640 1 -- start start 2024-04-25T00:16:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.109+0000 7fb47a129640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb474100610 0x7fb4741022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.109+0000 7fb47a129640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb474100fd0 0x7fb4741027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.109+0000 7fb47a129640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4741066b0 0x7fb474102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.109+0000 7fb47a129640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb474110a80 con 0x7fb474100610 2024-04-25T00:16:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.109+0000 7fb47a129640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb474110900 con 0x7fb474100fd0 2024-04-25T00:16:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.109+0000 7fb47a129640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb474110c00 con 0x7fb4741066b0 2024-04-25T00:16:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.109+0000 7fb4737fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb474100610 0x7fb4741022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.109+0000 7fb4737fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb474100610 0x7fb4741022b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35210/0 (socket says 172.21.15.27:35210) 2024-04-25T00:16:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.109+0000 7fb4737fe640 1 -- 172.21.15.27:0/1307324130 learned_addr learned my addr 172.21.15.27:0/1307324130 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.110+0000 7fb473fff640 1 --2- 172.21.15.27:0/1307324130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4741066b0 0x7fb474102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.110+0000 7fb472ffd640 1 --2- 172.21.15.27:0/1307324130 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb474100fd0 0x7fb4741027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.110+0000 7fb4737fe640 1 -- 172.21.15.27:0/1307324130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4741066b0 msgr2=0x7fb474102d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.110+0000 7fb4737fe640 1 --2- 172.21.15.27:0/1307324130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4741066b0 0x7fb474102d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.110+0000 7fb4737fe640 1 -- 172.21.15.27:0/1307324130 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb474100fd0 msgr2=0x7fb4741027f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.110+0000 7fb4737fe640 1 --2- 172.21.15.27:0/1307324130 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb474100fd0 0x7fb4741027f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.110+0000 7fb4737fe640 1 -- 172.21.15.27:0/1307324130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4741a6480 con 0x7fb474100610 2024-04-25T00:16:24.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.110+0000 7fb472ffd640 1 --2- 172.21.15.27:0/1307324130 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb474100fd0 0x7fb4741027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:24.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.110+0000 7fb4737fe640 1 --2- 172.21.15.27:0/1307324130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb474100610 0x7fb4741022b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb46402a940 tx=0x7fb464003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.110+0000 7fb470ff9640 1 -- 172.21.15.27:0/1307324130 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46403f070 con 0x7fb474100610 2024-04-25T00:16:24.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.110+0000 7fb47a129640 1 -- 172.21.15.27:0/1307324130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4741a6710 con 0x7fb474100610 2024-04-25T00:16:24.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.110+0000 7fb470ff9640 1 -- 172.21.15.27:0/1307324130 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb464031c90 con 0x7fb474100610 2024-04-25T00:16:24.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.110+0000 7fb470ff9640 1 -- 172.21.15.27:0/1307324130 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb464035030 con 0x7fb474100610 2024-04-25T00:16:24.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.110+0000 7fb47a129640 1 -- 172.21.15.27:0/1307324130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4741a6c50 con 0x7fb474100610 2024-04-25T00:16:24.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.113+0000 7fb47ce37740 1 Processor -- start 2024-04-25T00:16:24.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.113+0000 7fb47ce37740 1 -- start start 2024-04-25T00:16:24.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb470ff9640 1 -- 172.21.15.27:0/1307324130 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fb464042070 con 0x7fb474100610 2024-04-25T00:16:24.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb47ce37740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dcc580 0x564105dcc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb47ce37740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dccea0 0x564105dd7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb47ce37740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dd7570 0x564105dd9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb47ce37740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564105d31670 con 0x564105dd7570 2024-04-25T00:16:24.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb47ce37740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564105cd9970 con 0x564105dccea0 2024-04-25T00:16:24.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb47ce37740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564105d2d630 con 0x564105dcc580 2024-04-25T00:16:24.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb472ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dd7570 0x564105dd9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb472ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dd7570 0x564105dd9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35220/0 (socket says 172.21.15.27:35220) 2024-04-25T00:16:24.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb472ffd640 1 -- 172.21.15.27:0/3492668635 learned_addr learned my addr 172.21.15.27:0/3492668635 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb4737fe640 1 --2- 172.21.15.27:0/3492668635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dccea0 0x564105dd7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb473fff640 1 --2- 172.21.15.27:0/3492668635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dcc580 0x564105dcc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb472ffd640 1 -- 172.21.15.27:0/3492668635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dcc580 msgr2=0x564105dcc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb472ffd640 1 --2- 172.21.15.27:0/3492668635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dcc580 0x564105dcc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb472ffd640 1 -- 172.21.15.27:0/3492668635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dccea0 msgr2=0x564105dd7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb472ffd640 1 --2- 172.21.15.27:0/3492668635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dccea0 0x564105dd7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb472ffd640 1 -- 172.21.15.27:0/3492668635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564105cfd400 con 0x564105dd7570 2024-04-25T00:16:24.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb472ffd640 1 --2- 172.21.15.27:0/3492668635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dd7570 0x564105dd9960 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb468009c00 tx=0x7fb468010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea883a09815eb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.114+0000 7fb470ff9640 1 --2- 172.21.15.27:0/1307324130 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb45c0a2ee0 0x7fb45c0a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.115+0000 7fb459ffb640 1 -- 172.21.15.27:0/3492668635 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb468005020 con 0x564105dd7570 2024-04-25T00:16:24.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.115+0000 7fb473fff640 1 --2- 172.21.15.27:0/1307324130 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb45c0a2ee0 0x7fb45c0a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.115+0000 7fb470ff9640 1 -- 172.21.15.27:0/1307324130 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fb4640f01d0 con 0x7fb474100610 2024-04-25T00:16:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.115+0000 7fb459ffb640 1 -- 172.21.15.27:0/3492668635 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4680047a0 con 0x564105dd7570 2024-04-25T00:16:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.115+0000 7fb459ffb640 1 -- 172.21.15.27:0/3492668635 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb468012760 con 0x564105dd7570 2024-04-25T00:16:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.115+0000 7fb47ce37740 1 -- 172.21.15.27:0/3492668635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dd7570 msgr2=0x564105dd9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.115+0000 7fb473fff640 1 --2- 172.21.15.27:0/1307324130 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb45c0a2ee0 0x7fb45c0a53a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb46c005fd0 tx=0x7fb46c009500 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.115+0000 7fb47ce37740 1 --2- 172.21.15.27:0/3492668635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dd7570 0x564105dd9960 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb468009c00 tx=0x7fb468010740 comp rx=0 tx=0).stop 2024-04-25T00:16:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.115+0000 7fb47ce37740 1 -- 172.21.15.27:0/3492668635 shutdown_connections 2024-04-25T00:16:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.115+0000 7fb47ce37740 1 --2- 172.21.15.27:0/3492668635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dd7570 0x564105dd9960 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.115+0000 7fb47ce37740 1 --2- 172.21.15.27:0/3492668635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dccea0 0x564105dd7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.115+0000 7fb47ce37740 1 --2- 172.21.15.27:0/3492668635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dcc580 0x564105dcc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.116+0000 7fb47ce37740 1 -- 172.21.15.27:0/3492668635 >> 172.21.15.27:0/3492668635 conn(0x564105dc88a0 msgr2=0x564105dc8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.116+0000 7fb47ce37740 1 -- 172.21.15.27:0/3492668635 shutdown_connections 2024-04-25T00:16:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.116+0000 7fb47ce37740 1 -- 172.21.15.27:0/3492668635 wait complete. 2024-04-25T00:16:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.116+0000 7fb47ce37740 1 -- start start 2024-04-25T00:16:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.116+0000 7fb47ce37740 1 Processor -- start 2024-04-25T00:16:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.116+0000 7fb47ce37740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dcc580 0x564105dd6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.116+0000 7fb47ce37740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dccea0 0x564105dd41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.116+0000 7fb47ce37740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dd7570 0x564105dd46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.116+0000 7fb47ce37740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564105dda360 con 0x564105dd7570 2024-04-25T00:16:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.116+0000 7fb47ce37740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564105ced050 con 0x564105dccea0 2024-04-25T00:16:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.116+0000 7fb47ce37740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564105dda4e0 con 0x564105dcc580 2024-04-25T00:16:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.116+0000 7fb47ce37740 10 client.0 _ll_register_callbacks cb 0x564105a44be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb473fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dd7570 0x564105dd46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb473fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dd7570 0x564105dd46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35222/0 (socket says 172.21.15.27:35222) 2024-04-25T00:16:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb4737fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dcc580 0x564105dd6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb473fff640 1 -- 172.21.15.27:0/2009147211 learned_addr learned my addr 172.21.15.27:0/2009147211 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb472ffd640 1 --2- 172.21.15.27:0/2009147211 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dccea0 0x564105dd41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb473fff640 1 -- 172.21.15.27:0/2009147211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dcc580 msgr2=0x564105dd6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb473fff640 1 --2- 172.21.15.27:0/2009147211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dcc580 0x564105dd6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb473fff640 1 -- 172.21.15.27:0/2009147211 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dccea0 msgr2=0x564105dd41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb473fff640 1 --2- 172.21.15.27:0/2009147211 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dccea0 0x564105dd41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb473fff640 1 -- 172.21.15.27:0/2009147211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564105cbedc0 con 0x564105dd7570 2024-04-25T00:16:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb473fff640 1 --2- 172.21.15.27:0/2009147211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dd7570 0x564105dd46e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb46c012830 tx=0x7fb46c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56e46d02020538a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb452ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb452ffd640 1 -- 172.21.15.27:0/2009147211 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c010be0 con 0x564105dd7570 2024-04-25T00:16:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb452ffd640 1 -- 172.21.15.27:0/2009147211 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fb46c010d80 con 0x564105dd7570 2024-04-25T00:16:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb47ce37740 1 -- 172.21.15.27:0/2009147211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4680047a0 con 0x564105dd7570 2024-04-25T00:16:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb452ffd640 1 -- 172.21.15.27:0/2009147211 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb46c01b460 con 0x564105dd7570 2024-04-25T00:16:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb47ce37740 10 client.15870 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.117+0000 7fb47ce37740 1 -- 172.21.15.27:0/2009147211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564105a453f0 con 0x564105dd7570 2024-04-25T00:16:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.118+0000 7fb452ffd640 1 -- 172.21.15.27:0/2009147211 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4680047a0 con 0x564105dd7570 2024-04-25T00:16:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.118+0000 7fb452ffd640 1 -- 172.21.15.27:0/2009147211 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb46c024070 con 0x564105dd7570 2024-04-25T00:16:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.118+0000 7fb47ce37740 10 client.15870 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.118+0000 7fb47ce37740 10 client.15870 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:16:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.118+0000 7fb47ce37740 20 client.15870 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.118+0000 7fb47ce37740 1 --2- 172.21.15.27:0/2009147211 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x564105de7600 0x564105de9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.118+0000 7fb47ce37740 4 client.15870 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:16:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.118+0000 7fb47ce37740 1 -- 172.21.15.27:0/2009147211 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564105d3ebb0 con 0x564105de7600 2024-04-25T00:16:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.118+0000 7fb4737fe640 1 --2- 172.21.15.27:0/2009147211 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x564105de7600 0x564105de9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.119+0000 7fb452ffd640 10 client.15870 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:16:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.119+0000 7fb4737fe640 1 --2- 172.21.15.27:0/2009147211 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x564105de7600 0x564105de9a80 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=64206b90886bb838 server_cookie=a8326a026ad56af in_seq=0 out_seq=0 2024-04-25T00:16:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.119+0000 7fb452ffd640 1 -- 172.21.15.27:0/2009147211 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564105d3ebb0 con 0x564105de7600 2024-04-25T00:16:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.119+0000 7fb452ffd640 10 client.15870 handle_command_reply: tid=0 2024-04-25T00:16:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.119+0000 7fb47ce37740 1 client.15870 shutdown 2024-04-25T00:16:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.120+0000 7fb47ce37740 1 -- 172.21.15.27:0/2009147211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dd7570 msgr2=0x564105dd46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.120+0000 7fb47ce37740 1 --2- 172.21.15.27:0/2009147211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dd7570 0x564105dd46e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb46c012830 tx=0x7fb46c004a40 comp rx=0 tx=0).stop 2024-04-25T00:16:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.120+0000 7fb47ce37740 1 -- 172.21.15.27:0/2009147211 shutdown_connections 2024-04-25T00:16:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.120+0000 7fb47ce37740 1 --2- 172.21.15.27:0/2009147211 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x564105de7600 0x564105de9a80 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.120+0000 7fb47ce37740 1 --2- 172.21.15.27:0/2009147211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dd7570 0x564105dd46e0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.120+0000 7fb47ce37740 1 --2- 172.21.15.27:0/2009147211 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dccea0 0x564105dd41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.120+0000 7fb452ffd640 0 client.15870 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:16:24.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.120+0000 7fb47ce37740 1 --2- 172.21.15.27:0/2009147211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dcc580 0x564105dd6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.120+0000 7fb47ce37740 1 -- 172.21.15.27:0/2009147211 >> 172.21.15.27:0/2009147211 conn(0x564105dc88a0 msgr2=0x564105dc7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.120+0000 7fb47ce37740 1 -- 172.21.15.27:0/2009147211 shutdown_connections 2024-04-25T00:16:24.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.120+0000 7fb47ce37740 1 -- 172.21.15.27:0/2009147211 wait complete. 2024-04-25T00:16:24.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.120+0000 7fb47ce37740 20 client.15870 trim_cache size 0 max 16384 2024-04-25T00:16:24.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.135+0000 7fb47ce37740 1 Processor -- start 2024-04-25T00:16:24.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.135+0000 7fb47ce37740 1 -- start start 2024-04-25T00:16:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.135+0000 7fb47ce37740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dc88a0 0x564105de3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.135+0000 7fb47ce37740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dcc580 0x564105dcf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.135+0000 7fb47ce37740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dccea0 0x564105dd69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.135+0000 7fb47ce37740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb4741b0e20 con 0x564105dcc580 2024-04-25T00:16:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.135+0000 7fb47ce37740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb4741b2960 con 0x564105dc88a0 2024-04-25T00:16:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.135+0000 7fb47ce37740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb4741b14f0 con 0x564105dccea0 2024-04-25T00:16:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.136+0000 7fb472ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dcc580 0x564105dcf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.136+0000 7fb472ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dcc580 0x564105dcf600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35234/0 (socket says 172.21.15.27:35234) 2024-04-25T00:16:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.136+0000 7fb472ffd640 1 -- 172.21.15.27:0/319661937 learned_addr learned my addr 172.21.15.27:0/319661937 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.136+0000 7fb4737fe640 1 --2- 172.21.15.27:0/319661937 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dc88a0 0x564105de3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.136+0000 7fb473fff640 1 --2- 172.21.15.27:0/319661937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dccea0 0x564105dd69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.136+0000 7fb472ffd640 1 -- 172.21.15.27:0/319661937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dccea0 msgr2=0x564105dd69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.136+0000 7fb472ffd640 1 --2- 172.21.15.27:0/319661937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dccea0 0x564105dd69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.136+0000 7fb472ffd640 1 -- 172.21.15.27:0/319661937 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dc88a0 msgr2=0x564105de3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.136+0000 7fb472ffd640 1 --2- 172.21.15.27:0/319661937 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dc88a0 0x564105de3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.136+0000 7fb472ffd640 1 -- 172.21.15.27:0/319661937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564105b0f7d0 con 0x564105dcc580 2024-04-25T00:16:24.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.136+0000 7fb472ffd640 1 --2- 172.21.15.27:0/319661937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dcc580 0x564105dcf600 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x564105dc9a30 tx=0x7fb468009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b35fe8b36dcfe392 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.136+0000 7fb4597fa640 1 -- 172.21.15.27:0/319661937 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb468005040 con 0x564105dcc580 2024-04-25T00:16:24.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.136+0000 7fb4597fa640 1 -- 172.21.15.27:0/319661937 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb468029070 con 0x564105dcc580 2024-04-25T00:16:24.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.136+0000 7fb4597fa640 1 -- 172.21.15.27:0/319661937 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4680126b0 con 0x564105dcc580 2024-04-25T00:16:24.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.137+0000 7fb47ce37740 1 -- 172.21.15.27:0/319661937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dcc580 msgr2=0x564105dcf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.137+0000 7fb47ce37740 1 --2- 172.21.15.27:0/319661937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dcc580 0x564105dcf600 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x564105dc9a30 tx=0x7fb468009800 comp rx=0 tx=0).stop 2024-04-25T00:16:24.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.137+0000 7fb47ce37740 1 -- 172.21.15.27:0/319661937 shutdown_connections 2024-04-25T00:16:24.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.137+0000 7fb47ce37740 1 --2- 172.21.15.27:0/319661937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dccea0 0x564105dd69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.137+0000 7fb47ce37740 1 --2- 172.21.15.27:0/319661937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dcc580 0x564105dcf600 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.137+0000 7fb47ce37740 1 --2- 172.21.15.27:0/319661937 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dc88a0 0x564105de3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.137+0000 7fb47ce37740 1 -- 172.21.15.27:0/319661937 >> 172.21.15.27:0/319661937 conn(0x564105dd4840 msgr2=0x564105dceef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.137+0000 7fb47ce37740 1 -- 172.21.15.27:0/319661937 shutdown_connections 2024-04-25T00:16:24.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.137+0000 7fb47ce37740 1 -- 172.21.15.27:0/319661937 wait complete. 2024-04-25T00:16:24.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.137+0000 7fb47ce37740 1 -- start start 2024-04-25T00:16:24.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb47ce37740 1 Processor -- start 2024-04-25T00:16:24.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb47ce37740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dc88a0 0x564105ddf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb47ce37740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dcc580 0x564105dd0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb47ce37740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dccea0 0x564105dd0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb47ce37740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb4741b2290 con 0x564105dcc580 2024-04-25T00:16:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb47ce37740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb4741b1bc0 con 0x564105dccea0 2024-04-25T00:16:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb47ce37740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb4741aede0 con 0x564105dc88a0 2024-04-25T00:16:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb47ce37740 10 client.0 _ll_register_callbacks cb 0x7fb46c009ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb472ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dcc580 0x564105dd0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb472ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dcc580 0x564105dd0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35248/0 (socket says 172.21.15.27:35248) 2024-04-25T00:16:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb472ffd640 1 -- 172.21.15.27:0/739421990 learned_addr learned my addr 172.21.15.27:0/739421990 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb473fff640 1 --2- 172.21.15.27:0/739421990 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dccea0 0x564105dd0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb4737fe640 1 --2- 172.21.15.27:0/739421990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dc88a0 0x564105ddf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb473fff640 1 -- 172.21.15.27:0/739421990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dc88a0 msgr2=0x564105ddf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb473fff640 1 --2- 172.21.15.27:0/739421990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dc88a0 0x564105ddf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb473fff640 1 -- 172.21.15.27:0/739421990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dcc580 msgr2=0x564105dd0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb473fff640 1 --2- 172.21.15.27:0/739421990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dcc580 0x564105dd0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb473fff640 1 -- 172.21.15.27:0/739421990 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564105dd1510 con 0x564105dccea0 2024-04-25T00:16:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb4737fe640 1 --2- 172.21.15.27:0/739421990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dc88a0 0x564105ddf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.138+0000 7fb472ffd640 1 --2- 172.21.15.27:0/739421990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dcc580 0x564105dd0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.139+0000 7fb473fff640 1 --2- 172.21.15.27:0/739421990 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dccea0 0x564105dd0f50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb46c026930 tx=0x7fb46c0266f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6083a7a54505ed2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.139+0000 7fb452ffd640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T00:16:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.139+0000 7fb452ffd640 1 -- 172.21.15.27:0/739421990 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c038020 con 0x564105dccea0 2024-04-25T00:16:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.139+0000 7fb47ce37740 1 -- 172.21.15.27:0/739421990 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb468029070 con 0x564105dccea0 2024-04-25T00:16:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.139+0000 7fb452ffd640 1 -- 172.21.15.27:0/739421990 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fb46c010a30 con 0x564105dccea0 2024-04-25T00:16:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.139+0000 7fb452ffd640 1 -- 172.21.15.27:0/739421990 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb46c010c30 con 0x564105dccea0 2024-04-25T00:16:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.139+0000 7fb47ce37740 10 client.25529 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.139+0000 7fb47ce37740 1 -- 172.21.15.27:0/739421990 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564105e4d980 con 0x564105dccea0 2024-04-25T00:16:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.139+0000 7fb452ffd640 1 -- 172.21.15.27:0/739421990 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb468029070 con 0x564105dccea0 2024-04-25T00:16:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.139+0000 7fb452ffd640 1 -- 172.21.15.27:0/739421990 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb46c024040 con 0x564105dccea0 2024-04-25T00:16:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.140+0000 7fb47ce37740 10 client.25529 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.140+0000 7fb47ce37740 10 client.25529 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:16:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.140+0000 7fb47ce37740 20 client.25529 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.140+0000 7fb47ce37740 1 --2- 172.21.15.27:0/739421990 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x564105dd7570 0x564105e501c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.140+0000 7fb47ce37740 4 client.25529 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:16:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.140+0000 7fb47ce37740 1 -- 172.21.15.27:0/739421990 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564105d38f00 con 0x564105dd7570 2024-04-25T00:16:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.140+0000 7fb4737fe640 1 --2- 172.21.15.27:0/739421990 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x564105dd7570 0x564105e501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.140+0000 7fb452ffd640 10 client.25529 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:16:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.140+0000 7fb4737fe640 1 --2- 172.21.15.27:0/739421990 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x564105dd7570 0x564105e501c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=844dad047dffa40e server_cookie=dbcc86b1320f35c2 in_seq=0 out_seq=0 2024-04-25T00:16:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.140+0000 7fb452ffd640 1 -- 172.21.15.27:0/739421990 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564105d38f00 con 0x564105dd7570 2024-04-25T00:16:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.140+0000 7fb452ffd640 10 client.25529 handle_command_reply: tid=0 2024-04-25T00:16:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47ce37740 1 client.25529 shutdown 2024-04-25T00:16:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47ce37740 1 -- 172.21.15.27:0/739421990 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dccea0 msgr2=0x564105dd0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47ce37740 1 --2- 172.21.15.27:0/739421990 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dccea0 0x564105dd0f50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb46c026930 tx=0x7fb46c0266f0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47ce37740 1 -- 172.21.15.27:0/739421990 shutdown_connections 2024-04-25T00:16:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47ce37740 1 --2- 172.21.15.27:0/739421990 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x564105dd7570 0x564105e501c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47ce37740 1 --2- 172.21.15.27:0/739421990 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564105dccea0 0x564105dd0f50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47ce37740 1 --2- 172.21.15.27:0/739421990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564105dcc580 0x564105dd0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47ce37740 1 --2- 172.21.15.27:0/739421990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564105dc88a0 0x564105ddf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb452ffd640 0 client.25529 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:16:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47ce37740 1 -- 172.21.15.27:0/739421990 >> 172.21.15.27:0/739421990 conn(0x564105dd4840 msgr2=0x564105dceed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47ce37740 1 -- 172.21.15.27:0/739421990 shutdown_connections 2024-04-25T00:16:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47ce37740 1 -- 172.21.15.27:0/739421990 wait complete. 2024-04-25T00:16:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47ce37740 20 client.25529 trim_cache size 0 max 16384 2024-04-25T00:16:24.149 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:16:24.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47a129640 1 -- 172.21.15.27:0/1307324130 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb45c0a2ee0 msgr2=0x7fb45c0a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47a129640 1 --2- 172.21.15.27:0/1307324130 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb45c0a2ee0 0x7fb45c0a53a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb46c005fd0 tx=0x7fb46c009500 comp rx=0 tx=0).stop 2024-04-25T00:16:24.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47a129640 1 -- 172.21.15.27:0/1307324130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb474100610 msgr2=0x7fb4741022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47a129640 1 --2- 172.21.15.27:0/1307324130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb474100610 0x7fb4741022b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb46402a940 tx=0x7fb464003b20 comp rx=0 tx=0).stop 2024-04-25T00:16:24.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47a129640 1 -- 172.21.15.27:0/1307324130 shutdown_connections 2024-04-25T00:16:24.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47a129640 1 --2- 172.21.15.27:0/1307324130 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb45c0a2ee0 0x7fb45c0a53a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47a129640 1 --2- 172.21.15.27:0/1307324130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4741066b0 0x7fb474102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47a129640 1 --2- 172.21.15.27:0/1307324130 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb474100fd0 0x7fb4741027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.141+0000 7fb47a129640 1 --2- 172.21.15.27:0/1307324130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb474100610 0x7fb4741022b0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.142+0000 7fb47a129640 1 -- 172.21.15.27:0/1307324130 >> 172.21.15.27:0/1307324130 conn(0x7fb474077740 msgr2=0x7fb4740ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.142+0000 7fb47a129640 1 -- 172.21.15.27:0/1307324130 shutdown_connections 2024-04-25T00:16:24.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.142+0000 7fb47a129640 1 -- 172.21.15.27:0/1307324130 wait complete. 2024-04-25T00:16:24.152 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T00:16:24.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.256+0000 7fc7add7e640 1 -- 172.21.15.27:0/301480972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a81029c0 msgr2=0x7fc7a8102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.256+0000 7fc7add7e640 1 --2- 172.21.15.27:0/301480972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a81029c0 0x7fc7a8102da0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc79c009900 tx=0x7fc79c02a510 comp rx=0 tx=0).stop 2024-04-25T00:16:24.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.256+0000 7fc7add7e640 1 -- 172.21.15.27:0/301480972 shutdown_connections 2024-04-25T00:16:24.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.256+0000 7fc7add7e640 1 --2- 172.21.15.27:0/301480972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7a810bd00 0x7fc7a810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.256+0000 7fc7add7e640 1 --2- 172.21.15.27:0/301480972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7a81032e0 0x7fc7a810b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.256+0000 7fc7add7e640 1 --2- 172.21.15.27:0/301480972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a81029c0 0x7fc7a8102da0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.256+0000 7fc7add7e640 1 -- 172.21.15.27:0/301480972 >> 172.21.15.27:0/301480972 conn(0x7fc7a8077740 msgr2=0x7fc7a8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.256+0000 7fc7add7e640 1 -- 172.21.15.27:0/301480972 shutdown_connections 2024-04-25T00:16:24.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.256+0000 7fc7add7e640 1 -- 172.21.15.27:0/301480972 wait complete. 2024-04-25T00:16:24.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.257+0000 7fc7add7e640 1 Processor -- start 2024-04-25T00:16:24.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.257+0000 7fc7add7e640 1 -- start start 2024-04-25T00:16:24.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.257+0000 7fc7add7e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7a81029c0 0x7fc7a8198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.257+0000 7fc7add7e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a81032e0 0x7fc7a8198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.257+0000 7fc7add7e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7a810bd00 0x7fc7a819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.257+0000 7fc7add7e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a8111b00 con 0x7fc7a81032e0 2024-04-25T00:16:24.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.257+0000 7fc7add7e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a8111980 con 0x7fc7a810bd00 2024-04-25T00:16:24.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.257+0000 7fc7add7e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a8111c80 con 0x7fc7a81029c0 2024-04-25T00:16:24.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.258+0000 7fc7a6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a81032e0 0x7fc7a8198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.258+0000 7fc7a6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a81032e0 0x7fc7a8198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35260/0 (socket says 172.21.15.27:35260) 2024-04-25T00:16:24.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.258+0000 7fc7a6ffd640 1 -- 172.21.15.27:0/489897862 learned_addr learned my addr 172.21.15.27:0/489897862 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.258+0000 7fc7a77fe640 1 --2- 172.21.15.27:0/489897862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7a81029c0 0x7fc7a8198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.258+0000 7fc7a7fff640 1 --2- 172.21.15.27:0/489897862 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7a810bd00 0x7fc7a819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.258+0000 7fc7a6ffd640 1 -- 172.21.15.27:0/489897862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7a81029c0 msgr2=0x7fc7a8198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.258+0000 7fc7a6ffd640 1 --2- 172.21.15.27:0/489897862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7a81029c0 0x7fc7a8198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.258+0000 7fc7a6ffd640 1 -- 172.21.15.27:0/489897862 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7a810bd00 msgr2=0x7fc7a819c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.258+0000 7fc7a6ffd640 1 --2- 172.21.15.27:0/489897862 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7a810bd00 0x7fc7a819c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.258+0000 7fc7a6ffd640 1 -- 172.21.15.27:0/489897862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7a819d010 con 0x7fc7a81032e0 2024-04-25T00:16:24.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.258+0000 7fc7a6ffd640 1 --2- 172.21.15.27:0/489897862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a81032e0 0x7fc7a8198560 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc79800d950 tx=0x7fc79800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.258+0000 7fc7a4ff9640 1 -- 172.21.15.27:0/489897862 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc798017030 con 0x7fc7a81032e0 2024-04-25T00:16:24.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.258+0000 7fc7a4ff9640 1 -- 172.21.15.27:0/489897862 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc798004020 con 0x7fc7a81032e0 2024-04-25T00:16:24.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.258+0000 7fc7add7e640 1 -- 172.21.15.27:0/489897862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7a819d300 con 0x7fc7a81032e0 2024-04-25T00:16:24.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.258+0000 7fc7a4ff9640 1 -- 172.21.15.27:0/489897862 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7980047a0 con 0x7fc7a81032e0 2024-04-25T00:16:24.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.258+0000 7fc7add7e640 1 -- 172.21.15.27:0/489897862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7a81a4b90 con 0x7fc7a81032e0 2024-04-25T00:16:24.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.261+0000 7fc7a4ff9640 1 -- 172.21.15.27:0/489897862 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fc7980041d0 con 0x7fc7a81032e0 2024-04-25T00:16:24.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.261+0000 7fc7b0b6b740 1 Processor -- start 2024-04-25T00:16:24.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.261+0000 7fc7b0b6b740 1 -- start start 2024-04-25T00:16:24.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.261+0000 7fc7b0b6b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec42580 0x55f86ec42960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.261+0000 7fc7b0b6b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec42ea0 0x55f86ec4d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.261+0000 7fc7b0b6b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec4d570 0x55f86ec4f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.261+0000 7fc7b0b6b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f86eba7670 con 0x55f86ec4d570 2024-04-25T00:16:24.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.261+0000 7fc7b0b6b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f86eb4f970 con 0x55f86ec42ea0 2024-04-25T00:16:24.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.261+0000 7fc7b0b6b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f86eba3630 con 0x55f86ec42580 2024-04-25T00:16:24.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.261+0000 7fc7a6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec4d570 0x55f86ec4f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.261+0000 7fc7a4ff9640 1 --2- 172.21.15.27:0/489897862 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc7900a2eb0 0x7fc7900a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.261+0000 7fc7a6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec4d570 0x55f86ec4f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35262/0 (socket says 172.21.15.27:35262) 2024-04-25T00:16:24.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.261+0000 7fc7a6ffd640 1 -- 172.21.15.27:0/742008858 learned_addr learned my addr 172.21.15.27:0/742008858 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.262+0000 7fc7a7fff640 1 --2- 172.21.15.27:0/742008858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec42580 0x55f86ec42960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.262+0000 7fc7a77fe640 1 --2- 172.21.15.27:0/742008858 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec42ea0 0x55f86ec4d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.262+0000 7fc7a7fff640 1 --2- 172.21.15.27:0/489897862 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc7900a2eb0 0x7fc7900a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.262+0000 7fc7a4ff9640 1 -- 172.21.15.27:0/489897862 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fc7980cf460 con 0x7fc7a81032e0 2024-04-25T00:16:24.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.262+0000 7fc7a6ffd640 1 -- 172.21.15.27:0/742008858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec42580 msgr2=0x55f86ec42960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.262+0000 7fc7a6ffd640 1 --2- 172.21.15.27:0/742008858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec42580 0x55f86ec42960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.262+0000 7fc7a6ffd640 1 -- 172.21.15.27:0/742008858 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec42ea0 msgr2=0x55f86ec4d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.262+0000 7fc7a6ffd640 1 --2- 172.21.15.27:0/742008858 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec42ea0 0x55f86ec4d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.262+0000 7fc7a6ffd640 1 -- 172.21.15.27:0/742008858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f86eb73400 con 0x55f86ec4d570 2024-04-25T00:16:24.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.262+0000 7fc7a77fe640 1 --2- 172.21.15.27:0/742008858 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec42ea0 0x55f86ec4d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:24.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.262+0000 7fc7a7fff640 1 --2- 172.21.15.27:0/489897862 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc7900a2eb0 0x7fc7900a5370 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0009370 tx=0x7fc7a0009240 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.262+0000 7fc7a6ffd640 1 --2- 172.21.15.27:0/742008858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec4d570 0x55f86ec4f960 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fc7980cc570 tx=0x7fc79809af10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f19f6d05ece35fb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.262+0000 7fc78dffb640 1 -- 172.21.15.27:0/742008858 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79809bad0 con 0x55f86ec4d570 2024-04-25T00:16:24.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.262+0000 7fc78dffb640 1 -- 172.21.15.27:0/742008858 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc79809f780 con 0x55f86ec4d570 2024-04-25T00:16:24.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.262+0000 7fc78dffb640 1 -- 172.21.15.27:0/742008858 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79809a290 con 0x55f86ec4d570 2024-04-25T00:16:24.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.263+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/742008858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec4d570 msgr2=0x55f86ec4f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.263+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/742008858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec4d570 0x55f86ec4f960 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fc7980cc570 tx=0x7fc79809af10 comp rx=0 tx=0).stop 2024-04-25T00:16:24.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.263+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/742008858 shutdown_connections 2024-04-25T00:16:24.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.263+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/742008858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec4d570 0x55f86ec4f960 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.263+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/742008858 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec42ea0 0x55f86ec4d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.263+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/742008858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec42580 0x55f86ec42960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.263+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/742008858 >> 172.21.15.27:0/742008858 conn(0x55f86ec3e8a0 msgr2=0x55f86ec3ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.263+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/742008858 shutdown_connections 2024-04-25T00:16:24.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.263+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/742008858 wait complete. 2024-04-25T00:16:24.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.263+0000 7fc7b0b6b740 1 -- start start 2024-04-25T00:16:24.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7b0b6b740 1 Processor -- start 2024-04-25T00:16:24.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7b0b6b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec42580 0x55f86ec4ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7b0b6b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec42ea0 0x55f86ec4a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7b0b6b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec4d570 0x55f86ec4a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7b0b6b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f86ec50300 con 0x55f86ec4d570 2024-04-25T00:16:24.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7b0b6b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f86eb63050 con 0x55f86ec42ea0 2024-04-25T00:16:24.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7b0b6b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f86ec50480 con 0x55f86ec42580 2024-04-25T00:16:24.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7b0b6b740 10 client.0 _ll_register_callbacks cb 0x55f86e8babe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:24.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7a7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec4d570 0x55f86ec4a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7a7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec4d570 0x55f86ec4a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35274/0 (socket says 172.21.15.27:35274) 2024-04-25T00:16:24.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7a7fff640 1 -- 172.21.15.27:0/902395738 learned_addr learned my addr 172.21.15.27:0/902395738 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7a6ffd640 1 --2- 172.21.15.27:0/902395738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec42ea0 0x55f86ec4a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7a77fe640 1 --2- 172.21.15.27:0/902395738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec42580 0x55f86ec4ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7a7fff640 1 -- 172.21.15.27:0/902395738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec42580 msgr2=0x55f86ec4ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7a7fff640 1 --2- 172.21.15.27:0/902395738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec42580 0x55f86ec4ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7a7fff640 1 -- 172.21.15.27:0/902395738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec42ea0 msgr2=0x55f86ec4a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7a7fff640 1 --2- 172.21.15.27:0/902395738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec42ea0 0x55f86ec4a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.264+0000 7fc7a7fff640 1 -- 172.21.15.27:0/902395738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f86eb34dc0 con 0x55f86ec4d570 2024-04-25T00:16:24.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.265+0000 7fc7a7fff640 1 --2- 172.21.15.27:0/902395738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec4d570 0x55f86ec4a680 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00092b0 tx=0x7fc7a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7efa15caadaba48 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.265+0000 7fc786ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:24.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.265+0000 7fc786ffd640 1 -- 172.21.15.27:0/902395738 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0026020 con 0x55f86ec4d570 2024-04-25T00:16:24.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.265+0000 7fc786ffd640 1 -- 172.21.15.27:0/902395738 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fc7a0022870 con 0x55f86ec4d570 2024-04-25T00:16:24.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.265+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/902395738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc79809f780 con 0x55f86ec4d570 2024-04-25T00:16:24.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.265+0000 7fc786ffd640 1 -- 172.21.15.27:0/902395738 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc7a0031050 con 0x55f86ec4d570 2024-04-25T00:16:24.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.265+0000 7fc7b0b6b740 10 client.15888 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:24.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.265+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/902395738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f86e8bb3f0 con 0x55f86ec4d570 2024-04-25T00:16:24.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.265+0000 7fc786ffd640 1 -- 172.21.15.27:0/902395738 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc79809f780 con 0x55f86ec4d570 2024-04-25T00:16:24.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.265+0000 7fc786ffd640 1 -- 172.21.15.27:0/902395738 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc7a000f960 con 0x55f86ec4d570 2024-04-25T00:16:24.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.265+0000 7fc7b0b6b740 10 client.15888 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:24.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.265+0000 7fc7b0b6b740 10 client.15888 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:16:24.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.265+0000 7fc7b0b6b740 20 client.15888 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:24.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.265+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/902395738 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55f86ec5d5b0 0x55f86ec5fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.265+0000 7fc7b0b6b740 4 client.15888 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:16:24.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.265+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/902395738 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f86ebb4bb0 con 0x55f86ec5d5b0 2024-04-25T00:16:24.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.266+0000 7fc7a77fe640 1 --2- 172.21.15.27:0/902395738 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55f86ec5d5b0 0x55f86ec5fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.266+0000 7fc7a77fe640 1 --2- 172.21.15.27:0/902395738 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55f86ec5d5b0 0x55f86ec5fa00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3984d17917963646 server_cookie=b2a21639943c9d8b in_seq=0 out_seq=0 2024-04-25T00:16:24.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.266+0000 7fc786ffd640 10 client.15888 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:16:24.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.267+0000 7fc786ffd640 1 -- 172.21.15.27:0/902395738 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f86ebb4bb0 con 0x55f86ec5d5b0 2024-04-25T00:16:24.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.267+0000 7fc786ffd640 10 client.15888 handle_command_reply: tid=0 2024-04-25T00:16:24.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.267+0000 7fc7b0b6b740 1 client.15888 shutdown 2024-04-25T00:16:24.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.267+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/902395738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec4d570 msgr2=0x55f86ec4a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.267+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/902395738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec4d570 0x55f86ec4a680 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00092b0 tx=0x7fc7a0004a40 comp rx=0 tx=0).stop 2024-04-25T00:16:24.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.267+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/902395738 shutdown_connections 2024-04-25T00:16:24.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.267+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/902395738 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55f86ec5d5b0 0x55f86ec5fa00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.267+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/902395738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec4d570 0x55f86ec4a680 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.267+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/902395738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec42ea0 0x55f86ec4a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.267+0000 7fc786ffd640 0 client.15888 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:16:24.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.267+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/902395738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec42580 0x55f86ec4ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.267+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/902395738 >> 172.21.15.27:0/902395738 conn(0x55f86ec3e8a0 msgr2=0x55f86ec446a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.267+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/902395738 shutdown_connections 2024-04-25T00:16:24.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.267+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/902395738 wait complete. 2024-04-25T00:16:24.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.267+0000 7fc7b0b6b740 20 client.15888 trim_cache size 0 max 16384 2024-04-25T00:16:24.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7b0b6b740 1 Processor -- start 2024-04-25T00:16:24.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7b0b6b740 1 -- start start 2024-04-25T00:16:24.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7b0b6b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec42580 0x55f86ec591f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7b0b6b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec42ea0 0x55f86ec652d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7b0b6b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec4d570 0x55f86ec65810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7b0b6b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a81acad0 con 0x55f86ec42580 2024-04-25T00:16:24.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7b0b6b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a81ae610 con 0x55f86ec4d570 2024-04-25T00:16:24.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7b0b6b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a81ad1a0 con 0x55f86ec42ea0 2024-04-25T00:16:24.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7a77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec42580 0x55f86ec591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7a77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec42580 0x55f86ec591f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35276/0 (socket says 172.21.15.27:35276) 2024-04-25T00:16:24.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7a77fe640 1 -- 172.21.15.27:0/1772281452 learned_addr learned my addr 172.21.15.27:0/1772281452 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7a6ffd640 1 --2- 172.21.15.27:0/1772281452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec42ea0 0x55f86ec652d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7a7fff640 1 --2- 172.21.15.27:0/1772281452 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec4d570 0x55f86ec65810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7a77fe640 1 -- 172.21.15.27:0/1772281452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec42ea0 msgr2=0x55f86ec652d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7a77fe640 1 --2- 172.21.15.27:0/1772281452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec42ea0 0x55f86ec652d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7a77fe640 1 -- 172.21.15.27:0/1772281452 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec4d570 msgr2=0x55f86ec65810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7a77fe640 1 --2- 172.21.15.27:0/1772281452 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec4d570 0x55f86ec65810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7a77fe640 1 -- 172.21.15.27:0/1772281452 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f86e9857d0 con 0x55f86ec42580 2024-04-25T00:16:24.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7a6ffd640 1 --2- 172.21.15.27:0/1772281452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec42ea0 0x55f86ec652d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:24.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7a7fff640 1 --2- 172.21.15.27:0/1772281452 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec4d570 0x55f86ec65810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.281+0000 7fc7a77fe640 1 --2- 172.21.15.27:0/1772281452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec42580 0x55f86ec591f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc79c009230 tx=0x7fc79c0058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1799adabf64b9194 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.282+0000 7fc78d7fa640 1 -- 172.21.15.27:0/1772281452 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79c035040 con 0x55f86ec42580 2024-04-25T00:16:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.282+0000 7fc78d7fa640 1 -- 172.21.15.27:0/1772281452 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc79c005500 con 0x55f86ec42580 2024-04-25T00:16:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.282+0000 7fc78d7fa640 1 -- 172.21.15.27:0/1772281452 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79c004cd0 con 0x55f86ec42580 2024-04-25T00:16:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.282+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/1772281452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec42580 msgr2=0x55f86ec591f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.282+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/1772281452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec42580 0x55f86ec591f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc79c009230 tx=0x7fc79c0058d0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.282+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/1772281452 shutdown_connections 2024-04-25T00:16:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.282+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/1772281452 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec4d570 0x55f86ec65810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.282+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/1772281452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec42ea0 0x55f86ec652d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.282+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/1772281452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec42580 0x55f86ec591f0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.282+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/1772281452 >> 172.21.15.27:0/1772281452 conn(0x55f86ec3e8a0 msgr2=0x55f86ec3db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.282+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/1772281452 shutdown_connections 2024-04-25T00:16:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.282+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/1772281452 wait complete. 2024-04-25T00:16:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7b0b6b740 1 -- start start 2024-04-25T00:16:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7b0b6b740 1 Processor -- start 2024-04-25T00:16:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7b0b6b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec42580 0x55f86ec558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7b0b6b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec42ea0 0x55f86ec52c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7b0b6b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec4d570 0x55f86ec531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7b0b6b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a81adf40 con 0x55f86ec42580 2024-04-25T00:16:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7b0b6b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a81ad870 con 0x55f86ec42ea0 2024-04-25T00:16:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7b0b6b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a81aaa90 con 0x55f86ec4d570 2024-04-25T00:16:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7b0b6b740 10 client.0 _ll_register_callbacks cb 0x7fc7a0014790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7a77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec42580 0x55f86ec558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7a77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec42580 0x55f86ec558b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35288/0 (socket says 172.21.15.27:35288) 2024-04-25T00:16:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7a77fe640 1 -- 172.21.15.27:0/612132084 learned_addr learned my addr 172.21.15.27:0/612132084 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7a7fff640 1 --2- 172.21.15.27:0/612132084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec4d570 0x55f86ec531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7a77fe640 1 -- 172.21.15.27:0/612132084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec4d570 msgr2=0x55f86ec531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7a77fe640 1 --2- 172.21.15.27:0/612132084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec4d570 0x55f86ec531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7a77fe640 1 -- 172.21.15.27:0/612132084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec42ea0 msgr2=0x55f86ec52c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7a77fe640 1 --2- 172.21.15.27:0/612132084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec42ea0 0x55f86ec52c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.283+0000 7fc7a77fe640 1 -- 172.21.15.27:0/612132084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f86ec537c0 con 0x55f86ec42580 2024-04-25T00:16:24.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.284+0000 7fc7a77fe640 1 --2- 172.21.15.27:0/612132084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec42580 0x55f86ec558b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fc79c005760 tx=0x7fc79c0316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe95d1bc41bf922 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.284+0000 7fc786ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:24.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.284+0000 7fc786ffd640 1 -- 172.21.15.27:0/612132084 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79c058070 con 0x55f86ec42580 2024-04-25T00:16:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.284+0000 7fc786ffd640 1 -- 172.21.15.27:0/612132084 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fc79c0048b0 con 0x55f86ec42580 2024-04-25T00:16:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.284+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/612132084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc79c005500 con 0x55f86ec42580 2024-04-25T00:16:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.284+0000 7fc786ffd640 1 -- 172.21.15.27:0/612132084 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc79c035040 con 0x55f86ec42580 2024-04-25T00:16:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.284+0000 7fc7b0b6b740 10 client.15894 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.284+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/612132084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f86ecc38a0 con 0x55f86ec42580 2024-04-25T00:16:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.284+0000 7fc786ffd640 1 -- 172.21.15.27:0/612132084 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc79c005500 con 0x55f86ec42580 2024-04-25T00:16:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.284+0000 7fc786ffd640 1 -- 172.21.15.27:0/612132084 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc79c05d560 con 0x55f86ec42580 2024-04-25T00:16:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.284+0000 7fc7b0b6b740 10 client.15894 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.284+0000 7fc7b0b6b740 10 client.15894 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:16:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.284+0000 7fc7b0b6b740 20 client.15894 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.284+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/612132084 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55f86ecc3f30 0x55f86ecc6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.284+0000 7fc7b0b6b740 4 client.15894 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:16:24.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.284+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/612132084 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f86ebaef00 con 0x55f86ecc3f30 2024-04-25T00:16:24.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.285+0000 7fc7a6ffd640 1 --2- 172.21.15.27:0/612132084 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55f86ecc3f30 0x55f86ecc6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.285+0000 7fc7a6ffd640 1 --2- 172.21.15.27:0/612132084 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55f86ecc3f30 0x55f86ecc6370 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6731f2bbe3a81071 server_cookie=cf4691be86fe01ab in_seq=0 out_seq=0 2024-04-25T00:16:24.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.285+0000 7fc786ffd640 10 client.15894 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:16:24.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.285+0000 7fc786ffd640 1 -- 172.21.15.27:0/612132084 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f86ebaef00 con 0x55f86ecc3f30 2024-04-25T00:16:24.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.285+0000 7fc786ffd640 10 client.15894 handle_command_reply: tid=0 2024-04-25T00:16:24.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.285+0000 7fc7b0b6b740 1 client.15894 shutdown 2024-04-25T00:16:24.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/612132084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec42580 msgr2=0x55f86ec558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/612132084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec42580 0x55f86ec558b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fc79c005760 tx=0x7fc79c0316d0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/612132084 shutdown_connections 2024-04-25T00:16:24.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/612132084 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55f86ecc3f30 0x55f86ecc6370 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/612132084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86ec4d570 0x55f86ec531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/612132084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f86ec42ea0 0x55f86ec52c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc786ffd640 0 client.15894 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:16:24.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7b0b6b740 1 --2- 172.21.15.27:0/612132084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86ec42580 0x55f86ec558b0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/612132084 >> 172.21.15.27:0/612132084 conn(0x55f86ec3e8a0 msgr2=0x55f86ec581e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/612132084 shutdown_connections 2024-04-25T00:16:24.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7b0b6b740 1 -- 172.21.15.27:0/612132084 wait complete. 2024-04-25T00:16:24.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7b0b6b740 20 client.15894 trim_cache size 0 max 16384 2024-04-25T00:16:24.293 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:16:24.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7add7e640 1 -- 172.21.15.27:0/489897862 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc7900a2eb0 msgr2=0x7fc7900a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7add7e640 1 --2- 172.21.15.27:0/489897862 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc7900a2eb0 0x7fc7900a5370 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0009370 tx=0x7fc7a0009240 comp rx=0 tx=0).stop 2024-04-25T00:16:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7add7e640 1 -- 172.21.15.27:0/489897862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a81032e0 msgr2=0x7fc7a8198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7add7e640 1 --2- 172.21.15.27:0/489897862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a81032e0 0x7fc7a8198560 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc79800d950 tx=0x7fc79800de10 comp rx=0 tx=0).stop 2024-04-25T00:16:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7add7e640 1 -- 172.21.15.27:0/489897862 shutdown_connections 2024-04-25T00:16:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7add7e640 1 --2- 172.21.15.27:0/489897862 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc7900a2eb0 0x7fc7900a5370 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7add7e640 1 --2- 172.21.15.27:0/489897862 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7a810bd00 0x7fc7a819c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7add7e640 1 --2- 172.21.15.27:0/489897862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a81032e0 0x7fc7a8198560 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7add7e640 1 --2- 172.21.15.27:0/489897862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7a81029c0 0x7fc7a8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.286+0000 7fc7add7e640 1 -- 172.21.15.27:0/489897862 >> 172.21.15.27:0/489897862 conn(0x7fc7a8077740 msgr2=0x7fc7a80fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.287+0000 7fc7add7e640 1 -- 172.21.15.27:0/489897862 shutdown_connections 2024-04-25T00:16:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.287+0000 7fc7add7e640 1 -- 172.21.15.27:0/489897862 wait complete. 2024-04-25T00:16:24.296 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:24 smithi027 ceph-mon[29208]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T00:16:24.296 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:24 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4188219530' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:16:24.297 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T00:16:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.403+0000 7fda0e56a640 1 -- 172.21.15.27:0/499831170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda08100610 msgr2=0x7fda08100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.403+0000 7fda0e56a640 1 --2- 172.21.15.27:0/499831170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda08100610 0x7fda08100a90 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009930 tx=0x7fd9fc02a420 comp rx=0 tx=0).stop 2024-04-25T00:16:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.403+0000 7fda0e56a640 1 -- 172.21.15.27:0/499831170 shutdown_connections 2024-04-25T00:16:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.403+0000 7fda0e56a640 1 --2- 172.21.15.27:0/499831170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda08100fd0 0x7fda0810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.403+0000 7fda0e56a640 1 --2- 172.21.15.27:0/499831170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda08100610 0x7fda08100a90 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.403+0000 7fda0e56a640 1 --2- 172.21.15.27:0/499831170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda081066b0 0x7fda08106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.403+0000 7fda0e56a640 1 -- 172.21.15.27:0/499831170 >> 172.21.15.27:0/499831170 conn(0x7fda08077740 msgr2=0x7fda08077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.403+0000 7fda0e56a640 1 -- 172.21.15.27:0/499831170 shutdown_connections 2024-04-25T00:16:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.403+0000 7fda0e56a640 1 -- 172.21.15.27:0/499831170 wait complete. 2024-04-25T00:16:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.403+0000 7fda0e56a640 1 Processor -- start 2024-04-25T00:16:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.403+0000 7fda0e56a640 1 -- start start 2024-04-25T00:16:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.404+0000 7fda0e56a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda08100610 0x7fda08198db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.404+0000 7fda0e56a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda08100fd0 0x7fda081992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.404+0000 7fda0e56a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda081066b0 0x7fda081a3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.404+0000 7fda0e56a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fda08110a30 con 0x7fda081066b0 2024-04-25T00:16:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.404+0000 7fda0e56a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fda081108b0 con 0x7fda08100fd0 2024-04-25T00:16:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.404+0000 7fda0e56a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fda08110bb0 con 0x7fda08100610 2024-04-25T00:16:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.404+0000 7fda0cb07640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda081066b0 0x7fda081a3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.404+0000 7fda0cb07640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda081066b0 0x7fda081a3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35306/0 (socket says 172.21.15.27:35306) 2024-04-25T00:16:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.404+0000 7fda0cb07640 1 -- 172.21.15.27:0/1540472689 learned_addr learned my addr 172.21.15.27:0/1540472689 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.404+0000 7fda07fff640 1 --2- 172.21.15.27:0/1540472689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda08100610 0x7fda08198db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.404+0000 7fda077fe640 1 --2- 172.21.15.27:0/1540472689 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda08100fd0 0x7fda081992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.404+0000 7fda0cb07640 1 -- 172.21.15.27:0/1540472689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda08100610 msgr2=0x7fda08198db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.404+0000 7fda0cb07640 1 --2- 172.21.15.27:0/1540472689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda08100610 0x7fda08198db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.404+0000 7fda0cb07640 1 -- 172.21.15.27:0/1540472689 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda08100fd0 msgr2=0x7fda081992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.404+0000 7fda0cb07640 1 --2- 172.21.15.27:0/1540472689 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda08100fd0 0x7fda081992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.404+0000 7fda0cb07640 1 -- 172.21.15.27:0/1540472689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda081a4370 con 0x7fda081066b0 2024-04-25T00:16:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.405+0000 7fda0cb07640 1 --2- 172.21.15.27:0/1540472689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda081066b0 0x7fda081a3e30 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fda0000eb10 tx=0x7fda0000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.405+0000 7fda057fa640 1 -- 172.21.15.27:0/1540472689 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00004b20 con 0x7fda081066b0 2024-04-25T00:16:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.405+0000 7fda057fa640 1 -- 172.21.15.27:0/1540472689 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda00004cc0 con 0x7fda081066b0 2024-04-25T00:16:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.405+0000 7fda057fa640 1 -- 172.21.15.27:0/1540472689 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00010760 con 0x7fda081066b0 2024-04-25T00:16:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.405+0000 7fda0e56a640 1 -- 172.21.15.27:0/1540472689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda081a45a0 con 0x7fda081066b0 2024-04-25T00:16:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.405+0000 7fda0e56a640 1 -- 172.21.15.27:0/1540472689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda081a4ac0 con 0x7fda081066b0 2024-04-25T00:16:24.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.408+0000 7fda1127f740 1 Processor -- start 2024-04-25T00:16:24.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.408+0000 7fda057fa640 1 -- 172.21.15.27:0/1540472689 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fda00005020 con 0x7fda081066b0 2024-04-25T00:16:24.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.408+0000 7fda1127f740 1 -- start start 2024-04-25T00:16:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.408+0000 7fda1127f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556084862580 0x556084862960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.408+0000 7fda1127f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556084862ea0 0x55608486d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.408+0000 7fda1127f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55608486d570 0x55608486f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.408+0000 7fda1127f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5560847c7670 con 0x55608486d570 2024-04-25T00:16:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.408+0000 7fda1127f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55608476f970 con 0x556084862ea0 2024-04-25T00:16:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.408+0000 7fda1127f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5560847c3630 con 0x556084862580 2024-04-25T00:16:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.408+0000 7fda077fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55608486d570 0x55608486f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.409+0000 7fda077fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55608486d570 0x55608486f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35318/0 (socket says 172.21.15.27:35318) 2024-04-25T00:16:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.409+0000 7fda077fe640 1 -- 172.21.15.27:0/4044304181 learned_addr learned my addr 172.21.15.27:0/4044304181 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.409+0000 7fda077fe640 1 --2- 172.21.15.27:0/4044304181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556084862580 0x556084862960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.409+0000 7fda07fff640 1 --2- 172.21.15.27:0/4044304181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556084862ea0 0x55608486d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.409+0000 7fda057fa640 1 --2- 172.21.15.27:0/1540472689 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd9f00a2eb0 0x7fd9f00a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.409+0000 7fda077fe640 1 -- 172.21.15.27:0/4044304181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556084862580 msgr2=0x556084862960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.409+0000 7fda077fe640 1 --2- 172.21.15.27:0/4044304181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556084862580 0x556084862960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.409+0000 7fda077fe640 1 -- 172.21.15.27:0/4044304181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556084862ea0 msgr2=0x55608486d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.409+0000 7fda077fe640 1 --2- 172.21.15.27:0/4044304181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556084862ea0 0x55608486d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.409+0000 7fda077fe640 1 -- 172.21.15.27:0/4044304181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556084793400 con 0x55608486d570 2024-04-25T00:16:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.409+0000 7fda0cb07640 1 --2- 172.21.15.27:0/1540472689 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd9f00a2eb0 0x7fd9f00a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.409+0000 7fda057fa640 1 -- 172.21.15.27:0/1540472689 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fda000d01f0 con 0x7fda081066b0 2024-04-25T00:16:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.409+0000 7fda077fe640 1 --2- 172.21.15.27:0/4044304181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55608486d570 0x55608486f960 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009930 tx=0x7fd9fc005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f567b425026049 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.410+0000 7fda0cb07640 1 --2- 172.21.15.27:0/1540472689 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd9f00a2eb0 0x7fd9f00a5370 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fda000cd3f0 tx=0x7fda00097dc0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.410+0000 7fd9ee7fc640 1 -- 172.21.15.27:0/4044304181 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc035040 con 0x55608486d570 2024-04-25T00:16:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.410+0000 7fd9ee7fc640 1 -- 172.21.15.27:0/4044304181 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9fc031840 con 0x55608486d570 2024-04-25T00:16:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.410+0000 7fd9ee7fc640 1 -- 172.21.15.27:0/4044304181 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc003da0 con 0x55608486d570 2024-04-25T00:16:24.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.410+0000 7fda1127f740 1 -- 172.21.15.27:0/4044304181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55608486d570 msgr2=0x55608486f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.410+0000 7fda1127f740 1 --2- 172.21.15.27:0/4044304181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55608486d570 0x55608486f960 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009930 tx=0x7fd9fc005b50 comp rx=0 tx=0).stop 2024-04-25T00:16:24.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.410+0000 7fda1127f740 1 -- 172.21.15.27:0/4044304181 shutdown_connections 2024-04-25T00:16:24.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.410+0000 7fda1127f740 1 --2- 172.21.15.27:0/4044304181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55608486d570 0x55608486f960 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.410+0000 7fda1127f740 1 --2- 172.21.15.27:0/4044304181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556084862ea0 0x55608486d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.410+0000 7fda1127f740 1 --2- 172.21.15.27:0/4044304181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556084862580 0x556084862960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.410+0000 7fda1127f740 1 -- 172.21.15.27:0/4044304181 >> 172.21.15.27:0/4044304181 conn(0x55608485e8a0 msgr2=0x55608485ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.410+0000 7fda1127f740 1 -- 172.21.15.27:0/4044304181 shutdown_connections 2024-04-25T00:16:24.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.410+0000 7fda1127f740 1 -- 172.21.15.27:0/4044304181 wait complete. 2024-04-25T00:16:24.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.411+0000 7fda1127f740 1 -- start start 2024-04-25T00:16:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.411+0000 7fda1127f740 1 Processor -- start 2024-04-25T00:16:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.411+0000 7fda1127f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556084862580 0x55608486cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.411+0000 7fda1127f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556084862ea0 0x55608486a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.411+0000 7fda1127f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55608486d570 0x55608486a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.411+0000 7fda1127f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556084870300 con 0x55608486d570 2024-04-25T00:16:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.411+0000 7fda1127f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556084783050 con 0x556084862ea0 2024-04-25T00:16:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.411+0000 7fda1127f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556084870480 con 0x556084862580 2024-04-25T00:16:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.411+0000 7fda1127f740 10 client.0 _ll_register_callbacks cb 0x5560844dabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.411+0000 7fda077fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55608486d570 0x55608486a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.411+0000 7fda077fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55608486d570 0x55608486a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35326/0 (socket says 172.21.15.27:35326) 2024-04-25T00:16:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.411+0000 7fda077fe640 1 -- 172.21.15.27:0/1394876151 learned_addr learned my addr 172.21.15.27:0/1394876151 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.411+0000 7fda077fe640 1 --2- 172.21.15.27:0/1394876151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556084862580 0x55608486cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.412+0000 7fda07fff640 1 --2- 172.21.15.27:0/1394876151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556084862ea0 0x55608486a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.412+0000 7fda077fe640 1 -- 172.21.15.27:0/1394876151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556084862580 msgr2=0x55608486cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.412+0000 7fda077fe640 1 --2- 172.21.15.27:0/1394876151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556084862580 0x55608486cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.412+0000 7fda077fe640 1 -- 172.21.15.27:0/1394876151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556084862ea0 msgr2=0x55608486a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.412+0000 7fda077fe640 1 --2- 172.21.15.27:0/1394876151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556084862ea0 0x55608486a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.412+0000 7fda077fe640 1 -- 172.21.15.27:0/1394876151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556084754dc0 con 0x55608486d570 2024-04-25T00:16:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.412+0000 7fda077fe640 1 --2- 172.21.15.27:0/1394876151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55608486d570 0x55608486a870 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc003a80 tx=0x7fd9fc005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d4c0879a086771 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.412+0000 7fd9e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.412+0000 7fd9e77fe640 1 -- 172.21.15.27:0/1394876151 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc035040 con 0x55608486d570 2024-04-25T00:16:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.412+0000 7fd9e77fe640 1 -- 172.21.15.27:0/1394876151 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fd9fc004390 con 0x55608486d570 2024-04-25T00:16:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.412+0000 7fda1127f740 1 -- 172.21.15.27:0/1394876151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9fc031840 con 0x55608486d570 2024-04-25T00:16:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.412+0000 7fd9e77fe640 1 -- 172.21.15.27:0/1394876151 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd9fc05e050 con 0x55608486d570 2024-04-25T00:16:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.412+0000 7fda1127f740 10 client.15906 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.412+0000 7fda1127f740 1 -- 172.21.15.27:0/1394876151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5560844db3f0 con 0x55608486d570 2024-04-25T00:16:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.413+0000 7fd9e77fe640 1 -- 172.21.15.27:0/1394876151 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9fc031840 con 0x55608486d570 2024-04-25T00:16:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.413+0000 7fd9e77fe640 1 -- 172.21.15.27:0/1394876151 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd9fc03a040 con 0x55608486d570 2024-04-25T00:16:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.413+0000 7fda1127f740 10 client.15906 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.413+0000 7fda1127f740 10 client.15906 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:16:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.413+0000 7fda1127f740 20 client.15906 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.413+0000 7fda1127f740 1 --2- 172.21.15.27:0/1394876151 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55608487d4a0 0x55608487f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.413+0000 7fda1127f740 4 client.15906 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:16:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.413+0000 7fda1127f740 1 -- 172.21.15.27:0/1394876151 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560847d4bb0 con 0x55608487d4a0 2024-04-25T00:16:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.413+0000 7fda07fff640 1 --2- 172.21.15.27:0/1394876151 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55608487d4a0 0x55608487f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.414+0000 7fda07fff640 1 --2- 172.21.15.27:0/1394876151 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55608487d4a0 0x55608487f890 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d466c6867d4fa0bd server_cookie=c9183dcd88a564aa in_seq=0 out_seq=0 2024-04-25T00:16:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.414+0000 7fd9e77fe640 10 client.15906 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:16:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.414+0000 7fd9e77fe640 1 -- 172.21.15.27:0/1394876151 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5560847d4bb0 con 0x55608487d4a0 2024-04-25T00:16:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.414+0000 7fd9e77fe640 10 client.15906 handle_command_reply: tid=0 2024-04-25T00:16:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.415+0000 7fda1127f740 1 client.15906 shutdown 2024-04-25T00:16:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.415+0000 7fda1127f740 1 -- 172.21.15.27:0/1394876151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55608486d570 msgr2=0x55608486a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.415+0000 7fda1127f740 1 --2- 172.21.15.27:0/1394876151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55608486d570 0x55608486a870 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc003a80 tx=0x7fd9fc005a20 comp rx=0 tx=0).stop 2024-04-25T00:16:24.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.415+0000 7fda1127f740 1 -- 172.21.15.27:0/1394876151 shutdown_connections 2024-04-25T00:16:24.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.415+0000 7fda1127f740 1 --2- 172.21.15.27:0/1394876151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55608486d570 0x55608486a870 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.415+0000 7fda1127f740 1 --2- 172.21.15.27:0/1394876151 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55608487d4a0 0x55608487f890 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.415+0000 7fda1127f740 1 --2- 172.21.15.27:0/1394876151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556084862ea0 0x55608486a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.415+0000 7fda1127f740 1 --2- 172.21.15.27:0/1394876151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556084862580 0x55608486cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.415+0000 7fd9e77fe640 0 client.15906 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:16:24.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.415+0000 7fda1127f740 1 -- 172.21.15.27:0/1394876151 >> 172.21.15.27:0/1394876151 conn(0x55608485e8a0 msgr2=0x55608485db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.415+0000 7fda1127f740 1 -- 172.21.15.27:0/1394876151 shutdown_connections 2024-04-25T00:16:24.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.415+0000 7fda1127f740 1 -- 172.21.15.27:0/1394876151 wait complete. 2024-04-25T00:16:24.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.415+0000 7fda1127f740 20 client.15906 trim_cache size 0 max 16384 2024-04-25T00:16:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda1127f740 1 Processor -- start 2024-04-25T00:16:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda1127f740 1 -- start start 2024-04-25T00:16:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda1127f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556084862580 0x5560848828c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda1127f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556084862ea0 0x556084882e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda1127f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55608486d570 0x5560848674c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda1127f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fda081aebf0 con 0x556084862ea0 2024-04-25T00:16:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda1127f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fda081b0730 con 0x55608486d570 2024-04-25T00:16:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda1127f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fda081af2c0 con 0x556084862580 2024-04-25T00:16:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda07fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556084862ea0 0x556084882e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda07fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556084862ea0 0x556084882e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35342/0 (socket says 172.21.15.27:35342) 2024-04-25T00:16:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda07fff640 1 -- 172.21.15.27:0/2645070334 learned_addr learned my addr 172.21.15.27:0/2645070334 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda077fe640 1 --2- 172.21.15.27:0/2645070334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556084862580 0x5560848828c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda077fe640 1 --2- 172.21.15.27:0/2645070334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55608486d570 0x5560848674c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda07fff640 1 -- 172.21.15.27:0/2645070334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556084862580 msgr2=0x5560848828c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda07fff640 1 --2- 172.21.15.27:0/2645070334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556084862580 0x5560848828c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda07fff640 1 -- 172.21.15.27:0/2645070334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55608486d570 msgr2=0x5560848674c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda07fff640 1 --2- 172.21.15.27:0/2645070334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55608486d570 0x5560848674c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda07fff640 1 -- 172.21.15.27:0/2645070334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560845a57d0 con 0x556084862ea0 2024-04-25T00:16:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.429+0000 7fda077fe640 1 --2- 172.21.15.27:0/2645070334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55608486d570 0x5560848674c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.430+0000 7fda07fff640 1 --2- 172.21.15.27:0/2645070334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556084862ea0 0x556084882e00 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8009dd0 tx=0x7fd9f8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d24c3449c8dd507 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.430+0000 7fd9edffb640 1 -- 172.21.15.27:0/2645070334 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f801f630 con 0x556084862ea0 2024-04-25T00:16:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.430+0000 7fd9edffb640 1 -- 172.21.15.27:0/2645070334 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9f801f7d0 con 0x556084862ea0 2024-04-25T00:16:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.430+0000 7fd9edffb640 1 -- 172.21.15.27:0/2645070334 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f800f620 con 0x556084862ea0 2024-04-25T00:16:24.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.430+0000 7fda1127f740 1 -- 172.21.15.27:0/2645070334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556084862ea0 msgr2=0x556084882e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.430+0000 7fda1127f740 1 --2- 172.21.15.27:0/2645070334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556084862ea0 0x556084882e00 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8009dd0 tx=0x7fd9f8005f90 comp rx=0 tx=0).stop 2024-04-25T00:16:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.430+0000 7fda1127f740 1 -- 172.21.15.27:0/2645070334 shutdown_connections 2024-04-25T00:16:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.430+0000 7fda1127f740 1 --2- 172.21.15.27:0/2645070334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55608486d570 0x5560848674c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.430+0000 7fda1127f740 1 --2- 172.21.15.27:0/2645070334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556084862ea0 0x556084882e00 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.430+0000 7fda1127f740 1 --2- 172.21.15.27:0/2645070334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556084862580 0x5560848828c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.430+0000 7fda1127f740 1 -- 172.21.15.27:0/2645070334 >> 172.21.15.27:0/2645070334 conn(0x55608485e8a0 msgr2=0x556084878f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.431+0000 7fda1127f740 1 -- 172.21.15.27:0/2645070334 shutdown_connections 2024-04-25T00:16:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.431+0000 7fda1127f740 1 -- 172.21.15.27:0/2645070334 wait complete. 2024-04-25T00:16:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.431+0000 7fda1127f740 1 -- start start 2024-04-25T00:16:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.431+0000 7fda1127f740 1 Processor -- start 2024-04-25T00:16:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.431+0000 7fda1127f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556084862580 0x5560848d7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.431+0000 7fda1127f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556084862ea0 0x5560848d7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.431+0000 7fda1127f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55608486d570 0x5560848d8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.431+0000 7fda1127f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fda081b0060 con 0x556084862ea0 2024-04-25T00:16:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.431+0000 7fda1127f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fda081af990 con 0x556084862580 2024-04-25T00:16:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.431+0000 7fda1127f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fda081acbb0 con 0x55608486d570 2024-04-25T00:16:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.431+0000 7fda1127f740 10 client.0 _ll_register_callbacks cb 0x7fd9f80097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.431+0000 7fda07fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556084862ea0 0x5560848d7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.432+0000 7fda07fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556084862ea0 0x5560848d7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35354/0 (socket says 172.21.15.27:35354) 2024-04-25T00:16:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.432+0000 7fda07fff640 1 -- 172.21.15.27:0/2994753843 learned_addr learned my addr 172.21.15.27:0/2994753843 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.432+0000 7fda077fe640 1 --2- 172.21.15.27:0/2994753843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556084862580 0x5560848d7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.432+0000 7fda077fe640 1 --2- 172.21.15.27:0/2994753843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55608486d570 0x5560848d8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.432+0000 7fda07fff640 1 -- 172.21.15.27:0/2994753843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55608486d570 msgr2=0x5560848d8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.432+0000 7fda07fff640 1 --2- 172.21.15.27:0/2994753843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55608486d570 0x5560848d8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.432+0000 7fda07fff640 1 -- 172.21.15.27:0/2994753843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556084862580 msgr2=0x5560848d7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.432+0000 7fda07fff640 1 --2- 172.21.15.27:0/2994753843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556084862580 0x5560848d7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.432+0000 7fda07fff640 1 -- 172.21.15.27:0/2994753843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560848e2980 con 0x556084862ea0 2024-04-25T00:16:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.432+0000 7fda07fff640 1 --2- 172.21.15.27:0/2994753843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556084862ea0 0x5560848d7b50 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8000c00 tx=0x7fd9f8009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede0dd6ad4518812 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.432+0000 7fd9e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.432+0000 7fd9e77fe640 1 -- 172.21.15.27:0/2994753843 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f8035060 con 0x556084862ea0 2024-04-25T00:16:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.432+0000 7fda1127f740 1 -- 172.21.15.27:0/2994753843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9f801f7d0 con 0x556084862ea0 2024-04-25T00:16:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.432+0000 7fd9e77fe640 1 -- 172.21.15.27:0/2994753843 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fd9f801f430 con 0x556084862ea0 2024-04-25T00:16:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.432+0000 7fd9e77fe640 1 -- 172.21.15.27:0/2994753843 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd9f803c050 con 0x556084862ea0 2024-04-25T00:16:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.432+0000 7fda1127f740 10 client.15912 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.433+0000 7fda1127f740 1 -- 172.21.15.27:0/2994753843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5560848e3b90 con 0x556084862ea0 2024-04-25T00:16:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.433+0000 7fd9e77fe640 1 -- 172.21.15.27:0/2994753843 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9f801f7d0 con 0x556084862ea0 2024-04-25T00:16:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.433+0000 7fd9e77fe640 1 -- 172.21.15.27:0/2994753843 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd9f8030910 con 0x556084862ea0 2024-04-25T00:16:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.433+0000 7fda1127f740 10 client.15912 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.433+0000 7fda1127f740 10 client.15912 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:16:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.433+0000 7fda1127f740 20 client.15912 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:24.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.433+0000 7fda1127f740 1 --2- 172.21.15.27:0/2994753843 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5560848e4250 0x5560848e6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.433+0000 7fda1127f740 4 client.15912 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:16:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.433+0000 7fda1127f740 1 -- 172.21.15.27:0/2994753843 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5560847cef00 con 0x5560848e4250 2024-04-25T00:16:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.433+0000 7fda077fe640 1 --2- 172.21.15.27:0/2994753843 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5560848e4250 0x5560848e6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.434+0000 7fda077fe640 1 --2- 172.21.15.27:0/2994753843 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5560848e4250 0x5560848e6670 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9df3e48c767663ad server_cookie=9cee5acd555fd46b in_seq=0 out_seq=0 2024-04-25T00:16:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.434+0000 7fd9e77fe640 10 client.15912 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:16:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.434+0000 7fd9e77fe640 1 -- 172.21.15.27:0/2994753843 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5560847cef00 con 0x5560848e4250 2024-04-25T00:16:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.434+0000 7fd9e77fe640 10 client.15912 handle_command_reply: tid=0 2024-04-25T00:16:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.434+0000 7fda1127f740 1 client.15912 shutdown 2024-04-25T00:16:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.434+0000 7fda1127f740 1 -- 172.21.15.27:0/2994753843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556084862ea0 msgr2=0x5560848d7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.434+0000 7fda1127f740 1 --2- 172.21.15.27:0/2994753843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556084862ea0 0x5560848d7b50 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8000c00 tx=0x7fd9f8009b80 comp rx=0 tx=0).stop 2024-04-25T00:16:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.434+0000 7fda1127f740 1 -- 172.21.15.27:0/2994753843 shutdown_connections 2024-04-25T00:16:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.434+0000 7fda1127f740 1 --2- 172.21.15.27:0/2994753843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55608486d570 0x5560848d8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.434+0000 7fda1127f740 1 --2- 172.21.15.27:0/2994753843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556084862ea0 0x5560848d7b50 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.434+0000 7fda1127f740 1 --2- 172.21.15.27:0/2994753843 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5560848e4250 0x5560848e6670 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.434+0000 7fda1127f740 1 --2- 172.21.15.27:0/2994753843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556084862580 0x5560848d7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.434+0000 7fd9e77fe640 0 client.15912 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:16:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.434+0000 7fda1127f740 1 -- 172.21.15.27:0/2994753843 >> 172.21.15.27:0/2994753843 conn(0x55608485e8a0 msgr2=0x55608485d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.434+0000 7fda1127f740 1 -- 172.21.15.27:0/2994753843 shutdown_connections 2024-04-25T00:16:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.434+0000 7fda1127f740 1 -- 172.21.15.27:0/2994753843 wait complete. 2024-04-25T00:16:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.434+0000 7fda1127f740 20 client.15912 trim_cache size 0 max 16384 2024-04-25T00:16:24.440 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:16:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.435+0000 7fda0e56a640 1 -- 172.21.15.27:0/1540472689 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd9f00a2eb0 msgr2=0x7fd9f00a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.435+0000 7fda0e56a640 1 --2- 172.21.15.27:0/1540472689 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd9f00a2eb0 0x7fd9f00a5370 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fda000cd3f0 tx=0x7fda00097dc0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.435+0000 7fda0e56a640 1 -- 172.21.15.27:0/1540472689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda081066b0 msgr2=0x7fda081a3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.435+0000 7fda0e56a640 1 --2- 172.21.15.27:0/1540472689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda081066b0 0x7fda081a3e30 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fda0000eb10 tx=0x7fda0000c2d0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.435+0000 7fda0e56a640 1 -- 172.21.15.27:0/1540472689 shutdown_connections 2024-04-25T00:16:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.435+0000 7fda0e56a640 1 --2- 172.21.15.27:0/1540472689 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd9f00a2eb0 0x7fd9f00a5370 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.435+0000 7fda0e56a640 1 --2- 172.21.15.27:0/1540472689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda081066b0 0x7fda081a3e30 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.435+0000 7fda0e56a640 1 --2- 172.21.15.27:0/1540472689 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fda08100fd0 0x7fda081992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.435+0000 7fda0e56a640 1 --2- 172.21.15.27:0/1540472689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda08100610 0x7fda08198db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.435+0000 7fda0e56a640 1 -- 172.21.15.27:0/1540472689 >> 172.21.15.27:0/1540472689 conn(0x7fda08077740 msgr2=0x7fda080ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.435+0000 7fda0e56a640 1 -- 172.21.15.27:0/1540472689 shutdown_connections 2024-04-25T00:16:24.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.435+0000 7fda0e56a640 1 -- 172.21.15.27:0/1540472689 wait complete. 2024-04-25T00:16:24.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T00:16:24.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:24 smithi139 ceph-mon[32833]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T00:16:24.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:24 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4188219530' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:16:24.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:24 smithi153 ceph-mon[32896]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T00:16:24.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:24 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4188219530' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:16:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.555+0000 7f4ce55f9640 1 -- 172.21.15.27:0/124552898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce0100e60 msgr2=0x7f4ce010dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.555+0000 7f4ce55f9640 1 --2- 172.21.15.27:0/124552898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce0100e60 0x7f4ce010dbb0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009900 tx=0x7f4cd402a410 comp rx=0 tx=0).stop 2024-04-25T00:16:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.555+0000 7f4ce55f9640 1 -- 172.21.15.27:0/124552898 shutdown_connections 2024-04-25T00:16:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.555+0000 7f4ce55f9640 1 --2- 172.21.15.27:0/124552898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ce010e0f0 0x7f4ce01104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.555+0000 7f4ce55f9640 1 --2- 172.21.15.27:0/124552898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce0100e60 0x7f4ce010dbb0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.555+0000 7f4ce55f9640 1 --2- 172.21.15.27:0/124552898 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ce0100540 0x7f4ce0100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.555+0000 7f4ce55f9640 1 -- 172.21.15.27:0/124552898 >> 172.21.15.27:0/124552898 conn(0x7f4ce0077740 msgr2=0x7f4ce0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.555+0000 7f4ce55f9640 1 -- 172.21.15.27:0/124552898 shutdown_connections 2024-04-25T00:16:24.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.555+0000 7f4ce55f9640 1 -- 172.21.15.27:0/124552898 wait complete. 2024-04-25T00:16:24.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.555+0000 7f4ce55f9640 1 Processor -- start 2024-04-25T00:16:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4ce55f9640 1 -- start start 2024-04-25T00:16:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4ce55f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ce0100540 0x7f4ce019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4ce55f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce0100e60 0x7f4ce019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4ce55f9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ce010e0f0 0x7f4ce019eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4ce55f9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce0113d20 con 0x7f4ce0100e60 2024-04-25T00:16:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4ce55f9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce0113ba0 con 0x7f4ce010e0f0 2024-04-25T00:16:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4ce55f9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce0113ea0 con 0x7f4ce0100540 2024-04-25T00:16:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4cde7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce0100e60 0x7f4ce019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4cde7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce0100e60 0x7f4ce019a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35368/0 (socket says 172.21.15.27:35368) 2024-04-25T00:16:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4cde7fc640 1 -- 172.21.15.27:0/241386756 learned_addr learned my addr 172.21.15.27:0/241386756 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4cdeffd640 1 --2- 172.21.15.27:0/241386756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ce0100540 0x7f4ce019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4cdf7fe640 1 --2- 172.21.15.27:0/241386756 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ce010e0f0 0x7f4ce019eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4cde7fc640 1 -- 172.21.15.27:0/241386756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ce0100540 msgr2=0x7f4ce019a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4cde7fc640 1 --2- 172.21.15.27:0/241386756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ce0100540 0x7f4ce019a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4cde7fc640 1 -- 172.21.15.27:0/241386756 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ce010e0f0 msgr2=0x7f4ce019eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4cde7fc640 1 --2- 172.21.15.27:0/241386756 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ce010e0f0 0x7f4ce019eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4cde7fc640 1 -- 172.21.15.27:0/241386756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce019f1f0 con 0x7f4ce0100e60 2024-04-25T00:16:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.556+0000 7f4cdf7fe640 1 --2- 172.21.15.27:0/241386756 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ce010e0f0 0x7f4ce019eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.557+0000 7f4cde7fc640 1 --2- 172.21.15.27:0/241386756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce0100e60 0x7f4ce019a740 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4006010 tx=0x7f4cd4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.557+0000 7f4cc7fff640 1 -- 172.21.15.27:0/241386756 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd403f030 con 0x7f4ce0100e60 2024-04-25T00:16:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.557+0000 7f4ce55f9640 1 -- 172.21.15.27:0/241386756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ce019f480 con 0x7f4ce0100e60 2024-04-25T00:16:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.557+0000 7f4cc7fff640 1 -- 172.21.15.27:0/241386756 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cd4003ad0 con 0x7f4ce0100e60 2024-04-25T00:16:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.557+0000 7f4cc7fff640 1 -- 172.21.15.27:0/241386756 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4035070 con 0x7f4ce0100e60 2024-04-25T00:16:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.557+0000 7f4ce55f9640 1 -- 172.21.15.27:0/241386756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ce01a6d70 con 0x7f4ce0100e60 2024-04-25T00:16:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.559+0000 7f4cc7fff640 1 -- 172.21.15.27:0/241386756 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f4cd4042030 con 0x7f4ce0100e60 2024-04-25T00:16:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.560+0000 7f4cc7fff640 1 --2- 172.21.15.27:0/241386756 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4cc80a2eb0 0x7f4cc80a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.560+0000 7f4cdeffd640 1 --2- 172.21.15.27:0/241386756 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4cc80a2eb0 0x7f4cc80a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.560+0000 7f4cc7fff640 1 -- 172.21.15.27:0/241386756 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f4cd40f0280 con 0x7f4ce0100e60 2024-04-25T00:16:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.560+0000 7f4cdeffd640 1 --2- 172.21.15.27:0/241386756 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4cc80a2eb0 0x7f4cc80a5370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009b60 tx=0x7f4cd0005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.560+0000 7f4ce83e9740 1 Processor -- start 2024-04-25T00:16:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.560+0000 7f4ce83e9740 1 -- start start 2024-04-25T00:16:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4ce83e9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b41d580 0x55609b41d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4ce83e9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41dea0 0x55609b428030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4ce83e9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b428570 0x55609b42a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4ce83e9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55609b382670 con 0x55609b428570 2024-04-25T00:16:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4ce83e9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55609b32a970 con 0x55609b41d580 2024-04-25T00:16:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4ce83e9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55609b37e630 con 0x55609b41dea0 2024-04-25T00:16:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4cdf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b428570 0x55609b42a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4cdf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b428570 0x55609b42a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35382/0 (socket says 172.21.15.27:35382) 2024-04-25T00:16:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4cdf7fe640 1 -- 172.21.15.27:0/3272279928 learned_addr learned my addr 172.21.15.27:0/3272279928 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4cdeffd640 1 --2- 172.21.15.27:0/3272279928 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b41d580 0x55609b41d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4cde7fc640 1 --2- 172.21.15.27:0/3272279928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41dea0 0x55609b428030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4cdf7fe640 1 -- 172.21.15.27:0/3272279928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41dea0 msgr2=0x55609b428030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4cdf7fe640 1 --2- 172.21.15.27:0/3272279928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41dea0 0x55609b428030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4cdf7fe640 1 -- 172.21.15.27:0/3272279928 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b41d580 msgr2=0x55609b41d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4cdf7fe640 1 --2- 172.21.15.27:0/3272279928 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b41d580 0x55609b41d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4cdf7fe640 1 -- 172.21.15.27:0/3272279928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55609b34e400 con 0x55609b428570 2024-04-25T00:16:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4cdeffd640 1 --2- 172.21.15.27:0/3272279928 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b41d580 0x55609b41d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4cdf7fe640 1 --2- 172.21.15.27:0/3272279928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b428570 0x55609b42a960 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009e10 tx=0x7f4cd800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c1d050419fc1c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.561+0000 7f4cc57fa640 1 -- 172.21.15.27:0/3272279928 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8004d20 con 0x55609b428570 2024-04-25T00:16:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.562+0000 7f4cc57fa640 1 -- 172.21.15.27:0/3272279928 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cd8005500 con 0x55609b428570 2024-04-25T00:16:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.562+0000 7f4cc57fa640 1 -- 172.21.15.27:0/3272279928 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8011800 con 0x55609b428570 2024-04-25T00:16:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.562+0000 7f4ce83e9740 1 -- 172.21.15.27:0/3272279928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b428570 msgr2=0x55609b42a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.562+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/3272279928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b428570 0x55609b42a960 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009e10 tx=0x7f4cd800f730 comp rx=0 tx=0).stop 2024-04-25T00:16:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.562+0000 7f4ce83e9740 1 -- 172.21.15.27:0/3272279928 shutdown_connections 2024-04-25T00:16:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.562+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/3272279928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b428570 0x55609b42a960 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.562+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/3272279928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41dea0 0x55609b428030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.562+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/3272279928 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b41d580 0x55609b41d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.562+0000 7f4ce83e9740 1 -- 172.21.15.27:0/3272279928 >> 172.21.15.27:0/3272279928 conn(0x55609b4198a0 msgr2=0x55609b419c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.562+0000 7f4ce83e9740 1 -- 172.21.15.27:0/3272279928 shutdown_connections 2024-04-25T00:16:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.562+0000 7f4ce83e9740 1 -- 172.21.15.27:0/3272279928 wait complete. 2024-04-25T00:16:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4ce83e9740 1 -- start start 2024-04-25T00:16:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4ce83e9740 1 Processor -- start 2024-04-25T00:16:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4ce83e9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41d580 0x55609b427ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4ce83e9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b41dea0 0x55609b4251b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4ce83e9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b428570 0x55609b425710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4ce83e9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55609b42b360 con 0x55609b41dea0 2024-04-25T00:16:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4ce83e9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55609b33e050 con 0x55609b428570 2024-04-25T00:16:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4ce83e9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55609b42b4e0 con 0x55609b41d580 2024-04-25T00:16:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4ce83e9740 10 client.0 _ll_register_callbacks cb 0x55609b095be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4cde7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b41dea0 0x55609b4251b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4cde7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b41dea0 0x55609b4251b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35396/0 (socket says 172.21.15.27:35396) 2024-04-25T00:16:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4cde7fc640 1 -- 172.21.15.27:0/64025047 learned_addr learned my addr 172.21.15.27:0/64025047 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4cdeffd640 1 --2- 172.21.15.27:0/64025047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41d580 0x55609b427ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4cdf7fe640 1 --2- 172.21.15.27:0/64025047 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b428570 0x55609b425710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4cde7fc640 1 -- 172.21.15.27:0/64025047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41d580 msgr2=0x55609b427ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4cde7fc640 1 --2- 172.21.15.27:0/64025047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41d580 0x55609b427ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4cde7fc640 1 -- 172.21.15.27:0/64025047 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b428570 msgr2=0x55609b425710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4cde7fc640 1 --2- 172.21.15.27:0/64025047 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b428570 0x55609b425710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4cde7fc640 1 -- 172.21.15.27:0/64025047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55609b30fdc0 con 0x55609b41dea0 2024-04-25T00:16:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4cdeffd640 1 --2- 172.21.15.27:0/64025047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41d580 0x55609b427ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.563+0000 7f4cdf7fe640 1 --2- 172.21.15.27:0/64025047 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b428570 0x55609b425710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.564+0000 7f4cde7fc640 1 --2- 172.21.15.27:0/64025047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b41dea0 0x55609b4251b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4cd40bbfb0 tx=0x7f4cd40bbe70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f46f356a907ebd26 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.564+0000 7f4cc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:24.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.564+0000 7f4cc2ff5640 1 -- 172.21.15.27:0/64025047 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4049070 con 0x55609b41dea0 2024-04-25T00:16:24.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.564+0000 7f4cc2ff5640 1 -- 172.21.15.27:0/64025047 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f4cd40dc0d0 con 0x55609b41dea0 2024-04-25T00:16:24.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.564+0000 7f4ce83e9740 1 -- 172.21.15.27:0/64025047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cd8005500 con 0x55609b41dea0 2024-04-25T00:16:24.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.564+0000 7f4cc2ff5640 1 -- 172.21.15.27:0/64025047 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4cd40dc2d0 con 0x55609b41dea0 2024-04-25T00:16:24.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.564+0000 7f4ce83e9740 10 client.15924 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:24.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.564+0000 7f4ce83e9740 1 -- 172.21.15.27:0/64025047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55609b0963f0 con 0x55609b41dea0 2024-04-25T00:16:24.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.564+0000 7f4cc2ff5640 1 -- 172.21.15.27:0/64025047 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cd8005500 con 0x55609b41dea0 2024-04-25T00:16:24.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.564+0000 7f4cc2ff5640 1 -- 172.21.15.27:0/64025047 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4cd40bc510 con 0x55609b41dea0 2024-04-25T00:16:24.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.564+0000 7f4ce83e9740 10 client.15924 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:24.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.564+0000 7f4ce83e9740 10 client.15924 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:16:24.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.564+0000 7f4ce83e9740 20 client.15924 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:24.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.564+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/64025047 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55609b438560 0x55609b43a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.564+0000 7f4ce83e9740 4 client.15924 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:16:24.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.564+0000 7f4ce83e9740 1 -- 172.21.15.27:0/64025047 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55609b38fbb0 con 0x55609b438560 2024-04-25T00:16:24.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.565+0000 7f4cdeffd640 1 --2- 172.21.15.27:0/64025047 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55609b438560 0x55609b43a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.565+0000 7f4cdeffd640 1 --2- 172.21.15.27:0/64025047 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55609b438560 0x55609b43a9e0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c308b8fcd7320708 server_cookie=479920488dc9be7f in_seq=0 out_seq=0 2024-04-25T00:16:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.565+0000 7f4cc2ff5640 10 client.15924 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:16:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.566+0000 7f4cc2ff5640 1 -- 172.21.15.27:0/64025047 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55609b38fbb0 con 0x55609b438560 2024-04-25T00:16:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.566+0000 7f4cc2ff5640 10 client.15924 handle_command_reply: tid=0 2024-04-25T00:16:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.566+0000 7f4ce83e9740 1 client.15924 shutdown 2024-04-25T00:16:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.566+0000 7f4ce83e9740 1 -- 172.21.15.27:0/64025047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b41dea0 msgr2=0x55609b4251b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.566+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/64025047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b41dea0 0x55609b4251b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4cd40bbfb0 tx=0x7f4cd40bbe70 comp rx=0 tx=0).stop 2024-04-25T00:16:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.566+0000 7f4ce83e9740 1 -- 172.21.15.27:0/64025047 shutdown_connections 2024-04-25T00:16:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.566+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/64025047 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b428570 0x55609b425710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.566+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/64025047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b41dea0 0x55609b4251b0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.566+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/64025047 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55609b438560 0x55609b43a9e0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.566+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/64025047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41d580 0x55609b427ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.566+0000 7f4cc2ff5640 0 client.15924 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:16:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.566+0000 7f4ce83e9740 1 -- 172.21.15.27:0/64025047 >> 172.21.15.27:0/64025047 conn(0x55609b4198a0 msgr2=0x55609b41f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.566+0000 7f4ce83e9740 1 -- 172.21.15.27:0/64025047 shutdown_connections 2024-04-25T00:16:24.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.566+0000 7f4ce83e9740 1 -- 172.21.15.27:0/64025047 wait complete. 2024-04-25T00:16:24.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.566+0000 7f4ce83e9740 20 client.15924 trim_cache size 0 max 16384 2024-04-25T00:16:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.580+0000 7f4ce83e9740 1 Processor -- start 2024-04-25T00:16:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.580+0000 7f4ce83e9740 1 -- start start 2024-04-25T00:16:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.580+0000 7f4ce83e9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b41d580 0x55609b434220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.580+0000 7f4ce83e9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41dea0 0x55609b4402b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.580+0000 7f4ce83e9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b428570 0x55609b4407f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.580+0000 7f4ce83e9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce01b0d40 con 0x55609b428570 2024-04-25T00:16:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.580+0000 7f4ce83e9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce01b2880 con 0x55609b41d580 2024-04-25T00:16:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.580+0000 7f4ce83e9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce01b1410 con 0x55609b41dea0 2024-04-25T00:16:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.581+0000 7f4cdf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b428570 0x55609b4407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.581+0000 7f4cdf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b428570 0x55609b4407f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35412/0 (socket says 172.21.15.27:35412) 2024-04-25T00:16:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.581+0000 7f4cdf7fe640 1 -- 172.21.15.27:0/3626053279 learned_addr learned my addr 172.21.15.27:0/3626053279 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.581+0000 7f4cdeffd640 1 --2- 172.21.15.27:0/3626053279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b41d580 0x55609b434220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.581+0000 7f4cde7fc640 1 --2- 172.21.15.27:0/3626053279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41dea0 0x55609b4402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.581+0000 7f4cdf7fe640 1 -- 172.21.15.27:0/3626053279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41dea0 msgr2=0x55609b4402b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.581+0000 7f4cdf7fe640 1 --2- 172.21.15.27:0/3626053279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41dea0 0x55609b4402b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.581+0000 7f4cdf7fe640 1 -- 172.21.15.27:0/3626053279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b41d580 msgr2=0x55609b434220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.581+0000 7f4cdf7fe640 1 --2- 172.21.15.27:0/3626053279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b41d580 0x55609b434220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.581+0000 7f4cdf7fe640 1 -- 172.21.15.27:0/3626053279 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55609b424160 con 0x55609b428570 2024-04-25T00:16:24.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.581+0000 7f4cde7fc640 1 --2- 172.21.15.27:0/3626053279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41dea0 0x55609b4402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:24.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.581+0000 7f4cdf7fe640 1 --2- 172.21.15.27:0/3626053279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b428570 0x55609b4407f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009d30 tx=0x7f4cd800ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b156d484b22ed4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.581+0000 7f4cc4ff9640 1 -- 172.21.15.27:0/3626053279 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8011440 con 0x55609b428570 2024-04-25T00:16:24.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.581+0000 7f4cc4ff9640 1 -- 172.21.15.27:0/3626053279 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cd8011ac0 con 0x55609b428570 2024-04-25T00:16:24.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.581+0000 7f4cc4ff9640 1 -- 172.21.15.27:0/3626053279 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8019a90 con 0x55609b428570 2024-04-25T00:16:24.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.582+0000 7f4ce83e9740 1 -- 172.21.15.27:0/3626053279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b428570 msgr2=0x55609b4407f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.582+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/3626053279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b428570 0x55609b4407f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009d30 tx=0x7f4cd800ffd0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.582+0000 7f4ce83e9740 1 -- 172.21.15.27:0/3626053279 shutdown_connections 2024-04-25T00:16:24.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.582+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/3626053279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b428570 0x55609b4407f0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.582+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/3626053279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41dea0 0x55609b4402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.582+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/3626053279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b41d580 0x55609b434220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.582+0000 7f4ce83e9740 1 -- 172.21.15.27:0/3626053279 >> 172.21.15.27:0/3626053279 conn(0x55609b4198a0 msgr2=0x55609b418b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.582+0000 7f4ce83e9740 1 -- 172.21.15.27:0/3626053279 shutdown_connections 2024-04-25T00:16:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.582+0000 7f4ce83e9740 1 -- 172.21.15.27:0/3626053279 wait complete. 2024-04-25T00:16:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.582+0000 7f4ce83e9740 1 -- start start 2024-04-25T00:16:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4ce83e9740 1 Processor -- start 2024-04-25T00:16:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4ce83e9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b41d580 0x55609b42e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4ce83e9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41dea0 0x55609b42edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4ce83e9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b428570 0x55609b49db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4ce83e9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce01b21b0 con 0x55609b41d580 2024-04-25T00:16:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4ce83e9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce01b1ae0 con 0x55609b428570 2024-04-25T00:16:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4ce83e9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce01aed00 con 0x55609b41dea0 2024-04-25T00:16:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4ce83e9740 10 client.0 _ll_register_callbacks cb 0x7f4cd4005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4cdeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b41d580 0x55609b42e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4cdeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b41d580 0x55609b42e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35416/0 (socket says 172.21.15.27:35416) 2024-04-25T00:16:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4cdeffd640 1 -- 172.21.15.27:0/279836743 learned_addr learned my addr 172.21.15.27:0/279836743 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4cde7fc640 1 --2- 172.21.15.27:0/279836743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41dea0 0x55609b42edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4cdf7fe640 1 --2- 172.21.15.27:0/279836743 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b428570 0x55609b49db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4cdeffd640 1 -- 172.21.15.27:0/279836743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41dea0 msgr2=0x55609b42edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4cdeffd640 1 --2- 172.21.15.27:0/279836743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41dea0 0x55609b42edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4cdeffd640 1 -- 172.21.15.27:0/279836743 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b428570 msgr2=0x55609b49db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4cdeffd640 1 --2- 172.21.15.27:0/279836743 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b428570 0x55609b49db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4cdeffd640 1 -- 172.21.15.27:0/279836743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55609b1607d0 con 0x55609b41d580 2024-04-25T00:16:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4cdf7fe640 1 --2- 172.21.15.27:0/279836743 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b428570 0x55609b49db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4cde7fc640 1 --2- 172.21.15.27:0/279836743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41dea0 0x55609b42edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4cdeffd640 1 --2- 172.21.15.27:0/279836743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b41d580 0x55609b42e890 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x55609b43b220 tx=0x7f4cd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc63d19a0920398a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4cc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.583+0000 7f4cc2ff5640 1 -- 172.21.15.27:0/279836743 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0004220 con 0x55609b41d580 2024-04-25T00:16:24.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.584+0000 7f4cc2ff5640 1 -- 172.21.15.27:0/279836743 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f4cd00043c0 con 0x55609b41d580 2024-04-25T00:16:24.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.584+0000 7f4ce83e9740 1 -- 172.21.15.27:0/279836743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cd8011ac0 con 0x55609b41d580 2024-04-25T00:16:24.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.584+0000 7f4cc2ff5640 1 -- 172.21.15.27:0/279836743 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4cd0012760 con 0x55609b41d580 2024-04-25T00:16:24.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.584+0000 7f4ce83e9740 10 client.15930 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:24.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.584+0000 7f4ce83e9740 1 -- 172.21.15.27:0/279836743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55609b49f1a0 con 0x55609b41d580 2024-04-25T00:16:24.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.584+0000 7f4cc2ff5640 1 -- 172.21.15.27:0/279836743 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cd8011ac0 con 0x55609b41d580 2024-04-25T00:16:24.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.584+0000 7f4cc2ff5640 1 -- 172.21.15.27:0/279836743 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4cd0031070 con 0x55609b41d580 2024-04-25T00:16:24.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.584+0000 7f4ce83e9740 10 client.15930 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:24.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.584+0000 7f4ce83e9740 10 client.15930 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:16:24.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.584+0000 7f4ce83e9740 20 client.15930 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:24.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.584+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/279836743 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55609b49f950 0x55609b4a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.584+0000 7f4ce83e9740 4 client.15930 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:16:24.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.584+0000 7f4ce83e9740 1 -- 172.21.15.27:0/279836743 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55609b389f00 con 0x55609b49f950 2024-04-25T00:16:24.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.584+0000 7f4cde7fc640 1 --2- 172.21.15.27:0/279836743 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55609b49f950 0x55609b4a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.585+0000 7f4cde7fc640 1 --2- 172.21.15.27:0/279836743 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55609b49f950 0x55609b4a1db0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=95edb512fa6175d3 server_cookie=dd982f94ee5d87b in_seq=0 out_seq=0 2024-04-25T00:16:24.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.585+0000 7f4cc2ff5640 10 client.15930 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:16:24.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.585+0000 7f4cc2ff5640 1 -- 172.21.15.27:0/279836743 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55609b389f00 con 0x55609b49f950 2024-04-25T00:16:24.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.585+0000 7f4cc2ff5640 10 client.15930 handle_command_reply: tid=0 2024-04-25T00:16:24.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.585+0000 7f4ce83e9740 1 client.15930 shutdown 2024-04-25T00:16:24.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.585+0000 7f4ce83e9740 1 -- 172.21.15.27:0/279836743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b41d580 msgr2=0x55609b42e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.585+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/279836743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b41d580 0x55609b42e890 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x55609b43b220 tx=0x7f4cd00046c0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.585+0000 7f4ce83e9740 1 -- 172.21.15.27:0/279836743 shutdown_connections 2024-04-25T00:16:24.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.585+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/279836743 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55609b428570 0x55609b49db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.585+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/279836743 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55609b49f950 0x55609b4a1db0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.585+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/279836743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55609b41dea0 0x55609b42edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.585+0000 7f4cc2ff5640 0 client.15930 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:16:24.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.585+0000 7f4ce83e9740 1 --2- 172.21.15.27:0/279836743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55609b41d580 0x55609b42e890 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.585+0000 7f4ce83e9740 1 -- 172.21.15.27:0/279836743 >> 172.21.15.27:0/279836743 conn(0x55609b4198a0 msgr2=0x55609b433210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.585+0000 7f4ce83e9740 1 -- 172.21.15.27:0/279836743 shutdown_connections 2024-04-25T00:16:24.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.585+0000 7f4ce83e9740 1 -- 172.21.15.27:0/279836743 wait complete. 2024-04-25T00:16:24.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.585+0000 7f4ce83e9740 20 client.15930 trim_cache size 0 max 16384 2024-04-25T00:16:24.592 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:16:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.586+0000 7f4ce55f9640 1 -- 172.21.15.27:0/241386756 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4cc80a2eb0 msgr2=0x7f4cc80a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.586+0000 7f4ce55f9640 1 --2- 172.21.15.27:0/241386756 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4cc80a2eb0 0x7f4cc80a5370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009b60 tx=0x7f4cd0005f90 comp rx=0 tx=0).stop 2024-04-25T00:16:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.586+0000 7f4ce55f9640 1 -- 172.21.15.27:0/241386756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce0100e60 msgr2=0x7f4ce019a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.586+0000 7f4ce55f9640 1 --2- 172.21.15.27:0/241386756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce0100e60 0x7f4ce019a740 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4006010 tx=0x7f4cd4004270 comp rx=0 tx=0).stop 2024-04-25T00:16:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.586+0000 7f4ce55f9640 1 -- 172.21.15.27:0/241386756 shutdown_connections 2024-04-25T00:16:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.586+0000 7f4ce55f9640 1 --2- 172.21.15.27:0/241386756 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4cc80a2eb0 0x7f4cc80a5370 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.586+0000 7f4ce55f9640 1 --2- 172.21.15.27:0/241386756 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ce010e0f0 0x7f4ce019eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.586+0000 7f4ce55f9640 1 --2- 172.21.15.27:0/241386756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce0100e60 0x7f4ce019a740 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.586+0000 7f4ce55f9640 1 --2- 172.21.15.27:0/241386756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ce0100540 0x7f4ce019a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.586+0000 7f4ce55f9640 1 -- 172.21.15.27:0/241386756 >> 172.21.15.27:0/241386756 conn(0x7f4ce0077740 msgr2=0x7f4ce010f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.586+0000 7f4ce55f9640 1 -- 172.21.15.27:0/241386756 shutdown_connections 2024-04-25T00:16:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.586+0000 7f4ce55f9640 1 -- 172.21.15.27:0/241386756 wait complete. 2024-04-25T00:16:24.597 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T00:16:24.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.703+0000 7f61a7c79640 1 -- 172.21.15.27:0/2187787388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a01032e0 msgr2=0x7f61a010b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.703+0000 7f61a7c79640 1 --2- 172.21.15.27:0/2187787388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a01032e0 0x7f61a010b7c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f619c009900 tx=0x7f619c02a510 comp rx=0 tx=0).stop 2024-04-25T00:16:24.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.703+0000 7f61a7c79640 1 -- 172.21.15.27:0/2187787388 shutdown_connections 2024-04-25T00:16:24.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.703+0000 7f61a7c79640 1 --2- 172.21.15.27:0/2187787388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61a010bd00 0x7f61a010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.703+0000 7f61a7c79640 1 --2- 172.21.15.27:0/2187787388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a01032e0 0x7f61a010b7c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.703+0000 7f61a7c79640 1 --2- 172.21.15.27:0/2187787388 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61a01029c0 0x7f61a0102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.703+0000 7f61a7c79640 1 -- 172.21.15.27:0/2187787388 >> 172.21.15.27:0/2187787388 conn(0x7f61a0077740 msgr2=0x7f61a0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.703+0000 7f61a7c79640 1 -- 172.21.15.27:0/2187787388 shutdown_connections 2024-04-25T00:16:24.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.703+0000 7f61a7c79640 1 -- 172.21.15.27:0/2187787388 wait complete. 2024-04-25T00:16:24.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a7c79640 1 Processor -- start 2024-04-25T00:16:24.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a7c79640 1 -- start start 2024-04-25T00:16:24.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a7c79640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61a01029c0 0x7f61a0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a7c79640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a01032e0 0x7f61a0198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a7c79640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61a010bd00 0x7f61a019c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a7c79640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f61a0111b00 con 0x7f61a01032e0 2024-04-25T00:16:24.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a7c79640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f61a0111980 con 0x7f61a010bd00 2024-04-25T00:16:24.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a7c79640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f61a0111c80 con 0x7f61a01029c0 2024-04-25T00:16:24.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a5214640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a01032e0 0x7f61a0198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a5214640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a01032e0 0x7f61a0198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35444/0 (socket says 172.21.15.27:35444) 2024-04-25T00:16:24.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a5214640 1 -- 172.21.15.27:0/906824811 learned_addr learned my addr 172.21.15.27:0/906824811 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a6216640 1 --2- 172.21.15.27:0/906824811 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61a010bd00 0x7f61a019c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a5a15640 1 --2- 172.21.15.27:0/906824811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61a01029c0 0x7f61a0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a5214640 1 -- 172.21.15.27:0/906824811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61a01029c0 msgr2=0x7f61a0197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a5214640 1 --2- 172.21.15.27:0/906824811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61a01029c0 0x7f61a0197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a5214640 1 -- 172.21.15.27:0/906824811 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61a010bd00 msgr2=0x7f61a019c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a5214640 1 --2- 172.21.15.27:0/906824811 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61a010bd00 0x7f61a019c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a5214640 1 -- 172.21.15.27:0/906824811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a019cf60 con 0x7f61a01032e0 2024-04-25T00:16:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.704+0000 7f61a5a15640 1 --2- 172.21.15.27:0/906824811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61a01029c0 0x7f61a0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.705+0000 7f61a5214640 1 --2- 172.21.15.27:0/906824811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a01032e0 0x7f61a0198500 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f619c006010 tx=0x7f619c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.705+0000 7f6196ffd640 1 -- 172.21.15.27:0/906824811 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c03f030 con 0x7f61a01032e0 2024-04-25T00:16:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.705+0000 7f6196ffd640 1 -- 172.21.15.27:0/906824811 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f619c003ce0 con 0x7f61a01032e0 2024-04-25T00:16:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.705+0000 7f61a7c79640 1 -- 172.21.15.27:0/906824811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61a019d1f0 con 0x7f61a01032e0 2024-04-25T00:16:24.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.705+0000 7f6196ffd640 1 -- 172.21.15.27:0/906824811 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c035070 con 0x7f61a01032e0 2024-04-25T00:16:24.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.705+0000 7f61a7c79640 1 -- 172.21.15.27:0/906824811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61a01a4ae0 con 0x7f61a01032e0 2024-04-25T00:16:24.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.707+0000 7f6196ffd640 1 -- 172.21.15.27:0/906824811 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f619c042030 con 0x7f61a01032e0 2024-04-25T00:16:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.707+0000 7f61aa983740 1 Processor -- start 2024-04-25T00:16:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.708+0000 7f61aa983740 1 -- start start 2024-04-25T00:16:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.708+0000 7f61aa983740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759580 0x563296759960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.708+0000 7f61aa983740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759ea0 0x563296764030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.708+0000 7f61aa983740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296764570 0x563296766960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.708+0000 7f61aa983740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5632966be670 con 0x563296759580 2024-04-25T00:16:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.708+0000 7f61aa983740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563296666970 con 0x563296759ea0 2024-04-25T00:16:24.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.708+0000 7f61aa983740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5632966ba630 con 0x563296764570 2024-04-25T00:16:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.708+0000 7f61a6216640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759580 0x563296759960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.708+0000 7f61a6216640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759580 0x563296759960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35452/0 (socket says 172.21.15.27:35452) 2024-04-25T00:16:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.708+0000 7f61a6216640 1 -- 172.21.15.27:0/1462479154 learned_addr learned my addr 172.21.15.27:0/1462479154 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.708+0000 7f6196ffd640 1 --2- 172.21.15.27:0/906824811 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f618c0a2eb0 0x7f618c0a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.708+0000 7f61a5214640 1 --2- 172.21.15.27:0/1462479154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296764570 0x563296766960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.708+0000 7f61a5a15640 1 --2- 172.21.15.27:0/1462479154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759ea0 0x563296764030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.708+0000 7f6196ffd640 1 -- 172.21.15.27:0/906824811 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f619c0effb0 con 0x7f61a01032e0 2024-04-25T00:16:24.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.708+0000 7f61a6216640 1 --2- 172.21.15.27:0/906824811 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f618c0a2eb0 0x7f618c0a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.709+0000 7f61a6216640 1 -- 172.21.15.27:0/1462479154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296764570 msgr2=0x563296766960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.709+0000 7f61a6216640 1 --2- 172.21.15.27:0/1462479154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296764570 0x563296766960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.709+0000 7f61a6216640 1 -- 172.21.15.27:0/1462479154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759ea0 msgr2=0x563296764030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.709+0000 7f61a6216640 1 --2- 172.21.15.27:0/1462479154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759ea0 0x563296764030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.709+0000 7f61a6216640 1 -- 172.21.15.27:0/1462479154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56329668a400 con 0x563296759580 2024-04-25T00:16:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.709+0000 7f61a6216640 1 --2- 172.21.15.27:0/1462479154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759580 0x563296759960 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f6198014a60 tx=0x7f6198014f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84fcfb73e5cf1dd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.709+0000 7f618bfff640 1 -- 172.21.15.27:0/1462479154 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619801ec50 con 0x563296759580 2024-04-25T00:16:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.709+0000 7f618bfff640 1 -- 172.21.15.27:0/1462479154 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6198023070 con 0x563296759580 2024-04-25T00:16:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.709+0000 7f618bfff640 1 -- 172.21.15.27:0/1462479154 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6198005650 con 0x563296759580 2024-04-25T00:16:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.709+0000 7f61a6216640 1 --2- 172.21.15.27:0/906824811 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f618c0a2eb0 0x7f618c0a5370 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6198022000 tx=0x7f619801e000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.710+0000 7f61aa983740 1 -- 172.21.15.27:0/1462479154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759580 msgr2=0x563296759960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.710+0000 7f61aa983740 1 --2- 172.21.15.27:0/1462479154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759580 0x563296759960 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f6198014a60 tx=0x7f6198014f20 comp rx=0 tx=0).stop 2024-04-25T00:16:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.710+0000 7f61aa983740 1 -- 172.21.15.27:0/1462479154 shutdown_connections 2024-04-25T00:16:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.710+0000 7f61aa983740 1 --2- 172.21.15.27:0/1462479154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296764570 0x563296766960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.710+0000 7f61aa983740 1 --2- 172.21.15.27:0/1462479154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759ea0 0x563296764030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.710+0000 7f61aa983740 1 --2- 172.21.15.27:0/1462479154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759580 0x563296759960 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.710+0000 7f61aa983740 1 -- 172.21.15.27:0/1462479154 >> 172.21.15.27:0/1462479154 conn(0x5632967558a0 msgr2=0x563296755c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.710+0000 7f61aa983740 1 -- 172.21.15.27:0/1462479154 shutdown_connections 2024-04-25T00:16:24.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.710+0000 7f61aa983740 1 -- 172.21.15.27:0/1462479154 wait complete. 2024-04-25T00:16:24.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.710+0000 7f61aa983740 1 -- start start 2024-04-25T00:16:24.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.710+0000 7f61aa983740 1 Processor -- start 2024-04-25T00:16:24.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61aa983740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759580 0x563296763ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61aa983740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759ea0 0x5632967611c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61aa983740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296764570 0x563296761720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61aa983740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563296767360 con 0x563296759ea0 2024-04-25T00:16:24.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61aa983740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56329667a050 con 0x563296759580 2024-04-25T00:16:24.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61aa983740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5632967674e0 con 0x563296764570 2024-04-25T00:16:24.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61aa983740 10 client.0 _ll_register_callbacks cb 0x5632963d1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:24.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61a5214640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759ea0 0x5632967611c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61a5214640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759ea0 0x5632967611c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35464/0 (socket says 172.21.15.27:35464) 2024-04-25T00:16:24.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61a5214640 1 -- 172.21.15.27:0/1681941643 learned_addr learned my addr 172.21.15.27:0/1681941643 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61a5a15640 1 --2- 172.21.15.27:0/1681941643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759580 0x563296763ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61a6216640 1 --2- 172.21.15.27:0/1681941643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296764570 0x563296761720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61a5214640 1 -- 172.21.15.27:0/1681941643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296764570 msgr2=0x563296761720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61a5214640 1 --2- 172.21.15.27:0/1681941643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296764570 0x563296761720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61a5214640 1 -- 172.21.15.27:0/1681941643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759580 msgr2=0x563296763ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61a5214640 1 --2- 172.21.15.27:0/1681941643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759580 0x563296763ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61a5214640 1 -- 172.21.15.27:0/1681941643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56329664bdc0 con 0x563296759ea0 2024-04-25T00:16:24.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61a6216640 1 --2- 172.21.15.27:0/1681941643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296764570 0x563296761720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:24.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61a5214640 1 --2- 172.21.15.27:0/1681941643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759ea0 0x5632967611c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f619c0bbf70 tx=0x7f619c08afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6170a164cd26c0be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.711+0000 7f61897fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:24.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.712+0000 7f61897fa640 1 -- 172.21.15.27:0/1681941643 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c049040 con 0x563296759ea0 2024-04-25T00:16:24.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.712+0000 7f61897fa640 1 -- 172.21.15.27:0/1681941643 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f619c031dd0 con 0x563296759ea0 2024-04-25T00:16:24.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.712+0000 7f61aa983740 1 -- 172.21.15.27:0/1681941643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6198023070 con 0x563296759ea0 2024-04-25T00:16:24.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.712+0000 7f61897fa640 1 -- 172.21.15.27:0/1681941643 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f619c0bb940 con 0x563296759ea0 2024-04-25T00:16:24.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.712+0000 7f61aa983740 10 client.15942 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:24.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.712+0000 7f61aa983740 1 -- 172.21.15.27:0/1681941643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5632963d23f0 con 0x563296759ea0 2024-04-25T00:16:24.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.712+0000 7f61897fa640 1 -- 172.21.15.27:0/1681941643 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6198023070 con 0x563296759ea0 2024-04-25T00:16:24.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.712+0000 7f61897fa640 1 -- 172.21.15.27:0/1681941643 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f619c0da070 con 0x563296759ea0 2024-04-25T00:16:24.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.712+0000 7f61aa983740 10 client.15942 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:24.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.712+0000 7f61aa983740 10 client.15942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:16:24.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.712+0000 7f61aa983740 20 client.15942 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:24.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.712+0000 7f61aa983740 1 --2- 172.21.15.27:0/1681941643 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5632967744f0 0x563296776900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.712+0000 7f61aa983740 4 client.15942 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:16:24.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.712+0000 7f61aa983740 1 -- 172.21.15.27:0/1681941643 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632966cbbb0 con 0x5632967744f0 2024-04-25T00:16:24.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.713+0000 7f61a5a15640 1 --2- 172.21.15.27:0/1681941643 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5632967744f0 0x563296776900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.713+0000 7f61a5a15640 1 --2- 172.21.15.27:0/1681941643 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5632967744f0 0x563296776900 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cb4531df2aabc8b server_cookie=d2237b8ee67f4c60 in_seq=0 out_seq=0 2024-04-25T00:16:24.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.713+0000 7f61897fa640 10 client.15942 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:24.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.714+0000 7f61897fa640 1 -- 172.21.15.27:0/1681941643 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5632966cbbb0 con 0x5632967744f0 2024-04-25T00:16:24.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.714+0000 7f61897fa640 10 client.15942 handle_command_reply: tid=0 2024-04-25T00:16:24.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.714+0000 7f61aa983740 1 client.15942 shutdown 2024-04-25T00:16:24.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.714+0000 7f61aa983740 1 -- 172.21.15.27:0/1681941643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759ea0 msgr2=0x5632967611c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.714+0000 7f61aa983740 1 --2- 172.21.15.27:0/1681941643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759ea0 0x5632967611c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f619c0bbf70 tx=0x7f619c08afd0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.714+0000 7f61aa983740 1 -- 172.21.15.27:0/1681941643 shutdown_connections 2024-04-25T00:16:24.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.714+0000 7f61aa983740 1 --2- 172.21.15.27:0/1681941643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296764570 0x563296761720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.714+0000 7f61aa983740 1 --2- 172.21.15.27:0/1681941643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759ea0 0x5632967611c0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.714+0000 7f61aa983740 1 --2- 172.21.15.27:0/1681941643 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5632967744f0 0x563296776900 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.714+0000 7f61aa983740 1 --2- 172.21.15.27:0/1681941643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759580 0x563296763ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.714+0000 7f61897fa640 0 client.15942 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:24.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.714+0000 7f61aa983740 1 -- 172.21.15.27:0/1681941643 >> 172.21.15.27:0/1681941643 conn(0x5632967558a0 msgr2=0x563296754b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.714+0000 7f61aa983740 1 -- 172.21.15.27:0/1681941643 shutdown_connections 2024-04-25T00:16:24.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.714+0000 7f61aa983740 1 -- 172.21.15.27:0/1681941643 wait complete. 2024-04-25T00:16:24.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.714+0000 7f61aa983740 20 client.15942 trim_cache size 0 max 16384 2024-04-25T00:16:24.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61aa983740 1 Processor -- start 2024-04-25T00:16:24.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61aa983740 1 -- start start 2024-04-25T00:16:24.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61aa983740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759580 0x563296770250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61aa983740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759ea0 0x563296763a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61aa983740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296764570 0x56329677c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61aa983740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f61a01aeae0 con 0x563296759ea0 2024-04-25T00:16:24.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61aa983740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f61a01b0620 con 0x563296759580 2024-04-25T00:16:24.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61aa983740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f61a01af1b0 con 0x563296764570 2024-04-25T00:16:24.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61a5214640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759ea0 0x563296763a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61a5214640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759ea0 0x563296763a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35476/0 (socket says 172.21.15.27:35476) 2024-04-25T00:16:24.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61a5214640 1 -- 172.21.15.27:0/2795857490 learned_addr learned my addr 172.21.15.27:0/2795857490 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61a6216640 1 --2- 172.21.15.27:0/2795857490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296764570 0x56329677c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61a5a15640 1 --2- 172.21.15.27:0/2795857490 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759580 0x563296770250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61a5214640 1 -- 172.21.15.27:0/2795857490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296764570 msgr2=0x56329677c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61a5214640 1 --2- 172.21.15.27:0/2795857490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296764570 0x56329677c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61a5214640 1 -- 172.21.15.27:0/2795857490 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759580 msgr2=0x563296770250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61a5214640 1 --2- 172.21.15.27:0/2795857490 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759580 0x563296770250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61a5214640 1 -- 172.21.15.27:0/2795857490 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563296759cd0 con 0x563296759ea0 2024-04-25T00:16:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61a5a15640 1 --2- 172.21.15.27:0/2795857490 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759580 0x563296770250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.728+0000 7f61a6216640 1 --2- 172.21.15.27:0/2795857490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296764570 0x56329677c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:24.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.729+0000 7f61a5214640 1 --2- 172.21.15.27:0/2795857490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759ea0 0x563296763a00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f619c08f0e0 tx=0x7f619c0c47c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b5cabefb4cbb6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.729+0000 7f618b7fe640 1 -- 172.21.15.27:0/2795857490 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c049070 con 0x563296759ea0 2024-04-25T00:16:24.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.729+0000 7f618b7fe640 1 -- 172.21.15.27:0/2795857490 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f619c0c4a40 con 0x563296759ea0 2024-04-25T00:16:24.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.729+0000 7f618b7fe640 1 -- 172.21.15.27:0/2795857490 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c0044e0 con 0x563296759ea0 2024-04-25T00:16:24.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 1 -- 172.21.15.27:0/2795857490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759ea0 msgr2=0x563296763a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 1 --2- 172.21.15.27:0/2795857490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759ea0 0x563296763a00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f619c08f0e0 tx=0x7f619c0c47c0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 1 -- 172.21.15.27:0/2795857490 shutdown_connections 2024-04-25T00:16:24.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 1 --2- 172.21.15.27:0/2795857490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296764570 0x56329677c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 1 --2- 172.21.15.27:0/2795857490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296759ea0 0x563296763a00 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 1 --2- 172.21.15.27:0/2795857490 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759580 0x563296770250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 1 -- 172.21.15.27:0/2795857490 >> 172.21.15.27:0/2795857490 conn(0x5632967558a0 msgr2=0x56329675b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 1 -- 172.21.15.27:0/2795857490 shutdown_connections 2024-04-25T00:16:24.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 1 -- 172.21.15.27:0/2795857490 wait complete. 2024-04-25T00:16:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 1 -- start start 2024-04-25T00:16:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 1 Processor -- start 2024-04-25T00:16:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759580 0x5632967689d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296759ea0 0x563296768f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296764570 0x5632967d9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f61a01aff50 con 0x563296764570 2024-04-25T00:16:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f61a01af880 con 0x563296759580 2024-04-25T00:16:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f61a01acaa0 con 0x563296759ea0 2024-04-25T00:16:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.730+0000 7f61aa983740 10 client.0 _ll_register_callbacks cb 0x7f619c0b82e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61a6216640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296764570 0x5632967d9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61a6216640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296764570 0x5632967d9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35490/0 (socket says 172.21.15.27:35490) 2024-04-25T00:16:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61a6216640 1 -- 172.21.15.27:0/3762852382 learned_addr learned my addr 172.21.15.27:0/3762852382 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61a5a15640 1 --2- 172.21.15.27:0/3762852382 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759580 0x5632967689d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61a5214640 1 --2- 172.21.15.27:0/3762852382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296759ea0 0x563296768f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61a6216640 1 -- 172.21.15.27:0/3762852382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296759ea0 msgr2=0x563296768f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61a6216640 1 --2- 172.21.15.27:0/3762852382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296759ea0 0x563296768f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61a6216640 1 -- 172.21.15.27:0/3762852382 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759580 msgr2=0x5632967689d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61a6216640 1 --2- 172.21.15.27:0/3762852382 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759580 0x5632967689d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61a6216640 1 -- 172.21.15.27:0/3762852382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56329649c7d0 con 0x563296764570 2024-04-25T00:16:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61a6216640 1 --2- 172.21.15.27:0/3762852382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296764570 0x5632967d9a50 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f6198014a60 tx=0x7f6198012960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d939f01170a1aed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61897fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61897fa640 1 -- 172.21.15.27:0/3762852382 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6198037060 con 0x563296764570 2024-04-25T00:16:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61897fa640 1 -- 172.21.15.27:0/3762852382 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f619801ec50 con 0x563296764570 2024-04-25T00:16:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61aa983740 1 -- 172.21.15.27:0/3762852382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f619c0c4a40 con 0x563296764570 2024-04-25T00:16:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61897fa640 1 -- 172.21.15.27:0/3762852382 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f619803e050 con 0x563296764570 2024-04-25T00:16:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61aa983740 10 client.15948 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.731+0000 7f61aa983740 1 -- 172.21.15.27:0/3762852382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5632967db190 con 0x563296764570 2024-04-25T00:16:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.732+0000 7f61897fa640 1 -- 172.21.15.27:0/3762852382 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f619c0c4a40 con 0x563296764570 2024-04-25T00:16:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.732+0000 7f61897fa640 1 -- 172.21.15.27:0/3762852382 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6198031070 con 0x563296764570 2024-04-25T00:16:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.732+0000 7f61aa983740 10 client.15948 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.732+0000 7f61aa983740 10 client.15948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:16:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.732+0000 7f61aa983740 20 client.15948 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.732+0000 7f61aa983740 1 --2- 172.21.15.27:0/3762852382 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5632967db6b0 0x5632967ddad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.732+0000 7f61aa983740 4 client.15948 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:16:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.732+0000 7f61aa983740 1 -- 172.21.15.27:0/3762852382 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5632966c5f00 con 0x5632967db6b0 2024-04-25T00:16:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.732+0000 7f61a5a15640 1 --2- 172.21.15.27:0/3762852382 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5632967db6b0 0x5632967ddad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.733+0000 7f61897fa640 10 client.15948 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.733+0000 7f61a5a15640 1 --2- 172.21.15.27:0/3762852382 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5632967db6b0 0x5632967ddad0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef531629b25f6c1e server_cookie=2caf57a1a487aa51 in_seq=0 out_seq=0 2024-04-25T00:16:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.733+0000 7f61897fa640 1 -- 172.21.15.27:0/3762852382 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5632966c5f00 con 0x5632967db6b0 2024-04-25T00:16:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.733+0000 7f61897fa640 10 client.15948 handle_command_reply: tid=0 2024-04-25T00:16:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.733+0000 7f61aa983740 1 client.15948 shutdown 2024-04-25T00:16:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.733+0000 7f61aa983740 1 -- 172.21.15.27:0/3762852382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296764570 msgr2=0x5632967d9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.733+0000 7f61aa983740 1 --2- 172.21.15.27:0/3762852382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296764570 0x5632967d9a50 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f6198014a60 tx=0x7f6198012960 comp rx=0 tx=0).stop 2024-04-25T00:16:24.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.733+0000 7f61aa983740 1 -- 172.21.15.27:0/3762852382 shutdown_connections 2024-04-25T00:16:24.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.733+0000 7f61aa983740 1 --2- 172.21.15.27:0/3762852382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563296764570 0x5632967d9a50 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.733+0000 7f61aa983740 1 --2- 172.21.15.27:0/3762852382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563296759ea0 0x563296768f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.733+0000 7f61aa983740 1 --2- 172.21.15.27:0/3762852382 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5632967db6b0 0x5632967ddad0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.733+0000 7f61aa983740 1 --2- 172.21.15.27:0/3762852382 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563296759580 0x5632967689d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.733+0000 7f61897fa640 0 client.15948 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:24.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.733+0000 7f61aa983740 1 -- 172.21.15.27:0/3762852382 >> 172.21.15.27:0/3762852382 conn(0x5632967558a0 msgr2=0x56329675bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.733+0000 7f61aa983740 1 -- 172.21.15.27:0/3762852382 shutdown_connections 2024-04-25T00:16:24.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.733+0000 7f61aa983740 1 -- 172.21.15.27:0/3762852382 wait complete. 2024-04-25T00:16:24.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.733+0000 7f61aa983740 20 client.15948 trim_cache size 0 max 16384 2024-04-25T00:16:24.743 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:16:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.734+0000 7f61a7c79640 1 -- 172.21.15.27:0/906824811 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f618c0a2eb0 msgr2=0x7f618c0a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.734+0000 7f61a7c79640 1 --2- 172.21.15.27:0/906824811 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f618c0a2eb0 0x7f618c0a5370 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6198022000 tx=0x7f619801e000 comp rx=0 tx=0).stop 2024-04-25T00:16:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.734+0000 7f61a7c79640 1 -- 172.21.15.27:0/906824811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a01032e0 msgr2=0x7f61a0198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.734+0000 7f61a7c79640 1 --2- 172.21.15.27:0/906824811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a01032e0 0x7f61a0198500 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f619c006010 tx=0x7f619c0042f0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.734+0000 7f61a7c79640 1 -- 172.21.15.27:0/906824811 shutdown_connections 2024-04-25T00:16:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.734+0000 7f61a7c79640 1 --2- 172.21.15.27:0/906824811 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f618c0a2eb0 0x7f618c0a5370 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.734+0000 7f61a7c79640 1 --2- 172.21.15.27:0/906824811 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61a010bd00 0x7f61a019c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.734+0000 7f61a7c79640 1 --2- 172.21.15.27:0/906824811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61a01032e0 0x7f61a0198500 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.734+0000 7f61a7c79640 1 --2- 172.21.15.27:0/906824811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61a01029c0 0x7f61a0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.734+0000 7f61a7c79640 1 -- 172.21.15.27:0/906824811 >> 172.21.15.27:0/906824811 conn(0x7f61a0077740 msgr2=0x7f61a00fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.734+0000 7f61a7c79640 1 -- 172.21.15.27:0/906824811 shutdown_connections 2024-04-25T00:16:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:24.734+0000 7f61a7c79640 1 -- 172.21.15.27:0/906824811 wait complete. 2024-04-25T00:16:25.748 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T00:16:25.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:16:25.748 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T00:16:25.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.859+0000 7f6a64e76640 1 -- 172.21.15.27:0/4095308993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a601029c0 msgr2=0x7f6a60102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:25.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.859+0000 7f6a64e76640 1 --2- 172.21.15.27:0/4095308993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a601029c0 0x7f6a60102da0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f6a50009900 tx=0x7f6a5002a510 comp rx=0 tx=0).stop 2024-04-25T00:16:25.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.859+0000 7f6a64e76640 1 -- 172.21.15.27:0/4095308993 shutdown_connections 2024-04-25T00:16:25.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.859+0000 7f6a64e76640 1 --2- 172.21.15.27:0/4095308993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a6010bd00 0x7f6a6010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.859+0000 7f6a64e76640 1 --2- 172.21.15.27:0/4095308993 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a601032e0 0x7f6a6010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.859+0000 7f6a64e76640 1 --2- 172.21.15.27:0/4095308993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a601029c0 0x7f6a60102da0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.859+0000 7f6a64e76640 1 -- 172.21.15.27:0/4095308993 >> 172.21.15.27:0/4095308993 conn(0x7f6a60077740 msgr2=0x7f6a60077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:25.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.859+0000 7f6a64e76640 1 -- 172.21.15.27:0/4095308993 shutdown_connections 2024-04-25T00:16:25.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.860+0000 7f6a64e76640 1 -- 172.21.15.27:0/4095308993 wait complete. 2024-04-25T00:16:25.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.860+0000 7f6a64e76640 1 Processor -- start 2024-04-25T00:16:25.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.860+0000 7f6a64e76640 1 -- start start 2024-04-25T00:16:25.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.860+0000 7f6a64e76640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a601029c0 0x7f6a6019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.860+0000 7f6a64e76640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a601032e0 0x7f6a6019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.860+0000 7f6a64e76640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a6010bd00 0x7f6a6019ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.860+0000 7f6a64e76640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6a60111b00 con 0x7f6a601029c0 2024-04-25T00:16:25.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.860+0000 7f6a64e76640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6a60111980 con 0x7f6a6010bd00 2024-04-25T00:16:25.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.860+0000 7f6a64e76640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a60111c80 con 0x7f6a601032e0 2024-04-25T00:16:25.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.861+0000 7f6a5e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a601029c0 0x7f6a6019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.861+0000 7f6a5e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a601029c0 0x7f6a6019a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35512/0 (socket says 172.21.15.27:35512) 2024-04-25T00:16:25.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.861+0000 7f6a5e59c640 1 -- 172.21.15.27:0/1883699889 learned_addr learned my addr 172.21.15.27:0/1883699889 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:25.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.861+0000 7f6a5dd9b640 1 --2- 172.21.15.27:0/1883699889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a601032e0 0x7f6a6019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.861+0000 7f6a5ed9d640 1 --2- 172.21.15.27:0/1883699889 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a6010bd00 0x7f6a6019ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.861+0000 7f6a5e59c640 1 -- 172.21.15.27:0/1883699889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a601032e0 msgr2=0x7f6a6019a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:25.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.861+0000 7f6a5e59c640 1 --2- 172.21.15.27:0/1883699889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a601032e0 0x7f6a6019a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.861+0000 7f6a5e59c640 1 -- 172.21.15.27:0/1883699889 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a6010bd00 msgr2=0x7f6a6019ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:25.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.861+0000 7f6a5e59c640 1 --2- 172.21.15.27:0/1883699889 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a6010bd00 0x7f6a6019ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.861+0000 7f6a5e59c640 1 -- 172.21.15.27:0/1883699889 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a6019f1c0 con 0x7f6a601029c0 2024-04-25T00:16:25.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.861+0000 7f6a5ed9d640 1 --2- 172.21.15.27:0/1883699889 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a6010bd00 0x7f6a6019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:25.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.861+0000 7f6a5dd9b640 1 --2- 172.21.15.27:0/1883699889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a601032e0 0x7f6a6019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:25.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.861+0000 7f6a5e59c640 1 --2- 172.21.15.27:0/1883699889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a601029c0 0x7f6a6019a1d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6a50006010 tx=0x7f6a500042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:25.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.861+0000 7f6a4f7fe640 1 -- 172.21.15.27:0/1883699889 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5003f030 con 0x7f6a601029c0 2024-04-25T00:16:25.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.861+0000 7f6a64e76640 1 -- 172.21.15.27:0/1883699889 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a6019f450 con 0x7f6a601029c0 2024-04-25T00:16:25.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.861+0000 7f6a4f7fe640 1 -- 172.21.15.27:0/1883699889 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a50003ce0 con 0x7f6a601029c0 2024-04-25T00:16:25.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.862+0000 7f6a4f7fe640 1 -- 172.21.15.27:0/1883699889 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a50035070 con 0x7f6a601029c0 2024-04-25T00:16:25.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.862+0000 7f6a64e76640 1 -- 172.21.15.27:0/1883699889 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a601a6d40 con 0x7f6a601029c0 2024-04-25T00:16:25.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.864+0000 7f6a4f7fe640 1 -- 172.21.15.27:0/1883699889 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f6a50042030 con 0x7f6a601029c0 2024-04-25T00:16:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.864+0000 7f6a67b86740 1 Processor -- start 2024-04-25T00:16:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.864+0000 7f6a4f7fe640 1 --2- 172.21.15.27:0/1883699889 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6a480a2eb0 0x7f6a480a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.865+0000 7f6a4f7fe640 1 -- 172.21.15.27:0/1883699889 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f6a500f0000 con 0x7f6a601029c0 2024-04-25T00:16:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.865+0000 7f6a5ed9d640 1 --2- 172.21.15.27:0/1883699889 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6a480a2eb0 0x7f6a480a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.865+0000 7f6a67b86740 1 -- start start 2024-04-25T00:16:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.865+0000 7f6a67b86740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a31980 0x55ec67a2fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.865+0000 7f6a67b86740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a30290 0x55ec67a30710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.865+0000 7f6a67b86740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a3b010 0x55ec67a3d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.865+0000 7f6a67b86740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ec679942c0 con 0x55ec67a3b010 2024-04-25T00:16:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.865+0000 7f6a67b86740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ec6793c910 con 0x55ec67a30290 2024-04-25T00:16:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.865+0000 7f6a67b86740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ec67990280 con 0x55ec67a31980 2024-04-25T00:16:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.865+0000 7f6a5ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a3b010 0x55ec67a3d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.865+0000 7f6a5dd9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a30290 0x55ec67a30710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.865+0000 7f6a5ed9d640 1 --2- 172.21.15.27:0/1883699889 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6a480a2eb0 0x7f6a480a5370 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6a58005fd0 tx=0x7f6a58009500 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.865+0000 7f6a5ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a3b010 0x55ec67a3d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35526/0 (socket says 172.21.15.27:35526) 2024-04-25T00:16:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.865+0000 7f6a5e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a31980 0x55ec67a2fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.865+0000 7f6a5dd9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a30290 0x55ec67a30710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:45168/0 (socket says 172.21.15.27:45168) 2024-04-25T00:16:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.865+0000 7f6a5ed9d640 1 -- 172.21.15.27:0/755486418 learned_addr learned my addr 172.21.15.27:0/755486418 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.866+0000 7f6a5ed9d640 1 -- 172.21.15.27:0/755486418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a31980 msgr2=0x55ec67a2fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.866+0000 7f6a5ed9d640 1 --2- 172.21.15.27:0/755486418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a31980 0x55ec67a2fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.866+0000 7f6a5ed9d640 1 -- 172.21.15.27:0/755486418 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a30290 msgr2=0x55ec67a30710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.866+0000 7f6a5ed9d640 1 --2- 172.21.15.27:0/755486418 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a30290 0x55ec67a30710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.866+0000 7f6a5ed9d640 1 -- 172.21.15.27:0/755486418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec6794dfb0 con 0x55ec67a3b010 2024-04-25T00:16:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.866+0000 7f6a5ed9d640 1 --2- 172.21.15.27:0/755486418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a3b010 0x55ec67a3d400 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f6a58011740 tx=0x7f6a58013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=912566fa98845456 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.866+0000 7f6a4cff9640 1 -- 172.21.15.27:0/755486418 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5800e800 con 0x55ec67a3b010 2024-04-25T00:16:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.866+0000 7f6a4cff9640 1 -- 172.21.15.27:0/755486418 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a5800ee80 con 0x55ec67a3b010 2024-04-25T00:16:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.866+0000 7f6a4cff9640 1 -- 172.21.15.27:0/755486418 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a58026400 con 0x55ec67a3b010 2024-04-25T00:16:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.867+0000 7f6a67b86740 1 -- 172.21.15.27:0/755486418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a3b010 msgr2=0x55ec67a3d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.867+0000 7f6a67b86740 1 --2- 172.21.15.27:0/755486418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a3b010 0x55ec67a3d400 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f6a58011740 tx=0x7f6a58013000 comp rx=0 tx=0).stop 2024-04-25T00:16:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.867+0000 7f6a67b86740 1 -- 172.21.15.27:0/755486418 shutdown_connections 2024-04-25T00:16:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.867+0000 7f6a67b86740 1 --2- 172.21.15.27:0/755486418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a3b010 0x55ec67a3d400 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.867+0000 7f6a67b86740 1 --2- 172.21.15.27:0/755486418 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a30290 0x55ec67a30710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.867+0000 7f6a67b86740 1 --2- 172.21.15.27:0/755486418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a31980 0x55ec67a2fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.867+0000 7f6a67b86740 1 -- 172.21.15.27:0/755486418 >> 172.21.15.27:0/755486418 conn(0x55ec679f8950 msgr2=0x55ec67a2c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.867+0000 7f6a67b86740 1 -- 172.21.15.27:0/755486418 shutdown_connections 2024-04-25T00:16:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.867+0000 7f6a67b86740 1 -- 172.21.15.27:0/755486418 wait complete. 2024-04-25T00:16:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.867+0000 7f6a67b86740 1 -- start start 2024-04-25T00:16:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.867+0000 7f6a67b86740 1 Processor -- start 2024-04-25T00:16:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.867+0000 7f6a67b86740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a30290 0x55ec67a37f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.867+0000 7f6a67b86740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a31980 0x55ec67a38490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a67b86740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a3b010 0x55ec67a389d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a67b86740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ec67a3dea0 con 0x55ec67a30290 2024-04-25T00:16:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a67b86740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ec6794fe90 con 0x55ec67a3b010 2024-04-25T00:16:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a67b86740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ec67a3e020 con 0x55ec67a31980 2024-04-25T00:16:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a5e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a30290 0x55ec67a37f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a67b86740 10 client.0 _ll_register_callbacks cb 0x55ec6798b970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a5e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a30290 0x55ec67a37f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35542/0 (socket says 172.21.15.27:35542) 2024-04-25T00:16:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a5e59c640 1 -- 172.21.15.27:0/3192049018 learned_addr learned my addr 172.21.15.27:0/3192049018 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a5dd9b640 1 --2- 172.21.15.27:0/3192049018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a31980 0x55ec67a38490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a5ed9d640 1 --2- 172.21.15.27:0/3192049018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a3b010 0x55ec67a389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a5e59c640 1 -- 172.21.15.27:0/3192049018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a31980 msgr2=0x55ec67a38490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a5e59c640 1 --2- 172.21.15.27:0/3192049018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a31980 0x55ec67a38490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a5e59c640 1 -- 172.21.15.27:0/3192049018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a3b010 msgr2=0x55ec67a389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a5e59c640 1 --2- 172.21.15.27:0/3192049018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a3b010 0x55ec67a389d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a5e59c640 1 -- 172.21.15.27:0/3192049018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec677707e0 con 0x55ec67a30290 2024-04-25T00:16:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a5dd9b640 1 --2- 172.21.15.27:0/3192049018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a31980 0x55ec67a38490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a5e59c640 1 --2- 172.21.15.27:0/3192049018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a30290 0x55ec67a37f50 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f6a500bbfc0 tx=0x7f6a50009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9279240454751897 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:25.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:25.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a45ffb640 1 -- 172.21.15.27:0/3192049018 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a500d6070 con 0x55ec67a30290 2024-04-25T00:16:25.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a45ffb640 1 -- 172.21.15.27:0/3192049018 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f6a500c4810 con 0x55ec67a30290 2024-04-25T00:16:25.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a67b86740 1 -- 172.21.15.27:0/3192049018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ec679685a0 con 0x55ec67a30290 2024-04-25T00:16:25.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a45ffb640 1 -- 172.21.15.27:0/3192049018 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6a50079050 con 0x55ec67a30290 2024-04-25T00:16:25.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a67b86740 10 client.15960 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:25.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.868+0000 7f6a67b86740 1 -- 172.21.15.27:0/3192049018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ec677d0f70 con 0x55ec67a30290 2024-04-25T00:16:25.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.869+0000 7f6a45ffb640 1 -- 172.21.15.27:0/3192049018 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ec679685a0 con 0x55ec67a30290 2024-04-25T00:16:25.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.869+0000 7f6a45ffb640 1 -- 172.21.15.27:0/3192049018 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6a500dabb0 con 0x55ec67a30290 2024-04-25T00:16:25.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.869+0000 7f6a67b86740 10 client.15960 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:25.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.869+0000 7f6a67b86740 10 client.15960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:16:25.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.869+0000 7f6a67b86740 20 client.15960 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:25.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.869+0000 7f6a67b86740 1 --2- 172.21.15.27:0/3192049018 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55ec67a4b0e0 0x55ec67a4d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.869+0000 7f6a67b86740 4 client.15960 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:16:25.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.869+0000 7f6a67b86740 1 -- 172.21.15.27:0/3192049018 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec679a1800 con 0x55ec67a4b0e0 2024-04-25T00:16:25.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.869+0000 7f6a5dd9b640 1 --2- 172.21.15.27:0/3192049018 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55ec67a4b0e0 0x55ec67a4d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.870+0000 7f6a5dd9b640 1 --2- 172.21.15.27:0/3192049018 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55ec67a4b0e0 0x55ec67a4d530 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d186af28fc940443 server_cookie=d098abbc6c068072 in_seq=0 out_seq=0 2024-04-25T00:16:25.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.870+0000 7f6a45ffb640 10 client.15960 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:25.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.870+0000 7f6a45ffb640 1 -- 172.21.15.27:0/3192049018 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ec679a1800 con 0x55ec67a4b0e0 2024-04-25T00:16:25.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.870+0000 7f6a45ffb640 10 client.15960 handle_command_reply: tid=0 2024-04-25T00:16:25.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.870+0000 7f6a67b86740 1 client.15960 shutdown 2024-04-25T00:16:25.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.871+0000 7f6a67b86740 1 -- 172.21.15.27:0/3192049018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a30290 msgr2=0x55ec67a37f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:25.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.871+0000 7f6a67b86740 1 --2- 172.21.15.27:0/3192049018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a30290 0x55ec67a37f50 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f6a500bbfc0 tx=0x7f6a50009560 comp rx=0 tx=0).stop 2024-04-25T00:16:25.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.871+0000 7f6a67b86740 1 -- 172.21.15.27:0/3192049018 shutdown_connections 2024-04-25T00:16:25.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.871+0000 7f6a67b86740 1 --2- 172.21.15.27:0/3192049018 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55ec67a4b0e0 0x55ec67a4d530 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.871+0000 7f6a67b86740 1 --2- 172.21.15.27:0/3192049018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a3b010 0x55ec67a389d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.871+0000 7f6a67b86740 1 --2- 172.21.15.27:0/3192049018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a31980 0x55ec67a38490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.871+0000 7f6a67b86740 1 --2- 172.21.15.27:0/3192049018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a30290 0x55ec67a37f50 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.871+0000 7f6a67b86740 1 -- 172.21.15.27:0/3192049018 >> 172.21.15.27:0/3192049018 conn(0x55ec679f8950 msgr2=0x55ec67a32660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.871+0000 7f6a45ffb640 0 client.15960 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.871+0000 7f6a67b86740 1 -- 172.21.15.27:0/3192049018 shutdown_connections 2024-04-25T00:16:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.871+0000 7f6a67b86740 1 -- 172.21.15.27:0/3192049018 wait complete. 2024-04-25T00:16:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.871+0000 7f6a67b86740 20 client.15960 trim_cache size 0 max 16384 2024-04-25T00:16:25.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.888+0000 7f6a67b86740 1 Processor -- start 2024-04-25T00:16:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.888+0000 7f6a67b86740 1 -- start start 2024-04-25T00:16:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a67b86740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a30290 0x55ec67a46dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a67b86740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a31980 0x55ec67a2b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a67b86740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a3b010 0x55ec67a52e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a67b86740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6a601b0d70 con 0x55ec67a30290 2024-04-25T00:16:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a67b86740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6a601b28b0 con 0x55ec67a31980 2024-04-25T00:16:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a67b86740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a601b1440 con 0x55ec67a3b010 2024-04-25T00:16:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a5e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a30290 0x55ec67a46dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a5e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a30290 0x55ec67a46dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35550/0 (socket says 172.21.15.27:35550) 2024-04-25T00:16:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a5e59c640 1 -- 172.21.15.27:0/2999113122 learned_addr learned my addr 172.21.15.27:0/2999113122 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a5ed9d640 1 --2- 172.21.15.27:0/2999113122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a3b010 0x55ec67a52e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a5dd9b640 1 --2- 172.21.15.27:0/2999113122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a31980 0x55ec67a2b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a5e59c640 1 -- 172.21.15.27:0/2999113122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a3b010 msgr2=0x55ec67a52e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a5e59c640 1 --2- 172.21.15.27:0/2999113122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a3b010 0x55ec67a52e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a5e59c640 1 -- 172.21.15.27:0/2999113122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a31980 msgr2=0x55ec67a2b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a5e59c640 1 --2- 172.21.15.27:0/2999113122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a31980 0x55ec67a2b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a5e59c640 1 -- 172.21.15.27:0/2999113122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec67a36f40 con 0x55ec67a30290 2024-04-25T00:16:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a5dd9b640 1 --2- 172.21.15.27:0/2999113122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a31980 0x55ec67a2b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a5e59c640 1 --2- 172.21.15.27:0/2999113122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a30290 0x55ec67a46dd0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f6a50005ab0 tx=0x7f6a50032bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f954e27a828889e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.889+0000 7f6a47fff640 1 -- 172.21.15.27:0/2999113122 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a500d6070 con 0x55ec67a30290 2024-04-25T00:16:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.890+0000 7f6a47fff640 1 -- 172.21.15.27:0/2999113122 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a500bc730 con 0x55ec67a30290 2024-04-25T00:16:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.890+0000 7f6a47fff640 1 -- 172.21.15.27:0/2999113122 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a50049050 con 0x55ec67a30290 2024-04-25T00:16:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.890+0000 7f6a67b86740 1 -- 172.21.15.27:0/2999113122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a30290 msgr2=0x55ec67a46dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.890+0000 7f6a67b86740 1 --2- 172.21.15.27:0/2999113122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a30290 0x55ec67a46dd0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f6a50005ab0 tx=0x7f6a50032bc0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.890+0000 7f6a67b86740 1 -- 172.21.15.27:0/2999113122 shutdown_connections 2024-04-25T00:16:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.890+0000 7f6a67b86740 1 --2- 172.21.15.27:0/2999113122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a3b010 0x55ec67a52e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.890+0000 7f6a67b86740 1 --2- 172.21.15.27:0/2999113122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a31980 0x55ec67a2b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.890+0000 7f6a67b86740 1 --2- 172.21.15.27:0/2999113122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a30290 0x55ec67a46dd0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.890+0000 7f6a67b86740 1 -- 172.21.15.27:0/2999113122 >> 172.21.15.27:0/2999113122 conn(0x55ec679f8950 msgr2=0x55ec67a2b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.890+0000 7f6a67b86740 1 -- 172.21.15.27:0/2999113122 shutdown_connections 2024-04-25T00:16:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.890+0000 7f6a67b86740 1 -- 172.21.15.27:0/2999113122 wait complete. 2024-04-25T00:16:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a67b86740 1 -- start start 2024-04-25T00:16:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a67b86740 1 Processor -- start 2024-04-25T00:16:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a67b86740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a30290 0x55ec67a432e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a67b86740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a31980 0x55ec67a43820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a67b86740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a3b010 0x55ec67a402b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a67b86740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6a601b21e0 con 0x55ec67a3b010 2024-04-25T00:16:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a67b86740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6a601b1b10 con 0x55ec67a30290 2024-04-25T00:16:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a67b86740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a601aed30 con 0x55ec67a31980 2024-04-25T00:16:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a67b86740 10 client.0 _ll_register_callbacks cb 0x7f6a50003e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a5ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a3b010 0x55ec67a402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a5ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a3b010 0x55ec67a402b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35554/0 (socket says 172.21.15.27:35554) 2024-04-25T00:16:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a5ed9d640 1 -- 172.21.15.27:0/627298827 learned_addr learned my addr 172.21.15.27:0/627298827 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a5e59c640 1 --2- 172.21.15.27:0/627298827 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a30290 0x55ec67a432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a5dd9b640 1 --2- 172.21.15.27:0/627298827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a31980 0x55ec67a43820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a5ed9d640 1 -- 172.21.15.27:0/627298827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a31980 msgr2=0x55ec67a43820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a5ed9d640 1 --2- 172.21.15.27:0/627298827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a31980 0x55ec67a43820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a5ed9d640 1 -- 172.21.15.27:0/627298827 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a30290 msgr2=0x55ec67a432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a5ed9d640 1 --2- 172.21.15.27:0/627298827 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a30290 0x55ec67a432e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a5ed9d640 1 -- 172.21.15.27:0/627298827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec67668170 con 0x55ec67a3b010 2024-04-25T00:16:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.891+0000 7f6a5dd9b640 1 --2- 172.21.15.27:0/627298827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a31980 0x55ec67a43820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.892+0000 7f6a5ed9d640 1 --2- 172.21.15.27:0/627298827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a3b010 0x55ec67a402b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f6a58011740 tx=0x7f6a5800e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4657a3fd2e780db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.892+0000 7f6a45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.892+0000 7f6a45ffb640 1 -- 172.21.15.27:0/627298827 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5801e420 con 0x55ec67a3b010 2024-04-25T00:16:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.892+0000 7f6a45ffb640 1 -- 172.21.15.27:0/627298827 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f6a5801e5c0 con 0x55ec67a3b010 2024-04-25T00:16:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.892+0000 7f6a67b86740 1 -- 172.21.15.27:0/627298827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a500bc730 con 0x55ec67a3b010 2024-04-25T00:16:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.892+0000 7f6a45ffb640 1 -- 172.21.15.27:0/627298827 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6a5803d050 con 0x55ec67a3b010 2024-04-25T00:16:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.892+0000 7f6a67b86740 10 client.15966 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.892+0000 7f6a67b86740 1 -- 172.21.15.27:0/627298827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ec67ab1ed0 con 0x55ec67a3b010 2024-04-25T00:16:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.892+0000 7f6a45ffb640 1 -- 172.21.15.27:0/627298827 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a500bc730 con 0x55ec67a3b010 2024-04-25T00:16:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.892+0000 7f6a45ffb640 1 -- 172.21.15.27:0/627298827 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6a5802fad0 con 0x55ec67a3b010 2024-04-25T00:16:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.893+0000 7f6a67b86740 10 client.15966 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.893+0000 7f6a67b86740 10 client.15966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:16:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.893+0000 7f6a67b86740 20 client.15966 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.893+0000 7f6a67b86740 1 --2- 172.21.15.27:0/627298827 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55ec67ab24e0 0x55ec67ab4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.893+0000 7f6a67b86740 4 client.15966 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T00:16:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.893+0000 7f6a67b86740 1 -- 172.21.15.27:0/627298827 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ec6799bb50 con 0x55ec67ab24e0 2024-04-25T00:16:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.893+0000 7f6a5e59c640 1 --2- 172.21.15.27:0/627298827 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55ec67ab24e0 0x55ec67ab4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.893+0000 7f6a45ffb640 10 client.15966 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.893+0000 7f6a5e59c640 1 --2- 172.21.15.27:0/627298827 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55ec67ab24e0 0x55ec67ab4900 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a699258a86a42a9 server_cookie=57a2b82199460d9 in_seq=0 out_seq=0 2024-04-25T00:16:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.894+0000 7f6a45ffb640 1 -- 172.21.15.27:0/627298827 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ec6799bb50 con 0x55ec67ab24e0 2024-04-25T00:16:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.894+0000 7f6a45ffb640 10 client.15966 handle_command_reply: tid=0 2024-04-25T00:16:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.894+0000 7f6a67b86740 1 client.15966 shutdown 2024-04-25T00:16:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a67b86740 1 -- 172.21.15.27:0/627298827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a3b010 msgr2=0x55ec67a402b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a67b86740 1 --2- 172.21.15.27:0/627298827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a3b010 0x55ec67a402b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f6a58011740 tx=0x7f6a5800e830 comp rx=0 tx=0).stop 2024-04-25T00:16:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a67b86740 1 -- 172.21.15.27:0/627298827 shutdown_connections 2024-04-25T00:16:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a67b86740 1 --2- 172.21.15.27:0/627298827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec67a3b010 0x55ec67a402b0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a67b86740 1 --2- 172.21.15.27:0/627298827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ec67a31980 0x55ec67a43820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a67b86740 1 --2- 172.21.15.27:0/627298827 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55ec67ab24e0 0x55ec67ab4900 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a67b86740 1 --2- 172.21.15.27:0/627298827 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ec67a30290 0x55ec67a432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a67b86740 1 -- 172.21.15.27:0/627298827 >> 172.21.15.27:0/627298827 conn(0x55ec679f8950 msgr2=0x55ec67a2b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a45ffb640 0 client.15966 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a67b86740 1 -- 172.21.15.27:0/627298827 shutdown_connections 2024-04-25T00:16:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a67b86740 1 -- 172.21.15.27:0/627298827 wait complete. 2024-04-25T00:16:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a67b86740 20 client.15966 trim_cache size 0 max 16384 2024-04-25T00:16:25.902 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:16:25.902 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T00:16:25.902 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "abef993d-9c17-4d1d-a926-feb6d1cb9852", 2024-04-25T00:16:25.902 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T00:16:25.902 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:16:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a64e76640 1 -- 172.21.15.27:0/1883699889 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6a480a2eb0 msgr2=0x7f6a480a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a64e76640 1 --2- 172.21.15.27:0/1883699889 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6a480a2eb0 0x7f6a480a5370 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6a58005fd0 tx=0x7f6a58009500 comp rx=0 tx=0).stop 2024-04-25T00:16:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a64e76640 1 -- 172.21.15.27:0/1883699889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a601029c0 msgr2=0x7f6a6019a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a64e76640 1 --2- 172.21.15.27:0/1883699889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a601029c0 0x7f6a6019a1d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6a50006010 tx=0x7f6a500042f0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a64e76640 1 -- 172.21.15.27:0/1883699889 shutdown_connections 2024-04-25T00:16:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a64e76640 1 --2- 172.21.15.27:0/1883699889 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6a480a2eb0 0x7f6a480a5370 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a64e76640 1 --2- 172.21.15.27:0/1883699889 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a6010bd00 0x7f6a6019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a64e76640 1 --2- 172.21.15.27:0/1883699889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a601032e0 0x7f6a6019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a64e76640 1 --2- 172.21.15.27:0/1883699889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a601029c0 0x7f6a6019a1d0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.895+0000 7f6a64e76640 1 -- 172.21.15.27:0/1883699889 >> 172.21.15.27:0/1883699889 conn(0x7f6a60077740 msgr2=0x7f6a60107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:25.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.896+0000 7f6a64e76640 1 -- 172.21.15.27:0/1883699889 shutdown_connections 2024-04-25T00:16:25.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:25.896+0000 7f6a64e76640 1 -- 172.21.15.27:0/1883699889 wait complete. 2024-04-25T00:16:25.907 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:16:25.907 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:16:26.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.012+0000 7f48c9d28640 1 -- 172.21.15.27:0/2669014961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48c41066b0 msgr2=0x7f48c4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:26.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.012+0000 7f48c9d28640 1 --2- 172.21.15.27:0/2669014961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48c41066b0 0x7f48c4106a90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f48b4009930 tx=0x7f48b402a420 comp rx=0 tx=0).stop 2024-04-25T00:16:26.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.012+0000 7f48c9d28640 1 -- 172.21.15.27:0/2669014961 shutdown_connections 2024-04-25T00:16:26.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.012+0000 7f48c9d28640 1 --2- 172.21.15.27:0/2669014961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c4100fd0 0x7f48c410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.012+0000 7f48c9d28640 1 --2- 172.21.15.27:0/2669014961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48c4100610 0x7f48c4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.012+0000 7f48c9d28640 1 --2- 172.21.15.27:0/2669014961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48c41066b0 0x7f48c4106a90 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.012+0000 7f48c9d28640 1 -- 172.21.15.27:0/2669014961 >> 172.21.15.27:0/2669014961 conn(0x7f48c4077740 msgr2=0x7f48c4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:26.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.012+0000 7f48c9d28640 1 -- 172.21.15.27:0/2669014961 shutdown_connections 2024-04-25T00:16:26.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.012+0000 7f48c9d28640 1 -- 172.21.15.27:0/2669014961 wait complete. 2024-04-25T00:16:26.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.013+0000 7f48c9d28640 1 Processor -- start 2024-04-25T00:16:26.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.013+0000 7f48c9d28640 1 -- start start 2024-04-25T00:16:26.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.013+0000 7f48c9d28640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c4100610 0x7f48c419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.013+0000 7f48c9d28640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48c4100fd0 0x7f48c419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.013+0000 7f48c9d28640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48c41066b0 0x7f48c419ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.013+0000 7f48c9d28640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f48c4110a40 con 0x7f48c41066b0 2024-04-25T00:16:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.013+0000 7f48c9d28640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f48c41108c0 con 0x7f48c4100fd0 2024-04-25T00:16:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.013+0000 7f48c9d28640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f48c4110bc0 con 0x7f48c4100610 2024-04-25T00:16:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.013+0000 7f48c3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48c41066b0 0x7f48c419ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.014+0000 7f48c3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48c41066b0 0x7f48c419ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35574/0 (socket says 172.21.15.27:35574) 2024-04-25T00:16:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.014+0000 7f48c3fff640 1 -- 172.21.15.27:0/34654185 learned_addr learned my addr 172.21.15.27:0/34654185 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.014+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/34654185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48c4100fd0 0x7f48c419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.014+0000 7f48c37fe640 1 --2- 172.21.15.27:0/34654185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c4100610 0x7f48c419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.014+0000 7f48c3fff640 1 -- 172.21.15.27:0/34654185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c4100610 msgr2=0x7f48c419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.014+0000 7f48c3fff640 1 --2- 172.21.15.27:0/34654185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c4100610 0x7f48c419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.014+0000 7f48c3fff640 1 -- 172.21.15.27:0/34654185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48c4100fd0 msgr2=0x7f48c419a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.014+0000 7f48c3fff640 1 --2- 172.21.15.27:0/34654185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48c4100fd0 0x7f48c419a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.014+0000 7f48c3fff640 1 -- 172.21.15.27:0/34654185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c419f150 con 0x7f48c41066b0 2024-04-25T00:16:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.014+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/34654185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48c4100fd0 0x7f48c419a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.014+0000 7f48c37fe640 1 --2- 172.21.15.27:0/34654185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c4100610 0x7f48c419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.014+0000 7f48c3fff640 1 --2- 172.21.15.27:0/34654185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48c41066b0 0x7f48c419ea20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f48bc00eaa0 tx=0x7f48bc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.014+0000 7f48c0ff9640 1 -- 172.21.15.27:0/34654185 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc00cdb0 con 0x7f48c41066b0 2024-04-25T00:16:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.014+0000 7f48c0ff9640 1 -- 172.21.15.27:0/34654185 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48bc005020 con 0x7f48c41066b0 2024-04-25T00:16:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.014+0000 7f48c0ff9640 1 -- 172.21.15.27:0/34654185 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc0116b0 con 0x7f48c41066b0 2024-04-25T00:16:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.014+0000 7f48c9d28640 1 -- 172.21.15.27:0/34654185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48c419f440 con 0x7f48c41066b0 2024-04-25T00:16:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.014+0000 7f48c9d28640 1 -- 172.21.15.27:0/34654185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48c41a6cd0 con 0x7f48c41066b0 2024-04-25T00:16:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.017+0000 7f48c0ff9640 1 -- 172.21.15.27:0/34654185 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f48bc005560 con 0x7f48c41066b0 2024-04-25T00:16:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.017+0000 7f48cca30740 1 Processor -- start 2024-04-25T00:16:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.017+0000 7f48cca30740 1 -- start start 2024-04-25T00:16:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.017+0000 7f48cca30740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55610785b630 0x55610785ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.017+0000 7f48cca30740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55610785bf50 0x5561078660e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.017+0000 7f48cca30740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556107866620 0x556107868a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.017+0000 7f48cca30740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5561077c06b0 con 0x556107866620 2024-04-25T00:16:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.017+0000 7f48cca30740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556107768880 con 0x55610785b630 2024-04-25T00:16:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.017+0000 7f48cca30740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5561077bc670 con 0x55610785bf50 2024-04-25T00:16:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48c0ff9640 1 --2- 172.21.15.27:0/34654185 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f48ac0a2eb0 0x7f48ac0a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48c2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556107866620 0x556107868a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48c2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556107866620 0x556107868a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35590/0 (socket says 172.21.15.27:35590) 2024-04-25T00:16:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48c2ffd640 1 -- 172.21.15.27:0/4082217633 learned_addr learned my addr 172.21.15.27:0/4082217633 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48c37fe640 1 --2- 172.21.15.27:0/4082217633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55610785bf50 0x5561078660e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/4082217633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55610785b630 0x55610785ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48c3fff640 1 --2- 172.21.15.27:0/34654185 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f48ac0a2eb0 0x7f48ac0a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48c0ff9640 1 -- 172.21.15.27:0/34654185 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f48bc0cfc40 con 0x7f48c41066b0 2024-04-25T00:16:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48c2ffd640 1 -- 172.21.15.27:0/4082217633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55610785bf50 msgr2=0x5561078660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/4082217633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55610785bf50 0x5561078660e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48c2ffd640 1 -- 172.21.15.27:0/4082217633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55610785b630 msgr2=0x55610785ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/4082217633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55610785b630 0x55610785ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48c2ffd640 1 -- 172.21.15.27:0/4082217633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55610759e7d0 con 0x556107866620 2024-04-25T00:16:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/4082217633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556107866620 0x556107868a10 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f48b8011be0 tx=0x7f48b8009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a931311d945d05c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48c3fff640 1 --2- 172.21.15.27:0/34654185 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f48ac0a2eb0 0x7f48ac0a5370 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f48bc0cce40 tx=0x7f48bc04b040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48a9ffb640 1 -- 172.21.15.27:0/4082217633 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b800fc50 con 0x556107866620 2024-04-25T00:16:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48a9ffb640 1 -- 172.21.15.27:0/4082217633 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48b800fdf0 con 0x556107866620 2024-04-25T00:16:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.018+0000 7f48a9ffb640 1 -- 172.21.15.27:0/4082217633 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b800e790 con 0x556107866620 2024-04-25T00:16:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.019+0000 7f48cca30740 1 -- 172.21.15.27:0/4082217633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556107866620 msgr2=0x556107868a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.019+0000 7f48cca30740 1 --2- 172.21.15.27:0/4082217633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556107866620 0x556107868a10 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f48b8011be0 tx=0x7f48b8009ba0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.019+0000 7f48cca30740 1 -- 172.21.15.27:0/4082217633 shutdown_connections 2024-04-25T00:16:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.019+0000 7f48cca30740 1 --2- 172.21.15.27:0/4082217633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556107866620 0x556107868a10 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.019+0000 7f48cca30740 1 --2- 172.21.15.27:0/4082217633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55610785bf50 0x5561078660e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.019+0000 7f48cca30740 1 --2- 172.21.15.27:0/4082217633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55610785b630 0x55610785ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.019+0000 7f48cca30740 1 -- 172.21.15.27:0/4082217633 >> 172.21.15.27:0/4082217633 conn(0x556107857950 msgr2=0x556107857d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.019+0000 7f48cca30740 1 -- 172.21.15.27:0/4082217633 shutdown_connections 2024-04-25T00:16:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.019+0000 7f48cca30740 1 -- 172.21.15.27:0/4082217633 wait complete. 2024-04-25T00:16:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.020+0000 7f48cca30740 1 -- start start 2024-04-25T00:16:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.020+0000 7f48cca30740 1 Processor -- start 2024-04-25T00:16:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.020+0000 7f48cca30740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785b630 0x5561078635f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.020+0000 7f48cca30740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55610785bf50 0x556107863b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.020+0000 7f48cca30740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556107866620 0x556107864070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.020+0000 7f48cca30740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556107869410 con 0x55610785b630 2024-04-25T00:16:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.020+0000 7f48cca30740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55610777bf50 con 0x55610785bf50 2024-04-25T00:16:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.020+0000 7f48cca30740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556107869590 con 0x556107866620 2024-04-25T00:16:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.020+0000 7f48cca30740 10 client.0 _ll_register_callbacks cb 0x5561077b7000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.020+0000 7f48c2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785b630 0x5561078635f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.020+0000 7f48c2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785b630 0x5561078635f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35600/0 (socket says 172.21.15.27:35600) 2024-04-25T00:16:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.020+0000 7f48c2ffd640 1 -- 172.21.15.27:0/3310458500 learned_addr learned my addr 172.21.15.27:0/3310458500 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.021+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/3310458500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556107866620 0x556107864070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.021+0000 7f48c37fe640 1 --2- 172.21.15.27:0/3310458500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55610785bf50 0x556107863b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.021+0000 7f48c2ffd640 1 -- 172.21.15.27:0/3310458500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556107866620 msgr2=0x556107864070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.021+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/3310458500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556107866620 0x556107864070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.021+0000 7f48c2ffd640 1 -- 172.21.15.27:0/3310458500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55610785bf50 msgr2=0x556107863b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.021+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/3310458500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55610785bf50 0x556107863b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.021+0000 7f48c2ffd640 1 -- 172.21.15.27:0/3310458500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561074d43f0 con 0x55610785b630 2024-04-25T00:16:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.021+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/3310458500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785b630 0x5561078635f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f48b80056d0 tx=0x7f48b800e450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=405c38f54bcf1e6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.021+0000 7f48a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.021+0000 7f48a2ffd640 1 -- 172.21.15.27:0/3310458500 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b801fa40 con 0x55610785b630 2024-04-25T00:16:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.021+0000 7f48a2ffd640 1 -- 172.21.15.27:0/3310458500 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f48b801fbe0 con 0x55610785b630 2024-04-25T00:16:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.021+0000 7f48cca30740 1 -- 172.21.15.27:0/3310458500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48b800fdf0 con 0x55610785b630 2024-04-25T00:16:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.021+0000 7f48a2ffd640 1 -- 172.21.15.27:0/3310458500 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f48b8034050 con 0x55610785b630 2024-04-25T00:16:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.021+0000 7f48cca30740 10 client.15978 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.021+0000 7f48cca30740 1 -- 172.21.15.27:0/3310458500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5561074940f0 con 0x55610785b630 2024-04-25T00:16:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.022+0000 7f48a2ffd640 1 -- 172.21.15.27:0/3310458500 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48b800fdf0 con 0x55610785b630 2024-04-25T00:16:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.022+0000 7f48a2ffd640 1 -- 172.21.15.27:0/3310458500 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f48b801d020 con 0x55610785b630 2024-04-25T00:16:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.022+0000 7f48cca30740 10 client.15978 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:26.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.022+0000 7f48cca30740 10 client.15978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:16:26.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.022+0000 7f48cca30740 20 client.15978 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:26.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.022+0000 7f48cca30740 1 --2- 172.21.15.27:0/3310458500 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5561078767d0 0x556107878bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.022+0000 7f48cca30740 4 client.15978 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:16:26.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.022+0000 7f48cca30740 1 -- 172.21.15.27:0/3310458500 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561077cdbf0 con 0x5561078767d0 2024-04-25T00:16:26.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.022+0000 7f48c37fe640 1 --2- 172.21.15.27:0/3310458500 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5561078767d0 0x556107878bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.022+0000 7f48a2ffd640 10 client.15978 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:26.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.022+0000 7f48c37fe640 1 --2- 172.21.15.27:0/3310458500 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5561078767d0 0x556107878bc0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=385682550db3e8a8 server_cookie=88378c4bccdb8797 in_seq=0 out_seq=0 2024-04-25T00:16:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.023+0000 7f48a2ffd640 1 -- 172.21.15.27:0/3310458500 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5561077cdbf0 con 0x5561078767d0 2024-04-25T00:16:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.023+0000 7f48a2ffd640 10 client.15978 handle_command_reply: tid=0 2024-04-25T00:16:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.023+0000 7f48cca30740 1 client.15978 shutdown 2024-04-25T00:16:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.023+0000 7f48cca30740 1 -- 172.21.15.27:0/3310458500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785b630 msgr2=0x5561078635f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.023+0000 7f48cca30740 1 --2- 172.21.15.27:0/3310458500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785b630 0x5561078635f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f48b80056d0 tx=0x7f48b800e450 comp rx=0 tx=0).stop 2024-04-25T00:16:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.023+0000 7f48cca30740 1 -- 172.21.15.27:0/3310458500 shutdown_connections 2024-04-25T00:16:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.023+0000 7f48cca30740 1 --2- 172.21.15.27:0/3310458500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556107866620 0x556107864070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.023+0000 7f48cca30740 1 --2- 172.21.15.27:0/3310458500 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5561078767d0 0x556107878bc0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.023+0000 7f48cca30740 1 --2- 172.21.15.27:0/3310458500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55610785bf50 0x556107863b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.023+0000 7f48cca30740 1 --2- 172.21.15.27:0/3310458500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785b630 0x5561078635f0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.023+0000 7f48a2ffd640 0 client.15978 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.023+0000 7f48cca30740 1 -- 172.21.15.27:0/3310458500 >> 172.21.15.27:0/3310458500 conn(0x556107857950 msgr2=0x55610785d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.023+0000 7f48cca30740 1 -- 172.21.15.27:0/3310458500 shutdown_connections 2024-04-25T00:16:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.023+0000 7f48cca30740 1 -- 172.21.15.27:0/3310458500 wait complete. 2024-04-25T00:16:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.024+0000 7f48cca30740 20 client.15978 trim_cache size 0 max 16384 2024-04-25T00:16:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.041+0000 7f48cca30740 1 Processor -- start 2024-04-25T00:16:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.041+0000 7f48cca30740 1 -- start start 2024-04-25T00:16:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.041+0000 7f48cca30740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785b630 0x556107857270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.041+0000 7f48cca30740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55610785bf50 0x556107865c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.041+0000 7f48cca30740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556107866620 0x55610785e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.041+0000 7f48cca30740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f48c41b0d50 con 0x55610785b630 2024-04-25T00:16:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.041+0000 7f48cca30740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f48c41b2890 con 0x556107866620 2024-04-25T00:16:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.041+0000 7f48cca30740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f48c41b1420 con 0x55610785bf50 2024-04-25T00:16:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.041+0000 7f48c2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785b630 0x556107857270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.041+0000 7f48c2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785b630 0x556107857270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35614/0 (socket says 172.21.15.27:35614) 2024-04-25T00:16:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.041+0000 7f48c2ffd640 1 -- 172.21.15.27:0/1991387142 learned_addr learned my addr 172.21.15.27:0/1991387142 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.042+0000 7f48c37fe640 1 --2- 172.21.15.27:0/1991387142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55610785bf50 0x556107865c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.042+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/1991387142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556107866620 0x55610785e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.042+0000 7f48c2ffd640 1 -- 172.21.15.27:0/1991387142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55610785bf50 msgr2=0x556107865c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.042+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/1991387142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55610785bf50 0x556107865c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.042+0000 7f48c2ffd640 1 -- 172.21.15.27:0/1991387142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556107866620 msgr2=0x55610785e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.042+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/1991387142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556107866620 0x55610785e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.042+0000 7f48c2ffd640 1 -- 172.21.15.27:0/1991387142 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556107863c90 con 0x55610785b630 2024-04-25T00:16:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.042+0000 7f48c37fe640 1 --2- 172.21.15.27:0/1991387142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55610785bf50 0x556107865c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.042+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/1991387142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785b630 0x556107857270 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f48b8000c00 tx=0x7f48b8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387075c0dcc489fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.042+0000 7f48a97fa640 1 -- 172.21.15.27:0/1991387142 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b800fdf0 con 0x55610785b630 2024-04-25T00:16:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.042+0000 7f48a97fa640 1 -- 172.21.15.27:0/1991387142 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48b8011070 con 0x55610785b630 2024-04-25T00:16:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.042+0000 7f48a97fa640 1 -- 172.21.15.27:0/1991387142 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b80038e0 con 0x55610785b630 2024-04-25T00:16:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.043+0000 7f48cca30740 1 -- 172.21.15.27:0/1991387142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785b630 msgr2=0x556107857270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.043+0000 7f48cca30740 1 --2- 172.21.15.27:0/1991387142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785b630 0x556107857270 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f48b8000c00 tx=0x7f48b8005f90 comp rx=0 tx=0).stop 2024-04-25T00:16:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.043+0000 7f48cca30740 1 -- 172.21.15.27:0/1991387142 shutdown_connections 2024-04-25T00:16:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.043+0000 7f48cca30740 1 --2- 172.21.15.27:0/1991387142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556107866620 0x55610785e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.043+0000 7f48cca30740 1 --2- 172.21.15.27:0/1991387142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55610785bf50 0x556107865c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.043+0000 7f48cca30740 1 --2- 172.21.15.27:0/1991387142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785b630 0x556107857270 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.043+0000 7f48cca30740 1 -- 172.21.15.27:0/1991387142 >> 172.21.15.27:0/1991387142 conn(0x556107857950 msgr2=0x5561078712c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.043+0000 7f48cca30740 1 -- 172.21.15.27:0/1991387142 shutdown_connections 2024-04-25T00:16:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.043+0000 7f48cca30740 1 -- 172.21.15.27:0/1991387142 wait complete. 2024-04-25T00:16:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.043+0000 7f48cca30740 1 -- start start 2024-04-25T00:16:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.043+0000 7f48cca30740 1 Processor -- start 2024-04-25T00:16:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48cca30740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55610785b630 0x55610786ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48cca30740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785bf50 0x55610785ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48cca30740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556107866620 0x556107860520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48cca30740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f48c41b21c0 con 0x55610785bf50 2024-04-25T00:16:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48cca30740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f48c41b1af0 con 0x556107866620 2024-04-25T00:16:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48cca30740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f48c41aed10 con 0x55610785b630 2024-04-25T00:16:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48cca30740 10 client.0 _ll_register_callbacks cb 0x7f48b80116b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48c37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785bf50 0x55610785ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48c37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785bf50 0x55610785ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35624/0 (socket says 172.21.15.27:35624) 2024-04-25T00:16:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48c37fe640 1 -- 172.21.15.27:0/1003909682 learned_addr learned my addr 172.21.15.27:0/1003909682 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/1003909682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556107866620 0x556107860520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/1003909682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55610785b630 0x55610786ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48c37fe640 1 -- 172.21.15.27:0/1003909682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55610785b630 msgr2=0x55610786ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48c37fe640 1 --2- 172.21.15.27:0/1003909682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55610785b630 0x55610786ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48c37fe640 1 -- 172.21.15.27:0/1003909682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556107866620 msgr2=0x556107860520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48c37fe640 1 --2- 172.21.15.27:0/1003909682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556107866620 0x556107860520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48c37fe640 1 -- 172.21.15.27:0/1003909682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556107860d50 con 0x55610785bf50 2024-04-25T00:16:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48c37fe640 1 --2- 172.21.15.27:0/1003909682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785bf50 0x55610785ffe0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f48b4009930 tx=0x7f48b4009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5197460e7e9505fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48a2ffd640 1 -- 172.21.15.27:0/1003909682 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b4035040 con 0x55610785bf50 2024-04-25T00:16:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.044+0000 7f48a2ffd640 1 -- 172.21.15.27:0/1003909682 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f48b4046340 con 0x55610785bf50 2024-04-25T00:16:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.045+0000 7f48cca30740 1 -- 172.21.15.27:0/1003909682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48b8011070 con 0x55610785bf50 2024-04-25T00:16:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.045+0000 7f48a2ffd640 1 -- 172.21.15.27:0/1003909682 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f48b4055050 con 0x55610785bf50 2024-04-25T00:16:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.045+0000 7f48cca30740 10 client.15984 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.045+0000 7f48cca30740 1 -- 172.21.15.27:0/1003909682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5561078dcb60 con 0x55610785bf50 2024-04-25T00:16:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.045+0000 7f48a2ffd640 1 -- 172.21.15.27:0/1003909682 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48b8011070 con 0x55610785bf50 2024-04-25T00:16:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.045+0000 7f48a2ffd640 1 -- 172.21.15.27:0/1003909682 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f48b40315b0 con 0x55610785bf50 2024-04-25T00:16:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.045+0000 7f48cca30740 10 client.15984 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.045+0000 7f48cca30740 10 client.15984 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:16:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.045+0000 7f48cca30740 20 client.15984 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.045+0000 7f48cca30740 1 --2- 172.21.15.27:0/1003909682 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5561078dd360 0x5561078df780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.045+0000 7f48cca30740 4 client.15984 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:16:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.045+0000 7f48cca30740 1 -- 172.21.15.27:0/1003909682 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5561077c7f40 con 0x5561078dd360 2024-04-25T00:16:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.045+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/1003909682 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5561078dd360 0x5561078df780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.046+0000 7f48c2ffd640 1 --2- 172.21.15.27:0/1003909682 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5561078dd360 0x5561078df780 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c93353016c79dac server_cookie=9cc1238a250691df in_seq=0 out_seq=0 2024-04-25T00:16:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.046+0000 7f48a2ffd640 10 client.15984 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.046+0000 7f48a2ffd640 1 -- 172.21.15.27:0/1003909682 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5561077c7f40 con 0x5561078dd360 2024-04-25T00:16:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.046+0000 7f48a2ffd640 10 client.15984 handle_command_reply: tid=0 2024-04-25T00:16:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.046+0000 7f48cca30740 1 client.15984 shutdown 2024-04-25T00:16:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.046+0000 7f48cca30740 1 -- 172.21.15.27:0/1003909682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785bf50 msgr2=0x55610785ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.046+0000 7f48cca30740 1 --2- 172.21.15.27:0/1003909682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785bf50 0x55610785ffe0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f48b4009930 tx=0x7f48b4009680 comp rx=0 tx=0).stop 2024-04-25T00:16:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.046+0000 7f48cca30740 1 -- 172.21.15.27:0/1003909682 shutdown_connections 2024-04-25T00:16:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.046+0000 7f48cca30740 1 --2- 172.21.15.27:0/1003909682 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5561078dd360 0x5561078df780 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.046+0000 7f48cca30740 1 --2- 172.21.15.27:0/1003909682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556107866620 0x556107860520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.046+0000 7f48cca30740 1 --2- 172.21.15.27:0/1003909682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55610785bf50 0x55610785ffe0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.046+0000 7f48cca30740 1 --2- 172.21.15.27:0/1003909682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55610785b630 0x55610786ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.046+0000 7f48a2ffd640 0 client.15984 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.046+0000 7f48cca30740 1 -- 172.21.15.27:0/1003909682 >> 172.21.15.27:0/1003909682 conn(0x556107857950 msgr2=0x556107865a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.046+0000 7f48cca30740 1 -- 172.21.15.27:0/1003909682 shutdown_connections 2024-04-25T00:16:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.046+0000 7f48cca30740 1 -- 172.21.15.27:0/1003909682 wait complete. 2024-04-25T00:16:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.046+0000 7f48cca30740 20 client.15984 trim_cache size 0 max 16384 2024-04-25T00:16:26.053 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:16:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T00:16:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:16:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "abef993d-9c17-4d1d-a926-feb6d1cb9852": { 2024-04-25T00:16:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:16:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "abef993d-9c17-4d1d-a926-feb6d1cb9852", 2024-04-25T00:16:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:16:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:16:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:16:26.053 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:16:26.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.047+0000 7f48c9d28640 1 -- 172.21.15.27:0/34654185 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f48ac0a2eb0 msgr2=0x7f48ac0a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:26.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.047+0000 7f48c9d28640 1 --2- 172.21.15.27:0/34654185 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f48ac0a2eb0 0x7f48ac0a5370 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f48bc0cce40 tx=0x7f48bc04b040 comp rx=0 tx=0).stop 2024-04-25T00:16:26.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.047+0000 7f48c9d28640 1 -- 172.21.15.27:0/34654185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48c41066b0 msgr2=0x7f48c419ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:26.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.047+0000 7f48c9d28640 1 --2- 172.21.15.27:0/34654185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48c41066b0 0x7f48c419ea20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f48bc00eaa0 tx=0x7f48bc00ef60 comp rx=0 tx=0).stop 2024-04-25T00:16:26.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.047+0000 7f48c9d28640 1 -- 172.21.15.27:0/34654185 shutdown_connections 2024-04-25T00:16:26.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.047+0000 7f48c9d28640 1 --2- 172.21.15.27:0/34654185 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f48ac0a2eb0 0x7f48ac0a5370 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.047+0000 7f48c9d28640 1 --2- 172.21.15.27:0/34654185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48c41066b0 0x7f48c419ea20 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.047+0000 7f48c9d28640 1 --2- 172.21.15.27:0/34654185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48c4100fd0 0x7f48c419a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.047+0000 7f48c9d28640 1 --2- 172.21.15.27:0/34654185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c4100610 0x7f48c419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:26.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.047+0000 7f48c9d28640 1 -- 172.21.15.27:0/34654185 >> 172.21.15.27:0/34654185 conn(0x7f48c4077740 msgr2=0x7f48c40688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:26.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.047+0000 7f48c9d28640 1 -- 172.21.15.27:0/34654185 shutdown_connections 2024-04-25T00:16:26.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:26.047+0000 7f48c9d28640 1 -- 172.21.15.27:0/34654185 wait complete. 2024-04-25T00:16:26.058 INFO:tasks.cephfs.filesystem:scrub status for tag:abef993d-9c17-4d1d-a926-feb6d1cb9852 - {'path': '/', 'tag': 'abef993d-9c17-4d1d-a926-feb6d1cb9852', 'options': 'recursive,force'} 2024-04-25T00:16:26.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:26 smithi027 ceph-mon[29208]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:16:26.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:26 smithi139 ceph-mon[32833]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:16:26.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:26 smithi153 ceph-mon[32896]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:16:27.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:27 smithi027 ceph-mon[29208]: scrub queued for path: / 2024-04-25T00:16:27.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:27 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:16:27.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:27 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:16:27.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:27 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:16:27.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:27 smithi139 ceph-mon[32833]: scrub queued for path: / 2024-04-25T00:16:27.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:27 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:16:27.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:27 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:16:27.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:27 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:16:27.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:27 smithi153 ceph-mon[32896]: scrub queued for path: / 2024-04-25T00:16:27.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:27 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:16:27.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:27 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:16:27.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:27 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:16:28.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:28 smithi027 ceph-mon[29208]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 853 B/s wr, 12 op/s 2024-04-25T00:16:28.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:28 smithi139 ceph-mon[32833]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 853 B/s wr, 12 op/s 2024-04-25T00:16:28.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:28 smithi153 ceph-mon[32896]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 853 B/s wr, 12 op/s 2024-04-25T00:16:29.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:29 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:16:29.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:29 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:16:29.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:29 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:16:30.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:30 smithi027 ceph-mon[29208]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 853 B/s wr, 8 op/s 2024-04-25T00:16:30.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:30 smithi139 ceph-mon[32833]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 853 B/s wr, 8 op/s 2024-04-25T00:16:30.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:30 smithi153 ceph-mon[32896]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 853 B/s wr, 8 op/s 2024-04-25T00:16:32.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:32 smithi027 ceph-mon[29208]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 938 B/s wr, 12 op/s 2024-04-25T00:16:32.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:32 smithi139 ceph-mon[32833]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 938 B/s wr, 12 op/s 2024-04-25T00:16:32.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:32 smithi153 ceph-mon[32896]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 938 B/s wr, 12 op/s 2024-04-25T00:16:34.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:34 smithi027 ceph-mon[29208]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-25T00:16:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:34 smithi139 ceph-mon[32833]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-25T00:16:34.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:34 smithi153 ceph-mon[32896]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-25T00:16:36.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:36 smithi027 ceph-mon[29208]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-25T00:16:36.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:36 smithi139 ceph-mon[32833]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-25T00:16:36.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:36 smithi153 ceph-mon[32896]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-25T00:16:38.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:38 smithi027 ceph-mon[29208]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 938 B/s wr, 13 op/s 2024-04-25T00:16:38.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:38 smithi139 ceph-mon[32833]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 938 B/s wr, 13 op/s 2024-04-25T00:16:38.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:38 smithi153 ceph-mon[32896]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 938 B/s wr, 13 op/s 2024-04-25T00:16:40.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:40 smithi027 ceph-mon[29208]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T00:16:40.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:40 smithi139 ceph-mon[32833]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T00:16:40.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:40 smithi153 ceph-mon[32896]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T00:16:42.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:42 smithi027 ceph-mon[29208]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-25T00:16:42.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:42 smithi139 ceph-mon[32833]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-25T00:16:42.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:42 smithi153 ceph-mon[32896]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-25T00:16:44.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:44 smithi027 ceph-mon[29208]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:16:44.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:44 smithi139 ceph-mon[32833]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:16:44.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:44 smithi153 ceph-mon[32896]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:16:46.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:46 smithi027 ceph-mon[29208]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:16:46.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:46 smithi139 ceph-mon[32833]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:16:46.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:46 smithi153 ceph-mon[32896]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:16:48.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:48 smithi027 ceph-mon[29208]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T00:16:48.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:48 smithi139 ceph-mon[32833]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T00:16:48.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:48 smithi153 ceph-mon[32896]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T00:16:50.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:50 smithi027 ceph-mon[29208]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:16:50.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:50 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:16:50.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:50 smithi139 ceph-mon[32833]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:16:50.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:50 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:16:50.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:50 smithi153 ceph-mon[32896]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:16:50.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:50 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:16:51.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:51 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:51 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:51 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:51 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:51 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:51 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:51 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:16:51.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:51 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:16:51.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:51 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:51 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:51 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:51 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:51 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:51 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:51 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:51 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:16:51.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:51 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:16:51.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:51 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:51 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:51 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:51 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:51 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:51 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:51 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:51.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:51 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:16:51.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:51 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:16:51.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:51 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:16:52.655 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:52 smithi139 ceph-mon[32833]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T00:16:52.678 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:52 smithi153 ceph-mon[32896]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T00:16:52.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:52 smithi027 ceph-mon[29208]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T00:16:54.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:54 smithi027 ceph-mon[29208]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:16:54.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:54 smithi139 ceph-mon[32833]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:16:54.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:54 smithi153 ceph-mon[32896]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:16:56.059 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:16:56.060 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:16:56.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.184+0000 7f4144cbd640 1 -- 172.21.15.27:0/674686972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4140100fd0 msgr2=0x7f414010d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.184+0000 7f4144cbd640 1 --2- 172.21.15.27:0/674686972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4140100fd0 0x7f414010d4e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f4138009cd0 tx=0x7f413802a8e0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.184+0000 7f4144cbd640 1 -- 172.21.15.27:0/674686972 shutdown_connections 2024-04-25T00:16:56.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.184+0000 7f4144cbd640 1 --2- 172.21.15.27:0/674686972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4140100fd0 0x7f414010d4e0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.184+0000 7f4144cbd640 1 --2- 172.21.15.27:0/674686972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4140100610 0x7f4140100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.184+0000 7f4144cbd640 1 --2- 172.21.15.27:0/674686972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41401066b0 0x7f4140106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.184+0000 7f4144cbd640 1 -- 172.21.15.27:0/674686972 >> 172.21.15.27:0/674686972 conn(0x7f4140077740 msgr2=0x7f4140077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.184+0000 7f4144cbd640 1 -- 172.21.15.27:0/674686972 shutdown_connections 2024-04-25T00:16:56.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.184+0000 7f4144cbd640 1 -- 172.21.15.27:0/674686972 wait complete. 2024-04-25T00:16:56.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.185+0000 7f4144cbd640 1 Processor -- start 2024-04-25T00:16:56.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.185+0000 7f4144cbd640 1 -- start start 2024-04-25T00:16:56.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.185+0000 7f4144cbd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4140100610 0x7f4140197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.185+0000 7f4144cbd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4140100fd0 0x7f4140198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.185+0000 7f4144cbd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41401066b0 0x7f414019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.185+0000 7f4144cbd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f41401109f0 con 0x7f41401066b0 2024-04-25T00:16:56.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.186+0000 7f4144cbd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4140110870 con 0x7f4140100fd0 2024-04-25T00:16:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.186+0000 7f4144cbd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4140110b70 con 0x7f4140100610 2024-04-25T00:16:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.186+0000 7f413e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4140100610 0x7f4140197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.186+0000 7f413ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41401066b0 0x7f414019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.186+0000 7f413dd9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4140100fd0 0x7f4140198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.186+0000 7f413e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4140100610 0x7f4140197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:38606/0 (socket says 172.21.15.27:38606) 2024-04-25T00:16:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.186+0000 7f413ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41401066b0 0x7f414019c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50938/0 (socket says 172.21.15.27:50938) 2024-04-25T00:16:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.186+0000 7f413e59c640 1 -- 172.21.15.27:0/2369202842 learned_addr learned my addr 172.21.15.27:0/2369202842 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.186+0000 7f413e59c640 1 -- 172.21.15.27:0/2369202842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4140100fd0 msgr2=0x7f4140198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.186+0000 7f413e59c640 1 --2- 172.21.15.27:0/2369202842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4140100fd0 0x7f4140198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.186+0000 7f413e59c640 1 -- 172.21.15.27:0/2369202842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41401066b0 msgr2=0x7f414019c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.186+0000 7f413e59c640 1 --2- 172.21.15.27:0/2369202842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41401066b0 0x7f414019c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.186+0000 7f413e59c640 1 -- 172.21.15.27:0/2369202842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f414019cf20 con 0x7f4140100610 2024-04-25T00:16:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.186+0000 7f413dd9b640 1 --2- 172.21.15.27:0/2369202842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4140100fd0 0x7f4140198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.186+0000 7f413ed9d640 1 --2- 172.21.15.27:0/2369202842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41401066b0 0x7f414019c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.186+0000 7f413e59c640 1 --2- 172.21.15.27:0/2369202842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4140100610 0x7f4140197f30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f413000d950 tx=0x7f413000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.187+0000 7f412f7fe640 1 -- 172.21.15.27:0/2369202842 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4130017020 con 0x7f4140100610 2024-04-25T00:16:56.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.187+0000 7f412f7fe640 1 -- 172.21.15.27:0/2369202842 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4130009c30 con 0x7f4140100610 2024-04-25T00:16:56.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.187+0000 7f412f7fe640 1 -- 172.21.15.27:0/2369202842 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413000f720 con 0x7f4140100610 2024-04-25T00:16:56.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.187+0000 7f4144cbd640 1 -- 172.21.15.27:0/2369202842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f414019d210 con 0x7f4140100610 2024-04-25T00:16:56.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.187+0000 7f4144cbd640 1 -- 172.21.15.27:0/2369202842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41401a4aa0 con 0x7f4140100610 2024-04-25T00:16:56.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.190+0000 7f412f7fe640 1 -- 172.21.15.27:0/2369202842 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f413001a020 con 0x7f4140100610 2024-04-25T00:16:56.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.190+0000 7f41479d4740 1 Processor -- start 2024-04-25T00:16:56.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.191+0000 7f41479d4740 1 -- start start 2024-04-25T00:16:56.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.191+0000 7f41479d4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cc8510 0x558730cc88f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.191+0000 7f41479d4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cc8e30 0x558730cd74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.191+0000 7f41479d4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cd7a30 0x558730cd9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.191+0000 7f41479d4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558730c2d6b0 con 0x558730cc8510 2024-04-25T00:16:56.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.191+0000 7f41479d4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558730bd5880 con 0x558730cc8e30 2024-04-25T00:16:56.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.191+0000 7f41479d4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558730c29670 con 0x558730cd7a30 2024-04-25T00:16:56.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.191+0000 7f413ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cc8510 0x558730cc88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.191+0000 7f412f7fe640 1 --2- 172.21.15.27:0/2369202842 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f41280a2eb0 0x7f41280a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.191+0000 7f413ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cc8510 0x558730cc88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50948/0 (socket says 172.21.15.27:50948) 2024-04-25T00:16:56.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.191+0000 7f413ed9d640 1 -- 172.21.15.27:0/247656770 learned_addr learned my addr 172.21.15.27:0/247656770 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:56.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.191+0000 7f413dd9b640 1 --2- 172.21.15.27:0/247656770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cd7a30 0x558730cd9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.192+0000 7f412f7fe640 1 -- 172.21.15.27:0/2369202842 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f41300cf8c0 con 0x7f4140100610 2024-04-25T00:16:56.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.192+0000 7f413e59c640 1 --2- 172.21.15.27:0/247656770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cc8e30 0x558730cd74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.192+0000 7f413ed9d640 1 -- 172.21.15.27:0/247656770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cd7a30 msgr2=0x558730cd9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.192+0000 7f413ed9d640 1 --2- 172.21.15.27:0/247656770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cd7a30 0x558730cd9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.192+0000 7f413ed9d640 1 -- 172.21.15.27:0/247656770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cc8e30 msgr2=0x558730cd74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.192+0000 7f413ed9d640 1 --2- 172.21.15.27:0/247656770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cc8e30 0x558730cd74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.192+0000 7f413ed9d640 1 -- 172.21.15.27:0/247656770 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558730a0b7d0 con 0x558730cc8510 2024-04-25T00:16:56.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.192+0000 7f413ed9d640 1 --2- 172.21.15.27:0/247656770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cc8510 0x558730cc88f0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4138035000 tx=0x7f4138009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f69b3991b758294c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.192+0000 7f413ed9d640 1 --2- 172.21.15.27:0/2369202842 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f41280a2eb0 0x7f41280a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.192+0000 7f412cff9640 1 -- 172.21.15.27:0/247656770 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4138048070 con 0x558730cc8510 2024-04-25T00:16:56.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.193+0000 7f412cff9640 1 -- 172.21.15.27:0/247656770 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4138036070 con 0x558730cc8510 2024-04-25T00:16:56.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.193+0000 7f412cff9640 1 -- 172.21.15.27:0/247656770 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4138030800 con 0x558730cc8510 2024-04-25T00:16:56.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.193+0000 7f413ed9d640 1 --2- 172.21.15.27:0/2369202842 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f41280a2eb0 0x7f41280a5370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4138009bf0 tx=0x7f4138004920 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.193+0000 7f41479d4740 1 -- 172.21.15.27:0/247656770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cc8510 msgr2=0x558730cc88f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.193+0000 7f41479d4740 1 --2- 172.21.15.27:0/247656770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cc8510 0x558730cc88f0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4138035000 tx=0x7f4138009a20 comp rx=0 tx=0).stop 2024-04-25T00:16:56.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.194+0000 7f41479d4740 1 -- 172.21.15.27:0/247656770 shutdown_connections 2024-04-25T00:16:56.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.194+0000 7f41479d4740 1 --2- 172.21.15.27:0/247656770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cd7a30 0x558730cd9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.194+0000 7f41479d4740 1 --2- 172.21.15.27:0/247656770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cc8e30 0x558730cd74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.194+0000 7f41479d4740 1 --2- 172.21.15.27:0/247656770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cc8510 0x558730cc88f0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.194+0000 7f41479d4740 1 -- 172.21.15.27:0/247656770 >> 172.21.15.27:0/247656770 conn(0x558730cc4950 msgr2=0x558730cc4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.194+0000 7f41479d4740 1 -- 172.21.15.27:0/247656770 shutdown_connections 2024-04-25T00:16:56.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.194+0000 7f41479d4740 1 -- 172.21.15.27:0/247656770 wait complete. 2024-04-25T00:16:56.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.194+0000 7f41479d4740 1 -- start start 2024-04-25T00:16:56.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.194+0000 7f41479d4740 1 Processor -- start 2024-04-25T00:16:56.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f41479d4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cc8510 0x558730cd0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f41479d4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cc8e30 0x558730cd0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f41479d4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cd7a30 0x558730cd13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f413ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cd7a30 0x558730cd13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f413ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cd7a30 0x558730cd13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50954/0 (socket says 172.21.15.27:50954) 2024-04-25T00:16:56.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f413ed9d640 1 -- 172.21.15.27:0/4200476526 learned_addr learned my addr 172.21.15.27:0/4200476526 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:56.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f41479d4740 1 -- 172.21.15.27:0/4200476526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558730ccb140 con 0x558730cd7a30 2024-04-25T00:16:56.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f413e59c640 1 --2- 172.21.15.27:0/4200476526 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cc8510 0x558730cd0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f41479d4740 1 -- 172.21.15.27:0/4200476526 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558730be8f50 con 0x558730cc8510 2024-04-25T00:16:56.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f41479d4740 1 -- 172.21.15.27:0/4200476526 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558730ccb2c0 con 0x558730cc8e30 2024-04-25T00:16:56.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f413dd9b640 1 --2- 172.21.15.27:0/4200476526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cc8e30 0x558730cd0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f413dd9b640 1 -- 172.21.15.27:0/4200476526 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cc8510 msgr2=0x558730cd0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f413dd9b640 1 --2- 172.21.15.27:0/4200476526 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cc8510 0x558730cd0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f413dd9b640 1 -- 172.21.15.27:0/4200476526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cd7a30 msgr2=0x558730cd13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f413dd9b640 1 --2- 172.21.15.27:0/4200476526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cd7a30 0x558730cd13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f413dd9b640 1 -- 172.21.15.27:0/4200476526 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587309413f0 con 0x558730cc8e30 2024-04-25T00:16:56.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f413ed9d640 1 --2- 172.21.15.27:0/4200476526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cd7a30 0x558730cd13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:56.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.195+0000 7f41479d4740 10 client.0 _ll_register_callbacks cb 0x558730c24000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:56.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.196+0000 7f41479d4740 1 -- 172.21.15.27:0/4200476526 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4138036070 con 0x558730cc8e30 2024-04-25T00:16:56.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.196+0000 7f413dd9b640 1 --2- 172.21.15.27:0/4200476526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cc8e30 0x558730cd0eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4140105cf0 tx=0x7f4134011ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad1a259e6ad48c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.196+0000 7f4125ffb640 10 client.25579 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:16:56.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.196+0000 7f4125ffb640 1 -- 172.21.15.27:0/4200476526 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134004020 con 0x558730cc8e30 2024-04-25T00:16:56.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.196+0000 7f4125ffb640 1 -- 172.21.15.27:0/4200476526 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f41340041c0 con 0x558730cc8e30 2024-04-25T00:16:56.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.196+0000 7f4125ffb640 1 -- 172.21.15.27:0/4200476526 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f41340043e0 con 0x558730cc8e30 2024-04-25T00:16:56.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.196+0000 7f41479d4740 10 client.25579 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:56.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.196+0000 7f41479d4740 1 -- 172.21.15.27:0/4200476526 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({fsmap=29}) -- 0x5587309010f0 con 0x558730cc8e30 2024-04-25T00:16:56.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.197+0000 7f4125ffb640 1 -- 172.21.15.27:0/4200476526 <== mon.1 v2:172.21.15.153:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f41340108f0 con 0x558730cc8e30 2024-04-25T00:16:56.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.197+0000 7f41479d4740 10 client.25579 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:56.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.197+0000 7f41479d4740 10 client.25579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:16:56.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.197+0000 7f41479d4740 20 client.25579 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:56.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.197+0000 7f41479d4740 1 --2- 172.21.15.27:0/4200476526 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x558730ce35b0 0x558730ce5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.197+0000 7f41479d4740 4 client.25579 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:16:56.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.197+0000 7f41479d4740 1 -- 172.21.15.27:0/4200476526 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558730c3abf0 con 0x558730ce35b0 2024-04-25T00:16:56.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.197+0000 7f413e59c640 1 --2- 172.21.15.27:0/4200476526 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x558730ce35b0 0x558730ce5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.197+0000 7f4125ffb640 10 client.25579 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:56.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.197+0000 7f413e59c640 1 --2- 172.21.15.27:0/4200476526 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x558730ce35b0 0x558730ce5a40 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bed718f53bf5a869 server_cookie=48e55d727121b79 in_seq=0 out_seq=0 2024-04-25T00:16:56.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.198+0000 7f4125ffb640 1 -- 172.21.15.27:0/4200476526 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558730c3abf0 con 0x558730ce35b0 2024-04-25T00:16:56.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.198+0000 7f4125ffb640 10 client.25579 handle_command_reply: tid=0 2024-04-25T00:16:56.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.198+0000 7f41479d4740 1 client.25579 shutdown 2024-04-25T00:16:56.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.198+0000 7f41479d4740 1 -- 172.21.15.27:0/4200476526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cc8e30 msgr2=0x558730cd0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.198+0000 7f41479d4740 1 --2- 172.21.15.27:0/4200476526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cc8e30 0x558730cd0eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4140105cf0 tx=0x7f4134011ae0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.198+0000 7f41479d4740 1 -- 172.21.15.27:0/4200476526 shutdown_connections 2024-04-25T00:16:56.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.198+0000 7f41479d4740 1 --2- 172.21.15.27:0/4200476526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cd7a30 0x558730cd13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.198+0000 7f41479d4740 1 --2- 172.21.15.27:0/4200476526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cc8e30 0x558730cd0eb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.198+0000 7f41479d4740 1 --2- 172.21.15.27:0/4200476526 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x558730ce35b0 0x558730ce5a40 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.198+0000 7f41479d4740 1 --2- 172.21.15.27:0/4200476526 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cc8510 0x558730cd0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.198+0000 7f4125ffb640 0 client.25579 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:56.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.198+0000 7f41479d4740 1 -- 172.21.15.27:0/4200476526 >> 172.21.15.27:0/4200476526 conn(0x558730cc4950 msgr2=0x558730cca6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.199+0000 7f41479d4740 1 -- 172.21.15.27:0/4200476526 shutdown_connections 2024-04-25T00:16:56.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.199+0000 7f41479d4740 1 -- 172.21.15.27:0/4200476526 wait complete. 2024-04-25T00:16:56.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.199+0000 7f41479d4740 20 client.25579 trim_cache size 0 max 16384 2024-04-25T00:16:56.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.234+0000 7f41479d4740 1 Processor -- start 2024-04-25T00:16:56.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.234+0000 7f41479d4740 1 -- start start 2024-04-25T00:16:56.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.235+0000 7f41479d4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cc8510 0x558730cd7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.235+0000 7f41479d4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cc8e30 0x558730cd1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.235+0000 7f41479d4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cd7a30 0x558730ceb350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.235+0000 7f41479d4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f41401b50c0 con 0x558730cd7a30 2024-04-25T00:16:56.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.235+0000 7f41479d4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f41401b6c00 con 0x558730cc8510 2024-04-25T00:16:56.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.235+0000 7f41479d4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f41401b5790 con 0x558730cc8e30 2024-04-25T00:16:56.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.235+0000 7f413ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cd7a30 0x558730ceb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.235+0000 7f413ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cd7a30 0x558730ceb350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50962/0 (socket says 172.21.15.27:50962) 2024-04-25T00:16:56.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.235+0000 7f413ed9d640 1 -- 172.21.15.27:0/3294428682 learned_addr learned my addr 172.21.15.27:0/3294428682 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:56.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.235+0000 7f413e59c640 1 --2- 172.21.15.27:0/3294428682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cc8510 0x558730cd7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.235+0000 7f413dd9b640 1 --2- 172.21.15.27:0/3294428682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cc8e30 0x558730cd1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.235+0000 7f413ed9d640 1 -- 172.21.15.27:0/3294428682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cc8e30 msgr2=0x558730cd1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.235+0000 7f413ed9d640 1 --2- 172.21.15.27:0/3294428682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cc8e30 0x558730cd1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.235+0000 7f413ed9d640 1 -- 172.21.15.27:0/3294428682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cc8510 msgr2=0x558730cd7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.235+0000 7f413ed9d640 1 --2- 172.21.15.27:0/3294428682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cc8510 0x558730cd7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.235+0000 7f413ed9d640 1 -- 172.21.15.27:0/3294428682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558730cc8c60 con 0x558730cd7a30 2024-04-25T00:16:56.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.236+0000 7f413ed9d640 1 --2- 172.21.15.27:0/3294428682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cd7a30 0x558730ceb350 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f413803bfa0 tx=0x7f413804f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67741aa8d135a091 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.236+0000 7f4127fff640 1 -- 172.21.15.27:0/3294428682 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4138048040 con 0x558730cd7a30 2024-04-25T00:16:56.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.236+0000 7f4127fff640 1 -- 172.21.15.27:0/3294428682 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4138003e50 con 0x558730cd7a30 2024-04-25T00:16:56.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.236+0000 7f4127fff640 1 -- 172.21.15.27:0/3294428682 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4138004330 con 0x558730cd7a30 2024-04-25T00:16:56.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.237+0000 7f41479d4740 1 -- 172.21.15.27:0/3294428682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cd7a30 msgr2=0x558730ceb350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.237+0000 7f41479d4740 1 --2- 172.21.15.27:0/3294428682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cd7a30 0x558730ceb350 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f413803bfa0 tx=0x7f413804f000 comp rx=0 tx=0).stop 2024-04-25T00:16:56.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.237+0000 7f41479d4740 1 -- 172.21.15.27:0/3294428682 shutdown_connections 2024-04-25T00:16:56.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.237+0000 7f41479d4740 1 --2- 172.21.15.27:0/3294428682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cd7a30 0x558730ceb350 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.237+0000 7f41479d4740 1 --2- 172.21.15.27:0/3294428682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cc8e30 0x558730cd1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.237+0000 7f41479d4740 1 --2- 172.21.15.27:0/3294428682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cc8510 0x558730cd7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.237+0000 7f41479d4740 1 -- 172.21.15.27:0/3294428682 >> 172.21.15.27:0/3294428682 conn(0x558730cc4950 msgr2=0x558730cd0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.237+0000 7f41479d4740 1 -- 172.21.15.27:0/3294428682 shutdown_connections 2024-04-25T00:16:56.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.237+0000 7f41479d4740 1 -- 172.21.15.27:0/3294428682 wait complete. 2024-04-25T00:16:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.238+0000 7f41479d4740 1 -- start start 2024-04-25T00:16:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.238+0000 7f41479d4740 1 Processor -- start 2024-04-25T00:16:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.238+0000 7f41479d4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cc8510 0x558730cdc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.238+0000 7f41479d4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cc8e30 0x558730cdcca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.238+0000 7f41479d4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cd7a30 0x558730d48bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.238+0000 7f413dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cc8e30 0x558730cdcca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.238+0000 7f413dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cc8e30 0x558730cdcca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50964/0 (socket says 172.21.15.27:50964) 2024-04-25T00:16:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.239+0000 7f413dd9b640 1 -- 172.21.15.27:0/3578408026 learned_addr learned my addr 172.21.15.27:0/3578408026 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.239+0000 7f41479d4740 1 -- 172.21.15.27:0/3578408026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f41401b6530 con 0x558730cc8e30 2024-04-25T00:16:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.239+0000 7f41479d4740 1 -- 172.21.15.27:0/3578408026 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f41401b5e60 con 0x558730cd7a30 2024-04-25T00:16:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.239+0000 7f41479d4740 1 -- 172.21.15.27:0/3578408026 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f41401b3080 con 0x558730cc8510 2024-04-25T00:16:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.239+0000 7f413e59c640 1 --2- 172.21.15.27:0/3578408026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cc8510 0x558730cdc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.239+0000 7f41479d4740 10 client.0 _ll_register_callbacks cb 0x7f413400bf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.239+0000 7f413ed9d640 1 --2- 172.21.15.27:0/3578408026 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cd7a30 0x558730d48bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.239+0000 7f413dd9b640 1 -- 172.21.15.27:0/3578408026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cc8510 msgr2=0x558730cdc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.239+0000 7f413dd9b640 1 --2- 172.21.15.27:0/3578408026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cc8510 0x558730cdc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.239+0000 7f413dd9b640 1 -- 172.21.15.27:0/3578408026 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cd7a30 msgr2=0x558730d48bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.239+0000 7f413dd9b640 1 --2- 172.21.15.27:0/3578408026 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cd7a30 0x558730d48bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.239+0000 7f413dd9b640 1 -- 172.21.15.27:0/3578408026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558730d49100 con 0x558730cc8e30 2024-04-25T00:16:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.239+0000 7f413e59c640 1 --2- 172.21.15.27:0/3578408026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cc8510 0x558730cdc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.239+0000 7f413dd9b640 1 --2- 172.21.15.27:0/3578408026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cc8e30 0x558730cdcca0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x558730cd01d0 tx=0x7f41340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=249d08fb70fdf40e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.239+0000 7f4125ffb640 10 client.16002 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.239+0000 7f41479d4740 1 -- 172.21.15.27:0/3578408026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4138003e50 con 0x558730cc8e30 2024-04-25T00:16:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.240+0000 7f4125ffb640 1 -- 172.21.15.27:0/3578408026 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134010e40 con 0x558730cc8e30 2024-04-25T00:16:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.240+0000 7f4125ffb640 1 -- 172.21.15.27:0/3578408026 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f4134024940 con 0x558730cc8e30 2024-04-25T00:16:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.240+0000 7f4125ffb640 1 -- 172.21.15.27:0/3578408026 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4134024b60 con 0x558730cc8e30 2024-04-25T00:16:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.240+0000 7f41479d4740 10 client.16002 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.240+0000 7f41479d4740 1 -- 172.21.15.27:0/3578408026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({fsmap=29}) -- 0x558730d4a170 con 0x558730cc8e30 2024-04-25T00:16:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.240+0000 7f4125ffb640 1 -- 172.21.15.27:0/3578408026 <== mon.0 v2:172.21.15.27:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4134005020 con 0x558730cc8e30 2024-04-25T00:16:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.240+0000 7f41479d4740 10 client.16002 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.240+0000 7f41479d4740 10 client.16002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:16:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.240+0000 7f41479d4740 20 client.16002 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.240+0000 7f41479d4740 1 --2- 172.21.15.27:0/3578408026 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x558730d4a820 0x558730d4cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.240+0000 7f41479d4740 4 client.16002 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:16:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.240+0000 7f41479d4740 1 -- 172.21.15.27:0/3578408026 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558730c34f40 con 0x558730d4a820 2024-04-25T00:16:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.241+0000 7f413e59c640 1 --2- 172.21.15.27:0/3578408026 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x558730d4a820 0x558730d4cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.241+0000 7f4125ffb640 10 client.16002 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.241+0000 7f413e59c640 1 --2- 172.21.15.27:0/3578408026 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x558730d4a820 0x558730d4cc10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ced2fbb9bff1acc server_cookie=2ed157b1a863e121 in_seq=0 out_seq=0 2024-04-25T00:16:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.241+0000 7f4125ffb640 1 -- 172.21.15.27:0/3578408026 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558730c34f40 con 0x558730d4a820 2024-04-25T00:16:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.241+0000 7f4125ffb640 10 client.16002 handle_command_reply: tid=0 2024-04-25T00:16:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.241+0000 7f41479d4740 1 client.16002 shutdown 2024-04-25T00:16:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.241+0000 7f41479d4740 1 -- 172.21.15.27:0/3578408026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cc8e30 msgr2=0x558730cdcca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.241+0000 7f41479d4740 1 --2- 172.21.15.27:0/3578408026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cc8e30 0x558730cdcca0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x558730cd01d0 tx=0x7f41340046c0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.242+0000 7f41479d4740 1 -- 172.21.15.27:0/3578408026 shutdown_connections 2024-04-25T00:16:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.242+0000 7f41479d4740 1 --2- 172.21.15.27:0/3578408026 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x558730d4a820 0x558730d4cc10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.242+0000 7f41479d4740 1 --2- 172.21.15.27:0/3578408026 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558730cd7a30 0x558730d48bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.242+0000 7f4125ffb640 0 client.16002 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:56.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.242+0000 7f41479d4740 1 --2- 172.21.15.27:0/3578408026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558730cc8e30 0x558730cdcca0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.242+0000 7f41479d4740 1 --2- 172.21.15.27:0/3578408026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558730cc8510 0x558730cdc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.242+0000 7f41479d4740 1 -- 172.21.15.27:0/3578408026 >> 172.21.15.27:0/3578408026 conn(0x558730cc4950 msgr2=0x558730cd4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.242+0000 7f41479d4740 1 -- 172.21.15.27:0/3578408026 shutdown_connections 2024-04-25T00:16:56.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.242+0000 7f41479d4740 1 -- 172.21.15.27:0/3578408026 wait complete. 2024-04-25T00:16:56.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.242+0000 7f41479d4740 20 client.16002 trim_cache size 0 max 16384 2024-04-25T00:16:56.247 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:16:56.247 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T00:16:56.247 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T00:16:56.247 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:16:56.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.243+0000 7f4144cbd640 1 -- 172.21.15.27:0/2369202842 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f41280a2eb0 msgr2=0x7f41280a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.243+0000 7f4144cbd640 1 --2- 172.21.15.27:0/2369202842 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f41280a2eb0 0x7f41280a5370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4138009bf0 tx=0x7f4138004920 comp rx=0 tx=0).stop 2024-04-25T00:16:56.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.243+0000 7f4144cbd640 1 -- 172.21.15.27:0/2369202842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4140100610 msgr2=0x7f4140197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.243+0000 7f4144cbd640 1 --2- 172.21.15.27:0/2369202842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4140100610 0x7f4140197f30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f413000d950 tx=0x7f413000de10 comp rx=0 tx=0).stop 2024-04-25T00:16:56.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.243+0000 7f4144cbd640 1 -- 172.21.15.27:0/2369202842 shutdown_connections 2024-04-25T00:16:56.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.243+0000 7f4144cbd640 1 --2- 172.21.15.27:0/2369202842 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f41280a2eb0 0x7f41280a5370 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.243+0000 7f4144cbd640 1 --2- 172.21.15.27:0/2369202842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41401066b0 0x7f414019c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.243+0000 7f4144cbd640 1 --2- 172.21.15.27:0/2369202842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4140100fd0 0x7f4140198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.243+0000 7f4144cbd640 1 --2- 172.21.15.27:0/2369202842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4140100610 0x7f4140197f30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.243+0000 7f4144cbd640 1 -- 172.21.15.27:0/2369202842 >> 172.21.15.27:0/2369202842 conn(0x7f4140077740 msgr2=0x7f41400ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.243+0000 7f4144cbd640 1 -- 172.21.15.27:0/2369202842 shutdown_connections 2024-04-25T00:16:56.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.243+0000 7f4144cbd640 1 -- 172.21.15.27:0/2369202842 wait complete. 2024-04-25T00:16:56.258 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T00:16:56.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:56.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.366+0000 7fd9a543f640 1 -- 172.21.15.27:0/2219214339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a0101720 msgr2=0x7fd9a0101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.366+0000 7fd9a543f640 1 --2- 172.21.15.27:0/2219214339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a0101720 0x7fd9a0101ba0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd988009930 tx=0x7fd98802a420 comp rx=0 tx=0).stop 2024-04-25T00:16:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.366+0000 7fd9a543f640 1 -- 172.21.15.27:0/2219214339 shutdown_connections 2024-04-25T00:16:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.366+0000 7fd9a543f640 1 --2- 172.21.15.27:0/2219214339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9a01020e0 0x7fd9a010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.366+0000 7fd9a543f640 1 --2- 172.21.15.27:0/2219214339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a0101720 0x7fd9a0101ba0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.366+0000 7fd9a543f640 1 --2- 172.21.15.27:0/2219214339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd9a0107730 0x7fd9a0107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.366+0000 7fd9a543f640 1 -- 172.21.15.27:0/2219214339 >> 172.21.15.27:0/2219214339 conn(0x7fd9a00fd1d0 msgr2=0x7fd9a00ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.366+0000 7fd9a543f640 1 -- 172.21.15.27:0/2219214339 shutdown_connections 2024-04-25T00:16:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.366+0000 7fd9a543f640 1 -- 172.21.15.27:0/2219214339 wait complete. 2024-04-25T00:16:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.367+0000 7fd9a543f640 1 Processor -- start 2024-04-25T00:16:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.367+0000 7fd9a543f640 1 -- start start 2024-04-25T00:16:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.367+0000 7fd9a543f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9a0101720 0x7fd9a0073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.367+0000 7fd9a543f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a01020e0 0x7fd9a0073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.367+0000 7fd9a543f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd9a0107730 0x7fd9a00744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.367+0000 7fd9a543f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a0111b90 con 0x7fd9a01020e0 2024-04-25T00:16:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.367+0000 7fd9a543f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a0111a10 con 0x7fd9a0107730 2024-04-25T00:16:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.367+0000 7fd9a543f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a0111d10 con 0x7fd9a0101720 2024-04-25T00:16:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.367+0000 7fd99e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a01020e0 0x7fd9a0073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.368+0000 7fd99e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a01020e0 0x7fd9a0073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50988/0 (socket says 172.21.15.27:50988) 2024-04-25T00:16:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.368+0000 7fd99e7fc640 1 -- 172.21.15.27:0/753112718 learned_addr learned my addr 172.21.15.27:0/753112718 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.368+0000 7fd99effd640 1 --2- 172.21.15.27:0/753112718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9a0101720 0x7fd9a0073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.368+0000 7fd99f7fe640 1 --2- 172.21.15.27:0/753112718 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd9a0107730 0x7fd9a00744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.368+0000 7fd99e7fc640 1 -- 172.21.15.27:0/753112718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9a0101720 msgr2=0x7fd9a0073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.368+0000 7fd99e7fc640 1 --2- 172.21.15.27:0/753112718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9a0101720 0x7fd9a0073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.368+0000 7fd99e7fc640 1 -- 172.21.15.27:0/753112718 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd9a0107730 msgr2=0x7fd9a00744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.368+0000 7fd99e7fc640 1 --2- 172.21.15.27:0/753112718 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd9a0107730 0x7fd9a00744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.368+0000 7fd99e7fc640 1 -- 172.21.15.27:0/753112718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a01a6530 con 0x7fd9a01020e0 2024-04-25T00:16:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.368+0000 7fd99e7fc640 1 --2- 172.21.15.27:0/753112718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a01020e0 0x7fd9a0073f80 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd98802a940 tx=0x7fd988003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.368+0000 7fd997fff640 1 -- 172.21.15.27:0/753112718 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98803f070 con 0x7fd9a01020e0 2024-04-25T00:16:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.368+0000 7fd997fff640 1 -- 172.21.15.27:0/753112718 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd988031c90 con 0x7fd9a01020e0 2024-04-25T00:16:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.368+0000 7fd9a543f640 1 -- 172.21.15.27:0/753112718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9a01a67c0 con 0x7fd9a01020e0 2024-04-25T00:16:56.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.368+0000 7fd997fff640 1 -- 172.21.15.27:0/753112718 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd988035030 con 0x7fd9a01020e0 2024-04-25T00:16:56.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.368+0000 7fd9a543f640 1 -- 172.21.15.27:0/753112718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a01a6cb0 con 0x7fd9a01020e0 2024-04-25T00:16:56.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.370+0000 7fd9a543f640 1 -- 172.21.15.27:0/753112718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9a0068560 con 0x7fd9a01020e0 2024-04-25T00:16:56.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.372+0000 7fd997fff640 1 -- 172.21.15.27:0/753112718 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fd988042070 con 0x7fd9a01020e0 2024-04-25T00:16:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.372+0000 7fd997fff640 1 --2- 172.21.15.27:0/753112718 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd98c0a2eb0 0x7fd98c0a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.373+0000 7fd997fff640 1 -- 172.21.15.27:0/753112718 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fd9880f0840 con 0x7fd9a01020e0 2024-04-25T00:16:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.373+0000 7fd99effd640 1 --2- 172.21.15.27:0/753112718 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd98c0a2eb0 0x7fd98c0a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.373+0000 7fd99effd640 1 --2- 172.21.15.27:0/753112718 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd98c0a2eb0 0x7fd98c0a5370 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd990009b50 tx=0x7fd990005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.376+0000 7fd997fff640 1 -- 172.21.15.27:0/753112718 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd9880bc860 con 0x7fd9a01020e0 2024-04-25T00:16:56.409 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:56 smithi027 ceph-mon[29208]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:16:56.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.659+0000 7fd9a543f640 1 -- 172.21.15.27:0/753112718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9a0109c90 con 0x7fd9a01020e0 2024-04-25T00:16:56.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.660+0000 7fd997fff640 1 -- 172.21.15.27:0/753112718 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11944 (secure 0 0 0) 0x7fd9880c0b60 con 0x7fd9a01020e0 2024-04-25T00:16:56.661 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:16:56.661 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:15:43.587614+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24514,24430,24521,24428,24496]},"id":1}]} 2024-04-25T00:16:56.661 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 29 2024-04-25T00:16:56.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.662+0000 7fd9a543f640 1 -- 172.21.15.27:0/753112718 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd98c0a2eb0 msgr2=0x7fd98c0a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.662+0000 7fd9a543f640 1 --2- 172.21.15.27:0/753112718 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd98c0a2eb0 0x7fd98c0a5370 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd990009b50 tx=0x7fd990005f90 comp rx=0 tx=0).stop 2024-04-25T00:16:56.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.662+0000 7fd9a543f640 1 -- 172.21.15.27:0/753112718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a01020e0 msgr2=0x7fd9a0073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.662+0000 7fd9a543f640 1 --2- 172.21.15.27:0/753112718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a01020e0 0x7fd9a0073f80 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd98802a940 tx=0x7fd988003b90 comp rx=0 tx=0).stop 2024-04-25T00:16:56.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.662+0000 7fd9a543f640 1 -- 172.21.15.27:0/753112718 shutdown_connections 2024-04-25T00:16:56.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.662+0000 7fd9a543f640 1 --2- 172.21.15.27:0/753112718 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd98c0a2eb0 0x7fd98c0a5370 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.662+0000 7fd9a543f640 1 --2- 172.21.15.27:0/753112718 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd9a0107730 0x7fd9a00744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.662+0000 7fd9a543f640 1 --2- 172.21.15.27:0/753112718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a01020e0 0x7fd9a0073f80 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.662+0000 7fd9a543f640 1 --2- 172.21.15.27:0/753112718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9a0101720 0x7fd9a0073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.662+0000 7fd9a543f640 1 -- 172.21.15.27:0/753112718 >> 172.21.15.27:0/753112718 conn(0x7fd9a00fd1d0 msgr2=0x7fd9a00fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.662+0000 7fd9a543f640 1 -- 172.21.15.27:0/753112718 shutdown_connections 2024-04-25T00:16:56.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.662+0000 7fd9a543f640 1 -- 172.21.15.27:0/753112718 wait complete. 2024-04-25T00:16:56.675 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T00:16:56.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:56 smithi139 ceph-mon[32833]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:16:56.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:56 smithi153 ceph-mon[32896]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:16:56.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.780+0000 7f5957922640 1 -- 172.21.15.27:0/2091650929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59501020d0 msgr2=0x7f595010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.780+0000 7f5957922640 1 --2- 172.21.15.27:0/2091650929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59501020d0 0x7f595010e5e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f5948009cd0 tx=0x7f594802a8e0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.780+0000 7f5957922640 1 -- 172.21.15.27:0/2091650929 shutdown_connections 2024-04-25T00:16:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.780+0000 7f5957922640 1 --2- 172.21.15.27:0/2091650929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59501020d0 0x7f595010e5e0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.780+0000 7f5957922640 1 --2- 172.21.15.27:0/2091650929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5950101710 0x7f5950101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.780+0000 7f5957922640 1 --2- 172.21.15.27:0/2091650929 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5950107720 0x7f5950107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.780+0000 7f5957922640 1 -- 172.21.15.27:0/2091650929 >> 172.21.15.27:0/2091650929 conn(0x7f59500fd180 msgr2=0x7f59500ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.781+0000 7f5957922640 1 -- 172.21.15.27:0/2091650929 shutdown_connections 2024-04-25T00:16:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.781+0000 7f5957922640 1 -- 172.21.15.27:0/2091650929 wait complete. 2024-04-25T00:16:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.781+0000 7f5957922640 1 Processor -- start 2024-04-25T00:16:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.781+0000 7f5957922640 1 -- start start 2024-04-25T00:16:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.781+0000 7f5957922640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950101710 0x7f5950198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.781+0000 7f5957922640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59501020d0 0x7f5950198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.781+0000 7f5957922640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5950107720 0x7f595019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f59556be640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950101710 0x7f5950198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f5957922640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5950111b30 con 0x7f5950101710 2024-04-25T00:16:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f5957922640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f59501119b0 con 0x7f5950107720 2024-04-25T00:16:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f5957922640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5950111cb0 con 0x7f59501020d0 2024-04-25T00:16:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f59556be640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950101710 0x7f5950198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51250/0 (socket says 172.21.15.27:51250) 2024-04-25T00:16:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f59556be640 1 -- 172.21.15.27:0/1321507597 learned_addr learned my addr 172.21.15.27:0/1321507597 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f5954ebd640 1 --2- 172.21.15.27:0/1321507597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59501020d0 0x7f5950198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f5955ebf640 1 --2- 172.21.15.27:0/1321507597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5950107720 0x7f595019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f59556be640 1 -- 172.21.15.27:0/1321507597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59501020d0 msgr2=0x7f5950198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f59556be640 1 --2- 172.21.15.27:0/1321507597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59501020d0 0x7f5950198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f59556be640 1 -- 172.21.15.27:0/1321507597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5950107720 msgr2=0x7f595019c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f59556be640 1 --2- 172.21.15.27:0/1321507597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5950107720 0x7f595019c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f59556be640 1 -- 172.21.15.27:0/1321507597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f595019d000 con 0x7f5950101710 2024-04-25T00:16:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f5955ebf640 1 --2- 172.21.15.27:0/1321507597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5950107720 0x7f595019c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f5954ebd640 1 --2- 172.21.15.27:0/1321507597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59501020d0 0x7f5950198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f59556be640 1 --2- 172.21.15.27:0/1321507597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950101710 0x7f5950198010 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f594c00d950 tx=0x7f594c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f59467fc640 1 -- 172.21.15.27:0/1321507597 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594c017070 con 0x7f5950101710 2024-04-25T00:16:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f59467fc640 1 -- 172.21.15.27:0/1321507597 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f594c009c30 con 0x7f5950101710 2024-04-25T00:16:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.782+0000 7f5957922640 1 -- 172.21.15.27:0/1321507597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f595019d2f0 con 0x7f5950101710 2024-04-25T00:16:56.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.783+0000 7f59467fc640 1 -- 172.21.15.27:0/1321507597 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594c00f6b0 con 0x7f5950101710 2024-04-25T00:16:56.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.783+0000 7f5957922640 1 -- 172.21.15.27:0/1321507597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59501a4b80 con 0x7f5950101710 2024-04-25T00:16:56.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.786+0000 7f59467fc640 1 -- 172.21.15.27:0/1321507597 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f594c00f850 con 0x7f5950101710 2024-04-25T00:16:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.787+0000 7f595a634740 1 Processor -- start 2024-04-25T00:16:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.787+0000 7f595a634740 1 -- start start 2024-04-25T00:16:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.787+0000 7f59467fc640 1 --2- 172.21.15.27:0/1321507597 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f593c0a2e10 0x7f593c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.787+0000 7f595a634740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744e580 0x55d31744e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.787+0000 7f595a634740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d31744eea0 0x55d317459030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.787+0000 7f595a634740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 0x55d31745b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.787+0000 7f595a634740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d3173b3670 con 0x55d317459570 2024-04-25T00:16:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.787+0000 7f595a634740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d31735b970 con 0x55d31744e580 2024-04-25T00:16:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.787+0000 7f595a634740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d3173af630 con 0x55d31744eea0 2024-04-25T00:16:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.787+0000 7f59467fc640 1 -- 172.21.15.27:0/1321507597 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f594c0cf250 con 0x7f5950101710 2024-04-25T00:16:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.787+0000 7f5955ebf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 0x55d31745b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.787+0000 7f5955ebf640 1 --2- 172.21.15.27:0/1321507597 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f593c0a2e10 0x7f593c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.787+0000 7f5955ebf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 0x55d31745b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51256/0 (socket says 172.21.15.27:51256) 2024-04-25T00:16:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.787+0000 7f5955ebf640 1 -- 172.21.15.27:0/802678598 learned_addr learned my addr 172.21.15.27:0/802678598 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.787+0000 7f59556be640 1 --2- 172.21.15.27:0/802678598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744e580 0x55d31744e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.787+0000 7f5954ebd640 1 --2- 172.21.15.27:0/802678598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d31744eea0 0x55d317459030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.788+0000 7f5955ebf640 1 -- 172.21.15.27:0/802678598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d31744eea0 msgr2=0x55d317459030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.788+0000 7f5955ebf640 1 --2- 172.21.15.27:0/802678598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d31744eea0 0x55d317459030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.788+0000 7f5955ebf640 1 -- 172.21.15.27:0/802678598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744e580 msgr2=0x55d31744e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.788+0000 7f5955ebf640 1 --2- 172.21.15.27:0/802678598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744e580 0x55d31744e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.788+0000 7f5955ebf640 1 -- 172.21.15.27:0/802678598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d31737f400 con 0x55d317459570 2024-04-25T00:16:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.788+0000 7f5955ebf640 1 --2- 172.21.15.27:0/1321507597 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f593c0a2e10 0x7f593c0a52d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5948009cd0 tx=0x7f5948004a40 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.788+0000 7f5955ebf640 1 --2- 172.21.15.27:0/802678598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 0x55d31745b960 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f5948005cc0 tx=0x7f5948047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf332711d1fc854 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.788+0000 7f593b7fe640 1 -- 172.21.15.27:0/802678598 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594804d070 con 0x55d317459570 2024-04-25T00:16:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.788+0000 7f593b7fe640 1 -- 172.21.15.27:0/802678598 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5948036040 con 0x55d317459570 2024-04-25T00:16:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.788+0000 7f593b7fe640 1 -- 172.21.15.27:0/802678598 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5948030aa0 con 0x55d317459570 2024-04-25T00:16:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.789+0000 7f595a634740 1 -- 172.21.15.27:0/802678598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 msgr2=0x55d31745b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.789+0000 7f595a634740 1 --2- 172.21.15.27:0/802678598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 0x55d31745b960 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f5948005cc0 tx=0x7f5948047000 comp rx=0 tx=0).stop 2024-04-25T00:16:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.789+0000 7f595a634740 1 -- 172.21.15.27:0/802678598 shutdown_connections 2024-04-25T00:16:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.789+0000 7f595a634740 1 --2- 172.21.15.27:0/802678598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 0x55d31745b960 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.789+0000 7f595a634740 1 --2- 172.21.15.27:0/802678598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d31744eea0 0x55d317459030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.789+0000 7f595a634740 1 --2- 172.21.15.27:0/802678598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744e580 0x55d31744e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.789+0000 7f595a634740 1 -- 172.21.15.27:0/802678598 >> 172.21.15.27:0/802678598 conn(0x55d31744a8a0 msgr2=0x55d31744ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.789+0000 7f595a634740 1 -- 172.21.15.27:0/802678598 shutdown_connections 2024-04-25T00:16:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.789+0000 7f595a634740 1 -- 172.21.15.27:0/802678598 wait complete. 2024-04-25T00:16:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.789+0000 7f595a634740 1 -- start start 2024-04-25T00:16:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.789+0000 7f595a634740 1 Processor -- start 2024-04-25T00:16:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.790+0000 7f595a634740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d31744e580 0x55d317458c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.790+0000 7f595a634740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744eea0 0x55d317456160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.790+0000 7f595a634740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 0x55d3174566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.790+0000 7f595a634740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d317460760 con 0x55d317459570 2024-04-25T00:16:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.790+0000 7f595a634740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d31736f050 con 0x55d31744eea0 2024-04-25T00:16:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.790+0000 7f595a634740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d3174608e0 con 0x55d31744e580 2024-04-25T00:16:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.790+0000 7f595a634740 10 client.0 _ll_register_callbacks cb 0x55d3170c6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.790+0000 7f5954ebd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744eea0 0x55d317456160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.790+0000 7f5955ebf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 0x55d3174566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.790+0000 7f5954ebd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744eea0 0x55d317456160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:60254/0 (socket says 172.21.15.27:60254) 2024-04-25T00:16:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.790+0000 7f5954ebd640 1 -- 172.21.15.27:0/2372527775 learned_addr learned my addr 172.21.15.27:0/2372527775 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.790+0000 7f59556be640 1 --2- 172.21.15.27:0/2372527775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d31744e580 0x55d317458c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.791+0000 7f5954ebd640 1 -- 172.21.15.27:0/2372527775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d31744e580 msgr2=0x55d317458c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.791+0000 7f5954ebd640 1 --2- 172.21.15.27:0/2372527775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d31744e580 0x55d317458c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.791+0000 7f5954ebd640 1 -- 172.21.15.27:0/2372527775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 msgr2=0x55d3174566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.791+0000 7f5954ebd640 1 --2- 172.21.15.27:0/2372527775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 0x55d3174566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.791+0000 7f5954ebd640 1 -- 172.21.15.27:0/2372527775 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d317340dc0 con 0x55d31744eea0 2024-04-25T00:16:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.791+0000 7f5955ebf640 1 --2- 172.21.15.27:0/2372527775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 0x55d3174566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.791+0000 7f5954ebd640 1 --2- 172.21.15.27:0/2372527775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744eea0 0x55d317456160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5940012700 tx=0x7f5940012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88258cfb120188c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.791+0000 7f5938ff9640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T00:16:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.791+0000 7f5938ff9640 1 -- 172.21.15.27:0/2372527775 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5940010d40 con 0x55d31744eea0 2024-04-25T00:16:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.791+0000 7f595a634740 1 -- 172.21.15.27:0/2372527775 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5948036040 con 0x55d31744eea0 2024-04-25T00:16:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.791+0000 7f5938ff9640 1 -- 172.21.15.27:0/2372527775 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f5940004020 con 0x55d31744eea0 2024-04-25T00:16:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.791+0000 7f5938ff9640 1 -- 172.21.15.27:0/2372527775 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5940004220 con 0x55d31744eea0 2024-04-25T00:16:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.791+0000 7f595a634740 10 client.25646 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.791+0000 7f595a634740 1 -- 172.21.15.27:0/2372527775 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d3170c73f0 con 0x55d31744eea0 2024-04-25T00:16:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.792+0000 7f5938ff9640 1 -- 172.21.15.27:0/2372527775 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5948036040 con 0x55d31744eea0 2024-04-25T00:16:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.792+0000 7f5938ff9640 1 -- 172.21.15.27:0/2372527775 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5940004b00 con 0x55d31744eea0 2024-04-25T00:16:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.792+0000 7f595a634740 10 client.25646 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.792+0000 7f595a634740 10 client.25646 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:16:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.792+0000 7f595a634740 20 client.25646 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.792+0000 7f595a634740 1 --2- 172.21.15.27:0/2372527775 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55d3174722b0 0x55d3174746f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.792+0000 7f595a634740 4 client.25646 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:16:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.792+0000 7f595a634740 1 -- 172.21.15.27:0/2372527775 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3173c0bb0 con 0x55d3174722b0 2024-04-25T00:16:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.792+0000 7f59556be640 1 --2- 172.21.15.27:0/2372527775 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55d3174722b0 0x55d3174746f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.793+0000 7f59556be640 1 --2- 172.21.15.27:0/2372527775 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55d3174722b0 0x55d3174746f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=643a93bef28b2144 server_cookie=9d6b1aefcfc10664 in_seq=0 out_seq=0 2024-04-25T00:16:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.793+0000 7f5938ff9640 10 client.25646 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:16:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.793+0000 7f5938ff9640 1 -- 172.21.15.27:0/2372527775 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d3173c0bb0 con 0x55d3174722b0 2024-04-25T00:16:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.793+0000 7f5938ff9640 10 client.25646 handle_command_reply: tid=0 2024-04-25T00:16:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.793+0000 7f595a634740 1 client.25646 shutdown 2024-04-25T00:16:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.794+0000 7f595a634740 1 -- 172.21.15.27:0/2372527775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744eea0 msgr2=0x55d317456160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.794+0000 7f595a634740 1 --2- 172.21.15.27:0/2372527775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744eea0 0x55d317456160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5940012700 tx=0x7f5940012bc0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.794+0000 7f595a634740 1 -- 172.21.15.27:0/2372527775 shutdown_connections 2024-04-25T00:16:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.794+0000 7f595a634740 1 --2- 172.21.15.27:0/2372527775 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55d3174722b0 0x55d3174746f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.794+0000 7f595a634740 1 --2- 172.21.15.27:0/2372527775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 0x55d3174566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.794+0000 7f595a634740 1 --2- 172.21.15.27:0/2372527775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744eea0 0x55d317456160 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.794+0000 7f595a634740 1 --2- 172.21.15.27:0/2372527775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d31744e580 0x55d317458c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.794+0000 7f595a634740 1 -- 172.21.15.27:0/2372527775 >> 172.21.15.27:0/2372527775 conn(0x55d31744a8a0 msgr2=0x55d3174506a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.794+0000 7f5938ff9640 0 client.25646 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:16:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.794+0000 7f595a634740 1 -- 172.21.15.27:0/2372527775 shutdown_connections 2024-04-25T00:16:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.794+0000 7f595a634740 1 -- 172.21.15.27:0/2372527775 wait complete. 2024-04-25T00:16:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.794+0000 7f595a634740 20 client.25646 trim_cache size 0 max 16384 2024-04-25T00:16:56.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f595a634740 1 Processor -- start 2024-04-25T00:16:56.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f595a634740 1 -- start start 2024-04-25T00:16:56.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f595a634740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d31744e580 0x55d317456120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f595a634740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744eea0 0x55d3174589a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f595a634740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d317459570 0x55d31747a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f595a634740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f59501ac760 con 0x55d31744e580 2024-04-25T00:16:56.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f595a634740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f59501ae2a0 con 0x55d31744eea0 2024-04-25T00:16:56.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f595a634740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f59501ace30 con 0x55d317459570 2024-04-25T00:16:56.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f59556be640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d31744e580 0x55d317456120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f59556be640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d31744e580 0x55d317456120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51268/0 (socket says 172.21.15.27:51268) 2024-04-25T00:16:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f59556be640 1 -- 172.21.15.27:0/1098352999 learned_addr learned my addr 172.21.15.27:0/1098352999 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f5954ebd640 1 --2- 172.21.15.27:0/1098352999 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744eea0 0x55d3174589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f5955ebf640 1 --2- 172.21.15.27:0/1098352999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d317459570 0x55d31747a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f59556be640 1 -- 172.21.15.27:0/1098352999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d317459570 msgr2=0x55d31747a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f59556be640 1 --2- 172.21.15.27:0/1098352999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d317459570 0x55d31747a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f59556be640 1 -- 172.21.15.27:0/1098352999 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744eea0 msgr2=0x55d3174589a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f59556be640 1 --2- 172.21.15.27:0/1098352999 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744eea0 0x55d3174589a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f59556be640 1 -- 172.21.15.27:0/1098352999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d31744ecd0 con 0x55d31744e580 2024-04-25T00:16:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.808+0000 7f5954ebd640 1 --2- 172.21.15.27:0/1098352999 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744eea0 0x55d3174589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.809+0000 7f59556be640 1 --2- 172.21.15.27:0/1098352999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d31744e580 0x55d317456120 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f594c009610 tx=0x7f594c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6116486e77ba081 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.809+0000 7f593affd640 1 -- 172.21.15.27:0/1098352999 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594c020040 con 0x55d31744e580 2024-04-25T00:16:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.809+0000 7f593affd640 1 -- 172.21.15.27:0/1098352999 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f594c0a3e30 con 0x55d31744e580 2024-04-25T00:16:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.809+0000 7f593affd640 1 -- 172.21.15.27:0/1098352999 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594c0979b0 con 0x55d31744e580 2024-04-25T00:16:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.809+0000 7f595a634740 1 -- 172.21.15.27:0/1098352999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d31744e580 msgr2=0x55d317456120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.809+0000 7f595a634740 1 --2- 172.21.15.27:0/1098352999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d31744e580 0x55d317456120 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f594c009610 tx=0x7f594c005880 comp rx=0 tx=0).stop 2024-04-25T00:16:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.810+0000 7f595a634740 1 -- 172.21.15.27:0/1098352999 shutdown_connections 2024-04-25T00:16:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.810+0000 7f595a634740 1 --2- 172.21.15.27:0/1098352999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d317459570 0x55d31747a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.810+0000 7f595a634740 1 --2- 172.21.15.27:0/1098352999 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744eea0 0x55d3174589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.810+0000 7f595a634740 1 --2- 172.21.15.27:0/1098352999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d31744e580 0x55d317456120 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.810+0000 7f595a634740 1 -- 172.21.15.27:0/1098352999 >> 172.21.15.27:0/1098352999 conn(0x55d31744a8a0 msgr2=0x55d317451860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.810+0000 7f595a634740 1 -- 172.21.15.27:0/1098352999 shutdown_connections 2024-04-25T00:16:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.810+0000 7f595a634740 1 -- 172.21.15.27:0/1098352999 wait complete. 2024-04-25T00:16:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.810+0000 7f595a634740 1 -- start start 2024-04-25T00:16:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.810+0000 7f595a634740 1 Processor -- start 2024-04-25T00:16:56.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.810+0000 7f595a634740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744e580 0x55d31746ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.810+0000 7f595a634740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d31744eea0 0x55d31746a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f595a634740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 0x55d31746a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f595a634740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f59501adbd0 con 0x55d317459570 2024-04-25T00:16:56.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f595a634740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f59501ad500 con 0x55d31744e580 2024-04-25T00:16:56.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f595a634740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f59501aa720 con 0x55d31744eea0 2024-04-25T00:16:56.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f5954ebd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d31744eea0 0x55d31746a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f5955ebf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 0x55d31746a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f5954ebd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d31744eea0 0x55d31746a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:47284/0 (socket says 172.21.15.27:47284) 2024-04-25T00:16:56.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f5954ebd640 1 -- 172.21.15.27:0/1869184426 learned_addr learned my addr 172.21.15.27:0/1869184426 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:56.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f595a634740 10 client.0 _ll_register_callbacks cb 0x7f594000bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:56.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f59556be640 1 --2- 172.21.15.27:0/1869184426 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744e580 0x55d31746ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f5955ebf640 1 -- 172.21.15.27:0/1869184426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d31744eea0 msgr2=0x55d31746a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f5955ebf640 1 --2- 172.21.15.27:0/1869184426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d31744eea0 0x55d31746a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f5955ebf640 1 -- 172.21.15.27:0/1869184426 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744e580 msgr2=0x55d31746ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f5955ebf640 1 --2- 172.21.15.27:0/1869184426 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744e580 0x55d31746ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f5955ebf640 1 -- 172.21.15.27:0/1869184426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3171917d0 con 0x55d317459570 2024-04-25T00:16:56.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f5954ebd640 1 --2- 172.21.15.27:0/1869184426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d31744eea0 0x55d31746a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:56.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f59556be640 1 --2- 172.21.15.27:0/1869184426 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744e580 0x55d31746ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:56.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f5955ebf640 1 --2- 172.21.15.27:0/1869184426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 0x55d31746a5c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f5948009ca0 tx=0x7f5948009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2054f31f50f7c86c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f5938ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:56.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f5938ff9640 1 -- 172.21.15.27:0/1869184426 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594804d050 con 0x55d317459570 2024-04-25T00:16:56.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f5938ff9640 1 -- 172.21.15.27:0/1869184426 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f594802e040 con 0x55d317459570 2024-04-25T00:16:56.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.811+0000 7f595a634740 1 -- 172.21.15.27:0/1869184426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f594c0a3e30 con 0x55d317459570 2024-04-25T00:16:56.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.812+0000 7f5938ff9640 1 -- 172.21.15.27:0/1869184426 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5948031550 con 0x55d317459570 2024-04-25T00:16:56.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.812+0000 7f595a634740 10 client.16026 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:56.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.812+0000 7f595a634740 1 -- 172.21.15.27:0/1869184426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d3174d86a0 con 0x55d317459570 2024-04-25T00:16:56.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.812+0000 7f5938ff9640 1 -- 172.21.15.27:0/1869184426 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f594c0a3e30 con 0x55d317459570 2024-04-25T00:16:56.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.812+0000 7f5938ff9640 1 -- 172.21.15.27:0/1869184426 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5948030070 con 0x55d317459570 2024-04-25T00:16:56.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.812+0000 7f595a634740 10 client.16026 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:56.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.812+0000 7f595a634740 10 client.16026 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:16:56.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.812+0000 7f595a634740 20 client.16026 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:56.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.812+0000 7f595a634740 1 --2- 172.21.15.27:0/1869184426 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55d3174d8980 0x55d3174dad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.813+0000 7f595a634740 4 client.16026 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:16:56.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.813+0000 7f595a634740 1 -- 172.21.15.27:0/1869184426 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d3173baf00 con 0x55d3174d8980 2024-04-25T00:16:56.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.813+0000 7f59556be640 1 --2- 172.21.15.27:0/1869184426 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55d3174d8980 0x55d3174dad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.813+0000 7f5938ff9640 10 client.16026 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:16:56.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.813+0000 7f59556be640 1 --2- 172.21.15.27:0/1869184426 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55d3174d8980 0x55d3174dad70 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=eadde565708b02e4 server_cookie=30e82a50f0c463e1 in_seq=0 out_seq=0 2024-04-25T00:16:56.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.813+0000 7f5938ff9640 1 -- 172.21.15.27:0/1869184426 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d3173baf00 con 0x55d3174d8980 2024-04-25T00:16:56.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.813+0000 7f5938ff9640 10 client.16026 handle_command_reply: tid=0 2024-04-25T00:16:56.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.813+0000 7f595a634740 1 client.16026 shutdown 2024-04-25T00:16:56.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.813+0000 7f595a634740 1 -- 172.21.15.27:0/1869184426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 msgr2=0x55d31746a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.813+0000 7f595a634740 1 --2- 172.21.15.27:0/1869184426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 0x55d31746a5c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f5948009ca0 tx=0x7f5948009b20 comp rx=0 tx=0).stop 2024-04-25T00:16:56.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f595a634740 1 -- 172.21.15.27:0/1869184426 shutdown_connections 2024-04-25T00:16:56.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f595a634740 1 --2- 172.21.15.27:0/1869184426 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55d3174d8980 0x55d3174dad70 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f595a634740 1 --2- 172.21.15.27:0/1869184426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d317459570 0x55d31746a5c0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f595a634740 1 --2- 172.21.15.27:0/1869184426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d31744eea0 0x55d31746a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f595a634740 1 --2- 172.21.15.27:0/1869184426 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d31744e580 0x55d31746ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f595a634740 1 -- 172.21.15.27:0/1869184426 >> 172.21.15.27:0/1869184426 conn(0x55d31744a8a0 msgr2=0x55d31745dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f5938ff9640 0 client.16026 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:16:56.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f595a634740 1 -- 172.21.15.27:0/1869184426 shutdown_connections 2024-04-25T00:16:56.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f595a634740 1 -- 172.21.15.27:0/1869184426 wait complete. 2024-04-25T00:16:56.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f595a634740 20 client.16026 trim_cache size 0 max 16384 2024-04-25T00:16:56.821 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:16:56.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f5957922640 1 -- 172.21.15.27:0/1321507597 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f593c0a2e10 msgr2=0x7f593c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f5957922640 1 --2- 172.21.15.27:0/1321507597 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f593c0a2e10 0x7f593c0a52d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5948009cd0 tx=0x7f5948004a40 comp rx=0 tx=0).stop 2024-04-25T00:16:56.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f5957922640 1 -- 172.21.15.27:0/1321507597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950101710 msgr2=0x7f5950198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f5957922640 1 --2- 172.21.15.27:0/1321507597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950101710 0x7f5950198010 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f594c00d950 tx=0x7f594c00de10 comp rx=0 tx=0).stop 2024-04-25T00:16:56.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f5957922640 1 -- 172.21.15.27:0/1321507597 shutdown_connections 2024-04-25T00:16:56.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f5957922640 1 --2- 172.21.15.27:0/1321507597 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f593c0a2e10 0x7f593c0a52d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f5957922640 1 --2- 172.21.15.27:0/1321507597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5950107720 0x7f595019c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f5957922640 1 --2- 172.21.15.27:0/1321507597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59501020d0 0x7f5950198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.814+0000 7f5957922640 1 --2- 172.21.15.27:0/1321507597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950101710 0x7f5950198010 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.815+0000 7f5957922640 1 -- 172.21.15.27:0/1321507597 >> 172.21.15.27:0/1321507597 conn(0x7f59500fd180 msgr2=0x7f59500fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.815+0000 7f5957922640 1 -- 172.21.15.27:0/1321507597 shutdown_connections 2024-04-25T00:16:56.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.815+0000 7f5957922640 1 -- 172.21.15.27:0/1321507597 wait complete. 2024-04-25T00:16:56.827 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T00:16:56.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.933+0000 7f9bf35b0640 1 -- 172.21.15.27:0/1401409658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bec1020e0 msgr2=0x7f9bec10e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.933+0000 7f9bf35b0640 1 --2- 172.21.15.27:0/1401409658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bec1020e0 0x7f9bec10e5f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009cd0 tx=0x7f9be402a8e0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.933+0000 7f9bf35b0640 1 -- 172.21.15.27:0/1401409658 shutdown_connections 2024-04-25T00:16:56.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.933+0000 7f9bf35b0640 1 --2- 172.21.15.27:0/1401409658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bec1020e0 0x7f9bec10e5f0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.933+0000 7f9bf35b0640 1 --2- 172.21.15.27:0/1401409658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bec101720 0x7f9bec101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.933+0000 7f9bf35b0640 1 --2- 172.21.15.27:0/1401409658 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9bec107730 0x7f9bec107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.933+0000 7f9bf35b0640 1 -- 172.21.15.27:0/1401409658 >> 172.21.15.27:0/1401409658 conn(0x7f9bec078b40 msgr2=0x7f9bec078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.934+0000 7f9bf35b0640 1 -- 172.21.15.27:0/1401409658 shutdown_connections 2024-04-25T00:16:56.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.934+0000 7f9bf35b0640 1 -- 172.21.15.27:0/1401409658 wait complete. 2024-04-25T00:16:56.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.934+0000 7f9bf35b0640 1 Processor -- start 2024-04-25T00:16:56.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.934+0000 7f9bf35b0640 1 -- start start 2024-04-25T00:16:56.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.934+0000 7f9bf35b0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bec101720 0x7f9bec197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.934+0000 7f9bf35b0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bec1020e0 0x7f9bec198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.935+0000 7f9bf35b0640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9bec107730 0x7f9bec19c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.935+0000 7f9bf35b0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec111b90 con 0x7f9bec101720 2024-04-25T00:16:56.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.935+0000 7f9bf35b0640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec111a10 con 0x7f9bec107730 2024-04-25T00:16:56.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.935+0000 7f9bf35b0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec111d10 con 0x7f9bec1020e0 2024-04-25T00:16:56.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.935+0000 7f9bf134c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bec101720 0x7f9bec197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.935+0000 7f9bf134c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bec101720 0x7f9bec197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51288/0 (socket says 172.21.15.27:51288) 2024-04-25T00:16:56.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.935+0000 7f9bf134c640 1 -- 172.21.15.27:0/950531315 learned_addr learned my addr 172.21.15.27:0/950531315 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:56.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.935+0000 7f9bf1b4d640 1 --2- 172.21.15.27:0/950531315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9bec107730 0x7f9bec19c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.935+0000 7f9bf0b4b640 1 --2- 172.21.15.27:0/950531315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bec1020e0 0x7f9bec198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.935+0000 7f9bf134c640 1 -- 172.21.15.27:0/950531315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bec1020e0 msgr2=0x7f9bec198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.935+0000 7f9bf134c640 1 --2- 172.21.15.27:0/950531315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bec1020e0 0x7f9bec198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.935+0000 7f9bf134c640 1 -- 172.21.15.27:0/950531315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9bec107730 msgr2=0x7f9bec19c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.935+0000 7f9bf134c640 1 --2- 172.21.15.27:0/950531315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9bec107730 0x7f9bec19c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.935+0000 7f9bf134c640 1 -- 172.21.15.27:0/950531315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bec19cfe0 con 0x7f9bec101720 2024-04-25T00:16:56.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.935+0000 7f9bf134c640 1 --2- 172.21.15.27:0/950531315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bec101720 0x7f9bec197ff0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9be800e9f0 tx=0x7f9be800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.936+0000 7f9be27fc640 1 -- 172.21.15.27:0/950531315 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be800cd80 con 0x7f9bec101720 2024-04-25T00:16:56.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.936+0000 7f9be27fc640 1 -- 172.21.15.27:0/950531315 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9be8010910 con 0x7f9bec101720 2024-04-25T00:16:56.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.936+0000 7f9bf35b0640 1 -- 172.21.15.27:0/950531315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bec19d2d0 con 0x7f9bec101720 2024-04-25T00:16:56.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.936+0000 7f9bf35b0640 1 -- 172.21.15.27:0/950531315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bec073970 con 0x7f9bec101720 2024-04-25T00:16:56.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.938+0000 7f9be27fc640 1 -- 172.21.15.27:0/950531315 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be8018990 con 0x7f9bec101720 2024-04-25T00:16:56.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.939+0000 7f9bf639d740 1 Processor -- start 2024-04-25T00:16:56.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.939+0000 7f9bf639d740 1 -- start start 2024-04-25T00:16:56.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.939+0000 7f9bf639d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6c3580 0x55b4fa6c3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.939+0000 7f9bf639d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6c3ea0 0x55b4fa6ce030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.939+0000 7f9bf639d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6ce570 0x55b4fa6d0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.939+0000 7f9bf639d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b4fa628670 con 0x55b4fa6c3580 2024-04-25T00:16:56.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.939+0000 7f9bf639d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b4fa5d0970 con 0x55b4fa6c3ea0 2024-04-25T00:16:56.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.939+0000 7f9bf639d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b4fa624630 con 0x55b4fa6ce570 2024-04-25T00:16:56.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.940+0000 7f9be27fc640 1 -- 172.21.15.27:0/950531315 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f9be8018bf0 con 0x7f9bec101720 2024-04-25T00:16:56.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.940+0000 7f9bf1b4d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6c3580 0x55b4fa6c3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.940+0000 7f9bf1b4d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6c3580 0x55b4fa6c3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51302/0 (socket says 172.21.15.27:51302) 2024-04-25T00:16:56.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.940+0000 7f9bf1b4d640 1 -- 172.21.15.27:0/3483471399 learned_addr learned my addr 172.21.15.27:0/3483471399 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:56.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.940+0000 7f9bf0b4b640 1 --2- 172.21.15.27:0/3483471399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6ce570 0x55b4fa6d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.940+0000 7f9bf1b4d640 1 -- 172.21.15.27:0/3483471399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6ce570 msgr2=0x55b4fa6d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.940+0000 7f9bf1b4d640 1 --2- 172.21.15.27:0/3483471399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6ce570 0x55b4fa6d0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.940+0000 7f9bf1b4d640 1 -- 172.21.15.27:0/3483471399 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6c3ea0 msgr2=0x55b4fa6ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.940+0000 7f9bf1b4d640 1 --2- 172.21.15.27:0/3483471399 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6c3ea0 0x55b4fa6ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.940+0000 7f9bf1b4d640 1 -- 172.21.15.27:0/3483471399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4fa5f4400 con 0x55b4fa6c3580 2024-04-25T00:16:56.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.940+0000 7f9bf1b4d640 1 --2- 172.21.15.27:0/3483471399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6c3580 0x55b4fa6c3960 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009cd0 tx=0x7f9be4009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ec05ccff3831c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.940+0000 7f9bd77fe640 1 -- 172.21.15.27:0/3483471399 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be4049070 con 0x55b4fa6c3580 2024-04-25T00:16:56.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.941+0000 7f9bd77fe640 1 -- 172.21.15.27:0/3483471399 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9be4036040 con 0x55b4fa6c3580 2024-04-25T00:16:56.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.941+0000 7f9bd77fe640 1 -- 172.21.15.27:0/3483471399 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be4004040 con 0x55b4fa6c3580 2024-04-25T00:16:56.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.941+0000 7f9be27fc640 1 --2- 172.21.15.27:0/950531315 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9bd80a2f30 0x7f9bd80a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.941+0000 7f9bf134c640 1 --2- 172.21.15.27:0/950531315 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9bd80a2f30 0x7f9bd80a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.941+0000 7f9bf639d740 1 -- 172.21.15.27:0/3483471399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6c3580 msgr2=0x55b4fa6c3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.941+0000 7f9bf639d740 1 --2- 172.21.15.27:0/3483471399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6c3580 0x55b4fa6c3960 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009cd0 tx=0x7f9be4009a20 comp rx=0 tx=0).stop 2024-04-25T00:16:56.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.941+0000 7f9be27fc640 1 -- 172.21.15.27:0/950531315 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f9be80cfd40 con 0x7f9bec101720 2024-04-25T00:16:56.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.941+0000 7f9bf639d740 1 -- 172.21.15.27:0/3483471399 shutdown_connections 2024-04-25T00:16:56.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.941+0000 7f9bf639d740 1 --2- 172.21.15.27:0/3483471399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6ce570 0x55b4fa6d0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.941+0000 7f9bf639d740 1 --2- 172.21.15.27:0/3483471399 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6c3ea0 0x55b4fa6ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.941+0000 7f9bf639d740 1 --2- 172.21.15.27:0/3483471399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6c3580 0x55b4fa6c3960 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.941+0000 7f9bf639d740 1 -- 172.21.15.27:0/3483471399 >> 172.21.15.27:0/3483471399 conn(0x55b4fa6bf8a0 msgr2=0x55b4fa6bfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.942+0000 7f9bf134c640 1 --2- 172.21.15.27:0/950531315 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9bd80a2f30 0x7f9bd80a53f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9be80ccf40 tx=0x7f9be8097f50 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.942+0000 7f9bf639d740 1 -- 172.21.15.27:0/3483471399 shutdown_connections 2024-04-25T00:16:56.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.942+0000 7f9bf639d740 1 -- 172.21.15.27:0/3483471399 wait complete. 2024-04-25T00:16:56.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.942+0000 7f9bf639d740 1 -- start start 2024-04-25T00:16:56.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.942+0000 7f9bf639d740 1 Processor -- start 2024-04-25T00:16:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.943+0000 7f9bf639d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6c3580 0x55b4fa6cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.943+0000 7f9bf639d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6c3ea0 0x55b4fa6cb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.943+0000 7f9bf639d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6ce570 0x55b4fa6cb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.943+0000 7f9bf639d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b4fa6d1300 con 0x55b4fa6ce570 2024-04-25T00:16:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.943+0000 7f9bf639d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b4fa5e4050 con 0x55b4fa6c3ea0 2024-04-25T00:16:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.943+0000 7f9bf639d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b4fa6d1480 con 0x55b4fa6c3580 2024-04-25T00:16:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.943+0000 7f9bf639d740 10 client.0 _ll_register_callbacks cb 0x55b4fa33bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.943+0000 7f9bf1b4d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6ce570 0x55b4fa6cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.943+0000 7f9bf1b4d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6ce570 0x55b4fa6cb830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51314/0 (socket says 172.21.15.27:51314) 2024-04-25T00:16:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.943+0000 7f9bf1b4d640 1 -- 172.21.15.27:0/2668301972 learned_addr learned my addr 172.21.15.27:0/2668301972 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.943+0000 7f9bf1b4d640 1 --2- 172.21.15.27:0/2668301972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6c3580 0x55b4fa6cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.943+0000 7f9bf0b4b640 1 --2- 172.21.15.27:0/2668301972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6c3ea0 0x55b4fa6cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.944+0000 7f9bf1b4d640 1 -- 172.21.15.27:0/2668301972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6c3580 msgr2=0x55b4fa6cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.944+0000 7f9bf1b4d640 1 --2- 172.21.15.27:0/2668301972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6c3580 0x55b4fa6cdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.944+0000 7f9bf1b4d640 1 -- 172.21.15.27:0/2668301972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6c3ea0 msgr2=0x55b4fa6cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.944+0000 7f9bf1b4d640 1 --2- 172.21.15.27:0/2668301972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6c3ea0 0x55b4fa6cb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.944+0000 7f9bf1b4d640 1 -- 172.21.15.27:0/2668301972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4fa5b5dc0 con 0x55b4fa6ce570 2024-04-25T00:16:56.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.944+0000 7f9bf1b4d640 1 --2- 172.21.15.27:0/2668301972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6ce570 0x55b4fa6cb830 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009620 tx=0x7f9be4003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39554f48ba3c0436 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.944+0000 7f9bd4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:56.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.944+0000 7f9bd4ff9640 1 -- 172.21.15.27:0/2668301972 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be4049070 con 0x55b4fa6ce570 2024-04-25T00:16:56.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.944+0000 7f9bd4ff9640 1 -- 172.21.15.27:0/2668301972 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f9be40041a0 con 0x55b4fa6ce570 2024-04-25T00:16:56.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.944+0000 7f9bf639d740 1 -- 172.21.15.27:0/2668301972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9be4036040 con 0x55b4fa6ce570 2024-04-25T00:16:56.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.944+0000 7f9bd4ff9640 1 -- 172.21.15.27:0/2668301972 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9be4031d00 con 0x55b4fa6ce570 2024-04-25T00:16:56.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.944+0000 7f9bf639d740 10 client.16038 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:56.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.944+0000 7f9bf639d740 1 -- 172.21.15.27:0/2668301972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b4fa33c3f0 con 0x55b4fa6ce570 2024-04-25T00:16:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.945+0000 7f9bd4ff9640 1 -- 172.21.15.27:0/2668301972 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9be404a040 con 0x55b4fa6ce570 2024-04-25T00:16:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.945+0000 7f9bd4ff9640 1 -- 172.21.15.27:0/2668301972 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9be4030070 con 0x55b4fa6ce570 2024-04-25T00:16:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.945+0000 7f9bf639d740 10 client.16038 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.945+0000 7f9bf639d740 10 client.16038 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:16:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.945+0000 7f9bf639d740 20 client.16038 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.945+0000 7f9bf639d740 1 --2- 172.21.15.27:0/2668301972 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55b4fa6de480 0x55b4fa6e0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.945+0000 7f9bf639d740 4 client.16038 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:16:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.945+0000 7f9bf639d740 1 -- 172.21.15.27:0/2668301972 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9be4036040 con 0x55b4fa6de480 2024-04-25T00:16:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.945+0000 7f9bf0b4b640 1 --2- 172.21.15.27:0/2668301972 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55b4fa6de480 0x55b4fa6e0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.946+0000 7f9bd4ff9640 10 client.16038 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:16:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.946+0000 7f9bf0b4b640 1 --2- 172.21.15.27:0/2668301972 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55b4fa6de480 0x55b4fa6e0870 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3f4d6602e793ca19 server_cookie=3c1fc4d422e863e6 in_seq=0 out_seq=0 2024-04-25T00:16:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.947+0000 7f9bd4ff9640 1 -- 172.21.15.27:0/2668301972 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f9be4036040 con 0x55b4fa6de480 2024-04-25T00:16:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.947+0000 7f9bd4ff9640 10 client.16038 handle_command_reply: tid=0 2024-04-25T00:16:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.947+0000 7f9bf639d740 1 client.16038 shutdown 2024-04-25T00:16:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.947+0000 7f9bf639d740 1 -- 172.21.15.27:0/2668301972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6ce570 msgr2=0x55b4fa6cb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.947+0000 7f9bf639d740 1 --2- 172.21.15.27:0/2668301972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6ce570 0x55b4fa6cb830 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009620 tx=0x7f9be4003c20 comp rx=0 tx=0).stop 2024-04-25T00:16:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.947+0000 7f9bf639d740 1 -- 172.21.15.27:0/2668301972 shutdown_connections 2024-04-25T00:16:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.947+0000 7f9bf639d740 1 --2- 172.21.15.27:0/2668301972 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55b4fa6de480 0x55b4fa6e0870 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.947+0000 7f9bd4ff9640 0 client.16038 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:16:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.947+0000 7f9bf639d740 1 --2- 172.21.15.27:0/2668301972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6ce570 0x55b4fa6cb830 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.947+0000 7f9bf639d740 1 --2- 172.21.15.27:0/2668301972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6c3ea0 0x55b4fa6cb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.947+0000 7f9bf639d740 1 --2- 172.21.15.27:0/2668301972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6c3580 0x55b4fa6cdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.947+0000 7f9bf639d740 1 -- 172.21.15.27:0/2668301972 >> 172.21.15.27:0/2668301972 conn(0x55b4fa6bf8a0 msgr2=0x55b4fa6beb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.947+0000 7f9bf639d740 1 -- 172.21.15.27:0/2668301972 shutdown_connections 2024-04-25T00:16:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.947+0000 7f9bf639d740 1 -- 172.21.15.27:0/2668301972 wait complete. 2024-04-25T00:16:56.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.947+0000 7f9bf639d740 20 client.16038 trim_cache size 0 max 16384 2024-04-25T00:16:56.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.961+0000 7f9bf639d740 1 Processor -- start 2024-04-25T00:16:56.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.961+0000 7f9bf639d740 1 -- start start 2024-04-25T00:16:56.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.961+0000 7f9bf639d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6c3580 0x55b4fa6cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.961+0000 7f9bf639d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6c3ea0 0x55b4fa6e3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.961+0000 7f9bf639d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6ce570 0x55b4fa6e3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.961+0000 7f9bf639d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec1b0df0 con 0x55b4fa6ce570 2024-04-25T00:16:56.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.961+0000 7f9bf639d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec1b2930 con 0x55b4fa6c3ea0 2024-04-25T00:16:56.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.961+0000 7f9bf639d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec1b14c0 con 0x55b4fa6c3580 2024-04-25T00:16:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.961+0000 7f9bf1b4d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6ce570 0x55b4fa6e3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.961+0000 7f9bf1b4d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6ce570 0x55b4fa6e3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51328/0 (socket says 172.21.15.27:51328) 2024-04-25T00:16:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.961+0000 7f9bf1b4d640 1 -- 172.21.15.27:0/456071311 learned_addr learned my addr 172.21.15.27:0/456071311 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.961+0000 7f9bf1b4d640 1 --2- 172.21.15.27:0/456071311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6c3580 0x55b4fa6cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.962+0000 7f9bf0b4b640 1 --2- 172.21.15.27:0/456071311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6c3ea0 0x55b4fa6e3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.962+0000 7f9bf1b4d640 1 -- 172.21.15.27:0/456071311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6c3580 msgr2=0x55b4fa6cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.962+0000 7f9bf1b4d640 1 --2- 172.21.15.27:0/456071311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6c3580 0x55b4fa6cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.962+0000 7f9bf1b4d640 1 -- 172.21.15.27:0/456071311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6c3ea0 msgr2=0x55b4fa6e3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.962+0000 7f9bf1b4d640 1 --2- 172.21.15.27:0/456071311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6c3ea0 0x55b4fa6e3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.962+0000 7f9bf1b4d640 1 -- 172.21.15.27:0/456071311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4fa4067d0 con 0x55b4fa6ce570 2024-04-25T00:16:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.962+0000 7f9bf1b4d640 1 --2- 172.21.15.27:0/456071311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6ce570 0x55b4fa6e3dd0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f9be4002230 tx=0x7f9be402a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae4c82c189475db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.962+0000 7f9bd6ffd640 1 -- 172.21.15.27:0/456071311 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be4049070 con 0x55b4fa6ce570 2024-04-25T00:16:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.962+0000 7f9bd6ffd640 1 -- 172.21.15.27:0/456071311 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9be4036040 con 0x55b4fa6ce570 2024-04-25T00:16:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.962+0000 7f9bd6ffd640 1 -- 172.21.15.27:0/456071311 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be403f040 con 0x55b4fa6ce570 2024-04-25T00:16:56.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.963+0000 7f9bf639d740 1 -- 172.21.15.27:0/456071311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6ce570 msgr2=0x55b4fa6e3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.963+0000 7f9bf639d740 1 --2- 172.21.15.27:0/456071311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6ce570 0x55b4fa6e3dd0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f9be4002230 tx=0x7f9be402a8e0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.963+0000 7f9bf639d740 1 -- 172.21.15.27:0/456071311 shutdown_connections 2024-04-25T00:16:56.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.963+0000 7f9bf639d740 1 --2- 172.21.15.27:0/456071311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6ce570 0x55b4fa6e3dd0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.963+0000 7f9bf639d740 1 --2- 172.21.15.27:0/456071311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6c3ea0 0x55b4fa6e3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.963+0000 7f9bf639d740 1 --2- 172.21.15.27:0/456071311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6c3580 0x55b4fa6cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.963+0000 7f9bf639d740 1 -- 172.21.15.27:0/456071311 >> 172.21.15.27:0/456071311 conn(0x55b4fa6bf8a0 msgr2=0x55b4fa6d9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.963+0000 7f9bf639d740 1 -- 172.21.15.27:0/456071311 shutdown_connections 2024-04-25T00:16:56.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.963+0000 7f9bf639d740 1 -- 172.21.15.27:0/456071311 wait complete. 2024-04-25T00:16:56.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.964+0000 7f9bf639d740 1 -- start start 2024-04-25T00:16:56.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.964+0000 7f9bf639d740 1 Processor -- start 2024-04-25T00:16:56.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.964+0000 7f9bf639d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6c3580 0x55b4fa6e1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.964+0000 7f9bf639d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6c3ea0 0x55b4fa6e19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.964+0000 7f9bf639d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6ce570 0x55b4fa6e1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.964+0000 7f9bf639d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec1b2260 con 0x55b4fa6c3580 2024-04-25T00:16:56.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.964+0000 7f9bf639d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec1b1b90 con 0x55b4fa6ce570 2024-04-25T00:16:56.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.964+0000 7f9bf639d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec1aedb0 con 0x55b4fa6c3ea0 2024-04-25T00:16:56.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.964+0000 7f9bf639d740 10 client.0 _ll_register_callbacks cb 0x7f9be4047070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:56.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.965+0000 7f9bf1b4d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6c3580 0x55b4fa6e1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.965+0000 7f9bf0b4b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6c3ea0 0x55b4fa6e19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.965+0000 7f9bf1b4d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6c3580 0x55b4fa6e1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51338/0 (socket says 172.21.15.27:51338) 2024-04-25T00:16:56.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.965+0000 7f9bf1b4d640 1 -- 172.21.15.27:0/1198266612 learned_addr learned my addr 172.21.15.27:0/1198266612 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:56.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.965+0000 7f9bf1b4d640 1 --2- 172.21.15.27:0/1198266612 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6ce570 0x55b4fa6e1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.965+0000 7f9bf1b4d640 1 -- 172.21.15.27:0/1198266612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6c3ea0 msgr2=0x55b4fa6e19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.965+0000 7f9bf1b4d640 1 --2- 172.21.15.27:0/1198266612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6c3ea0 0x55b4fa6e19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.965+0000 7f9bf1b4d640 1 -- 172.21.15.27:0/1198266612 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6ce570 msgr2=0x55b4fa6e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:56.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.965+0000 7f9bf1b4d640 1 --2- 172.21.15.27:0/1198266612 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6ce570 0x55b4fa6e1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.965+0000 7f9bf1b4d640 1 -- 172.21.15.27:0/1198266612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4fa7438b0 con 0x55b4fa6c3580 2024-04-25T00:16:56.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.965+0000 7f9bf0b4b640 1 --2- 172.21.15.27:0/1198266612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6c3ea0 0x55b4fa6e19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.965+0000 7f9bf1b4d640 1 --2- 172.21.15.27:0/1198266612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6c3580 0x55b4fa6e1490 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9be402ae00 tx=0x7f9be4030ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=755276671c6eaf5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.965+0000 7f9bd4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.965+0000 7f9bd4ff9640 1 -- 172.21.15.27:0/1198266612 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be4066060 con 0x55b4fa6c3580 2024-04-25T00:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.965+0000 7f9bd4ff9640 1 -- 172.21.15.27:0/1198266612 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f9be4030d90 con 0x55b4fa6c3580 2024-04-25T00:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.965+0000 7f9bf639d740 1 -- 172.21.15.27:0/1198266612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9be4036040 con 0x55b4fa6c3580 2024-04-25T00:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.966+0000 7f9bd4ff9640 1 -- 172.21.15.27:0/1198266612 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9be4049030 con 0x55b4fa6c3580 2024-04-25T00:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.966+0000 7f9bf639d740 10 client.16044 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.966+0000 7f9bf639d740 1 -- 172.21.15.27:0/1198266612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b4fa744b70 con 0x55b4fa6c3580 2024-04-25T00:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.966+0000 7f9bd4ff9640 1 -- 172.21.15.27:0/1198266612 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9be404a040 con 0x55b4fa6c3580 2024-04-25T00:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.966+0000 7f9bd4ff9640 1 -- 172.21.15.27:0/1198266612 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9be405f070 con 0x55b4fa6c3580 2024-04-25T00:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.966+0000 7f9bf639d740 10 client.16044 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.966+0000 7f9bf639d740 10 client.16044 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.966+0000 7f9bf639d740 20 client.16044 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.966+0000 7f9bf639d740 1 --2- 172.21.15.27:0/1198266612 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55b4fa6de480 0x55b4fa747190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.966+0000 7f9bf639d740 4 client.16044 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.966+0000 7f9bf639d740 1 -- 172.21.15.27:0/1198266612 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f9be4036040 con 0x55b4fa6de480 2024-04-25T00:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.967+0000 7f9bf0b4b640 1 --2- 172.21.15.27:0/1198266612 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55b4fa6de480 0x55b4fa747190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.967+0000 7f9bd4ff9640 10 client.16044 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.967+0000 7f9bf0b4b640 1 --2- 172.21.15.27:0/1198266612 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55b4fa6de480 0x55b4fa747190 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ff4b6d3ce4b7a2a3 server_cookie=35fc1ab31c8a0b6a in_seq=0 out_seq=0 2024-04-25T00:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.967+0000 7f9bd4ff9640 1 -- 172.21.15.27:0/1198266612 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f9be4036040 con 0x55b4fa6de480 2024-04-25T00:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.967+0000 7f9bd4ff9640 10 client.16044 handle_command_reply: tid=0 2024-04-25T00:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.967+0000 7f9bf639d740 1 client.16044 shutdown 2024-04-25T00:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.967+0000 7f9bf639d740 1 -- 172.21.15.27:0/1198266612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6c3580 msgr2=0x55b4fa6e1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.967+0000 7f9bf639d740 1 --2- 172.21.15.27:0/1198266612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6c3580 0x55b4fa6e1490 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9be402ae00 tx=0x7f9be4030ba0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.967+0000 7f9bf639d740 1 -- 172.21.15.27:0/1198266612 shutdown_connections 2024-04-25T00:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.967+0000 7f9bf639d740 1 --2- 172.21.15.27:0/1198266612 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55b4fa6de480 0x55b4fa747190 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.967+0000 7f9bf639d740 1 --2- 172.21.15.27:0/1198266612 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4fa6ce570 0x55b4fa6e1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.967+0000 7f9bf639d740 1 --2- 172.21.15.27:0/1198266612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4fa6c3ea0 0x55b4fa6e19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.967+0000 7f9bf639d740 1 --2- 172.21.15.27:0/1198266612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4fa6c3580 0x55b4fa6e1490 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.967+0000 7f9bf639d740 1 -- 172.21.15.27:0/1198266612 >> 172.21.15.27:0/1198266612 conn(0x55b4fa6bf8a0 msgr2=0x55b4fa6c5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.967+0000 7f9bd4ff9640 0 client.16044 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:16:56.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.968+0000 7f9bf639d740 1 -- 172.21.15.27:0/1198266612 shutdown_connections 2024-04-25T00:16:56.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.968+0000 7f9bf639d740 1 -- 172.21.15.27:0/1198266612 wait complete. 2024-04-25T00:16:56.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.968+0000 7f9bf639d740 20 client.16044 trim_cache size 0 max 16384 2024-04-25T00:16:56.974 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:16:56.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.968+0000 7f9bf35b0640 1 -- 172.21.15.27:0/950531315 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9bd80a2f30 msgr2=0x7f9bd80a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.968+0000 7f9bf35b0640 1 --2- 172.21.15.27:0/950531315 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9bd80a2f30 0x7f9bd80a53f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9be80ccf40 tx=0x7f9be8097f50 comp rx=0 tx=0).stop 2024-04-25T00:16:56.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.968+0000 7f9bf35b0640 1 -- 172.21.15.27:0/950531315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bec101720 msgr2=0x7f9bec197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:56.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.968+0000 7f9bf35b0640 1 --2- 172.21.15.27:0/950531315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bec101720 0x7f9bec197ff0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9be800e9f0 tx=0x7f9be800eeb0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.968+0000 7f9bf35b0640 1 -- 172.21.15.27:0/950531315 shutdown_connections 2024-04-25T00:16:56.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.968+0000 7f9bf35b0640 1 --2- 172.21.15.27:0/950531315 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9bd80a2f30 0x7f9bd80a53f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.968+0000 7f9bf35b0640 1 --2- 172.21.15.27:0/950531315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9bec107730 0x7f9bec19c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.968+0000 7f9bf35b0640 1 --2- 172.21.15.27:0/950531315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bec1020e0 0x7f9bec198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.968+0000 7f9bf35b0640 1 --2- 172.21.15.27:0/950531315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bec101720 0x7f9bec197ff0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:56.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.968+0000 7f9bf35b0640 1 -- 172.21.15.27:0/950531315 >> 172.21.15.27:0/950531315 conn(0x7f9bec078b40 msgr2=0x7f9bec104010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:56.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.969+0000 7f9bf35b0640 1 -- 172.21.15.27:0/950531315 shutdown_connections 2024-04-25T00:16:56.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:56.969+0000 7f9bf35b0640 1 -- 172.21.15.27:0/950531315 wait complete. 2024-04-25T00:16:56.983 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T00:16:57.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.088+0000 7f0491de7640 1 -- 172.21.15.27:0/158135881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f048c107750 msgr2=0x7f048c107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.088+0000 7f0491de7640 1 --2- 172.21.15.27:0/158135881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f048c107750 0x7f048c107b30 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f047c009930 tx=0x7f047c02a420 comp rx=0 tx=0).stop 2024-04-25T00:16:57.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.088+0000 7f0491de7640 1 -- 172.21.15.27:0/158135881 shutdown_connections 2024-04-25T00:16:57.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.088+0000 7f0491de7640 1 --2- 172.21.15.27:0/158135881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f048c102100 0x7f048c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.088+0000 7f0491de7640 1 --2- 172.21.15.27:0/158135881 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f048c101740 0x7f048c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.088+0000 7f0491de7640 1 --2- 172.21.15.27:0/158135881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f048c107750 0x7f048c107b30 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.088+0000 7f0491de7640 1 -- 172.21.15.27:0/158135881 >> 172.21.15.27:0/158135881 conn(0x7f048c078bd0 msgr2=0x7f048c078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.089+0000 7f0491de7640 1 -- 172.21.15.27:0/158135881 shutdown_connections 2024-04-25T00:16:57.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.089+0000 7f0491de7640 1 -- 172.21.15.27:0/158135881 wait complete. 2024-04-25T00:16:57.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.089+0000 7f0491de7640 1 Processor -- start 2024-04-25T00:16:57.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.089+0000 7f0491de7640 1 -- start start 2024-04-25T00:16:57.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.090+0000 7f0491de7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f048c101740 0x7f048c073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.090+0000 7f0491de7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f048c102100 0x7f048c073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.090+0000 7f0491de7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f048c107750 0x7f048c0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.090+0000 7f0491de7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f048c111b60 con 0x7f048c101740 2024-04-25T00:16:57.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.090+0000 7f0491de7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f048c1119e0 con 0x7f048c107750 2024-04-25T00:16:57.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.090+0000 7f0491de7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f048c111ce0 con 0x7f048c102100 2024-04-25T00:16:57.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.090+0000 7f048b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f048c101740 0x7f048c073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.090+0000 7f048b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f048c101740 0x7f048c073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51366/0 (socket says 172.21.15.27:51366) 2024-04-25T00:16:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.090+0000 7f048b7fe640 1 -- 172.21.15.27:0/1150136032 learned_addr learned my addr 172.21.15.27:0/1150136032 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.090+0000 7f048affd640 1 --2- 172.21.15.27:0/1150136032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f048c102100 0x7f048c073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.090+0000 7f048bfff640 1 --2- 172.21.15.27:0/1150136032 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f048c107750 0x7f048c0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.090+0000 7f048affd640 1 -- 172.21.15.27:0/1150136032 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f048c107750 msgr2=0x7f048c0744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.090+0000 7f048affd640 1 --2- 172.21.15.27:0/1150136032 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f048c107750 0x7f048c0744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.090+0000 7f048affd640 1 -- 172.21.15.27:0/1150136032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f048c101740 msgr2=0x7f048c073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.090+0000 7f048affd640 1 --2- 172.21.15.27:0/1150136032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f048c101740 0x7f048c073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.090+0000 7f048affd640 1 -- 172.21.15.27:0/1150136032 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f048c1a6540 con 0x7f048c102100 2024-04-25T00:16:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.090+0000 7f048b7fe640 1 --2- 172.21.15.27:0/1150136032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f048c101740 0x7f048c073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.091+0000 7f048affd640 1 --2- 172.21.15.27:0/1150136032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f048c102100 0x7f048c073f80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f048000d6d0 tx=0x7f048000db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.091+0000 7f0488ff9640 1 -- 172.21.15.27:0/1150136032 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0480017070 con 0x7f048c102100 2024-04-25T00:16:57.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.091+0000 7f0491de7640 1 -- 172.21.15.27:0/1150136032 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f048c1a6830 con 0x7f048c102100 2024-04-25T00:16:57.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.091+0000 7f0488ff9640 1 -- 172.21.15.27:0/1150136032 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0480009820 con 0x7f048c102100 2024-04-25T00:16:57.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.091+0000 7f0491de7640 1 -- 172.21.15.27:0/1150136032 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f048c1a6dd0 con 0x7f048c102100 2024-04-25T00:16:57.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.091+0000 7f0488ff9640 1 -- 172.21.15.27:0/1150136032 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0480004cd0 con 0x7f048c102100 2024-04-25T00:16:57.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.094+0000 7f0488ff9640 1 -- 172.21.15.27:0/1150136032 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f04800099d0 con 0x7f048c102100 2024-04-25T00:16:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.094+0000 7f0494bd3740 1 Processor -- start 2024-04-25T00:16:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.095+0000 7f0494bd3740 1 -- start start 2024-04-25T00:16:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.095+0000 7f0494bd3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac54580 0x56449ac54960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.095+0000 7f0494bd3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac54ea0 0x56449ac5f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.095+0000 7f0494bd3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac5f570 0x56449ac61960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.095+0000 7f0494bd3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56449abb9670 con 0x56449ac54580 2024-04-25T00:16:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.095+0000 7f0494bd3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56449ab61970 con 0x56449ac54ea0 2024-04-25T00:16:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.095+0000 7f0494bd3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56449abb5630 con 0x56449ac5f570 2024-04-25T00:16:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.095+0000 7f048bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac54580 0x56449ac54960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.095+0000 7f048bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac54580 0x56449ac54960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51374/0 (socket says 172.21.15.27:51374) 2024-04-25T00:16:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.095+0000 7f048bfff640 1 -- 172.21.15.27:0/3875871111 learned_addr learned my addr 172.21.15.27:0/3875871111 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.095+0000 7f0488ff9640 1 --2- 172.21.15.27:0/1150136032 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f04740a2fb0 0x7f04740a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.095+0000 7f048b7fe640 1 --2- 172.21.15.27:0/3875871111 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac54ea0 0x56449ac5f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.095+0000 7f048affd640 1 --2- 172.21.15.27:0/3875871111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac5f570 0x56449ac61960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.096+0000 7f048bfff640 1 --2- 172.21.15.27:0/1150136032 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f04740a2fb0 0x7f04740a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.096+0000 7f048bfff640 1 -- 172.21.15.27:0/3875871111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac5f570 msgr2=0x56449ac61960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.096+0000 7f048bfff640 1 --2- 172.21.15.27:0/3875871111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac5f570 0x56449ac61960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.096+0000 7f048bfff640 1 -- 172.21.15.27:0/3875871111 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac54ea0 msgr2=0x56449ac5f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.096+0000 7f048bfff640 1 --2- 172.21.15.27:0/3875871111 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac54ea0 0x56449ac5f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.096+0000 7f048bfff640 1 -- 172.21.15.27:0/3875871111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56449ab85400 con 0x56449ac54580 2024-04-25T00:16:57.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.096+0000 7f0488ff9640 1 -- 172.21.15.27:0/1150136032 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f04800cf520 con 0x7f048c102100 2024-04-25T00:16:57.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.096+0000 7f048bfff640 1 --2- 172.21.15.27:0/3875871111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac54580 0x56449ac54960 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0484005f40 tx=0x7f0484014f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f36b11822703342 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.096+0000 7f048affd640 1 --2- 172.21.15.27:0/3875871111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac5f570 0x56449ac61960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:57.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.096+0000 7f0471ffb640 1 -- 172.21.15.27:0/3875871111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048401ee50 con 0x56449ac54580 2024-04-25T00:16:57.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.096+0000 7f048bfff640 1 --2- 172.21.15.27:0/1150136032 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f04740a2fb0 0x7f04740a5470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0484022000 tx=0x7f0484020000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.096+0000 7f0471ffb640 1 -- 172.21.15.27:0/3875871111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0484023070 con 0x56449ac54580 2024-04-25T00:16:57.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.097+0000 7f0471ffb640 1 -- 172.21.15.27:0/3875871111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04840054b0 con 0x56449ac54580 2024-04-25T00:16:57.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.097+0000 7f0494bd3740 1 -- 172.21.15.27:0/3875871111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac54580 msgr2=0x56449ac54960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.097+0000 7f0494bd3740 1 --2- 172.21.15.27:0/3875871111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac54580 0x56449ac54960 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0484005f40 tx=0x7f0484014f10 comp rx=0 tx=0).stop 2024-04-25T00:16:57.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.097+0000 7f0494bd3740 1 -- 172.21.15.27:0/3875871111 shutdown_connections 2024-04-25T00:16:57.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.097+0000 7f0494bd3740 1 --2- 172.21.15.27:0/3875871111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac5f570 0x56449ac61960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.097+0000 7f0494bd3740 1 --2- 172.21.15.27:0/3875871111 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac54ea0 0x56449ac5f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.097+0000 7f0494bd3740 1 --2- 172.21.15.27:0/3875871111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac54580 0x56449ac54960 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.097+0000 7f0494bd3740 1 -- 172.21.15.27:0/3875871111 >> 172.21.15.27:0/3875871111 conn(0x56449ac508a0 msgr2=0x56449ac50c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.097+0000 7f0494bd3740 1 -- 172.21.15.27:0/3875871111 shutdown_connections 2024-04-25T00:16:57.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.097+0000 7f0494bd3740 1 -- 172.21.15.27:0/3875871111 wait complete. 2024-04-25T00:16:57.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.098+0000 7f0494bd3740 1 -- start start 2024-04-25T00:16:57.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.098+0000 7f0494bd3740 1 Processor -- start 2024-04-25T00:16:57.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.098+0000 7f0494bd3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac54580 0x56449ac5eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.098+0000 7f0494bd3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac54ea0 0x56449ac5c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.098+0000 7f0494bd3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac5f570 0x56449ac5c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.098+0000 7f0494bd3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56449ac62360 con 0x56449ac5f570 2024-04-25T00:16:57.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.098+0000 7f0494bd3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56449ab75050 con 0x56449ac54580 2024-04-25T00:16:57.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.098+0000 7f0494bd3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56449ac624e0 con 0x56449ac54ea0 2024-04-25T00:16:57.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.098+0000 7f0494bd3740 10 client.0 _ll_register_callbacks cb 0x56449a8ccbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f048bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac5f570 0x56449ac5c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f048b7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac54580 0x56449ac5eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f048affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac54ea0 0x56449ac5c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f048b7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac54580 0x56449ac5eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:60364/0 (socket says 172.21.15.27:60364) 2024-04-25T00:16:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f048affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac54ea0 0x56449ac5c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:47366/0 (socket says 172.21.15.27:47366) 2024-04-25T00:16:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f048bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac5f570 0x56449ac5c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51384/0 (socket says 172.21.15.27:51384) 2024-04-25T00:16:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f048affd640 1 -- 172.21.15.27:0/648363646 learned_addr learned my addr 172.21.15.27:0/648363646 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f048affd640 1 -- 172.21.15.27:0/648363646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac54580 msgr2=0x56449ac5eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f048affd640 1 --2- 172.21.15.27:0/648363646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac54580 0x56449ac5eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f048affd640 1 -- 172.21.15.27:0/648363646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac5f570 msgr2=0x56449ac5c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f048affd640 1 --2- 172.21.15.27:0/648363646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac5f570 0x56449ac5c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f048affd640 1 -- 172.21.15.27:0/648363646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56449ab46dc0 con 0x56449ac54ea0 2024-04-25T00:16:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f048b7fe640 1 --2- 172.21.15.27:0/648363646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac54580 0x56449ac5eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f048bfff640 1 --2- 172.21.15.27:0/648363646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac5f570 0x56449ac5c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f048affd640 1 --2- 172.21.15.27:0/648363646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac54ea0 0x56449ac5c1d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f04800979c0 tx=0x7f048009b190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb3fc7f243c4b78 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f046affd640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:16:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f046affd640 1 -- 172.21.15.27:0/648363646 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0480020070 con 0x56449ac54ea0 2024-04-25T00:16:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f0494bd3740 1 -- 172.21.15.27:0/648363646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0484023070 con 0x56449ac54ea0 2024-04-25T00:16:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.099+0000 7f046affd640 1 -- 172.21.15.27:0/648363646 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f04800cf170 con 0x56449ac54ea0 2024-04-25T00:16:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.100+0000 7f046affd640 1 -- 172.21.15.27:0/648363646 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f04800a4120 con 0x56449ac54ea0 2024-04-25T00:16:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.100+0000 7f0494bd3740 10 client.25633 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.100+0000 7f0494bd3740 1 -- 172.21.15.27:0/648363646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56449a8cd3f0 con 0x56449ac54ea0 2024-04-25T00:16:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.100+0000 7f046affd640 1 -- 172.21.15.27:0/648363646 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0484023070 con 0x56449ac54ea0 2024-04-25T00:16:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.100+0000 7f046affd640 1 -- 172.21.15.27:0/648363646 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f04800d2b00 con 0x56449ac54ea0 2024-04-25T00:16:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.100+0000 7f0494bd3740 10 client.25633 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.100+0000 7f0494bd3740 10 client.25633 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:16:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.100+0000 7f0494bd3740 20 client.25633 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:57.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.100+0000 7f0494bd3740 1 --2- 172.21.15.27:0/648363646 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x56449ac6f4d0 0x56449ac718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.100+0000 7f0494bd3740 4 client.25633 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:16:57.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.100+0000 7f0494bd3740 1 -- 172.21.15.27:0/648363646 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56449abc6bb0 con 0x56449ac6f4d0 2024-04-25T00:16:57.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.100+0000 7f048b7fe640 1 --2- 172.21.15.27:0/648363646 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x56449ac6f4d0 0x56449ac718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.101+0000 7f046affd640 10 client.25633 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:16:57.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.101+0000 7f048b7fe640 1 --2- 172.21.15.27:0/648363646 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x56449ac6f4d0 0x56449ac718c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5e107ddd212e219d server_cookie=2d0a5bf5520de6f7 in_seq=0 out_seq=0 2024-04-25T00:16:57.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.102+0000 7f046affd640 1 -- 172.21.15.27:0/648363646 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56449abc6bb0 con 0x56449ac6f4d0 2024-04-25T00:16:57.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.102+0000 7f046affd640 10 client.25633 handle_command_reply: tid=0 2024-04-25T00:16:57.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.102+0000 7f0494bd3740 1 client.25633 shutdown 2024-04-25T00:16:57.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.102+0000 7f0494bd3740 1 -- 172.21.15.27:0/648363646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac54ea0 msgr2=0x56449ac5c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.102+0000 7f0494bd3740 1 --2- 172.21.15.27:0/648363646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac54ea0 0x56449ac5c1d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f04800979c0 tx=0x7f048009b190 comp rx=0 tx=0).stop 2024-04-25T00:16:57.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.102+0000 7f0494bd3740 1 -- 172.21.15.27:0/648363646 shutdown_connections 2024-04-25T00:16:57.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.102+0000 7f0494bd3740 1 --2- 172.21.15.27:0/648363646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac5f570 0x56449ac5c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.102+0000 7f0494bd3740 1 --2- 172.21.15.27:0/648363646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac54ea0 0x56449ac5c1d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.102+0000 7f0494bd3740 1 --2- 172.21.15.27:0/648363646 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x56449ac6f4d0 0x56449ac718c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.102+0000 7f0494bd3740 1 --2- 172.21.15.27:0/648363646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac54580 0x56449ac5eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.102+0000 7f046affd640 0 client.25633 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:16:57.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.102+0000 7f0494bd3740 1 -- 172.21.15.27:0/648363646 >> 172.21.15.27:0/648363646 conn(0x56449ac508a0 msgr2=0x56449ac4fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.102+0000 7f0494bd3740 1 -- 172.21.15.27:0/648363646 shutdown_connections 2024-04-25T00:16:57.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.102+0000 7f0494bd3740 1 -- 172.21.15.27:0/648363646 wait complete. 2024-04-25T00:16:57.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.102+0000 7f0494bd3740 20 client.25633 trim_cache size 0 max 16384 2024-04-25T00:16:57.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.128+0000 7f0494bd3740 1 Processor -- start 2024-04-25T00:16:57.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.129+0000 7f0494bd3740 1 -- start start 2024-04-25T00:16:57.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.129+0000 7f0494bd3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac54580 0x56449ac567e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.129+0000 7f0494bd3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac54ea0 0x56449ac5ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.129+0000 7f0494bd3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac5f570 0x56449ac77170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.129+0000 7f0494bd3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f048c1b0d00 con 0x56449ac5f570 2024-04-25T00:16:57.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.129+0000 7f0494bd3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f048c1b2840 con 0x56449ac54580 2024-04-25T00:16:57.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.129+0000 7f0494bd3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f048c1b13d0 con 0x56449ac54ea0 2024-04-25T00:16:57.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.129+0000 7f048bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac5f570 0x56449ac77170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.129+0000 7f048bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac5f570 0x56449ac77170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51392/0 (socket says 172.21.15.27:51392) 2024-04-25T00:16:57.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.129+0000 7f048bfff640 1 -- 172.21.15.27:0/791477651 learned_addr learned my addr 172.21.15.27:0/791477651 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:57.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.129+0000 7f048b7fe640 1 --2- 172.21.15.27:0/791477651 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac54580 0x56449ac567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.129+0000 7f048affd640 1 --2- 172.21.15.27:0/791477651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac54ea0 0x56449ac5ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.129+0000 7f048bfff640 1 -- 172.21.15.27:0/791477651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac54ea0 msgr2=0x56449ac5ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.129+0000 7f048bfff640 1 --2- 172.21.15.27:0/791477651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac54ea0 0x56449ac5ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.129+0000 7f048bfff640 1 -- 172.21.15.27:0/791477651 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac54580 msgr2=0x56449ac567e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.129+0000 7f048bfff640 1 --2- 172.21.15.27:0/791477651 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac54580 0x56449ac567e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.129+0000 7f048bfff640 1 -- 172.21.15.27:0/791477651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56449ac5caa0 con 0x56449ac5f570 2024-04-25T00:16:57.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.130+0000 7f048bfff640 1 --2- 172.21.15.27:0/791477651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac5f570 0x56449ac77170 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0484005f40 tx=0x7f04840099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56675fffe0031f3a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.130+0000 7f04717fa640 1 -- 172.21.15.27:0/791477651 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0484037070 con 0x56449ac5f570 2024-04-25T00:16:57.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.130+0000 7f04717fa640 1 -- 172.21.15.27:0/791477651 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0484023070 con 0x56449ac5f570 2024-04-25T00:16:57.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.130+0000 7f04717fa640 1 -- 172.21.15.27:0/791477651 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0484004c90 con 0x56449ac5f570 2024-04-25T00:16:57.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.131+0000 7f0494bd3740 1 -- 172.21.15.27:0/791477651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac5f570 msgr2=0x56449ac77170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.131+0000 7f0494bd3740 1 --2- 172.21.15.27:0/791477651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac5f570 0x56449ac77170 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0484005f40 tx=0x7f04840099f0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.131+0000 7f0494bd3740 1 -- 172.21.15.27:0/791477651 shutdown_connections 2024-04-25T00:16:57.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.131+0000 7f0494bd3740 1 --2- 172.21.15.27:0/791477651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac5f570 0x56449ac77170 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.131+0000 7f0494bd3740 1 --2- 172.21.15.27:0/791477651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac54ea0 0x56449ac5ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.131+0000 7f0494bd3740 1 --2- 172.21.15.27:0/791477651 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac54580 0x56449ac567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.131+0000 7f0494bd3740 1 -- 172.21.15.27:0/791477651 >> 172.21.15.27:0/791477651 conn(0x56449ac508a0 msgr2=0x56449ac57690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.131+0000 7f0494bd3740 1 -- 172.21.15.27:0/791477651 shutdown_connections 2024-04-25T00:16:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.131+0000 7f0494bd3740 1 -- 172.21.15.27:0/791477651 wait complete. 2024-04-25T00:16:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.131+0000 7f0494bd3740 1 -- start start 2024-04-25T00:16:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.132+0000 7f0494bd3740 1 Processor -- start 2024-04-25T00:16:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.132+0000 7f0494bd3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac54580 0x56449ac59570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.132+0000 7f0494bd3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac54ea0 0x56449ac59ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.132+0000 7f0494bd3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac5f570 0x56449acd49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.132+0000 7f0494bd3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f048c1b2170 con 0x56449ac54ea0 2024-04-25T00:16:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.132+0000 7f0494bd3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f048c1b1aa0 con 0x56449ac5f570 2024-04-25T00:16:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.132+0000 7f0494bd3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f048c1aecc0 con 0x56449ac54580 2024-04-25T00:16:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.132+0000 7f0494bd3740 10 client.0 _ll_register_callbacks cb 0x7f04800057f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.132+0000 7f048affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac54ea0 0x56449ac59ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.132+0000 7f048affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac54ea0 0x56449ac59ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51406/0 (socket says 172.21.15.27:51406) 2024-04-25T00:16:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.132+0000 7f048affd640 1 -- 172.21.15.27:0/3685074325 learned_addr learned my addr 172.21.15.27:0/3685074325 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.132+0000 7f048b7fe640 1 --2- 172.21.15.27:0/3685074325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac54580 0x56449ac59570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.132+0000 7f048bfff640 1 --2- 172.21.15.27:0/3685074325 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac5f570 0x56449acd49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.133+0000 7f048affd640 1 -- 172.21.15.27:0/3685074325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac54580 msgr2=0x56449ac59570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.133+0000 7f048affd640 1 --2- 172.21.15.27:0/3685074325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac54580 0x56449ac59570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.133+0000 7f048affd640 1 -- 172.21.15.27:0/3685074325 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac5f570 msgr2=0x56449acd49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.133+0000 7f048affd640 1 --2- 172.21.15.27:0/3685074325 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac5f570 0x56449acd49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.133+0000 7f048affd640 1 -- 172.21.15.27:0/3685074325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56449a9977d0 con 0x56449ac54ea0 2024-04-25T00:16:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.133+0000 7f048affd640 1 --2- 172.21.15.27:0/3685074325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac54ea0 0x56449ac59ab0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f048000d6a0 tx=0x7f04800cf420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa0f0d195c109180 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.133+0000 7f046affd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.133+0000 7f046affd640 1 -- 172.21.15.27:0/3685074325 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0480020040 con 0x56449ac54ea0 2024-04-25T00:16:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.133+0000 7f0494bd3740 1 -- 172.21.15.27:0/3685074325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0484023070 con 0x56449ac54ea0 2024-04-25T00:16:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.133+0000 7f046affd640 1 -- 172.21.15.27:0/3685074325 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f048009a9b0 con 0x56449ac54ea0 2024-04-25T00:16:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.133+0000 7f046affd640 1 -- 172.21.15.27:0/3685074325 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f04800d2d10 con 0x56449ac54ea0 2024-04-25T00:16:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.133+0000 7f0494bd3740 10 client.16062 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.133+0000 7f0494bd3740 1 -- 172.21.15.27:0/3685074325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56449acd5ff0 con 0x56449ac54ea0 2024-04-25T00:16:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.133+0000 7f046affd640 1 -- 172.21.15.27:0/3685074325 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0484023070 con 0x56449ac54ea0 2024-04-25T00:16:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.134+0000 7f046affd640 1 -- 172.21.15.27:0/3685074325 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f048009a420 con 0x56449ac54ea0 2024-04-25T00:16:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.134+0000 7f0494bd3740 10 client.16062 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.134+0000 7f0494bd3740 10 client.16062 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:16:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.134+0000 7f0494bd3740 20 client.16062 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.134+0000 7f0494bd3740 1 --2- 172.21.15.27:0/3685074325 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x56449acd6740 0x56449acd8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.134+0000 7f0494bd3740 4 client.16062 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:16:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.134+0000 7f0494bd3740 1 -- 172.21.15.27:0/3685074325 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56449abc0f00 con 0x56449acd6740 2024-04-25T00:16:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.134+0000 7f048b7fe640 1 --2- 172.21.15.27:0/3685074325 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x56449acd6740 0x56449acd8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.135+0000 7f046affd640 10 client.16062 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:16:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.135+0000 7f048b7fe640 1 --2- 172.21.15.27:0/3685074325 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x56449acd6740 0x56449acd8b60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2680fb2e21d1d017 server_cookie=72cf7889daa933e7 in_seq=0 out_seq=0 2024-04-25T00:16:57.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.135+0000 7f046affd640 1 -- 172.21.15.27:0/3685074325 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56449abc0f00 con 0x56449acd6740 2024-04-25T00:16:57.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.135+0000 7f046affd640 10 client.16062 handle_command_reply: tid=0 2024-04-25T00:16:57.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.135+0000 7f0494bd3740 1 client.16062 shutdown 2024-04-25T00:16:57.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.135+0000 7f0494bd3740 1 -- 172.21.15.27:0/3685074325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac54ea0 msgr2=0x56449ac59ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.135+0000 7f0494bd3740 1 --2- 172.21.15.27:0/3685074325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac54ea0 0x56449ac59ab0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f048000d6a0 tx=0x7f04800cf420 comp rx=0 tx=0).stop 2024-04-25T00:16:57.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.135+0000 7f0494bd3740 1 -- 172.21.15.27:0/3685074325 shutdown_connections 2024-04-25T00:16:57.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.135+0000 7f0494bd3740 1 --2- 172.21.15.27:0/3685074325 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x56449acd6740 0x56449acd8b60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.135+0000 7f0494bd3740 1 --2- 172.21.15.27:0/3685074325 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56449ac5f570 0x56449acd49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.135+0000 7f046affd640 0 client.16062 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:16:57.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.135+0000 7f0494bd3740 1 --2- 172.21.15.27:0/3685074325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56449ac54ea0 0x56449ac59ab0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.135+0000 7f0494bd3740 1 --2- 172.21.15.27:0/3685074325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56449ac54580 0x56449ac59570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.135+0000 7f0494bd3740 1 -- 172.21.15.27:0/3685074325 >> 172.21.15.27:0/3685074325 conn(0x56449ac508a0 msgr2=0x56449ac4f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.135+0000 7f0494bd3740 1 -- 172.21.15.27:0/3685074325 shutdown_connections 2024-04-25T00:16:57.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.135+0000 7f0494bd3740 1 -- 172.21.15.27:0/3685074325 wait complete. 2024-04-25T00:16:57.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.136+0000 7f0494bd3740 20 client.16062 trim_cache size 0 max 16384 2024-04-25T00:16:57.142 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:16:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.136+0000 7f0491de7640 1 -- 172.21.15.27:0/1150136032 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f04740a2fb0 msgr2=0x7f04740a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.136+0000 7f0491de7640 1 --2- 172.21.15.27:0/1150136032 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f04740a2fb0 0x7f04740a5470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0484022000 tx=0x7f0484020000 comp rx=0 tx=0).stop 2024-04-25T00:16:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.136+0000 7f0491de7640 1 -- 172.21.15.27:0/1150136032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f048c102100 msgr2=0x7f048c073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.136+0000 7f0491de7640 1 --2- 172.21.15.27:0/1150136032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f048c102100 0x7f048c073f80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f048000d6d0 tx=0x7f048000db90 comp rx=0 tx=0).stop 2024-04-25T00:16:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.136+0000 7f0491de7640 1 -- 172.21.15.27:0/1150136032 shutdown_connections 2024-04-25T00:16:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.136+0000 7f0491de7640 1 --2- 172.21.15.27:0/1150136032 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f04740a2fb0 0x7f04740a5470 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.136+0000 7f0491de7640 1 --2- 172.21.15.27:0/1150136032 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f048c107750 0x7f048c0744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.136+0000 7f0491de7640 1 --2- 172.21.15.27:0/1150136032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f048c102100 0x7f048c073f80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.136+0000 7f0491de7640 1 --2- 172.21.15.27:0/1150136032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f048c101740 0x7f048c073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.136+0000 7f0491de7640 1 -- 172.21.15.27:0/1150136032 >> 172.21.15.27:0/1150136032 conn(0x7f048c078bd0 msgr2=0x7f048c0ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.136+0000 7f0491de7640 1 -- 172.21.15.27:0/1150136032 shutdown_connections 2024-04-25T00:16:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.136+0000 7f0491de7640 1 -- 172.21.15.27:0/1150136032 wait complete. 2024-04-25T00:16:57.148 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T00:16:57.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.254+0000 7f9f3ed61640 1 -- 172.21.15.27:0/3184670845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f381066b0 msgr2=0x7f9f38106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.254+0000 7f9f3ed61640 1 --2- 172.21.15.27:0/3184670845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f381066b0 0x7f9f38106a90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009930 tx=0x7f9f3402a420 comp rx=0 tx=0).stop 2024-04-25T00:16:57.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.254+0000 7f9f3ed61640 1 -- 172.21.15.27:0/3184670845 shutdown_connections 2024-04-25T00:16:57.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.254+0000 7f9f3ed61640 1 --2- 172.21.15.27:0/3184670845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f38100fd0 0x7f9f3810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.254+0000 7f9f3ed61640 1 --2- 172.21.15.27:0/3184670845 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f38100610 0x7f9f38100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.254+0000 7f9f3ed61640 1 --2- 172.21.15.27:0/3184670845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f381066b0 0x7f9f38106a90 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.254+0000 7f9f3ed61640 1 -- 172.21.15.27:0/3184670845 >> 172.21.15.27:0/3184670845 conn(0x7f9f38077740 msgr2=0x7f9f38077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.254+0000 7f9f3ed61640 1 -- 172.21.15.27:0/3184670845 shutdown_connections 2024-04-25T00:16:57.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.254+0000 7f9f3ed61640 1 -- 172.21.15.27:0/3184670845 wait complete. 2024-04-25T00:16:57.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.255+0000 7f9f3ed61640 1 Processor -- start 2024-04-25T00:16:57.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.255+0000 7f9f3ed61640 1 -- start start 2024-04-25T00:16:57.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.255+0000 7f9f3ed61640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f38100610 0x7f9f38196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.256+0000 7f9f3ed61640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f38100fd0 0x7f9f381973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.256+0000 7f9f3ed61640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f381066b0 0x7f9f3819b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.256+0000 7f9f3ed61640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9f38110a80 con 0x7f9f38100fd0 2024-04-25T00:16:57.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.256+0000 7f9f3ed61640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9f38110900 con 0x7f9f381066b0 2024-04-25T00:16:57.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.256+0000 7f9f3ed61640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9f38110c00 con 0x7f9f38100610 2024-04-25T00:16:57.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.256+0000 7f9f2ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f38100fd0 0x7f9f381973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.256+0000 7f9f2ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f38100fd0 0x7f9f381973e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51424/0 (socket says 172.21.15.27:51424) 2024-04-25T00:16:57.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.256+0000 7f9f2ffff640 1 -- 172.21.15.27:0/2883811155 learned_addr learned my addr 172.21.15.27:0/2883811155 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:57.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.256+0000 7f9f3d2fe640 1 --2- 172.21.15.27:0/2883811155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f381066b0 0x7f9f3819b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.256+0000 7f9f3cafd640 1 --2- 172.21.15.27:0/2883811155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f38100610 0x7f9f38196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.256+0000 7f9f3d2fe640 1 -- 172.21.15.27:0/2883811155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f38100610 msgr2=0x7f9f38196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.256+0000 7f9f3d2fe640 1 --2- 172.21.15.27:0/2883811155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f38100610 0x7f9f38196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.256+0000 7f9f3d2fe640 1 -- 172.21.15.27:0/2883811155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f38100fd0 msgr2=0x7f9f381973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.256+0000 7f9f3d2fe640 1 --2- 172.21.15.27:0/2883811155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f38100fd0 0x7f9f381973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.256+0000 7f9f3d2fe640 1 -- 172.21.15.27:0/2883811155 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f3819be90 con 0x7f9f381066b0 2024-04-25T00:16:57.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.256+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/2883811155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f38100fd0 0x7f9f381973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:57.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.257+0000 7f9f3d2fe640 1 --2- 172.21.15.27:0/2883811155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f381066b0 0x7f9f3819b760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9f30009fd0 tx=0x7f9f3000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.257+0000 7f9f2dffb640 1 -- 172.21.15.27:0/2883811155 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3000ce20 con 0x7f9f381066b0 2024-04-25T00:16:57.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.257+0000 7f9f3ed61640 1 -- 172.21.15.27:0/2883811155 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f3819c180 con 0x7f9f381066b0 2024-04-25T00:16:57.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.257+0000 7f9f2dffb640 1 -- 172.21.15.27:0/2883811155 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f30005020 con 0x7f9f381066b0 2024-04-25T00:16:57.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.257+0000 7f9f2dffb640 1 -- 172.21.15.27:0/2883811155 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f30010720 con 0x7f9f381066b0 2024-04-25T00:16:57.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.257+0000 7f9f3ed61640 1 -- 172.21.15.27:0/2883811155 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f380688d0 con 0x7f9f381066b0 2024-04-25T00:16:57.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.260+0000 7f9f2dffb640 1 -- 172.21.15.27:0/2883811155 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f9f30005540 con 0x7f9f381066b0 2024-04-25T00:16:57.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.260+0000 7f9f41a6b740 1 Processor -- start 2024-04-25T00:16:57.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.260+0000 7f9f41a6b740 1 -- start start 2024-04-25T00:16:57.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f41a6b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddceb990 0x55cdddcebd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f41a6b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcec2b0 0x55cdddcf6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f41a6b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcf6980 0x55cdddcf8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f41a6b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cdddc4c670 con 0x55cdddcf6980 2024-04-25T00:16:57.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f41a6b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55cdddbf4970 con 0x55cdddceb990 2024-04-25T00:16:57.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f41a6b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cdddc48630 con 0x55cdddcec2b0 2024-04-25T00:16:57.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f2ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcf6980 0x55cdddcf8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f2ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcf6980 0x55cdddcf8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51430/0 (socket says 172.21.15.27:51430) 2024-04-25T00:16:57.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f2ffff640 1 -- 172.21.15.27:0/2655704258 learned_addr learned my addr 172.21.15.27:0/2655704258 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:57.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f2dffb640 1 --2- 172.21.15.27:0/2883811155 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9f200a2f30 0x7f9f200a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f3cafd640 1 --2- 172.21.15.27:0/2655704258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcec2b0 0x55cdddcf6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/2655704258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddceb990 0x55cdddcebd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f2dffb640 1 -- 172.21.15.27:0/2883811155 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f9f300d0020 con 0x7f9f381066b0 2024-04-25T00:16:57.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f2ffff640 1 -- 172.21.15.27:0/2655704258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcec2b0 msgr2=0x55cdddcf6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/2655704258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcec2b0 0x55cdddcf6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f2ffff640 1 -- 172.21.15.27:0/2655704258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddceb990 msgr2=0x55cdddcebd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/2655704258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddceb990 0x55cdddcebd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.261+0000 7f9f2ffff640 1 -- 172.21.15.27:0/2655704258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdddc18400 con 0x55cdddcf6980 2024-04-25T00:16:57.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.262+0000 7f9f3d2fe640 1 --2- 172.21.15.27:0/2883811155 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9f200a2f30 0x7f9f200a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.262+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/2655704258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcf6980 0x55cdddcf8d70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9f28012a40 tx=0x7f9f28005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cbcb25c3dd2fe7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.262+0000 7f9f1effd640 1 -- 172.21.15.27:0/2655704258 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f28010b30 con 0x55cdddcf6980 2024-04-25T00:16:57.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.262+0000 7f9f1effd640 1 -- 172.21.15.27:0/2655704258 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f28005500 con 0x55cdddcf6980 2024-04-25T00:16:57.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.262+0000 7f9f1effd640 1 -- 172.21.15.27:0/2655704258 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2801c390 con 0x55cdddcf6980 2024-04-25T00:16:57.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.263+0000 7f9f41a6b740 1 -- 172.21.15.27:0/2655704258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcf6980 msgr2=0x55cdddcf8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.263+0000 7f9f3d2fe640 1 --2- 172.21.15.27:0/2883811155 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9f200a2f30 0x7f9f200a53f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9f300cd220 tx=0x7f9f30005b40 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.263+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/2655704258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcf6980 0x55cdddcf8d70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9f28012a40 tx=0x7f9f28005f30 comp rx=0 tx=0).stop 2024-04-25T00:16:57.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.263+0000 7f9f41a6b740 1 -- 172.21.15.27:0/2655704258 shutdown_connections 2024-04-25T00:16:57.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.263+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/2655704258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcf6980 0x55cdddcf8d70 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.263+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/2655704258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcec2b0 0x55cdddcf6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.263+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/2655704258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddceb990 0x55cdddcebd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.263+0000 7f9f41a6b740 1 -- 172.21.15.27:0/2655704258 >> 172.21.15.27:0/2655704258 conn(0x55cdddce7cb0 msgr2=0x55cdddce8090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.263+0000 7f9f41a6b740 1 -- 172.21.15.27:0/2655704258 shutdown_connections 2024-04-25T00:16:57.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.263+0000 7f9f41a6b740 1 -- 172.21.15.27:0/2655704258 wait complete. 2024-04-25T00:16:57.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.263+0000 7f9f41a6b740 1 -- start start 2024-04-25T00:16:57.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.263+0000 7f9f41a6b740 1 Processor -- start 2024-04-25T00:16:57.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.264+0000 7f9f41a6b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddceb990 0x55cdddcf61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.264+0000 7f9f41a6b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcec2b0 0x55cdddcef450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.264+0000 7f9f41a6b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcef990 0x55cdddcefe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.264+0000 7f9f41a6b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cdddcf9770 con 0x55cdddcef990 2024-04-25T00:16:57.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.264+0000 7f9f41a6b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55cdddc08050 con 0x55cdddceb990 2024-04-25T00:16:57.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.264+0000 7f9f41a6b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cdddcf98f0 con 0x55cdddcec2b0 2024-04-25T00:16:57.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.264+0000 7f9f2ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcef990 0x55cdddcefe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.264+0000 7f9f41a6b740 10 client.0 _ll_register_callbacks cb 0x55cddd95fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:57.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.264+0000 7f9f2ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcef990 0x55cdddcefe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51446/0 (socket says 172.21.15.27:51446) 2024-04-25T00:16:57.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.264+0000 7f9f2ffff640 1 -- 172.21.15.27:0/2548487173 learned_addr learned my addr 172.21.15.27:0/2548487173 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:57.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.264+0000 7f9f3cafd640 1 --2- 172.21.15.27:0/2548487173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcec2b0 0x55cdddcef450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.265+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/2548487173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddceb990 0x55cdddcf61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.265+0000 7f9f2ffff640 1 -- 172.21.15.27:0/2548487173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcec2b0 msgr2=0x55cdddcef450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.265+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/2548487173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcec2b0 0x55cdddcef450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.265+0000 7f9f2ffff640 1 -- 172.21.15.27:0/2548487173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddceb990 msgr2=0x55cdddcf61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.265+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/2548487173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddceb990 0x55cdddcf61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.265+0000 7f9f2ffff640 1 -- 172.21.15.27:0/2548487173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdddbd9dc0 con 0x55cdddcef990 2024-04-25T00:16:57.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.265+0000 7f9f3cafd640 1 --2- 172.21.15.27:0/2548487173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcec2b0 0x55cdddcef450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:57.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.265+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/2548487173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcef990 0x55cdddcefe10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f9f28010740 tx=0x7f9f2800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25e7dffc9cde4b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.265+0000 7f9f17fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:57.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.265+0000 7f9f17fff640 1 -- 172.21.15.27:0/2548487173 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2802e060 con 0x55cdddcef990 2024-04-25T00:16:57.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.265+0000 7f9f17fff640 1 -- 172.21.15.27:0/2548487173 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f9f28004260 con 0x55cdddcef990 2024-04-25T00:16:57.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.265+0000 7f9f41a6b740 1 -- 172.21.15.27:0/2548487173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f28005500 con 0x55cdddcef990 2024-04-25T00:16:57.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.265+0000 7f9f17fff640 1 -- 172.21.15.27:0/2548487173 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9f28004460 con 0x55cdddcef990 2024-04-25T00:16:57.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.265+0000 7f9f41a6b740 10 client.16074 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:57.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.266+0000 7f9f41a6b740 1 -- 172.21.15.27:0/2548487173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cddd9603f0 con 0x55cdddcef990 2024-04-25T00:16:57.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.266+0000 7f9f17fff640 1 -- 172.21.15.27:0/2548487173 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f28005500 con 0x55cdddcef990 2024-04-25T00:16:57.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.266+0000 7f9f17fff640 1 -- 172.21.15.27:0/2548487173 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9f28028070 con 0x55cdddcef990 2024-04-25T00:16:57.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.266+0000 7f9f41a6b740 10 client.16074 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:57.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.266+0000 7f9f41a6b740 10 client.16074 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:16:57.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.266+0000 7f9f41a6b740 20 client.16074 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.266+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/2548487173 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55cdddd0b2b0 0x55cdddd0d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.266+0000 7f9f41a6b740 4 client.16074 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:16:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.266+0000 7f9f41a6b740 1 -- 172.21.15.27:0/2548487173 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cdddc59bb0 con 0x55cdddd0b2b0 2024-04-25T00:16:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.267+0000 7f9f3cafd640 1 --2- 172.21.15.27:0/2548487173 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55cdddd0b2b0 0x55cdddd0d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.267+0000 7f9f17fff640 10 client.16074 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:16:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.267+0000 7f9f3cafd640 1 --2- 172.21.15.27:0/2548487173 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55cdddd0b2b0 0x55cdddd0d6a0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=277046ef5ac179d6 server_cookie=cc8cffcc47aab746 in_seq=0 out_seq=0 2024-04-25T00:16:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.268+0000 7f9f17fff640 1 -- 172.21.15.27:0/2548487173 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cdddc59bb0 con 0x55cdddd0b2b0 2024-04-25T00:16:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.268+0000 7f9f17fff640 10 client.16074 handle_command_reply: tid=0 2024-04-25T00:16:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.268+0000 7f9f41a6b740 1 client.16074 shutdown 2024-04-25T00:16:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.268+0000 7f9f41a6b740 1 -- 172.21.15.27:0/2548487173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcef990 msgr2=0x55cdddcefe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.268+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/2548487173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcef990 0x55cdddcefe10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f9f28010740 tx=0x7f9f2800b4a0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.268+0000 7f9f41a6b740 1 -- 172.21.15.27:0/2548487173 shutdown_connections 2024-04-25T00:16:57.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.268+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/2548487173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcef990 0x55cdddcefe10 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.268+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/2548487173 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55cdddd0b2b0 0x55cdddd0d6a0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.268+0000 7f9f17fff640 0 client.16074 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:16:57.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.268+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/2548487173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcec2b0 0x55cdddcef450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.268+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/2548487173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddceb990 0x55cdddcf61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.268+0000 7f9f41a6b740 1 -- 172.21.15.27:0/2548487173 >> 172.21.15.27:0/2548487173 conn(0x55cdddce7cb0 msgr2=0x55cdddced220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.268+0000 7f9f41a6b740 1 -- 172.21.15.27:0/2548487173 shutdown_connections 2024-04-25T00:16:57.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.268+0000 7f9f41a6b740 1 -- 172.21.15.27:0/2548487173 wait complete. 2024-04-25T00:16:57.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.268+0000 7f9f41a6b740 20 client.16074 trim_cache size 0 max 16384 2024-04-25T00:16:57.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.282+0000 7f9f41a6b740 1 Processor -- start 2024-04-25T00:16:57.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.282+0000 7f9f41a6b740 1 -- start start 2024-04-25T00:16:57.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.282+0000 7f9f41a6b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddceb990 0x55cdddcf5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.282+0000 7f9f41a6b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcec2b0 0x55cdddcf6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.282+0000 7f9f41a6b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddcef990 0x55cdddce5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.282+0000 7f9f41a6b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9f381ac9c0 con 0x55cdddceb990 2024-04-25T00:16:57.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.282+0000 7f9f41a6b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9f381ae500 con 0x55cdddcef990 2024-04-25T00:16:57.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.282+0000 7f9f41a6b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9f381ad090 con 0x55cdddcec2b0 2024-04-25T00:16:57.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.282+0000 7f9f2ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddceb990 0x55cdddcf5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.282+0000 7f9f2ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddceb990 0x55cdddcf5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51452/0 (socket says 172.21.15.27:51452) 2024-04-25T00:16:57.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.282+0000 7f9f2ffff640 1 -- 172.21.15.27:0/21879233 learned_addr learned my addr 172.21.15.27:0/21879233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:57.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.283+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/21879233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddcef990 0x55cdddce5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.283+0000 7f9f3cafd640 1 --2- 172.21.15.27:0/21879233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcec2b0 0x55cdddcf6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.283+0000 7f9f2ffff640 1 -- 172.21.15.27:0/21879233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcec2b0 msgr2=0x55cdddcf6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.283+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/21879233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcec2b0 0x55cdddcf6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.283+0000 7f9f2ffff640 1 -- 172.21.15.27:0/21879233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddcef990 msgr2=0x55cdddce5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.283+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/21879233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddcef990 0x55cdddce5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.283+0000 7f9f2ffff640 1 -- 172.21.15.27:0/21879233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdddcec0e0 con 0x55cdddceb990 2024-04-25T00:16:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.283+0000 7f9f3cafd640 1 --2- 172.21.15.27:0/21879233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcec2b0 0x55cdddcf6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.283+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/21879233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddceb990 0x55cdddcf5bc0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9f28004aa0 tx=0x7f9f28005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d180cb857ab5a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.283+0000 7f9f1e7fc640 1 -- 172.21.15.27:0/21879233 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2802e040 con 0x55cdddceb990 2024-04-25T00:16:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.283+0000 7f9f1e7fc640 1 -- 172.21.15.27:0/21879233 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f28005020 con 0x55cdddceb990 2024-04-25T00:16:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.283+0000 7f9f1e7fc640 1 -- 172.21.15.27:0/21879233 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f28038080 con 0x55cdddceb990 2024-04-25T00:16:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.284+0000 7f9f41a6b740 1 -- 172.21.15.27:0/21879233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddceb990 msgr2=0x55cdddcf5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.284+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/21879233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddceb990 0x55cdddcf5bc0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9f28004aa0 tx=0x7f9f28005880 comp rx=0 tx=0).stop 2024-04-25T00:16:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.284+0000 7f9f41a6b740 1 -- 172.21.15.27:0/21879233 shutdown_connections 2024-04-25T00:16:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.284+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/21879233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddcef990 0x55cdddce5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.284+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/21879233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcec2b0 0x55cdddcf6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.284+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/21879233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddceb990 0x55cdddcf5bc0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.284+0000 7f9f41a6b740 1 -- 172.21.15.27:0/21879233 >> 172.21.15.27:0/21879233 conn(0x55cdddce7cb0 msgr2=0x55cdddcf3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.284+0000 7f9f41a6b740 1 -- 172.21.15.27:0/21879233 shutdown_connections 2024-04-25T00:16:57.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.284+0000 7f9f41a6b740 1 -- 172.21.15.27:0/21879233 wait complete. 2024-04-25T00:16:57.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.284+0000 7f9f41a6b740 1 -- start start 2024-04-25T00:16:57.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.284+0000 7f9f41a6b740 1 Processor -- start 2024-04-25T00:16:57.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f41a6b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddceb990 0x55cdddd07ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f41a6b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcec2b0 0x55cdddd04e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f41a6b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcef990 0x55cdddd053b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f41a6b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9f381ade30 con 0x55cdddcec2b0 2024-04-25T00:16:57.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f41a6b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9f381ad760 con 0x55cdddceb990 2024-04-25T00:16:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f41a6b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9f381aa980 con 0x55cdddcef990 2024-04-25T00:16:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f41a6b740 10 client.0 _ll_register_callbacks cb 0x7f9f34009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f3cafd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcec2b0 0x55cdddd04e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f3cafd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcec2b0 0x55cdddd04e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51460/0 (socket says 172.21.15.27:51460) 2024-04-25T00:16:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f3cafd640 1 -- 172.21.15.27:0/3127966046 learned_addr learned my addr 172.21.15.27:0/3127966046 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/3127966046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddceb990 0x55cdddd07ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/3127966046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcef990 0x55cdddd053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f3cafd640 1 -- 172.21.15.27:0/3127966046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcef990 msgr2=0x55cdddd053b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f3cafd640 1 --2- 172.21.15.27:0/3127966046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcef990 0x55cdddd053b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f3cafd640 1 -- 172.21.15.27:0/3127966046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddceb990 msgr2=0x55cdddd07ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f3cafd640 1 --2- 172.21.15.27:0/3127966046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddceb990 0x55cdddd07ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f3cafd640 1 -- 172.21.15.27:0/3127966046 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cddda2a7d0 con 0x55cdddcec2b0 2024-04-25T00:16:57.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/3127966046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddceb990 0x55cdddd07ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:57.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f3cafd640 1 --2- 172.21.15.27:0/3127966046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcec2b0 0x55cdddd04e70 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9f34000c00 tx=0x7f9f34004400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6277a9abb609be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.285+0000 7f9f17fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:57.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.286+0000 7f9f17fff640 1 -- 172.21.15.27:0/3127966046 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f34035040 con 0x55cdddcec2b0 2024-04-25T00:16:57.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.286+0000 7f9f41a6b740 1 -- 172.21.15.27:0/3127966046 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f28005020 con 0x55cdddcec2b0 2024-04-25T00:16:57.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.286+0000 7f9f17fff640 1 -- 172.21.15.27:0/3127966046 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f9f34044540 con 0x55cdddcec2b0 2024-04-25T00:16:57.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.286+0000 7f9f17fff640 1 -- 172.21.15.27:0/3127966046 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9f34044740 con 0x55cdddcec2b0 2024-04-25T00:16:57.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.286+0000 7f9f41a6b740 10 client.16080 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:57.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.286+0000 7f9f41a6b740 1 -- 172.21.15.27:0/3127966046 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cdddd711b0 con 0x55cdddcec2b0 2024-04-25T00:16:57.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.286+0000 7f9f17fff640 1 -- 172.21.15.27:0/3127966046 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f28005020 con 0x55cdddcec2b0 2024-04-25T00:16:57.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.286+0000 7f9f17fff640 1 -- 172.21.15.27:0/3127966046 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9f34004b00 con 0x55cdddcec2b0 2024-04-25T00:16:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.286+0000 7f9f41a6b740 10 client.16080 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.286+0000 7f9f41a6b740 10 client.16080 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:16:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.286+0000 7f9f41a6b740 20 client.16080 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.286+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/3127966046 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55cdddd0b2b0 0x55cdddd738c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.286+0000 7f9f41a6b740 4 client.16080 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:16:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.286+0000 7f9f41a6b740 1 -- 172.21.15.27:0/3127966046 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cdddc53f00 con 0x55cdddd0b2b0 2024-04-25T00:16:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.287+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/3127966046 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55cdddd0b2b0 0x55cdddd738c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.287+0000 7f9f17fff640 10 client.16080 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:16:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.287+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/3127966046 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55cdddd0b2b0 0x55cdddd738c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fe6275ec6ad1f2ef server_cookie=385b89d321fdcebf in_seq=0 out_seq=0 2024-04-25T00:16:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.287+0000 7f9f17fff640 1 -- 172.21.15.27:0/3127966046 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cdddc53f00 con 0x55cdddd0b2b0 2024-04-25T00:16:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.287+0000 7f9f17fff640 10 client.16080 handle_command_reply: tid=0 2024-04-25T00:16:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.287+0000 7f9f41a6b740 1 client.16080 shutdown 2024-04-25T00:16:57.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.287+0000 7f9f41a6b740 1 -- 172.21.15.27:0/3127966046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcec2b0 msgr2=0x55cdddd04e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.287+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/3127966046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcec2b0 0x55cdddd04e70 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9f34000c00 tx=0x7f9f34004400 comp rx=0 tx=0).stop 2024-04-25T00:16:57.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.287+0000 7f9f41a6b740 1 -- 172.21.15.27:0/3127966046 shutdown_connections 2024-04-25T00:16:57.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.287+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/3127966046 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55cdddd0b2b0 0x55cdddd738c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.288+0000 7f9f17fff640 0 client.16080 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:16:57.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.288+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/3127966046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cdddcef990 0x55cdddd053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.288+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/3127966046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cdddcec2b0 0x55cdddd04e70 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.288+0000 7f9f41a6b740 1 --2- 172.21.15.27:0/3127966046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cdddceb990 0x55cdddd07ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.288+0000 7f9f41a6b740 1 -- 172.21.15.27:0/3127966046 >> 172.21.15.27:0/3127966046 conn(0x55cdddce7cb0 msgr2=0x55cdddce3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.288+0000 7f9f41a6b740 1 -- 172.21.15.27:0/3127966046 shutdown_connections 2024-04-25T00:16:57.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.288+0000 7f9f41a6b740 1 -- 172.21.15.27:0/3127966046 wait complete. 2024-04-25T00:16:57.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.288+0000 7f9f41a6b740 20 client.16080 trim_cache size 0 max 16384 2024-04-25T00:16:57.297 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:16:57.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.288+0000 7f9f3ed61640 1 -- 172.21.15.27:0/2883811155 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9f200a2f30 msgr2=0x7f9f200a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.288+0000 7f9f3ed61640 1 --2- 172.21.15.27:0/2883811155 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9f200a2f30 0x7f9f200a53f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9f300cd220 tx=0x7f9f30005b40 comp rx=0 tx=0).stop 2024-04-25T00:16:57.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.288+0000 7f9f3ed61640 1 -- 172.21.15.27:0/2883811155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f381066b0 msgr2=0x7f9f3819b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.288+0000 7f9f3ed61640 1 --2- 172.21.15.27:0/2883811155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f381066b0 0x7f9f3819b760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9f30009fd0 tx=0x7f9f3000ef10 comp rx=0 tx=0).stop 2024-04-25T00:16:57.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.289+0000 7f9f3ed61640 1 -- 172.21.15.27:0/2883811155 shutdown_connections 2024-04-25T00:16:57.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.289+0000 7f9f3ed61640 1 --2- 172.21.15.27:0/2883811155 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9f200a2f30 0x7f9f200a53f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.289+0000 7f9f3ed61640 1 --2- 172.21.15.27:0/2883811155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f381066b0 0x7f9f3819b760 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.289+0000 7f9f3ed61640 1 --2- 172.21.15.27:0/2883811155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f38100fd0 0x7f9f381973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.289+0000 7f9f3ed61640 1 --2- 172.21.15.27:0/2883811155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f38100610 0x7f9f38196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.289+0000 7f9f3ed61640 1 -- 172.21.15.27:0/2883811155 >> 172.21.15.27:0/2883811155 conn(0x7f9f38077740 msgr2=0x7f9f380ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.289+0000 7f9f3ed61640 1 -- 172.21.15.27:0/2883811155 shutdown_connections 2024-04-25T00:16:57.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.289+0000 7f9f3ed61640 1 -- 172.21.15.27:0/2883811155 wait complete. 2024-04-25T00:16:57.304 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T00:16:57.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.448+0000 7ff38b6b3640 1 -- 172.21.15.27:0/2334249871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff384100540 msgr2=0x7ff384100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.448+0000 7ff38b6b3640 1 --2- 172.21.15.27:0/2334249871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff384100540 0x7ff384100920 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ff380009900 tx=0x7ff38002a410 comp rx=0 tx=0).stop 2024-04-25T00:16:57.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.448+0000 7ff38b6b3640 1 -- 172.21.15.27:0/2334249871 shutdown_connections 2024-04-25T00:16:57.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.448+0000 7ff38b6b3640 1 --2- 172.21.15.27:0/2334249871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff38410e0f0 0x7ff3841104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.448+0000 7ff38b6b3640 1 --2- 172.21.15.27:0/2334249871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff384100e60 0x7ff38410dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.448+0000 7ff38b6b3640 1 --2- 172.21.15.27:0/2334249871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff384100540 0x7ff384100920 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.448+0000 7ff38b6b3640 1 -- 172.21.15.27:0/2334249871 >> 172.21.15.27:0/2334249871 conn(0x7ff384077740 msgr2=0x7ff384077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.448+0000 7ff38b6b3640 1 -- 172.21.15.27:0/2334249871 shutdown_connections 2024-04-25T00:16:57.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.448+0000 7ff38b6b3640 1 -- 172.21.15.27:0/2334249871 wait complete. 2024-04-25T00:16:57.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.449+0000 7ff38b6b3640 1 Processor -- start 2024-04-25T00:16:57.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.449+0000 7ff38b6b3640 1 -- start start 2024-04-25T00:16:57.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.449+0000 7ff38b6b3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff384100540 0x7ff38419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.449+0000 7ff38b6b3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff384100e60 0x7ff38419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.449+0000 7ff38b6b3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff38410e0f0 0x7ff38419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.449+0000 7ff38b6b3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff384113cf0 con 0x7ff38410e0f0 2024-04-25T00:16:57.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.449+0000 7ff38b6b3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff384113b70 con 0x7ff384100e60 2024-04-25T00:16:57.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.449+0000 7ff38b6b3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff384113e70 con 0x7ff384100540 2024-04-25T00:16:57.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.450+0000 7ff38944f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff384100540 0x7ff38419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.450+0000 7ff389c50640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff38410e0f0 0x7ff38419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.450+0000 7ff38944f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff384100540 0x7ff38419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:47468/0 (socket says 172.21.15.27:47468) 2024-04-25T00:16:57.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.450+0000 7ff38944f640 1 -- 172.21.15.27:0/4025909078 learned_addr learned my addr 172.21.15.27:0/4025909078 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:57.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.450+0000 7ff388c4e640 1 --2- 172.21.15.27:0/4025909078 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff384100e60 0x7ff38419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.450+0000 7ff389c50640 1 -- 172.21.15.27:0/4025909078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff384100540 msgr2=0x7ff38419a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.450+0000 7ff389c50640 1 --2- 172.21.15.27:0/4025909078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff384100540 0x7ff38419a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.450+0000 7ff389c50640 1 -- 172.21.15.27:0/4025909078 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff384100e60 msgr2=0x7ff38419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.450+0000 7ff389c50640 1 --2- 172.21.15.27:0/4025909078 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff384100e60 0x7ff38419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.450+0000 7ff389c50640 1 -- 172.21.15.27:0/4025909078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff38419f200 con 0x7ff38410e0f0 2024-04-25T00:16:57.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.450+0000 7ff389c50640 1 --2- 172.21.15.27:0/4025909078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff38410e0f0 0x7ff38419ead0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ff37c005fd0 tx=0x7ff37c00edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.450+0000 7ff37a7fc640 1 -- 172.21.15.27:0/4025909078 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c004d20 con 0x7ff38410e0f0 2024-04-25T00:16:57.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.450+0000 7ff38b6b3640 1 -- 172.21.15.27:0/4025909078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff38419f4f0 con 0x7ff38410e0f0 2024-04-25T00:16:57.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.450+0000 7ff37a7fc640 1 -- 172.21.15.27:0/4025909078 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff37c005020 con 0x7ff38410e0f0 2024-04-25T00:16:57.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.450+0000 7ff37a7fc640 1 -- 172.21.15.27:0/4025909078 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c0116d0 con 0x7ff38410e0f0 2024-04-25T00:16:57.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.450+0000 7ff38b6b3640 1 -- 172.21.15.27:0/4025909078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3841a6d80 con 0x7ff38410e0f0 2024-04-25T00:16:57.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.453+0000 7ff37a7fc640 1 -- 172.21.15.27:0/4025909078 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7ff37c005540 con 0x7ff38410e0f0 2024-04-25T00:16:57.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.454+0000 7ff37a7fc640 1 --2- 172.21.15.27:0/4025909078 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff3700a2eb0 0x7ff3700a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.454+0000 7ff38e3bf740 1 Processor -- start 2024-04-25T00:16:57.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.454+0000 7ff38e3bf740 1 -- start start 2024-04-25T00:16:57.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.454+0000 7ff38944f640 1 --2- 172.21.15.27:0/4025909078 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff3700a2eb0 0x7ff3700a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.454+0000 7ff37a7fc640 1 -- 172.21.15.27:0/4025909078 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7ff37c0cf7a0 con 0x7ff38410e0f0 2024-04-25T00:16:57.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.454+0000 7ff38e3bf740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb13580 0x56104eb13960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.454+0000 7ff38e3bf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13ea0 0x56104eb1e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.454+0000 7ff38e3bf740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb1e570 0x56104eb20960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.454+0000 7ff38e3bf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56104ea78670 con 0x56104eb13ea0 2024-04-25T00:16:57.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.454+0000 7ff38e3bf740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56104ea20970 con 0x56104eb13580 2024-04-25T00:16:57.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.454+0000 7ff38e3bf740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56104ea74630 con 0x56104eb1e570 2024-04-25T00:16:57.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.455+0000 7ff388c4e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13ea0 0x56104eb1e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.455+0000 7ff388c4e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13ea0 0x56104eb1e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51488/0 (socket says 172.21.15.27:51488) 2024-04-25T00:16:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.455+0000 7ff388c4e640 1 -- 172.21.15.27:0/3837918986 learned_addr learned my addr 172.21.15.27:0/3837918986 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.455+0000 7ff389c50640 1 --2- 172.21.15.27:0/3837918986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb1e570 0x56104eb20960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.455+0000 7ff38944f640 1 --2- 172.21.15.27:0/3837918986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb13580 0x56104eb13960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.455+0000 7ff38944f640 1 --2- 172.21.15.27:0/4025909078 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff3700a2eb0 0x7ff3700a5370 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff3800096c0 tx=0x7ff380005e50 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.455+0000 7ff388c4e640 1 -- 172.21.15.27:0/3837918986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb1e570 msgr2=0x56104eb20960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.455+0000 7ff388c4e640 1 --2- 172.21.15.27:0/3837918986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb1e570 0x56104eb20960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.455+0000 7ff388c4e640 1 -- 172.21.15.27:0/3837918986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb13580 msgr2=0x56104eb13960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.455+0000 7ff388c4e640 1 --2- 172.21.15.27:0/3837918986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb13580 0x56104eb13960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.455+0000 7ff388c4e640 1 -- 172.21.15.27:0/3837918986 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56104ea44400 con 0x56104eb13ea0 2024-04-25T00:16:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.455+0000 7ff389c50640 1 --2- 172.21.15.27:0/3837918986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb1e570 0x56104eb20960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:57.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.455+0000 7ff38944f640 1 --2- 172.21.15.27:0/3837918986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb13580 0x56104eb13960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:57.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.455+0000 7ff388c4e640 1 --2- 172.21.15.27:0/3837918986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13ea0 0x56104eb1e030 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff374009c90 tx=0x7ff374011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ddc8f2e1f07a63b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.455+0000 7ff36f7fe640 1 -- 172.21.15.27:0/3837918986 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37401e070 con 0x56104eb13ea0 2024-04-25T00:16:57.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.456+0000 7ff36f7fe640 1 -- 172.21.15.27:0/3837918986 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff374019b10 con 0x56104eb13ea0 2024-04-25T00:16:57.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.456+0000 7ff36f7fe640 1 -- 172.21.15.27:0/3837918986 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37400ea00 con 0x56104eb13ea0 2024-04-25T00:16:57.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.456+0000 7ff38e3bf740 1 -- 172.21.15.27:0/3837918986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13ea0 msgr2=0x56104eb1e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.456+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/3837918986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13ea0 0x56104eb1e030 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff374009c90 tx=0x7ff374011810 comp rx=0 tx=0).stop 2024-04-25T00:16:57.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.457+0000 7ff38e3bf740 1 -- 172.21.15.27:0/3837918986 shutdown_connections 2024-04-25T00:16:57.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.457+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/3837918986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb1e570 0x56104eb20960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.457+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/3837918986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13ea0 0x56104eb1e030 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.457+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/3837918986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb13580 0x56104eb13960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.457+0000 7ff38e3bf740 1 -- 172.21.15.27:0/3837918986 >> 172.21.15.27:0/3837918986 conn(0x56104eb0f8a0 msgr2=0x56104eb0fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.457+0000 7ff38e3bf740 1 -- 172.21.15.27:0/3837918986 shutdown_connections 2024-04-25T00:16:57.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.457+0000 7ff38e3bf740 1 -- 172.21.15.27:0/3837918986 wait complete. 2024-04-25T00:16:57.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.457+0000 7ff38e3bf740 1 -- start start 2024-04-25T00:16:57.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.457+0000 7ff38e3bf740 1 Processor -- start 2024-04-25T00:16:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.458+0000 7ff38e3bf740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb13580 0x56104eb1df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.458+0000 7ff38e3bf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13ea0 0x56104eb1b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.458+0000 7ff38e3bf740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb1e570 0x56104eb1b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.458+0000 7ff388c4e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13ea0 0x56104eb1b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.458+0000 7ff38e3bf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56104eb21360 con 0x56104eb13ea0 2024-04-25T00:16:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.458+0000 7ff388c4e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13ea0 0x56104eb1b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51494/0 (socket says 172.21.15.27:51494) 2024-04-25T00:16:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.458+0000 7ff388c4e640 1 -- 172.21.15.27:0/1551544369 learned_addr learned my addr 172.21.15.27:0/1551544369 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.458+0000 7ff38e3bf740 1 -- 172.21.15.27:0/1551544369 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56104ea34050 con 0x56104eb1e570 2024-04-25T00:16:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.459+0000 7ff38e3bf740 1 -- 172.21.15.27:0/1551544369 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56104eb214e0 con 0x56104eb13580 2024-04-25T00:16:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.459+0000 7ff38944f640 1 --2- 172.21.15.27:0/1551544369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb13580 0x56104eb1df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.459+0000 7ff389c50640 1 --2- 172.21.15.27:0/1551544369 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb1e570 0x56104eb1b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.459+0000 7ff388c4e640 1 -- 172.21.15.27:0/1551544369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb13580 msgr2=0x56104eb1df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.459+0000 7ff388c4e640 1 --2- 172.21.15.27:0/1551544369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb13580 0x56104eb1df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.459+0000 7ff388c4e640 1 -- 172.21.15.27:0/1551544369 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb1e570 msgr2=0x56104eb1b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.459+0000 7ff388c4e640 1 --2- 172.21.15.27:0/1551544369 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb1e570 0x56104eb1b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.459+0000 7ff388c4e640 1 -- 172.21.15.27:0/1551544369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56104ea05dc0 con 0x56104eb13ea0 2024-04-25T00:16:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.459+0000 7ff38e3bf740 10 client.0 _ll_register_callbacks cb 0x56104e78bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.459+0000 7ff38e3bf740 1 -- 172.21.15.27:0/1551544369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff374019b10 con 0x56104eb13ea0 2024-04-25T00:16:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.459+0000 7ff388c4e640 1 --2- 172.21.15.27:0/1551544369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13ea0 0x56104eb1b1f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7ff374019660 tx=0x7ff374019690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c159445c91cc6f0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.459+0000 7ff36cff9640 10 client.16092 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.459+0000 7ff36cff9640 1 -- 172.21.15.27:0/1551544369 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37401e040 con 0x56104eb13ea0 2024-04-25T00:16:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.459+0000 7ff36cff9640 1 -- 172.21.15.27:0/1551544369 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7ff374004030 con 0x56104eb13ea0 2024-04-25T00:16:57.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.460+0000 7ff36cff9640 1 -- 172.21.15.27:0/1551544369 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff374004270 con 0x56104eb13ea0 2024-04-25T00:16:57.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.460+0000 7ff38e3bf740 10 client.16092 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:57.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.460+0000 7ff38e3bf740 1 -- 172.21.15.27:0/1551544369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({fsmap=29}) -- 0x56104e78c3f0 con 0x56104eb13ea0 2024-04-25T00:16:57.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.460+0000 7ff36cff9640 1 -- 172.21.15.27:0/1551544369 <== mon.0 v2:172.21.15.27:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff374005040 con 0x56104eb13ea0 2024-04-25T00:16:57.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.460+0000 7ff38e3bf740 10 client.16092 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:57.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.460+0000 7ff38e3bf740 10 client.16092 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:16:57.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.460+0000 7ff38e3bf740 20 client.16092 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:57.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.460+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/1551544369 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56104eb2e330 0x56104eb30720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.460+0000 7ff38e3bf740 4 client.16092 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:16:57.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.460+0000 7ff38e3bf740 1 -- 172.21.15.27:0/1551544369 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56104ea85bb0 con 0x56104eb2e330 2024-04-25T00:16:57.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.460+0000 7ff38944f640 1 --2- 172.21.15.27:0/1551544369 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56104eb2e330 0x56104eb30720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.461+0000 7ff36cff9640 10 client.16092 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:57.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.461+0000 7ff38944f640 1 --2- 172.21.15.27:0/1551544369 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56104eb2e330 0x56104eb30720 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5660a9246e80b64 server_cookie=381b07fd0ce782fb in_seq=0 out_seq=0 2024-04-25T00:16:57.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.461+0000 7ff36cff9640 1 -- 172.21.15.27:0/1551544369 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56104ea85bb0 con 0x56104eb2e330 2024-04-25T00:16:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.461+0000 7ff36cff9640 10 client.16092 handle_command_reply: tid=0 2024-04-25T00:16:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.461+0000 7ff38e3bf740 1 client.16092 shutdown 2024-04-25T00:16:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.462+0000 7ff38e3bf740 1 -- 172.21.15.27:0/1551544369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13ea0 msgr2=0x56104eb1b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.462+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/1551544369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13ea0 0x56104eb1b1f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7ff374019660 tx=0x7ff374019690 comp rx=0 tx=0).stop 2024-04-25T00:16:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.462+0000 7ff38e3bf740 1 -- 172.21.15.27:0/1551544369 shutdown_connections 2024-04-25T00:16:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.462+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/1551544369 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56104eb2e330 0x56104eb30720 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.462+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/1551544369 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb1e570 0x56104eb1b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.462+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/1551544369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13ea0 0x56104eb1b1f0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.462+0000 7ff36cff9640 0 client.16092 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.462+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/1551544369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb13580 0x56104eb1df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.462+0000 7ff38e3bf740 1 -- 172.21.15.27:0/1551544369 >> 172.21.15.27:0/1551544369 conn(0x56104eb0f8a0 msgr2=0x56104eb0eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.462+0000 7ff38e3bf740 1 -- 172.21.15.27:0/1551544369 shutdown_connections 2024-04-25T00:16:57.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.462+0000 7ff38e3bf740 1 -- 172.21.15.27:0/1551544369 wait complete. 2024-04-25T00:16:57.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.462+0000 7ff38e3bf740 20 client.16092 trim_cache size 0 max 16384 2024-04-25T00:16:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff38e3bf740 1 Processor -- start 2024-04-25T00:16:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff38e3bf740 1 -- start start 2024-04-25T00:16:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff38e3bf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13580 0x56104eb2a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff38e3bf740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb13ea0 0x56104eb35e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff38e3bf740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb1e570 0x56104eb36350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff38e3bf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff3841b0e00 con 0x56104eb13580 2024-04-25T00:16:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff38e3bf740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff3841b2940 con 0x56104eb13ea0 2024-04-25T00:16:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff38e3bf740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff3841b14d0 con 0x56104eb1e570 2024-04-25T00:16:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff38944f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13580 0x56104eb2a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff38944f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13580 0x56104eb2a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51504/0 (socket says 172.21.15.27:51504) 2024-04-25T00:16:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff38944f640 1 -- 172.21.15.27:0/2849547334 learned_addr learned my addr 172.21.15.27:0/2849547334 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff388c4e640 1 --2- 172.21.15.27:0/2849547334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb13ea0 0x56104eb35e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff389c50640 1 --2- 172.21.15.27:0/2849547334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb1e570 0x56104eb36350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff38944f640 1 -- 172.21.15.27:0/2849547334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb1e570 msgr2=0x56104eb36350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff38944f640 1 --2- 172.21.15.27:0/2849547334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb1e570 0x56104eb36350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff38944f640 1 -- 172.21.15.27:0/2849547334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb13ea0 msgr2=0x56104eb35e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff38944f640 1 --2- 172.21.15.27:0/2849547334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb13ea0 0x56104eb35e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.476+0000 7ff38944f640 1 -- 172.21.15.27:0/2849547334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56104e8567d0 con 0x56104eb13580 2024-04-25T00:16:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.477+0000 7ff38944f640 1 --2- 172.21.15.27:0/2849547334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13580 0x56104eb2a220 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7ff380005900 tx=0x7ff38003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3569fc18acce72b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.477+0000 7ff36effd640 1 -- 172.21.15.27:0/2849547334 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff380058070 con 0x56104eb13580 2024-04-25T00:16:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.477+0000 7ff36effd640 1 -- 172.21.15.27:0/2849547334 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56104ea85bb0 con 0x56104eb13580 2024-04-25T00:16:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.477+0000 7ff36effd640 1 -- 172.21.15.27:0/2849547334 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff380035040 con 0x56104eb13580 2024-04-25T00:16:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.478+0000 7ff38e3bf740 1 -- 172.21.15.27:0/2849547334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13580 msgr2=0x56104eb2a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.478+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/2849547334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13580 0x56104eb2a220 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7ff380005900 tx=0x7ff38003f000 comp rx=0 tx=0).stop 2024-04-25T00:16:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.478+0000 7ff38e3bf740 1 -- 172.21.15.27:0/2849547334 shutdown_connections 2024-04-25T00:16:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.478+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/2849547334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb1e570 0x56104eb36350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.478+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/2849547334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb13ea0 0x56104eb35e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.478+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/2849547334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13580 0x56104eb2a220 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.478+0000 7ff38e3bf740 1 -- 172.21.15.27:0/2849547334 >> 172.21.15.27:0/2849547334 conn(0x56104eb0f8a0 msgr2=0x56104eb0eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.478+0000 7ff38e3bf740 1 -- 172.21.15.27:0/2849547334 shutdown_connections 2024-04-25T00:16:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.478+0000 7ff38e3bf740 1 -- 172.21.15.27:0/2849547334 wait complete. 2024-04-25T00:16:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.478+0000 7ff38e3bf740 1 -- start start 2024-04-25T00:16:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.478+0000 7ff38e3bf740 1 Processor -- start 2024-04-25T00:16:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff38e3bf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13580 0x56104eb26b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff38e3bf740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb13ea0 0x56104eb23f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff38e3bf740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb1e570 0x56104eb244a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff38e3bf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff3841b2270 con 0x56104eb13580 2024-04-25T00:16:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff38e3bf740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff3841b1ba0 con 0x56104eb1e570 2024-04-25T00:16:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff38e3bf740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff3841aedc0 con 0x56104eb13ea0 2024-04-25T00:16:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff38944f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13580 0x56104eb26b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff38944f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13580 0x56104eb26b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51516/0 (socket says 172.21.15.27:51516) 2024-04-25T00:16:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff38944f640 1 -- 172.21.15.27:0/322663713 learned_addr learned my addr 172.21.15.27:0/322663713 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff38e3bf740 10 client.0 _ll_register_callbacks cb 0x7ff37401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff388c4e640 1 --2- 172.21.15.27:0/322663713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb13ea0 0x56104eb23f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff38944f640 1 -- 172.21.15.27:0/322663713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb13ea0 msgr2=0x56104eb23f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff38944f640 1 --2- 172.21.15.27:0/322663713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb13ea0 0x56104eb23f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff38944f640 1 -- 172.21.15.27:0/322663713 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb1e570 msgr2=0x56104eb244a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff389c50640 1 --2- 172.21.15.27:0/322663713 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb1e570 0x56104eb244a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff38944f640 1 --2- 172.21.15.27:0/322663713 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb1e570 0x56104eb244a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff38944f640 1 -- 172.21.15.27:0/322663713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56104eb24cd0 con 0x56104eb13580 2024-04-25T00:16:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff388c4e640 1 --2- 172.21.15.27:0/322663713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb13ea0 0x56104eb23f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff389c50640 1 --2- 172.21.15.27:0/322663713 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb1e570 0x56104eb244a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.479+0000 7ff38944f640 1 --2- 172.21.15.27:0/322663713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13580 0x56104eb26b90 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff380000c00 tx=0x7ff38002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1fb37cda0337a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.480+0000 7ff36cff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.480+0000 7ff36cff9640 1 -- 172.21.15.27:0/322663713 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff380004450 con 0x56104eb13580 2024-04-25T00:16:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.480+0000 7ff36cff9640 1 -- 172.21.15.27:0/322663713 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7ff38005f980 con 0x56104eb13580 2024-04-25T00:16:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.480+0000 7ff38e3bf740 1 -- 172.21.15.27:0/322663713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56104ea85bb0 con 0x56104eb13580 2024-04-25T00:16:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.480+0000 7ff36cff9640 1 -- 172.21.15.27:0/322663713 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff38005fb80 con 0x56104eb13580 2024-04-25T00:16:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.480+0000 7ff38e3bf740 10 client.16098 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.480+0000 7ff38e3bf740 1 -- 172.21.15.27:0/322663713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56104eb94710 con 0x56104eb13580 2024-04-25T00:16:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.480+0000 7ff36cff9640 1 -- 172.21.15.27:0/322663713 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff380005510 con 0x56104eb13580 2024-04-25T00:16:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.480+0000 7ff36cff9640 1 -- 172.21.15.27:0/322663713 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff380004c60 con 0x56104eb13580 2024-04-25T00:16:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.480+0000 7ff38e3bf740 10 client.16098 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.480+0000 7ff38e3bf740 10 client.16098 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:16:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.480+0000 7ff38e3bf740 20 client.16098 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.480+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/322663713 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56104eb94dc0 0x56104eb971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.480+0000 7ff38e3bf740 4 client.16098 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:16:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.481+0000 7ff38e3bf740 1 -- 172.21.15.27:0/322663713 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56104ea85bb0 con 0x56104eb94dc0 2024-04-25T00:16:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.481+0000 7ff388c4e640 1 --2- 172.21.15.27:0/322663713 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56104eb94dc0 0x56104eb971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.481+0000 7ff388c4e640 1 --2- 172.21.15.27:0/322663713 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56104eb94dc0 0x56104eb971e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=809754c69c226453 server_cookie=193f36b4a544cadf in_seq=0 out_seq=0 2024-04-25T00:16:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.481+0000 7ff36cff9640 10 client.16098 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.481+0000 7ff36cff9640 1 -- 172.21.15.27:0/322663713 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56104ea85bb0 con 0x56104eb94dc0 2024-04-25T00:16:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.481+0000 7ff36cff9640 10 client.16098 handle_command_reply: tid=0 2024-04-25T00:16:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.481+0000 7ff38e3bf740 1 client.16098 shutdown 2024-04-25T00:16:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.481+0000 7ff38e3bf740 1 -- 172.21.15.27:0/322663713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13580 msgr2=0x56104eb26b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.481+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/322663713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13580 0x56104eb26b90 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff380000c00 tx=0x7ff38002a410 comp rx=0 tx=0).stop 2024-04-25T00:16:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.482+0000 7ff38e3bf740 1 -- 172.21.15.27:0/322663713 shutdown_connections 2024-04-25T00:16:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.482+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/322663713 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56104eb94dc0 0x56104eb971e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.482+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/322663713 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56104eb1e570 0x56104eb244a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.482+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/322663713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56104eb13ea0 0x56104eb23f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.482+0000 7ff36cff9640 0 client.16098 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.482+0000 7ff38e3bf740 1 --2- 172.21.15.27:0/322663713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56104eb13580 0x56104eb26b90 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.482+0000 7ff38e3bf740 1 -- 172.21.15.27:0/322663713 >> 172.21.15.27:0/322663713 conn(0x56104eb0f8a0 msgr2=0x56104eb29210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.482+0000 7ff38e3bf740 1 -- 172.21.15.27:0/322663713 shutdown_connections 2024-04-25T00:16:57.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.482+0000 7ff38e3bf740 1 -- 172.21.15.27:0/322663713 wait complete. 2024-04-25T00:16:57.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.482+0000 7ff38e3bf740 20 client.16098 trim_cache size 0 max 16384 2024-04-25T00:16:57.489 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:16:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.482+0000 7ff38b6b3640 1 -- 172.21.15.27:0/4025909078 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff3700a2eb0 msgr2=0x7ff3700a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.482+0000 7ff38b6b3640 1 --2- 172.21.15.27:0/4025909078 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff3700a2eb0 0x7ff3700a5370 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff3800096c0 tx=0x7ff380005e50 comp rx=0 tx=0).stop 2024-04-25T00:16:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.482+0000 7ff38b6b3640 1 -- 172.21.15.27:0/4025909078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff38410e0f0 msgr2=0x7ff38419ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.482+0000 7ff38b6b3640 1 --2- 172.21.15.27:0/4025909078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff38410e0f0 0x7ff38419ead0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ff37c005fd0 tx=0x7ff37c00edd0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.483+0000 7ff38b6b3640 1 -- 172.21.15.27:0/4025909078 shutdown_connections 2024-04-25T00:16:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.483+0000 7ff38b6b3640 1 --2- 172.21.15.27:0/4025909078 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff3700a2eb0 0x7ff3700a5370 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.483+0000 7ff38b6b3640 1 --2- 172.21.15.27:0/4025909078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff38410e0f0 0x7ff38419ead0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.483+0000 7ff38b6b3640 1 --2- 172.21.15.27:0/4025909078 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff384100e60 0x7ff38419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.483+0000 7ff38b6b3640 1 --2- 172.21.15.27:0/4025909078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff384100540 0x7ff38419a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.483+0000 7ff38b6b3640 1 -- 172.21.15.27:0/4025909078 >> 172.21.15.27:0/4025909078 conn(0x7ff384077740 msgr2=0x7ff38410f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.483+0000 7ff38b6b3640 1 -- 172.21.15.27:0/4025909078 shutdown_connections 2024-04-25T00:16:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:57.483+0000 7ff38b6b3640 1 -- 172.21.15.27:0/4025909078 wait complete. 2024-04-25T00:16:57.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:57 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/753112718' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:16:57.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:57 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/753112718' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:16:57.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:57 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/753112718' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:16:58.496 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T00:16:58.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:16:58.497 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T00:16:58.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.663+0000 7f32fc55c640 1 -- 172.21.15.27:0/1591513826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f41029c0 msgr2=0x7f32f4102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.663+0000 7f32fc55c640 1 --2- 172.21.15.27:0/1591513826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f41029c0 0x7f32f4102da0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f32f0009900 tx=0x7f32f002a510 comp rx=0 tx=0).stop 2024-04-25T00:16:58.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.663+0000 7f32fc55c640 1 -- 172.21.15.27:0/1591513826 shutdown_connections 2024-04-25T00:16:58.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.663+0000 7f32fc55c640 1 --2- 172.21.15.27:0/1591513826 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32f410bd00 0x7f32f410e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.663+0000 7f32fc55c640 1 --2- 172.21.15.27:0/1591513826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f41032e0 0x7f32f410b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.663+0000 7f32fc55c640 1 --2- 172.21.15.27:0/1591513826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f41029c0 0x7f32f4102da0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.663+0000 7f32fc55c640 1 -- 172.21.15.27:0/1591513826 >> 172.21.15.27:0/1591513826 conn(0x7f32f4077740 msgr2=0x7f32f4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:58.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.663+0000 7f32fc55c640 1 -- 172.21.15.27:0/1591513826 shutdown_connections 2024-04-25T00:16:58.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.663+0000 7f32fc55c640 1 -- 172.21.15.27:0/1591513826 wait complete. 2024-04-25T00:16:58.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32fc55c640 1 Processor -- start 2024-04-25T00:16:58.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32fc55c640 1 -- start start 2024-04-25T00:16:58.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32fc55c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f41029c0 0x7f32f4198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32fc55c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f41032e0 0x7f32f4198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32fc55c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32f410bd00 0x7f32f419c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32fc55c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f32f4111b00 con 0x7f32f41029c0 2024-04-25T00:16:58.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32fc55c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f32f4111980 con 0x7f32f410bd00 2024-04-25T00:16:58.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32fc55c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f32f4111c80 con 0x7f32f41032e0 2024-04-25T00:16:58.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32fa2f8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f41029c0 0x7f32f4198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32fa2f8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f41029c0 0x7f32f4198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51538/0 (socket says 172.21.15.27:51538) 2024-04-25T00:16:58.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32fa2f8640 1 -- 172.21.15.27:0/3734362658 learned_addr learned my addr 172.21.15.27:0/3734362658 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:58.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32f9af7640 1 --2- 172.21.15.27:0/3734362658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f41032e0 0x7f32f4198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32faaf9640 1 --2- 172.21.15.27:0/3734362658 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32f410bd00 0x7f32f419c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32fa2f8640 1 -- 172.21.15.27:0/3734362658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f41032e0 msgr2=0x7f32f4198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32fa2f8640 1 --2- 172.21.15.27:0/3734362658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f41032e0 0x7f32f4198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32fa2f8640 1 -- 172.21.15.27:0/3734362658 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32f410bd00 msgr2=0x7f32f419c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32fa2f8640 1 --2- 172.21.15.27:0/3734362658 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32f410bd00 0x7f32f419c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32fa2f8640 1 -- 172.21.15.27:0/3734362658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f419cff0 con 0x7f32f41029c0 2024-04-25T00:16:58.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.664+0000 7f32f9af7640 1 --2- 172.21.15.27:0/3734362658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f41032e0 0x7f32f4198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:58.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.665+0000 7f32fa2f8640 1 --2- 172.21.15.27:0/3734362658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f41029c0 0x7f32f4198000 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f32f0006010 tx=0x7f32f0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:58.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.665+0000 7f32eb7fe640 1 -- 172.21.15.27:0/3734362658 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f003f030 con 0x7f32f41029c0 2024-04-25T00:16:58.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.665+0000 7f32fc55c640 1 -- 172.21.15.27:0/3734362658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32f419d280 con 0x7f32f41029c0 2024-04-25T00:16:58.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.665+0000 7f32fc55c640 1 -- 172.21.15.27:0/3734362658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32f419d6f0 con 0x7f32f41029c0 2024-04-25T00:16:58.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.665+0000 7f32eb7fe640 1 -- 172.21.15.27:0/3734362658 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32f0004440 con 0x7f32f41029c0 2024-04-25T00:16:58.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.665+0000 7f32eb7fe640 1 -- 172.21.15.27:0/3734362658 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f0035070 con 0x7f32f41029c0 2024-04-25T00:16:58.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.668+0000 7f32eb7fe640 1 -- 172.21.15.27:0/3734362658 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f32f0042030 con 0x7f32f41029c0 2024-04-25T00:16:58.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.668+0000 7f32ff25f740 1 Processor -- start 2024-04-25T00:16:58.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.668+0000 7f32ff25f740 1 -- start start 2024-04-25T00:16:58.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32ff25f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f38980 0x564b43f36cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32ff25f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f37290 0x564b43f37710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32ff25f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f42010 0x564b43f44400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32ff25f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564b43e9b2c0 con 0x564b43f37290 2024-04-25T00:16:58.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32ff25f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564b43e43910 con 0x564b43f42010 2024-04-25T00:16:58.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32ff25f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564b43e97280 con 0x564b43f38980 2024-04-25T00:16:58.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32fa2f8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f37290 0x564b43f37710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32faaf9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f38980 0x564b43f36cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32faaf9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f38980 0x564b43f36cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:47542/0 (socket says 172.21.15.27:47542) 2024-04-25T00:16:58.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32faaf9640 1 -- 172.21.15.27:0/3448575023 learned_addr learned my addr 172.21.15.27:0/3448575023 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:58.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32f9af7640 1 --2- 172.21.15.27:0/3448575023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f42010 0x564b43f44400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32eb7fe640 1 --2- 172.21.15.27:0/3734362658 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f32e00a2f30 0x7f32e00a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32faaf9640 1 -- 172.21.15.27:0/3448575023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f42010 msgr2=0x564b43f44400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:58.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32faaf9640 1 --2- 172.21.15.27:0/3448575023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f42010 0x564b43f44400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32faaf9640 1 -- 172.21.15.27:0/3448575023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f37290 msgr2=0x564b43f37710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32faaf9640 1 --2- 172.21.15.27:0/3448575023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f37290 0x564b43f37710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32faaf9640 1 -- 172.21.15.27:0/3448575023 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b43e54fb0 con 0x564b43f38980 2024-04-25T00:16:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32eb7fe640 1 -- 172.21.15.27:0/3734362658 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f32f00efd90 con 0x7f32f41029c0 2024-04-25T00:16:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.669+0000 7f32faaf9640 1 --2- 172.21.15.27:0/3734362658 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f32e00a2f30 0x7f32e00a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.670+0000 7f32fa2f8640 1 --2- 172.21.15.27:0/3448575023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f37290 0x564b43f37710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.670+0000 7f32faaf9640 1 --2- 172.21.15.27:0/3448575023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f38980 0x564b43f36cc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f32ec009fb0 tx=0x7f32ec00f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cdbe02bfb460208 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.670+0000 7f32faaf9640 1 --2- 172.21.15.27:0/3734362658 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f32e00a2f30 0x7f32e00a53f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f32ec020000 tx=0x7f32ec01d000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.670+0000 7f32e8bfe640 1 -- 172.21.15.27:0/3448575023 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec004b20 con 0x564b43f38980 2024-04-25T00:16:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.670+0000 7f32e8bfe640 1 -- 172.21.15.27:0/3448575023 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32ec004d80 con 0x564b43f38980 2024-04-25T00:16:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.670+0000 7f32e8bfe640 1 -- 172.21.15.27:0/3448575023 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec01c8c0 con 0x564b43f38980 2024-04-25T00:16:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.671+0000 7f32ff25f740 1 -- 172.21.15.27:0/3448575023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f38980 msgr2=0x564b43f36cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.671+0000 7f32ff25f740 1 --2- 172.21.15.27:0/3448575023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f38980 0x564b43f36cc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f32ec009fb0 tx=0x7f32ec00f7f0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.671+0000 7f32ff25f740 1 -- 172.21.15.27:0/3448575023 shutdown_connections 2024-04-25T00:16:58.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.671+0000 7f32ff25f740 1 --2- 172.21.15.27:0/3448575023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f42010 0x564b43f44400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.671+0000 7f32ff25f740 1 --2- 172.21.15.27:0/3448575023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f37290 0x564b43f37710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.671+0000 7f32ff25f740 1 --2- 172.21.15.27:0/3448575023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f38980 0x564b43f36cc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.671+0000 7f32ff25f740 1 -- 172.21.15.27:0/3448575023 >> 172.21.15.27:0/3448575023 conn(0x564b43eff950 msgr2=0x564b43f337f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:58.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.671+0000 7f32ff25f740 1 -- 172.21.15.27:0/3448575023 shutdown_connections 2024-04-25T00:16:58.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.671+0000 7f32ff25f740 1 -- 172.21.15.27:0/3448575023 wait complete. 2024-04-25T00:16:58.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.671+0000 7f32ff25f740 1 -- start start 2024-04-25T00:16:58.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.672+0000 7f32ff25f740 1 Processor -- start 2024-04-25T00:16:58.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.672+0000 7f32ff25f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f37290 0x564b43f3eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.672+0000 7f32ff25f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f38980 0x564b43f3f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.672+0000 7f32ff25f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f42010 0x564b43f3f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.672+0000 7f32ff25f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564b43f44ea0 con 0x564b43f42010 2024-04-25T00:16:58.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.672+0000 7f32ff25f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564b43e56e90 con 0x564b43f38980 2024-04-25T00:16:58.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.672+0000 7f32ff25f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564b43f45020 con 0x564b43f37290 2024-04-25T00:16:58.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.672+0000 7f32ff25f740 10 client.0 _ll_register_callbacks cb 0x564b43e92970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:58.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.672+0000 7f32fa2f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f37290 0x564b43f3eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.672+0000 7f32fa2f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f37290 0x564b43f3eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:47546/0 (socket says 172.21.15.27:47546) 2024-04-25T00:16:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.672+0000 7f32fa2f8640 1 -- 172.21.15.27:0/2888876258 learned_addr learned my addr 172.21.15.27:0/2888876258 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.672+0000 7f32f9af7640 1 --2- 172.21.15.27:0/2888876258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f38980 0x564b43f3f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.672+0000 7f32faaf9640 1 --2- 172.21.15.27:0/2888876258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f42010 0x564b43f3f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.672+0000 7f32f9af7640 1 -- 172.21.15.27:0/2888876258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f37290 msgr2=0x564b43f3eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.672+0000 7f32f9af7640 1 --2- 172.21.15.27:0/2888876258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f37290 0x564b43f3eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.672+0000 7f32f9af7640 1 -- 172.21.15.27:0/2888876258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f42010 msgr2=0x564b43f3f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.673+0000 7f32f9af7640 1 --2- 172.21.15.27:0/2888876258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f42010 0x564b43f3f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.673+0000 7f32f9af7640 1 -- 172.21.15.27:0/2888876258 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b43c777e0 con 0x564b43f38980 2024-04-25T00:16:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.673+0000 7f32fa2f8640 1 --2- 172.21.15.27:0/2888876258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f37290 0x564b43f3eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.673+0000 7f32faaf9640 1 --2- 172.21.15.27:0/2888876258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f42010 0x564b43f3f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.673+0000 7f32f9af7640 1 --2- 172.21.15.27:0/2888876258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f38980 0x564b43f3f3e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f32e4012680 tx=0x7f32e4012b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed6391e392735be6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.673+0000 7f32ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T00:16:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.673+0000 7f32ddffb640 1 -- 172.21.15.27:0/2888876258 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e4011c70 con 0x564b43f38980 2024-04-25T00:16:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.673+0000 7f32ff25f740 1 -- 172.21.15.27:0/2888876258 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32ec004d80 con 0x564b43f38980 2024-04-25T00:16:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.673+0000 7f32ddffb640 1 -- 172.21.15.27:0/2888876258 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f32e4011e10 con 0x564b43f38980 2024-04-25T00:16:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.673+0000 7f32ddffb640 1 -- 172.21.15.27:0/2888876258 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f32e4005020 con 0x564b43f38980 2024-04-25T00:16:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.673+0000 7f32ff25f740 10 client.25724 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.673+0000 7f32ff25f740 1 -- 172.21.15.27:0/2888876258 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564b43cd7f70 con 0x564b43f38980 2024-04-25T00:16:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.673+0000 7f32ddffb640 1 -- 172.21.15.27:0/2888876258 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32ec004d80 con 0x564b43f38980 2024-04-25T00:16:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.674+0000 7f32ddffb640 1 -- 172.21.15.27:0/2888876258 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f32e401c470 con 0x564b43f38980 2024-04-25T00:16:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.674+0000 7f32ff25f740 10 client.25724 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.674+0000 7f32ff25f740 10 client.25724 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:16:58.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.674+0000 7f32ff25f740 20 client.25724 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:58.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.674+0000 7f32ff25f740 1 --2- 172.21.15.27:0/2888876258 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564b43f52140 0x564b43f545a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.674+0000 7f32ff25f740 4 client.25724 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:16:58.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.674+0000 7f32ff25f740 1 -- 172.21.15.27:0/2888876258 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b43e6f5a0 con 0x564b43f52140 2024-04-25T00:16:58.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.674+0000 7f32fa2f8640 1 --2- 172.21.15.27:0/2888876258 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564b43f52140 0x564b43f545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.674+0000 7f32ddffb640 10 client.25724 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:58.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.674+0000 7f32fa2f8640 1 --2- 172.21.15.27:0/2888876258 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564b43f52140 0x564b43f545a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bece2fa0aa151a0 server_cookie=de297552ef08d75d in_seq=0 out_seq=0 2024-04-25T00:16:58.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.675+0000 7f32ddffb640 1 -- 172.21.15.27:0/2888876258 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564b43e6f5a0 con 0x564b43f52140 2024-04-25T00:16:58.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.675+0000 7f32ddffb640 10 client.25724 handle_command_reply: tid=0 2024-04-25T00:16:58.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.675+0000 7f32ff25f740 1 client.25724 shutdown 2024-04-25T00:16:58.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.675+0000 7f32ff25f740 1 -- 172.21.15.27:0/2888876258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f38980 msgr2=0x564b43f3f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.675+0000 7f32ff25f740 1 --2- 172.21.15.27:0/2888876258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f38980 0x564b43f3f3e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f32e4012680 tx=0x7f32e4012b40 comp rx=0 tx=0).stop 2024-04-25T00:16:58.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.675+0000 7f32ff25f740 1 -- 172.21.15.27:0/2888876258 shutdown_connections 2024-04-25T00:16:58.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.675+0000 7f32ff25f740 1 --2- 172.21.15.27:0/2888876258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f42010 0x564b43f3f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.675+0000 7f32ff25f740 1 --2- 172.21.15.27:0/2888876258 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564b43f52140 0x564b43f545a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.675+0000 7f32ff25f740 1 --2- 172.21.15.27:0/2888876258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f38980 0x564b43f3f3e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.675+0000 7f32ddffb640 0 client.25724 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:58.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.675+0000 7f32ff25f740 1 --2- 172.21.15.27:0/2888876258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f37290 0x564b43f3eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.675+0000 7f32ff25f740 1 -- 172.21.15.27:0/2888876258 >> 172.21.15.27:0/2888876258 conn(0x564b43eff950 msgr2=0x564b43f38f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:58.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.675+0000 7f32ff25f740 1 -- 172.21.15.27:0/2888876258 shutdown_connections 2024-04-25T00:16:58.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.676+0000 7f32ff25f740 1 -- 172.21.15.27:0/2888876258 wait complete. 2024-04-25T00:16:58.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.676+0000 7f32ff25f740 20 client.25724 trim_cache size 0 max 16384 2024-04-25T00:16:58.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:58 smithi027 ceph-mon[29208]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T00:16:58.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.701+0000 7f32ff25f740 1 Processor -- start 2024-04-25T00:16:58.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.702+0000 7f32ff25f740 1 -- start start 2024-04-25T00:16:58.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.702+0000 7f32ff25f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f37290 0x564b43f4de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.702+0000 7f32ff25f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f38980 0x564b43f3de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.702+0000 7f32ff25f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f42010 0x564b43f59e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.702+0000 7f32ff25f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f32d800fe80 con 0x564b43f38980 2024-04-25T00:16:58.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.702+0000 7f32ff25f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f32d80119c0 con 0x564b43f42010 2024-04-25T00:16:58.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.702+0000 7f32ff25f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f32d8010550 con 0x564b43f37290 2024-04-25T00:16:58.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.702+0000 7f32f9af7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f38980 0x564b43f3de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.702+0000 7f32f9af7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f38980 0x564b43f3de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51572/0 (socket says 172.21.15.27:51572) 2024-04-25T00:16:58.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.702+0000 7f32f9af7640 1 -- 172.21.15.27:0/325598721 learned_addr learned my addr 172.21.15.27:0/325598721 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:58.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.702+0000 7f32fa2f8640 1 --2- 172.21.15.27:0/325598721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f37290 0x564b43f4de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.702+0000 7f32f9af7640 1 -- 172.21.15.27:0/325598721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f37290 msgr2=0x564b43f4de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:58.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.702+0000 7f32f9af7640 1 --2- 172.21.15.27:0/325598721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f37290 0x564b43f4de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.702+0000 7f32f9af7640 1 -- 172.21.15.27:0/325598721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f42010 msgr2=0x564b43f59e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:58.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.702+0000 7f32f9af7640 1 --2- 172.21.15.27:0/325598721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f42010 0x564b43f59e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.702+0000 7f32f9af7640 1 -- 172.21.15.27:0/325598721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b43b6f170 con 0x564b43f38980 2024-04-25T00:16:58.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.703+0000 7f32f9af7640 1 --2- 172.21.15.27:0/325598721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f38980 0x564b43f3de80 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f32e401caa0 tx=0x7f32e4009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b7ae65dd5690f03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:58.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.703+0000 7f32dffff640 1 -- 172.21.15.27:0/325598721 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e4011d30 con 0x564b43f38980 2024-04-25T00:16:58.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.703+0000 7f32dffff640 1 -- 172.21.15.27:0/325598721 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32e401c470 con 0x564b43f38980 2024-04-25T00:16:58.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.703+0000 7f32dffff640 1 -- 172.21.15.27:0/325598721 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e4004660 con 0x564b43f38980 2024-04-25T00:16:58.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.704+0000 7f32ff25f740 1 -- 172.21.15.27:0/325598721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f38980 msgr2=0x564b43f3de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.704+0000 7f32ff25f740 1 --2- 172.21.15.27:0/325598721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f38980 0x564b43f3de80 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f32e401caa0 tx=0x7f32e4009ed0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.704+0000 7f32ff25f740 1 -- 172.21.15.27:0/325598721 shutdown_connections 2024-04-25T00:16:58.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.704+0000 7f32ff25f740 1 --2- 172.21.15.27:0/325598721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f42010 0x564b43f59e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.704+0000 7f32ff25f740 1 --2- 172.21.15.27:0/325598721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f38980 0x564b43f3de80 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.704+0000 7f32ff25f740 1 --2- 172.21.15.27:0/325598721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f37290 0x564b43f4de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.704+0000 7f32ff25f740 1 -- 172.21.15.27:0/325598721 >> 172.21.15.27:0/325598721 conn(0x564b43eff950 msgr2=0x564b43f3d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:58.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.704+0000 7f32ff25f740 1 -- 172.21.15.27:0/325598721 shutdown_connections 2024-04-25T00:16:58.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.704+0000 7f32ff25f740 1 -- 172.21.15.27:0/325598721 wait complete. 2024-04-25T00:16:58.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.705+0000 7f32ff25f740 1 -- start start 2024-04-25T00:16:58.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.705+0000 7f32ff25f740 1 Processor -- start 2024-04-25T00:16:58.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.705+0000 7f32ff25f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f37290 0x564b43f4a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.705+0000 7f32ff25f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f38980 0x564b43f42820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.705+0000 7f32ff25f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f42010 0x564b43f42d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.706+0000 7f32faaf9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f42010 0x564b43f42d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.706+0000 7f32faaf9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f42010 0x564b43f42d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51584/0 (socket says 172.21.15.27:51584) 2024-04-25T00:16:58.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.706+0000 7f32fa2f8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f37290 0x564b43f4a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.706+0000 7f32faaf9640 1 -- 172.21.15.27:0/3574758291 learned_addr learned my addr 172.21.15.27:0/3574758291 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:58.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.706+0000 7f32f9af7640 1 --2- 172.21.15.27:0/3574758291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f38980 0x564b43f42820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.706+0000 7f32ff25f740 1 -- 172.21.15.27:0/3574758291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f32d80112f0 con 0x564b43f42010 2024-04-25T00:16:58.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.706+0000 7f32ff25f740 1 -- 172.21.15.27:0/3574758291 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f32d8010c20 con 0x564b43f37290 2024-04-25T00:16:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.706+0000 7f32ff25f740 1 -- 172.21.15.27:0/3574758291 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f32d800de40 con 0x564b43f38980 2024-04-25T00:16:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.706+0000 7f32ff25f740 10 client.0 _ll_register_callbacks cb 0x7f32e4009860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.706+0000 7f32faaf9640 1 -- 172.21.15.27:0/3574758291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f38980 msgr2=0x564b43f42820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.706+0000 7f32faaf9640 1 --2- 172.21.15.27:0/3574758291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f38980 0x564b43f42820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.706+0000 7f32faaf9640 1 -- 172.21.15.27:0/3574758291 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f37290 msgr2=0x564b43f4a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.706+0000 7f32faaf9640 1 --2- 172.21.15.27:0/3574758291 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f37290 0x564b43f4a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.706+0000 7f32faaf9640 1 -- 172.21.15.27:0/3574758291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b43f43590 con 0x564b43f42010 2024-04-25T00:16:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.706+0000 7f32fa2f8640 1 --2- 172.21.15.27:0/3574758291 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f37290 0x564b43f4a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.706+0000 7f32faaf9640 1 --2- 172.21.15.27:0/3574758291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f42010 0x564b43f42d60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f32ec01c510 tx=0x7f32ec009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22e3e360333a80d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.706+0000 7f32ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.707+0000 7f32ddffb640 1 -- 172.21.15.27:0/3574758291 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec01c540 con 0x564b43f42010 2024-04-25T00:16:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.707+0000 7f32ddffb640 1 -- 172.21.15.27:0/3574758291 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f32ec01c6e0 con 0x564b43f42010 2024-04-25T00:16:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.707+0000 7f32ff25f740 1 -- 172.21.15.27:0/3574758291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32e401c470 con 0x564b43f42010 2024-04-25T00:16:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.707+0000 7f32ddffb640 1 -- 172.21.15.27:0/3574758291 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f32ec03d050 con 0x564b43f42010 2024-04-25T00:16:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.707+0000 7f32ff25f740 10 client.16116 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.707+0000 7f32ff25f740 1 -- 172.21.15.27:0/3574758291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564b43fb86e0 con 0x564b43f42010 2024-04-25T00:16:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.707+0000 7f32ddffb640 1 -- 172.21.15.27:0/3574758291 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32ec034070 con 0x564b43f42010 2024-04-25T00:16:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.707+0000 7f32ddffb640 1 -- 172.21.15.27:0/3574758291 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f32ec005650 con 0x564b43f42010 2024-04-25T00:16:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.707+0000 7f32ff25f740 10 client.16116 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.707+0000 7f32ff25f740 10 client.16116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:16:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.708+0000 7f32ff25f740 20 client.16116 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.708+0000 7f32ff25f740 1 --2- 172.21.15.27:0/3574758291 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564b43f52140 0x564b43fbacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.708+0000 7f32ff25f740 4 client.16116 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T00:16:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.708+0000 7f32ff25f740 1 -- 172.21.15.27:0/3574758291 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f32e401c470 con 0x564b43f52140 2024-04-25T00:16:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.708+0000 7f32fa2f8640 1 --2- 172.21.15.27:0/3574758291 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564b43f52140 0x564b43fbacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.708+0000 7f32ddffb640 10 client.16116 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.708+0000 7f32fa2f8640 1 --2- 172.21.15.27:0/3574758291 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564b43f52140 0x564b43fbacf0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4710c4521f834ef server_cookie=c023cf615d7bea9d in_seq=0 out_seq=0 2024-04-25T00:16:58.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.709+0000 7f32ddffb640 1 -- 172.21.15.27:0/3574758291 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f32e401c470 con 0x564b43f52140 2024-04-25T00:16:58.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.709+0000 7f32ddffb640 10 client.16116 handle_command_reply: tid=0 2024-04-25T00:16:58.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.709+0000 7f32ff25f740 1 client.16116 shutdown 2024-04-25T00:16:58.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.710+0000 7f32ff25f740 1 -- 172.21.15.27:0/3574758291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f42010 msgr2=0x564b43f42d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.710+0000 7f32ff25f740 1 --2- 172.21.15.27:0/3574758291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f42010 0x564b43f42d60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f32ec01c510 tx=0x7f32ec009bf0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.710+0000 7f32ff25f740 1 -- 172.21.15.27:0/3574758291 shutdown_connections 2024-04-25T00:16:58.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.710+0000 7f32ff25f740 1 --2- 172.21.15.27:0/3574758291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564b43f42010 0x564b43f42d60 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.710+0000 7f32ff25f740 1 --2- 172.21.15.27:0/3574758291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564b43f38980 0x564b43f42820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.710+0000 7f32ff25f740 1 --2- 172.21.15.27:0/3574758291 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564b43f52140 0x564b43fbacf0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.710+0000 7f32ff25f740 1 --2- 172.21.15.27:0/3574758291 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564b43f37290 0x564b43f4a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.710+0000 7f32ff25f740 1 -- 172.21.15.27:0/3574758291 >> 172.21.15.27:0/3574758291 conn(0x564b43eff950 msgr2=0x564b43f3e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:58.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.710+0000 7f32ddffb640 0 client.16116 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:58.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.710+0000 7f32ff25f740 1 -- 172.21.15.27:0/3574758291 shutdown_connections 2024-04-25T00:16:58.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.710+0000 7f32ff25f740 1 -- 172.21.15.27:0/3574758291 wait complete. 2024-04-25T00:16:58.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.710+0000 7f32ff25f740 20 client.16116 trim_cache size 0 max 16384 2024-04-25T00:16:58.715 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:16:58.715 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T00:16:58.715 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "c8feec42-54b9-4da2-8629-87ce7cff7af1", 2024-04-25T00:16:58.715 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T00:16:58.716 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:16:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.710+0000 7f32fc55c640 1 -- 172.21.15.27:0/3734362658 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f32e00a2f30 msgr2=0x7f32e00a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.710+0000 7f32fc55c640 1 --2- 172.21.15.27:0/3734362658 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f32e00a2f30 0x7f32e00a53f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f32ec020000 tx=0x7f32ec01d000 comp rx=0 tx=0).stop 2024-04-25T00:16:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.710+0000 7f32fc55c640 1 -- 172.21.15.27:0/3734362658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f41029c0 msgr2=0x7f32f4198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.710+0000 7f32fc55c640 1 --2- 172.21.15.27:0/3734362658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f41029c0 0x7f32f4198000 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f32f0006010 tx=0x7f32f0004040 comp rx=0 tx=0).stop 2024-04-25T00:16:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.711+0000 7f32fc55c640 1 -- 172.21.15.27:0/3734362658 shutdown_connections 2024-04-25T00:16:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.711+0000 7f32fc55c640 1 --2- 172.21.15.27:0/3734362658 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f32e00a2f30 0x7f32e00a53f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.711+0000 7f32fc55c640 1 --2- 172.21.15.27:0/3734362658 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32f410bd00 0x7f32f419c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.711+0000 7f32fc55c640 1 --2- 172.21.15.27:0/3734362658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f41032e0 0x7f32f4198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.711+0000 7f32fc55c640 1 --2- 172.21.15.27:0/3734362658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f41029c0 0x7f32f4198000 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.711+0000 7f32fc55c640 1 -- 172.21.15.27:0/3734362658 >> 172.21.15.27:0/3734362658 conn(0x7f32f4077740 msgr2=0x7f32f40fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.711+0000 7f32fc55c640 1 -- 172.21.15.27:0/3734362658 shutdown_connections 2024-04-25T00:16:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.711+0000 7f32fc55c640 1 -- 172.21.15.27:0/3734362658 wait complete. 2024-04-25T00:16:58.723 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:16:58.724 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:16:58.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:58 smithi139 ceph-mon[32833]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T00:16:58.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:58 smithi153 ceph-mon[32896]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T00:16:58.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.828+0000 7f67bbe91640 1 -- 172.21.15.27:0/2301064279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67b41066b0 msgr2=0x7f67b4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.828+0000 7f67bbe91640 1 --2- 172.21.15.27:0/2301064279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67b41066b0 0x7f67b4106a90 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f67b00099b0 tx=0x7f67b002e850 comp rx=0 tx=0).stop 2024-04-25T00:16:58.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.828+0000 7f67bbe91640 1 -- 172.21.15.27:0/2301064279 shutdown_connections 2024-04-25T00:16:58.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.828+0000 7f67bbe91640 1 --2- 172.21.15.27:0/2301064279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67b4100fd0 0x7f67b410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.828+0000 7f67bbe91640 1 --2- 172.21.15.27:0/2301064279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b4100610 0x7f67b4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.828+0000 7f67bbe91640 1 --2- 172.21.15.27:0/2301064279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67b41066b0 0x7f67b4106a90 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.828+0000 7f67bbe91640 1 -- 172.21.15.27:0/2301064279 >> 172.21.15.27:0/2301064279 conn(0x7f67b4077740 msgr2=0x7f67b4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:58.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.828+0000 7f67bbe91640 1 -- 172.21.15.27:0/2301064279 shutdown_connections 2024-04-25T00:16:58.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.829+0000 7f67bbe91640 1 -- 172.21.15.27:0/2301064279 wait complete. 2024-04-25T00:16:58.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.829+0000 7f67bbe91640 1 Processor -- start 2024-04-25T00:16:58.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.829+0000 7f67bbe91640 1 -- start start 2024-04-25T00:16:58.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.830+0000 7f67bbe91640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67b4100610 0x7f67b4195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.830+0000 7f67bbe91640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b4100fd0 0x7f67b41962c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.830+0000 7f67bbe91640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67b41066b0 0x7f67b419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.830+0000 7f67b9c2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67b4100610 0x7f67b4195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.830+0000 7f67b9c2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67b4100610 0x7f67b4195d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51604/0 (socket says 172.21.15.27:51604) 2024-04-25T00:16:58.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.830+0000 7f67b9c2d640 1 -- 172.21.15.27:0/3036684021 learned_addr learned my addr 172.21.15.27:0/3036684021 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:58.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.830+0000 7f67b942c640 1 --2- 172.21.15.27:0/3036684021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b4100fd0 0x7f67b41962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.830+0000 7f67bbe91640 1 -- 172.21.15.27:0/3036684021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f67b410f920 con 0x7f67b4100610 2024-04-25T00:16:58.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.830+0000 7f67bbe91640 1 -- 172.21.15.27:0/3036684021 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f67b410f7a0 con 0x7f67b41066b0 2024-04-25T00:16:58.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.830+0000 7f67bbe91640 1 -- 172.21.15.27:0/3036684021 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f67b410faa0 con 0x7f67b4100fd0 2024-04-25T00:16:58.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.830+0000 7f67ba42e640 1 --2- 172.21.15.27:0/3036684021 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67b41066b0 0x7f67b419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.830+0000 7f67b9c2d640 1 -- 172.21.15.27:0/3036684021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b4100fd0 msgr2=0x7f67b41962c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.830+0000 7f67b9c2d640 1 --2- 172.21.15.27:0/3036684021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b4100fd0 0x7f67b41962c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.830+0000 7f67b9c2d640 1 -- 172.21.15.27:0/3036684021 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67b41066b0 msgr2=0x7f67b419a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.830+0000 7f67b9c2d640 1 --2- 172.21.15.27:0/3036684021 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67b41066b0 0x7f67b419a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.830+0000 7f67b9c2d640 1 -- 172.21.15.27:0/3036684021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b419ad70 con 0x7f67b4100610 2024-04-25T00:16:58.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.830+0000 7f67ba42e640 1 --2- 172.21.15.27:0/3036684021 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67b41066b0 0x7f67b419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:58.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.831+0000 7f67b9c2d640 1 --2- 172.21.15.27:0/3036684021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67b4100610 0x7f67b4195d80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f67b0009230 tx=0x7f67b0035420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:58.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.831+0000 7f67aaffd640 1 -- 172.21.15.27:0/3036684021 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b0046070 con 0x7f67b4100610 2024-04-25T00:16:58.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.831+0000 7f67bbe91640 1 -- 172.21.15.27:0/3036684021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67b419b000 con 0x7f67b4100610 2024-04-25T00:16:58.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.831+0000 7f67aaffd640 1 -- 172.21.15.27:0/3036684021 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67b0036400 con 0x7f67b4100610 2024-04-25T00:16:58.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.831+0000 7f67aaffd640 1 -- 172.21.15.27:0/3036684021 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b0004920 con 0x7f67b4100610 2024-04-25T00:16:58.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.831+0000 7f67bbe91640 1 -- 172.21.15.27:0/3036684021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67b41a28f0 con 0x7f67b4100610 2024-04-25T00:16:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.834+0000 7f67beb95740 1 Processor -- start 2024-04-25T00:16:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.835+0000 7f67beb95740 1 -- start start 2024-04-25T00:16:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.835+0000 7f67beb95740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562dae40 0x55b1562db220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.835+0000 7f67beb95740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562db760 0x55b1562e58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.835+0000 7f67beb95740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562e5e30 0x55b1562e8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.835+0000 7f67beb95740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b1562376b0 con 0x55b1562e5e30 2024-04-25T00:16:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.835+0000 7f67beb95740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b1561df880 con 0x55b1562db760 2024-04-25T00:16:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.835+0000 7f67beb95740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b156233670 con 0x55b1562dae40 2024-04-25T00:16:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.835+0000 7f67aaffd640 1 -- 172.21.15.27:0/3036684021 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f67b0049070 con 0x7f67b4100610 2024-04-25T00:16:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.835+0000 7f67ba42e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562dae40 0x55b1562db220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.835+0000 7f67b942c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562e5e30 0x55b1562e8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.835+0000 7f67ba42e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562dae40 0x55b1562db220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:47616/0 (socket says 172.21.15.27:47616) 2024-04-25T00:16:58.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.835+0000 7f67ba42e640 1 -- 172.21.15.27:0/2321379590 learned_addr learned my addr 172.21.15.27:0/2321379590 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:58.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.835+0000 7f67b942c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562e5e30 0x55b1562e8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51610/0 (socket says 172.21.15.27:51610) 2024-04-25T00:16:58.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.835+0000 7f67ba42e640 1 -- 172.21.15.27:0/2321379590 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562db760 msgr2=0x55b1562e58f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:16:58.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.835+0000 7f67ba42e640 1 --2- 172.21.15.27:0/2321379590 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562db760 0x55b1562e58f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.835+0000 7f67ba42e640 1 -- 172.21.15.27:0/2321379590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562e5e30 msgr2=0x55b1562e8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:58.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.836+0000 7f67ba42e640 1 --2- 172.21.15.27:0/2321379590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562e5e30 0x55b1562e8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.836+0000 7f67ba42e640 1 -- 172.21.15.27:0/2321379590 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1560157d0 con 0x55b1562dae40 2024-04-25T00:16:58.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.836+0000 7f67b942c640 1 --2- 172.21.15.27:0/2321379590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562e5e30 0x55b1562e8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:58.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.836+0000 7f67ba42e640 1 --2- 172.21.15.27:0/2321379590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562dae40 0x55b1562db220 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f67ac010750 tx=0x7f67ac010c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5de5a3dfa73e65aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:58.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.836+0000 7f67aaffd640 1 --2- 172.21.15.27:0/3036684021 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f67a00a2f30 0x7f67a00a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.836+0000 7f67b9c2d640 1 --2- 172.21.15.27:0/3036684021 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f67a00a2f30 0x7f67a00a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.836+0000 7f679ffff640 1 -- 172.21.15.27:0/2321379590 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac00ead0 con 0x55b1562dae40 2024-04-25T00:16:58.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.836+0000 7f679ffff640 1 -- 172.21.15.27:0/2321379590 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67ac010e60 con 0x55b1562dae40 2024-04-25T00:16:58.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.836+0000 7f679ffff640 1 -- 172.21.15.27:0/2321379590 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac011640 con 0x55b1562dae40 2024-04-25T00:16:58.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.836+0000 7f67aaffd640 1 -- 172.21.15.27:0/3036684021 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f67b00efe40 con 0x7f67b4100610 2024-04-25T00:16:58.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.837+0000 7f67beb95740 1 -- 172.21.15.27:0/2321379590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562dae40 msgr2=0x55b1562db220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.837+0000 7f67beb95740 1 --2- 172.21.15.27:0/2321379590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562dae40 0x55b1562db220 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f67ac010750 tx=0x7f67ac010c10 comp rx=0 tx=0).stop 2024-04-25T00:16:58.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.837+0000 7f67b9c2d640 1 --2- 172.21.15.27:0/3036684021 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f67a00a2f30 0x7f67a00a53f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f67b00ed040 tx=0x7f67b0052f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:58.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.837+0000 7f67beb95740 1 -- 172.21.15.27:0/2321379590 shutdown_connections 2024-04-25T00:16:58.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.837+0000 7f67beb95740 1 --2- 172.21.15.27:0/2321379590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562e5e30 0x55b1562e8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.837+0000 7f67beb95740 1 --2- 172.21.15.27:0/2321379590 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562db760 0x55b1562e58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.837+0000 7f67beb95740 1 --2- 172.21.15.27:0/2321379590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562dae40 0x55b1562db220 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.837+0000 7f67beb95740 1 -- 172.21.15.27:0/2321379590 >> 172.21.15.27:0/2321379590 conn(0x55b1562d7160 msgr2=0x55b1562d7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:58.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.837+0000 7f67beb95740 1 -- 172.21.15.27:0/2321379590 shutdown_connections 2024-04-25T00:16:58.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.837+0000 7f67beb95740 1 -- 172.21.15.27:0/2321379590 wait complete. 2024-04-25T00:16:58.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.837+0000 7f67beb95740 1 -- start start 2024-04-25T00:16:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.838+0000 7f67beb95740 1 Processor -- start 2024-04-25T00:16:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.838+0000 7f67beb95740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562dae40 0x55b1562e2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.838+0000 7f67beb95740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562db760 0x55b1562e3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.838+0000 7f67beb95740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562e5e30 0x55b1562e38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.838+0000 7f67beb95740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b1562e8c20 con 0x55b1562e5e30 2024-04-25T00:16:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.838+0000 7f67beb95740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b1561f2f50 con 0x55b1562db760 2024-04-25T00:16:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.838+0000 7f67beb95740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b1562e8da0 con 0x55b1562dae40 2024-04-25T00:16:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.838+0000 7f67beb95740 10 client.0 _ll_register_callbacks cb 0x55b15622e000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.838+0000 7f67ba42e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562e5e30 0x55b1562e38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.838+0000 7f67ba42e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562e5e30 0x55b1562e38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51622/0 (socket says 172.21.15.27:51622) 2024-04-25T00:16:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.838+0000 7f67ba42e640 1 -- 172.21.15.27:0/2045887533 learned_addr learned my addr 172.21.15.27:0/2045887533 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.838+0000 7f67ba42e640 1 --2- 172.21.15.27:0/2045887533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562dae40 0x55b1562e2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.838+0000 7f67b942c640 1 --2- 172.21.15.27:0/2045887533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562db760 0x55b1562e3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.839+0000 7f67ba42e640 1 -- 172.21.15.27:0/2045887533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562dae40 msgr2=0x55b1562e2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.839+0000 7f67ba42e640 1 --2- 172.21.15.27:0/2045887533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562dae40 0x55b1562e2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.839+0000 7f67ba42e640 1 -- 172.21.15.27:0/2045887533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562db760 msgr2=0x55b1562e3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.839+0000 7f67ba42e640 1 --2- 172.21.15.27:0/2045887533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562db760 0x55b1562e3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.839+0000 7f67ba42e640 1 -- 172.21.15.27:0/2045887533 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b155f4b3f0 con 0x55b1562e5e30 2024-04-25T00:16:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.839+0000 7f67ba42e640 1 --2- 172.21.15.27:0/2045887533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562e5e30 0x55b1562e38a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f67ac014000 tx=0x7f67ac012dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377f10e1905f01e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.839+0000 7f679d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.839+0000 7f679d7fa640 1 -- 172.21.15.27:0/2045887533 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac025040 con 0x55b1562e5e30 2024-04-25T00:16:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.839+0000 7f67beb95740 1 -- 172.21.15.27:0/2045887533 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67ac010e60 con 0x55b1562e5e30 2024-04-25T00:16:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.839+0000 7f679d7fa640 1 -- 172.21.15.27:0/2045887533 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f67ac00ed70 con 0x55b1562e5e30 2024-04-25T00:16:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.839+0000 7f679d7fa640 1 -- 172.21.15.27:0/2045887533 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f67ac02c430 con 0x55b1562e5e30 2024-04-25T00:16:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.839+0000 7f67beb95740 10 client.16128 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.839+0000 7f67beb95740 1 -- 172.21.15.27:0/2045887533 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b155f0b0f0 con 0x55b1562e5e30 2024-04-25T00:16:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.840+0000 7f679d7fa640 1 -- 172.21.15.27:0/2045887533 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67ac010e60 con 0x55b1562e5e30 2024-04-25T00:16:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.840+0000 7f679d7fa640 1 -- 172.21.15.27:0/2045887533 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f67ac00f040 con 0x55b1562e5e30 2024-04-25T00:16:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.840+0000 7f67beb95740 10 client.16128 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.840+0000 7f67beb95740 10 client.16128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:16:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.840+0000 7f67beb95740 20 client.16128 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.840+0000 7f67beb95740 1 --2- 172.21.15.27:0/2045887533 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55b1562f6480 0x55b1562f88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.840+0000 7f67beb95740 4 client.16128 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:16:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.840+0000 7f67beb95740 1 -- 172.21.15.27:0/2045887533 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b156244bf0 con 0x55b1562f6480 2024-04-25T00:16:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.840+0000 7f67b942c640 1 --2- 172.21.15.27:0/2045887533 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55b1562f6480 0x55b1562f88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.841+0000 7f679d7fa640 10 client.16128 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.841+0000 7f67b942c640 1 --2- 172.21.15.27:0/2045887533 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55b1562f6480 0x55b1562f88c0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33d6367f19cf762e server_cookie=40cbe7a9927dfc8f in_seq=0 out_seq=0 2024-04-25T00:16:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.841+0000 7f679d7fa640 1 -- 172.21.15.27:0/2045887533 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b156244bf0 con 0x55b1562f6480 2024-04-25T00:16:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.841+0000 7f679d7fa640 10 client.16128 handle_command_reply: tid=0 2024-04-25T00:16:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.842+0000 7f67beb95740 1 client.16128 shutdown 2024-04-25T00:16:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.842+0000 7f67beb95740 1 -- 172.21.15.27:0/2045887533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562e5e30 msgr2=0x55b1562e38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.842+0000 7f67beb95740 1 --2- 172.21.15.27:0/2045887533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562e5e30 0x55b1562e38a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f67ac014000 tx=0x7f67ac012dd0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.842+0000 7f67beb95740 1 -- 172.21.15.27:0/2045887533 shutdown_connections 2024-04-25T00:16:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.842+0000 7f67beb95740 1 --2- 172.21.15.27:0/2045887533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562e5e30 0x55b1562e38a0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.842+0000 7f67beb95740 1 --2- 172.21.15.27:0/2045887533 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55b1562f6480 0x55b1562f88c0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.842+0000 7f67beb95740 1 --2- 172.21.15.27:0/2045887533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562db760 0x55b1562e3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.842+0000 7f67beb95740 1 --2- 172.21.15.27:0/2045887533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562dae40 0x55b1562e2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.842+0000 7f67beb95740 1 -- 172.21.15.27:0/2045887533 >> 172.21.15.27:0/2045887533 conn(0x55b1562d7160 msgr2=0x55b1562dcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.842+0000 7f679d7fa640 0 client.16128 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.842+0000 7f67beb95740 1 -- 172.21.15.27:0/2045887533 shutdown_connections 2024-04-25T00:16:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.842+0000 7f67beb95740 1 -- 172.21.15.27:0/2045887533 wait complete. 2024-04-25T00:16:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.842+0000 7f67beb95740 20 client.16128 trim_cache size 0 max 16384 2024-04-25T00:16:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.860+0000 7f67beb95740 1 Processor -- start 2024-04-25T00:16:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.860+0000 7f67beb95740 1 -- start start 2024-04-25T00:16:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.860+0000 7f67beb95740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562dae40 0x55b1562e5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.860+0000 7f67beb95740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562db760 0x55b1562f1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.860+0000 7f67beb95740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562e5e30 0x55b1562e1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.860+0000 7f67beb95740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f67b41ac9a0 con 0x55b1562db760 2024-04-25T00:16:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.860+0000 7f67beb95740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f67b41ae4e0 con 0x55b1562dae40 2024-04-25T00:16:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.860+0000 7f67beb95740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f67b41ad070 con 0x55b1562e5e30 2024-04-25T00:16:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.861+0000 7f67b942c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562db760 0x55b1562f1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.861+0000 7f67b942c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562db760 0x55b1562f1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51632/0 (socket says 172.21.15.27:51632) 2024-04-25T00:16:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.861+0000 7f67b942c640 1 -- 172.21.15.27:0/77429747 learned_addr learned my addr 172.21.15.27:0/77429747 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.861+0000 7f67ba42e640 1 --2- 172.21.15.27:0/77429747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562dae40 0x55b1562e5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.861+0000 7f67ba42e640 1 --2- 172.21.15.27:0/77429747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562e5e30 0x55b1562e1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.861+0000 7f67b942c640 1 -- 172.21.15.27:0/77429747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562e5e30 msgr2=0x55b1562e1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.861+0000 7f67b942c640 1 --2- 172.21.15.27:0/77429747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562e5e30 0x55b1562e1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.861+0000 7f67b942c640 1 -- 172.21.15.27:0/77429747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562dae40 msgr2=0x55b1562e5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.861+0000 7f67b942c640 1 --2- 172.21.15.27:0/77429747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562dae40 0x55b1562e5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.861+0000 7f67b942c640 1 -- 172.21.15.27:0/77429747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1562e1370 con 0x55b1562db760 2024-04-25T00:16:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.861+0000 7f67ba42e640 1 --2- 172.21.15.27:0/77429747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562e5e30 0x55b1562e1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.861+0000 7f67b942c640 1 --2- 172.21.15.27:0/77429747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562db760 0x55b1562f1ae0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f67b4105d20 tx=0x7f67a400b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2540e29efb5447 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.861+0000 7f679f7fe640 1 -- 172.21.15.27:0/77429747 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a4003b20 con 0x55b1562db760 2024-04-25T00:16:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.861+0000 7f679f7fe640 1 -- 172.21.15.27:0/77429747 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67a4023020 con 0x55b1562db760 2024-04-25T00:16:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.861+0000 7f679f7fe640 1 -- 172.21.15.27:0/77429747 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a4011730 con 0x55b1562db760 2024-04-25T00:16:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.862+0000 7f67beb95740 1 -- 172.21.15.27:0/77429747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562db760 msgr2=0x55b1562f1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.862+0000 7f67beb95740 1 --2- 172.21.15.27:0/77429747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562db760 0x55b1562f1ae0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f67b4105d20 tx=0x7f67a400b390 comp rx=0 tx=0).stop 2024-04-25T00:16:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.862+0000 7f67beb95740 1 -- 172.21.15.27:0/77429747 shutdown_connections 2024-04-25T00:16:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.862+0000 7f67beb95740 1 --2- 172.21.15.27:0/77429747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562e5e30 0x55b1562e1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.862+0000 7f67beb95740 1 --2- 172.21.15.27:0/77429747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562db760 0x55b1562f1ae0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.862+0000 7f67beb95740 1 --2- 172.21.15.27:0/77429747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562dae40 0x55b1562e5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.862+0000 7f67beb95740 1 -- 172.21.15.27:0/77429747 >> 172.21.15.27:0/77429747 conn(0x55b1562d7160 msgr2=0x55b1562e16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.862+0000 7f67beb95740 1 -- 172.21.15.27:0/77429747 shutdown_connections 2024-04-25T00:16:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.862+0000 7f67beb95740 1 -- 172.21.15.27:0/77429747 wait complete. 2024-04-25T00:16:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.863+0000 7f67beb95740 1 -- start start 2024-04-25T00:16:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.863+0000 7f67beb95740 1 Processor -- start 2024-04-25T00:16:58.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.863+0000 7f67beb95740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562dae40 0x55b1562ee480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.863+0000 7f67beb95740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562db760 0x55b1562e0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.863+0000 7f67beb95740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562e5e30 0x55b1562e0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.863+0000 7f67beb95740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f67b41ade10 con 0x55b1562db760 2024-04-25T00:16:58.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.863+0000 7f67beb95740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f67b41ad740 con 0x55b1562e5e30 2024-04-25T00:16:58.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.863+0000 7f67beb95740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f67b41aa960 con 0x55b1562dae40 2024-04-25T00:16:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.863+0000 7f67b942c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562db760 0x55b1562e0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.863+0000 7f67beb95740 10 client.0 _ll_register_callbacks cb 0x7f67ac016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:16:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.863+0000 7f67b942c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562db760 0x55b1562e0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51646/0 (socket says 172.21.15.27:51646) 2024-04-25T00:16:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.863+0000 7f67b942c640 1 -- 172.21.15.27:0/1023789458 learned_addr learned my addr 172.21.15.27:0/1023789458 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:16:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.863+0000 7f67ba42e640 1 --2- 172.21.15.27:0/1023789458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562dae40 0x55b1562ee480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.864+0000 7f67ba42e640 1 --2- 172.21.15.27:0/1023789458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562e5e30 0x55b1562e0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.864+0000 7f67b942c640 1 -- 172.21.15.27:0/1023789458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562dae40 msgr2=0x55b1562ee480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.864+0000 7f67b942c640 1 --2- 172.21.15.27:0/1023789458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562dae40 0x55b1562ee480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.864+0000 7f67b942c640 1 -- 172.21.15.27:0/1023789458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562e5e30 msgr2=0x55b1562e0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.864+0000 7f67b942c640 1 --2- 172.21.15.27:0/1023789458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562e5e30 0x55b1562e0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.864+0000 7f67b942c640 1 -- 172.21.15.27:0/1023789458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1562df190 con 0x55b1562db760 2024-04-25T00:16:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.864+0000 7f67ba42e640 1 --2- 172.21.15.27:0/1023789458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562e5e30 0x55b1562e0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.864+0000 7f67b942c640 1 --2- 172.21.15.27:0/1023789458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562db760 0x55b1562e0370 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f67a4022040 tx=0x7f67a401f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83866479ca458a13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.864+0000 7f679d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:16:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.864+0000 7f679d7fa640 1 -- 172.21.15.27:0/1023789458 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a402c020 con 0x55b1562db760 2024-04-25T00:16:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.864+0000 7f679d7fa640 1 -- 172.21.15.27:0/1023789458 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f67a4003b20 con 0x55b1562db760 2024-04-25T00:16:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.864+0000 7f67beb95740 1 -- 172.21.15.27:0/1023789458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67a4023020 con 0x55b1562db760 2024-04-25T00:16:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.864+0000 7f679d7fa640 1 -- 172.21.15.27:0/1023789458 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f67a4037050 con 0x55b1562db760 2024-04-25T00:16:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.864+0000 7f67beb95740 10 client.16134 fetch_fsmap learned FSMap version 29 2024-04-25T00:16:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.864+0000 7f67beb95740 1 -- 172.21.15.27:0/1023789458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b15635bdb0 con 0x55b1562db760 2024-04-25T00:16:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.865+0000 7f679d7fa640 1 -- 172.21.15.27:0/1023789458 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67a4023020 con 0x55b1562db760 2024-04-25T00:16:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.865+0000 7f679d7fa640 1 -- 172.21.15.27:0/1023789458 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f67a40048f0 con 0x55b1562db760 2024-04-25T00:16:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.865+0000 7f67beb95740 10 client.16134 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T00:16:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.865+0000 7f67beb95740 10 client.16134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:16:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.865+0000 7f67beb95740 20 client.16134 populate_metadata read hostname 'smithi027' 2024-04-25T00:16:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.865+0000 7f67beb95740 1 --2- 172.21.15.27:0/1023789458 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55b15635c420 0x55b15635e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.865+0000 7f67beb95740 4 client.16134 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:16:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.865+0000 7f67beb95740 1 -- 172.21.15.27:0/1023789458 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b15623ef40 con 0x55b15635c420 2024-04-25T00:16:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.865+0000 7f67ba42e640 1 --2- 172.21.15.27:0/1023789458 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55b15635c420 0x55b15635e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.866+0000 7f679d7fa640 10 client.16134 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.866+0000 7f67ba42e640 1 --2- 172.21.15.27:0/1023789458 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55b15635c420 0x55b15635e840 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71bd6754cca25c3 server_cookie=890d0321465f22a9 in_seq=0 out_seq=0 2024-04-25T00:16:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.866+0000 7f679d7fa640 1 -- 172.21.15.27:0/1023789458 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b15623ef40 con 0x55b15635c420 2024-04-25T00:16:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.866+0000 7f679d7fa640 10 client.16134 handle_command_reply: tid=0 2024-04-25T00:16:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.866+0000 7f67beb95740 1 client.16134 shutdown 2024-04-25T00:16:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.866+0000 7f67beb95740 1 -- 172.21.15.27:0/1023789458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562db760 msgr2=0x55b1562e0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.866+0000 7f67beb95740 1 --2- 172.21.15.27:0/1023789458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562db760 0x55b1562e0370 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f67a4022040 tx=0x7f67a401f810 comp rx=0 tx=0).stop 2024-04-25T00:16:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.866+0000 7f67beb95740 1 -- 172.21.15.27:0/1023789458 shutdown_connections 2024-04-25T00:16:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.866+0000 7f67beb95740 1 --2- 172.21.15.27:0/1023789458 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55b15635c420 0x55b15635e840 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.866+0000 7f67beb95740 1 --2- 172.21.15.27:0/1023789458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b1562e5e30 0x55b1562e0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.866+0000 7f67beb95740 1 --2- 172.21.15.27:0/1023789458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b1562db760 0x55b1562e0370 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.866+0000 7f679d7fa640 0 client.16134 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:16:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.866+0000 7f67beb95740 1 --2- 172.21.15.27:0/1023789458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b1562dae40 0x55b1562ee480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.866+0000 7f67beb95740 1 -- 172.21.15.27:0/1023789458 >> 172.21.15.27:0/1023789458 conn(0x55b1562d7160 msgr2=0x55b1562e1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.866+0000 7f67beb95740 1 -- 172.21.15.27:0/1023789458 shutdown_connections 2024-04-25T00:16:58.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.866+0000 7f67beb95740 1 -- 172.21.15.27:0/1023789458 wait complete. 2024-04-25T00:16:58.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.866+0000 7f67beb95740 20 client.16134 trim_cache size 0 max 16384 2024-04-25T00:16:58.873 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:16:58.873 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T00:16:58.873 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:16:58.873 INFO:teuthology.orchestra.run.smithi027.stdout: "c8feec42-54b9-4da2-8629-87ce7cff7af1": { 2024-04-25T00:16:58.873 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:16:58.874 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "c8feec42-54b9-4da2-8629-87ce7cff7af1", 2024-04-25T00:16:58.874 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:16:58.874 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:16:58.874 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:16:58.874 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:16:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.867+0000 7f67bbe91640 1 -- 172.21.15.27:0/3036684021 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f67a00a2f30 msgr2=0x7f67a00a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.867+0000 7f67bbe91640 1 --2- 172.21.15.27:0/3036684021 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f67a00a2f30 0x7f67a00a53f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f67b00ed040 tx=0x7f67b0052f90 comp rx=0 tx=0).stop 2024-04-25T00:16:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.867+0000 7f67bbe91640 1 -- 172.21.15.27:0/3036684021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67b4100610 msgr2=0x7f67b4195d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.867+0000 7f67bbe91640 1 --2- 172.21.15.27:0/3036684021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67b4100610 0x7f67b4195d80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f67b0009230 tx=0x7f67b0035420 comp rx=0 tx=0).stop 2024-04-25T00:16:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.867+0000 7f67bbe91640 1 -- 172.21.15.27:0/3036684021 shutdown_connections 2024-04-25T00:16:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.867+0000 7f67bbe91640 1 --2- 172.21.15.27:0/3036684021 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f67a00a2f30 0x7f67a00a53f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.867+0000 7f67bbe91640 1 --2- 172.21.15.27:0/3036684021 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67b41066b0 0x7f67b419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.867+0000 7f67bbe91640 1 --2- 172.21.15.27:0/3036684021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67b4100fd0 0x7f67b41962c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.867+0000 7f67bbe91640 1 --2- 172.21.15.27:0/3036684021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67b4100610 0x7f67b4195d80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.867+0000 7f67bbe91640 1 -- 172.21.15.27:0/3036684021 >> 172.21.15.27:0/3036684021 conn(0x7f67b4077740 msgr2=0x7f67b40ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.867+0000 7f67bbe91640 1 -- 172.21.15.27:0/3036684021 shutdown_connections 2024-04-25T00:16:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:16:58.867+0000 7f67bbe91640 1 -- 172.21.15.27:0/3036684021 wait complete. 2024-04-25T00:16:58.879 INFO:tasks.cephfs.filesystem:scrub status for tag:c8feec42-54b9-4da2-8629-87ce7cff7af1 - {'path': '/', 'tag': 'c8feec42-54b9-4da2-8629-87ce7cff7af1', 'options': 'recursive,force'} 2024-04-25T00:16:59.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:59 smithi027 ceph-mon[29208]: scrub queued for path: / 2024-04-25T00:16:59.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:59 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:16:59.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:59 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:16:59.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:16:59 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:16:59.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:59 smithi139 ceph-mon[32833]: scrub queued for path: / 2024-04-25T00:16:59.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:59 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:16:59.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:59 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:16:59.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:16:59 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:16:59.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:59 smithi153 ceph-mon[32896]: scrub queued for path: / 2024-04-25T00:16:59.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:59 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:16:59.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:59 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:16:59.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:16:59 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:00.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:00 smithi027 ceph-mon[29208]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:17:00.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:00 smithi139 ceph-mon[32833]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:17:00.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:00 smithi153 ceph-mon[32896]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T00:17:02.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:02 smithi027 ceph-mon[29208]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T00:17:02.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:02 smithi139 ceph-mon[32833]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T00:17:02.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:02 smithi153 ceph-mon[32896]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T00:17:03.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:03 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:17:03.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:03 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:17:03.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:03 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:17:03.904 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 95% (11940/12492) Updating files: 96% (11993/12492) Updating files: 97% (12118/12492) Updating files: 98% (12243/12492) Updating files: 99% (12368/12492) Updating files: 100% (12492/12492) Updating files: 100% (12492/12492), done. 2024-04-25T00:17:04.691 INFO:tasks.workunit.client.0.smithi027.stderr:Note: switching to 'd2c45d977142e024a59b68a8f1dce8f0b9840551'. 2024-04-25T00:17:04.691 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-04-25T00:17:04.691 INFO:tasks.workunit.client.0.smithi027.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T00:17:04.691 INFO:tasks.workunit.client.0.smithi027.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T00:17:04.691 INFO:tasks.workunit.client.0.smithi027.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T00:17:04.691 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-04-25T00:17:04.691 INFO:tasks.workunit.client.0.smithi027.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T00:17:04.691 INFO:tasks.workunit.client.0.smithi027.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T00:17:04.692 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-04-25T00:17:04.692 INFO:tasks.workunit.client.0.smithi027.stderr: git switch -c 2024-04-25T00:17:04.692 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-04-25T00:17:04.692 INFO:tasks.workunit.client.0.smithi027.stderr:Or undo this operation with: 2024-04-25T00:17:04.692 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-04-25T00:17:04.692 INFO:tasks.workunit.client.0.smithi027.stderr: git switch - 2024-04-25T00:17:04.692 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-04-25T00:17:04.692 INFO:tasks.workunit.client.0.smithi027.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T00:17:04.692 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-04-25T00:17:04.692 INFO:tasks.workunit.client.0.smithi027.stderr:HEAD is now at d2c45d97714 Merge pull request #56528 from adk3798/squid-mypy-bump 2024-04-25T00:17:04.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:04 smithi027 ceph-mon[29208]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T00:17:04.695 DEBUG:teuthology.orchestra.run.smithi027:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T00:17:04.734 INFO:tasks.workunit.client.0.smithi027.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-25T00:17:04.736 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T00:17:04.736 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-25T00:17:04.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:04 smithi139 ceph-mon[32833]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T00:17:04.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:04 smithi153 ceph-mon[32896]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T00:17:05.281 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-25T00:17:05.335 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-25T00:17:05.375 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T00:17:05.377 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T00:17:05.377 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-25T00:17:05.417 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T00:17:05.422 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:17:05.422 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T00:17:05.478 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-04-25T00:17:05.479 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-04-25T00:17:05.479 DEBUG:teuthology.orchestra.run.smithi027:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d2c45d977142e024a59b68a8f1dce8f0b9840551 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-04-25T00:17:05.540 INFO:tasks.workunit.client.0.smithi027.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-04-25T00:17:05.540 INFO:tasks.workunit.client.0.smithi027.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-04-25T00:17:05.541 INFO:tasks.workunit.client.0.smithi027.stderr:+ trap -- retry EXIT 2024-04-25T00:17:05.541 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm -rf ffsb 2024-04-25T00:17:05.541 INFO:tasks.workunit.client.0.smithi027.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-04-25T00:17:05.544 INFO:tasks.workunit.client.0.smithi027.stderr:Cloning into 'ffsb'... 2024-04-25T00:17:06.509 INFO:tasks.workunit.client.0.smithi027.stderr:+ trap - EXIT 2024-04-25T00:17:06.509 INFO:tasks.workunit.client.0.smithi027.stderr:+ cd ffsb 2024-04-25T00:17:06.509 INFO:tasks.workunit.client.0.smithi027.stderr:+ ./configure 2024-04-25T00:17:06.632 INFO:tasks.workunit.client.0.smithi027.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-25T00:17:06.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:06 smithi027 ceph-mon[29208]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T00:17:06.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:06 smithi139 ceph-mon[32833]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T00:17:06.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:06 smithi153 ceph-mon[32896]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T00:17:07.642 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether build environment is sane... yes 2024-04-25T00:17:07.651 INFO:tasks.workunit.client.0.smithi027.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-04-25T00:17:07.651 INFO:tasks.workunit.client.0.smithi027.stdout:checking for gawk... gawk 2024-04-25T00:17:07.668 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether make sets $(MAKE)... yes 2024-04-25T00:17:07.702 INFO:tasks.workunit.client.0.smithi027.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-25T00:17:07.702 INFO:tasks.workunit.client.0.smithi027.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-25T00:17:07.713 INFO:tasks.workunit.client.0.smithi027.stdout:checking for style of include used by make... GNU 2024-04-25T00:17:07.718 INFO:tasks.workunit.client.0.smithi027.stdout:checking for gcc... gcc 2024-04-25T00:17:07.773 INFO:tasks.workunit.client.0.smithi027.stdout:checking for C compiler default output file name... a.out 2024-04-25T00:17:07.775 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether the C compiler works... yes 2024-04-25T00:17:07.778 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether we are cross compiling... no 2024-04-25T00:17:07.816 INFO:tasks.workunit.client.0.smithi027.stdout:checking for suffix of executables... 2024-04-25T00:17:07.866 INFO:tasks.workunit.client.0.smithi027.stdout:checking for suffix of object files... o 2024-04-25T00:17:07.935 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether we are using the GNU C compiler... yes 2024-04-25T00:17:07.966 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether gcc accepts -g... yes 2024-04-25T00:17:08.045 INFO:tasks.workunit.client.0.smithi027.stdout:checking for gcc option to accept ISO C89... none needed 2024-04-25T00:17:08.151 INFO:tasks.workunit.client.0.smithi027.stdout:checking dependency style of gcc... gcc3 2024-04-25T00:17:08.206 INFO:tasks.workunit.client.0.smithi027.stdout:checking how to run the C preprocessor... gcc -E 2024-04-25T00:17:08.248 INFO:tasks.workunit.client.0.smithi027.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-25T00:17:08.249 INFO:tasks.workunit.client.0.smithi027.stdout:checking for egrep... /usr/bin/grep -E 2024-04-25T00:17:08.258 INFO:tasks.workunit.client.0.smithi027.stdout:checking for AIX... no 2024-04-25T00:17:08.278 INFO:tasks.workunit.client.0.smithi027.stdout:checking for gcc... (cached) gcc 2024-04-25T00:17:08.285 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-04-25T00:17:08.287 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-25T00:17:08.287 INFO:tasks.workunit.client.0.smithi027.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-04-25T00:17:08.287 INFO:tasks.workunit.client.0.smithi027.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-25T00:17:08.339 INFO:tasks.workunit.client.0.smithi027.stdout:checking for main in -lm... yes 2024-04-25T00:17:08.393 INFO:tasks.workunit.client.0.smithi027.stdout:checking for main in -lpthread... yes 2024-04-25T00:17:08.566 INFO:tasks.workunit.client.0.smithi027.stdout:checking for ANSI C header files... yes 2024-04-25T00:17:08.620 INFO:tasks.workunit.client.0.smithi027.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-25T00:17:08.673 INFO:tasks.workunit.client.0.smithi027.stdout:checking for sys/types.h... yes 2024-04-25T00:17:08.725 INFO:tasks.workunit.client.0.smithi027.stdout:checking for sys/stat.h... yes 2024-04-25T00:17:08.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:08 smithi139 ceph-mon[32833]: pgmap v44: 129 pgs: 129 active+clean; 2.2 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 94 KiB/s wr, 28 op/s 2024-04-25T00:17:08.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:08 smithi153 ceph-mon[32896]: pgmap v44: 129 pgs: 129 active+clean; 2.2 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 94 KiB/s wr, 28 op/s 2024-04-25T00:17:08.778 INFO:tasks.workunit.client.0.smithi027.stdout:checking for stdlib.h... yes 2024-04-25T00:17:08.831 INFO:tasks.workunit.client.0.smithi027.stdout:checking for string.h... yes 2024-04-25T00:17:08.884 INFO:tasks.workunit.client.0.smithi027.stdout:checking for memory.h... yes 2024-04-25T00:17:08.932 INFO:tasks.workunit.client.0.smithi027.stdout:checking for strings.h... yes 2024-04-25T00:17:08.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:08 smithi027 ceph-mon[29208]: pgmap v44: 129 pgs: 129 active+clean; 2.2 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 94 KiB/s wr, 28 op/s 2024-04-25T00:17:08.981 INFO:tasks.workunit.client.0.smithi027.stdout:checking for inttypes.h... yes 2024-04-25T00:17:09.029 INFO:tasks.workunit.client.0.smithi027.stdout:checking for stdint.h... yes 2024-04-25T00:17:09.079 INFO:tasks.workunit.client.0.smithi027.stdout:checking for unistd.h... yes 2024-04-25T00:17:09.140 INFO:tasks.workunit.client.0.smithi027.stdout:checking pthread.h usability... yes 2024-04-25T00:17:09.163 INFO:tasks.workunit.client.0.smithi027.stdout:checking pthread.h presence... yes 2024-04-25T00:17:09.163 INFO:tasks.workunit.client.0.smithi027.stdout:checking for pthread.h... yes 2024-04-25T00:17:09.216 INFO:tasks.workunit.client.0.smithi027.stdout:checking fcntl.h usability... yes 2024-04-25T00:17:09.239 INFO:tasks.workunit.client.0.smithi027.stdout:checking fcntl.h presence... yes 2024-04-25T00:17:09.239 INFO:tasks.workunit.client.0.smithi027.stdout:checking for fcntl.h... yes 2024-04-25T00:17:09.297 INFO:tasks.workunit.client.0.smithi027.stdout:checking limits.h usability... yes 2024-04-25T00:17:09.321 INFO:tasks.workunit.client.0.smithi027.stdout:checking limits.h presence... yes 2024-04-25T00:17:09.321 INFO:tasks.workunit.client.0.smithi027.stdout:checking for limits.h... yes 2024-04-25T00:17:09.337 INFO:tasks.workunit.client.0.smithi027.stdout:checking for stdint.h... (cached) yes 2024-04-25T00:17:09.384 INFO:tasks.workunit.client.0.smithi027.stdout:checking sys/time.h usability... yes 2024-04-25T00:17:09.409 INFO:tasks.workunit.client.0.smithi027.stdout:checking sys/time.h presence... yes 2024-04-25T00:17:09.409 INFO:tasks.workunit.client.0.smithi027.stdout:checking for sys/time.h... yes 2024-04-25T00:17:09.424 INFO:tasks.workunit.client.0.smithi027.stdout:checking for unistd.h... (cached) yes 2024-04-25T00:17:09.482 INFO:tasks.workunit.client.0.smithi027.stdout:checking sys/vfs.h usability... yes 2024-04-25T00:17:09.500 INFO:tasks.workunit.client.0.smithi027.stdout:checking sys/vfs.h presence... yes 2024-04-25T00:17:09.501 INFO:tasks.workunit.client.0.smithi027.stdout:checking for sys/vfs.h... yes 2024-04-25T00:17:09.548 INFO:tasks.workunit.client.0.smithi027.stdout:checking sys/limits.h usability... no 2024-04-25T00:17:09.569 INFO:tasks.workunit.client.0.smithi027.stdout:checking sys/limits.h presence... no 2024-04-25T00:17:09.570 INFO:tasks.workunit.client.0.smithi027.stdout:checking for sys/limits.h... no 2024-04-25T00:17:09.603 INFO:tasks.workunit.client.0.smithi027.stdout:checking for an ANSI C-conforming const... yes 2024-04-25T00:17:09.638 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-04-25T00:17:09.639 INFO:tasks.workunit.client.0.smithi027.stdout:checking for function prototypes... yes 2024-04-25T00:17:09.668 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether setvbuf arguments are reversed... no 2024-04-25T00:17:09.745 INFO:tasks.workunit.client.0.smithi027.stdout:checking for system... yes 2024-04-25T00:17:09.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:09 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:17:09.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:09 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:17:09.811 INFO:tasks.workunit.client.0.smithi027.stdout:checking for gettimeofday... yes 2024-04-25T00:17:09.874 INFO:tasks.workunit.client.0.smithi027.stdout:checking for mkdir... yes 2024-04-25T00:17:09.937 INFO:tasks.workunit.client.0.smithi027.stdout:checking for strerror... yes 2024-04-25T00:17:09.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:09 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:17:10.000 INFO:tasks.workunit.client.0.smithi027.stdout:checking for open64... yes 2024-04-25T00:17:10.065 INFO:tasks.workunit.client.0.smithi027.stdout:checking for stat64... yes 2024-04-25T00:17:10.128 INFO:tasks.workunit.client.0.smithi027.stdout:checking for fseeko64... yes 2024-04-25T00:17:10.191 INFO:tasks.workunit.client.0.smithi027.stdout:checking for lrand48_r... yes 2024-04-25T00:17:10.253 INFO:tasks.workunit.client.0.smithi027.stdout:checking for srand48_r... yes 2024-04-25T00:17:10.276 INFO:tasks.workunit.client.0.smithi027.stdout:configure: creating ./config.status 2024-04-25T00:17:10.374 INFO:tasks.workunit.client.0.smithi027.stdout:config.status: creating Makefile 2024-04-25T00:17:10.388 INFO:tasks.workunit.client.0.smithi027.stdout:config.status: creating config.h 2024-04-25T00:17:10.405 INFO:tasks.workunit.client.0.smithi027.stdout:config.status: executing depfiles commands 2024-04-25T00:17:10.452 INFO:tasks.workunit.client.0.smithi027.stdout:config.status: executing default commands 2024-04-25T00:17:10.501 INFO:tasks.workunit.client.0.smithi027.stderr:+ make 2024-04-25T00:17:10.510 INFO:tasks.workunit.client.0.smithi027.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-04-25T00:17:10.513 INFO:tasks.workunit.client.0.smithi027.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-04-25T00:17:10.514 INFO:tasks.workunit.client.0.smithi027.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-04-25T00:17:10.514 INFO:tasks.workunit.client.0.smithi027.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-04-25T00:17:10.514 INFO:tasks.workunit.client.0.smithi027.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-04-25T00:17:10.514 INFO:tasks.workunit.client.0.smithi027.stderr: any GNU archive site. 2024-04-25T00:17:10.516 INFO:tasks.workunit.client.0.smithi027.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-04-25T00:17:10.517 INFO:tasks.workunit.client.0.smithi027.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-04-25T00:17:10.518 INFO:tasks.workunit.client.0.smithi027.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-04-25T00:17:10.518 INFO:tasks.workunit.client.0.smithi027.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-04-25T00:17:10.518 INFO:tasks.workunit.client.0.smithi027.stderr: You might want to install the `Automake' and `Perl' packages. 2024-04-25T00:17:10.518 INFO:tasks.workunit.client.0.smithi027.stderr: Grab them from any GNU archive site. 2024-04-25T00:17:10.723 INFO:tasks.workunit.client.0.smithi027.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-04-25T00:17:10.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:10 smithi139 ceph-mon[32833]: pgmap v45: 129 pgs: 129 active+clean; 2.2 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 94 KiB/s wr, 24 op/s 2024-04-25T00:17:10.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:10 smithi153 ceph-mon[32896]: pgmap v45: 129 pgs: 129 active+clean; 2.2 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 94 KiB/s wr, 24 op/s 2024-04-25T00:17:10.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:10 smithi027 ceph-mon[29208]: pgmap v45: 129 pgs: 129 active+clean; 2.2 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 94 KiB/s wr, 24 op/s 2024-04-25T00:17:10.974 INFO:tasks.workunit.client.0.smithi027.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-04-25T00:17:10.974 INFO:tasks.workunit.client.0.smithi027.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-04-25T00:17:10.974 INFO:tasks.workunit.client.0.smithi027.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-04-25T00:17:10.974 INFO:tasks.workunit.client.0.smithi027.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-04-25T00:17:11.096 INFO:tasks.workunit.client.0.smithi027.stdout:/bin/sh ./config.status --recheck 2024-04-25T00:17:11.149 INFO:tasks.workunit.client.0.smithi027.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-04-25T00:17:11.250 INFO:tasks.workunit.client.0.smithi027.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-25T00:17:12.258 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether build environment is sane... yes 2024-04-25T00:17:12.262 INFO:tasks.workunit.client.0.smithi027.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-04-25T00:17:12.262 INFO:tasks.workunit.client.0.smithi027.stdout:checking for gawk... gawk 2024-04-25T00:17:12.294 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether make sets $(MAKE)... yes 2024-04-25T00:17:12.324 INFO:tasks.workunit.client.0.smithi027.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-25T00:17:12.325 INFO:tasks.workunit.client.0.smithi027.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-25T00:17:12.337 INFO:tasks.workunit.client.0.smithi027.stdout:checking for style of include used by make... GNU 2024-04-25T00:17:12.342 INFO:tasks.workunit.client.0.smithi027.stdout:checking for gcc... gcc 2024-04-25T00:17:12.440 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether the C compiler works... yes 2024-04-25T00:17:12.440 INFO:tasks.workunit.client.0.smithi027.stdout:checking for C compiler default output file name... a.out 2024-04-25T00:17:12.485 INFO:tasks.workunit.client.0.smithi027.stdout:checking for suffix of executables... 2024-04-25T00:17:12.522 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether we are cross compiling... no 2024-04-25T00:17:12.567 INFO:tasks.workunit.client.0.smithi027.stdout:checking for suffix of object files... o 2024-04-25T00:17:12.592 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether we are using the GNU C compiler... yes 2024-04-25T00:17:12.617 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether gcc accepts -g... yes 2024-04-25T00:17:12.654 INFO:tasks.workunit.client.0.smithi027.stdout:checking for gcc option to accept ISO C89... none needed 2024-04-25T00:17:12.767 INFO:tasks.workunit.client.0.smithi027.stdout:checking dependency style of gcc... gcc3 2024-04-25T00:17:12.814 INFO:tasks.workunit.client.0.smithi027.stdout:checking how to run the C preprocessor... gcc -E 2024-04-25T00:17:12.865 INFO:tasks.workunit.client.0.smithi027.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-25T00:17:12.866 INFO:tasks.workunit.client.0.smithi027.stdout:checking for egrep... /usr/bin/grep -E 2024-04-25T00:17:12.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:12 smithi027 ceph-mon[29208]: pgmap v46: 129 pgs: 129 active+clean; 6.0 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 463 KiB/s wr, 69 op/s 2024-04-25T00:17:13.000 INFO:tasks.workunit.client.0.smithi027.stdout:checking for ANSI C header files... yes 2024-04-25T00:17:13.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:12 smithi139 ceph-mon[32833]: pgmap v46: 129 pgs: 129 active+clean; 6.0 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 463 KiB/s wr, 69 op/s 2024-04-25T00:17:13.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:12 smithi153 ceph-mon[32896]: pgmap v46: 129 pgs: 129 active+clean; 6.0 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 463 KiB/s wr, 69 op/s 2024-04-25T00:17:13.034 INFO:tasks.workunit.client.0.smithi027.stdout:checking for sys/types.h... yes 2024-04-25T00:17:13.074 INFO:tasks.workunit.client.0.smithi027.stdout:checking for sys/stat.h... yes 2024-04-25T00:17:13.113 INFO:tasks.workunit.client.0.smithi027.stdout:checking for stdlib.h... yes 2024-04-25T00:17:13.158 INFO:tasks.workunit.client.0.smithi027.stdout:checking for string.h... yes 2024-04-25T00:17:13.212 INFO:tasks.workunit.client.0.smithi027.stdout:checking for memory.h... yes 2024-04-25T00:17:13.253 INFO:tasks.workunit.client.0.smithi027.stdout:checking for strings.h... yes 2024-04-25T00:17:13.294 INFO:tasks.workunit.client.0.smithi027.stdout:checking for inttypes.h... yes 2024-04-25T00:17:13.336 INFO:tasks.workunit.client.0.smithi027.stdout:checking for stdint.h... yes 2024-04-25T00:17:13.377 INFO:tasks.workunit.client.0.smithi027.stdout:checking for unistd.h... yes 2024-04-25T00:17:13.434 INFO:tasks.workunit.client.0.smithi027.stdout:checking minix/config.h usability... no 2024-04-25T00:17:13.464 INFO:tasks.workunit.client.0.smithi027.stdout:checking minix/config.h presence... no 2024-04-25T00:17:13.464 INFO:tasks.workunit.client.0.smithi027.stdout:checking for minix/config.h... no 2024-04-25T00:17:13.501 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-04-25T00:17:13.501 INFO:tasks.workunit.client.0.smithi027.stdout:checking for gcc... (cached) gcc 2024-04-25T00:17:13.537 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-04-25T00:17:13.537 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-25T00:17:13.537 INFO:tasks.workunit.client.0.smithi027.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-04-25T00:17:13.537 INFO:tasks.workunit.client.0.smithi027.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-25T00:17:13.583 INFO:tasks.workunit.client.0.smithi027.stdout:checking for main in -lm... yes 2024-04-25T00:17:13.628 INFO:tasks.workunit.client.0.smithi027.stdout:checking for main in -lpthread... yes 2024-04-25T00:17:13.628 INFO:tasks.workunit.client.0.smithi027.stdout:checking for ANSI C header files... (cached) yes 2024-04-25T00:17:13.663 INFO:tasks.workunit.client.0.smithi027.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-25T00:17:13.730 INFO:tasks.workunit.client.0.smithi027.stdout:checking pthread.h usability... yes 2024-04-25T00:17:13.749 INFO:tasks.workunit.client.0.smithi027.stdout:checking pthread.h presence... yes 2024-04-25T00:17:13.749 INFO:tasks.workunit.client.0.smithi027.stdout:checking for pthread.h... yes 2024-04-25T00:17:13.798 INFO:tasks.workunit.client.0.smithi027.stdout:checking fcntl.h usability... yes 2024-04-25T00:17:13.815 INFO:tasks.workunit.client.0.smithi027.stdout:checking fcntl.h presence... yes 2024-04-25T00:17:13.815 INFO:tasks.workunit.client.0.smithi027.stdout:checking for fcntl.h... yes 2024-04-25T00:17:13.868 INFO:tasks.workunit.client.0.smithi027.stdout:checking limits.h usability... yes 2024-04-25T00:17:13.890 INFO:tasks.workunit.client.0.smithi027.stdout:checking limits.h presence... yes 2024-04-25T00:17:13.891 INFO:tasks.workunit.client.0.smithi027.stdout:checking for limits.h... yes 2024-04-25T00:17:13.905 INFO:tasks.workunit.client.0.smithi027.stdout:checking for stdint.h... (cached) yes 2024-04-25T00:17:13.942 INFO:tasks.workunit.client.0.smithi027.stdout:checking sys/time.h usability... yes 2024-04-25T00:17:13.959 INFO:tasks.workunit.client.0.smithi027.stdout:checking sys/time.h presence... yes 2024-04-25T00:17:13.959 INFO:tasks.workunit.client.0.smithi027.stdout:checking for sys/time.h... yes 2024-04-25T00:17:13.973 INFO:tasks.workunit.client.0.smithi027.stdout:checking for unistd.h... (cached) yes 2024-04-25T00:17:14.010 INFO:tasks.workunit.client.0.smithi027.stdout:checking sys/vfs.h usability... yes 2024-04-25T00:17:14.031 INFO:tasks.workunit.client.0.smithi027.stdout:checking sys/vfs.h presence... yes 2024-04-25T00:17:14.032 INFO:tasks.workunit.client.0.smithi027.stdout:checking for sys/vfs.h... yes 2024-04-25T00:17:14.084 INFO:tasks.workunit.client.0.smithi027.stdout:checking sys/limits.h usability... no 2024-04-25T00:17:14.113 INFO:tasks.workunit.client.0.smithi027.stdout:checking sys/limits.h presence... no 2024-04-25T00:17:14.113 INFO:tasks.workunit.client.0.smithi027.stdout:checking for sys/limits.h... no 2024-04-25T00:17:14.144 INFO:tasks.workunit.client.0.smithi027.stdout:checking for an ANSI C-conforming const... yes 2024-04-25T00:17:14.176 INFO:tasks.workunit.client.0.smithi027.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-04-25T00:17:14.239 INFO:tasks.workunit.client.0.smithi027.stdout:checking for system... yes 2024-04-25T00:17:14.296 INFO:tasks.workunit.client.0.smithi027.stdout:checking for gettimeofday... yes 2024-04-25T00:17:14.352 INFO:tasks.workunit.client.0.smithi027.stdout:checking for mkdir... yes 2024-04-25T00:17:14.408 INFO:tasks.workunit.client.0.smithi027.stdout:checking for strerror... yes 2024-04-25T00:17:14.465 INFO:tasks.workunit.client.0.smithi027.stdout:checking for open64... yes 2024-04-25T00:17:14.520 INFO:tasks.workunit.client.0.smithi027.stdout:checking for stat64... yes 2024-04-25T00:17:14.577 INFO:tasks.workunit.client.0.smithi027.stdout:checking for fseeko64... yes 2024-04-25T00:17:14.634 INFO:tasks.workunit.client.0.smithi027.stdout:checking for lrand48_r... yes 2024-04-25T00:17:14.694 INFO:tasks.workunit.client.0.smithi027.stdout:checking for srand48_r... yes 2024-04-25T00:17:14.718 INFO:tasks.workunit.client.0.smithi027.stdout:configure: creating ./config.status 2024-04-25T00:17:14.780 INFO:tasks.workunit.client.0.smithi027.stdout: /bin/sh ./config.status 2024-04-25T00:17:14.826 INFO:tasks.workunit.client.0.smithi027.stdout:config.status: creating Makefile 2024-04-25T00:17:14.838 INFO:tasks.workunit.client.0.smithi027.stdout:config.status: creating config.h 2024-04-25T00:17:14.859 INFO:tasks.workunit.client.0.smithi027.stdout:config.status: executing depfiles commands 2024-04-25T00:17:14.869 INFO:tasks.workunit.client.0.smithi027.stdout:config.status: executing default commands 2024-04-25T00:17:14.897 INFO:tasks.workunit.client.0.smithi027.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-04-25T00:17:14.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:14 smithi027 ceph-mon[29208]: pgmap v47: 129 pgs: 129 active+clean; 6.0 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 440 KiB/s wr, 50 op/s 2024-04-25T00:17:15.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:14 smithi139 ceph-mon[32833]: pgmap v47: 129 pgs: 129 active+clean; 6.0 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 440 KiB/s wr, 50 op/s 2024-04-25T00:17:15.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:14 smithi153 ceph-mon[32896]: pgmap v47: 129 pgs: 129 active+clean; 6.0 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 440 KiB/s wr, 50 op/s 2024-04-25T00:17:15.055 INFO:tasks.workunit.client.0.smithi027.stdout:rm -f stamp-h1 2024-04-25T00:17:15.058 INFO:tasks.workunit.client.0.smithi027.stdout:touch config.h.in 2024-04-25T00:17:15.059 INFO:tasks.workunit.client.0.smithi027.stdout:cd . && /bin/sh ./config.status config.h 2024-04-25T00:17:15.100 INFO:tasks.workunit.client.0.smithi027.stdout:config.status: creating config.h 2024-04-25T00:17:15.126 INFO:tasks.workunit.client.0.smithi027.stdout:make all-am 2024-04-25T00:17:15.132 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-04-25T00:17:15.132 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-04-25T00:17:15.297 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-04-25T00:17:15.300 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-04-25T00:17:15.385 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-04-25T00:17:15.388 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-04-25T00:17:15.472 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-04-25T00:17:15.475 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-04-25T00:17:15.563 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-04-25T00:17:15.566 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-04-25T00:17:15.688 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-04-25T00:17:15.691 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-04-25T00:17:15.740 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-04-25T00:17:15.743 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-04-25T00:17:15.785 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-04-25T00:17:15.788 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-04-25T00:17:15.835 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-04-25T00:17:15.839 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-04-25T00:17:16.050 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-04-25T00:17:16.053 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-04-25T00:17:16.179 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-04-25T00:17:16.182 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-04-25T00:17:16.328 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-04-25T00:17:16.332 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-04-25T00:17:16.404 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-04-25T00:17:16.407 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-04-25T00:17:16.477 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-04-25T00:17:16.480 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-04-25T00:17:16.585 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-04-25T00:17:16.588 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-04-25T00:17:16.616 INFO:tasks.workunit.client.0.smithi027.stderr:In file included from parser.h:22, 2024-04-25T00:17:16.616 INFO:tasks.workunit.client.0.smithi027.stderr: from parser.c:25: 2024-04-25T00:17:16.617 INFO:tasks.workunit.client.0.smithi027.stderr:parser.c: In function ‘init_filesys’: 2024-04-25T00:17:16.617 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:17:16.617 INFO:tasks.workunit.client.0.smithi027.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:17:16.617 INFO:tasks.workunit.client.0.smithi027.stderr: | ^ 2024-04-25T00:17:16.617 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-25T00:17:16.617 INFO:tasks.workunit.client.0.smithi027.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:17:16.617 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~ 2024-04-25T00:17:16.617 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-25T00:17:16.617 INFO:tasks.workunit.client.0.smithi027.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:17:16.617 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:17:16.617 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-25T00:17:16.617 INFO:tasks.workunit.client.0.smithi027.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-25T00:17:16.617 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~ 2024-04-25T00:17:16.618 INFO:tasks.workunit.client.0.smithi027.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-25T00:17:16.618 INFO:tasks.workunit.client.0.smithi027.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-25T00:17:16.618 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:17:16.618 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:17:16.618 INFO:tasks.workunit.client.0.smithi027.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:17:16.618 INFO:tasks.workunit.client.0.smithi027.stderr: | ^ 2024-04-25T00:17:16.618 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-25T00:17:16.618 INFO:tasks.workunit.client.0.smithi027.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:17:16.618 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~ 2024-04-25T00:17:16.618 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-25T00:17:16.618 INFO:tasks.workunit.client.0.smithi027.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:17:16.618 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:17:16.618 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-25T00:17:16.618 INFO:tasks.workunit.client.0.smithi027.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-25T00:17:16.619 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~ 2024-04-25T00:17:16.619 INFO:tasks.workunit.client.0.smithi027.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-25T00:17:16.619 INFO:tasks.workunit.client.0.smithi027.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-25T00:17:16.619 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:17:16.619 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:17:16.619 INFO:tasks.workunit.client.0.smithi027.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:17:16.619 INFO:tasks.workunit.client.0.smithi027.stderr: | ^ 2024-04-25T00:17:16.619 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-25T00:17:16.619 INFO:tasks.workunit.client.0.smithi027.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:17:16.619 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~ 2024-04-25T00:17:16.619 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-25T00:17:16.619 INFO:tasks.workunit.client.0.smithi027.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:17:16.619 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:17:16.620 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-25T00:17:16.620 INFO:tasks.workunit.client.0.smithi027.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-25T00:17:16.620 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~ 2024-04-25T00:17:16.620 INFO:tasks.workunit.client.0.smithi027.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-25T00:17:16.620 INFO:tasks.workunit.client.0.smithi027.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-25T00:17:16.620 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:17:16.620 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:17:16.620 INFO:tasks.workunit.client.0.smithi027.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:17:16.620 INFO:tasks.workunit.client.0.smithi027.stderr: | ^ 2024-04-25T00:17:16.620 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-25T00:17:16.620 INFO:tasks.workunit.client.0.smithi027.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:17:16.620 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~ 2024-04-25T00:17:16.620 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-25T00:17:16.620 INFO:tasks.workunit.client.0.smithi027.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:17:16.620 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:17:16.621 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-25T00:17:16.621 INFO:tasks.workunit.client.0.smithi027.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-25T00:17:16.621 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~ 2024-04-25T00:17:16.621 INFO:tasks.workunit.client.0.smithi027.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-25T00:17:16.621 INFO:tasks.workunit.client.0.smithi027.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-25T00:17:16.621 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:17:16.621 INFO:tasks.workunit.client.0.smithi027.stderr:parser.c: In function ‘init_tg_stats’: 2024-04-25T00:17:16.621 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:17:16.621 INFO:tasks.workunit.client.0.smithi027.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:17:16.621 INFO:tasks.workunit.client.0.smithi027.stderr: | ^ 2024-04-25T00:17:16.621 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-25T00:17:16.621 INFO:tasks.workunit.client.0.smithi027.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:17:16.621 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~ 2024-04-25T00:17:16.621 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-25T00:17:16.622 INFO:tasks.workunit.client.0.smithi027.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:17:16.622 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:17:16.622 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-25T00:17:16.622 INFO:tasks.workunit.client.0.smithi027.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-25T00:17:16.622 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~ 2024-04-25T00:17:16.622 INFO:tasks.workunit.client.0.smithi027.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-25T00:17:16.622 INFO:tasks.workunit.client.0.smithi027.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-25T00:17:16.622 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:17:16.622 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:17:16.623 INFO:tasks.workunit.client.0.smithi027.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:17:16.623 INFO:tasks.workunit.client.0.smithi027.stderr: | ^ 2024-04-25T00:17:16.623 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-25T00:17:16.623 INFO:tasks.workunit.client.0.smithi027.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:17:16.623 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~ 2024-04-25T00:17:16.623 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-25T00:17:16.623 INFO:tasks.workunit.client.0.smithi027.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:17:16.623 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:17:16.623 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-25T00:17:16.623 INFO:tasks.workunit.client.0.smithi027.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-25T00:17:16.623 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~ 2024-04-25T00:17:16.623 INFO:tasks.workunit.client.0.smithi027.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-25T00:17:16.623 INFO:tasks.workunit.client.0.smithi027.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-25T00:17:16.623 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:17:16.623 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:17:16.624 INFO:tasks.workunit.client.0.smithi027.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:17:16.624 INFO:tasks.workunit.client.0.smithi027.stderr: | ^ 2024-04-25T00:17:16.624 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-25T00:17:16.624 INFO:tasks.workunit.client.0.smithi027.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:17:16.624 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~ 2024-04-25T00:17:16.624 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-25T00:17:16.624 INFO:tasks.workunit.client.0.smithi027.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:17:16.624 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:17:16.624 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-25T00:17:16.624 INFO:tasks.workunit.client.0.smithi027.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-25T00:17:16.624 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~ 2024-04-25T00:17:16.624 INFO:tasks.workunit.client.0.smithi027.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-25T00:17:16.624 INFO:tasks.workunit.client.0.smithi027.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-25T00:17:16.624 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:17:16.625 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:17:16.625 INFO:tasks.workunit.client.0.smithi027.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:17:16.625 INFO:tasks.workunit.client.0.smithi027.stderr: | ^ 2024-04-25T00:17:16.625 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-25T00:17:16.625 INFO:tasks.workunit.client.0.smithi027.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:17:16.625 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~ 2024-04-25T00:17:16.625 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-25T00:17:16.625 INFO:tasks.workunit.client.0.smithi027.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:17:16.625 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:17:16.625 INFO:tasks.workunit.client.0.smithi027.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-25T00:17:16.625 INFO:tasks.workunit.client.0.smithi027.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-25T00:17:16.625 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~ 2024-04-25T00:17:16.625 INFO:tasks.workunit.client.0.smithi027.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-25T00:17:16.625 INFO:tasks.workunit.client.0.smithi027.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-25T00:17:16.625 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:17:16.676 INFO:tasks.workunit.client.0.smithi027.stderr:parser.c: In function ‘get_optstr’: 2024-04-25T00:17:16.676 INFO:tasks.workunit.client.0.smithi027.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-04-25T00:17:16.677 INFO:tasks.workunit.client.0.smithi027.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-04-25T00:17:16.677 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:17:16.677 INFO:tasks.workunit.client.0.smithi027.stderr:parser.c:200:14: note: source object declared here 2024-04-25T00:17:16.677 INFO:tasks.workunit.client.0.smithi027.stderr: 200 | char temp[BUFSIZE]; 2024-04-25T00:17:16.677 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~ 2024-04-25T00:17:16.683 INFO:tasks.workunit.client.0.smithi027.stderr:parser.c: In function ‘size64_convert’: 2024-04-25T00:17:16.683 INFO:tasks.workunit.client.0.smithi027.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-04-25T00:17:16.683 INFO:tasks.workunit.client.0.smithi027.stderr: 119 | memcpy(unit, "\0", 3); 2024-04-25T00:17:16.683 INFO:tasks.workunit.client.0.smithi027.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-04-25T00:17:16.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:16 smithi027 ceph-mon[29208]: pgmap v48: 129 pgs: 129 active+clean; 6.0 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 440 KiB/s wr, 50 op/s 2024-04-25T00:17:16.971 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-04-25T00:17:16.974 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-04-25T00:17:17.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:16 smithi139 ceph-mon[32833]: pgmap v48: 129 pgs: 129 active+clean; 6.0 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 440 KiB/s wr, 50 op/s 2024-04-25T00:17:17.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:16 smithi153 ceph-mon[32896]: pgmap v48: 129 pgs: 129 active+clean; 6.0 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 440 KiB/s wr, 50 op/s 2024-04-25T00:17:17.031 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-04-25T00:17:17.035 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-04-25T00:17:17.125 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-04-25T00:17:17.128 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-04-25T00:17:17.158 INFO:tasks.workunit.client.0.smithi027.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-04-25T00:17:17.162 INFO:tasks.workunit.client.0.smithi027.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-04-25T00:17:17.195 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-04-25T00:17:17.196 INFO:tasks.workunit.client.0.smithi027.stderr:+ cd .. 2024-04-25T00:17:17.196 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir tmp 2024-04-25T00:17:17.198 INFO:tasks.workunit.client.0.smithi027.stderr:+ cd tmp 2024-04-25T00:17:17.198 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in $mydir/*.ffsb 2024-04-25T00:17:17.198 INFO:tasks.workunit.client.0.smithi027.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-04-25T00:17:17.198 INFO:tasks.workunit.client.0.smithi027.stdout:FFSB version 6.0-RC2 started 2024-04-25T00:17:17.198 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:17:17.201 INFO:tasks.workunit.client.0.smithi027.stdout:benchmark time = 300 2024-04-25T00:17:17.201 INFO:tasks.workunit.client.0.smithi027.stdout:ThreadGroup 0 2024-04-25T00:17:17.201 INFO:tasks.workunit.client.0.smithi027.stdout:================ 2024-04-25T00:17:17.201 INFO:tasks.workunit.client.0.smithi027.stdout: num_threads = 32 2024-04-25T00:17:17.201 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:17:17.202 INFO:tasks.workunit.client.0.smithi027.stdout: read_random = off 2024-04-25T00:17:17.202 INFO:tasks.workunit.client.0.smithi027.stdout: read_size = 0 (0B) 2024-04-25T00:17:17.202 INFO:tasks.workunit.client.0.smithi027.stdout: read_blocksize = 0 (0B) 2024-04-25T00:17:17.202 INFO:tasks.workunit.client.0.smithi027.stdout: read_skip = off 2024-04-25T00:17:17.202 INFO:tasks.workunit.client.0.smithi027.stdout: read_skipsize = 0 (0B) 2024-04-25T00:17:17.202 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:17:17.202 INFO:tasks.workunit.client.0.smithi027.stdout: write_random = on 2024-04-25T00:17:17.202 INFO:tasks.workunit.client.0.smithi027.stdout: write_size = 5242880 (5MB) 2024-04-25T00:17:17.202 INFO:tasks.workunit.client.0.smithi027.stdout: fsync_file = 0 2024-04-25T00:17:17.202 INFO:tasks.workunit.client.0.smithi027.stdout: write_blocksize = 4096 (4KB) 2024-04-25T00:17:17.202 INFO:tasks.workunit.client.0.smithi027.stdout: wait time = 0 2024-04-25T00:17:17.202 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:17:17.202 INFO:tasks.workunit.client.0.smithi027.stdout: op weights 2024-04-25T00:17:17.202 INFO:tasks.workunit.client.0.smithi027.stdout: read = 0 (0.00%) 2024-04-25T00:17:17.203 INFO:tasks.workunit.client.0.smithi027.stdout: readall = 0 (0.00%) 2024-04-25T00:17:17.203 INFO:tasks.workunit.client.0.smithi027.stdout: write = 1 (100.00%) 2024-04-25T00:17:17.203 INFO:tasks.workunit.client.0.smithi027.stdout: create = 0 (0.00%) 2024-04-25T00:17:17.203 INFO:tasks.workunit.client.0.smithi027.stdout: append = 0 (0.00%) 2024-04-25T00:17:17.203 INFO:tasks.workunit.client.0.smithi027.stdout: delete = 0 (0.00%) 2024-04-25T00:17:17.203 INFO:tasks.workunit.client.0.smithi027.stdout: metaop = 0 (0.00%) 2024-04-25T00:17:17.203 INFO:tasks.workunit.client.0.smithi027.stdout: createdir = 0 (0.00%) 2024-04-25T00:17:17.203 INFO:tasks.workunit.client.0.smithi027.stdout: stat = 0 (0.00%) 2024-04-25T00:17:17.203 INFO:tasks.workunit.client.0.smithi027.stdout: writeall = 0 (0.00%) 2024-04-25T00:17:17.203 INFO:tasks.workunit.client.0.smithi027.stdout: writeall_fsync = 0 (0.00%) 2024-04-25T00:17:17.203 INFO:tasks.workunit.client.0.smithi027.stdout: open_close = 0 (0.00%) 2024-04-25T00:17:17.203 INFO:tasks.workunit.client.0.smithi027.stdout: write_fsync = 0 (0.00%) 2024-04-25T00:17:17.203 INFO:tasks.workunit.client.0.smithi027.stdout: create_fsync = 0 (0.00%) 2024-04-25T00:17:17.203 INFO:tasks.workunit.client.0.smithi027.stdout: append_fsync = 0 (0.00%) 2024-04-25T00:17:17.203 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:17:17.203 INFO:tasks.workunit.client.0.smithi027.stdout:FileSystem . 2024-04-25T00:17:17.204 INFO:tasks.workunit.client.0.smithi027.stdout:========== 2024-04-25T00:17:17.204 INFO:tasks.workunit.client.0.smithi027.stdout: num_dirs = 0 2024-04-25T00:17:17.204 INFO:tasks.workunit.client.0.smithi027.stdout: starting files = 128 2024-04-25T00:17:17.204 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:17:17.204 INFO:tasks.workunit.client.0.smithi027.stdout: min file size = 104857600 (100MB) 2024-04-25T00:17:17.204 INFO:tasks.workunit.client.0.smithi027.stdout: max file size = 104857600 (100MB) 2024-04-25T00:17:17.204 INFO:tasks.workunit.client.0.smithi027.stdout: directio = off 2024-04-25T00:17:17.204 INFO:tasks.workunit.client.0.smithi027.stdout: alignedio = on 2024-04-25T00:17:17.204 INFO:tasks.workunit.client.0.smithi027.stdout: bufferedio = off 2024-04-25T00:17:17.204 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:17:17.204 INFO:tasks.workunit.client.0.smithi027.stdout: aging is off 2024-04-25T00:17:17.204 INFO:tasks.workunit.client.0.smithi027.stdout: current utilization = 0.03% 2024-04-25T00:17:17.204 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:17:17.204 INFO:tasks.workunit.client.0.smithi027.stdout:checking existing fs: . 2024-04-25T00:17:17.205 INFO:tasks.workunit.client.0.smithi027.stdout:recreating new fileset 2024-04-25T00:17:17.205 INFO:tasks.workunit.client.0.smithi027.stderr:opendir: No such file or directory 2024-04-25T00:17:18.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:18 smithi027 ceph-mon[29208]: pgmap v49: 129 pgs: 129 active+clean; 7.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 600 KiB/s wr, 75 op/s 2024-04-25T00:17:18.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:18 smithi027 ceph-mon[29208]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-25T00:17:19.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:18 smithi139 ceph-mon[32833]: pgmap v49: 129 pgs: 129 active+clean; 7.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 600 KiB/s wr, 75 op/s 2024-04-25T00:17:19.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:18 smithi139 ceph-mon[32833]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-25T00:17:19.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:18 smithi153 ceph-mon[32896]: pgmap v49: 129 pgs: 129 active+clean; 7.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 600 KiB/s wr, 75 op/s 2024-04-25T00:17:19.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:18 smithi153 ceph-mon[32896]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-25T00:17:20.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:20 smithi027 ceph-mon[29208]: pgmap v50: 129 pgs: 129 active+clean; 7.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 529 KiB/s wr, 70 op/s 2024-04-25T00:17:21.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:20 smithi139 ceph-mon[32833]: pgmap v50: 129 pgs: 129 active+clean; 7.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 529 KiB/s wr, 70 op/s 2024-04-25T00:17:21.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:20 smithi153 ceph-mon[32896]: pgmap v50: 129 pgs: 129 active+clean; 7.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 529 KiB/s wr, 70 op/s 2024-04-25T00:17:22.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:22 smithi027 ceph-mon[29208]: pgmap v51: 129 pgs: 129 active+clean; 1.7 GiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 147 MiB/s wr, 180 op/s 2024-04-25T00:17:23.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:22 smithi139 ceph-mon[32833]: pgmap v51: 129 pgs: 129 active+clean; 1.7 GiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 147 MiB/s wr, 180 op/s 2024-04-25T00:17:23.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:22 smithi153 ceph-mon[32896]: pgmap v51: 129 pgs: 129 active+clean; 1.7 GiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 147 MiB/s wr, 180 op/s 2024-04-25T00:17:24.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:24 smithi027 ceph-mon[29208]: pgmap v52: 129 pgs: 129 active+clean; 1.7 GiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 147 MiB/s wr, 135 op/s 2024-04-25T00:17:25.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:24 smithi139 ceph-mon[32833]: pgmap v52: 129 pgs: 129 active+clean; 1.7 GiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 147 MiB/s wr, 135 op/s 2024-04-25T00:17:25.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:24 smithi153 ceph-mon[32896]: pgmap v52: 129 pgs: 129 active+clean; 1.7 GiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 147 MiB/s wr, 135 op/s 2024-04-25T00:17:26.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:26 smithi027 ceph-mon[29208]: pgmap v53: 129 pgs: 129 active+clean; 1.7 GiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 147 MiB/s wr, 135 op/s 2024-04-25T00:17:27.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:26 smithi139 ceph-mon[32833]: pgmap v53: 129 pgs: 129 active+clean; 1.7 GiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 147 MiB/s wr, 135 op/s 2024-04-25T00:17:27.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:26 smithi153 ceph-mon[32896]: pgmap v53: 129 pgs: 129 active+clean; 1.7 GiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 147 MiB/s wr, 135 op/s 2024-04-25T00:17:28.880 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:17:28.880 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:17:28.919 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:28 smithi027 ceph-mon[29208]: pgmap v54: 129 pgs: 129 active+clean; 5.7 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 489 MiB/s wr, 263 op/s 2024-04-25T00:17:29.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:28 smithi139 ceph-mon[32833]: pgmap v54: 129 pgs: 129 active+clean; 5.7 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 489 MiB/s wr, 263 op/s 2024-04-25T00:17:29.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:28 smithi153 ceph-mon[32896]: pgmap v54: 129 pgs: 129 active+clean; 5.7 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 489 MiB/s wr, 263 op/s 2024-04-25T00:17:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.090+0000 7f236d5b7640 1 -- 172.21.15.27:0/2384246887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2368100b20 msgr2=0x7f2368100fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.090+0000 7f236d5b7640 1 --2- 172.21.15.27:0/2384246887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2368100b20 0x7f2368100fa0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f235c009930 tx=0x7f235c02a420 comp rx=0 tx=0).stop 2024-04-25T00:17:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.091+0000 7f236d5b7640 1 -- 172.21.15.27:0/2384246887 shutdown_connections 2024-04-25T00:17:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.091+0000 7f236d5b7640 1 --2- 172.21.15.27:0/2384246887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23681014e0 0x7f236810d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.091+0000 7f236d5b7640 1 --2- 172.21.15.27:0/2384246887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2368100b20 0x7f2368100fa0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.091+0000 7f236d5b7640 1 --2- 172.21.15.27:0/2384246887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23681066a0 0x7f2368106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.091+0000 7f236d5b7640 1 -- 172.21.15.27:0/2384246887 >> 172.21.15.27:0/2384246887 conn(0x7f23680761c0 msgr2=0x7f23680765d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.091+0000 7f236d5b7640 1 -- 172.21.15.27:0/2384246887 shutdown_connections 2024-04-25T00:17:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.091+0000 7f236d5b7640 1 -- 172.21.15.27:0/2384246887 wait complete. 2024-04-25T00:17:29.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.092+0000 7f236d5b7640 1 Processor -- start 2024-04-25T00:17:29.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.092+0000 7f236d5b7640 1 -- start start 2024-04-25T00:17:29.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.093+0000 7f236d5b7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2368100b20 0x7f236810d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.093+0000 7f236d5b7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23681014e0 0x7f236810a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.093+0000 7f236d5b7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23681066a0 0x7f236810a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.093+0000 7f236d5b7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f236810f980 con 0x7f23681066a0 2024-04-25T00:17:29.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.093+0000 7f23677fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23681066a0 0x7f236810a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.093+0000 7f23677fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23681066a0 0x7f236810a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59656/0 (socket says 172.21.15.27:59656) 2024-04-25T00:17:29.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.093+0000 7f23677fe640 1 -- 172.21.15.27:0/1429505714 learned_addr learned my addr 172.21.15.27:0/1429505714 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:29.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.093+0000 7f236d5b7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f236810f800 con 0x7f23681014e0 2024-04-25T00:17:29.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.094+0000 7f236d5b7640 1 -- 172.21.15.27:0/1429505714 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f236810fb00 con 0x7f2368100b20 2024-04-25T00:17:29.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.095+0000 7f2366ffd640 1 --2- 172.21.15.27:0/1429505714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2368100b20 0x7f236810d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.096+0000 7f23667fc640 1 --2- 172.21.15.27:0/1429505714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23681014e0 0x7f236810a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.096+0000 7f23677fe640 1 -- 172.21.15.27:0/1429505714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2368100b20 msgr2=0x7f236810d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:29.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.096+0000 7f23677fe640 1 --2- 172.21.15.27:0/1429505714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2368100b20 0x7f236810d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.096+0000 7f23677fe640 1 -- 172.21.15.27:0/1429505714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23681014e0 msgr2=0x7f236810a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:29.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.096+0000 7f23677fe640 1 --2- 172.21.15.27:0/1429505714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23681014e0 0x7f236810a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.096+0000 7f23677fe640 1 -- 172.21.15.27:0/1429505714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f236810abb0 con 0x7f23681066a0 2024-04-25T00:17:29.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.097+0000 7f23677fe640 1 --2- 172.21.15.27:0/1429505714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23681066a0 0x7f236810a5f0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f236000b970 tx=0x7f236000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:29.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.097+0000 7f234ffff640 1 -- 172.21.15.27:0/1429505714 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2360019070 con 0x7f23681066a0 2024-04-25T00:17:29.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.097+0000 7f234ffff640 1 -- 172.21.15.27:0/1429505714 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f236000cb40 con 0x7f23681066a0 2024-04-25T00:17:29.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.097+0000 7f234ffff640 1 -- 172.21.15.27:0/1429505714 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2360007670 con 0x7f23681066a0 2024-04-25T00:17:29.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.097+0000 7f236d5b7640 1 -- 172.21.15.27:0/1429505714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f236810aea0 con 0x7f23681066a0 2024-04-25T00:17:29.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.097+0000 7f236d5b7640 1 -- 172.21.15.27:0/1429505714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23681aafe0 con 0x7f23681066a0 2024-04-25T00:17:29.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.102+0000 7f234ffff640 1 -- 172.21.15.27:0/1429505714 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f236001c070 con 0x7f23681066a0 2024-04-25T00:17:29.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.103+0000 7f234ffff640 1 --2- 172.21.15.27:0/1429505714 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f23500a2ed0 0x7f23500a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.105+0000 7f234ffff640 1 -- 172.21.15.27:0/1429505714 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f23600cea60 con 0x7f23681066a0 2024-04-25T00:17:29.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.108+0000 7f2366ffd640 1 --2- 172.21.15.27:0/1429505714 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f23500a2ed0 0x7f23500a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.111+0000 7f2366ffd640 1 --2- 172.21.15.27:0/1429505714 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f23500a2ed0 0x7f23500a5390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f23680777a0 tx=0x7f2358005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:29.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.113+0000 7f23703a4740 1 Processor -- start 2024-04-25T00:17:29.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.113+0000 7f23703a4740 1 -- start start 2024-04-25T00:17:29.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.119+0000 7f23703a4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 0x560cafed3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.119+0000 7f23703a4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafed3f50 0x560cafede0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.119+0000 7f23703a4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafede620 0x560cafee0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.119+0000 7f23703a4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560cafe386b0 con 0x560cafede620 2024-04-25T00:17:29.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.119+0000 7f23703a4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560cafde0880 con 0x560cafed3f50 2024-04-25T00:17:29.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.119+0000 7f23703a4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560cafe34670 con 0x560cafed3630 2024-04-25T00:17:29.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.121+0000 7f2366ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 0x560cafed3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.124+0000 7f2366ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 0x560cafed3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:35996/0 (socket says 172.21.15.27:35996) 2024-04-25T00:17:29.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.125+0000 7f2366ffd640 1 -- 172.21.15.27:0/1066731411 learned_addr learned my addr 172.21.15.27:0/1066731411 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:29.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.125+0000 7f23677fe640 1 --2- 172.21.15.27:0/1066731411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafede620 0x560cafee0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.129+0000 7f23667fc640 1 --2- 172.21.15.27:0/1066731411 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafed3f50 0x560cafede0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.131+0000 7f2366ffd640 1 -- 172.21.15.27:0/1066731411 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafed3f50 msgr2=0x560cafede0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:29.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.131+0000 7f2366ffd640 1 --2- 172.21.15.27:0/1066731411 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafed3f50 0x560cafede0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.131+0000 7f2366ffd640 1 -- 172.21.15.27:0/1066731411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafede620 msgr2=0x560cafee0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:29.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.131+0000 7f2366ffd640 1 --2- 172.21.15.27:0/1066731411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafede620 0x560cafee0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.131+0000 7f2366ffd640 1 -- 172.21.15.27:0/1066731411 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cafc167d0 con 0x560cafed3630 2024-04-25T00:17:29.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.138+0000 7f2366ffd640 1 --2- 172.21.15.27:0/1066731411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 0x560cafed3a10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f23580058c0 tx=0x7f2358014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d91c1454209e594 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:29.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.145+0000 7f234d7fa640 1 -- 172.21.15.27:0/1066731411 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2358026430 con 0x560cafed3630 2024-04-25T00:17:29.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.146+0000 7f234d7fa640 1 -- 172.21.15.27:0/1066731411 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f235801c8f0 con 0x560cafed3630 2024-04-25T00:17:29.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.146+0000 7f234d7fa640 1 -- 172.21.15.27:0/1066731411 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235801da60 con 0x560cafed3630 2024-04-25T00:17:29.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.149+0000 7f23703a4740 1 -- 172.21.15.27:0/1066731411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 msgr2=0x560cafed3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:29.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.149+0000 7f23703a4740 1 --2- 172.21.15.27:0/1066731411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 0x560cafed3a10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f23580058c0 tx=0x7f2358014000 comp rx=0 tx=0).stop 2024-04-25T00:17:29.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.160+0000 7f23703a4740 1 -- 172.21.15.27:0/1066731411 shutdown_connections 2024-04-25T00:17:29.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.160+0000 7f23703a4740 1 --2- 172.21.15.27:0/1066731411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafede620 0x560cafee0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.160+0000 7f23703a4740 1 --2- 172.21.15.27:0/1066731411 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafed3f50 0x560cafede0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.160+0000 7f23703a4740 1 --2- 172.21.15.27:0/1066731411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 0x560cafed3a10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.160+0000 7f23703a4740 1 -- 172.21.15.27:0/1066731411 >> 172.21.15.27:0/1066731411 conn(0x560cafecf950 msgr2=0x560cafecfd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:29.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.166+0000 7f23703a4740 1 -- 172.21.15.27:0/1066731411 shutdown_connections 2024-04-25T00:17:29.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.173+0000 7f23703a4740 1 -- 172.21.15.27:0/1066731411 wait complete. 2024-04-25T00:17:29.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.174+0000 7f23703a4740 1 -- start start 2024-04-25T00:17:29.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.174+0000 7f23703a4740 1 Processor -- start 2024-04-25T00:17:29.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.181+0000 7f23703a4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafed3630 0x560cafedb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.181+0000 7f23703a4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafed3f50 0x560cafedba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.182+0000 7f23703a4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafede620 0x560cafedbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.182+0000 7f23703a4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560cafee1410 con 0x560cafed3f50 2024-04-25T00:17:29.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.182+0000 7f23703a4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560cafdf3f50 con 0x560cafed3630 2024-04-25T00:17:29.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.182+0000 7f23703a4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560cafee1590 con 0x560cafede620 2024-04-25T00:17:29.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.182+0000 7f23703a4740 10 client.0 _ll_register_callbacks cb 0x560cafe2f000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:17:29.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.183+0000 7f23667fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafed3f50 0x560cafedba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.183+0000 7f23667fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafed3f50 0x560cafedba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59670/0 (socket says 172.21.15.27:59670) 2024-04-25T00:17:29.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.183+0000 7f23667fc640 1 -- 172.21.15.27:0/1559482875 learned_addr learned my addr 172.21.15.27:0/1559482875 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:29.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.183+0000 7f23677fe640 1 --2- 172.21.15.27:0/1559482875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafede620 0x560cafedbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.184+0000 7f23667fc640 1 -- 172.21.15.27:0/1559482875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafede620 msgr2=0x560cafedbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:29.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.184+0000 7f23667fc640 1 --2- 172.21.15.27:0/1559482875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafede620 0x560cafedbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.184+0000 7f23667fc640 1 -- 172.21.15.27:0/1559482875 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafed3630 msgr2=0x560cafedb540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:17:29.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.184+0000 7f23667fc640 1 --2- 172.21.15.27:0/1559482875 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafed3630 0x560cafedb540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.184+0000 7f23667fc640 1 -- 172.21.15.27:0/1559482875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cafb4c3f0 con 0x560cafed3f50 2024-04-25T00:17:29.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.187+0000 7f23667fc640 1 --2- 172.21.15.27:0/1559482875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafed3f50 0x560cafedba80 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f235c009930 tx=0x7f235c004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a2b6fea2af26b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:29.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.187+0000 7f234aff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:17:29.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.188+0000 7f234aff5640 1 -- 172.21.15.27:0/1559482875 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235c035030 con 0x560cafed3f50 2024-04-25T00:17:29.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.188+0000 7f234aff5640 1 -- 172.21.15.27:0/1559482875 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f235c003ac0 con 0x560cafed3f50 2024-04-25T00:17:29.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.191+0000 7f23703a4740 1 -- 172.21.15.27:0/1559482875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f235801c8f0 con 0x560cafed3f50 2024-04-25T00:17:29.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.193+0000 7f234aff5640 1 -- 172.21.15.27:0/1559482875 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f235c003cc0 con 0x560cafed3f50 2024-04-25T00:17:29.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.195+0000 7f23703a4740 10 client.16146 fetch_fsmap learned FSMap version 30 2024-04-25T00:17:29.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.195+0000 7f23703a4740 1 -- 172.21.15.27:0/1559482875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560cafb0c0f0 con 0x560cafed3f50 2024-04-25T00:17:29.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.197+0000 7f234aff5640 1 -- 172.21.15.27:0/1559482875 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f235801c8f0 con 0x560cafed3f50 2024-04-25T00:17:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.199+0000 7f234aff5640 1 -- 172.21.15.27:0/1559482875 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f235c004cc0 con 0x560cafed3f50 2024-04-25T00:17:29.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.201+0000 7f23703a4740 10 client.16146 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:17:29.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.201+0000 7f23703a4740 10 client.16146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:17:29.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.201+0000 7f23703a4740 20 client.16146 populate_metadata read hostname 'smithi027' 2024-04-25T00:17:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.201+0000 7f23703a4740 1 --2- 172.21.15.27:0/1559482875 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560cafeee990 0x560cafef0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.201+0000 7f23703a4740 4 client.16146 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:17:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.201+0000 7f23703a4740 1 -- 172.21.15.27:0/1559482875 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560cafe45bf0 con 0x560cafeee990 2024-04-25T00:17:29.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.210+0000 7f2366ffd640 1 --2- 172.21.15.27:0/1559482875 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560cafeee990 0x560cafef0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.218+0000 7f2366ffd640 1 --2- 172.21.15.27:0/1559482875 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560cafeee990 0x560cafef0e00 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b4feae8ca4f8005 server_cookie=b27fe8df0d2631bf in_seq=0 out_seq=0 2024-04-25T00:17:29.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.219+0000 7f234aff5640 10 client.16146 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:17:29.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.224+0000 7f234aff5640 1 -- 172.21.15.27:0/1559482875 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560cafe45bf0 con 0x560cafeee990 2024-04-25T00:17:29.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.224+0000 7f234aff5640 10 client.16146 handle_command_reply: tid=0 2024-04-25T00:17:29.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.228+0000 7f23703a4740 1 client.16146 shutdown 2024-04-25T00:17:29.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.232+0000 7f23703a4740 1 -- 172.21.15.27:0/1559482875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafed3f50 msgr2=0x560cafedba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:29.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.232+0000 7f23703a4740 1 --2- 172.21.15.27:0/1559482875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafed3f50 0x560cafedba80 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f235c009930 tx=0x7f235c004570 comp rx=0 tx=0).stop 2024-04-25T00:17:29.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.232+0000 7f23703a4740 1 -- 172.21.15.27:0/1559482875 shutdown_connections 2024-04-25T00:17:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.232+0000 7f23703a4740 1 --2- 172.21.15.27:0/1559482875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafede620 0x560cafedbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.232+0000 7f23703a4740 1 --2- 172.21.15.27:0/1559482875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafed3f50 0x560cafedba80 secure :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f235c009930 tx=0x7f235c004570 comp rx=0 tx=0).stop 2024-04-25T00:17:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.232+0000 7f23703a4740 1 --2- 172.21.15.27:0/1559482875 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560cafeee990 0x560cafef0e00 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.232+0000 7f23703a4740 1 --2- 172.21.15.27:0/1559482875 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafed3630 0x560cafedb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.232+0000 7f23703a4740 1 -- 172.21.15.27:0/1559482875 >> 172.21.15.27:0/1559482875 conn(0x560cafecf950 msgr2=0x560cafed5650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.232+0000 7f234aff5640 0 client.16146 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:17:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.233+0000 7f23703a4740 1 -- 172.21.15.27:0/1559482875 shutdown_connections 2024-04-25T00:17:29.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.237+0000 7f23703a4740 1 -- 172.21.15.27:0/1559482875 wait complete. 2024-04-25T00:17:29.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.238+0000 7f23703a4740 20 client.16146 trim_cache size 0 max 16384 2024-04-25T00:17:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.275+0000 7f23703a4740 1 Processor -- start 2024-04-25T00:17:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.275+0000 7f23703a4740 1 -- start start 2024-04-25T00:17:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.275+0000 7f23703a4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 0x560cafeea300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.275+0000 7f23703a4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafede620 0x560cafecaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.275+0000 7f23703a4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafeda6c0 0x560cafed5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.275+0000 7f23703a4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f235400ffb0 con 0x560cafeda6c0 2024-04-25T00:17:29.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.275+0000 7f23703a4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2354011bb0 con 0x560cafede620 2024-04-25T00:17:29.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.275+0000 7f23703a4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2354010680 con 0x560cafed3630 2024-04-25T00:17:29.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.275+0000 7f2366ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 0x560cafeea300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.275+0000 7f2366ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 0x560cafeea300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:36018/0 (socket says 172.21.15.27:36018) 2024-04-25T00:17:29.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.275+0000 7f2366ffd640 1 -- 172.21.15.27:0/3625768014 learned_addr learned my addr 172.21.15.27:0/3625768014 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:29.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.275+0000 7f23667fc640 1 --2- 172.21.15.27:0/3625768014 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafede620 0x560cafecaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.275+0000 7f23677fe640 1 --2- 172.21.15.27:0/3625768014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafeda6c0 0x560cafed5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.276+0000 7f2366ffd640 1 -- 172.21.15.27:0/3625768014 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafede620 msgr2=0x560cafecaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:29.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.276+0000 7f2366ffd640 1 --2- 172.21.15.27:0/3625768014 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafede620 0x560cafecaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.276+0000 7f2366ffd640 1 -- 172.21.15.27:0/3625768014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafeda6c0 msgr2=0x560cafed5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:29.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.276+0000 7f2366ffd640 1 --2- 172.21.15.27:0/3625768014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafeda6c0 0x560cafed5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.276+0000 7f2366ffd640 1 -- 172.21.15.27:0/3625768014 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cafec1090 con 0x560cafed3630 2024-04-25T00:17:29.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.276+0000 7f23667fc640 1 --2- 172.21.15.27:0/3625768014 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafede620 0x560cafecaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:17:29.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.276+0000 7f2366ffd640 1 --2- 172.21.15.27:0/3625768014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 0x560cafeea300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f23580118d0 tx=0x7f235801d520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35ee4f3b896f1451 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:29.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.276+0000 7f234cff9640 1 -- 172.21.15.27:0/3625768014 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235802b040 con 0x560cafed3630 2024-04-25T00:17:29.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.276+0000 7f234cff9640 1 -- 172.21.15.27:0/3625768014 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2358005520 con 0x560cafed3630 2024-04-25T00:17:29.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.276+0000 7f234cff9640 1 -- 172.21.15.27:0/3625768014 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235801daf0 con 0x560cafed3630 2024-04-25T00:17:29.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.277+0000 7f23703a4740 1 -- 172.21.15.27:0/3625768014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 msgr2=0x560cafeea300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.277+0000 7f23703a4740 1 --2- 172.21.15.27:0/3625768014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 0x560cafeea300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f23580118d0 tx=0x7f235801d520 comp rx=0 tx=0).stop 2024-04-25T00:17:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.277+0000 7f23703a4740 1 -- 172.21.15.27:0/3625768014 shutdown_connections 2024-04-25T00:17:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.277+0000 7f23703a4740 1 --2- 172.21.15.27:0/3625768014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafeda6c0 0x560cafed5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.277+0000 7f23703a4740 1 --2- 172.21.15.27:0/3625768014 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafede620 0x560cafecaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.277+0000 7f23703a4740 1 --2- 172.21.15.27:0/3625768014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 0x560cafeea300 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.277+0000 7f23703a4740 1 -- 172.21.15.27:0/3625768014 >> 172.21.15.27:0/3625768014 conn(0x560cafecf950 msgr2=0x560cafedb500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.277+0000 7f23703a4740 1 -- 172.21.15.27:0/3625768014 shutdown_connections 2024-04-25T00:17:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.277+0000 7f23703a4740 1 -- 172.21.15.27:0/3625768014 wait complete. 2024-04-25T00:17:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.278+0000 7f23703a4740 1 -- start start 2024-04-25T00:17:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.278+0000 7f23703a4740 1 Processor -- start 2024-04-25T00:17:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.278+0000 7f23703a4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 0x560cafee6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.278+0000 7f23703a4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafeda6c0 0x560cafed8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.278+0000 7f23703a4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafede620 0x560cafed8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.278+0000 7f2366ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 0x560cafee6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.278+0000 7f23667fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafeda6c0 0x560cafed8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.278+0000 7f2366ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 0x560cafee6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:36034/0 (socket says 172.21.15.27:36034) 2024-04-25T00:17:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.278+0000 7f23667fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafeda6c0 0x560cafed8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59690/0 (socket says 172.21.15.27:59690) 2024-04-25T00:17:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.278+0000 7f2366ffd640 1 -- 172.21.15.27:0/697242281 learned_addr learned my addr 172.21.15.27:0/697242281 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.278+0000 7f23677fe640 1 --2- 172.21.15.27:0/697242281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafede620 0x560cafed8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.278+0000 7f23703a4740 1 -- 172.21.15.27:0/697242281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2354011420 con 0x560cafeda6c0 2024-04-25T00:17:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.278+0000 7f23703a4740 1 -- 172.21.15.27:0/697242281 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2354010d50 con 0x560cafede620 2024-04-25T00:17:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.278+0000 7f23703a4740 1 -- 172.21.15.27:0/697242281 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f235400df70 con 0x560cafed3630 2024-04-25T00:17:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.278+0000 7f23703a4740 10 client.0 _ll_register_callbacks cb 0x7f235c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:17:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.279+0000 7f23667fc640 1 -- 172.21.15.27:0/697242281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 msgr2=0x560cafee6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.279+0000 7f23667fc640 1 --2- 172.21.15.27:0/697242281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 0x560cafee6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.279+0000 7f23667fc640 1 -- 172.21.15.27:0/697242281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafede620 msgr2=0x560cafed8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.279+0000 7f23667fc640 1 --2- 172.21.15.27:0/697242281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafede620 0x560cafed8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.279+0000 7f23667fc640 1 -- 172.21.15.27:0/697242281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560caff53f80 con 0x560cafeda6c0 2024-04-25T00:17:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.279+0000 7f23667fc640 1 --2- 172.21.15.27:0/697242281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafeda6c0 0x560cafed8050 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x560cafedc600 tx=0x7f235c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb8e3044f9d1171d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.279+0000 7f234aff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:17:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.279+0000 7f234aff5640 1 -- 172.21.15.27:0/697242281 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235c053040 con 0x560cafeda6c0 2024-04-25T00:17:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.279+0000 7f234aff5640 1 -- 172.21.15.27:0/697242281 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f235c005220 con 0x560cafeda6c0 2024-04-25T00:17:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.279+0000 7f23703a4740 1 -- 172.21.15.27:0/697242281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2358005520 con 0x560cafeda6c0 2024-04-25T00:17:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.279+0000 7f234aff5640 1 -- 172.21.15.27:0/697242281 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f235c060050 con 0x560cafeda6c0 2024-04-25T00:17:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.279+0000 7f23703a4740 10 client.16152 fetch_fsmap learned FSMap version 30 2024-04-25T00:17:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.279+0000 7f23703a4740 1 -- 172.21.15.27:0/697242281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560caff55050 con 0x560cafeda6c0 2024-04-25T00:17:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.279+0000 7f234aff5640 1 -- 172.21.15.27:0/697242281 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f235c040070 con 0x560cafeda6c0 2024-04-25T00:17:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.280+0000 7f234aff5640 1 -- 172.21.15.27:0/697242281 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f235c005420 con 0x560cafeda6c0 2024-04-25T00:17:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.280+0000 7f23703a4740 10 client.16152 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:17:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.280+0000 7f23703a4740 10 client.16152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:17:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.280+0000 7f23703a4740 20 client.16152 populate_metadata read hostname 'smithi027' 2024-04-25T00:17:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.280+0000 7f23703a4740 1 --2- 172.21.15.27:0/697242281 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560caff55800 0x560caff57c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.280+0000 7f23703a4740 4 client.16152 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:17:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.280+0000 7f23703a4740 1 -- 172.21.15.27:0/697242281 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f2358005520 con 0x560caff55800 2024-04-25T00:17:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.280+0000 7f2366ffd640 1 --2- 172.21.15.27:0/697242281 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560caff55800 0x560caff57c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.281+0000 7f234aff5640 10 client.16152 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:17:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.281+0000 7f2366ffd640 1 --2- 172.21.15.27:0/697242281 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560caff55800 0x560caff57c20 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56e170f079f716bc server_cookie=d2fad0d7d5624f57 in_seq=0 out_seq=0 2024-04-25T00:17:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.281+0000 7f234aff5640 1 -- 172.21.15.27:0/697242281 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f2358005520 con 0x560caff55800 2024-04-25T00:17:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.281+0000 7f234aff5640 10 client.16152 handle_command_reply: tid=0 2024-04-25T00:17:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.281+0000 7f23703a4740 1 client.16152 shutdown 2024-04-25T00:17:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.281+0000 7f23703a4740 1 -- 172.21.15.27:0/697242281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafeda6c0 msgr2=0x560cafed8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.281+0000 7f23703a4740 1 --2- 172.21.15.27:0/697242281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafeda6c0 0x560cafed8050 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x560cafedc600 tx=0x7f235c005fd0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.281+0000 7f23703a4740 1 -- 172.21.15.27:0/697242281 shutdown_connections 2024-04-25T00:17:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.281+0000 7f23703a4740 1 --2- 172.21.15.27:0/697242281 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560caff55800 0x560caff57c20 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.281+0000 7f23703a4740 1 --2- 172.21.15.27:0/697242281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560cafede620 0x560cafed8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.281+0000 7f23703a4740 1 --2- 172.21.15.27:0/697242281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560cafeda6c0 0x560cafed8050 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.281+0000 7f234aff5640 0 client.16152 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:17:29.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.281+0000 7f23703a4740 1 --2- 172.21.15.27:0/697242281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560cafed3630 0x560cafee6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.281+0000 7f23703a4740 1 -- 172.21.15.27:0/697242281 >> 172.21.15.27:0/697242281 conn(0x560cafecf950 msgr2=0x560cafedbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:29.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.281+0000 7f23703a4740 1 -- 172.21.15.27:0/697242281 shutdown_connections 2024-04-25T00:17:29.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.281+0000 7f23703a4740 1 -- 172.21.15.27:0/697242281 wait complete. 2024-04-25T00:17:29.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.282+0000 7f23703a4740 20 client.16152 trim_cache size 0 max 16384 2024-04-25T00:17:29.289 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:17:29.289 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T00:17:29.289 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T00:17:29.289 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:17:29.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.282+0000 7f236d5b7640 1 -- 172.21.15.27:0/1429505714 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f23500a2ed0 msgr2=0x7f23500a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:29.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.282+0000 7f236d5b7640 1 --2- 172.21.15.27:0/1429505714 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f23500a2ed0 0x7f23500a5390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f23680777a0 tx=0x7f2358005f90 comp rx=0 tx=0).stop 2024-04-25T00:17:29.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.282+0000 7f236d5b7640 1 -- 172.21.15.27:0/1429505714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23681066a0 msgr2=0x7f236810a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.282+0000 7f236d5b7640 1 --2- 172.21.15.27:0/1429505714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23681066a0 0x7f236810a5f0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f236000b970 tx=0x7f236000be30 comp rx=0 tx=0).stop 2024-04-25T00:17:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.282+0000 7f236d5b7640 1 -- 172.21.15.27:0/1429505714 shutdown_connections 2024-04-25T00:17:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.282+0000 7f236d5b7640 1 --2- 172.21.15.27:0/1429505714 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f23500a2ed0 0x7f23500a5390 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.282+0000 7f236d5b7640 1 --2- 172.21.15.27:0/1429505714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23681066a0 0x7f236810a5f0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.282+0000 7f236d5b7640 1 --2- 172.21.15.27:0/1429505714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23681014e0 0x7f236810a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.282+0000 7f236d5b7640 1 --2- 172.21.15.27:0/1429505714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2368100b20 0x7f236810d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.282+0000 7f236d5b7640 1 -- 172.21.15.27:0/1429505714 >> 172.21.15.27:0/1429505714 conn(0x7f23680761c0 msgr2=0x7f23681102d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.282+0000 7f236d5b7640 1 -- 172.21.15.27:0/1429505714 shutdown_connections 2024-04-25T00:17:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.282+0000 7f236d5b7640 1 -- 172.21.15.27:0/1429505714 wait complete. 2024-04-25T00:17:29.295 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T00:17:29.295 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.542+0000 7fc88829a640 1 -- 172.21.15.27:0/3887376085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8801032f0 msgr2=0x7fc88010b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.542+0000 7fc88829a640 1 --2- 172.21.15.27:0/3887376085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8801032f0 0x7fc88010b7d0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc870009900 tx=0x7fc87002a510 comp rx=0 tx=0).stop 2024-04-25T00:17:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.543+0000 7fc88829a640 1 -- 172.21.15.27:0/3887376085 shutdown_connections 2024-04-25T00:17:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.543+0000 7fc88829a640 1 --2- 172.21.15.27:0/3887376085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc88010bd10 0x7fc88010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.543+0000 7fc88829a640 1 --2- 172.21.15.27:0/3887376085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8801032f0 0x7fc88010b7d0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.543+0000 7fc88829a640 1 --2- 172.21.15.27:0/3887376085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8801029d0 0x7fc880102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.543+0000 7fc88829a640 1 -- 172.21.15.27:0/3887376085 >> 172.21.15.27:0/3887376085 conn(0x7fc880077740 msgr2=0x7fc880077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.543+0000 7fc88829a640 1 -- 172.21.15.27:0/3887376085 shutdown_connections 2024-04-25T00:17:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.543+0000 7fc88829a640 1 -- 172.21.15.27:0/3887376085 wait complete. 2024-04-25T00:17:29.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.544+0000 7fc88829a640 1 Processor -- start 2024-04-25T00:17:29.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.544+0000 7fc88829a640 1 -- start start 2024-04-25T00:17:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.545+0000 7fc88829a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8801029d0 0x7fc880100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.545+0000 7fc88829a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8801032f0 0x7fc880100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.545+0000 7fc88829a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc88010bd10 0x7fc8801010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.545+0000 7fc88829a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc880111c20 con 0x7fc88010bd10 2024-04-25T00:17:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.545+0000 7fc88829a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc880111aa0 con 0x7fc8801032f0 2024-04-25T00:17:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.545+0000 7fc88829a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc880111da0 con 0x7fc8801029d0 2024-04-25T00:17:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.545+0000 7fc886837640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc88010bd10 0x7fc8801010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.545+0000 7fc886837640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc88010bd10 0x7fc8801010a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59708/0 (socket says 172.21.15.27:59708) 2024-04-25T00:17:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.545+0000 7fc886837640 1 -- 172.21.15.27:0/1858887703 learned_addr learned my addr 172.21.15.27:0/1858887703 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.545+0000 7fc885835640 1 --2- 172.21.15.27:0/1858887703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8801032f0 0x7fc880100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.545+0000 7fc886837640 1 -- 172.21.15.27:0/1858887703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8801029d0 msgr2=0x7fc880100620 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:17:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.545+0000 7fc886837640 1 --2- 172.21.15.27:0/1858887703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8801029d0 0x7fc880100620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.545+0000 7fc886837640 1 -- 172.21.15.27:0/1858887703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8801032f0 msgr2=0x7fc880100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.545+0000 7fc886837640 1 --2- 172.21.15.27:0/1858887703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8801032f0 0x7fc880100b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.546+0000 7fc886837640 1 -- 172.21.15.27:0/1858887703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8801a6460 con 0x7fc88010bd10 2024-04-25T00:17:29.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.546+0000 7fc886837640 1 --2- 172.21.15.27:0/1858887703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc88010bd10 0x7fc8801010a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc87800b970 tx=0x7fc87800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:29.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.546+0000 7fc8777fe640 1 -- 172.21.15.27:0/1858887703 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc878019070 con 0x7fc88010bd10 2024-04-25T00:17:29.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.546+0000 7fc88829a640 1 -- 172.21.15.27:0/1858887703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8801a6750 con 0x7fc88010bd10 2024-04-25T00:17:29.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.546+0000 7fc8777fe640 1 -- 172.21.15.27:0/1858887703 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc87800cb40 con 0x7fc88010bd10 2024-04-25T00:17:29.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.547+0000 7fc8777fe640 1 -- 172.21.15.27:0/1858887703 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87801d610 con 0x7fc88010bd10 2024-04-25T00:17:29.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.547+0000 7fc88829a640 1 -- 172.21.15.27:0/1858887703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8801a6c90 con 0x7fc88010bd10 2024-04-25T00:17:29.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.549+0000 7fc88829a640 1 -- 172.21.15.27:0/1858887703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc880068560 con 0x7fc88010bd10 2024-04-25T00:17:29.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.550+0000 7fc8777fe640 1 -- 172.21.15.27:0/1858887703 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc87801c070 con 0x7fc88010bd10 2024-04-25T00:17:29.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.550+0000 7fc8777fe640 1 --2- 172.21.15.27:0/1858887703 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc86c0a2d80 0x7fc86c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.551+0000 7fc8777fe640 1 -- 172.21.15.27:0/1858887703 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fc8780cecf0 con 0x7fc88010bd10 2024-04-25T00:17:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.551+0000 7fc886036640 1 --2- 172.21.15.27:0/1858887703 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc86c0a2d80 0x7fc86c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:29.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.552+0000 7fc886036640 1 --2- 172.21.15.27:0/1858887703 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc86c0a2d80 0x7fc86c0a5240 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0096b0 tx=0x7fc87c006c20 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:29.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.559+0000 7fc8777fe640 1 -- 172.21.15.27:0/1858887703 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc87809b6b0 con 0x7fc88010bd10 2024-04-25T00:17:29.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.961+0000 7fc88829a640 1 -- 172.21.15.27:0/1858887703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8801a6f50 con 0x7fc88010bd10 2024-04-25T00:17:29.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.962+0000 7fc8777fe640 1 -- 172.21.15.27:0/1858887703 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11951 (secure 0 0 0) 0x7fc87809afb0 con 0x7fc88010bd10 2024-04-25T00:17:29.964 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:17:29.964 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:17:08.383555+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24496,24428,24521,24514,24430]},"id":1}]} 2024-04-25T00:17:29.964 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 30 2024-04-25T00:17:29.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.965+0000 7fc88829a640 1 -- 172.21.15.27:0/1858887703 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc86c0a2d80 msgr2=0x7fc86c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:29.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.965+0000 7fc88829a640 1 --2- 172.21.15.27:0/1858887703 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc86c0a2d80 0x7fc86c0a5240 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0096b0 tx=0x7fc87c006c20 comp rx=0 tx=0).stop 2024-04-25T00:17:29.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.965+0000 7fc88829a640 1 -- 172.21.15.27:0/1858887703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc88010bd10 msgr2=0x7fc8801010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:29.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.965+0000 7fc88829a640 1 --2- 172.21.15.27:0/1858887703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc88010bd10 0x7fc8801010a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc87800b970 tx=0x7fc87800be30 comp rx=0 tx=0).stop 2024-04-25T00:17:29.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.966+0000 7fc88829a640 1 -- 172.21.15.27:0/1858887703 shutdown_connections 2024-04-25T00:17:29.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.966+0000 7fc88829a640 1 --2- 172.21.15.27:0/1858887703 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc86c0a2d80 0x7fc86c0a5240 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.966+0000 7fc88829a640 1 --2- 172.21.15.27:0/1858887703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc88010bd10 0x7fc8801010a0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.966+0000 7fc88829a640 1 --2- 172.21.15.27:0/1858887703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8801032f0 0x7fc880100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.966+0000 7fc88829a640 1 --2- 172.21.15.27:0/1858887703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8801029d0 0x7fc880100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.966+0000 7fc88829a640 1 -- 172.21.15.27:0/1858887703 >> 172.21.15.27:0/1858887703 conn(0x7fc880077740 msgr2=0x7fc8800fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:29.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.966+0000 7fc88829a640 1 -- 172.21.15.27:0/1858887703 shutdown_connections 2024-04-25T00:17:29.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:29.966+0000 7fc88829a640 1 -- 172.21.15.27:0/1858887703 wait complete. 2024-04-25T00:17:29.980 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T00:17:30.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.247+0000 7fe141458640 1 -- 172.21.15.27:0/2942365737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe13c10ff90 msgr2=0x7fe13c10da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.247+0000 7fe141458640 1 --2- 172.21.15.27:0/2942365737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe13c10ff90 0x7fe13c10da80 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fe130009930 tx=0x7fe13002a440 comp rx=0 tx=0).stop 2024-04-25T00:17:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.248+0000 7fe141458640 1 -- 172.21.15.27:0/2942365737 shutdown_connections 2024-04-25T00:17:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.248+0000 7fe141458640 1 --2- 172.21.15.27:0/2942365737 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe13c11af20 0x7fe13c11d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.248+0000 7fe141458640 1 --2- 172.21.15.27:0/2942365737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe13c10dfc0 0x7fe13c10e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.248+0000 7fe141458640 1 --2- 172.21.15.27:0/2942365737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe13c10ff90 0x7fe13c10da80 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.248+0000 7fe141458640 1 -- 172.21.15.27:0/2942365737 >> 172.21.15.27:0/2942365737 conn(0x7fe13c06c850 msgr2=0x7fe13c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.248+0000 7fe141458640 1 -- 172.21.15.27:0/2942365737 shutdown_connections 2024-04-25T00:17:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.248+0000 7fe141458640 1 -- 172.21.15.27:0/2942365737 wait complete. 2024-04-25T00:17:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.249+0000 7fe141458640 1 Processor -- start 2024-04-25T00:17:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.249+0000 7fe141458640 1 -- start start 2024-04-25T00:17:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.250+0000 7fe141458640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe13c10dfc0 0x7fe13c1a6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.251+0000 7fe141458640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe13c10ff90 0x7fe13c1a74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.251+0000 7fe141458640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe13c11af20 0x7fe13c1ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.251+0000 7fe141458640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c120980 con 0x7fe13c10ff90 2024-04-25T00:17:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.251+0000 7fe141458640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c120800 con 0x7fe13c11af20 2024-04-25T00:17:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.251+0000 7fe141458640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c120b00 con 0x7fe13c10dfc0 2024-04-25T00:17:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.252+0000 7fe13b7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe13c11af20 0x7fe13c1ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.253+0000 7fe13b7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe13c11af20 0x7fe13c1ab840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:60852/0 (socket says 172.21.15.27:60852) 2024-04-25T00:17:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.253+0000 7fe13b7fe640 1 -- 172.21.15.27:0/2502802388 learned_addr learned my addr 172.21.15.27:0/2502802388 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.254+0000 7fe13affd640 1 --2- 172.21.15.27:0/2502802388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe13c10dfc0 0x7fe13c1a6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.254+0000 7fe13a7fc640 1 --2- 172.21.15.27:0/2502802388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe13c10ff90 0x7fe13c1a74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.255+0000 7fe13a7fc640 1 -- 172.21.15.27:0/2502802388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe13c10dfc0 msgr2=0x7fe13c1a6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.255+0000 7fe13a7fc640 1 --2- 172.21.15.27:0/2502802388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe13c10dfc0 0x7fe13c1a6f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.255+0000 7fe13a7fc640 1 -- 172.21.15.27:0/2502802388 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe13c11af20 msgr2=0x7fe13c1ab840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.255+0000 7fe13a7fc640 1 --2- 172.21.15.27:0/2502802388 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe13c11af20 0x7fe13c1ab840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.255+0000 7fe13a7fc640 1 -- 172.21.15.27:0/2502802388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe13c1abf70 con 0x7fe13c10ff90 2024-04-25T00:17:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.255+0000 7fe13a7fc640 1 --2- 172.21.15.27:0/2502802388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe13c10ff90 0x7fe13c1a74c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009fd0 tx=0x7fe12c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.256+0000 7fe123fff640 1 -- 172.21.15.27:0/2502802388 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c015020 con 0x7fe13c10ff90 2024-04-25T00:17:30.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.256+0000 7fe123fff640 1 -- 172.21.15.27:0/2502802388 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe12c011d10 con 0x7fe13c10ff90 2024-04-25T00:17:30.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.256+0000 7fe123fff640 1 -- 172.21.15.27:0/2502802388 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c00f2e0 con 0x7fe13c10ff90 2024-04-25T00:17:30.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.257+0000 7fe141458640 1 -- 172.21.15.27:0/2502802388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe13c1ac260 con 0x7fe13c10ff90 2024-04-25T00:17:30.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.257+0000 7fe141458640 1 -- 172.21.15.27:0/2502802388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe13c1b3af0 con 0x7fe13c10ff90 2024-04-25T00:17:30.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.262+0000 7fe144165740 1 Processor -- start 2024-04-25T00:17:30.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.262+0000 7fe144165740 1 -- start start 2024-04-25T00:17:30.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.263+0000 7fe144165740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01ea580 0x5561d01ea960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.263+0000 7fe144165740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01eaea0 0x5561d01f5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.263+0000 7fe144165740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01f5570 0x5561d01f7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.263+0000 7fe144165740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5561d014f670 con 0x5561d01ea580 2024-04-25T00:17:30.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.263+0000 7fe144165740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5561d00f7970 con 0x5561d01eaea0 2024-04-25T00:17:30.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.263+0000 7fe144165740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5561d014b630 con 0x5561d01f5570 2024-04-25T00:17:30.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.264+0000 7fe13affd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01eaea0 0x5561d01f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.264+0000 7fe13affd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01eaea0 0x5561d01f5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:60856/0 (socket says 172.21.15.27:60856) 2024-04-25T00:17:30.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.264+0000 7fe13affd640 1 -- 172.21.15.27:0/1737817921 learned_addr learned my addr 172.21.15.27:0/1737817921 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:30.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.265+0000 7fe13b7fe640 1 --2- 172.21.15.27:0/1737817921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01ea580 0x5561d01ea960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.265+0000 7fe123fff640 1 -- 172.21.15.27:0/2502802388 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe12c018020 con 0x7fe13c10ff90 2024-04-25T00:17:30.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.266+0000 7fe13affd640 1 -- 172.21.15.27:0/1737817921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01f5570 msgr2=0x5561d01f7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.266+0000 7fe13affd640 1 --2- 172.21.15.27:0/1737817921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01f5570 0x5561d01f7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.266+0000 7fe13affd640 1 -- 172.21.15.27:0/1737817921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01ea580 msgr2=0x5561d01ea960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.266+0000 7fe13affd640 1 --2- 172.21.15.27:0/1737817921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01ea580 0x5561d01ea960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.266+0000 7fe13affd640 1 -- 172.21.15.27:0/1737817921 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561d011b400 con 0x5561d01eaea0 2024-04-25T00:17:30.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.266+0000 7fe123fff640 1 --2- 172.21.15.27:0/2502802388 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fe1240a2f00 0x7fe1240a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.266+0000 7fe123fff640 1 -- 172.21.15.27:0/2502802388 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fe12c0ceda0 con 0x7fe13c10ff90 2024-04-25T00:17:30.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.267+0000 7fe13b7fe640 1 --2- 172.21.15.27:0/2502802388 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fe1240a2f00 0x7fe1240a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.268+0000 7fe13affd640 1 --2- 172.21.15.27:0/1737817921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01eaea0 0x5561d01f5030 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe130009a50 tx=0x7fe1300040f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3bbae719e889f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.268+0000 7fe13b7fe640 1 --2- 172.21.15.27:0/2502802388 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fe1240a2f00 0x7fe1240a53c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe134007f70 tx=0x7fe134007740 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.268+0000 7fe1217fa640 1 -- 172.21.15.27:0/1737817921 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13003f070 con 0x5561d01eaea0 2024-04-25T00:17:30.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.269+0000 7fe1217fa640 1 -- 172.21.15.27:0/1737817921 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe130005520 con 0x5561d01eaea0 2024-04-25T00:17:30.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.269+0000 7fe1217fa640 1 -- 172.21.15.27:0/1737817921 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe130035030 con 0x5561d01eaea0 2024-04-25T00:17:30.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.270+0000 7fe144165740 1 -- 172.21.15.27:0/1737817921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01eaea0 msgr2=0x5561d01f5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.270+0000 7fe144165740 1 --2- 172.21.15.27:0/1737817921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01eaea0 0x5561d01f5030 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe130009a50 tx=0x7fe1300040f0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.270+0000 7fe144165740 1 -- 172.21.15.27:0/1737817921 shutdown_connections 2024-04-25T00:17:30.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.270+0000 7fe144165740 1 --2- 172.21.15.27:0/1737817921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01f5570 0x5561d01f7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.270+0000 7fe144165740 1 --2- 172.21.15.27:0/1737817921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01eaea0 0x5561d01f5030 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.270+0000 7fe144165740 1 --2- 172.21.15.27:0/1737817921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01ea580 0x5561d01ea960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.270+0000 7fe144165740 1 -- 172.21.15.27:0/1737817921 >> 172.21.15.27:0/1737817921 conn(0x5561d01e68a0 msgr2=0x5561d01e6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:30.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.270+0000 7fe144165740 1 -- 172.21.15.27:0/1737817921 shutdown_connections 2024-04-25T00:17:30.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.270+0000 7fe144165740 1 -- 172.21.15.27:0/1737817921 wait complete. 2024-04-25T00:17:30.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.271+0000 7fe144165740 1 -- start start 2024-04-25T00:17:30.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.271+0000 7fe144165740 1 Processor -- start 2024-04-25T00:17:30.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.271+0000 7fe144165740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01ea580 0x5561d01f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.271+0000 7fe144165740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01eaea0 0x5561d01f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.271+0000 7fe144165740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01f5570 0x5561d01f2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.271+0000 7fe144165740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5561d01f8350 con 0x5561d01ea580 2024-04-25T00:17:30.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.271+0000 7fe144165740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5561d010b050 con 0x5561d01f5570 2024-04-25T00:17:30.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.271+0000 7fe144165740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5561d01f84d0 con 0x5561d01eaea0 2024-04-25T00:17:30.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.271+0000 7fe144165740 10 client.0 _ll_register_callbacks cb 0x5561cfe62be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:17:30.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.271+0000 7fe13affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01ea580 0x5561d01f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.271+0000 7fe13affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01ea580 0x5561d01f4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59758/0 (socket says 172.21.15.27:59758) 2024-04-25T00:17:30.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.271+0000 7fe13affd640 1 -- 172.21.15.27:0/1673808822 learned_addr learned my addr 172.21.15.27:0/1673808822 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:30.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.272+0000 7fe13b7fe640 1 --2- 172.21.15.27:0/1673808822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01f5570 0x5561d01f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.272+0000 7fe13affd640 1 -- 172.21.15.27:0/1673808822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01eaea0 msgr2=0x5561d01f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.272+0000 7fe13affd640 1 --2- 172.21.15.27:0/1673808822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01eaea0 0x5561d01f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.272+0000 7fe13affd640 1 -- 172.21.15.27:0/1673808822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01f5570 msgr2=0x5561d01f2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.272+0000 7fe13affd640 1 --2- 172.21.15.27:0/1673808822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01f5570 0x5561d01f2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.272+0000 7fe13affd640 1 -- 172.21.15.27:0/1673808822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561d00dcdc0 con 0x5561d01ea580 2024-04-25T00:17:30.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.272+0000 7fe13affd640 1 --2- 172.21.15.27:0/1673808822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01ea580 0x5561d01f4f00 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe130009930 tx=0x7fe130009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b391a93d71c2ad4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.272+0000 7fe11eff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:17:30.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.273+0000 7fe11eff5640 1 -- 172.21.15.27:0/1673808822 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13004c070 con 0x5561d01ea580 2024-04-25T00:17:30.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.273+0000 7fe144165740 1 -- 172.21.15.27:0/1673808822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe130005520 con 0x5561d01ea580 2024-04-25T00:17:30.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.273+0000 7fe11eff5640 1 -- 172.21.15.27:0/1673808822 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fe1300046c0 con 0x5561d01ea580 2024-04-25T00:17:30.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.273+0000 7fe11eff5640 1 -- 172.21.15.27:0/1673808822 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe13003f070 con 0x5561d01ea580 2024-04-25T00:17:30.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.273+0000 7fe144165740 10 client.16170 fetch_fsmap learned FSMap version 30 2024-04-25T00:17:30.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.273+0000 7fe144165740 1 -- 172.21.15.27:0/1673808822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5561cfe633f0 con 0x5561d01ea580 2024-04-25T00:17:30.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.273+0000 7fe11eff5640 1 -- 172.21.15.27:0/1673808822 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe130005520 con 0x5561d01ea580 2024-04-25T00:17:30.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.274+0000 7fe11eff5640 1 -- 172.21.15.27:0/1673808822 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fe1300035f0 con 0x5561d01ea580 2024-04-25T00:17:30.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.274+0000 7fe144165740 10 client.16170 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:17:30.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.274+0000 7fe144165740 10 client.16170 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:17:30.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.274+0000 7fe144165740 20 client.16170 populate_metadata read hostname 'smithi027' 2024-04-25T00:17:30.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.274+0000 7fe144165740 1 --2- 172.21.15.27:0/1673808822 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5561d02054b0 0x5561d02078a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.274+0000 7fe144165740 4 client.16170 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:17:30.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.274+0000 7fe144165740 1 -- 172.21.15.27:0/1673808822 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561d015cbb0 con 0x5561d02054b0 2024-04-25T00:17:30.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.274+0000 7fe13a7fc640 1 --2- 172.21.15.27:0/1673808822 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5561d02054b0 0x5561d02078a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.275+0000 7fe13a7fc640 1 --2- 172.21.15.27:0/1673808822 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5561d02054b0 0x5561d02078a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bcd989c9e86807f server_cookie=d6cd4530a1c13e57 in_seq=0 out_seq=0 2024-04-25T00:17:30.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.275+0000 7fe11eff5640 10 client.16170 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:17:30.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.276+0000 7fe11eff5640 1 -- 172.21.15.27:0/1673808822 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5561d015cbb0 con 0x5561d02054b0 2024-04-25T00:17:30.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.276+0000 7fe11eff5640 10 client.16170 handle_command_reply: tid=0 2024-04-25T00:17:30.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.276+0000 7fe144165740 1 client.16170 shutdown 2024-04-25T00:17:30.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.276+0000 7fe144165740 1 -- 172.21.15.27:0/1673808822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01ea580 msgr2=0x5561d01f4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.276+0000 7fe144165740 1 --2- 172.21.15.27:0/1673808822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01ea580 0x5561d01f4f00 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe130009930 tx=0x7fe130009520 comp rx=0 tx=0).stop 2024-04-25T00:17:30.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.276+0000 7fe144165740 1 -- 172.21.15.27:0/1673808822 shutdown_connections 2024-04-25T00:17:30.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.276+0000 7fe144165740 1 --2- 172.21.15.27:0/1673808822 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5561d02054b0 0x5561d02078a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.276+0000 7fe144165740 1 --2- 172.21.15.27:0/1673808822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01f5570 0x5561d01f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.276+0000 7fe144165740 1 --2- 172.21.15.27:0/1673808822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01eaea0 0x5561d01f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.276+0000 7fe144165740 1 --2- 172.21.15.27:0/1673808822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01ea580 0x5561d01f4f00 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.276+0000 7fe11eff5640 0 client.16170 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:17:30.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.276+0000 7fe144165740 1 -- 172.21.15.27:0/1673808822 >> 172.21.15.27:0/1673808822 conn(0x5561d01e68a0 msgr2=0x5561d01e5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:30.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.277+0000 7fe144165740 1 -- 172.21.15.27:0/1673808822 shutdown_connections 2024-04-25T00:17:30.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.277+0000 7fe144165740 1 -- 172.21.15.27:0/1673808822 wait complete. 2024-04-25T00:17:30.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.277+0000 7fe144165740 20 client.16170 trim_cache size 0 max 16384 2024-04-25T00:17:30.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.297+0000 7fe144165740 1 Processor -- start 2024-04-25T00:17:30.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.297+0000 7fe144165740 1 -- start start 2024-04-25T00:17:30.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.297+0000 7fe144165740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01eaea0 0x5561d01e59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.297+0000 7fe144165740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01f5570 0x5561d01f4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.297+0000 7fe144165740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01e1ea0 0x5561d01ef470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.298+0000 7fe144165740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c1bd950 con 0x5561d01eaea0 2024-04-25T00:17:30.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.298+0000 7fe144165740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c1bf550 con 0x5561d01f5570 2024-04-25T00:17:30.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.298+0000 7fe13affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01eaea0 0x5561d01e59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.298+0000 7fe13affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01eaea0 0x5561d01e59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59766/0 (socket says 172.21.15.27:59766) 2024-04-25T00:17:30.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.298+0000 7fe13affd640 1 -- 172.21.15.27:0/1469191915 learned_addr learned my addr 172.21.15.27:0/1469191915 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:30.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.298+0000 7fe13a7fc640 1 --2- 172.21.15.27:0/1469191915 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01f5570 0x5561d01f4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.298+0000 7fe144165740 1 -- 172.21.15.27:0/1469191915 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c1be020 con 0x5561d01e1ea0 2024-04-25T00:17:30.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.298+0000 7fe13b7fe640 1 --2- 172.21.15.27:0/1469191915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01e1ea0 0x5561d01ef470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.298+0000 7fe13affd640 1 -- 172.21.15.27:0/1469191915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01e1ea0 msgr2=0x5561d01ef470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.298+0000 7fe13affd640 1 --2- 172.21.15.27:0/1469191915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01e1ea0 0x5561d01ef470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.298+0000 7fe13affd640 1 -- 172.21.15.27:0/1469191915 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01f5570 msgr2=0x5561d01f4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.298+0000 7fe13affd640 1 --2- 172.21.15.27:0/1469191915 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01f5570 0x5561d01f4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.298+0000 7fe13affd640 1 -- 172.21.15.27:0/1469191915 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561d01eacd0 con 0x5561d01eaea0 2024-04-25T00:17:30.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.298+0000 7fe13a7fc640 1 --2- 172.21.15.27:0/1469191915 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01f5570 0x5561d01f4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:17:30.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.298+0000 7fe13affd640 1 --2- 172.21.15.27:0/1469191915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01eaea0 0x5561d01e59c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe13002aa30 tx=0x7fe130005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb55018de696622 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.299+0000 7fe120ff9640 1 -- 172.21.15.27:0/1469191915 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13004c070 con 0x5561d01eaea0 2024-04-25T00:17:30.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.299+0000 7fe120ff9640 1 -- 172.21.15.27:0/1469191915 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1300035f0 con 0x5561d01eaea0 2024-04-25T00:17:30.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.299+0000 7fe120ff9640 1 -- 172.21.15.27:0/1469191915 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe130035040 con 0x5561d01eaea0 2024-04-25T00:17:30.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.300+0000 7fe144165740 1 -- 172.21.15.27:0/1469191915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01eaea0 msgr2=0x5561d01e59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.300+0000 7fe144165740 1 --2- 172.21.15.27:0/1469191915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01eaea0 0x5561d01e59c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe13002aa30 tx=0x7fe130005fd0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.300+0000 7fe144165740 1 -- 172.21.15.27:0/1469191915 shutdown_connections 2024-04-25T00:17:30.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.300+0000 7fe144165740 1 --2- 172.21.15.27:0/1469191915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01e1ea0 0x5561d01ef470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.300+0000 7fe144165740 1 --2- 172.21.15.27:0/1469191915 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d01f5570 0x5561d01f4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.300+0000 7fe144165740 1 --2- 172.21.15.27:0/1469191915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01eaea0 0x5561d01e59c0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.300+0000 7fe144165740 1 -- 172.21.15.27:0/1469191915 >> 172.21.15.27:0/1469191915 conn(0x5561d01e68a0 msgr2=0x5561d01e52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:30.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.300+0000 7fe144165740 1 -- 172.21.15.27:0/1469191915 shutdown_connections 2024-04-25T00:17:30.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.300+0000 7fe144165740 1 -- 172.21.15.27:0/1469191915 wait complete. 2024-04-25T00:17:30.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.301+0000 7fe144165740 1 -- start start 2024-04-25T00:17:30.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.301+0000 7fe144165740 1 Processor -- start 2024-04-25T00:17:30.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.301+0000 7fe144165740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01eaea0 0x5561d0208570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.301+0000 7fe144165740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01f5570 0x5561d0208ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.301+0000 7fe144165740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d0208ff0 0x5561d026a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.301+0000 7fe144165740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c1bedc0 con 0x5561d01eaea0 2024-04-25T00:17:30.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.301+0000 7fe144165740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c1be6f0 con 0x5561d0208ff0 2024-04-25T00:17:30.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.301+0000 7fe144165740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c1bb910 con 0x5561d01f5570 2024-04-25T00:17:30.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.301+0000 7fe144165740 10 client.0 _ll_register_callbacks cb 0x7fe1340075e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:17:30.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.301+0000 7fe13affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01eaea0 0x5561d0208570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.301+0000 7fe13affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01eaea0 0x5561d0208570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59772/0 (socket says 172.21.15.27:59772) 2024-04-25T00:17:30.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.301+0000 7fe13affd640 1 -- 172.21.15.27:0/1168891560 learned_addr learned my addr 172.21.15.27:0/1168891560 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:30.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.301+0000 7fe13b7fe640 1 --2- 172.21.15.27:0/1168891560 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d0208ff0 0x5561d026a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.301+0000 7fe13a7fc640 1 --2- 172.21.15.27:0/1168891560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01f5570 0x5561d0208ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.302+0000 7fe13affd640 1 -- 172.21.15.27:0/1168891560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01f5570 msgr2=0x5561d0208ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.302+0000 7fe13affd640 1 --2- 172.21.15.27:0/1168891560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01f5570 0x5561d0208ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.302+0000 7fe13affd640 1 -- 172.21.15.27:0/1168891560 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d0208ff0 msgr2=0x5561d026a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.302+0000 7fe13affd640 1 --2- 172.21.15.27:0/1168891560 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d0208ff0 0x5561d026a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.302+0000 7fe13affd640 1 -- 172.21.15.27:0/1168891560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561cff2d7d0 con 0x5561d01eaea0 2024-04-25T00:17:30.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.302+0000 7fe13b7fe640 1 --2- 172.21.15.27:0/1168891560 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d0208ff0 0x5561d026a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:17:30.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.302+0000 7fe13a7fc640 1 --2- 172.21.15.27:0/1168891560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01f5570 0x5561d0208ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:17:30.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.302+0000 7fe13affd640 1 --2- 172.21.15.27:0/1168891560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01eaea0 0x5561d0208570 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe130004100 tx=0x7fe130005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef0f1541d240b26e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.302+0000 7fe11eff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:17:30.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.302+0000 7fe11eff5640 1 -- 172.21.15.27:0/1168891560 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe130035030 con 0x5561d01eaea0 2024-04-25T00:17:30.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.302+0000 7fe11eff5640 1 -- 172.21.15.27:0/1168891560 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fe130005410 con 0x5561d01eaea0 2024-04-25T00:17:30.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.302+0000 7fe144165740 1 -- 172.21.15.27:0/1168891560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1300035f0 con 0x5561d01eaea0 2024-04-25T00:17:30.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.302+0000 7fe11eff5640 1 -- 172.21.15.27:0/1168891560 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe13006a050 con 0x5561d01eaea0 2024-04-25T00:17:30.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.302+0000 7fe144165740 10 client.16176 fetch_fsmap learned FSMap version 30 2024-04-25T00:17:30.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.302+0000 7fe144165740 1 -- 172.21.15.27:0/1168891560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5561d026bdd0 con 0x5561d01eaea0 2024-04-25T00:17:30.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.302+0000 7fe11eff5640 1 -- 172.21.15.27:0/1168891560 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1300035f0 con 0x5561d01eaea0 2024-04-25T00:17:30.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.303+0000 7fe11eff5640 1 -- 172.21.15.27:0/1168891560 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fe13003b8f0 con 0x5561d01eaea0 2024-04-25T00:17:30.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.303+0000 7fe144165740 10 client.16176 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:17:30.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.303+0000 7fe144165740 10 client.16176 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:17:30.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.303+0000 7fe144165740 20 client.16176 populate_metadata read hostname 'smithi027' 2024-04-25T00:17:30.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.303+0000 7fe144165740 1 --2- 172.21.15.27:0/1168891560 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5561d026c2a0 0x5561d026e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.303+0000 7fe144165740 4 client.16176 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:17:30.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.303+0000 7fe144165740 1 -- 172.21.15.27:0/1168891560 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5561d0156f00 con 0x5561d026c2a0 2024-04-25T00:17:30.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.303+0000 7fe13a7fc640 1 --2- 172.21.15.27:0/1168891560 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5561d026c2a0 0x5561d026e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.304+0000 7fe13a7fc640 1 --2- 172.21.15.27:0/1168891560 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5561d026c2a0 0x5561d026e690 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=43b49f09ff9962f0 server_cookie=2a0c4bca72573677 in_seq=0 out_seq=0 2024-04-25T00:17:30.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.304+0000 7fe11eff5640 10 client.16176 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:17:30.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.304+0000 7fe11eff5640 1 -- 172.21.15.27:0/1168891560 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5561d0156f00 con 0x5561d026c2a0 2024-04-25T00:17:30.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.304+0000 7fe11eff5640 10 client.16176 handle_command_reply: tid=0 2024-04-25T00:17:30.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.304+0000 7fe144165740 1 client.16176 shutdown 2024-04-25T00:17:30.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.304+0000 7fe144165740 1 -- 172.21.15.27:0/1168891560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01eaea0 msgr2=0x5561d0208570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.304+0000 7fe144165740 1 --2- 172.21.15.27:0/1168891560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01eaea0 0x5561d0208570 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe130004100 tx=0x7fe130005bc0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.304+0000 7fe144165740 1 -- 172.21.15.27:0/1168891560 shutdown_connections 2024-04-25T00:17:30.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.304+0000 7fe144165740 1 --2- 172.21.15.27:0/1168891560 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x5561d026c2a0 0x5561d026e690 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.304+0000 7fe144165740 1 --2- 172.21.15.27:0/1168891560 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5561d0208ff0 0x5561d026a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.304+0000 7fe144165740 1 --2- 172.21.15.27:0/1168891560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561d01f5570 0x5561d0208ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.304+0000 7fe11eff5640 0 client.16176 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:17:30.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.304+0000 7fe144165740 1 --2- 172.21.15.27:0/1168891560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561d01eaea0 0x5561d0208570 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.304+0000 7fe144165740 1 -- 172.21.15.27:0/1168891560 >> 172.21.15.27:0/1168891560 conn(0x5561d01e68a0 msgr2=0x5561d01e5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:30.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.304+0000 7fe144165740 1 -- 172.21.15.27:0/1168891560 shutdown_connections 2024-04-25T00:17:30.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.304+0000 7fe144165740 1 -- 172.21.15.27:0/1168891560 wait complete. 2024-04-25T00:17:30.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.304+0000 7fe144165740 20 client.16176 trim_cache size 0 max 16384 2024-04-25T00:17:30.310 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:17:30.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.305+0000 7fe141458640 1 -- 172.21.15.27:0/2502802388 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fe1240a2f00 msgr2=0x7fe1240a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.305+0000 7fe141458640 1 --2- 172.21.15.27:0/2502802388 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fe1240a2f00 0x7fe1240a53c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe134007f70 tx=0x7fe134007740 comp rx=0 tx=0).stop 2024-04-25T00:17:30.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.305+0000 7fe141458640 1 -- 172.21.15.27:0/2502802388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe13c10ff90 msgr2=0x7fe13c1a74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.305+0000 7fe141458640 1 --2- 172.21.15.27:0/2502802388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe13c10ff90 0x7fe13c1a74c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009fd0 tx=0x7fe12c00b870 comp rx=0 tx=0).stop 2024-04-25T00:17:30.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.305+0000 7fe141458640 1 -- 172.21.15.27:0/2502802388 shutdown_connections 2024-04-25T00:17:30.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.305+0000 7fe141458640 1 --2- 172.21.15.27:0/2502802388 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fe1240a2f00 0x7fe1240a53c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.305+0000 7fe141458640 1 --2- 172.21.15.27:0/2502802388 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe13c11af20 0x7fe13c1ab840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.305+0000 7fe141458640 1 --2- 172.21.15.27:0/2502802388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe13c10ff90 0x7fe13c1a74c0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.305+0000 7fe141458640 1 --2- 172.21.15.27:0/2502802388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe13c10dfc0 0x7fe13c1a6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.305+0000 7fe141458640 1 -- 172.21.15.27:0/2502802388 >> 172.21.15.27:0/2502802388 conn(0x7fe13c06c850 msgr2=0x7fe13c070230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:30.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.305+0000 7fe141458640 1 -- 172.21.15.27:0/2502802388 shutdown_connections 2024-04-25T00:17:30.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.305+0000 7fe141458640 1 -- 172.21.15.27:0/2502802388 wait complete. 2024-04-25T00:17:30.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T00:17:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.536+0000 7fdc498f5640 1 -- 172.21.15.27:0/1716896710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc44101710 msgr2=0x7fdc44101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.536+0000 7fdc498f5640 1 --2- 172.21.15.27:0/1716896710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc44101710 0x7fdc44101b90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009890 tx=0x7fdc3802a380 comp rx=0 tx=0).stop 2024-04-25T00:17:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.536+0000 7fdc498f5640 1 -- 172.21.15.27:0/1716896710 shutdown_connections 2024-04-25T00:17:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.536+0000 7fdc498f5640 1 --2- 172.21.15.27:0/1716896710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc441020d0 0x7fdc4410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.536+0000 7fdc498f5640 1 --2- 172.21.15.27:0/1716896710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc44101710 0x7fdc44101b90 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.536+0000 7fdc498f5640 1 --2- 172.21.15.27:0/1716896710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc44107720 0x7fdc44107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.536+0000 7fdc498f5640 1 -- 172.21.15.27:0/1716896710 >> 172.21.15.27:0/1716896710 conn(0x7fdc44078b90 msgr2=0x7fdc44078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.536+0000 7fdc498f5640 1 -- 172.21.15.27:0/1716896710 shutdown_connections 2024-04-25T00:17:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.536+0000 7fdc498f5640 1 -- 172.21.15.27:0/1716896710 wait complete. 2024-04-25T00:17:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.537+0000 7fdc498f5640 1 Processor -- start 2024-04-25T00:17:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.537+0000 7fdc498f5640 1 -- start start 2024-04-25T00:17:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.537+0000 7fdc498f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc44101710 0x7fdc441980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.537+0000 7fdc498f5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc441020d0 0x7fdc44198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.537+0000 7fdc498f5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc44107720 0x7fdc4419c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.538+0000 7fdc498f5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdc44111b30 con 0x7fdc441020d0 2024-04-25T00:17:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.538+0000 7fdc498f5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdc441119b0 con 0x7fdc44107720 2024-04-25T00:17:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.538+0000 7fdc498f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdc44111cb0 con 0x7fdc44101710 2024-04-25T00:17:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.538+0000 7fdc427fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc441020d0 0x7fdc44198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.538+0000 7fdc427fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc441020d0 0x7fdc44198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59798/0 (socket says 172.21.15.27:59798) 2024-04-25T00:17:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.538+0000 7fdc427fc640 1 -- 172.21.15.27:0/533225867 learned_addr learned my addr 172.21.15.27:0/533225867 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.538+0000 7fdc42ffd640 1 --2- 172.21.15.27:0/533225867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc44101710 0x7fdc441980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.538+0000 7fdc427fc640 1 -- 172.21.15.27:0/533225867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc44101710 msgr2=0x7fdc441980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.538+0000 7fdc427fc640 1 --2- 172.21.15.27:0/533225867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc44101710 0x7fdc441980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.538+0000 7fdc427fc640 1 -- 172.21.15.27:0/533225867 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc44107720 msgr2=0x7fdc4419c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.538+0000 7fdc427fc640 1 --2- 172.21.15.27:0/533225867 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc44107720 0x7fdc4419c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.538+0000 7fdc427fc640 1 -- 172.21.15.27:0/533225867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc4419d080 con 0x7fdc441020d0 2024-04-25T00:17:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.538+0000 7fdc427fc640 1 --2- 172.21.15.27:0/533225867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc441020d0 0x7fdc44198620 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009860 tx=0x7fdc38033d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.538+0000 7fdc488f3640 1 -- 172.21.15.27:0/533225867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3803f020 con 0x7fdc441020d0 2024-04-25T00:17:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.539+0000 7fdc488f3640 1 -- 172.21.15.27:0/533225867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc38032770 con 0x7fdc441020d0 2024-04-25T00:17:30.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.539+0000 7fdc498f5640 1 -- 172.21.15.27:0/533225867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc4419d310 con 0x7fdc441020d0 2024-04-25T00:17:30.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.539+0000 7fdc488f3640 1 -- 172.21.15.27:0/533225867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc38037040 con 0x7fdc441020d0 2024-04-25T00:17:30.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.539+0000 7fdc498f5640 1 -- 172.21.15.27:0/533225867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc441a4c00 con 0x7fdc441020d0 2024-04-25T00:17:30.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.542+0000 7fdc488f3640 1 -- 172.21.15.27:0/533225867 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fdc38042020 con 0x7fdc441020d0 2024-04-25T00:17:30.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.543+0000 7fdc4c5fe740 1 Processor -- start 2024-04-25T00:17:30.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.543+0000 7fdc488f3640 1 --2- 172.21.15.27:0/533225867 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fdc2c0a2d80 0x7fdc2c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.543+0000 7fdc488f3640 1 -- 172.21.15.27:0/533225867 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fdc380ef6f0 con 0x7fdc441020d0 2024-04-25T00:17:30.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.544+0000 7fdc437fe640 1 --2- 172.21.15.27:0/533225867 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fdc2c0a2d80 0x7fdc2c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.544+0000 7fdc4c5fe740 1 -- start start 2024-04-25T00:17:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.544+0000 7fdc4c5fe740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264b82b0 0x55d3264b8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.544+0000 7fdc4c5fe740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b8c60 0x55d3264c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.544+0000 7fdc4c5fe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264c7980 0x55d3264c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.544+0000 7fdc4c5fe740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d32641d670 con 0x55d3264b8c60 2024-04-25T00:17:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.544+0000 7fdc4c5fe740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d3263c5970 con 0x55d3264b82b0 2024-04-25T00:17:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.544+0000 7fdc4c5fe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d326419630 con 0x55d3264c7980 2024-04-25T00:17:30.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.545+0000 7fdc427fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b8c60 0x55d3264c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.545+0000 7fdc427fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b8c60 0x55d3264c7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59800/0 (socket says 172.21.15.27:59800) 2024-04-25T00:17:30.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.545+0000 7fdc427fc640 1 -- 172.21.15.27:0/4061865257 learned_addr learned my addr 172.21.15.27:0/4061865257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.545+0000 7fdc42ffd640 1 --2- 172.21.15.27:0/4061865257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264b82b0 0x55d3264b8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.545+0000 7fdc437fe640 1 --2- 172.21.15.27:0/4061865257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264c7980 0x55d3264c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.545+0000 7fdc437fe640 1 --2- 172.21.15.27:0/533225867 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fdc2c0a2d80 0x7fdc2c0a5240 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c005c40 tx=0x7fdc3c005bd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.545+0000 7fdc427fc640 1 -- 172.21.15.27:0/4061865257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264c7980 msgr2=0x55d3264c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.545+0000 7fdc427fc640 1 --2- 172.21.15.27:0/4061865257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264c7980 0x55d3264c9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.545+0000 7fdc427fc640 1 -- 172.21.15.27:0/4061865257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264b82b0 msgr2=0x55d3264b8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.545+0000 7fdc427fc640 1 --2- 172.21.15.27:0/4061865257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264b82b0 0x55d3264b8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.545+0000 7fdc427fc640 1 -- 172.21.15.27:0/4061865257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3263e9400 con 0x55d3264b8c60 2024-04-25T00:17:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.546+0000 7fdc427fc640 1 --2- 172.21.15.27:0/4061865257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b8c60 0x55d3264c7440 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fdc38035040 tx=0x7fdc380bb4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca0cb08ae62c838 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.546+0000 7fdc29ffb640 1 -- 172.21.15.27:0/4061865257 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc380bc0b0 con 0x55d3264b8c60 2024-04-25T00:17:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.546+0000 7fdc29ffb640 1 -- 172.21.15.27:0/4061865257 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc380bb9b0 con 0x55d3264b8c60 2024-04-25T00:17:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.546+0000 7fdc29ffb640 1 -- 172.21.15.27:0/4061865257 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc38004240 con 0x55d3264b8c60 2024-04-25T00:17:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.547+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/4061865257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b8c60 msgr2=0x55d3264c7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.547+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/4061865257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b8c60 0x55d3264c7440 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fdc38035040 tx=0x7fdc380bb4f0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.547+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/4061865257 shutdown_connections 2024-04-25T00:17:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.547+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/4061865257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264c7980 0x55d3264c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.547+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/4061865257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b8c60 0x55d3264c7440 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.547+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/4061865257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264b82b0 0x55d3264b8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.547+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/4061865257 >> 172.21.15.27:0/4061865257 conn(0x55d3264b48a0 msgr2=0x55d3264b4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.548+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/4061865257 shutdown_connections 2024-04-25T00:17:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.548+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/4061865257 wait complete. 2024-04-25T00:17:30.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.548+0000 7fdc4c5fe740 1 -- start start 2024-04-25T00:17:30.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.549+0000 7fdc4c5fe740 1 Processor -- start 2024-04-25T00:17:30.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.549+0000 7fdc4c5fe740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b82b0 0x55d3264c7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.549+0000 7fdc4c5fe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264b8c60 0x55d3264c4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.549+0000 7fdc4c5fe740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264c7980 0x55d3264c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.549+0000 7fdc4c5fe740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d3264ca500 con 0x55d3264b82b0 2024-04-25T00:17:30.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.549+0000 7fdc42ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b82b0 0x55d3264c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.549+0000 7fdc42ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b82b0 0x55d3264c7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59802/0 (socket says 172.21.15.27:59802) 2024-04-25T00:17:30.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.549+0000 7fdc42ffd640 1 -- 172.21.15.27:0/2064763611 learned_addr learned my addr 172.21.15.27:0/2064763611 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:30.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.550+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/2064763611 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d3263d9050 con 0x55d3264c7980 2024-04-25T00:17:30.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.550+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/2064763611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d3264ca680 con 0x55d3264b8c60 2024-04-25T00:17:30.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.550+0000 7fdc427fc640 1 --2- 172.21.15.27:0/2064763611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264b8c60 0x55d3264c4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.550+0000 7fdc437fe640 1 --2- 172.21.15.27:0/2064763611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264c7980 0x55d3264c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.550+0000 7fdc4c5fe740 10 client.0 _ll_register_callbacks cb 0x55d326130be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:17:30.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.550+0000 7fdc42ffd640 1 -- 172.21.15.27:0/2064763611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264b8c60 msgr2=0x55d3264c4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.550+0000 7fdc42ffd640 1 --2- 172.21.15.27:0/2064763611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264b8c60 0x55d3264c4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.550+0000 7fdc42ffd640 1 -- 172.21.15.27:0/2064763611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264c7980 msgr2=0x55d3264c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.550+0000 7fdc42ffd640 1 --2- 172.21.15.27:0/2064763611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264c7980 0x55d3264c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.550+0000 7fdc42ffd640 1 -- 172.21.15.27:0/2064763611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3263aadc0 con 0x55d3264b82b0 2024-04-25T00:17:30.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.550+0000 7fdc42ffd640 1 --2- 172.21.15.27:0/2064763611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b82b0 0x55d3264c7390 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fdc34012720 tx=0x7fdc34012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce791175a2f5698 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.550+0000 7fdc22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:17:30.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.550+0000 7fdc22ffd640 1 -- 172.21.15.27:0/2064763611 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3400f920 con 0x55d3264b82b0 2024-04-25T00:17:30.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.550+0000 7fdc22ffd640 1 -- 172.21.15.27:0/2064763611 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fdc3400fac0 con 0x55d3264b82b0 2024-04-25T00:17:30.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.550+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/2064763611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc380bb9b0 con 0x55d3264b82b0 2024-04-25T00:17:30.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.551+0000 7fdc22ffd640 1 -- 172.21.15.27:0/2064763611 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdc34025050 con 0x55d3264b82b0 2024-04-25T00:17:30.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.551+0000 7fdc4c5fe740 10 client.16188 fetch_fsmap learned FSMap version 30 2024-04-25T00:17:30.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.551+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/2064763611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d3261313f0 con 0x55d3264b82b0 2024-04-25T00:17:30.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.551+0000 7fdc22ffd640 1 -- 172.21.15.27:0/2064763611 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc3400fcc0 con 0x55d3264b82b0 2024-04-25T00:17:30.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.551+0000 7fdc22ffd640 1 -- 172.21.15.27:0/2064763611 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fdc340053e0 con 0x55d3264b82b0 2024-04-25T00:17:30.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.551+0000 7fdc4c5fe740 10 client.16188 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:17:30.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.551+0000 7fdc4c5fe740 10 client.16188 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:17:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.551+0000 7fdc4c5fe740 20 client.16188 populate_metadata read hostname 'smithi027' 2024-04-25T00:17:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.551+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/2064763611 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55d3264d77c0 0x55d3264d9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.551+0000 7fdc4c5fe740 4 client.16188 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:17:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.551+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/2064763611 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fdc380bb9b0 con 0x55d3264d77c0 2024-04-25T00:17:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.551+0000 7fdc427fc640 1 --2- 172.21.15.27:0/2064763611 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55d3264d77c0 0x55d3264d9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.552+0000 7fdc22ffd640 10 client.16188 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:17:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.552+0000 7fdc427fc640 1 --2- 172.21.15.27:0/2064763611 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55d3264d77c0 0x55d3264d9bd0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=909eb1c382d8cce6 server_cookie=efbb5fc4ae2f0457 in_seq=0 out_seq=0 2024-04-25T00:17:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.553+0000 7fdc22ffd640 1 -- 172.21.15.27:0/2064763611 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fdc380bb9b0 con 0x55d3264d77c0 2024-04-25T00:17:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.553+0000 7fdc22ffd640 10 client.16188 handle_command_reply: tid=0 2024-04-25T00:17:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.553+0000 7fdc4c5fe740 1 client.16188 shutdown 2024-04-25T00:17:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.553+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/2064763611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b82b0 msgr2=0x55d3264c7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.553+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/2064763611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b82b0 0x55d3264c7390 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fdc34012720 tx=0x7fdc34012be0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.553+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/2064763611 shutdown_connections 2024-04-25T00:17:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.553+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/2064763611 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55d3264d77c0 0x55d3264d9bd0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.553+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/2064763611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264c7980 0x55d3264c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.553+0000 7fdc22ffd640 0 client.16188 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:17:30.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.553+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/2064763611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264b8c60 0x55d3264c4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.553+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/2064763611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b82b0 0x55d3264c7390 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.553+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/2064763611 >> 172.21.15.27:0/2064763611 conn(0x55d3264b48a0 msgr2=0x55d3264ba6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:30.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.554+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/2064763611 shutdown_connections 2024-04-25T00:17:30.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.554+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/2064763611 wait complete. 2024-04-25T00:17:30.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.554+0000 7fdc4c5fe740 20 client.16188 trim_cache size 0 max 16384 2024-04-25T00:17:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.568+0000 7fdc4c5fe740 1 Processor -- start 2024-04-25T00:17:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.568+0000 7fdc4c5fe740 1 -- start start 2024-04-25T00:17:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.568+0000 7fdc4c5fe740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264b82b0 0x55d3264c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.568+0000 7fdc4c5fe740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b8c60 0x55d3264df4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.568+0000 7fdc4c5fe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264c7980 0x55d3264df9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.568+0000 7fdc4c5fe740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdc441aca40 con 0x55d3264b8c60 2024-04-25T00:17:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.568+0000 7fdc4c5fe740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdc441ae640 con 0x55d3264b82b0 2024-04-25T00:17:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.568+0000 7fdc4c5fe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdc441ad110 con 0x55d3264c7980 2024-04-25T00:17:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.568+0000 7fdc427fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b8c60 0x55d3264df4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.569+0000 7fdc427fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b8c60 0x55d3264df4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59810/0 (socket says 172.21.15.27:59810) 2024-04-25T00:17:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.569+0000 7fdc42ffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264b82b0 0x55d3264c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.569+0000 7fdc427fc640 1 -- 172.21.15.27:0/4273883373 learned_addr learned my addr 172.21.15.27:0/4273883373 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.569+0000 7fdc437fe640 1 --2- 172.21.15.27:0/4273883373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264c7980 0x55d3264df9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.569+0000 7fdc427fc640 1 -- 172.21.15.27:0/4273883373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264c7980 msgr2=0x55d3264df9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.569+0000 7fdc427fc640 1 --2- 172.21.15.27:0/4273883373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264c7980 0x55d3264df9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.569+0000 7fdc427fc640 1 -- 172.21.15.27:0/4273883373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264b82b0 msgr2=0x55d3264c6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.569+0000 7fdc427fc640 1 --2- 172.21.15.27:0/4273883373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264b82b0 0x55d3264c6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.569+0000 7fdc427fc640 1 -- 172.21.15.27:0/4273883373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3261fb7d0 con 0x55d3264b8c60 2024-04-25T00:17:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.569+0000 7fdc42ffd640 1 --2- 172.21.15.27:0/4273883373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264b82b0 0x55d3264c6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:17:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.569+0000 7fdc427fc640 1 --2- 172.21.15.27:0/4273883373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b8c60 0x55d3264df4a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fdc380b7e10 tx=0x7fdc380bb620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebdddc9db60a221a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.569+0000 7fdc297fa640 1 -- 172.21.15.27:0/4273883373 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc380bb9b0 con 0x55d3264b8c60 2024-04-25T00:17:30.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.569+0000 7fdc297fa640 1 -- 172.21.15.27:0/4273883373 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc380c4620 con 0x55d3264b8c60 2024-04-25T00:17:30.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.569+0000 7fdc297fa640 1 -- 172.21.15.27:0/4273883373 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc380c4190 con 0x55d3264b8c60 2024-04-25T00:17:30.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.570+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/4273883373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b8c60 msgr2=0x55d3264df4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.570+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/4273883373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b8c60 0x55d3264df4a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fdc380b7e10 tx=0x7fdc380bb620 comp rx=0 tx=0).stop 2024-04-25T00:17:30.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.570+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/4273883373 shutdown_connections 2024-04-25T00:17:30.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.570+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/4273883373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264c7980 0x55d3264df9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.570+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/4273883373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264b8c60 0x55d3264df4a0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.570+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/4273883373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264b82b0 0x55d3264c6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.570+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/4273883373 >> 172.21.15.27:0/4273883373 conn(0x55d3264b48a0 msgr2=0x55d3264bfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:30.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.570+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/4273883373 shutdown_connections 2024-04-25T00:17:30.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.570+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/4273883373 wait complete. 2024-04-25T00:17:30.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.570+0000 7fdc4c5fe740 1 -- start start 2024-04-25T00:17:30.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc4c5fe740 1 Processor -- start 2024-04-25T00:17:30.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc4c5fe740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264b82b0 0x55d3264cb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc4c5fe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264b8c60 0x55d3264cb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc4c5fe740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264c7980 0x55d3264cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc437fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264c7980 0x55d3264cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc42ffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264b82b0 0x55d3264cb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc437fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264c7980 0x55d3264cbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59812/0 (socket says 172.21.15.27:59812) 2024-04-25T00:17:30.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc437fe640 1 -- 172.21.15.27:0/74770656 learned_addr learned my addr 172.21.15.27:0/74770656 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:30.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc427fc640 1 --2- 172.21.15.27:0/74770656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264b8c60 0x55d3264cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/74770656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdc441adeb0 con 0x55d3264c7980 2024-04-25T00:17:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/74770656 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdc441ad7e0 con 0x55d3264b82b0 2024-04-25T00:17:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/74770656 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdc441aaa00 con 0x55d3264b8c60 2024-04-25T00:17:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc4c5fe740 10 client.0 _ll_register_callbacks cb 0x7fdc34009870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:17:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc437fe640 1 -- 172.21.15.27:0/74770656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264b8c60 msgr2=0x55d3264cb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc437fe640 1 --2- 172.21.15.27:0/74770656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264b8c60 0x55d3264cb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc437fe640 1 -- 172.21.15.27:0/74770656 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264b82b0 msgr2=0x55d3264cb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc437fe640 1 --2- 172.21.15.27:0/74770656 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264b82b0 0x55d3264cb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.571+0000 7fdc437fe640 1 -- 172.21.15.27:0/74770656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d32653cd10 con 0x55d3264c7980 2024-04-25T00:17:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.572+0000 7fdc437fe640 1 --2- 172.21.15.27:0/74770656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264c7980 0x55d3264cbcc0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c01c960 tx=0x7fdc3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e3556a3c6c8a71 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.572+0000 7fdc22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:17:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.572+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/74770656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc380c4620 con 0x55d3264c7980 2024-04-25T00:17:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.572+0000 7fdc22ffd640 1 -- 172.21.15.27:0/74770656 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c010cc0 con 0x55d3264c7980 2024-04-25T00:17:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.572+0000 7fdc22ffd640 1 -- 172.21.15.27:0/74770656 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fdc3c030450 con 0x55d3264c7980 2024-04-25T00:17:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.572+0000 7fdc22ffd640 1 -- 172.21.15.27:0/74770656 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdc3c010e60 con 0x55d3264c7980 2024-04-25T00:17:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.572+0000 7fdc4c5fe740 10 client.16194 fetch_fsmap learned FSMap version 30 2024-04-25T00:17:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.572+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/74770656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({fsmap=30}) -- 0x55d32653de70 con 0x55d3264c7980 2024-04-25T00:17:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.572+0000 7fdc22ffd640 1 -- 172.21.15.27:0/74770656 <== mon.0 v2:172.21.15.27:3300/0 4 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fdc3c019040 con 0x55d3264c7980 2024-04-25T00:17:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.572+0000 7fdc4c5fe740 10 client.16194 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:17:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.572+0000 7fdc4c5fe740 10 client.16194 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:17:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.572+0000 7fdc4c5fe740 20 client.16194 populate_metadata read hostname 'smithi027' 2024-04-25T00:17:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.572+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/74770656 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55d3264d77c0 0x55d326540540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.572+0000 7fdc4c5fe740 4 client.16194 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:17:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.572+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/74770656 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fdc380c4620 con 0x55d3264d77c0 2024-04-25T00:17:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.573+0000 7fdc42ffd640 1 --2- 172.21.15.27:0/74770656 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55d3264d77c0 0x55d326540540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.573+0000 7fdc42ffd640 1 --2- 172.21.15.27:0/74770656 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55d3264d77c0 0x55d326540540 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c64f56c06c56f10d server_cookie=2c02b2e31a528f0f in_seq=0 out_seq=0 2024-04-25T00:17:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.573+0000 7fdc22ffd640 10 client.16194 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:17:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.573+0000 7fdc22ffd640 1 -- 172.21.15.27:0/74770656 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fdc380c4620 con 0x55d3264d77c0 2024-04-25T00:17:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.573+0000 7fdc22ffd640 10 client.16194 handle_command_reply: tid=0 2024-04-25T00:17:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.573+0000 7fdc4c5fe740 1 client.16194 shutdown 2024-04-25T00:17:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.573+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/74770656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264c7980 msgr2=0x55d3264cbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.573+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/74770656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264c7980 0x55d3264cbcc0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c01c960 tx=0x7fdc3c004a40 comp rx=0 tx=0).stop 2024-04-25T00:17:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.573+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/74770656 shutdown_connections 2024-04-25T00:17:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.573+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/74770656 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55d3264d77c0 0x55d326540540 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc22ffd640 0 client.16194 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:17:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/74770656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3264c7980 0x55d3264cbcc0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/74770656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3264b8c60 0x55d3264cb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc4c5fe740 1 --2- 172.21.15.27:0/74770656 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3264b82b0 0x55d3264cb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/74770656 >> 172.21.15.27:0/74770656 conn(0x55d3264b48a0 msgr2=0x55d3264b9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/74770656 shutdown_connections 2024-04-25T00:17:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc4c5fe740 1 -- 172.21.15.27:0/74770656 wait complete. 2024-04-25T00:17:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc4c5fe740 20 client.16194 trim_cache size 0 max 16384 2024-04-25T00:17:30.581 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:17:30.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc498f5640 1 -- 172.21.15.27:0/533225867 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fdc2c0a2d80 msgr2=0x7fdc2c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc498f5640 1 --2- 172.21.15.27:0/533225867 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fdc2c0a2d80 0x7fdc2c0a5240 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c005c40 tx=0x7fdc3c005bd0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc498f5640 1 -- 172.21.15.27:0/533225867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc441020d0 msgr2=0x7fdc44198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc498f5640 1 --2- 172.21.15.27:0/533225867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc441020d0 0x7fdc44198620 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009860 tx=0x7fdc38033d10 comp rx=0 tx=0).stop 2024-04-25T00:17:30.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc498f5640 1 -- 172.21.15.27:0/533225867 shutdown_connections 2024-04-25T00:17:30.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc498f5640 1 --2- 172.21.15.27:0/533225867 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fdc2c0a2d80 0x7fdc2c0a5240 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc498f5640 1 --2- 172.21.15.27:0/533225867 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc44107720 0x7fdc4419c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc498f5640 1 --2- 172.21.15.27:0/533225867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc441020d0 0x7fdc44198620 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc498f5640 1 --2- 172.21.15.27:0/533225867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc44101710 0x7fdc441980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.574+0000 7fdc498f5640 1 -- 172.21.15.27:0/533225867 >> 172.21.15.27:0/533225867 conn(0x7fdc44078b90 msgr2=0x7fdc440ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.575+0000 7fdc498f5640 1 -- 172.21.15.27:0/533225867 shutdown_connections 2024-04-25T00:17:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.575+0000 7fdc498f5640 1 -- 172.21.15.27:0/533225867 wait complete. 2024-04-25T00:17:30.590 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T00:17:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.736+0000 7f387bfff640 1 -- 172.21.15.27:0/2466272442 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3874035020 con 0x7f3884100620 2024-04-25T00:17:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.738+0000 7f388b309640 1 -- 172.21.15.27:0/2466272442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3884100620 msgr2=0x7f3884100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.738+0000 7f388b309640 1 --2- 172.21.15.27:0/2466272442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3884100620 0x7f3884100aa0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3874009930 tx=0x7f387402a480 comp rx=0 tx=0).stop 2024-04-25T00:17:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.738+0000 7f388b309640 1 -- 172.21.15.27:0/2466272442 shutdown_connections 2024-04-25T00:17:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.738+0000 7f388b309640 1 --2- 172.21.15.27:0/2466272442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3884100fe0 0x7f388410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.738+0000 7f388b309640 1 --2- 172.21.15.27:0/2466272442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3884100620 0x7f3884100aa0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.738+0000 7f388b309640 1 --2- 172.21.15.27:0/2466272442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3884106630 0x7f3884106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.738+0000 7f388b309640 1 -- 172.21.15.27:0/2466272442 >> 172.21.15.27:0/2466272442 conn(0x7f38840769c0 msgr2=0x7f3884076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.739+0000 7f388b309640 1 -- 172.21.15.27:0/2466272442 shutdown_connections 2024-04-25T00:17:30.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.739+0000 7f388b309640 1 -- 172.21.15.27:0/2466272442 wait complete. 2024-04-25T00:17:30.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.740+0000 7f388b309640 1 Processor -- start 2024-04-25T00:17:30.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.740+0000 7f388b309640 1 -- start start 2024-04-25T00:17:30.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.741+0000 7f388b309640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3884100620 0x7f3884109190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.741+0000 7f388b309640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3884100fe0 0x7f3884107590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.741+0000 7f388b309640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3884106630 0x7f3884107bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.741+0000 7f388b309640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3884110b50 con 0x7f3884106630 2024-04-25T00:17:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.741+0000 7f388b309640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f38841109d0 con 0x7f3884100620 2024-04-25T00:17:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.741+0000 7f388b309640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3884110cd0 con 0x7f3884100fe0 2024-04-25T00:17:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.741+0000 7f38898a6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3884106630 0x7f3884107bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.741+0000 7f38898a6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3884106630 0x7f3884107bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59840/0 (socket says 172.21.15.27:59840) 2024-04-25T00:17:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.741+0000 7f38898a6640 1 -- 172.21.15.27:0/4061590445 learned_addr learned my addr 172.21.15.27:0/4061590445 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.742+0000 7f38898a6640 1 -- 172.21.15.27:0/4061590445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3884100fe0 msgr2=0x7f3884107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.742+0000 7f38898a6640 1 --2- 172.21.15.27:0/4061590445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3884100fe0 0x7f3884107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.742+0000 7f38898a6640 1 -- 172.21.15.27:0/4061590445 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3884100620 msgr2=0x7f3884109190 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:17:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.742+0000 7f38898a6640 1 --2- 172.21.15.27:0/4061590445 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3884100620 0x7f3884109190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.742+0000 7f38898a6640 1 -- 172.21.15.27:0/4061590445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388419c350 con 0x7f3884106630 2024-04-25T00:17:30.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.742+0000 7f38898a6640 1 --2- 172.21.15.27:0/4061590445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3884106630 0x7f3884107bd0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f387c00b990 tx=0x7f387c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.742+0000 7f387a7fc640 1 -- 172.21.15.27:0/4061590445 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c019070 con 0x7f3884106630 2024-04-25T00:17:30.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.742+0000 7f387a7fc640 1 -- 172.21.15.27:0/4061590445 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f387c00cba0 con 0x7f3884106630 2024-04-25T00:17:30.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.744+0000 7f387a7fc640 1 -- 172.21.15.27:0/4061590445 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c007650 con 0x7f3884106630 2024-04-25T00:17:30.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.744+0000 7f388b309640 1 -- 172.21.15.27:0/4061590445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f388419c640 con 0x7f3884106630 2024-04-25T00:17:30.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.744+0000 7f388b309640 1 -- 172.21.15.27:0/4061590445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f388419cb10 con 0x7f3884106630 2024-04-25T00:17:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.749+0000 7f388e00f740 1 Processor -- start 2024-04-25T00:17:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.750+0000 7f388e00f740 1 -- start start 2024-04-25T00:17:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.750+0000 7f387a7fc640 1 -- 172.21.15.27:0/4061590445 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f387c0077f0 con 0x7f3884106630 2024-04-25T00:17:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.750+0000 7f388e00f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7e6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.750+0000 7f388e00f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7e6ea0 0x55f2ec7f1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.750+0000 7f388e00f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7f1570 0x55f2ec7f3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.750+0000 7f38888a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7e6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.750+0000 7f388e00f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f2ec74b670 con 0x55f2ec7e6580 2024-04-25T00:17:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.750+0000 7f388e00f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f2ec6f3970 con 0x55f2ec7e6ea0 2024-04-25T00:17:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.750+0000 7f38888a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7e6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59846/0 (socket says 172.21.15.27:59846) 2024-04-25T00:17:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.750+0000 7f388e00f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f2ec747630 con 0x55f2ec7f1570 2024-04-25T00:17:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.750+0000 7f38888a4640 1 -- 172.21.15.27:0/1806818242 learned_addr learned my addr 172.21.15.27:0/1806818242 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.750+0000 7f38890a5640 1 --2- 172.21.15.27:0/1806818242 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7e6ea0 0x55f2ec7f1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.750+0000 7f38888a4640 1 --2- 172.21.15.27:0/1806818242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7f1570 0x55f2ec7f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.751+0000 7f38890a5640 1 -- 172.21.15.27:0/1806818242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7f1570 msgr2=0x55f2ec7f3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.751+0000 7f38890a5640 1 --2- 172.21.15.27:0/1806818242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7f1570 0x55f2ec7f3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.751+0000 7f38890a5640 1 -- 172.21.15.27:0/1806818242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 msgr2=0x55f2ec7e6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.751+0000 7f38890a5640 1 --2- 172.21.15.27:0/1806818242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7e6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.751+0000 7f38890a5640 1 -- 172.21.15.27:0/1806818242 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2ec717400 con 0x55f2ec7e6ea0 2024-04-25T00:17:30.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.751+0000 7f387a7fc640 1 --2- 172.21.15.27:0/4061590445 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f38700a2ce0 0x7f38700a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.751+0000 7f38888a4640 1 --2- 172.21.15.27:0/1806818242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7f1570 0x55f2ec7f3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:17:30.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.751+0000 7f38888a4640 1 --2- 172.21.15.27:0/4061590445 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f38700a2ce0 0x7f38700a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.752+0000 7f38890a5640 1 --2- 172.21.15.27:0/1806818242 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7e6ea0 0x55f2ec7f1030 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3880009690 tx=0x7f3880009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d22a1e6acdc0f3ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.752+0000 7f387a7fc640 1 -- 172.21.15.27:0/4061590445 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f387c0cec20 con 0x7f3884106630 2024-04-25T00:17:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.752+0000 7f386f7fe640 1 -- 172.21.15.27:0/1806818242 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880015020 con 0x55f2ec7e6ea0 2024-04-25T00:17:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.752+0000 7f386f7fe640 1 -- 172.21.15.27:0/1806818242 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f388000bda0 con 0x55f2ec7e6ea0 2024-04-25T00:17:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.752+0000 7f386f7fe640 1 -- 172.21.15.27:0/1806818242 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880010380 con 0x55f2ec7e6ea0 2024-04-25T00:17:30.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.754+0000 7f388e00f740 1 -- 172.21.15.27:0/1806818242 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7e6ea0 msgr2=0x55f2ec7f1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.754+0000 7f388e00f740 1 --2- 172.21.15.27:0/1806818242 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7e6ea0 0x55f2ec7f1030 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3880009690 tx=0x7f3880009b50 comp rx=0 tx=0).stop 2024-04-25T00:17:30.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.755+0000 7f38888a4640 1 --2- 172.21.15.27:0/4061590445 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f38700a2ce0 0x7f38700a51a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f38740096c0 tx=0x7f3874005b50 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.756+0000 7f388e00f740 1 -- 172.21.15.27:0/1806818242 shutdown_connections 2024-04-25T00:17:30.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.756+0000 7f388e00f740 1 --2- 172.21.15.27:0/1806818242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7f1570 0x55f2ec7f3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.756+0000 7f388e00f740 1 --2- 172.21.15.27:0/1806818242 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7e6ea0 0x55f2ec7f1030 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.756+0000 7f388e00f740 1 --2- 172.21.15.27:0/1806818242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7e6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.756+0000 7f388e00f740 1 -- 172.21.15.27:0/1806818242 >> 172.21.15.27:0/1806818242 conn(0x55f2ec7e28a0 msgr2=0x55f2ec7e2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:30.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.758+0000 7f388e00f740 1 -- 172.21.15.27:0/1806818242 shutdown_connections 2024-04-25T00:17:30.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.758+0000 7f388e00f740 1 -- 172.21.15.27:0/1806818242 wait complete. 2024-04-25T00:17:30.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.759+0000 7f388e00f740 1 -- start start 2024-04-25T00:17:30.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.760+0000 7f388e00f740 1 Processor -- start 2024-04-25T00:17:30.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.761+0000 7f388e00f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7f0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.761+0000 7f388e00f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7e6ea0 0x55f2ec7ee210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.761+0000 7f38890a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7f0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.761+0000 7f38890a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7f0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59854/0 (socket says 172.21.15.27:59854) 2024-04-25T00:17:30.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.762+0000 7f38890a5640 1 -- 172.21.15.27:0/678173215 learned_addr learned my addr 172.21.15.27:0/678173215 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:30.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.762+0000 7f388e00f740 1 --2- 172.21.15.27:0/678173215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7f1570 0x55f2ec7ee7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.762+0000 7f388e00f740 1 -- 172.21.15.27:0/678173215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f2ec7f4360 con 0x55f2ec7e6580 2024-04-25T00:17:30.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.762+0000 7f388e00f740 1 -- 172.21.15.27:0/678173215 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f2ec707050 con 0x55f2ec7f1570 2024-04-25T00:17:30.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.762+0000 7f388e00f740 1 -- 172.21.15.27:0/678173215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f2ec7f44e0 con 0x55f2ec7e6ea0 2024-04-25T00:17:30.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.762+0000 7f38898a6640 1 --2- 172.21.15.27:0/678173215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7f1570 0x55f2ec7ee7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.763+0000 7f388e00f740 10 client.0 _ll_register_callbacks cb 0x55f2ec45ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:17:30.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.763+0000 7f38898a6640 1 -- 172.21.15.27:0/678173215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7e6ea0 msgr2=0x55f2ec7ee210 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:17:30.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.763+0000 7f38898a6640 1 --2- 172.21.15.27:0/678173215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7e6ea0 0x55f2ec7ee210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.763+0000 7f38898a6640 1 -- 172.21.15.27:0/678173215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 msgr2=0x55f2ec7f0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:30.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.763+0000 7f38898a6640 1 --2- 172.21.15.27:0/678173215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7f0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.763+0000 7f38898a6640 1 -- 172.21.15.27:0/678173215 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2ec6d8dc0 con 0x55f2ec7f1570 2024-04-25T00:17:30.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.764+0000 7f38898a6640 1 --2- 172.21.15.27:0/678173215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7f1570 0x55f2ec7ee7b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f387c0cc0d0 tx=0x7f387c09b660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbb238d6749e05d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.765+0000 7f386cff9640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T00:17:30.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.765+0000 7f386cff9640 1 -- 172.21.15.27:0/678173215 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c0a3b10 con 0x55f2ec7f1570 2024-04-25T00:17:30.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.765+0000 7f386cff9640 1 -- 172.21.15.27:0/678173215 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f387c097510 con 0x55f2ec7f1570 2024-04-25T00:17:30.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.765+0000 7f388e00f740 1 -- 172.21.15.27:0/678173215 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f388000bda0 con 0x55f2ec7f1570 2024-04-25T00:17:30.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.766+0000 7f386cff9640 1 -- 172.21.15.27:0/678173215 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f387c0cc240 con 0x55f2ec7f1570 2024-04-25T00:17:30.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.766+0000 7f388e00f740 10 client.25790 fetch_fsmap learned FSMap version 30 2024-04-25T00:17:30.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.766+0000 7f388e00f740 1 -- 172.21.15.27:0/678173215 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f2ec45f3f0 con 0x55f2ec7f1570 2024-04-25T00:17:30.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.767+0000 7f386cff9640 1 -- 172.21.15.27:0/678173215 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f387c0b7030 con 0x55f2ec7f1570 2024-04-25T00:17:30.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.767+0000 7f386cff9640 1 -- 172.21.15.27:0/678173215 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f387c09a490 con 0x55f2ec7f1570 2024-04-25T00:17:30.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.768+0000 7f388e00f740 10 client.25790 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:17:30.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.768+0000 7f388e00f740 10 client.25790 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:17:30.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.768+0000 7f388e00f740 20 client.25790 populate_metadata read hostname 'smithi027' 2024-04-25T00:17:30.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.768+0000 7f388e00f740 1 --2- 172.21.15.27:0/678173215 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55f2ec801430 0x55f2ec803820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.768+0000 7f388e00f740 4 client.25790 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:17:30.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.768+0000 7f388e00f740 1 -- 172.21.15.27:0/678173215 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f388000bda0 con 0x55f2ec801430 2024-04-25T00:17:30.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.769+0000 7f38890a5640 1 --2- 172.21.15.27:0/678173215 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55f2ec801430 0x55f2ec803820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.770+0000 7f386cff9640 10 client.25790 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:17:30.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.770+0000 7f38890a5640 1 --2- 172.21.15.27:0/678173215 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55f2ec801430 0x55f2ec803820 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=73779f57171f52d5 server_cookie=e86143c2f1ce1fde in_seq=0 out_seq=0 2024-04-25T00:17:30.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.772+0000 7f386cff9640 1 -- 172.21.15.27:0/678173215 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f388000bda0 con 0x55f2ec801430 2024-04-25T00:17:30.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.772+0000 7f386cff9640 10 client.25790 handle_command_reply: tid=0 2024-04-25T00:17:30.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.773+0000 7f388e00f740 1 client.25790 shutdown 2024-04-25T00:17:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.773+0000 7f388e00f740 1 -- 172.21.15.27:0/678173215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7f1570 msgr2=0x55f2ec7ee7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.773+0000 7f388e00f740 1 --2- 172.21.15.27:0/678173215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7f1570 0x55f2ec7ee7b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f387c0cc0d0 tx=0x7f387c09b660 comp rx=0 tx=0).stop 2024-04-25T00:17:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.773+0000 7f388e00f740 1 -- 172.21.15.27:0/678173215 shutdown_connections 2024-04-25T00:17:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.773+0000 7f388e00f740 1 --2- 172.21.15.27:0/678173215 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55f2ec801430 0x55f2ec803820 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.773+0000 7f388e00f740 1 --2- 172.21.15.27:0/678173215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7f1570 0x55f2ec7ee7b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.773+0000 7f388e00f740 1 --2- 172.21.15.27:0/678173215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7e6ea0 0x55f2ec7ee210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.773+0000 7f388e00f740 1 --2- 172.21.15.27:0/678173215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7f0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.773+0000 7f386cff9640 0 client.25790 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:17:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.773+0000 7f388e00f740 1 -- 172.21.15.27:0/678173215 >> 172.21.15.27:0/678173215 conn(0x55f2ec7e28a0 msgr2=0x55f2ec7e1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.773+0000 7f388e00f740 1 -- 172.21.15.27:0/678173215 shutdown_connections 2024-04-25T00:17:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.773+0000 7f388e00f740 1 -- 172.21.15.27:0/678173215 wait complete. 2024-04-25T00:17:30.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.774+0000 7f388e00f740 20 client.25790 trim_cache size 0 max 16384 2024-04-25T00:17:30.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.799+0000 7f388e00f740 1 Processor -- start 2024-04-25T00:17:30.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.800+0000 7f388e00f740 1 -- start start 2024-04-25T00:17:30.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.800+0000 7f388e00f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7fd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.800+0000 7f388e00f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7e6ea0 0x55f2ec7f0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.800+0000 7f388e00f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7f1570 0x55f2ec8090b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.800+0000 7f388e00f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f38841bd920 con 0x55f2ec7e6580 2024-04-25T00:17:30.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.800+0000 7f388e00f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f38841bf520 con 0x55f2ec7e6ea0 2024-04-25T00:17:30.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.800+0000 7f388e00f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f38841bdff0 con 0x55f2ec7f1570 2024-04-25T00:17:30.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.801+0000 7f38890a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7fd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.801+0000 7f38890a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7fd220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59864/0 (socket says 172.21.15.27:59864) 2024-04-25T00:17:30.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.801+0000 7f38890a5640 1 -- 172.21.15.27:0/2245499162 learned_addr learned my addr 172.21.15.27:0/2245499162 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:30.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.801+0000 7f38890a5640 1 -- 172.21.15.27:0/2245499162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7f1570 msgr2=0x55f2ec8090b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:17:30.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.801+0000 7f38890a5640 1 --2- 172.21.15.27:0/2245499162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7f1570 0x55f2ec8090b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.801+0000 7f38890a5640 1 -- 172.21.15.27:0/2245499162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7e6ea0 msgr2=0x55f2ec7f0a50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:17:30.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.801+0000 7f38890a5640 1 --2- 172.21.15.27:0/2245499162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7e6ea0 0x55f2ec7f0a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.801+0000 7f38890a5640 1 -- 172.21.15.27:0/2245499162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2ec5297d0 con 0x55f2ec7e6580 2024-04-25T00:17:30.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.801+0000 7f38890a5640 1 --2- 172.21.15.27:0/2245499162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7fd220 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3880011bf0 tx=0x7f3880009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2880eb7bae0341a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.802+0000 7f386effd640 1 -- 172.21.15.27:0/2245499162 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880015040 con 0x55f2ec7e6580 2024-04-25T00:17:30.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.802+0000 7f386effd640 1 -- 172.21.15.27:0/2245499162 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3880004520 con 0x55f2ec7e6580 2024-04-25T00:17:30.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.802+0000 7f386effd640 1 -- 172.21.15.27:0/2245499162 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880003690 con 0x55f2ec7e6580 2024-04-25T00:17:30.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.804+0000 7f388e00f740 1 -- 172.21.15.27:0/2245499162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 msgr2=0x55f2ec7fd220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:30.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.804+0000 7f388e00f740 1 --2- 172.21.15.27:0/2245499162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7fd220 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3880011bf0 tx=0x7f3880009040 comp rx=0 tx=0).stop 2024-04-25T00:17:30.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.804+0000 7f388e00f740 1 -- 172.21.15.27:0/2245499162 shutdown_connections 2024-04-25T00:17:30.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.804+0000 7f388e00f740 1 --2- 172.21.15.27:0/2245499162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7f1570 0x55f2ec8090b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.804+0000 7f388e00f740 1 --2- 172.21.15.27:0/2245499162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7e6ea0 0x55f2ec7f0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.804+0000 7f388e00f740 1 --2- 172.21.15.27:0/2245499162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7fd220 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.804+0000 7f388e00f740 1 -- 172.21.15.27:0/2245499162 >> 172.21.15.27:0/2245499162 conn(0x55f2ec7e28a0 msgr2=0x55f2ec7e8560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:30.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.805+0000 7f388e00f740 1 -- 172.21.15.27:0/2245499162 shutdown_connections 2024-04-25T00:17:30.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.805+0000 7f388e00f740 1 -- 172.21.15.27:0/2245499162 wait complete. 2024-04-25T00:17:30.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.806+0000 7f388e00f740 1 -- start start 2024-04-25T00:17:30.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.806+0000 7f388e00f740 1 Processor -- start 2024-04-25T00:17:30.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.806+0000 7f388e00f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7eb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.806+0000 7f388e00f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7e6ea0 0x55f2ec7eb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.806+0000 7f388e00f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7f1570 0x55f2ec7eba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.806+0000 7f388e00f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f38841bed90 con 0x55f2ec7e6580 2024-04-25T00:17:30.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.806+0000 7f388e00f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f38841be6c0 con 0x55f2ec7e6ea0 2024-04-25T00:17:30.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.806+0000 7f388e00f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3884069590 con 0x55f2ec7f1570 2024-04-25T00:17:30.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.806+0000 7f388e00f740 10 client.0 _ll_register_callbacks cb 0x7f387c097140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:17:30.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.806+0000 7f38890a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7eb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:30.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.807+0000 7f38890a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7eb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59866/0 (socket says 172.21.15.27:59866) 2024-04-25T00:17:30.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.807+0000 7f38890a5640 1 -- 172.21.15.27:0/2749802289 learned_addr learned my addr 172.21.15.27:0/2749802289 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:30.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.807+0000 7f38890a5640 1 -- 172.21.15.27:0/2749802289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7f1570 msgr2=0x55f2ec7eba90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:17:30.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.807+0000 7f38890a5640 1 --2- 172.21.15.27:0/2749802289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7f1570 0x55f2ec7eba90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.807+0000 7f38890a5640 1 -- 172.21.15.27:0/2749802289 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7e6ea0 msgr2=0x55f2ec7eb550 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:17:30.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.807+0000 7f38890a5640 1 --2- 172.21.15.27:0/2749802289 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7e6ea0 0x55f2ec7eb550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:30.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.807+0000 7f38890a5640 1 -- 172.21.15.27:0/2749802289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2ec8669f0 con 0x55f2ec7e6580 2024-04-25T00:17:30.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.807+0000 7f38890a5640 1 --2- 172.21.15.27:0/2749802289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7eb010 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3880006c80 tx=0x7f38800117f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e64a32e48995e5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:30.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.807+0000 7f386cff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:17:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.807+0000 7f386cff9640 1 -- 172.21.15.27:0/2749802289 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880032060 con 0x55f2ec7e6580 2024-04-25T00:17:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.808+0000 7f386cff9640 1 -- 172.21.15.27:0/2749802289 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f388000bb30 con 0x55f2ec7e6580 2024-04-25T00:17:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.808+0000 7f388e00f740 1 -- 172.21.15.27:0/2749802289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3880004520 con 0x55f2ec7e6580 2024-04-25T00:17:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.808+0000 7f386cff9640 1 -- 172.21.15.27:0/2749802289 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3880015050 con 0x55f2ec7e6580 2024-04-25T00:17:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.808+0000 7f388e00f740 10 client.16212 fetch_fsmap learned FSMap version 30 2024-04-25T00:17:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.808+0000 7f388e00f740 1 -- 172.21.15.27:0/2749802289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f2ec867cf0 con 0x55f2ec7e6580 2024-04-25T00:17:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.809+0000 7f386cff9640 1 -- 172.21.15.27:0/2749802289 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3880004520 con 0x55f2ec7e6580 2024-04-25T00:17:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.809+0000 7f386cff9640 1 -- 172.21.15.27:0/2749802289 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f388001a420 con 0x55f2ec7e6580 2024-04-25T00:17:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.809+0000 7f388e00f740 10 client.16212 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:17:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.809+0000 7f388e00f740 10 client.16212 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:17:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.809+0000 7f388e00f740 20 client.16212 populate_metadata read hostname 'smithi027' 2024-04-25T00:17:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.809+0000 7f388e00f740 1 --2- 172.21.15.27:0/2749802289 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55f2ec8681e0 0x55f2ec86a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:30.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.809+0000 7f388e00f740 4 client.16212 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:17:30.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:30.810+0000 7f388e00f740 1 -- 172.21.15.27:0/2749802289 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f2ec758bb0 con 0x55f2ec8681e0 2024-04-25T00:17:30.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:30 smithi027 ceph-mon[29208]: pgmap v55: 129 pgs: 129 active+clean; 5.7 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 489 MiB/s wr, 238 op/s 2024-04-25T00:17:30.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:30 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1858887703' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:17:31.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:30 smithi139 ceph-mon[32833]: pgmap v55: 129 pgs: 129 active+clean; 5.7 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 489 MiB/s wr, 238 op/s 2024-04-25T00:17:31.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:30 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1858887703' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:17:31.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:30 smithi153 ceph-mon[32896]: pgmap v55: 129 pgs: 129 active+clean; 5.7 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 489 MiB/s wr, 238 op/s 2024-04-25T00:17:31.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:30 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1858887703' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:17:31.038 INFO:tasks.workunit.client.0.smithi027.stdout:fs setup took 13 secs 2024-04-25T00:17:31.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.816+0000 7f38888a4640 1 --2- 172.21.15.27:0/2749802289 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55f2ec8681e0 0x55f2ec86a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.816+0000 7f38888a4640 1 --2- 172.21.15.27:0/2749802289 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55f2ec8681e0 0x55f2ec86a600 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=791e61bf0ecec2b1 server_cookie=a7e712a2c75743e3 in_seq=0 out_seq=0 2024-04-25T00:17:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.816+0000 7f386cff9640 10 client.16212 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:17:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.816+0000 7f386cff9640 1 -- 172.21.15.27:0/2749802289 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f2ec758bb0 con 0x55f2ec8681e0 2024-04-25T00:17:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.816+0000 7f386cff9640 10 client.16212 handle_command_reply: tid=0 2024-04-25T00:17:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.816+0000 7f388e00f740 1 client.16212 shutdown 2024-04-25T00:17:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.817+0000 7f388e00f740 1 -- 172.21.15.27:0/2749802289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 msgr2=0x55f2ec7eb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.817+0000 7f388e00f740 1 --2- 172.21.15.27:0/2749802289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7eb010 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3880006c80 tx=0x7f38800117f0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.817+0000 7f388e00f740 1 -- 172.21.15.27:0/2749802289 shutdown_connections 2024-04-25T00:17:31.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.817+0000 7f388e00f740 1 --2- 172.21.15.27:0/2749802289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2ec7f1570 0x55f2ec7eba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.817+0000 7f388e00f740 1 --2- 172.21.15.27:0/2749802289 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55f2ec8681e0 0x55f2ec86a600 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.817+0000 7f388e00f740 1 --2- 172.21.15.27:0/2749802289 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f2ec7e6ea0 0x55f2ec7eb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.817+0000 7f388e00f740 1 --2- 172.21.15.27:0/2749802289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f2ec7e6580 0x55f2ec7eb010 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.817+0000 7f386cff9640 0 client.16212 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:17:31.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.817+0000 7f388e00f740 1 -- 172.21.15.27:0/2749802289 >> 172.21.15.27:0/2749802289 conn(0x55f2ec7e28a0 msgr2=0x55f2ec7e8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:31.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.817+0000 7f388e00f740 1 -- 172.21.15.27:0/2749802289 shutdown_connections 2024-04-25T00:17:31.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.817+0000 7f388e00f740 1 -- 172.21.15.27:0/2749802289 wait complete. 2024-04-25T00:17:31.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.817+0000 7f388e00f740 20 client.16212 trim_cache size 0 max 16384 2024-04-25T00:17:31.819 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:17:31.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.818+0000 7f388b309640 1 -- 172.21.15.27:0/4061590445 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f38700a2ce0 msgr2=0x7f38700a51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:31.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.818+0000 7f388b309640 1 --2- 172.21.15.27:0/4061590445 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f38700a2ce0 0x7f38700a51a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f38740096c0 tx=0x7f3874005b50 comp rx=0 tx=0).stop 2024-04-25T00:17:31.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.818+0000 7f388b309640 1 -- 172.21.15.27:0/4061590445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3884106630 msgr2=0x7f3884107bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:31.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.818+0000 7f388b309640 1 --2- 172.21.15.27:0/4061590445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3884106630 0x7f3884107bd0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f387c00b990 tx=0x7f387c00be50 comp rx=0 tx=0).stop 2024-04-25T00:17:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.818+0000 7f388b309640 1 -- 172.21.15.27:0/4061590445 shutdown_connections 2024-04-25T00:17:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.818+0000 7f388b309640 1 --2- 172.21.15.27:0/4061590445 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f38700a2ce0 0x7f38700a51a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.818+0000 7f388b309640 1 --2- 172.21.15.27:0/4061590445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3884106630 0x7f3884107bd0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.818+0000 7f388b309640 1 --2- 172.21.15.27:0/4061590445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3884100fe0 0x7f3884107590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.818+0000 7f388b309640 1 --2- 172.21.15.27:0/4061590445 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3884100620 0x7f3884109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.818+0000 7f388b309640 1 -- 172.21.15.27:0/4061590445 >> 172.21.15.27:0/4061590445 conn(0x7f38840769c0 msgr2=0x7f38840fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.818+0000 7f388b309640 1 -- 172.21.15.27:0/4061590445 shutdown_connections 2024-04-25T00:17:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.818+0000 7f388b309640 1 -- 172.21.15.27:0/4061590445 wait complete. 2024-04-25T00:17:31.834 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T00:17:31.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.949+0000 7f36d64fc640 1 -- 172.21.15.27:0/1330318208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0100e60 msgr2=0x7f36d010dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:31.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.949+0000 7f36d64fc640 1 --2- 172.21.15.27:0/1330318208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0100e60 0x7f36d010dbb0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f36c4009900 tx=0x7f36c402a410 comp rx=0 tx=0).stop 2024-04-25T00:17:31.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.949+0000 7f36d64fc640 1 -- 172.21.15.27:0/1330318208 shutdown_connections 2024-04-25T00:17:31.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.949+0000 7f36d64fc640 1 --2- 172.21.15.27:0/1330318208 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36d010e0f0 0x7f36d01104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.949+0000 7f36d64fc640 1 --2- 172.21.15.27:0/1330318208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0100e60 0x7f36d010dbb0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.949+0000 7f36d64fc640 1 --2- 172.21.15.27:0/1330318208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d0100540 0x7f36d0100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.949+0000 7f36d64fc640 1 -- 172.21.15.27:0/1330318208 >> 172.21.15.27:0/1330318208 conn(0x7f36d0077740 msgr2=0x7f36d0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:31.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.950+0000 7f36d64fc640 1 -- 172.21.15.27:0/1330318208 shutdown_connections 2024-04-25T00:17:31.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.950+0000 7f36d64fc640 1 -- 172.21.15.27:0/1330318208 wait complete. 2024-04-25T00:17:31.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.950+0000 7f36d64fc640 1 Processor -- start 2024-04-25T00:17:31.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.951+0000 7f36d64fc640 1 -- start start 2024-04-25T00:17:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.951+0000 7f36d64fc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36d0100540 0x7f36d019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.951+0000 7f36d64fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0100e60 0x7f36d019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.951+0000 7f36d64fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d010e0f0 0x7f36d019eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.951+0000 7f36d64fc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f36d0113d70 con 0x7f36d0100e60 2024-04-25T00:17:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.951+0000 7f36d64fc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f36d0113bf0 con 0x7f36d0100540 2024-04-25T00:17:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.951+0000 7f36d64fc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f36d0113ef0 con 0x7f36d010e0f0 2024-04-25T00:17:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.951+0000 7f36d4a99640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d010e0f0 0x7f36d019eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.951+0000 7f36cf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0100e60 0x7f36d019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.951+0000 7f36d4a99640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d010e0f0 0x7f36d019eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:36214/0 (socket says 172.21.15.27:36214) 2024-04-25T00:17:31.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.952+0000 7f36d4a99640 1 -- 172.21.15.27:0/3054052262 learned_addr learned my addr 172.21.15.27:0/3054052262 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:31.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.952+0000 7f36cf7fe640 1 -- 172.21.15.27:0/3054052262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d010e0f0 msgr2=0x7f36d019eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:31.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.952+0000 7f36cf7fe640 1 --2- 172.21.15.27:0/3054052262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d010e0f0 0x7f36d019eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.952+0000 7f36cf7fe640 1 -- 172.21.15.27:0/3054052262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36d0100540 msgr2=0x7f36d019a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:31.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.952+0000 7f36cf7fe640 1 --2- 172.21.15.27:0/3054052262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36d0100540 0x7f36d019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.952+0000 7f36cf7fe640 1 -- 172.21.15.27:0/3054052262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d019f240 con 0x7f36d0100e60 2024-04-25T00:17:31.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.952+0000 7f36d4a99640 1 --2- 172.21.15.27:0/3054052262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d010e0f0 0x7f36d019eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:17:31.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.952+0000 7f36cf7fe640 1 --2- 172.21.15.27:0/3054052262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0100e60 0x7f36d019a790 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f36c402a930 tx=0x7f36c4003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:31.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.953+0000 7f36cd7fa640 1 -- 172.21.15.27:0/3054052262 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c403f070 con 0x7f36d0100e60 2024-04-25T00:17:31.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.953+0000 7f36d64fc640 1 -- 172.21.15.27:0/3054052262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36d019f4d0 con 0x7f36d0100e60 2024-04-25T00:17:31.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.953+0000 7f36cd7fa640 1 -- 172.21.15.27:0/3054052262 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36c4031c90 con 0x7f36d0100e60 2024-04-25T00:17:31.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.953+0000 7f36cd7fa640 1 -- 172.21.15.27:0/3054052262 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c4035030 con 0x7f36d0100e60 2024-04-25T00:17:31.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.953+0000 7f36d64fc640 1 -- 172.21.15.27:0/3054052262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36d01a6dc0 con 0x7f36d0100e60 2024-04-25T00:17:31.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.957+0000 7f36d9206740 1 Processor -- start 2024-04-25T00:17:31.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.957+0000 7f36d9206740 1 -- start start 2024-04-25T00:17:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.957+0000 7f36d9206740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f30879d2d0 0x55f30879d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.957+0000 7f36d9206740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f30879dc80 0x55f3087b0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.958+0000 7f36d9206740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3087b0d80 0x55f3087b3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.958+0000 7f36d9206740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f308702670 con 0x55f30879d2d0 2024-04-25T00:17:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.958+0000 7f36d9206740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f3086aa970 con 0x55f30879dc80 2024-04-25T00:17:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.958+0000 7f36d9206740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f3086fe630 con 0x55f3087b0d80 2024-04-25T00:17:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.958+0000 7f36d4a99640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f30879d2d0 0x55f30879d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.958+0000 7f36d4a99640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f30879d2d0 0x55f30879d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59900/0 (socket says 172.21.15.27:59900) 2024-04-25T00:17:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.958+0000 7f36d4a99640 1 -- 172.21.15.27:0/2862643395 learned_addr learned my addr 172.21.15.27:0/2862643395 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.958+0000 7f36cffff640 1 --2- 172.21.15.27:0/2862643395 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f30879dc80 0x55f3087b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.958+0000 7f36d4a99640 1 -- 172.21.15.27:0/2862643395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3087b0d80 msgr2=0x55f3087b3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:17:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.958+0000 7f36d4a99640 1 --2- 172.21.15.27:0/2862643395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3087b0d80 0x55f3087b3170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.958+0000 7f36d4a99640 1 -- 172.21.15.27:0/2862643395 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f30879dc80 msgr2=0x55f3087b0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.958+0000 7f36d4a99640 1 --2- 172.21.15.27:0/2862643395 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f30879dc80 0x55f3087b0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.958+0000 7f36d4a99640 1 -- 172.21.15.27:0/2862643395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3086ce400 con 0x55f30879d2d0 2024-04-25T00:17:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.959+0000 7f36d4a99640 1 --2- 172.21.15.27:0/2862643395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f30879d2d0 0x55f30879d6b0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f36c8011840 tx=0x7f36c8011d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da9f541eba30157 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.959+0000 7f36b67fc640 1 -- 172.21.15.27:0/2862643395 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c8011330 con 0x55f30879d2d0 2024-04-25T00:17:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.959+0000 7f36b67fc640 1 -- 172.21.15.27:0/2862643395 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36c800fe80 con 0x55f30879d2d0 2024-04-25T00:17:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.959+0000 7f36b67fc640 1 -- 172.21.15.27:0/2862643395 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c8010e60 con 0x55f30879d2d0 2024-04-25T00:17:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.959+0000 7f36cd7fa640 1 -- 172.21.15.27:0/3054052262 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f36c4042070 con 0x7f36d0100e60 2024-04-25T00:17:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.960+0000 7f36d9206740 1 -- 172.21.15.27:0/2862643395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f30879d2d0 msgr2=0x55f30879d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.960+0000 7f36d9206740 1 --2- 172.21.15.27:0/2862643395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f30879d2d0 0x55f30879d6b0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f36c8011840 tx=0x7f36c8011d00 comp rx=0 tx=0).stop 2024-04-25T00:17:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.960+0000 7f36cd7fa640 1 --2- 172.21.15.27:0/3054052262 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f36b80a2d80 0x7f36b80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.961+0000 7f36cd7fa640 1 -- 172.21.15.27:0/3054052262 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f36c40efee0 con 0x7f36d0100e60 2024-04-25T00:17:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.961+0000 7f36d4a99640 1 --2- 172.21.15.27:0/3054052262 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f36b80a2d80 0x7f36b80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.962+0000 7f36d4a99640 1 --2- 172.21.15.27:0/3054052262 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f36b80a2d80 0x7f36b80a5240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f36c8011680 tx=0x7f36c8011570 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.963+0000 7f36d9206740 1 -- 172.21.15.27:0/2862643395 shutdown_connections 2024-04-25T00:17:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.963+0000 7f36d9206740 1 --2- 172.21.15.27:0/2862643395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f3087b0d80 0x55f3087b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.963+0000 7f36d9206740 1 --2- 172.21.15.27:0/2862643395 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f30879dc80 0x55f3087b0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.963+0000 7f36d9206740 1 --2- 172.21.15.27:0/2862643395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f30879d2d0 0x55f30879d6b0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.963+0000 7f36d9206740 1 -- 172.21.15.27:0/2862643395 >> 172.21.15.27:0/2862643395 conn(0x55f3087998a0 msgr2=0x55f308799c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.963+0000 7f36d9206740 1 -- 172.21.15.27:0/2862643395 shutdown_connections 2024-04-25T00:17:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.963+0000 7f36d9206740 1 -- 172.21.15.27:0/2862643395 wait complete. 2024-04-25T00:17:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.963+0000 7f36d9206740 1 -- start start 2024-04-25T00:17:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.963+0000 7f36d9206740 1 Processor -- start 2024-04-25T00:17:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.964+0000 7f36d9206740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f30879d2d0 0x55f3087b0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.964+0000 7f36d9206740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f30879dc80 0x55f3087ad960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.964+0000 7f36d9206740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f3087b0d80 0x55f3087adea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.964+0000 7f36d9206740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f3087b3b40 con 0x55f30879d2d0 2024-04-25T00:17:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.964+0000 7f36d9206740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f3086be050 con 0x55f3087b0d80 2024-04-25T00:17:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.964+0000 7f36d9206740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f3087b3cc0 con 0x55f30879dc80 2024-04-25T00:17:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.964+0000 7f36d9206740 10 client.0 _ll_register_callbacks cb 0x55f308415be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:17:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.965+0000 7f36cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f30879d2d0 0x55f3087b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.965+0000 7f36cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f30879d2d0 0x55f3087b0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59914/0 (socket says 172.21.15.27:59914) 2024-04-25T00:17:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.965+0000 7f36cffff640 1 -- 172.21.15.27:0/3166204664 learned_addr learned my addr 172.21.15.27:0/3166204664 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.965+0000 7f36cf7fe640 1 --2- 172.21.15.27:0/3166204664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f30879dc80 0x55f3087ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.965+0000 7f36d4a99640 1 --2- 172.21.15.27:0/3166204664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f3087b0d80 0x55f3087adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.965+0000 7f36cffff640 1 -- 172.21.15.27:0/3166204664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f30879dc80 msgr2=0x55f3087ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:31.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.965+0000 7f36cffff640 1 --2- 172.21.15.27:0/3166204664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f30879dc80 0x55f3087ad960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.965+0000 7f36cffff640 1 -- 172.21.15.27:0/3166204664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f3087b0d80 msgr2=0x55f3087adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:31.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.965+0000 7f36cffff640 1 --2- 172.21.15.27:0/3166204664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f3087b0d80 0x55f3087adea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.965+0000 7f36cffff640 1 -- 172.21.15.27:0/3166204664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f30868fdc0 con 0x55f30879d2d0 2024-04-25T00:17:31.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.966+0000 7f36cffff640 1 --2- 172.21.15.27:0/3166204664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f30879d2d0 0x55f3087b0670 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f36c0009fd0 tx=0x7f36c0011c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c4108c9325fe53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:31.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.966+0000 7f36b3ff7640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:17:31.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.966+0000 7f36b3ff7640 1 -- 172.21.15.27:0/3166204664 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c001f060 con 0x55f30879d2d0 2024-04-25T00:17:31.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.966+0000 7f36b3ff7640 1 -- 172.21.15.27:0/3166204664 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f36c000fac0 con 0x55f30879d2d0 2024-04-25T00:17:31.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.966+0000 7f36d9206740 1 -- 172.21.15.27:0/3166204664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f30870fbb0 con 0x55f30879d2d0 2024-04-25T00:17:31.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.967+0000 7f36b3ff7640 1 -- 172.21.15.27:0/3166204664 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f36c0026050 con 0x55f30879d2d0 2024-04-25T00:17:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.967+0000 7f36d9206740 10 client.16224 fetch_fsmap learned FSMap version 30 2024-04-25T00:17:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.967+0000 7f36d9206740 1 -- 172.21.15.27:0/3166204664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f3084163f0 con 0x55f30879d2d0 2024-04-25T00:17:31.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.968+0000 7f36b3ff7640 1 -- 172.21.15.27:0/3166204664 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f30870fbb0 con 0x55f30879d2d0 2024-04-25T00:17:31.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.969+0000 7f36b3ff7640 1 -- 172.21.15.27:0/3166204664 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f36c001a440 con 0x55f30879d2d0 2024-04-25T00:17:31.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.969+0000 7f36d9206740 10 client.16224 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:17:31.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.969+0000 7f36d9206740 10 client.16224 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:17:31.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.969+0000 7f36d9206740 20 client.16224 populate_metadata read hostname 'smithi027' 2024-04-25T00:17:31.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.969+0000 7f36d9206740 1 --2- 172.21.15.27:0/3166204664 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55f3087a3de0 0x55f3087a6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.969+0000 7f36d9206740 4 client.16224 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:17:31.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.969+0000 7f36d9206740 1 -- 172.21.15.27:0/3166204664 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f308709f00 con 0x55f3087a3de0 2024-04-25T00:17:31.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.970+0000 7f36cf7fe640 1 --2- 172.21.15.27:0/3166204664 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55f3087a3de0 0x55f3087a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.970+0000 7f36b3ff7640 10 client.16224 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:17:31.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.970+0000 7f36cf7fe640 1 --2- 172.21.15.27:0/3166204664 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55f3087a3de0 0x55f3087a6250 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3bb6c999cc00d511 server_cookie=66d715ddca5e5c42 in_seq=0 out_seq=0 2024-04-25T00:17:31.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.971+0000 7f36b3ff7640 1 -- 172.21.15.27:0/3166204664 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f308709f00 con 0x55f3087a3de0 2024-04-25T00:17:31.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.971+0000 7f36b3ff7640 10 client.16224 handle_command_reply: tid=0 2024-04-25T00:17:31.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.971+0000 7f36d9206740 1 client.16224 shutdown 2024-04-25T00:17:31.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.973+0000 7f36d9206740 1 -- 172.21.15.27:0/3166204664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f30879d2d0 msgr2=0x55f3087b0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:31.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.973+0000 7f36d9206740 1 --2- 172.21.15.27:0/3166204664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f30879d2d0 0x55f3087b0670 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f36c0009fd0 tx=0x7f36c0011c30 comp rx=0 tx=0).stop 2024-04-25T00:17:31.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.973+0000 7f36d9206740 1 -- 172.21.15.27:0/3166204664 shutdown_connections 2024-04-25T00:17:31.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.973+0000 7f36d9206740 1 --2- 172.21.15.27:0/3166204664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f3087b0d80 0x55f3087adea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.973+0000 7f36d9206740 1 --2- 172.21.15.27:0/3166204664 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55f3087a3de0 0x55f3087a6250 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.973+0000 7f36d9206740 1 --2- 172.21.15.27:0/3166204664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f30879dc80 0x55f3087ad960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.973+0000 7f36d9206740 1 --2- 172.21.15.27:0/3166204664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f30879d2d0 0x55f3087b0670 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.973+0000 7f36d9206740 1 -- 172.21.15.27:0/3166204664 >> 172.21.15.27:0/3166204664 conn(0x55f3087998a0 msgr2=0x55f30879f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:31.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.973+0000 7f36b3ff7640 0 client.16224 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:17:31.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.975+0000 7f36d9206740 1 -- 172.21.15.27:0/3166204664 shutdown_connections 2024-04-25T00:17:31.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.977+0000 7f36d9206740 1 -- 172.21.15.27:0/3166204664 wait complete. 2024-04-25T00:17:31.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:31.977+0000 7f36d9206740 20 client.16224 trim_cache size 0 max 16384 2024-04-25T00:17:32.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.015+0000 7f36d9206740 1 Processor -- start 2024-04-25T00:17:32.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.015+0000 7f36d9206740 1 -- start start 2024-04-25T00:17:32.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.016+0000 7f36d9206740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f3087998a0 0x55f3087bca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.017+0000 7f36d9206740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f30879d2d0 0x55f3087b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.017+0000 7f36d9206740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f30879dc80 0x55f3087d3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.017+0000 7f36d9206740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f36d01aca40 con 0x55f3087998a0 2024-04-25T00:17:32.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.017+0000 7f36d9206740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f36d01ae640 con 0x55f30879d2d0 2024-04-25T00:17:32.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.017+0000 7f36d9206740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f36d01ad110 con 0x55f30879dc80 2024-04-25T00:17:32.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.017+0000 7f36cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f3087998a0 0x55f3087bca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.017+0000 7f36cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f3087998a0 0x55f3087bca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59926/0 (socket says 172.21.15.27:59926) 2024-04-25T00:17:32.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.017+0000 7f36cffff640 1 -- 172.21.15.27:0/498848312 learned_addr learned my addr 172.21.15.27:0/498848312 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:32.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.019+0000 7f36cffff640 1 -- 172.21.15.27:0/498848312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f30879dc80 msgr2=0x55f3087d3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:32.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.019+0000 7f36cffff640 1 --2- 172.21.15.27:0/498848312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f30879dc80 0x55f3087d3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.019+0000 7f36cffff640 1 -- 172.21.15.27:0/498848312 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f30879d2d0 msgr2=0x55f3087b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:32.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.019+0000 7f36cffff640 1 --2- 172.21.15.27:0/498848312 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f30879d2d0 0x55f3087b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.019+0000 7f36cffff640 1 -- 172.21.15.27:0/498848312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3084e07d0 con 0x55f3087998a0 2024-04-25T00:17:32.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.019+0000 7f36cffff640 1 --2- 172.21.15.27:0/498848312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f3087998a0 0x55f3087bca40 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f36c0001eb0 tx=0x7f36c00097f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=105403cdb63b944d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:32.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.020+0000 7f36b5ffb640 1 -- 172.21.15.27:0/498848312 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c002e070 con 0x55f3087998a0 2024-04-25T00:17:32.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.020+0000 7f36b5ffb640 1 -- 172.21.15.27:0/498848312 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36c001a640 con 0x55f3087998a0 2024-04-25T00:17:32.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.020+0000 7f36b5ffb640 1 -- 172.21.15.27:0/498848312 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c0004a00 con 0x55f3087998a0 2024-04-25T00:17:32.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.024+0000 7f36d9206740 1 -- 172.21.15.27:0/498848312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f3087998a0 msgr2=0x55f3087bca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:32.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.024+0000 7f36d9206740 1 --2- 172.21.15.27:0/498848312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f3087998a0 0x55f3087bca40 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f36c0001eb0 tx=0x7f36c00097f0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.028+0000 7f36d9206740 1 -- 172.21.15.27:0/498848312 shutdown_connections 2024-04-25T00:17:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.028+0000 7f36d9206740 1 --2- 172.21.15.27:0/498848312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f30879dc80 0x55f3087d3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.028+0000 7f36d9206740 1 --2- 172.21.15.27:0/498848312 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f30879d2d0 0x55f3087b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.028+0000 7f36d9206740 1 --2- 172.21.15.27:0/498848312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f3087998a0 0x55f3087bca40 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.028+0000 7f36d9206740 1 -- 172.21.15.27:0/498848312 >> 172.21.15.27:0/498848312 conn(0x55f3087ae000 msgr2=0x55f3087982b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:32.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.034+0000 7f36d9206740 1 -- 172.21.15.27:0/498848312 shutdown_connections 2024-04-25T00:17:32.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.041+0000 7f36d9206740 1 -- 172.21.15.27:0/498848312 wait complete. 2024-04-25T00:17:32.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.041+0000 7f36d9206740 1 -- start start 2024-04-25T00:17:32.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.042+0000 7f36d9206740 1 Processor -- start 2024-04-25T00:17:32.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.042+0000 7f36d9206740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f3087998a0 0x55f3087b9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.043+0000 7f36d9206740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f30879d2d0 0x55f3087b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.043+0000 7f36d9206740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f30879dc80 0x55f3087b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.043+0000 7f36d9206740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f36d01adeb0 con 0x55f3087998a0 2024-04-25T00:17:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.043+0000 7f36d9206740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f36d01ad7e0 con 0x55f30879dc80 2024-04-25T00:17:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.043+0000 7f36d9206740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f36d01aaa00 con 0x55f30879d2d0 2024-04-25T00:17:32.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.044+0000 7f36cf7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f30879d2d0 0x55f3087b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.044+0000 7f36cf7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f30879d2d0 0x55f3087b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:36244/0 (socket says 172.21.15.27:36244) 2024-04-25T00:17:32.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.044+0000 7f36cf7fe640 1 -- 172.21.15.27:0/485269477 learned_addr learned my addr 172.21.15.27:0/485269477 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:32.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.044+0000 7f36d4a99640 1 --2- 172.21.15.27:0/485269477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f30879dc80 0x55f3087b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.044+0000 7f36d9206740 10 client.0 _ll_register_callbacks cb 0x7f36c00113e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:17:32.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.053+0000 7f36d4a99640 1 -- 172.21.15.27:0/485269477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f30879d2d0 msgr2=0x55f3087b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:32.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.053+0000 7f36d4a99640 1 --2- 172.21.15.27:0/485269477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f30879d2d0 0x55f3087b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.053+0000 7f36d4a99640 1 -- 172.21.15.27:0/485269477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f3087998a0 msgr2=0x55f3087b9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:32.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.053+0000 7f36d4a99640 1 --2- 172.21.15.27:0/485269477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f3087998a0 0x55f3087b9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.053+0000 7f36d4a99640 1 -- 172.21.15.27:0/485269477 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3087b51a0 con 0x55f30879dc80 2024-04-25T00:17:32.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.058+0000 7f36d4a99640 1 --2- 172.21.15.27:0/485269477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f30879dc80 0x55f3087b4ab0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x55f3087b0d50 tx=0x7f36c800fd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a83cdb7845e89272 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:32.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.059+0000 7f36b3ff7640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T00:17:32.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.059+0000 7f36b3ff7640 1 -- 172.21.15.27:0/485269477 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c802c040 con 0x55f30879dc80 2024-04-25T00:17:32.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.059+0000 7f36b3ff7640 1 -- 172.21.15.27:0/485269477 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f36c8011130 con 0x55f30879dc80 2024-04-25T00:17:32.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.063+0000 7f36d9206740 1 -- 172.21.15.27:0/485269477 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36c001a640 con 0x55f30879dc80 2024-04-25T00:17:32.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.066+0000 7f36b3ff7640 1 -- 172.21.15.27:0/485269477 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f36c801f040 con 0x55f30879dc80 2024-04-25T00:17:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.068+0000 7f36d9206740 10 client.25802 fetch_fsmap learned FSMap version 30 2024-04-25T00:17:32.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.069+0000 7f36d9206740 1 -- 172.21.15.27:0/485269477 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f30882f750 con 0x55f30879dc80 2024-04-25T00:17:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.073+0000 7f36b3ff7640 1 -- 172.21.15.27:0/485269477 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36c80055a0 con 0x55f30879dc80 2024-04-25T00:17:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.073+0000 7f36b3ff7640 1 -- 172.21.15.27:0/485269477 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f36c8018070 con 0x55f30879dc80 2024-04-25T00:17:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.073+0000 7f36d9206740 10 client.25802 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:17:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.073+0000 7f36d9206740 10 client.25802 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:17:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.073+0000 7f36d9206740 20 client.25802 populate_metadata read hostname 'smithi027' 2024-04-25T00:17:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.073+0000 7f36d9206740 1 --2- 172.21.15.27:0/485269477 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55f3087b0d80 0x55f308831c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.075+0000 7f36d9206740 4 client.25802 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:17:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.075+0000 7f36d9206740 1 -- 172.21.15.27:0/485269477 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f36c001a640 con 0x55f3087b0d80 2024-04-25T00:17:32.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.082+0000 7f36cffff640 1 --2- 172.21.15.27:0/485269477 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55f3087b0d80 0x55f308831c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.092+0000 7f36cffff640 1 --2- 172.21.15.27:0/485269477 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55f3087b0d80 0x55f308831c90 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=14832a330c82a26f server_cookie=9fd8eda436524e8f in_seq=0 out_seq=0 2024-04-25T00:17:32.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.092+0000 7f36b3ff7640 10 client.25802 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:17:32.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.092+0000 7f36b3ff7640 1 -- 172.21.15.27:0/485269477 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f36c001a640 con 0x55f3087b0d80 2024-04-25T00:17:32.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.092+0000 7f36b3ff7640 10 client.25802 handle_command_reply: tid=0 2024-04-25T00:17:32.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.092+0000 7f36d9206740 1 client.25802 shutdown 2024-04-25T00:17:32.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.095+0000 7f36d9206740 1 -- 172.21.15.27:0/485269477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f30879dc80 msgr2=0x55f3087b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:32.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.095+0000 7f36d9206740 1 --2- 172.21.15.27:0/485269477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f30879dc80 0x55f3087b4ab0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x55f3087b0d50 tx=0x7f36c800fd70 comp rx=0 tx=0).stop 2024-04-25T00:17:32.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.099+0000 7f36d9206740 1 -- 172.21.15.27:0/485269477 shutdown_connections 2024-04-25T00:17:32.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.099+0000 7f36d9206740 1 --2- 172.21.15.27:0/485269477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f30879dc80 0x55f3087b4ab0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.099+0000 7f36d9206740 1 --2- 172.21.15.27:0/485269477 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55f3087b0d80 0x55f308831c90 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.099+0000 7f36d9206740 1 --2- 172.21.15.27:0/485269477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f30879d2d0 0x55f3087b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.099+0000 7f36d9206740 1 --2- 172.21.15.27:0/485269477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f3087998a0 0x55f3087b9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.099+0000 7f36d9206740 1 -- 172.21.15.27:0/485269477 >> 172.21.15.27:0/485269477 conn(0x55f3087ae000 msgr2=0x55f308798b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:32.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.099+0000 7f36b3ff7640 0 client.25802 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:17:32.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.104+0000 7f36d9206740 1 -- 172.21.15.27:0/485269477 shutdown_connections 2024-04-25T00:17:32.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.108+0000 7f36d9206740 1 -- 172.21.15.27:0/485269477 wait complete. 2024-04-25T00:17:32.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.110+0000 7f36d9206740 20 client.25802 trim_cache size 0 max 16384 2024-04-25T00:17:32.116 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:17:32.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.118+0000 7f36d64fc640 1 -- 172.21.15.27:0/3054052262 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f36b80a2d80 msgr2=0x7f36b80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:32.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.118+0000 7f36d64fc640 1 --2- 172.21.15.27:0/3054052262 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f36b80a2d80 0x7f36b80a5240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f36c8011680 tx=0x7f36c8011570 comp rx=0 tx=0).stop 2024-04-25T00:17:32.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.119+0000 7f36d64fc640 1 -- 172.21.15.27:0/3054052262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0100e60 msgr2=0x7f36d019a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:32.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.119+0000 7f36d64fc640 1 --2- 172.21.15.27:0/3054052262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0100e60 0x7f36d019a790 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f36c402a930 tx=0x7f36c4003b20 comp rx=0 tx=0).stop 2024-04-25T00:17:32.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.127+0000 7f36d64fc640 1 -- 172.21.15.27:0/3054052262 shutdown_connections 2024-04-25T00:17:32.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.127+0000 7f36d64fc640 1 --2- 172.21.15.27:0/3054052262 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f36b80a2d80 0x7f36b80a5240 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.127+0000 7f36d64fc640 1 --2- 172.21.15.27:0/3054052262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d010e0f0 0x7f36d019eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.127+0000 7f36d64fc640 1 --2- 172.21.15.27:0/3054052262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0100e60 0x7f36d019a790 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.128+0000 7f36d64fc640 1 --2- 172.21.15.27:0/3054052262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36d0100540 0x7f36d019a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.128+0000 7f36d64fc640 1 -- 172.21.15.27:0/3054052262 >> 172.21.15.27:0/3054052262 conn(0x7f36d0077740 msgr2=0x7f36d010f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.131+0000 7f36d64fc640 1 -- 172.21.15.27:0/3054052262 shutdown_connections 2024-04-25T00:17:32.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.134+0000 7f36d64fc640 1 -- 172.21.15.27:0/3054052262 wait complete. 2024-04-25T00:17:32.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T00:17:32.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.504+0000 7f8c4a888640 1 -- 172.21.15.27:0/870676602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c4410ff90 msgr2=0x7f8c4410da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:32.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.504+0000 7f8c4a888640 1 --2- 172.21.15.27:0/870676602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c4410ff90 0x7f8c4410da80 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8c34009930 tx=0x7f8c3402a460 comp rx=0 tx=0).stop 2024-04-25T00:17:32.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.504+0000 7f8c4a888640 1 -- 172.21.15.27:0/870676602 shutdown_connections 2024-04-25T00:17:32.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.504+0000 7f8c4a888640 1 --2- 172.21.15.27:0/870676602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c4411af20 0x7f8c4411d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.504+0000 7f8c4a888640 1 --2- 172.21.15.27:0/870676602 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c4410dfc0 0x7f8c4410e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.504+0000 7f8c4a888640 1 --2- 172.21.15.27:0/870676602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c4410ff90 0x7f8c4410da80 secure :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8c34009930 tx=0x7f8c3402a460 comp rx=0 tx=0).stop 2024-04-25T00:17:32.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.504+0000 7f8c4a888640 1 -- 172.21.15.27:0/870676602 >> 172.21.15.27:0/870676602 conn(0x7f8c4406c850 msgr2=0x7f8c4406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:32.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.510+0000 7f8c4a888640 1 -- 172.21.15.27:0/870676602 shutdown_connections 2024-04-25T00:17:32.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.516+0000 7f8c4a888640 1 -- 172.21.15.27:0/870676602 wait complete. 2024-04-25T00:17:32.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.517+0000 7f8c4a888640 1 Processor -- start 2024-04-25T00:17:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.519+0000 7f8c4a888640 1 -- start start 2024-04-25T00:17:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.520+0000 7f8c4a888640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c4410dfc0 0x7f8c441186f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.520+0000 7f8c4a888640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c4411af20 0x7f8c44111770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.520+0000 7f8c4a888640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c44111cb0 0x7f8c44112160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.520+0000 7f8c4a888640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8c44120980 con 0x7f8c44111cb0 2024-04-25T00:17:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.520+0000 7f8c4a888640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8c44120800 con 0x7f8c4410dfc0 2024-04-25T00:17:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.520+0000 7f8c4a888640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8c44120b00 con 0x7f8c4411af20 2024-04-25T00:17:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.521+0000 7f8c48e25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c44111cb0 0x7f8c44112160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.521+0000 7f8c48e25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c44111cb0 0x7f8c44112160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59938/0 (socket says 172.21.15.27:59938) 2024-04-25T00:17:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.521+0000 7f8c48e25640 1 -- 172.21.15.27:0/3434432808 learned_addr learned my addr 172.21.15.27:0/3434432808 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.525+0000 7f8c437fe640 1 --2- 172.21.15.27:0/3434432808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c4411af20 0x7f8c44111770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.525+0000 7f8c43fff640 1 --2- 172.21.15.27:0/3434432808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c4410dfc0 0x7f8c441186f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.526+0000 7f8c48e25640 1 -- 172.21.15.27:0/3434432808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c4411af20 msgr2=0x7f8c44111770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.526+0000 7f8c48e25640 1 --2- 172.21.15.27:0/3434432808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c4411af20 0x7f8c44111770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.526+0000 7f8c48e25640 1 -- 172.21.15.27:0/3434432808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c4410dfc0 msgr2=0x7f8c441186f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.526+0000 7f8c48e25640 1 --2- 172.21.15.27:0/3434432808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c4410dfc0 0x7f8c441186f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.526+0000 7f8c48e25640 1 -- 172.21.15.27:0/3434432808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c44116310 con 0x7f8c44111cb0 2024-04-25T00:17:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.535+0000 7f8c48e25640 1 --2- 172.21.15.27:0/3434432808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c44111cb0 0x7f8c44112160 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800c970 tx=0x7f8c3800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:32.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.536+0000 7f8c417fa640 1 -- 172.21.15.27:0/3434432808 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c38007bb0 con 0x7f8c44111cb0 2024-04-25T00:17:32.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.536+0000 7f8c417fa640 1 -- 172.21.15.27:0/3434432808 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c38007d50 con 0x7f8c44111cb0 2024-04-25T00:17:32.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.536+0000 7f8c417fa640 1 -- 172.21.15.27:0/3434432808 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3800f450 con 0x7f8c44111cb0 2024-04-25T00:17:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.539+0000 7f8c4a888640 1 -- 172.21.15.27:0/3434432808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c44116600 con 0x7f8c44111cb0 2024-04-25T00:17:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.539+0000 7f8c4a888640 1 -- 172.21.15.27:0/3434432808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c44116b00 con 0x7f8c44111cb0 2024-04-25T00:17:32.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.550+0000 7f8c417fa640 1 -- 172.21.15.27:0/3434432808 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8c38004f10 con 0x7f8c44111cb0 2024-04-25T00:17:32.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.550+0000 7f8c4d590740 1 Processor -- start 2024-04-25T00:17:32.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.551+0000 7f8c4d590740 1 -- start start 2024-04-25T00:17:32.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.552+0000 7f8c4d590740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84409580 0x557b84409960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.552+0000 7f8c4d590740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84409ea0 0x557b84414030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.552+0000 7f8c4d590740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84414570 0x557b84416960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.552+0000 7f8c4d590740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557b8436e670 con 0x557b84409ea0 2024-04-25T00:17:32.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.552+0000 7f8c4d590740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557b84316970 con 0x557b84414570 2024-04-25T00:17:32.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.552+0000 7f8c4d590740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557b8436a630 con 0x557b84409580 2024-04-25T00:17:32.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.552+0000 7f8c437fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84409580 0x557b84409960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.552+0000 7f8c437fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84409580 0x557b84409960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:36260/0 (socket says 172.21.15.27:36260) 2024-04-25T00:17:32.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.552+0000 7f8c437fe640 1 -- 172.21.15.27:0/246024345 learned_addr learned my addr 172.21.15.27:0/246024345 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:32.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.556+0000 7f8c437fe640 1 --2- 172.21.15.27:0/246024345 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84414570 0x557b84416960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.562+0000 7f8c437fe640 1 -- 172.21.15.27:0/246024345 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84414570 msgr2=0x557b84416960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:32.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.562+0000 7f8c437fe640 1 --2- 172.21.15.27:0/246024345 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84414570 0x557b84416960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.562+0000 7f8c437fe640 1 -- 172.21.15.27:0/246024345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84409ea0 msgr2=0x557b84414030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:32.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.562+0000 7f8c437fe640 1 --2- 172.21.15.27:0/246024345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84409ea0 0x557b84414030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.562+0000 7f8c437fe640 1 -- 172.21.15.27:0/246024345 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b8433a400 con 0x557b84409580 2024-04-25T00:17:32.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.567+0000 7f8c417fa640 1 --2- 172.21.15.27:0/3434432808 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8c2c0a2f00 0x7f8c2c0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.568+0000 7f8c417fa640 1 -- 172.21.15.27:0/3434432808 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8c380cf760 con 0x7f8c44111cb0 2024-04-25T00:17:32.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.568+0000 7f8c437fe640 1 --2- 172.21.15.27:0/246024345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84409580 0x557b84409960 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00c940 tx=0x7f8c3c0099f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b15846a16fc8cc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:32.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.570+0000 7f8c2a7fc640 1 -- 172.21.15.27:0/246024345 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c00c2d0 con 0x557b84409580 2024-04-25T00:17:32.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.570+0000 7f8c2a7fc640 1 -- 172.21.15.27:0/246024345 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c3c010030 con 0x557b84409580 2024-04-25T00:17:32.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.570+0000 7f8c2a7fc640 1 -- 172.21.15.27:0/246024345 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c00ab40 con 0x557b84409580 2024-04-25T00:17:32.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.571+0000 7f8c43fff640 1 --2- 172.21.15.27:0/3434432808 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8c2c0a2f00 0x7f8c2c0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.581+0000 7f8c4d590740 1 -- 172.21.15.27:0/246024345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84409580 msgr2=0x557b84409960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:32.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.581+0000 7f8c4d590740 1 --2- 172.21.15.27:0/246024345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84409580 0x557b84409960 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00c940 tx=0x7f8c3c0099f0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.583+0000 7f8c4d590740 1 -- 172.21.15.27:0/246024345 shutdown_connections 2024-04-25T00:17:32.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.583+0000 7f8c4d590740 1 --2- 172.21.15.27:0/246024345 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84414570 0x557b84416960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.583+0000 7f8c4d590740 1 --2- 172.21.15.27:0/246024345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84409ea0 0x557b84414030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.583+0000 7f8c4d590740 1 --2- 172.21.15.27:0/246024345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84409580 0x557b84409960 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.583+0000 7f8c4d590740 1 -- 172.21.15.27:0/246024345 >> 172.21.15.27:0/246024345 conn(0x557b844058a0 msgr2=0x557b84405c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:32.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.583+0000 7f8c43fff640 1 --2- 172.21.15.27:0/3434432808 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8c2c0a2f00 0x7f8c2c0a53c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8c34009230 tx=0x7f8c34005fd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:32.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.584+0000 7f8c4d590740 1 -- 172.21.15.27:0/246024345 shutdown_connections 2024-04-25T00:17:32.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.588+0000 7f8c4d590740 1 -- 172.21.15.27:0/246024345 wait complete. 2024-04-25T00:17:32.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.589+0000 7f8c4d590740 1 -- start start 2024-04-25T00:17:32.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.589+0000 7f8c4d590740 1 Processor -- start 2024-04-25T00:17:32.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.590+0000 7f8c4d590740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84409580 0x557b84413f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.590+0000 7f8c4d590740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84409ea0 0x557b84411250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.590+0000 7f8c4d590740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84414570 0x557b84411860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.590+0000 7f8c4d590740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557b84417360 con 0x557b84409580 2024-04-25T00:17:32.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.590+0000 7f8c4d590740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557b8432a050 con 0x557b84409ea0 2024-04-25T00:17:32.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.590+0000 7f8c4d590740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557b844174e0 con 0x557b84414570 2024-04-25T00:17:32.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.590+0000 7f8c4d590740 10 client.0 _ll_register_callbacks cb 0x557b84081be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:17:32.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.594+0000 7f8c48e25640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84414570 0x557b84411860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.594+0000 7f8c48e25640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84414570 0x557b84411860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:36262/0 (socket says 172.21.15.27:36262) 2024-04-25T00:17:32.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.594+0000 7f8c48e25640 1 -- 172.21.15.27:0/4113491605 learned_addr learned my addr 172.21.15.27:0/4113491605 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:32.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.594+0000 7f8c437fe640 1 --2- 172.21.15.27:0/4113491605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84409ea0 0x557b84411250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.598+0000 7f8c48e25640 1 -- 172.21.15.27:0/4113491605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84409ea0 msgr2=0x557b84411250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:32.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.598+0000 7f8c48e25640 1 --2- 172.21.15.27:0/4113491605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84409ea0 0x557b84411250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.598+0000 7f8c48e25640 1 -- 172.21.15.27:0/4113491605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84409580 msgr2=0x557b84413f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:32.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.598+0000 7f8c48e25640 1 --2- 172.21.15.27:0/4113491605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84409580 0x557b84413f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.598+0000 7f8c48e25640 1 -- 172.21.15.27:0/4113491605 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b842fbdc0 con 0x557b84414570 2024-04-25T00:17:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.604+0000 7f8c48e25640 1 --2- 172.21.15.27:0/4113491605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84414570 0x557b84411860 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8c380a4090 tx=0x7f8c38004800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=590df81663fe9bf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.605+0000 7f8c237fe640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:17:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.608+0000 7f8c237fe640 1 -- 172.21.15.27:0/4113491605 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c380b3070 con 0x557b84414570 2024-04-25T00:17:32.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.608+0000 7f8c237fe640 1 -- 172.21.15.27:0/4113491605 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8c38005200 con 0x557b84414570 2024-04-25T00:17:32.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.615+0000 7f8c4d590740 1 -- 172.21.15.27:0/4113491605 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c3c010030 con 0x557b84414570 2024-04-25T00:17:32.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.621+0000 7f8c237fe640 1 -- 172.21.15.27:0/4113491605 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8c380a45c0 con 0x557b84414570 2024-04-25T00:17:32.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.621+0000 7f8c4d590740 10 client.25768 fetch_fsmap learned FSMap version 30 2024-04-25T00:17:32.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.621+0000 7f8c4d590740 1 -- 172.21.15.27:0/4113491605 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557b840823f0 con 0x557b84414570 2024-04-25T00:17:32.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.622+0000 7f8c237fe640 1 -- 172.21.15.27:0/4113491605 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c3c010030 con 0x557b84414570 2024-04-25T00:17:32.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.624+0000 7f8c237fe640 1 -- 172.21.15.27:0/4113491605 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f8c380cf3a0 con 0x557b84414570 2024-04-25T00:17:32.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.624+0000 7f8c4d590740 10 client.25768 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:17:32.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.624+0000 7f8c4d590740 10 client.25768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:17:32.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.624+0000 7f8c4d590740 20 client.25768 populate_metadata read hostname 'smithi027' 2024-04-25T00:17:32.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.624+0000 7f8c4d590740 1 --2- 172.21.15.27:0/4113491605 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x557b84424b50 0x557b84426ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.624+0000 7f8c4d590740 4 client.25768 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:17:32.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.624+0000 7f8c4d590740 1 -- 172.21.15.27:0/4113491605 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b8437bbb0 con 0x557b84424b50 2024-04-25T00:17:32.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.632+0000 7f8c43fff640 1 --2- 172.21.15.27:0/4113491605 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x557b84424b50 0x557b84426ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.654+0000 7f8c43fff640 1 --2- 172.21.15.27:0/4113491605 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x557b84424b50 0x557b84426ff0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5aba10a0704c538e server_cookie=efb8bb79535925a7 in_seq=0 out_seq=0 2024-04-25T00:17:32.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.655+0000 7f8c237fe640 10 client.25768 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:17:32.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.658+0000 7f8c237fe640 1 -- 172.21.15.27:0/4113491605 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557b8437bbb0 con 0x557b84424b50 2024-04-25T00:17:32.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.658+0000 7f8c237fe640 10 client.25768 handle_command_reply: tid=0 2024-04-25T00:17:32.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.658+0000 7f8c4d590740 1 client.25768 shutdown 2024-04-25T00:17:32.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.675+0000 7f8c4d590740 1 -- 172.21.15.27:0/4113491605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84414570 msgr2=0x557b84411860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:32.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.675+0000 7f8c4d590740 1 --2- 172.21.15.27:0/4113491605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84414570 0x557b84411860 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8c380a4090 tx=0x7f8c38004800 comp rx=0 tx=0).stop 2024-04-25T00:17:32.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.679+0000 7f8c4d590740 1 -- 172.21.15.27:0/4113491605 shutdown_connections 2024-04-25T00:17:32.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.679+0000 7f8c4d590740 1 --2- 172.21.15.27:0/4113491605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84414570 0x557b84411860 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.679+0000 7f8c4d590740 1 --2- 172.21.15.27:0/4113491605 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x557b84424b50 0x557b84426ff0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.679+0000 7f8c4d590740 1 --2- 172.21.15.27:0/4113491605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84409ea0 0x557b84411250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.679+0000 7f8c4d590740 1 --2- 172.21.15.27:0/4113491605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84409580 0x557b84413f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.679+0000 7f8c4d590740 1 -- 172.21.15.27:0/4113491605 >> 172.21.15.27:0/4113491605 conn(0x557b844058a0 msgr2=0x557b8440b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:32.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.679+0000 7f8c237fe640 0 client.25768 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:17:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.681+0000 7f8c4d590740 1 -- 172.21.15.27:0/4113491605 shutdown_connections 2024-04-25T00:17:32.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.683+0000 7f8c4d590740 1 -- 172.21.15.27:0/4113491605 wait complete. 2024-04-25T00:17:32.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.684+0000 7f8c4d590740 20 client.25768 trim_cache size 0 max 16384 2024-04-25T00:17:32.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.724+0000 7f8c4d590740 1 Processor -- start 2024-04-25T00:17:32.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.724+0000 7f8c4d590740 1 -- start start 2024-04-25T00:17:32.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.725+0000 7f8c4d590740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84409580 0x557b84413a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.725+0000 7f8c4d590740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84409ea0 0x557b8442c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.725+0000 7f8c4d590740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84414570 0x557b8442ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.726+0000 7f8c4d590740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8c441c67d0 con 0x557b84414570 2024-04-25T00:17:32.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.726+0000 7f8c48e25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84414570 0x557b8442ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.726+0000 7f8c4d590740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8c441c83d0 con 0x557b84409580 2024-04-25T00:17:32.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.726+0000 7f8c4d590740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8c441c6ea0 con 0x557b84409ea0 2024-04-25T00:17:32.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.727+0000 7f8c48e25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84414570 0x557b8442ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59960/0 (socket says 172.21.15.27:59960) 2024-04-25T00:17:32.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.727+0000 7f8c48e25640 1 -- 172.21.15.27:0/2595254434 learned_addr learned my addr 172.21.15.27:0/2595254434 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:32.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.727+0000 7f8c437fe640 1 --2- 172.21.15.27:0/2595254434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84409ea0 0x557b8442c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.728+0000 7f8c43fff640 1 --2- 172.21.15.27:0/2595254434 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84409580 0x557b84413a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.730+0000 7f8c48e25640 1 -- 172.21.15.27:0/2595254434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84409ea0 msgr2=0x557b8442c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:32.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.730+0000 7f8c48e25640 1 --2- 172.21.15.27:0/2595254434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84409ea0 0x557b8442c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.730+0000 7f8c48e25640 1 -- 172.21.15.27:0/2595254434 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84409580 msgr2=0x557b84413a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:32.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.730+0000 7f8c48e25640 1 --2- 172.21.15.27:0/2595254434 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84409580 0x557b84413a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.730+0000 7f8c48e25640 1 -- 172.21.15.27:0/2595254434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b84409cd0 con 0x557b84414570 2024-04-25T00:17:32.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.735+0000 7f8c48e25640 1 --2- 172.21.15.27:0/2595254434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84414570 0x557b8442ce40 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8c38097db0 tx=0x7f8c380049b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef4416db017fe44 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:32.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.738+0000 7f8c29ffb640 1 -- 172.21.15.27:0/2595254434 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c380b3040 con 0x557b84414570 2024-04-25T00:17:32.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.739+0000 7f8c29ffb640 1 -- 172.21.15.27:0/2595254434 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c38005200 con 0x557b84414570 2024-04-25T00:17:32.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.739+0000 7f8c29ffb640 1 -- 172.21.15.27:0/2595254434 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c380a9080 con 0x557b84414570 2024-04-25T00:17:32.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.741+0000 7f8c4d590740 1 -- 172.21.15.27:0/2595254434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84414570 msgr2=0x557b8442ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:32.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.741+0000 7f8c4d590740 1 --2- 172.21.15.27:0/2595254434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84414570 0x557b8442ce40 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8c38097db0 tx=0x7f8c380049b0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.746+0000 7f8c4d590740 1 -- 172.21.15.27:0/2595254434 shutdown_connections 2024-04-25T00:17:32.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.746+0000 7f8c4d590740 1 --2- 172.21.15.27:0/2595254434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84414570 0x557b8442ce40 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.746+0000 7f8c4d590740 1 --2- 172.21.15.27:0/2595254434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84409ea0 0x557b8442c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.746+0000 7f8c4d590740 1 --2- 172.21.15.27:0/2595254434 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84409580 0x557b84413a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.746+0000 7f8c4d590740 1 -- 172.21.15.27:0/2595254434 >> 172.21.15.27:0/2595254434 conn(0x557b844058a0 msgr2=0x557b84411210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.752+0000 7f8c4d590740 1 -- 172.21.15.27:0/2595254434 shutdown_connections 2024-04-25T00:17:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.752+0000 7f8c4d590740 1 -- 172.21.15.27:0/2595254434 wait complete. 2024-04-25T00:17:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.753+0000 7f8c4d590740 1 -- start start 2024-04-25T00:17:32.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.753+0000 7f8c4d590740 1 Processor -- start 2024-04-25T00:17:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.754+0000 7f8c4d590740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84409580 0x557b844195a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.754+0000 7f8c4d590740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84409ea0 0x557b84419ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.754+0000 7f8c4d590740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84414570 0x557b84489170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.754+0000 7f8c4d590740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8c441c7c40 con 0x557b84409580 2024-04-25T00:17:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.754+0000 7f8c4d590740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8c441c7570 con 0x557b84414570 2024-04-25T00:17:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.754+0000 7f8c4d590740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8c441c4860 con 0x557b84409ea0 2024-04-25T00:17:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.754+0000 7f8c4d590740 10 client.0 _ll_register_callbacks cb 0x7f8c3800a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:17:32.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.755+0000 7f8c43fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84409580 0x557b844195a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.755+0000 7f8c43fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84409580 0x557b844195a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59976/0 (socket says 172.21.15.27:59976) 2024-04-25T00:17:32.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.755+0000 7f8c43fff640 1 -- 172.21.15.27:0/1103468645 learned_addr learned my addr 172.21.15.27:0/1103468645 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.757+0000 7f8c43fff640 1 -- 172.21.15.27:0/1103468645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84409ea0 msgr2=0x557b84419ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.757+0000 7f8c43fff640 1 --2- 172.21.15.27:0/1103468645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84409ea0 0x557b84419ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.757+0000 7f8c43fff640 1 -- 172.21.15.27:0/1103468645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84414570 msgr2=0x557b84489170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.757+0000 7f8c43fff640 1 --2- 172.21.15.27:0/1103468645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84414570 0x557b84489170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.757+0000 7f8c43fff640 1 -- 172.21.15.27:0/1103468645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b8414c7d0 con 0x557b84409580 2024-04-25T00:17:32.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.759+0000 7f8c43fff640 1 --2- 172.21.15.27:0/1103468645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84409580 0x557b844195a0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8c340092a0 tx=0x7f8c34009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9133ef5e8c82ed4d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:32.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.759+0000 7f8c237fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:17:32.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.766+0000 7f8c237fe640 1 -- 172.21.15.27:0/1103468645 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c34035050 con 0x557b84409580 2024-04-25T00:17:32.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.767+0000 7f8c237fe640 1 -- 172.21.15.27:0/1103468645 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8c34003ac0 con 0x557b84409580 2024-04-25T00:17:32.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.767+0000 7f8c4d590740 1 -- 172.21.15.27:0/1103468645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c38005200 con 0x557b84409580 2024-04-25T00:17:32.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.767+0000 7f8c237fe640 1 -- 172.21.15.27:0/1103468645 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8c34003cc0 con 0x557b84409580 2024-04-25T00:17:32.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.768+0000 7f8c4d590740 10 client.16239 fetch_fsmap learned FSMap version 30 2024-04-25T00:17:32.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.768+0000 7f8c4d590740 1 -- 172.21.15.27:0/1103468645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557b8448a600 con 0x557b84409580 2024-04-25T00:17:32.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.769+0000 7f8c237fe640 1 -- 172.21.15.27:0/1103468645 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c38005200 con 0x557b84409580 2024-04-25T00:17:32.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.769+0000 7f8c237fe640 1 -- 172.21.15.27:0/1103468645 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f8c3403b6f0 con 0x557b84409580 2024-04-25T00:17:32.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.769+0000 7f8c4d590740 10 client.16239 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:17:32.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.769+0000 7f8c4d590740 10 client.16239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:17:32.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.769+0000 7f8c4d590740 20 client.16239 populate_metadata read hostname 'smithi027' 2024-04-25T00:17:32.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.769+0000 7f8c4d590740 1 --2- 172.21.15.27:0/1103468645 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x557b8448add0 0x557b8448d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:32.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.769+0000 7f8c4d590740 4 client.16239 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:17:32.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.769+0000 7f8c4d590740 1 -- 172.21.15.27:0/1103468645 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557b84375f00 con 0x557b8448add0 2024-04-25T00:17:32.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.772+0000 7f8c437fe640 1 --2- 172.21.15.27:0/1103468645 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x557b8448add0 0x557b8448d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:32.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.776+0000 7f8c437fe640 1 --2- 172.21.15.27:0/1103468645 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x557b8448add0 0x557b8448d250 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f04f8cbe2c0a007 server_cookie=6e7bcd16c9bdc739 in_seq=0 out_seq=0 2024-04-25T00:17:32.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.776+0000 7f8c237fe640 10 client.16239 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:17:32.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.777+0000 7f8c237fe640 1 -- 172.21.15.27:0/1103468645 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557b84375f00 con 0x557b8448add0 2024-04-25T00:17:32.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.777+0000 7f8c237fe640 10 client.16239 handle_command_reply: tid=0 2024-04-25T00:17:32.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.777+0000 7f8c4d590740 1 client.16239 shutdown 2024-04-25T00:17:32.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.777+0000 7f8c4d590740 1 -- 172.21.15.27:0/1103468645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84409580 msgr2=0x557b844195a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:32.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.777+0000 7f8c4d590740 1 --2- 172.21.15.27:0/1103468645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84409580 0x557b844195a0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8c340092a0 tx=0x7f8c34009570 comp rx=0 tx=0).stop 2024-04-25T00:17:32.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.777+0000 7f8c4d590740 1 -- 172.21.15.27:0/1103468645 shutdown_connections 2024-04-25T00:17:32.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.777+0000 7f8c4d590740 1 --2- 172.21.15.27:0/1103468645 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x557b8448add0 0x557b8448d250 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.777+0000 7f8c4d590740 1 --2- 172.21.15.27:0/1103468645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557b84414570 0x557b84489170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.777+0000 7f8c4d590740 1 --2- 172.21.15.27:0/1103468645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b84409ea0 0x557b84419ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.777+0000 7f8c4d590740 1 --2- 172.21.15.27:0/1103468645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b84409580 0x557b844195a0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.777+0000 7f8c4d590740 1 -- 172.21.15.27:0/1103468645 >> 172.21.15.27:0/1103468645 conn(0x557b844058a0 msgr2=0x557b844042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:32.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.777+0000 7f8c237fe640 0 client.16239 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:17:32.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.780+0000 7f8c4d590740 1 -- 172.21.15.27:0/1103468645 shutdown_connections 2024-04-25T00:17:32.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.783+0000 7f8c4d590740 1 -- 172.21.15.27:0/1103468645 wait complete. 2024-04-25T00:17:32.786 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:17:32.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.783+0000 7f8c4d590740 20 client.16239 trim_cache size 0 max 16384 2024-04-25T00:17:32.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.785+0000 7f8c4a888640 1 -- 172.21.15.27:0/3434432808 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8c2c0a2f00 msgr2=0x7f8c2c0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:32.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.785+0000 7f8c4a888640 1 --2- 172.21.15.27:0/3434432808 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8c2c0a2f00 0x7f8c2c0a53c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8c34009230 tx=0x7f8c34005fd0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.785+0000 7f8c4a888640 1 -- 172.21.15.27:0/3434432808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c44111cb0 msgr2=0x7f8c44112160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:32.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.785+0000 7f8c4a888640 1 --2- 172.21.15.27:0/3434432808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c44111cb0 0x7f8c44112160 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800c970 tx=0x7f8c3800ce30 comp rx=0 tx=0).stop 2024-04-25T00:17:32.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.786+0000 7f8c4a888640 1 -- 172.21.15.27:0/3434432808 shutdown_connections 2024-04-25T00:17:32.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.786+0000 7f8c4a888640 1 --2- 172.21.15.27:0/3434432808 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8c2c0a2f00 0x7f8c2c0a53c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.786+0000 7f8c4a888640 1 --2- 172.21.15.27:0/3434432808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c44111cb0 0x7f8c44112160 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.786+0000 7f8c4a888640 1 --2- 172.21.15.27:0/3434432808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c4411af20 0x7f8c44111770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.786+0000 7f8c4a888640 1 --2- 172.21.15.27:0/3434432808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c4410dfc0 0x7f8c441186f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:32.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.786+0000 7f8c4a888640 1 -- 172.21.15.27:0/3434432808 >> 172.21.15.27:0/3434432808 conn(0x7f8c4406c850 msgr2=0x7f8c44119ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:32.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.786+0000 7f8c4a888640 1 -- 172.21.15.27:0/3434432808 shutdown_connections 2024-04-25T00:17:32.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:32.786+0000 7f8c4a888640 1 -- 172.21.15.27:0/3434432808 wait complete. 2024-04-25T00:17:32.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:32 smithi027 ceph-mon[29208]: pgmap v56: 129 pgs: 129 active+clean; 8.9 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 756 MiB/s wr, 344 op/s 2024-04-25T00:17:33.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:32 smithi139 ceph-mon[32833]: pgmap v56: 129 pgs: 129 active+clean; 8.9 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 756 MiB/s wr, 344 op/s 2024-04-25T00:17:33.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:32 smithi153 ceph-mon[32896]: pgmap v56: 129 pgs: 129 active+clean; 8.9 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 756 MiB/s wr, 344 op/s 2024-04-25T00:17:33.833 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T00:17:33.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:17:33.834 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T00:17:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.053+0000 7f2527fff640 1 -- 172.21.15.27:0/71645711 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c0319b0 con 0x7f25301005a0 2024-04-25T00:17:34.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.053+0000 7f2537777640 1 -- 172.21.15.27:0/71645711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25301005a0 msgr2=0x7f2530100980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:34.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.053+0000 7f2537777640 1 --2- 172.21.15.27:0/71645711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25301005a0 0x7f2530100980 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f252c009900 tx=0x7f252c02a410 comp rx=0 tx=0).stop 2024-04-25T00:17:34.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.053+0000 7f2537777640 1 -- 172.21.15.27:0/71645711 shutdown_connections 2024-04-25T00:17:34.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.053+0000 7f2537777640 1 --2- 172.21.15.27:0/71645711 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f253010e150 0x7f2530110540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.053+0000 7f2537777640 1 --2- 172.21.15.27:0/71645711 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2530100ec0 0x7f253010dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.053+0000 7f2537777640 1 --2- 172.21.15.27:0/71645711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25301005a0 0x7f2530100980 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.053+0000 7f2537777640 1 -- 172.21.15.27:0/71645711 >> 172.21.15.27:0/71645711 conn(0x7f2530077770 msgr2=0x7f2530077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.054+0000 7f2537777640 1 -- 172.21.15.27:0/71645711 shutdown_connections 2024-04-25T00:17:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.054+0000 7f2537777640 1 -- 172.21.15.27:0/71645711 wait complete. 2024-04-25T00:17:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.054+0000 7f2537777640 1 Processor -- start 2024-04-25T00:17:34.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.055+0000 7f2537777640 1 -- start start 2024-04-25T00:17:34.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.055+0000 7f2537777640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25301005a0 0x7f253019a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.055+0000 7f2537777640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2530100ec0 0x7f253019a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.055+0000 7f2537777640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f253010e150 0x7f253019ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.055+0000 7f2537777640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2530113e60 con 0x7f2530100ec0 2024-04-25T00:17:34.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.055+0000 7f2537777640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2530113ce0 con 0x7f253010e150 2024-04-25T00:17:34.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.055+0000 7f2537777640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2530113fe0 con 0x7f25301005a0 2024-04-25T00:17:34.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.055+0000 7f2534d12640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2530100ec0 0x7f253019a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.055+0000 7f2534d12640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2530100ec0 0x7f253019a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59992/0 (socket says 172.21.15.27:59992) 2024-04-25T00:17:34.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.055+0000 7f2534d12640 1 -- 172.21.15.27:0/4221035297 learned_addr learned my addr 172.21.15.27:0/4221035297 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:34.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.056+0000 7f2534d12640 1 -- 172.21.15.27:0/4221035297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25301005a0 msgr2=0x7f253019a420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:17:34.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.056+0000 7f2534d12640 1 --2- 172.21.15.27:0/4221035297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25301005a0 0x7f253019a420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.056+0000 7f2534d12640 1 -- 172.21.15.27:0/4221035297 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f253010e150 msgr2=0x7f253019ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:34.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.056+0000 7f2534d12640 1 --2- 172.21.15.27:0/4221035297 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f253010e150 0x7f253019ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.056+0000 7f2534d12640 1 -- 172.21.15.27:0/4221035297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253019f3c0 con 0x7f2530100ec0 2024-04-25T00:17:34.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.056+0000 7f2534d12640 1 --2- 172.21.15.27:0/4221035297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2530100ec0 0x7f253019a960 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f252000b4a0 tx=0x7f252000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:34.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.056+0000 7f25267fc640 1 -- 172.21.15.27:0/4221035297 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520014070 con 0x7f2530100ec0 2024-04-25T00:17:34.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.056+0000 7f2537777640 1 -- 172.21.15.27:0/4221035297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253019f6b0 con 0x7f2530100ec0 2024-04-25T00:17:34.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.057+0000 7f2537777640 1 -- 172.21.15.27:0/4221035297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25301a6f40 con 0x7f2530100ec0 2024-04-25T00:17:34.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.057+0000 7f25267fc640 1 -- 172.21.15.27:0/4221035297 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2520010af0 con 0x7f2530100ec0 2024-04-25T00:17:34.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.057+0000 7f25267fc640 1 -- 172.21.15.27:0/4221035297 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520017d10 con 0x7f2530100ec0 2024-04-25T00:17:34.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.061+0000 7f25267fc640 1 -- 172.21.15.27:0/4221035297 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2520017070 con 0x7f2530100ec0 2024-04-25T00:17:34.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.061+0000 7f253a563740 1 Processor -- start 2024-04-25T00:17:34.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.061+0000 7f253a563740 1 -- start start 2024-04-25T00:17:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.061+0000 7f253a563740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.061+0000 7f253a563740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593d83d0290 0x5593d83d0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.061+0000 7f253a563740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593d83db010 0x5593d83dd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.061+0000 7f253a563740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5593d83342c0 con 0x5593d83d1980 2024-04-25T00:17:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.061+0000 7f253a563740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5593d82dc910 con 0x5593d83db010 2024-04-25T00:17:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.061+0000 7f253a563740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5593d8330280 con 0x5593d83d0290 2024-04-25T00:17:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.062+0000 7f2535d14640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.062+0000 7f2535d14640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83cfcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60008/0 (socket says 172.21.15.27:60008) 2024-04-25T00:17:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.062+0000 7f2535d14640 1 -- 172.21.15.27:0/2784511991 learned_addr learned my addr 172.21.15.27:0/2784511991 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.062+0000 7f2535d14640 1 -- 172.21.15.27:0/2784511991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593d83d0290 msgr2=0x5593d83d0710 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:17:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.062+0000 7f2535d14640 1 --2- 172.21.15.27:0/2784511991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593d83d0290 0x5593d83d0710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.062+0000 7f2535d14640 1 -- 172.21.15.27:0/2784511991 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593d83db010 msgr2=0x5593d83dd400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:17:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.062+0000 7f2535d14640 1 --2- 172.21.15.27:0/2784511991 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593d83db010 0x5593d83dd400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.062+0000 7f2535d14640 1 -- 172.21.15.27:0/2784511991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593d82edfb0 con 0x5593d83d1980 2024-04-25T00:17:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.062+0000 7f25267fc640 1 --2- 172.21.15.27:0/4221035297 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f251c0a2d00 0x7f251c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.062+0000 7f2535d14640 1 --2- 172.21.15.27:0/2784511991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83cfcc0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f2528007c90 tx=0x7f252800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609d029dc5bb055f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.062+0000 7f25267fc640 1 -- 172.21.15.27:0/4221035297 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f25200ce920 con 0x7f2530100ec0 2024-04-25T00:17:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.062+0000 7f251b7fe640 1 -- 172.21.15.27:0/2784511991 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2528015d50 con 0x5593d83d1980 2024-04-25T00:17:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.063+0000 7f2535513640 1 --2- 172.21.15.27:0/4221035297 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f251c0a2d00 0x7f251c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.063+0000 7f251b7fe640 1 -- 172.21.15.27:0/2784511991 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f252801d8d0 con 0x5593d83d1980 2024-04-25T00:17:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.063+0000 7f251b7fe640 1 -- 172.21.15.27:0/2784511991 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252801c7e0 con 0x5593d83d1980 2024-04-25T00:17:34.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.064+0000 7f253a563740 1 -- 172.21.15.27:0/2784511991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 msgr2=0x5593d83cfcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:34.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.064+0000 7f253a563740 1 --2- 172.21.15.27:0/2784511991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83cfcc0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f2528007c90 tx=0x7f252800c6a0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.064+0000 7f253a563740 1 -- 172.21.15.27:0/2784511991 shutdown_connections 2024-04-25T00:17:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.064+0000 7f253a563740 1 --2- 172.21.15.27:0/2784511991 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593d83db010 0x5593d83dd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.064+0000 7f253a563740 1 --2- 172.21.15.27:0/2784511991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593d83d0290 0x5593d83d0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.064+0000 7f253a563740 1 --2- 172.21.15.27:0/2784511991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83cfcc0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.064+0000 7f253a563740 1 -- 172.21.15.27:0/2784511991 >> 172.21.15.27:0/2784511991 conn(0x5593d8398950 msgr2=0x5593d83cc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.064+0000 7f253a563740 1 -- 172.21.15.27:0/2784511991 shutdown_connections 2024-04-25T00:17:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.064+0000 7f253a563740 1 -- 172.21.15.27:0/2784511991 wait complete. 2024-04-25T00:17:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.064+0000 7f2535513640 1 --2- 172.21.15.27:0/4221035297 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f251c0a2d00 0x7f251c0a51c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f252c009230 tx=0x7f252c005fd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.065+0000 7f253a563740 1 -- start start 2024-04-25T00:17:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.065+0000 7f253a563740 1 Processor -- start 2024-04-25T00:17:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.065+0000 7f253a563740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d0290 0x5593d83d7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.065+0000 7f253a563740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593d83d1980 0x5593d83d8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.065+0000 7f253a563740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593d83db010 0x5593d83d89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.065+0000 7f253a563740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5593d83ddea0 con 0x5593d83d0290 2024-04-25T00:17:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.065+0000 7f253a563740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5593d82efe90 con 0x5593d83d1980 2024-04-25T00:17:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.065+0000 7f253a563740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5593d83de020 con 0x5593d83db010 2024-04-25T00:17:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.065+0000 7f2535513640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d0290 0x5593d83d7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.066+0000 7f253a563740 10 client.0 _ll_register_callbacks cb 0x5593d832b970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:17:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.066+0000 7f2535513640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d0290 0x5593d83d7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60020/0 (socket says 172.21.15.27:60020) 2024-04-25T00:17:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.066+0000 7f2535513640 1 -- 172.21.15.27:0/3445160388 learned_addr learned my addr 172.21.15.27:0/3445160388 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:34.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.066+0000 7f2535513640 1 -- 172.21.15.27:0/3445160388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593d83db010 msgr2=0x5593d83d89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:34.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.066+0000 7f2535513640 1 --2- 172.21.15.27:0/3445160388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593d83db010 0x5593d83d89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.066+0000 7f2535513640 1 -- 172.21.15.27:0/3445160388 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593d83d1980 msgr2=0x5593d83d8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:34.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.066+0000 7f2535513640 1 --2- 172.21.15.27:0/3445160388 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593d83d1980 0x5593d83d8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.066+0000 7f2535513640 1 -- 172.21.15.27:0/3445160388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593d81107e0 con 0x5593d83d0290 2024-04-25T00:17:34.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.066+0000 7f2535513640 1 --2- 172.21.15.27:0/3445160388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d0290 0x5593d83d7f30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f252c0045a0 tx=0x7f252c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa2084a1d147b6e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:34.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.066+0000 7f2518ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:17:34.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.066+0000 7f2518ff9640 1 -- 172.21.15.27:0/3445160388 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c04b020 con 0x5593d83d0290 2024-04-25T00:17:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.066+0000 7f2518ff9640 1 -- 172.21.15.27:0/3445160388 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f252c003be0 con 0x5593d83d0290 2024-04-25T00:17:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.066+0000 7f253a563740 1 -- 172.21.15.27:0/3445160388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f252801d8d0 con 0x5593d83d0290 2024-04-25T00:17:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.067+0000 7f2518ff9640 1 -- 172.21.15.27:0/3445160388 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f252c035040 con 0x5593d83d0290 2024-04-25T00:17:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.067+0000 7f253a563740 10 client.16251 fetch_fsmap learned FSMap version 30 2024-04-25T00:17:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.067+0000 7f253a563740 1 -- 172.21.15.27:0/3445160388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5593d8170f70 con 0x5593d83d0290 2024-04-25T00:17:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.067+0000 7f2518ff9640 1 -- 172.21.15.27:0/3445160388 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f252801d8d0 con 0x5593d83d0290 2024-04-25T00:17:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.067+0000 7f2518ff9640 1 -- 172.21.15.27:0/3445160388 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f252c0474d0 con 0x5593d83d0290 2024-04-25T00:17:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.067+0000 7f253a563740 10 client.16251 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:17:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.067+0000 7f253a563740 10 client.16251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:17:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.067+0000 7f253a563740 20 client.16251 populate_metadata read hostname 'smithi027' 2024-04-25T00:17:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.067+0000 7f253a563740 1 --2- 172.21.15.27:0/3445160388 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5593d83eb2a0 0x5593d83ed760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.067+0000 7f253a563740 4 client.16251 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:17:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.067+0000 7f253a563740 1 -- 172.21.15.27:0/3445160388 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593d83085a0 con 0x5593d83eb2a0 2024-04-25T00:17:34.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.068+0000 7f2534d12640 1 --2- 172.21.15.27:0/3445160388 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5593d83eb2a0 0x5593d83ed760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.070+0000 7f2534d12640 1 --2- 172.21.15.27:0/3445160388 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5593d83eb2a0 0x5593d83ed760 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=681b4a240ed6527 server_cookie=d573e09b7f71eae3 in_seq=0 out_seq=0 2024-04-25T00:17:34.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.070+0000 7f2518ff9640 10 client.16251 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:17:34.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.071+0000 7f2518ff9640 1 -- 172.21.15.27:0/3445160388 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5593d83085a0 con 0x5593d83eb2a0 2024-04-25T00:17:34.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.071+0000 7f2518ff9640 10 client.16251 handle_command_reply: tid=0 2024-04-25T00:17:34.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.071+0000 7f253a563740 1 client.16251 shutdown 2024-04-25T00:17:34.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.072+0000 7f253a563740 1 -- 172.21.15.27:0/3445160388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d0290 msgr2=0x5593d83d7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:34.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.072+0000 7f253a563740 1 --2- 172.21.15.27:0/3445160388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d0290 0x5593d83d7f30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f252c0045a0 tx=0x7f252c02a410 comp rx=0 tx=0).stop 2024-04-25T00:17:34.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.072+0000 7f253a563740 1 -- 172.21.15.27:0/3445160388 shutdown_connections 2024-04-25T00:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.072+0000 7f253a563740 1 --2- 172.21.15.27:0/3445160388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593d83db010 0x5593d83d89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.072+0000 7f253a563740 1 --2- 172.21.15.27:0/3445160388 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5593d83eb2a0 0x5593d83ed760 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.072+0000 7f253a563740 1 --2- 172.21.15.27:0/3445160388 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593d83d1980 0x5593d83d8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.072+0000 7f253a563740 1 --2- 172.21.15.27:0/3445160388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d0290 0x5593d83d7f30 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.072+0000 7f2518ff9640 0 client.16251 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.072+0000 7f253a563740 1 -- 172.21.15.27:0/3445160388 >> 172.21.15.27:0/3445160388 conn(0x5593d8398950 msgr2=0x5593d83d2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.072+0000 7f253a563740 1 -- 172.21.15.27:0/3445160388 shutdown_connections 2024-04-25T00:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.072+0000 7f253a563740 1 -- 172.21.15.27:0/3445160388 wait complete. 2024-04-25T00:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.072+0000 7f253a563740 20 client.16251 trim_cache size 0 max 16384 2024-04-25T00:17:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.097+0000 7f253a563740 1 Processor -- start 2024-04-25T00:17:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.097+0000 7f253a563740 1 -- start start 2024-04-25T00:17:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.097+0000 7f253a563740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593d83d0290 0x5593d83e6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.097+0000 7f253a563740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.097+0000 7f253a563740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593d83db010 0x5593d83f3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.097+0000 7f253a563740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f251400fe80 con 0x5593d83d1980 2024-04-25T00:17:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.097+0000 7f253a563740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2514011a80 con 0x5593d83d0290 2024-04-25T00:17:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.097+0000 7f253a563740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2514010550 con 0x5593d83db010 2024-04-25T00:17:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.098+0000 7f2534d12640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.098+0000 7f2534d12640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83cb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60026/0 (socket says 172.21.15.27:60026) 2024-04-25T00:17:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.098+0000 7f2534d12640 1 -- 172.21.15.27:0/3075738744 learned_addr learned my addr 172.21.15.27:0/3075738744 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.098+0000 7f2534d12640 1 -- 172.21.15.27:0/3075738744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593d83db010 msgr2=0x5593d83f3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.098+0000 7f2534d12640 1 --2- 172.21.15.27:0/3075738744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593d83db010 0x5593d83f3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.098+0000 7f2534d12640 1 -- 172.21.15.27:0/3075738744 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593d83d0290 msgr2=0x5593d83e6e30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:17:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.098+0000 7f2534d12640 1 --2- 172.21.15.27:0/3075738744 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593d83d0290 0x5593d83e6e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.098+0000 7f2534d12640 1 -- 172.21.15.27:0/3075738744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593d8008170 con 0x5593d83d1980 2024-04-25T00:17:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.098+0000 7f2534d12640 1 --2- 172.21.15.27:0/3075738744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83cb7d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f25200975b0 tx=0x7f25200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6338aa5aadef25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.098+0000 7f251affd640 1 -- 172.21.15.27:0/3075738744 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252001e040 con 0x5593d83d1980 2024-04-25T00:17:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.098+0000 7f251affd640 1 -- 172.21.15.27:0/3075738744 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25200b1070 con 0x5593d83d1980 2024-04-25T00:17:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.098+0000 7f251affd640 1 -- 172.21.15.27:0/3075738744 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252009a290 con 0x5593d83d1980 2024-04-25T00:17:34.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.099+0000 7f253a563740 1 -- 172.21.15.27:0/3075738744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 msgr2=0x5593d83cb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:34.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.099+0000 7f253a563740 1 --2- 172.21.15.27:0/3075738744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83cb7d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f25200975b0 tx=0x7f25200046c0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.099+0000 7f253a563740 1 -- 172.21.15.27:0/3075738744 shutdown_connections 2024-04-25T00:17:34.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.099+0000 7f253a563740 1 --2- 172.21.15.27:0/3075738744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593d83db010 0x5593d83f3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.099+0000 7f253a563740 1 --2- 172.21.15.27:0/3075738744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83cb7d0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.099+0000 7f253a563740 1 --2- 172.21.15.27:0/3075738744 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593d83d0290 0x5593d83e6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.099+0000 7f253a563740 1 -- 172.21.15.27:0/3075738744 >> 172.21.15.27:0/3075738744 conn(0x5593d8398950 msgr2=0x5593d83cb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:34.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.099+0000 7f253a563740 1 -- 172.21.15.27:0/3075738744 shutdown_connections 2024-04-25T00:17:34.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.099+0000 7f253a563740 1 -- 172.21.15.27:0/3075738744 wait complete. 2024-04-25T00:17:34.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.100+0000 7f253a563740 1 -- start start 2024-04-25T00:17:34.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.100+0000 7f253a563740 1 Processor -- start 2024-04-25T00:17:34.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.100+0000 7f253a563740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593d83d0290 0x5593d83df8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.100+0000 7f253a563740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83dfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.100+0000 7f253a563740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593d83db010 0x5593d84498a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.100+0000 7f253a563740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f25140112f0 con 0x5593d83d1980 2024-04-25T00:17:34.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.100+0000 7f253a563740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2514010c20 con 0x5593d83db010 2024-04-25T00:17:34.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.100+0000 7f253a563740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f251400de40 con 0x5593d83d0290 2024-04-25T00:17:34.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.100+0000 7f253a563740 10 client.0 _ll_register_callbacks cb 0x7f252c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:17:34.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.100+0000 7f2534d12640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83dfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.100+0000 7f2534d12640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83dfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60040/0 (socket says 172.21.15.27:60040) 2024-04-25T00:17:34.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.100+0000 7f2534d12640 1 -- 172.21.15.27:0/293466919 learned_addr learned my addr 172.21.15.27:0/293466919 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.101+0000 7f2534d12640 1 -- 172.21.15.27:0/293466919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593d83d0290 msgr2=0x5593d83df8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.101+0000 7f2534d12640 1 --2- 172.21.15.27:0/293466919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593d83d0290 0x5593d83df8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.101+0000 7f2534d12640 1 -- 172.21.15.27:0/293466919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593d83db010 msgr2=0x5593d84498a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:17:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.101+0000 7f2534d12640 1 --2- 172.21.15.27:0/293466919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593d83db010 0x5593d84498a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.101+0000 7f2534d12640 1 -- 172.21.15.27:0/293466919 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593d8449f60 con 0x5593d83d1980 2024-04-25T00:17:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.101+0000 7f2534d12640 1 --2- 172.21.15.27:0/293466919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83dfe20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f252009a8e0 tx=0x7f25200974a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d01132a8f494b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.101+0000 7f2518ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:17:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.101+0000 7f2518ff9640 1 -- 172.21.15.27:0/293466919 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252009b2e0 con 0x5593d83d1980 2024-04-25T00:17:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.101+0000 7f2518ff9640 1 -- 172.21.15.27:0/293466919 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f25200b9370 con 0x5593d83d1980 2024-04-25T00:17:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.101+0000 7f253a563740 1 -- 172.21.15.27:0/293466919 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25200b1070 con 0x5593d83d1980 2024-04-25T00:17:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.101+0000 7f2518ff9640 1 -- 172.21.15.27:0/293466919 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f25200b9570 con 0x5593d83d1980 2024-04-25T00:17:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.101+0000 7f253a563740 10 client.16257 fetch_fsmap learned FSMap version 30 2024-04-25T00:17:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.101+0000 7f253a563740 1 -- 172.21.15.27:0/293466919 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5593d8452440 con 0x5593d83d1980 2024-04-25T00:17:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.102+0000 7f2518ff9640 1 -- 172.21.15.27:0/293466919 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25200b1070 con 0x5593d83d1980 2024-04-25T00:17:34.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.102+0000 7f2518ff9640 1 -- 172.21.15.27:0/293466919 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f2520096930 con 0x5593d83d1980 2024-04-25T00:17:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.102+0000 7f253a563740 10 client.16257 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:17:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.102+0000 7f253a563740 10 client.16257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:17:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.102+0000 7f253a563740 20 client.16257 populate_metadata read hostname 'smithi027' 2024-04-25T00:17:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.102+0000 7f253a563740 1 --2- 172.21.15.27:0/293466919 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5593d8452b40 0x5593d8454f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.102+0000 7f253a563740 4 client.16257 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T00:17:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.102+0000 7f253a563740 1 -- 172.21.15.27:0/293466919 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5593d8341800 con 0x5593d8452b40 2024-04-25T00:17:34.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.103+0000 7f2535513640 1 --2- 172.21.15.27:0/293466919 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5593d8452b40 0x5593d8454f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.104+0000 7f2518ff9640 10 client.16257 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:17:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.104+0000 7f2535513640 1 --2- 172.21.15.27:0/293466919 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5593d8452b40 0x5593d8454f60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=718aa5fcc5062ed8 server_cookie=ae29f2694f2eced6 in_seq=0 out_seq=0 2024-04-25T00:17:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.105+0000 7f2518ff9640 1 -- 172.21.15.27:0/293466919 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5593d8341800 con 0x5593d8452b40 2024-04-25T00:17:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.105+0000 7f2518ff9640 10 client.16257 handle_command_reply: tid=0 2024-04-25T00:17:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.105+0000 7f253a563740 1 client.16257 shutdown 2024-04-25T00:17:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.105+0000 7f253a563740 1 -- 172.21.15.27:0/293466919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 msgr2=0x5593d83dfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:34.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.105+0000 7f253a563740 1 --2- 172.21.15.27:0/293466919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83dfe20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f252009a8e0 tx=0x7f25200974a0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.105+0000 7f253a563740 1 -- 172.21.15.27:0/293466919 shutdown_connections 2024-04-25T00:17:34.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.105+0000 7f253a563740 1 --2- 172.21.15.27:0/293466919 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5593d8452b40 0x5593d8454f60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.105+0000 7f253a563740 1 --2- 172.21.15.27:0/293466919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5593d83db010 0x5593d84498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.105+0000 7f253a563740 1 --2- 172.21.15.27:0/293466919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593d83d1980 0x5593d83dfe20 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.105+0000 7f253a563740 1 --2- 172.21.15.27:0/293466919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5593d83d0290 0x5593d83df8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.105+0000 7f253a563740 1 -- 172.21.15.27:0/293466919 >> 172.21.15.27:0/293466919 conn(0x5593d8398950 msgr2=0x5593d83cb950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:34.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.105+0000 7f2518ff9640 0 client.16257 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:17:34.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.106+0000 7f253a563740 1 -- 172.21.15.27:0/293466919 shutdown_connections 2024-04-25T00:17:34.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.106+0000 7f253a563740 1 -- 172.21.15.27:0/293466919 wait complete. 2024-04-25T00:17:34.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.106+0000 7f253a563740 20 client.16257 trim_cache size 0 max 16384 2024-04-25T00:17:34.109 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:17:34.110 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T00:17:34.110 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "4b378baa-7184-4771-a5eb-086427f3134f", 2024-04-25T00:17:34.110 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T00:17:34.110 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:17:34.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.107+0000 7f2537777640 1 -- 172.21.15.27:0/4221035297 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f251c0a2d00 msgr2=0x7f251c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:34.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.107+0000 7f2537777640 1 --2- 172.21.15.27:0/4221035297 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f251c0a2d00 0x7f251c0a51c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f252c009230 tx=0x7f252c005fd0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.107+0000 7f2537777640 1 -- 172.21.15.27:0/4221035297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2530100ec0 msgr2=0x7f253019a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:34.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.107+0000 7f2537777640 1 --2- 172.21.15.27:0/4221035297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2530100ec0 0x7f253019a960 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f252000b4a0 tx=0x7f252000b960 comp rx=0 tx=0).stop 2024-04-25T00:17:34.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.107+0000 7f2537777640 1 -- 172.21.15.27:0/4221035297 shutdown_connections 2024-04-25T00:17:34.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.107+0000 7f2537777640 1 --2- 172.21.15.27:0/4221035297 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f251c0a2d00 0x7f251c0a51c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.107+0000 7f2537777640 1 --2- 172.21.15.27:0/4221035297 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f253010e150 0x7f253019ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.107+0000 7f2537777640 1 --2- 172.21.15.27:0/4221035297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2530100ec0 0x7f253019a960 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.107+0000 7f2537777640 1 --2- 172.21.15.27:0/4221035297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25301005a0 0x7f253019a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.107+0000 7f2537777640 1 -- 172.21.15.27:0/4221035297 >> 172.21.15.27:0/4221035297 conn(0x7f2530077770 msgr2=0x7f253010f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:34.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.107+0000 7f2537777640 1 -- 172.21.15.27:0/4221035297 shutdown_connections 2024-04-25T00:17:34.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.107+0000 7f2537777640 1 -- 172.21.15.27:0/4221035297 wait complete. 2024-04-25T00:17:34.126 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:17:34.126 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:17:34.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.264+0000 7f1bf1142640 1 -- 172.21.15.27:0/4163543161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bec101710 msgr2=0x7f1bec101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:34.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.264+0000 7f1bf1142640 1 --2- 172.21.15.27:0/4163543161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bec101710 0x7f1bec101b90 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4009930 tx=0x7f1bd402a420 comp rx=0 tx=0).stop 2024-04-25T00:17:34.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.264+0000 7f1bf1142640 1 -- 172.21.15.27:0/4163543161 shutdown_connections 2024-04-25T00:17:34.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.264+0000 7f1bf1142640 1 --2- 172.21.15.27:0/4163543161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bec1020d0 0x7f1bec10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.264+0000 7f1bf1142640 1 --2- 172.21.15.27:0/4163543161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bec101710 0x7f1bec101b90 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.264+0000 7f1bf1142640 1 --2- 172.21.15.27:0/4163543161 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bec107720 0x7f1bec107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.264+0000 7f1bf1142640 1 -- 172.21.15.27:0/4163543161 >> 172.21.15.27:0/4163543161 conn(0x7f1bec078b90 msgr2=0x7f1bec078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:34.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.264+0000 7f1bf1142640 1 -- 172.21.15.27:0/4163543161 shutdown_connections 2024-04-25T00:17:34.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.264+0000 7f1bf1142640 1 -- 172.21.15.27:0/4163543161 wait complete. 2024-04-25T00:17:34.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.265+0000 7f1bf1142640 1 Processor -- start 2024-04-25T00:17:34.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.265+0000 7f1bf1142640 1 -- start start 2024-04-25T00:17:34.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.266+0000 7f1bf1142640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bec101710 0x7f1bec1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.266+0000 7f1bf1142640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bec1020d0 0x7f1bec1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.266+0000 7f1bf1142640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bec107720 0x7f1bec19ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.266+0000 7f1bf1142640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec111bb0 con 0x7f1bec101710 2024-04-25T00:17:34.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.266+0000 7f1bf1142640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec111a30 con 0x7f1bec107720 2024-04-25T00:17:34.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.266+0000 7f1bf1142640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec111d30 con 0x7f1bec1020d0 2024-04-25T00:17:34.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.266+0000 7f1bead9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bec101710 0x7f1bec1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.266+0000 7f1bead9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bec101710 0x7f1bec1981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60054/0 (socket says 172.21.15.27:60054) 2024-04-25T00:17:34.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.266+0000 7f1bead9d640 1 -- 172.21.15.27:0/287760203 learned_addr learned my addr 172.21.15.27:0/287760203 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:34.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.266+0000 7f1be259c640 1 --2- 172.21.15.27:0/287760203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bec1020d0 0x7f1bec1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.267+0000 7f1bead9d640 1 -- 172.21.15.27:0/287760203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bec1020d0 msgr2=0x7f1bec1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:34.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.267+0000 7f1bead9d640 1 --2- 172.21.15.27:0/287760203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bec1020d0 0x7f1bec1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.267+0000 7f1beb59e640 1 --2- 172.21.15.27:0/287760203 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bec107720 0x7f1bec19ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.267+0000 7f1bead9d640 1 -- 172.21.15.27:0/287760203 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bec107720 msgr2=0x7f1bec19ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:34.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.267+0000 7f1bead9d640 1 --2- 172.21.15.27:0/287760203 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bec107720 0x7f1bec19ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.267+0000 7f1bead9d640 1 -- 172.21.15.27:0/287760203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bec19d1a0 con 0x7f1bec101710 2024-04-25T00:17:34.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.267+0000 7f1bead9d640 1 --2- 172.21.15.27:0/287760203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bec101710 0x7f1bec1981b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00b4a0 tx=0x7f1bdc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:34.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.267+0000 7f1be8d99640 1 -- 172.21.15.27:0/287760203 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc014070 con 0x7f1bec101710 2024-04-25T00:17:34.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.267+0000 7f1bf1142640 1 -- 172.21.15.27:0/287760203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bec19d490 con 0x7f1bec101710 2024-04-25T00:17:34.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.267+0000 7f1be8d99640 1 -- 172.21.15.27:0/287760203 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bdc010af0 con 0x7f1bec101710 2024-04-25T00:17:34.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.267+0000 7f1bf1142640 1 -- 172.21.15.27:0/287760203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bec1a4d20 con 0x7f1bec101710 2024-04-25T00:17:34.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.268+0000 7f1be8d99640 1 -- 172.21.15.27:0/287760203 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc01e060 con 0x7f1bec101710 2024-04-25T00:17:34.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.271+0000 7f1bf3e53740 1 Processor -- start 2024-04-25T00:17:34.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.271+0000 7f1bf3e53740 1 -- start start 2024-04-25T00:17:34.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.272+0000 7f1bf3e53740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0f312d630 0x55f0f312da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.272+0000 7f1bf3e53740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f312df50 0x55f0f31380e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.272+0000 7f1bf3e53740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f313aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.273+0000 7f1bf3e53740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f0f30926b0 con 0x55f0f3138620 2024-04-25T00:17:34.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.273+0000 7f1bf3e53740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f0f303a880 con 0x55f0f312df50 2024-04-25T00:17:34.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.273+0000 7f1bf3e53740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f0f308e670 con 0x55f0f312d630 2024-04-25T00:17:34.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.273+0000 7f1be259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f313aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.273+0000 7f1be259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f313aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60058/0 (socket says 172.21.15.27:60058) 2024-04-25T00:17:34.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.273+0000 7f1be259c640 1 -- 172.21.15.27:0/4039887120 learned_addr learned my addr 172.21.15.27:0/4039887120 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:34.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.273+0000 7f1be259c640 1 -- 172.21.15.27:0/4039887120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0f312d630 msgr2=0x55f0f312da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:34.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.273+0000 7f1be259c640 1 --2- 172.21.15.27:0/4039887120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0f312d630 0x55f0f312da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.273+0000 7f1be259c640 1 -- 172.21.15.27:0/4039887120 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f312df50 msgr2=0x55f0f31380e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:17:34.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.273+0000 7f1be259c640 1 --2- 172.21.15.27:0/4039887120 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f312df50 0x55f0f31380e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.273+0000 7f1be259c640 1 -- 172.21.15.27:0/4039887120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0f2e707d0 con 0x55f0f3138620 2024-04-25T00:17:34.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.273+0000 7f1be8d99640 1 -- 172.21.15.27:0/287760203 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1bdc017070 con 0x7f1bec101710 2024-04-25T00:17:34.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.274+0000 7f1be259c640 1 --2- 172.21.15.27:0/4039887120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f313aa10 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4005ab0 tx=0x7f1bd4031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc599b81cc7d47f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:34.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.274+0000 7f1be159a640 1 -- 172.21.15.27:0/4039887120 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4035070 con 0x55f0f3138620 2024-04-25T00:17:34.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.274+0000 7f1be159a640 1 -- 172.21.15.27:0/4039887120 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bd4031a20 con 0x55f0f3138620 2024-04-25T00:17:34.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.274+0000 7f1be159a640 1 -- 172.21.15.27:0/4039887120 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd403eac0 con 0x55f0f3138620 2024-04-25T00:17:34.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.275+0000 7f1be8d99640 1 --2- 172.21.15.27:0/287760203 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1bd80a2d00 0x7f1bd80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.275+0000 7f1beb59e640 1 --2- 172.21.15.27:0/287760203 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1bd80a2d00 0x7f1bd80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.275+0000 7f1bf3e53740 1 -- 172.21.15.27:0/4039887120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 msgr2=0x55f0f313aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:34.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.275+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/4039887120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f313aa10 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4005ab0 tx=0x7f1bd4031630 comp rx=0 tx=0).stop 2024-04-25T00:17:34.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.276+0000 7f1bf3e53740 1 -- 172.21.15.27:0/4039887120 shutdown_connections 2024-04-25T00:17:34.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.276+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/4039887120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f313aa10 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.276+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/4039887120 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f312df50 0x55f0f31380e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.276+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/4039887120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0f312d630 0x55f0f312da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.276+0000 7f1bf3e53740 1 -- 172.21.15.27:0/4039887120 >> 172.21.15.27:0/4039887120 conn(0x55f0f3129950 msgr2=0x55f0f3129d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:34.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.276+0000 7f1be8d99640 1 -- 172.21.15.27:0/287760203 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f1bdc0ceac0 con 0x7f1bec101710 2024-04-25T00:17:34.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.276+0000 7f1bf3e53740 1 -- 172.21.15.27:0/4039887120 shutdown_connections 2024-04-25T00:17:34.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.276+0000 7f1beb59e640 1 --2- 172.21.15.27:0/287760203 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1bd80a2d00 0x7f1bd80a51c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1be4008bf0 tx=0x7f1be4008260 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:34.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.276+0000 7f1bf3e53740 1 -- 172.21.15.27:0/4039887120 wait complete. 2024-04-25T00:17:34.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.276+0000 7f1bf3e53740 1 -- start start 2024-04-25T00:17:34.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.277+0000 7f1bf3e53740 1 Processor -- start 2024-04-25T00:17:34.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.277+0000 7f1bf3e53740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0f312d630 0x55f0f3135580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.277+0000 7f1bf3e53740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f312df50 0x55f0f3135ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.277+0000 7f1bf3e53740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f3136000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.277+0000 7f1bf3e53740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f0f313b3b0 con 0x55f0f3138620 2024-04-25T00:17:34.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.277+0000 7f1bf3e53740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f0f304df50 con 0x55f0f312df50 2024-04-25T00:17:34.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.277+0000 7f1bf3e53740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f0f313b530 con 0x55f0f312d630 2024-04-25T00:17:34.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.277+0000 7f1beb59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f3136000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.277+0000 7f1beb59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f3136000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60064/0 (socket says 172.21.15.27:60064) 2024-04-25T00:17:34.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.277+0000 7f1beb59e640 1 -- 172.21.15.27:0/3446893856 learned_addr learned my addr 172.21.15.27:0/3446893856 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:34.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.277+0000 7f1bf3e53740 10 client.0 _ll_register_callbacks cb 0x55f0f3089000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:17:34.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.278+0000 7f1bead9d640 1 --2- 172.21.15.27:0/3446893856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0f312d630 0x55f0f3135580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.278+0000 7f1beb59e640 1 -- 172.21.15.27:0/3446893856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0f312d630 msgr2=0x55f0f3135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:34.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.278+0000 7f1beb59e640 1 --2- 172.21.15.27:0/3446893856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0f312d630 0x55f0f3135580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.278+0000 7f1beb59e640 1 -- 172.21.15.27:0/3446893856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f312df50 msgr2=0x55f0f3135ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:34.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.278+0000 7f1be259c640 1 --2- 172.21.15.27:0/3446893856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f312df50 0x55f0f3135ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.278+0000 7f1beb59e640 1 --2- 172.21.15.27:0/3446893856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f312df50 0x55f0f3135ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.278+0000 7f1beb59e640 1 -- 172.21.15.27:0/3446893856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0f2da63f0 con 0x55f0f3138620 2024-04-25T00:17:34.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.278+0000 7f1be259c640 1 --2- 172.21.15.27:0/3446893856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f312df50 0x55f0f3135ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:34.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.278+0000 7f1beb59e640 1 --2- 172.21.15.27:0/3446893856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f3136000 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1be4014a50 tx=0x7f1be4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e638c47918f67443 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:34.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.278+0000 7f1bd27fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:17:34.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.279+0000 7f1bd27fc640 1 -- 172.21.15.27:0/3446893856 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be4024060 con 0x55f0f3138620 2024-04-25T00:17:34.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.279+0000 7f1bd27fc640 1 -- 172.21.15.27:0/3446893856 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f1be4012db0 con 0x55f0f3138620 2024-04-25T00:17:34.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.279+0000 7f1bf3e53740 1 -- 172.21.15.27:0/3446893856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bd4031a20 con 0x55f0f3138620 2024-04-25T00:17:34.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.279+0000 7f1bd27fc640 1 -- 172.21.15.27:0/3446893856 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1be400de80 con 0x55f0f3138620 2024-04-25T00:17:34.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.279+0000 7f1bf3e53740 10 client.16269 fetch_fsmap learned FSMap version 30 2024-04-25T00:17:34.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.279+0000 7f1bf3e53740 1 -- 172.21.15.27:0/3446893856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f0f2d660f0 con 0x55f0f3138620 2024-04-25T00:17:34.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.280+0000 7f1bd27fc640 1 -- 172.21.15.27:0/3446893856 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bd4031a20 con 0x55f0f3138620 2024-04-25T00:17:34.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.280+0000 7f1bd27fc640 1 -- 172.21.15.27:0/3446893856 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f1be40136f0 con 0x55f0f3138620 2024-04-25T00:17:34.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.280+0000 7f1bf3e53740 10 client.16269 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:17:34.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.280+0000 7f1bf3e53740 10 client.16269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:17:34.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.280+0000 7f1bf3e53740 20 client.16269 populate_metadata read hostname 'smithi027' 2024-04-25T00:17:34.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.280+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/3446893856 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f0f31488a0 0x55f0f314ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.280+0000 7f1bf3e53740 4 client.16269 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:17:34.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.280+0000 7f1bf3e53740 1 -- 172.21.15.27:0/3446893856 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0f309fbf0 con 0x55f0f31488a0 2024-04-25T00:17:34.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.281+0000 7f1bead9d640 1 --2- 172.21.15.27:0/3446893856 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f0f31488a0 0x55f0f314ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.282+0000 7f1bd27fc640 10 client.16269 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:17:34.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.282+0000 7f1bead9d640 1 --2- 172.21.15.27:0/3446893856 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f0f31488a0 0x55f0f314ad00 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb53ce5be8d67e1e server_cookie=704ff1c95fd7bc32 in_seq=0 out_seq=0 2024-04-25T00:17:34.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.283+0000 7f1bd27fc640 1 -- 172.21.15.27:0/3446893856 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f0f309fbf0 con 0x55f0f31488a0 2024-04-25T00:17:34.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.283+0000 7f1bd27fc640 10 client.16269 handle_command_reply: tid=0 2024-04-25T00:17:34.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.283+0000 7f1bf3e53740 1 client.16269 shutdown 2024-04-25T00:17:34.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.283+0000 7f1bf3e53740 1 -- 172.21.15.27:0/3446893856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 msgr2=0x55f0f3136000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:34.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.283+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/3446893856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f3136000 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1be4014a50 tx=0x7f1be4004a40 comp rx=0 tx=0).stop 2024-04-25T00:17:34.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.283+0000 7f1bf3e53740 1 -- 172.21.15.27:0/3446893856 shutdown_connections 2024-04-25T00:17:34.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.283+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/3446893856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f3136000 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.283+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/3446893856 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f0f31488a0 0x55f0f314ad00 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.283+0000 7f1bd27fc640 0 client.16269 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:17:34.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.283+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/3446893856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f312df50 0x55f0f3135ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.283+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/3446893856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0f312d630 0x55f0f3135580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.283+0000 7f1bf3e53740 1 -- 172.21.15.27:0/3446893856 >> 172.21.15.27:0/3446893856 conn(0x55f0f3129950 msgr2=0x55f0f312f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:34.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.283+0000 7f1bf3e53740 1 -- 172.21.15.27:0/3446893856 shutdown_connections 2024-04-25T00:17:34.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.283+0000 7f1bf3e53740 1 -- 172.21.15.27:0/3446893856 wait complete. 2024-04-25T00:17:34.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.283+0000 7f1bf3e53740 20 client.16269 trim_cache size 0 max 16384 2024-04-25T00:17:34.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.305+0000 7f1bf3e53740 1 Processor -- start 2024-04-25T00:17:34.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.305+0000 7f1bf3e53740 1 -- start start 2024-04-25T00:17:34.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.305+0000 7f1bf3e53740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0f312d630 0x55f0f3137cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.305+0000 7f1bf3e53740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f312df50 0x55f0f3129270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.305+0000 7f1bf3e53740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f3138620 0x55f0f312fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.306+0000 7f1bf3e53740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec1acaf0 con 0x55f0f312df50 2024-04-25T00:17:34.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.306+0000 7f1bf3e53740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec1ae6f0 con 0x55f0f3138620 2024-04-25T00:17:34.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.306+0000 7f1bf3e53740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec1ad1c0 con 0x55f0f312d630 2024-04-25T00:17:34.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.306+0000 7f1be259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f312df50 0x55f0f3129270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.306+0000 7f1be259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f312df50 0x55f0f3129270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60070/0 (socket says 172.21.15.27:60070) 2024-04-25T00:17:34.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.306+0000 7f1be259c640 1 -- 172.21.15.27:0/1537163900 learned_addr learned my addr 172.21.15.27:0/1537163900 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:34.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.306+0000 7f1be259c640 1 -- 172.21.15.27:0/1537163900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0f312d630 msgr2=0x55f0f3137cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:17:34.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.306+0000 7f1be259c640 1 --2- 172.21.15.27:0/1537163900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0f312d630 0x55f0f3137cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.306+0000 7f1be259c640 1 -- 172.21.15.27:0/1537163900 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f3138620 msgr2=0x55f0f312fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:34.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.306+0000 7f1be259c640 1 --2- 172.21.15.27:0/1537163900 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f3138620 0x55f0f312fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.306+0000 7f1be259c640 1 -- 172.21.15.27:0/1537163900 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0f311b090 con 0x55f0f312df50 2024-04-25T00:17:34.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.306+0000 7f1be259c640 1 --2- 172.21.15.27:0/1537163900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f312df50 0x55f0f3129270 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4045000 tx=0x7f1bd4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a1b40741dddb29d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:34.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.307+0000 7f1be0d99640 1 -- 172.21.15.27:0/1537163900 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4035050 con 0x55f0f312df50 2024-04-25T00:17:34.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.307+0000 7f1be0d99640 1 -- 172.21.15.27:0/1537163900 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bd4005020 con 0x55f0f312df50 2024-04-25T00:17:34.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.307+0000 7f1be0d99640 1 -- 172.21.15.27:0/1537163900 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd403ea30 con 0x55f0f312df50 2024-04-25T00:17:34.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.307+0000 7f1bf3e53740 1 -- 172.21.15.27:0/1537163900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f312df50 msgr2=0x55f0f3129270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:34.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.307+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/1537163900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f312df50 0x55f0f3129270 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4045000 tx=0x7f1bd4000bc0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.308+0000 7f1bf3e53740 1 -- 172.21.15.27:0/1537163900 shutdown_connections 2024-04-25T00:17:34.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.308+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/1537163900 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f3138620 0x55f0f312fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.308+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/1537163900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f312df50 0x55f0f3129270 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.308+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/1537163900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0f312d630 0x55f0f3137cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.308+0000 7f1bf3e53740 1 -- 172.21.15.27:0/1537163900 >> 172.21.15.27:0/1537163900 conn(0x55f0f3129950 msgr2=0x55f0f3135540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:34.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.308+0000 7f1bf3e53740 1 -- 172.21.15.27:0/1537163900 shutdown_connections 2024-04-25T00:17:34.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.308+0000 7f1bf3e53740 1 -- 172.21.15.27:0/1537163900 wait complete. 2024-04-25T00:17:34.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.308+0000 7f1bf3e53740 1 -- start start 2024-04-25T00:17:34.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.308+0000 7f1bf3e53740 1 Processor -- start 2024-04-25T00:17:34.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.309+0000 7f1bf3e53740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f312d630 0x55f0f3140970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.309+0000 7f1bf3e53740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0f312df50 0x55f0f3131c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.309+0000 7f1bf3e53740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f3132140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.309+0000 7f1beb59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f3132140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.309+0000 7f1bf3e53740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec1adf60 con 0x55f0f3138620 2024-04-25T00:17:34.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.309+0000 7f1bf3e53740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec1ad890 con 0x55f0f312d630 2024-04-25T00:17:34.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.309+0000 7f1bf3e53740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec1aaab0 con 0x55f0f312df50 2024-04-25T00:17:34.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.309+0000 7f1beb59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f3132140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60080/0 (socket says 172.21.15.27:60080) 2024-04-25T00:17:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.309+0000 7f1beb59e640 1 -- 172.21.15.27:0/61404055 learned_addr learned my addr 172.21.15.27:0/61404055 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:17:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.309+0000 7f1bf3e53740 10 client.0 _ll_register_callbacks cb 0x7f1be4008960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:17:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.309+0000 7f1bead9d640 1 --2- 172.21.15.27:0/61404055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f312d630 0x55f0f3140970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.310+0000 7f1beb59e640 1 -- 172.21.15.27:0/61404055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0f312df50 msgr2=0x55f0f3131c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.310+0000 7f1beb59e640 1 --2- 172.21.15.27:0/61404055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0f312df50 0x55f0f3131c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.310+0000 7f1beb59e640 1 -- 172.21.15.27:0/61404055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f312d630 msgr2=0x55f0f3140970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.310+0000 7f1beb59e640 1 --2- 172.21.15.27:0/61404055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f312d630 0x55f0f3140970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.310+0000 7f1beb59e640 1 -- 172.21.15.27:0/61404055 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0f3132680 con 0x55f0f3138620 2024-04-25T00:17:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.310+0000 7f1beb59e640 1 --2- 172.21.15.27:0/61404055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f3132140 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f1be400d630 tx=0x7f1be4012e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70cfaf2ce2e5052 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.310+0000 7f1bd27fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:17:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.310+0000 7f1bd27fc640 1 -- 172.21.15.27:0/61404055 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be402e080 con 0x55f0f3138620 2024-04-25T00:17:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.311+0000 7f1bd27fc640 1 -- 172.21.15.27:0/61404055 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f1be40137e0 con 0x55f0f3138620 2024-04-25T00:17:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.311+0000 7f1bf3e53740 1 -- 172.21.15.27:0/61404055 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bd4005020 con 0x55f0f3138620 2024-04-25T00:17:34.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.311+0000 7f1bd27fc640 1 -- 172.21.15.27:0/61404055 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1be4024020 con 0x55f0f3138620 2024-04-25T00:17:34.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.311+0000 7f1bf3e53740 10 client.16275 fetch_fsmap learned FSMap version 30 2024-04-25T00:17:34.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.311+0000 7f1bf3e53740 1 -- 172.21.15.27:0/61404055 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f0f31aead0 con 0x55f0f3138620 2024-04-25T00:17:34.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.311+0000 7f1bd27fc640 1 -- 172.21.15.27:0/61404055 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bd4005020 con 0x55f0f3138620 2024-04-25T00:17:34.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.312+0000 7f1bd27fc640 1 -- 172.21.15.27:0/61404055 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f1be401f440 con 0x55f0f3138620 2024-04-25T00:17:34.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.312+0000 7f1bf3e53740 10 client.16275 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:17:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.312+0000 7f1bf3e53740 10 client.16275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:17:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.312+0000 7f1bf3e53740 20 client.16275 populate_metadata read hostname 'smithi027' 2024-04-25T00:17:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.312+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/61404055 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f0f31af290 0x55f0f31b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.312+0000 7f1bf3e53740 4 client.16275 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:17:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.312+0000 7f1bf3e53740 1 -- 172.21.15.27:0/61404055 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f0f3099f40 con 0x55f0f31af290 2024-04-25T00:17:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.312+0000 7f1bead9d640 1 --2- 172.21.15.27:0/61404055 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f0f31af290 0x55f0f31b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.313+0000 7f1bd27fc640 10 client.16275 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:17:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.313+0000 7f1bead9d640 1 --2- 172.21.15.27:0/61404055 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f0f31af290 0x55f0f31b16b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12367d0825bb90a5 server_cookie=17ea253a3abc0c98 in_seq=0 out_seq=0 2024-04-25T00:17:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.313+0000 7f1bd27fc640 1 -- 172.21.15.27:0/61404055 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f0f3099f40 con 0x55f0f31af290 2024-04-25T00:17:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.313+0000 7f1bd27fc640 10 client.16275 handle_command_reply: tid=0 2024-04-25T00:17:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.313+0000 7f1bf3e53740 1 client.16275 shutdown 2024-04-25T00:17:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.313+0000 7f1bf3e53740 1 -- 172.21.15.27:0/61404055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 msgr2=0x55f0f3132140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.313+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/61404055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f3132140 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f1be400d630 tx=0x7f1be4012e60 comp rx=0 tx=0).stop 2024-04-25T00:17:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.314+0000 7f1bf3e53740 1 -- 172.21.15.27:0/61404055 shutdown_connections 2024-04-25T00:17:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.314+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/61404055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0f3138620 0x55f0f3132140 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.314+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/61404055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0f312df50 0x55f0f3131c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.314+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/61404055 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f0f31af290 0x55f0f31b16b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.314+0000 7f1bf3e53740 1 --2- 172.21.15.27:0/61404055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f0f312d630 0x55f0f3140970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.314+0000 7f1bd27fc640 0 client.16275 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:17:34.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.314+0000 7f1bf3e53740 1 -- 172.21.15.27:0/61404055 >> 172.21.15.27:0/61404055 conn(0x55f0f3129950 msgr2=0x55f0f3135db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:34.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.314+0000 7f1bf3e53740 1 -- 172.21.15.27:0/61404055 shutdown_connections 2024-04-25T00:17:34.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.314+0000 7f1bf3e53740 1 -- 172.21.15.27:0/61404055 wait complete. 2024-04-25T00:17:34.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.314+0000 7f1bf3e53740 20 client.16275 trim_cache size 0 max 16384 2024-04-25T00:17:34.318 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:17:34.318 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T00:17:34.318 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:17:34.319 INFO:teuthology.orchestra.run.smithi027.stdout: "4b378baa-7184-4771-a5eb-086427f3134f": { 2024-04-25T00:17:34.319 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:17:34.319 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "4b378baa-7184-4771-a5eb-086427f3134f", 2024-04-25T00:17:34.319 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:17:34.319 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:17:34.319 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:17:34.319 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:17:34.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.315+0000 7f1bf1142640 1 -- 172.21.15.27:0/287760203 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1bd80a2d00 msgr2=0x7f1bd80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:34.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.315+0000 7f1bf1142640 1 --2- 172.21.15.27:0/287760203 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1bd80a2d00 0x7f1bd80a51c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1be4008bf0 tx=0x7f1be4008260 comp rx=0 tx=0).stop 2024-04-25T00:17:34.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.315+0000 7f1bf1142640 1 -- 172.21.15.27:0/287760203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bec101710 msgr2=0x7f1bec1981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:34.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.315+0000 7f1bf1142640 1 --2- 172.21.15.27:0/287760203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bec101710 0x7f1bec1981b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00b4a0 tx=0x7f1bdc00b960 comp rx=0 tx=0).stop 2024-04-25T00:17:34.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.315+0000 7f1bf1142640 1 -- 172.21.15.27:0/287760203 shutdown_connections 2024-04-25T00:17:34.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.315+0000 7f1bf1142640 1 --2- 172.21.15.27:0/287760203 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1bd80a2d00 0x7f1bd80a51c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.315+0000 7f1bf1142640 1 --2- 172.21.15.27:0/287760203 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bec107720 0x7f1bec19ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.315+0000 7f1bf1142640 1 --2- 172.21.15.27:0/287760203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bec1020d0 0x7f1bec1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.315+0000 7f1bf1142640 1 --2- 172.21.15.27:0/287760203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bec101710 0x7f1bec1981b0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:34.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.315+0000 7f1bf1142640 1 -- 172.21.15.27:0/287760203 >> 172.21.15.27:0/287760203 conn(0x7f1bec078b90 msgr2=0x7f1bec0ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:34.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.315+0000 7f1bf1142640 1 -- 172.21.15.27:0/287760203 shutdown_connections 2024-04-25T00:17:34.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:17:34.315+0000 7f1bf1142640 1 -- 172.21.15.27:0/287760203 wait complete. 2024-04-25T00:17:34.332 INFO:tasks.cephfs.filesystem:scrub status for tag:4b378baa-7184-4771-a5eb-086427f3134f - {'path': '/', 'tag': '4b378baa-7184-4771-a5eb-086427f3134f', 'options': 'recursive,force'} 2024-04-25T00:17:34.782 INFO:tasks.workunit.client.0.smithi027.stdout:Syncing()...3 sec 2024-04-25T00:17:34.783 INFO:tasks.workunit.client.0.smithi027.stdout:Starting Actual Benchmark At: Thu Apr 25 00:17:34 2024 2024-04-25T00:17:34.783 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:17:34.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:34 smithi027 ceph-mon[29208]: pgmap v57: 129 pgs: 129 active+clean; 9.7 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 683 MiB/s wr, 253 op/s 2024-04-25T00:17:34.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:34 smithi027 ceph-mon[29208]: scrub queued for path: / 2024-04-25T00:17:34.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:34 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:34.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:34 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:17:34.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:34 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:17:34.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:34 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:34.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:34 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:17:34.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:34 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:34.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:34 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:17:34.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:34 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:34.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:34 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:17:34.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:34 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:34.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:34 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:35.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:34 smithi139 ceph-mon[32833]: pgmap v57: 129 pgs: 129 active+clean; 9.7 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 683 MiB/s wr, 253 op/s 2024-04-25T00:17:35.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:34 smithi139 ceph-mon[32833]: scrub queued for path: / 2024-04-25T00:17:35.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:34 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:35.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:34 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:17:35.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:34 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:17:35.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:34 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:35.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:34 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:17:35.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:34 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:35.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:34 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:17:35.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:34 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:35.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:34 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:17:35.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:34 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:35.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:34 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:35.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:34 smithi153 ceph-mon[32896]: pgmap v57: 129 pgs: 129 active+clean; 9.7 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 683 MiB/s wr, 253 op/s 2024-04-25T00:17:35.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:34 smithi153 ceph-mon[32896]: scrub queued for path: / 2024-04-25T00:17:35.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:34 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:35.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:34 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:17:35.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:34 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:17:35.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:34 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:35.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:34 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:17:35.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:34 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:35.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:34 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:17:35.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:34 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:35.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:34 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:17:35.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:34 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:35.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:34 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:36.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:35 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:17:36.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:35 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:36.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:35 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:17:36.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:35 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:17:36.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:35 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:36.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:35 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:17:36.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:35 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:36.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:35 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:36.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:35 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:17:36.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:35 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:36.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:35 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:17:36.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:35 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:17:36.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:35 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:36.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:35 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:17:36.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:35 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:36.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:35 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:36.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:35 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:17:36.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:35 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:36.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:35 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:17:36.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:35 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:17:36.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:35 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:36.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:35 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:17:36.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:35 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:36.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:35 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:17:37.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:36 smithi139 ceph-mon[32833]: pgmap v58: 129 pgs: 129 active+clean; 9.7 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 683 MiB/s wr, 253 op/s 2024-04-25T00:17:37.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:36 smithi153 ceph-mon[32896]: pgmap v58: 129 pgs: 129 active+clean; 9.7 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 683 MiB/s wr, 253 op/s 2024-04-25T00:17:37.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:36 smithi027 ceph-mon[29208]: pgmap v58: 129 pgs: 129 active+clean; 9.7 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 683 MiB/s wr, 253 op/s 2024-04-25T00:17:39.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:38 smithi139 ceph-mon[32833]: pgmap v59: 129 pgs: 129 active+clean; 13 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 926 MiB/s wr, 1.41k op/s 2024-04-25T00:17:39.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:38 smithi153 ceph-mon[32896]: pgmap v59: 129 pgs: 129 active+clean; 13 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 926 MiB/s wr, 1.41k op/s 2024-04-25T00:17:39.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:38 smithi027 ceph-mon[29208]: pgmap v59: 129 pgs: 129 active+clean; 13 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 926 MiB/s wr, 1.41k op/s 2024-04-25T00:17:41.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:40 smithi139 ceph-mon[32833]: pgmap v60: 129 pgs: 129 active+clean; 13 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 583 MiB/s wr, 1.28k op/s 2024-04-25T00:17:41.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:40 smithi153 ceph-mon[32896]: pgmap v60: 129 pgs: 129 active+clean; 13 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 583 MiB/s wr, 1.28k op/s 2024-04-25T00:17:41.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:40 smithi027 ceph-mon[29208]: pgmap v60: 129 pgs: 129 active+clean; 13 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 583 MiB/s wr, 1.28k op/s 2024-04-25T00:17:43.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:42 smithi139 ceph-mon[32833]: pgmap v61: 129 pgs: 129 active+clean; 13 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 586 MiB/s wr, 1.70k op/s 2024-04-25T00:17:43.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:42 smithi153 ceph-mon[32896]: pgmap v61: 129 pgs: 129 active+clean; 13 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 586 MiB/s wr, 1.70k op/s 2024-04-25T00:17:43.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:42 smithi027 ceph-mon[29208]: pgmap v61: 129 pgs: 129 active+clean; 13 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 586 MiB/s wr, 1.70k op/s 2024-04-25T00:17:44.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:43 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:17:44.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:43 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:17:44.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:43 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:17:44.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:43 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:17:44.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:43 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:17:44.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:43 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:17:44.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:43 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:17:44.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:43 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:17:44.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:43 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:17:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:44 smithi139 ceph-mon[32833]: pgmap v62: 129 pgs: 129 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 363 MiB/s wr, 8.25k op/s 2024-04-25T00:17:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:44 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:17:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:44 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:17:45.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:44 smithi153 ceph-mon[32896]: pgmap v62: 129 pgs: 129 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 363 MiB/s wr, 8.25k op/s 2024-04-25T00:17:45.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:44 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:17:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:44 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:17:45.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:44 smithi027 ceph-mon[29208]: pgmap v62: 129 pgs: 129 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 363 MiB/s wr, 8.25k op/s 2024-04-25T00:17:45.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:44 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:17:45.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:44 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:17:47.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:46 smithi139 ceph-mon[32833]: pgmap v63: 129 pgs: 129 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 290 MiB/s wr, 8.23k op/s 2024-04-25T00:17:47.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:46 smithi153 ceph-mon[32896]: pgmap v63: 129 pgs: 129 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 290 MiB/s wr, 8.23k op/s 2024-04-25T00:17:47.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:46 smithi027 ceph-mon[29208]: pgmap v63: 129 pgs: 129 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 290 MiB/s wr, 8.23k op/s 2024-04-25T00:17:49.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:48 smithi139 ceph-mon[32833]: pgmap v64: 129 pgs: 129 active+clean; 13 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 339 MiB/s wr, 14.67k op/s 2024-04-25T00:17:49.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:48 smithi153 ceph-mon[32896]: pgmap v64: 129 pgs: 129 active+clean; 13 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 339 MiB/s wr, 14.67k op/s 2024-04-25T00:17:49.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:48 smithi027 ceph-mon[29208]: pgmap v64: 129 pgs: 129 active+clean; 13 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 339 MiB/s wr, 14.67k op/s 2024-04-25T00:17:51.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:50 smithi153 ceph-mon[32896]: pgmap v65: 129 pgs: 129 active+clean; 13 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 96 MiB/s wr, 13.52k op/s 2024-04-25T00:17:51.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:50 smithi027 ceph-mon[29208]: pgmap v65: 129 pgs: 129 active+clean; 13 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 96 MiB/s wr, 13.52k op/s 2024-04-25T00:17:51.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:50 smithi139 ceph-mon[32833]: pgmap v65: 129 pgs: 129 active+clean; 13 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 96 MiB/s wr, 13.52k op/s 2024-04-25T00:17:52.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:51 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:17:52.154 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:51 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:17:52.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:51 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:17:53.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:52 smithi153 ceph-mon[32896]: pgmap v66: 129 pgs: 129 active+clean; 13 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 99 MiB/s wr, 13.92k op/s 2024-04-25T00:17:53.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:52 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:17:53.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:52 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:17:53.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:52 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:17:53.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:52 smithi027 ceph-mon[29208]: pgmap v66: 129 pgs: 129 active+clean; 13 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 99 MiB/s wr, 13.92k op/s 2024-04-25T00:17:53.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:52 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:17:53.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:52 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:17:53.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:52 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:17:53.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:52 smithi139 ceph-mon[32833]: pgmap v66: 129 pgs: 129 active+clean; 13 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 99 MiB/s wr, 13.92k op/s 2024-04-25T00:17:53.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:52 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:17:53.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:52 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:17:53.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:52 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:17:55.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:54 smithi153 ceph-mon[32896]: pgmap v67: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 147 MiB/s wr, 19.71k op/s 2024-04-25T00:17:55.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:54 smithi027 ceph-mon[29208]: pgmap v67: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 147 MiB/s wr, 19.71k op/s 2024-04-25T00:17:55.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:54 smithi139 ceph-mon[32833]: pgmap v67: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 147 MiB/s wr, 19.71k op/s 2024-04-25T00:17:57.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:56 smithi153 ceph-mon[32896]: pgmap v68: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 103 MiB/s wr, 13.06k op/s 2024-04-25T00:17:57.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:56 smithi027 ceph-mon[29208]: pgmap v68: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 103 MiB/s wr, 13.06k op/s 2024-04-25T00:17:57.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:56 smithi139 ceph-mon[32833]: pgmap v68: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 103 MiB/s wr, 13.06k op/s 2024-04-25T00:17:59.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:17:58 smithi027 ceph-mon[29208]: pgmap v69: 129 pgs: 129 active+clean; 13 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 148 MiB/s wr, 18.82k op/s 2024-04-25T00:17:59.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:17:58 smithi139 ceph-mon[32833]: pgmap v69: 129 pgs: 129 active+clean; 13 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 148 MiB/s wr, 18.82k op/s 2024-04-25T00:17:59.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:17:58 smithi153 ceph-mon[32896]: pgmap v69: 129 pgs: 129 active+clean; 13 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 148 MiB/s wr, 18.82k op/s 2024-04-25T00:18:01.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:00 smithi027 ceph-mon[29208]: pgmap v70: 129 pgs: 129 active+clean; 13 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 107 MiB/s wr, 13.29k op/s 2024-04-25T00:18:01.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:00 smithi139 ceph-mon[32833]: pgmap v70: 129 pgs: 129 active+clean; 13 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 107 MiB/s wr, 13.29k op/s 2024-04-25T00:18:01.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:00 smithi153 ceph-mon[32896]: pgmap v70: 129 pgs: 129 active+clean; 13 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 107 MiB/s wr, 13.29k op/s 2024-04-25T00:18:03.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:02 smithi027 ceph-mon[29208]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 109 MiB/s wr, 13.68k op/s 2024-04-25T00:18:03.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:02 smithi139 ceph-mon[32833]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 109 MiB/s wr, 13.68k op/s 2024-04-25T00:18:03.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:02 smithi153 ceph-mon[32896]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 109 MiB/s wr, 13.68k op/s 2024-04-25T00:18:04.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:18:04.334 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:18:04.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.512+0000 7f636c81c640 1 -- 172.21.15.27:0/1537637089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f636810ff90 msgr2=0x7f636810da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:04.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.512+0000 7f636c81c640 1 --2- 172.21.15.27:0/1537637089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f636810ff90 0x7f636810da80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6360009ca0 tx=0x7f636002a8d0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.513+0000 7f636c81c640 1 -- 172.21.15.27:0/1537637089 shutdown_connections 2024-04-25T00:18:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.513+0000 7f636c81c640 1 --2- 172.21.15.27:0/1537637089 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636811af20 0x7f636811d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.513+0000 7f636c81c640 1 --2- 172.21.15.27:0/1537637089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f636810dfc0 0x7f636810e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.513+0000 7f636c81c640 1 --2- 172.21.15.27:0/1537637089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f636810ff90 0x7f636810da80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.513+0000 7f636c81c640 1 -- 172.21.15.27:0/1537637089 >> 172.21.15.27:0/1537637089 conn(0x7f636806c850 msgr2=0x7f636806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:04.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.517+0000 7f636c81c640 1 -- 172.21.15.27:0/1537637089 shutdown_connections 2024-04-25T00:18:04.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.519+0000 7f636c81c640 1 -- 172.21.15.27:0/1537637089 wait complete. 2024-04-25T00:18:04.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.520+0000 7f636c81c640 1 Processor -- start 2024-04-25T00:18:04.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.521+0000 7f636c81c640 1 -- start start 2024-04-25T00:18:04.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.521+0000 7f636c81c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f636810dfc0 0x7f636811a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.521+0000 7f636c81c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636810ff90 0x7f636811a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.521+0000 7f636c81c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f636811af20 0x7f6368113740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.521+0000 7f636c81c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6368120980 con 0x7f636810dfc0 2024-04-25T00:18:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.521+0000 7f636c81c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6368120800 con 0x7f636810ff90 2024-04-25T00:18:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.521+0000 7f636c81c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6368120b00 con 0x7f636811af20 2024-04-25T00:18:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.522+0000 7f6365d9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636810ff90 0x7f636811a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.522+0000 7f6365d9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636810ff90 0x7f636811a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:59832/0 (socket says 172.21.15.27:59832) 2024-04-25T00:18:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.522+0000 7f6365d9b640 1 -- 172.21.15.27:0/1018773710 learned_addr learned my addr 172.21.15.27:0/1018773710 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.522+0000 7f6366d9d640 1 --2- 172.21.15.27:0/1018773710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f636811af20 0x7f6368113740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.522+0000 7f6365d9b640 1 -- 172.21.15.27:0/1018773710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f636811af20 msgr2=0x7f6368113740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.522+0000 7f6365d9b640 1 --2- 172.21.15.27:0/1018773710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f636811af20 0x7f6368113740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.522+0000 7f6365d9b640 1 -- 172.21.15.27:0/1018773710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f636810dfc0 msgr2=0x7f636811a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.522+0000 7f6365d9b640 1 --2- 172.21.15.27:0/1018773710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f636810dfc0 0x7f636811a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.522+0000 7f6365d9b640 1 -- 172.21.15.27:0/1018773710 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6368114000 con 0x7f636810ff90 2024-04-25T00:18:04.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.523+0000 7f6365d9b640 1 --2- 172.21.15.27:0/1018773710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636810ff90 0x7f636811a9b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f635800b450 tx=0x7f635800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:04.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.524+0000 7f63577fe640 1 -- 172.21.15.27:0/1018773710 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6358015020 con 0x7f636810ff90 2024-04-25T00:18:04.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.524+0000 7f636c81c640 1 -- 172.21.15.27:0/1018773710 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63681142f0 con 0x7f636810ff90 2024-04-25T00:18:04.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.524+0000 7f636c81c640 1 -- 172.21.15.27:0/1018773710 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6368118230 con 0x7f636810ff90 2024-04-25T00:18:04.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.528+0000 7f636f606740 1 Processor -- start 2024-04-25T00:18:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.528+0000 7f636f606740 1 -- start start 2024-04-25T00:18:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.528+0000 7f636f606740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317ed2630 0x565317ed2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.528+0000 7f636f606740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2f50 0x565317edd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.528+0000 7f636f606740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317edd620 0x565317edfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.528+0000 7f636f606740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x565317e376b0 con 0x565317ed2f50 2024-04-25T00:18:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.528+0000 7f636f606740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x565317ddf880 con 0x565317ed2630 2024-04-25T00:18:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.528+0000 7f636f606740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x565317e33670 con 0x565317edd620 2024-04-25T00:18:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.529+0000 7f63577fe640 1 -- 172.21.15.27:0/1018773710 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6358011d70 con 0x7f636810ff90 2024-04-25T00:18:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.530+0000 7f6366d9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317ed2630 0x565317ed2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.531+0000 7f6366d9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317ed2630 0x565317ed2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:59840/0 (socket says 172.21.15.27:59840) 2024-04-25T00:18:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.531+0000 7f6366d9d640 1 -- 172.21.15.27:0/3317235429 learned_addr learned my addr 172.21.15.27:0/3317235429 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.532+0000 7f636659c640 1 --2- 172.21.15.27:0/3317235429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2f50 0x565317edd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.532+0000 7f6366d9d640 1 -- 172.21.15.27:0/3317235429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317edd620 msgr2=0x565317edfa10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:18:04.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.532+0000 7f6366d9d640 1 --2- 172.21.15.27:0/3317235429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317edd620 0x565317edfa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.533+0000 7f6366d9d640 1 -- 172.21.15.27:0/3317235429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2f50 msgr2=0x565317edd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.533+0000 7f6366d9d640 1 --2- 172.21.15.27:0/3317235429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2f50 0x565317edd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.533+0000 7f6366d9d640 1 -- 172.21.15.27:0/3317235429 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565317c157d0 con 0x565317ed2630 2024-04-25T00:18:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.533+0000 7f6366d9d640 1 --2- 172.21.15.27:0/3317235429 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317ed2630 0x565317ed2a10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f635c00ac10 tx=0x7f635c00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a37be69028cf0a82 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.535+0000 7f63577fe640 1 -- 172.21.15.27:0/1018773710 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635801a400 con 0x7f636810ff90 2024-04-25T00:18:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.535+0000 7f6354ff9640 1 -- 172.21.15.27:0/3317235429 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c005220 con 0x565317ed2630 2024-04-25T00:18:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.535+0000 7f6354ff9640 1 -- 172.21.15.27:0/3317235429 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f635c004c80 con 0x565317ed2630 2024-04-25T00:18:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.536+0000 7f63577fe640 1 -- 172.21.15.27:0/1018773710 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6358018020 con 0x7f636810ff90 2024-04-25T00:18:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.536+0000 7f636f606740 1 -- 172.21.15.27:0/3317235429 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317ed2630 msgr2=0x565317ed2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.536+0000 7f636f606740 1 --2- 172.21.15.27:0/3317235429 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317ed2630 0x565317ed2a10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f635c00ac10 tx=0x7f635c00f790 comp rx=0 tx=0).stop 2024-04-25T00:18:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.537+0000 7f63577fe640 1 --2- 172.21.15.27:0/1018773710 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f63500a2ca0 0x7f63500a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.537+0000 7f63577fe640 1 -- 172.21.15.27:0/1018773710 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f63580cf1d0 con 0x7f636810ff90 2024-04-25T00:18:04.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.538+0000 7f6366d9d640 1 --2- 172.21.15.27:0/1018773710 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f63500a2ca0 0x7f63500a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.539+0000 7f636f606740 1 -- 172.21.15.27:0/3317235429 shutdown_connections 2024-04-25T00:18:04.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.539+0000 7f636f606740 1 --2- 172.21.15.27:0/3317235429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317edd620 0x565317edfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.539+0000 7f636f606740 1 --2- 172.21.15.27:0/3317235429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2f50 0x565317edd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.539+0000 7f636f606740 1 --2- 172.21.15.27:0/3317235429 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317ed2630 0x565317ed2a10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.539+0000 7f636f606740 1 -- 172.21.15.27:0/3317235429 >> 172.21.15.27:0/3317235429 conn(0x565317ece950 msgr2=0x565317eced30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:04.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.540+0000 7f636f606740 1 -- 172.21.15.27:0/3317235429 shutdown_connections 2024-04-25T00:18:04.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.540+0000 7f6366d9d640 1 --2- 172.21.15.27:0/1018773710 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f63500a2ca0 0x7f63500a5160 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f635c01a040 tx=0x7f635c012040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.542+0000 7f636f606740 1 -- 172.21.15.27:0/3317235429 wait complete. 2024-04-25T00:18:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.543+0000 7f636f606740 1 -- start start 2024-04-25T00:18:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.543+0000 7f636f606740 1 Processor -- start 2024-04-25T00:18:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.544+0000 7f636f606740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2630 0x565317edd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.544+0000 7f636f606740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317ed2f50 0x565317eda470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.544+0000 7f636659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2630 0x565317edd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.544+0000 7f636659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2630 0x565317edd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51752/0 (socket says 172.21.15.27:51752) 2024-04-25T00:18:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.544+0000 7f636659c640 1 -- 172.21.15.27:0/3993263678 learned_addr learned my addr 172.21.15.27:0/3993263678 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.544+0000 7f636f606740 1 --2- 172.21.15.27:0/3993263678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317edd620 0x565317edabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.544+0000 7f636f606740 1 -- 172.21.15.27:0/3993263678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x565317ee0460 con 0x565317ed2630 2024-04-25T00:18:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.544+0000 7f636f606740 1 -- 172.21.15.27:0/3993263678 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x565317df2f50 con 0x565317edd620 2024-04-25T00:18:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.544+0000 7f636f606740 1 -- 172.21.15.27:0/3993263678 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x565317ee05e0 con 0x565317ed2f50 2024-04-25T00:18:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.544+0000 7f6365d9b640 1 --2- 172.21.15.27:0/3993263678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317ed2f50 0x565317eda470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.545+0000 7f6366d9d640 1 --2- 172.21.15.27:0/3993263678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317edd620 0x565317edabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.545+0000 7f636f606740 10 client.0 _ll_register_callbacks cb 0x565317e2e000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.546+0000 7f6366d9d640 1 -- 172.21.15.27:0/3993263678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317ed2f50 msgr2=0x565317eda470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.546+0000 7f6366d9d640 1 --2- 172.21.15.27:0/3993263678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317ed2f50 0x565317eda470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.546+0000 7f6366d9d640 1 -- 172.21.15.27:0/3993263678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2630 msgr2=0x565317edd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.546+0000 7f6366d9d640 1 --2- 172.21.15.27:0/3993263678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2630 0x565317edd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.546+0000 7f6366d9d640 1 -- 172.21.15.27:0/3993263678 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565317b4b3f0 con 0x565317edd620 2024-04-25T00:18:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.546+0000 7f6366d9d640 1 --2- 172.21.15.27:0/3993263678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317edd620 0x565317edabb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f635c00abe0 tx=0x7f635c00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b66a07487c62fcc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.547+0000 7f634dffb640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T00:18:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.547+0000 7f634dffb640 1 -- 172.21.15.27:0/3993263678 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c02b050 con 0x565317edd620 2024-04-25T00:18:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.547+0000 7f636f606740 1 -- 172.21.15.27:0/3993263678 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f635c004c80 con 0x565317edd620 2024-04-25T00:18:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.547+0000 7f634dffb640 1 -- 172.21.15.27:0/3993263678 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f635c004990 con 0x565317edd620 2024-04-25T00:18:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.548+0000 7f634dffb640 1 -- 172.21.15.27:0/3993263678 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f635c03e050 con 0x565317edd620 2024-04-25T00:18:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.548+0000 7f636f606740 10 client.25826 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.548+0000 7f636f606740 1 -- 172.21.15.27:0/3993263678 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x565317b0b0f0 con 0x565317edd620 2024-04-25T00:18:04.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.550+0000 7f634dffb640 1 -- 172.21.15.27:0/3993263678 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f635c004c80 con 0x565317edd620 2024-04-25T00:18:04.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.550+0000 7f634dffb640 1 -- 172.21.15.27:0/3993263678 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f635c0305c0 con 0x565317edd620 2024-04-25T00:18:04.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.550+0000 7f636f606740 10 client.25826 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:04.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.550+0000 7f636f606740 10 client.25826 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:18:04.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.550+0000 7f636f606740 20 client.25826 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:04.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.550+0000 7f636f606740 1 --2- 172.21.15.27:0/3993263678 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x565317eedd10 0x565317ef0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.550+0000 7f636f606740 4 client.25826 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:18:04.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.550+0000 7f636f606740 1 -- 172.21.15.27:0/3993263678 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565317e44bf0 con 0x565317eedd10 2024-04-25T00:18:04.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.553+0000 7f636659c640 1 --2- 172.21.15.27:0/3993263678 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x565317eedd10 0x565317ef0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.556+0000 7f636659c640 1 --2- 172.21.15.27:0/3993263678 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x565317eedd10 0x565317ef0150 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3478c680787ddbc4 server_cookie=a5bc1ff6acbf17f2 in_seq=0 out_seq=0 2024-04-25T00:18:04.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.557+0000 7f634dffb640 10 client.25826 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:04.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.560+0000 7f634dffb640 1 -- 172.21.15.27:0/3993263678 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565317e44bf0 con 0x565317eedd10 2024-04-25T00:18:04.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.560+0000 7f634dffb640 10 client.25826 handle_command_reply: tid=0 2024-04-25T00:18:04.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.560+0000 7f636f606740 1 client.25826 shutdown 2024-04-25T00:18:04.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.560+0000 7f636f606740 1 -- 172.21.15.27:0/3993263678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317edd620 msgr2=0x565317edabb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:04.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.560+0000 7f636f606740 1 --2- 172.21.15.27:0/3993263678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317edd620 0x565317edabb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f635c00abe0 tx=0x7f635c00f790 comp rx=0 tx=0).stop 2024-04-25T00:18:04.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.564+0000 7f636f606740 1 -- 172.21.15.27:0/3993263678 shutdown_connections 2024-04-25T00:18:04.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.564+0000 7f636f606740 1 --2- 172.21.15.27:0/3993263678 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x565317eedd10 0x565317ef0150 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.564+0000 7f636f606740 1 --2- 172.21.15.27:0/3993263678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317edd620 0x565317edabb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.564+0000 7f636f606740 1 --2- 172.21.15.27:0/3993263678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317ed2f50 0x565317eda470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.564+0000 7f636f606740 1 --2- 172.21.15.27:0/3993263678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2630 0x565317edd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.564+0000 7f636f606740 1 -- 172.21.15.27:0/3993263678 >> 172.21.15.27:0/3993263678 conn(0x565317ece950 msgr2=0x565317ecdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:04.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.564+0000 7f634dffb640 0 client.25826 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:04.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.569+0000 7f636f606740 1 -- 172.21.15.27:0/3993263678 shutdown_connections 2024-04-25T00:18:04.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.569+0000 7f636f606740 1 -- 172.21.15.27:0/3993263678 wait complete. 2024-04-25T00:18:04.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.570+0000 7f636f606740 20 client.25826 trim_cache size 0 max 16384 2024-04-25T00:18:04.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.617+0000 7f636f606740 1 Processor -- start 2024-04-25T00:18:04.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.617+0000 7f636f606740 1 -- start start 2024-04-25T00:18:04.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.618+0000 7f636f606740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317ed2630 0x565317ee9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.618+0000 7f636f606740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2f50 0x565317ef5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.618+0000 7f636f606740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317edd620 0x565317ef5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.618+0000 7f636f606740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f634800ff40 con 0x565317ed2f50 2024-04-25T00:18:04.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.618+0000 7f636f606740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6348011b40 con 0x565317edd620 2024-04-25T00:18:04.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.618+0000 7f636f606740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6348010610 con 0x565317ed2630 2024-04-25T00:18:04.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.619+0000 7f6365d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2f50 0x565317ef5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.619+0000 7f6365d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2f50 0x565317ef5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51758/0 (socket says 172.21.15.27:51758) 2024-04-25T00:18:04.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.619+0000 7f6365d9b640 1 -- 172.21.15.27:0/1579852535 learned_addr learned my addr 172.21.15.27:0/1579852535 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:04.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.619+0000 7f636659c640 1 --2- 172.21.15.27:0/1579852535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317ed2630 0x565317ee9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.619+0000 7f6366d9d640 1 --2- 172.21.15.27:0/1579852535 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317edd620 0x565317ef5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.620+0000 7f6366d9d640 1 -- 172.21.15.27:0/1579852535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317ed2630 msgr2=0x565317ee9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:04.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.620+0000 7f6366d9d640 1 --2- 172.21.15.27:0/1579852535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317ed2630 0x565317ee9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.620+0000 7f6366d9d640 1 -- 172.21.15.27:0/1579852535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2f50 msgr2=0x565317ef5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:04.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.620+0000 7f6366d9d640 1 --2- 172.21.15.27:0/1579852535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2f50 0x565317ef5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.620+0000 7f6366d9d640 1 -- 172.21.15.27:0/1579852535 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565317eca2a0 con 0x565317edd620 2024-04-25T00:18:04.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.622+0000 7f6366d9d640 1 --2- 172.21.15.27:0/1579852535 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317edd620 0x565317ef5fa0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f635c030bd0 tx=0x7f635c018580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bb812da14dc5548 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:04.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.623+0000 7f634ffff640 1 -- 172.21.15.27:0/1579852535 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c046070 con 0x565317edd620 2024-04-25T00:18:04.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.623+0000 7f634ffff640 1 -- 172.21.15.27:0/1579852535 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f635c035070 con 0x565317edd620 2024-04-25T00:18:04.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.624+0000 7f636f606740 1 -- 172.21.15.27:0/1579852535 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317edd620 msgr2=0x565317ef5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:04.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.624+0000 7f636f606740 1 --2- 172.21.15.27:0/1579852535 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317edd620 0x565317ef5fa0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f635c030bd0 tx=0x7f635c018580 comp rx=0 tx=0).stop 2024-04-25T00:18:04.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.624+0000 7f636f606740 1 -- 172.21.15.27:0/1579852535 shutdown_connections 2024-04-25T00:18:04.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.624+0000 7f636f606740 1 --2- 172.21.15.27:0/1579852535 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317edd620 0x565317ef5fa0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.625+0000 7f636f606740 1 --2- 172.21.15.27:0/1579852535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2f50 0x565317ef5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.625+0000 7f636f606740 1 --2- 172.21.15.27:0/1579852535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317ed2630 0x565317ee9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.625+0000 7f636f606740 1 -- 172.21.15.27:0/1579852535 >> 172.21.15.27:0/1579852535 conn(0x565317ece950 msgr2=0x565317ed5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:04.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.627+0000 7f636f606740 1 -- 172.21.15.27:0/1579852535 shutdown_connections 2024-04-25T00:18:04.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.627+0000 7f636f606740 1 -- 172.21.15.27:0/1579852535 wait complete. 2024-04-25T00:18:04.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.628+0000 7f636f606740 1 -- start start 2024-04-25T00:18:04.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.628+0000 7f636f606740 1 Processor -- start 2024-04-25T00:18:04.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.628+0000 7f636f606740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2630 0x565317ee5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.628+0000 7f636f606740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317ed2f50 0x565317ed7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.628+0000 7f636f606740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317edd620 0x565317ed7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.628+0000 7f636f606740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f63480113b0 con 0x565317ed2630 2024-04-25T00:18:04.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.628+0000 7f636f606740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6348010ce0 con 0x565317edd620 2024-04-25T00:18:04.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.628+0000 7f636f606740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f634800df00 con 0x565317ed2f50 2024-04-25T00:18:04.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.628+0000 7f636f606740 10 client.0 _ll_register_callbacks cb 0x7f635c00aa60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:04.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.629+0000 7f636659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2630 0x565317ee5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.629+0000 7f636659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2630 0x565317ee5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51774/0 (socket says 172.21.15.27:51774) 2024-04-25T00:18:04.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.629+0000 7f636659c640 1 -- 172.21.15.27:0/2567709866 learned_addr learned my addr 172.21.15.27:0/2567709866 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:04.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.629+0000 7f6365d9b640 1 --2- 172.21.15.27:0/2567709866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317ed2f50 0x565317ed7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.629+0000 7f6366d9d640 1 --2- 172.21.15.27:0/2567709866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317edd620 0x565317ed7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.631+0000 7f6365d9b640 1 -- 172.21.15.27:0/2567709866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317edd620 msgr2=0x565317ed7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:04.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.631+0000 7f6365d9b640 1 --2- 172.21.15.27:0/2567709866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317edd620 0x565317ed7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.631+0000 7f6365d9b640 1 -- 172.21.15.27:0/2567709866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2630 msgr2=0x565317ee5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:04.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.631+0000 7f6365d9b640 1 --2- 172.21.15.27:0/2567709866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2630 0x565317ee5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.631+0000 7f6365d9b640 1 -- 172.21.15.27:0/2567709866 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565317ed6700 con 0x565317ed2f50 2024-04-25T00:18:04.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.631+0000 7f6365d9b640 1 --2- 172.21.15.27:0/2567709866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317ed2f50 0x565317ed7830 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6358097f10 tx=0x7f635809b4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b5af92275ca0fc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:04.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.631+0000 7f634dffb640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:18:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.636+0000 7f634dffb640 1 -- 172.21.15.27:0/2567709866 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6358065020 con 0x565317ed2f50 2024-04-25T00:18:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.636+0000 7f636f606740 1 -- 172.21.15.27:0/2567709866 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x565317e3ef40 con 0x565317ed2f50 2024-04-25T00:18:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.636+0000 7f634dffb640 1 -- 172.21.15.27:0/2567709866 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f635809af00 con 0x565317ed2f50 2024-04-25T00:18:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.636+0000 7f634dffb640 1 -- 172.21.15.27:0/2567709866 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6358084050 con 0x565317ed2f50 2024-04-25T00:18:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.636+0000 7f636f606740 10 client.25792 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.636+0000 7f636f606740 1 -- 172.21.15.27:0/2567709866 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x565317ee4b80 con 0x565317ed2f50 2024-04-25T00:18:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.638+0000 7f634dffb640 1 -- 172.21.15.27:0/2567709866 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x565317e3ef40 con 0x565317ed2f50 2024-04-25T00:18:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.638+0000 7f634dffb640 1 -- 172.21.15.27:0/2567709866 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f63580103e0 con 0x565317ed2f50 2024-04-25T00:18:04.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.638+0000 7f636f606740 10 client.25792 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:04.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.638+0000 7f636f606740 10 client.25792 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:18:04.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.638+0000 7f636f606740 20 client.25792 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:04.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.638+0000 7f636f606740 1 --2- 172.21.15.27:0/2567709866 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x565317ee5440 0x565317f782f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.638+0000 7f636f606740 4 client.25792 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:18:04.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.638+0000 7f636f606740 1 -- 172.21.15.27:0/2567709866 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x565317dbc6f0 con 0x565317ee5440 2024-04-25T00:18:04.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.644+0000 7f636659c640 1 --2- 172.21.15.27:0/2567709866 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x565317ee5440 0x565317f782f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.653+0000 7f636659c640 1 --2- 172.21.15.27:0/2567709866 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x565317ee5440 0x565317f782f0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97a7558b772783ed server_cookie=ab3fd411a9fafe68 in_seq=0 out_seq=0 2024-04-25T00:18:04.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.653+0000 7f634dffb640 10 client.25792 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:04.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.654+0000 7f634dffb640 1 -- 172.21.15.27:0/2567709866 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x565317dbc6f0 con 0x565317ee5440 2024-04-25T00:18:04.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.654+0000 7f634dffb640 10 client.25792 handle_command_reply: tid=0 2024-04-25T00:18:04.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.654+0000 7f636f606740 1 client.25792 shutdown 2024-04-25T00:18:04.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.655+0000 7f636f606740 1 -- 172.21.15.27:0/2567709866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317ed2f50 msgr2=0x565317ed7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:04.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.655+0000 7f636f606740 1 --2- 172.21.15.27:0/2567709866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317ed2f50 0x565317ed7830 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6358097f10 tx=0x7f635809b4e0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.655+0000 7f636f606740 1 -- 172.21.15.27:0/2567709866 shutdown_connections 2024-04-25T00:18:04.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.655+0000 7f636f606740 1 --2- 172.21.15.27:0/2567709866 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x565317ee5440 0x565317f782f0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.655+0000 7f636f606740 1 --2- 172.21.15.27:0/2567709866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565317edd620 0x565317ed7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.655+0000 7f636f606740 1 --2- 172.21.15.27:0/2567709866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565317ed2f50 0x565317ed7830 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.655+0000 7f636f606740 1 --2- 172.21.15.27:0/2567709866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565317ed2630 0x565317ee5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.655+0000 7f636f606740 1 -- 172.21.15.27:0/2567709866 >> 172.21.15.27:0/2567709866 conn(0x565317ece950 msgr2=0x565317ed8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:04.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.657+0000 7f634dffb640 0 client.25792 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:04.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.657+0000 7f636f606740 1 -- 172.21.15.27:0/2567709866 shutdown_connections 2024-04-25T00:18:04.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.657+0000 7f636f606740 1 -- 172.21.15.27:0/2567709866 wait complete. 2024-04-25T00:18:04.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.659+0000 7f636f606740 20 client.25792 trim_cache size 0 max 16384 2024-04-25T00:18:04.662 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:18:04.662 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T00:18:04.662 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T00:18:04.662 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:18:04.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.662+0000 7f636c81c640 1 -- 172.21.15.27:0/1018773710 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f63500a2ca0 msgr2=0x7f63500a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:04.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.662+0000 7f636c81c640 1 --2- 172.21.15.27:0/1018773710 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f63500a2ca0 0x7f63500a5160 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f635c01a040 tx=0x7f635c012040 comp rx=0 tx=0).stop 2024-04-25T00:18:04.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.662+0000 7f636c81c640 1 -- 172.21.15.27:0/1018773710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636810ff90 msgr2=0x7f636811a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:04.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.662+0000 7f636c81c640 1 --2- 172.21.15.27:0/1018773710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636810ff90 0x7f636811a9b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f635800b450 tx=0x7f635800b910 comp rx=0 tx=0).stop 2024-04-25T00:18:04.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.663+0000 7f636c81c640 1 -- 172.21.15.27:0/1018773710 shutdown_connections 2024-04-25T00:18:04.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.663+0000 7f636c81c640 1 --2- 172.21.15.27:0/1018773710 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f63500a2ca0 0x7f63500a5160 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.663+0000 7f636c81c640 1 --2- 172.21.15.27:0/1018773710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f636811af20 0x7f6368113740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.663+0000 7f636c81c640 1 --2- 172.21.15.27:0/1018773710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636810ff90 0x7f636811a9b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.663+0000 7f636c81c640 1 --2- 172.21.15.27:0/1018773710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f636810dfc0 0x7f636811a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.663+0000 7f636c81c640 1 -- 172.21.15.27:0/1018773710 >> 172.21.15.27:0/1018773710 conn(0x7f636806c850 msgr2=0x7f6368072240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:04.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.666+0000 7f636c81c640 1 -- 172.21.15.27:0/1018773710 shutdown_connections 2024-04-25T00:18:04.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.667+0000 7f636c81c640 1 -- 172.21.15.27:0/1018773710 wait complete. 2024-04-25T00:18:04.691 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T00:18:04.691 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.877+0000 7f42a1f7e640 1 -- 172.21.15.27:0/807829359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f429c0694f0 msgr2=0x7f429c100070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.877+0000 7f42a1f7e640 1 --2- 172.21.15.27:0/807829359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f429c0694f0 0x7f429c100070 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4290009900 tx=0x7f429002a400 comp rx=0 tx=0).stop 2024-04-25T00:18:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.877+0000 7f42a1f7e640 1 -- 172.21.15.27:0/807829359 shutdown_connections 2024-04-25T00:18:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.877+0000 7f42a1f7e640 1 --2- 172.21.15.27:0/807829359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c069950 0x7f429c1005b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.877+0000 7f42a1f7e640 1 --2- 172.21.15.27:0/807829359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f429c0694f0 0x7f429c100070 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.877+0000 7f42a1f7e640 1 --2- 172.21.15.27:0/807829359 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f429c105e40 0x7f429c106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.877+0000 7f42a1f7e640 1 -- 172.21.15.27:0/807829359 >> 172.21.15.27:0/807829359 conn(0x7f429c0759c0 msgr2=0x7f429c075dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.878+0000 7f42a1f7e640 1 -- 172.21.15.27:0/807829359 shutdown_connections 2024-04-25T00:18:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.878+0000 7f42a1f7e640 1 -- 172.21.15.27:0/807829359 wait complete. 2024-04-25T00:18:04.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.879+0000 7f42a1f7e640 1 Processor -- start 2024-04-25T00:18:04.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.879+0000 7f42a1f7e640 1 -- start start 2024-04-25T00:18:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.879+0000 7f42a1f7e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f429c0694f0 0x7f429c113d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.879+0000 7f42a1f7e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c069950 0x7f429c110e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.879+0000 7f42a1f7e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f429c105e40 0x7f429c111360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.879+0000 7f42a1f7e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f429c10f970 con 0x7f429c069950 2024-04-25T00:18:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.879+0000 7f42a1f7e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f429c10f7f0 con 0x7f429c0694f0 2024-04-25T00:18:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.879+0000 7f42a1f7e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f429c10faf0 con 0x7f429c105e40 2024-04-25T00:18:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.880+0000 7f429bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f429c105e40 0x7f429c111360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.880+0000 7f429bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f429c105e40 0x7f429c111360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:46886/0 (socket says 172.21.15.27:46886) 2024-04-25T00:18:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.880+0000 7f429bfff640 1 -- 172.21.15.27:0/1758759482 learned_addr learned my addr 172.21.15.27:0/1758759482 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.880+0000 7f429affd640 1 --2- 172.21.15.27:0/1758759482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c069950 0x7f429c110e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.880+0000 7f429b7fe640 1 --2- 172.21.15.27:0/1758759482 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f429c0694f0 0x7f429c113d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.880+0000 7f429affd640 1 -- 172.21.15.27:0/1758759482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f429c105e40 msgr2=0x7f429c111360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.880+0000 7f429affd640 1 --2- 172.21.15.27:0/1758759482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f429c105e40 0x7f429c111360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.881+0000 7f429affd640 1 -- 172.21.15.27:0/1758759482 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f429c0694f0 msgr2=0x7f429c113d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.881+0000 7f429affd640 1 --2- 172.21.15.27:0/1758759482 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f429c0694f0 0x7f429c113d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.881+0000 7f429affd640 1 -- 172.21.15.27:0/1758759482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f429c111c20 con 0x7f429c069950 2024-04-25T00:18:04.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.882+0000 7f429affd640 1 --2- 172.21.15.27:0/1758759482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c069950 0x7f429c110e00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f429002a9f0 tx=0x7f4290003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.882+0000 7f4298ff9640 1 -- 172.21.15.27:0/1758759482 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429003f070 con 0x7f429c069950 2024-04-25T00:18:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.882+0000 7f4298ff9640 1 -- 172.21.15.27:0/1758759482 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4290031c90 con 0x7f429c069950 2024-04-25T00:18:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.883+0000 7f4298ff9640 1 -- 172.21.15.27:0/1758759482 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4290035030 con 0x7f429c069950 2024-04-25T00:18:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.883+0000 7f42a1f7e640 1 -- 172.21.15.27:0/1758759482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f429c06b220 con 0x7f429c069950 2024-04-25T00:18:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.884+0000 7f42a1f7e640 1 -- 172.21.15.27:0/1758759482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f429c06b690 con 0x7f429c069950 2024-04-25T00:18:04.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.886+0000 7f42a1f7e640 1 -- 172.21.15.27:0/1758759482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f429c1014c0 con 0x7f429c069950 2024-04-25T00:18:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.888+0000 7f4298ff9640 1 -- 172.21.15.27:0/1758759482 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f429003bd10 con 0x7f429c069950 2024-04-25T00:18:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.890+0000 7f4298ff9640 1 --2- 172.21.15.27:0/1758759482 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f42840a2d60 0x7f42840a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.890+0000 7f4298ff9640 1 -- 172.21.15.27:0/1758759482 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f42900efba0 con 0x7f429c069950 2024-04-25T00:18:04.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.895+0000 7f429b7fe640 1 --2- 172.21.15.27:0/1758759482 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f42840a2d60 0x7f42840a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:04.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.896+0000 7f429b7fe640 1 --2- 172.21.15.27:0/1758759482 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f42840a2d60 0x7f42840a5220 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f428c00bb60 tx=0x7f428c00b390 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:04.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:04.899+0000 7f4298ff9640 1 -- 172.21.15.27:0/1758759482 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4290049020 con 0x7f429c069950 2024-04-25T00:18:04.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:04 smithi027 ceph-mon[29208]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 159 MiB/s wr, 19.68k op/s 2024-04-25T00:18:05.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:04 smithi139 ceph-mon[32833]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 159 MiB/s wr, 19.68k op/s 2024-04-25T00:18:05.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.273+0000 7f42a1f7e640 1 -- 172.21.15.27:0/1758759482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f429c1122e0 con 0x7f429c069950 2024-04-25T00:18:05.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.275+0000 7f4298ff9640 1 -- 172.21.15.27:0/1758759482 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11951 (secure 0 0 0) 0x7f42900bc410 con 0x7f429c069950 2024-04-25T00:18:05.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:04 smithi153 ceph-mon[32896]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 159 MiB/s wr, 19.68k op/s 2024-04-25T00:18:05.277 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:18:05.277 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:17:08.383555+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24496,24428,24521,24514,24430]},"id":1}]} 2024-04-25T00:18:05.277 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 30 2024-04-25T00:18:05.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.279+0000 7f42a1f7e640 1 -- 172.21.15.27:0/1758759482 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f42840a2d60 msgr2=0x7f42840a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.279+0000 7f42a1f7e640 1 --2- 172.21.15.27:0/1758759482 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f42840a2d60 0x7f42840a5220 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f428c00bb60 tx=0x7f428c00b390 comp rx=0 tx=0).stop 2024-04-25T00:18:05.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.279+0000 7f42a1f7e640 1 -- 172.21.15.27:0/1758759482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c069950 msgr2=0x7f429c110e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.279+0000 7f42a1f7e640 1 --2- 172.21.15.27:0/1758759482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c069950 0x7f429c110e00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f429002a9f0 tx=0x7f4290003b90 comp rx=0 tx=0).stop 2024-04-25T00:18:05.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.279+0000 7f42a1f7e640 1 -- 172.21.15.27:0/1758759482 shutdown_connections 2024-04-25T00:18:05.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.279+0000 7f42a1f7e640 1 --2- 172.21.15.27:0/1758759482 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f42840a2d60 0x7f42840a5220 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.279+0000 7f42a1f7e640 1 --2- 172.21.15.27:0/1758759482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f429c105e40 0x7f429c111360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.279+0000 7f42a1f7e640 1 --2- 172.21.15.27:0/1758759482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c069950 0x7f429c110e00 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.279+0000 7f42a1f7e640 1 --2- 172.21.15.27:0/1758759482 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f429c0694f0 0x7f429c113d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.279+0000 7f42a1f7e640 1 -- 172.21.15.27:0/1758759482 >> 172.21.15.27:0/1758759482 conn(0x7f429c0759c0 msgr2=0x7f429c0fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.279+0000 7f42a1f7e640 1 -- 172.21.15.27:0/1758759482 shutdown_connections 2024-04-25T00:18:05.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.279+0000 7f42a1f7e640 1 -- 172.21.15.27:0/1758759482 wait complete. 2024-04-25T00:18:05.301 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T00:18:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.459+0000 7f30a6584640 1 -- 172.21.15.27:0/1134339938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30a01009a0 msgr2=0x7f30a01049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.459+0000 7f30a6584640 1 --2- 172.21.15.27:0/1134339938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30a01009a0 0x7f30a01049e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f3094009900 tx=0x7f309402a410 comp rx=0 tx=0).stop 2024-04-25T00:18:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.459+0000 7f30a6584640 1 -- 172.21.15.27:0/1134339938 shutdown_connections 2024-04-25T00:18:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.459+0000 7f30a6584640 1 --2- 172.21.15.27:0/1134339938 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30a01050c0 0x7f30a010cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.460+0000 7f30a6584640 1 --2- 172.21.15.27:0/1134339938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30a01009a0 0x7f30a01049e0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.460+0000 7f30a6584640 1 --2- 172.21.15.27:0/1134339938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a00ffff0 0x7f30a01003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.460+0000 7f30a6584640 1 -- 172.21.15.27:0/1134339938 >> 172.21.15.27:0/1134339938 conn(0x7f30a0076be0 msgr2=0x7f30a00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.460+0000 7f30a6584640 1 -- 172.21.15.27:0/1134339938 shutdown_connections 2024-04-25T00:18:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.460+0000 7f30a6584640 1 -- 172.21.15.27:0/1134339938 wait complete. 2024-04-25T00:18:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.460+0000 7f30a6584640 1 Processor -- start 2024-04-25T00:18:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.461+0000 7f30a6584640 1 -- start start 2024-04-25T00:18:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.461+0000 7f30a6584640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a00ffff0 0x7f30a01a4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.461+0000 7f30a6584640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30a01050c0 0x7f30a01a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.461+0000 7f30a6584640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30a01a2f80 0x7f30a01a3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.461+0000 7f30a6584640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f30a0110150 con 0x7f30a01050c0 2024-04-25T00:18:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.461+0000 7f30a6584640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f30a010ffd0 con 0x7f30a01a2f80 2024-04-25T00:18:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.461+0000 7f30a6584640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f30a01102d0 con 0x7f30a00ffff0 2024-04-25T00:18:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.462+0000 7f309f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30a01050c0 0x7f30a01a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.462+0000 7f309f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30a01050c0 0x7f30a01a4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51790/0 (socket says 172.21.15.27:51790) 2024-04-25T00:18:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.462+0000 7f309f7fe640 1 -- 172.21.15.27:0/3383529480 learned_addr learned my addr 172.21.15.27:0/3383529480 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.462+0000 7f30a4b21640 1 --2- 172.21.15.27:0/3383529480 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30a01a2f80 0x7f30a01a3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.462+0000 7f309f7fe640 1 -- 172.21.15.27:0/3383529480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a00ffff0 msgr2=0x7f30a01a4220 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:18:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.462+0000 7f309f7fe640 1 --2- 172.21.15.27:0/3383529480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a00ffff0 0x7f30a01a4220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.462+0000 7f309f7fe640 1 -- 172.21.15.27:0/3383529480 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30a01a2f80 msgr2=0x7f30a01a3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.462+0000 7f309f7fe640 1 --2- 172.21.15.27:0/3383529480 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30a01a2f80 0x7f30a01a3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.462+0000 7f309f7fe640 1 -- 172.21.15.27:0/3383529480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30a01a3a70 con 0x7f30a01050c0 2024-04-25T00:18:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.462+0000 7f309f7fe640 1 --2- 172.21.15.27:0/3383529480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30a01050c0 0x7f30a01a4760 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f3094005940 tx=0x7f3094004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.463+0000 7f309d7fa640 1 -- 172.21.15.27:0/3383529480 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309403f030 con 0x7f30a01050c0 2024-04-25T00:18:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.463+0000 7f30a6584640 1 -- 172.21.15.27:0/3383529480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30a01a3d00 con 0x7f30a01050c0 2024-04-25T00:18:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.463+0000 7f30a6584640 1 -- 172.21.15.27:0/3383529480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30a01ab560 con 0x7f30a01050c0 2024-04-25T00:18:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.463+0000 7f309d7fa640 1 -- 172.21.15.27:0/3383529480 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3094003ad0 con 0x7f30a01050c0 2024-04-25T00:18:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.464+0000 7f309d7fa640 1 -- 172.21.15.27:0/3383529480 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3094049060 con 0x7f30a01050c0 2024-04-25T00:18:05.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.467+0000 7f30a9368740 1 Processor -- start 2024-04-25T00:18:05.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.467+0000 7f30a9368740 1 -- start start 2024-04-25T00:18:05.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.468+0000 7f30a9368740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df13580 0x558c7df13960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.468+0000 7f30a9368740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df13ea0 0x558c7df1e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.468+0000 7f30a9368740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df1e570 0x558c7df20960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.468+0000 7f30a9368740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558c7de78670 con 0x558c7df1e570 2024-04-25T00:18:05.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.468+0000 7f30a9368740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558c7de20970 con 0x558c7df13ea0 2024-04-25T00:18:05.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.468+0000 7f30a9368740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558c7de74630 con 0x558c7df13580 2024-04-25T00:18:05.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.468+0000 7f309ffff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df13ea0 0x558c7df1e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.468+0000 7f309ffff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df13ea0 0x558c7df1e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:59910/0 (socket says 172.21.15.27:59910) 2024-04-25T00:18:05.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.468+0000 7f309ffff640 1 -- 172.21.15.27:0/1330187148 learned_addr learned my addr 172.21.15.27:0/1330187148 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:05.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.468+0000 7f30a4b21640 1 --2- 172.21.15.27:0/1330187148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df13580 0x558c7df13960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.468+0000 7f309d7fa640 1 -- 172.21.15.27:0/3383529480 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3094042030 con 0x7f30a01050c0 2024-04-25T00:18:05.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.469+0000 7f309ffff640 1 -- 172.21.15.27:0/1330187148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df13580 msgr2=0x558c7df13960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.469+0000 7f309ffff640 1 --2- 172.21.15.27:0/1330187148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df13580 0x558c7df13960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.469+0000 7f309f7fe640 1 --2- 172.21.15.27:0/1330187148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df1e570 0x558c7df20960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.469+0000 7f309ffff640 1 -- 172.21.15.27:0/1330187148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df1e570 msgr2=0x558c7df20960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.469+0000 7f309ffff640 1 --2- 172.21.15.27:0/1330187148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df1e570 0x558c7df20960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.469+0000 7f309ffff640 1 -- 172.21.15.27:0/1330187148 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c7de44400 con 0x558c7df13ea0 2024-04-25T00:18:05.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.469+0000 7f30a4b21640 1 --2- 172.21.15.27:0/1330187148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df13580 0x558c7df13960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:18:05.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.469+0000 7f309f7fe640 1 --2- 172.21.15.27:0/1330187148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df1e570 0x558c7df20960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:05.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.469+0000 7f309ffff640 1 --2- 172.21.15.27:0/1330187148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df13ea0 0x558c7df1e030 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3090009bd0 tx=0x7f309000e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=682b1792a9867af1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.469+0000 7f30867fc640 1 -- 172.21.15.27:0/1330187148 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3090005020 con 0x558c7df13ea0 2024-04-25T00:18:05.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.469+0000 7f30867fc640 1 -- 172.21.15.27:0/1330187148 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30900047a0 con 0x558c7df13ea0 2024-04-25T00:18:05.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.469+0000 7f309d7fa640 1 --2- 172.21.15.27:0/3383529480 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f30880a2d80 0x7f30880a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.470+0000 7f30867fc640 1 -- 172.21.15.27:0/1330187148 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3090010690 con 0x558c7df13ea0 2024-04-25T00:18:05.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.470+0000 7f30a4b21640 1 --2- 172.21.15.27:0/3383529480 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f30880a2d80 0x7f30880a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.470+0000 7f309d7fa640 1 -- 172.21.15.27:0/3383529480 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f30940efbb0 con 0x7f30a01050c0 2024-04-25T00:18:05.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.470+0000 7f30a9368740 1 -- 172.21.15.27:0/1330187148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df13ea0 msgr2=0x558c7df1e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.470+0000 7f30a4b21640 1 --2- 172.21.15.27:0/3383529480 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f30880a2d80 0x7f30880a5240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f309800af00 tx=0x7f309800a4d0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.470+0000 7f30a9368740 1 --2- 172.21.15.27:0/1330187148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df13ea0 0x558c7df1e030 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3090009bd0 tx=0x7f309000e720 comp rx=0 tx=0).stop 2024-04-25T00:18:05.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.470+0000 7f30a9368740 1 -- 172.21.15.27:0/1330187148 shutdown_connections 2024-04-25T00:18:05.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.470+0000 7f30a9368740 1 --2- 172.21.15.27:0/1330187148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df1e570 0x558c7df20960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.470+0000 7f30a9368740 1 --2- 172.21.15.27:0/1330187148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df13ea0 0x558c7df1e030 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.470+0000 7f30a9368740 1 --2- 172.21.15.27:0/1330187148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df13580 0x558c7df13960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.471+0000 7f30a9368740 1 -- 172.21.15.27:0/1330187148 >> 172.21.15.27:0/1330187148 conn(0x558c7df0f8a0 msgr2=0x558c7df0fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.471+0000 7f30a9368740 1 -- 172.21.15.27:0/1330187148 shutdown_connections 2024-04-25T00:18:05.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.471+0000 7f30a9368740 1 -- 172.21.15.27:0/1330187148 wait complete. 2024-04-25T00:18:05.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.471+0000 7f30a9368740 1 -- start start 2024-04-25T00:18:05.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.471+0000 7f30a9368740 1 Processor -- start 2024-04-25T00:18:05.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f30a9368740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df13580 0x558c7df1ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f30a9368740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 0x558c7df1b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f30a9368740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df1e570 0x558c7df1b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f30a9368740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558c7df21400 con 0x558c7df13ea0 2024-04-25T00:18:05.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f30a9368740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558c7de34050 con 0x558c7df1e570 2024-04-25T00:18:05.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f30a9368740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558c7df21580 con 0x558c7df13580 2024-04-25T00:18:05.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f309f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 0x558c7df1b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f30a9368740 10 client.0 _ll_register_callbacks cb 0x558c7db8bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:05.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f309f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 0x558c7df1b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51810/0 (socket says 172.21.15.27:51810) 2024-04-25T00:18:05.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f309f7fe640 1 -- 172.21.15.27:0/70913570 learned_addr learned my addr 172.21.15.27:0/70913570 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:05.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f30a4b21640 1 --2- 172.21.15.27:0/70913570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df1e570 0x558c7df1b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f309ffff640 1 --2- 172.21.15.27:0/70913570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df13580 0x558c7df1ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f309f7fe640 1 -- 172.21.15.27:0/70913570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df13580 msgr2=0x558c7df1ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f309f7fe640 1 --2- 172.21.15.27:0/70913570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df13580 0x558c7df1ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f309f7fe640 1 -- 172.21.15.27:0/70913570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df1e570 msgr2=0x558c7df1b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f309f7fe640 1 --2- 172.21.15.27:0/70913570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df1e570 0x558c7df1b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f309f7fe640 1 -- 172.21.15.27:0/70913570 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c7de05dc0 con 0x558c7df13ea0 2024-04-25T00:18:05.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f30a4b21640 1 --2- 172.21.15.27:0/70913570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df1e570 0x558c7df1b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:18:05.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.472+0000 7f309ffff640 1 --2- 172.21.15.27:0/70913570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df13580 0x558c7df1ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:18:05.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.473+0000 7f309f7fe640 1 --2- 172.21.15.27:0/70913570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 0x558c7df1b0b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f30940bbcb0 tx=0x7f30940bb740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41341cb483e9b2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.473+0000 7f307f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:18:05.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.473+0000 7f307f7fe640 1 -- 172.21.15.27:0/70913570 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3094035070 con 0x558c7df13ea0 2024-04-25T00:18:05.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.473+0000 7f307f7fe640 1 -- 172.21.15.27:0/70913570 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f30940b8390 con 0x558c7df13ea0 2024-04-25T00:18:05.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.473+0000 7f30a9368740 1 -- 172.21.15.27:0/70913570 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30900047a0 con 0x558c7df13ea0 2024-04-25T00:18:05.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.473+0000 7f307f7fe640 1 -- 172.21.15.27:0/70913570 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f30940bbd20 con 0x558c7df13ea0 2024-04-25T00:18:05.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.473+0000 7f30a9368740 10 client.16302 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:05.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.473+0000 7f30a9368740 1 -- 172.21.15.27:0/70913570 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558c7db8c3f0 con 0x558c7df13ea0 2024-04-25T00:18:05.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.474+0000 7f307f7fe640 1 -- 172.21.15.27:0/70913570 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30900047a0 con 0x558c7df13ea0 2024-04-25T00:18:05.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.474+0000 7f307f7fe640 1 -- 172.21.15.27:0/70913570 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f30940bbf80 con 0x558c7df13ea0 2024-04-25T00:18:05.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.474+0000 7f30a9368740 10 client.16302 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:05.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.474+0000 7f30a9368740 10 client.16302 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:18:05.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.474+0000 7f30a9368740 20 client.16302 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:05.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.474+0000 7f30a9368740 1 --2- 172.21.15.27:0/70913570 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x558c7df2e5b0 0x558c7df30a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.474+0000 7f30a9368740 4 client.16302 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:18:05.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.474+0000 7f30a9368740 1 -- 172.21.15.27:0/70913570 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c7de85bb0 con 0x558c7df2e5b0 2024-04-25T00:18:05.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.474+0000 7f309ffff640 1 --2- 172.21.15.27:0/70913570 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x558c7df2e5b0 0x558c7df30a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.475+0000 7f309ffff640 1 --2- 172.21.15.27:0/70913570 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x558c7df2e5b0 0x558c7df30a10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ca97e53d4a39daee server_cookie=130cc7ab8bdecf44 in_seq=0 out_seq=0 2024-04-25T00:18:05.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.475+0000 7f307f7fe640 10 client.16302 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:18:05.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.476+0000 7f307f7fe640 1 -- 172.21.15.27:0/70913570 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c7de85bb0 con 0x558c7df2e5b0 2024-04-25T00:18:05.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.476+0000 7f307f7fe640 10 client.16302 handle_command_reply: tid=0 2024-04-25T00:18:05.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.476+0000 7f30a9368740 1 client.16302 shutdown 2024-04-25T00:18:05.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.477+0000 7f30a9368740 1 -- 172.21.15.27:0/70913570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 msgr2=0x558c7df1b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.477+0000 7f30a9368740 1 --2- 172.21.15.27:0/70913570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 0x558c7df1b0b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f30940bbcb0 tx=0x7f30940bb740 comp rx=0 tx=0).stop 2024-04-25T00:18:05.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.477+0000 7f30a9368740 1 -- 172.21.15.27:0/70913570 shutdown_connections 2024-04-25T00:18:05.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.477+0000 7f30a9368740 1 --2- 172.21.15.27:0/70913570 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x558c7df2e5b0 0x558c7df30a10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.478+0000 7f30a9368740 1 --2- 172.21.15.27:0/70913570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df1e570 0x558c7df1b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.478+0000 7f307f7fe640 0 client.16302 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:18:05.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.478+0000 7f30a9368740 1 --2- 172.21.15.27:0/70913570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 0x558c7df1b0b0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.478+0000 7f30a9368740 1 --2- 172.21.15.27:0/70913570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df13580 0x558c7df1ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.478+0000 7f30a9368740 1 -- 172.21.15.27:0/70913570 >> 172.21.15.27:0/70913570 conn(0x558c7df0f8a0 msgr2=0x558c7df0eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.478+0000 7f30a9368740 1 -- 172.21.15.27:0/70913570 shutdown_connections 2024-04-25T00:18:05.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.478+0000 7f30a9368740 1 -- 172.21.15.27:0/70913570 wait complete. 2024-04-25T00:18:05.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.478+0000 7f30a9368740 20 client.16302 trim_cache size 0 max 16384 2024-04-25T00:18:05.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.511+0000 7f30a9368740 1 Processor -- start 2024-04-25T00:18:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.511+0000 7f30a9368740 1 -- start start 2024-04-25T00:18:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.511+0000 7f30a9368740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df13580 0x558c7df0f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.511+0000 7f30a9368740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 0x558c7df2a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.511+0000 7f30a9368740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df1e570 0x558c7df36320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.511+0000 7f30a9368740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f308c00fe90 con 0x558c7df13ea0 2024-04-25T00:18:05.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.511+0000 7f30a9368740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f308c011a90 con 0x558c7df13580 2024-04-25T00:18:05.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.511+0000 7f30a9368740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f308c010560 con 0x558c7df1e570 2024-04-25T00:18:05.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.513+0000 7f30a4b21640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df1e570 0x558c7df36320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.513+0000 7f30a4b21640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df1e570 0x558c7df36320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:46934/0 (socket says 172.21.15.27:46934) 2024-04-25T00:18:05.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.513+0000 7f30a4b21640 1 -- 172.21.15.27:0/326943048 learned_addr learned my addr 172.21.15.27:0/326943048 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.513+0000 7f309f7fe640 1 --2- 172.21.15.27:0/326943048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 0x558c7df2a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.513+0000 7f309ffff640 1 --2- 172.21.15.27:0/326943048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df13580 0x558c7df0f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.514+0000 7f309f7fe640 1 -- 172.21.15.27:0/326943048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df1e570 msgr2=0x558c7df36320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.514+0000 7f309f7fe640 1 --2- 172.21.15.27:0/326943048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df1e570 0x558c7df36320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.514+0000 7f309f7fe640 1 -- 172.21.15.27:0/326943048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df13580 msgr2=0x558c7df0f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.514+0000 7f309f7fe640 1 --2- 172.21.15.27:0/326943048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df13580 0x558c7df0f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.514+0000 7f309f7fe640 1 -- 172.21.15.27:0/326943048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c7dc567d0 con 0x558c7df13ea0 2024-04-25T00:18:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.514+0000 7f309f7fe640 1 --2- 172.21.15.27:0/326943048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 0x558c7df2a2e0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f30940b9fd0 tx=0x7f30940bb7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27dcaf747394015d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.515+0000 7f3085ffb640 1 -- 172.21.15.27:0/326943048 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3094035070 con 0x558c7df13ea0 2024-04-25T00:18:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.515+0000 7f3085ffb640 1 -- 172.21.15.27:0/326943048 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30940c00d0 con 0x558c7df13ea0 2024-04-25T00:18:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.515+0000 7f3085ffb640 1 -- 172.21.15.27:0/326943048 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30940bbf80 con 0x558c7df13ea0 2024-04-25T00:18:05.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.516+0000 7f30a9368740 1 -- 172.21.15.27:0/326943048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 msgr2=0x558c7df2a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.516+0000 7f30a9368740 1 --2- 172.21.15.27:0/326943048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 0x558c7df2a2e0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f30940b9fd0 tx=0x7f30940bb7b0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.516+0000 7f30a9368740 1 -- 172.21.15.27:0/326943048 shutdown_connections 2024-04-25T00:18:05.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.516+0000 7f30a9368740 1 --2- 172.21.15.27:0/326943048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df1e570 0x558c7df36320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.516+0000 7f30a9368740 1 --2- 172.21.15.27:0/326943048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 0x558c7df2a2e0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.516+0000 7f30a9368740 1 --2- 172.21.15.27:0/326943048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df13580 0x558c7df0f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.516+0000 7f30a9368740 1 -- 172.21.15.27:0/326943048 >> 172.21.15.27:0/326943048 conn(0x558c7df0f8a0 msgr2=0x558c7df29bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.518+0000 7f30a9368740 1 -- 172.21.15.27:0/326943048 shutdown_connections 2024-04-25T00:18:05.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.521+0000 7f30a9368740 1 -- 172.21.15.27:0/326943048 wait complete. 2024-04-25T00:18:05.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.521+0000 7f30a9368740 1 -- start start 2024-04-25T00:18:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.521+0000 7f30a9368740 1 Processor -- start 2024-04-25T00:18:05.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.522+0000 7f30a9368740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df13580 0x558c7df268b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.522+0000 7f30a9368740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 0x558c7df26df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.522+0000 7f30a9368740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df1e570 0x558c7df24e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.522+0000 7f30a9368740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f308c011300 con 0x558c7df13ea0 2024-04-25T00:18:05.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.522+0000 7f30a9368740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f308c010c30 con 0x558c7df13580 2024-04-25T00:18:05.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.522+0000 7f30a9368740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f308c00de50 con 0x558c7df1e570 2024-04-25T00:18:05.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.522+0000 7f30a9368740 10 client.0 _ll_register_callbacks cb 0x7f3094005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:05.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.523+0000 7f309f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 0x558c7df26df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.523+0000 7f309f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 0x558c7df26df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51828/0 (socket says 172.21.15.27:51828) 2024-04-25T00:18:05.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.523+0000 7f309f7fe640 1 -- 172.21.15.27:0/789975251 learned_addr learned my addr 172.21.15.27:0/789975251 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:05.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.523+0000 7f30a4b21640 1 --2- 172.21.15.27:0/789975251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df1e570 0x558c7df24e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.524+0000 7f30a4b21640 1 -- 172.21.15.27:0/789975251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df13580 msgr2=0x558c7df268b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:18:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.524+0000 7f30a4b21640 1 --2- 172.21.15.27:0/789975251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df13580 0x558c7df268b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.524+0000 7f30a4b21640 1 -- 172.21.15.27:0/789975251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 msgr2=0x558c7df26df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.524+0000 7f30a4b21640 1 --2- 172.21.15.27:0/789975251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 0x558c7df26df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.524+0000 7f30a4b21640 1 -- 172.21.15.27:0/789975251 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c7df25650 con 0x558c7df1e570 2024-04-25T00:18:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.524+0000 7f30a4b21640 1 --2- 172.21.15.27:0/789975251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df1e570 0x558c7df24e50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3098005fb0 tx=0x7f3098004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf231223526ecdb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.524+0000 7f307f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:18:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.524+0000 7f307f7fe640 1 -- 172.21.15.27:0/789975251 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309802c060 con 0x558c7df1e570 2024-04-25T00:18:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.524+0000 7f30a9368740 1 -- 172.21.15.27:0/789975251 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30940c00d0 con 0x558c7df1e570 2024-04-25T00:18:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.524+0000 7f307f7fe640 1 -- 172.21.15.27:0/789975251 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f309801edc0 con 0x558c7df1e570 2024-04-25T00:18:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.525+0000 7f307f7fe640 1 -- 172.21.15.27:0/789975251 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3098033050 con 0x558c7df1e570 2024-04-25T00:18:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.525+0000 7f30a9368740 10 client.25813 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:05.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.525+0000 7f30a9368740 1 -- 172.21.15.27:0/789975251 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558c7df8f8a0 con 0x558c7df1e570 2024-04-25T00:18:05.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.525+0000 7f307f7fe640 1 -- 172.21.15.27:0/789975251 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30940c00d0 con 0x558c7df1e570 2024-04-25T00:18:05.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.525+0000 7f307f7fe640 1 -- 172.21.15.27:0/789975251 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f3098026070 con 0x558c7df1e570 2024-04-25T00:18:05.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.525+0000 7f30a9368740 10 client.25813 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:05.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.525+0000 7f30a9368740 10 client.25813 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:18:05.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.525+0000 7f30a9368740 20 client.25813 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:05.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.525+0000 7f30a9368740 1 --2- 172.21.15.27:0/789975251 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x558c7df900a0 0x558c7df904b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.525+0000 7f30a9368740 4 client.25813 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:18:05.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.525+0000 7f30a9368740 1 -- 172.21.15.27:0/789975251 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558c7de7ff00 con 0x558c7df900a0 2024-04-25T00:18:05.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.530+0000 7f309ffff640 1 --2- 172.21.15.27:0/789975251 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x558c7df900a0 0x558c7df904b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.531+0000 7f309ffff640 1 --2- 172.21.15.27:0/789975251 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x558c7df900a0 0x558c7df904b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=23cd631b73166079 server_cookie=655bc8bb4442af9 in_seq=0 out_seq=0 2024-04-25T00:18:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.531+0000 7f307f7fe640 10 client.25813 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:18:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.532+0000 7f307f7fe640 1 -- 172.21.15.27:0/789975251 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558c7de7ff00 con 0x558c7df900a0 2024-04-25T00:18:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.532+0000 7f307f7fe640 10 client.25813 handle_command_reply: tid=0 2024-04-25T00:18:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.532+0000 7f30a9368740 1 client.25813 shutdown 2024-04-25T00:18:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.532+0000 7f30a9368740 1 -- 172.21.15.27:0/789975251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df1e570 msgr2=0x558c7df24e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.532+0000 7f30a9368740 1 --2- 172.21.15.27:0/789975251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df1e570 0x558c7df24e50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3098005fb0 tx=0x7f3098004a40 comp rx=0 tx=0).stop 2024-04-25T00:18:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.532+0000 7f30a9368740 1 -- 172.21.15.27:0/789975251 shutdown_connections 2024-04-25T00:18:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.532+0000 7f30a9368740 1 --2- 172.21.15.27:0/789975251 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x558c7df900a0 0x558c7df904b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.532+0000 7f30a9368740 1 --2- 172.21.15.27:0/789975251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c7df1e570 0x558c7df24e50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.532+0000 7f30a9368740 1 --2- 172.21.15.27:0/789975251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c7df13ea0 0x558c7df26df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.532+0000 7f30a9368740 1 --2- 172.21.15.27:0/789975251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c7df13580 0x558c7df268b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.532+0000 7f30a9368740 1 -- 172.21.15.27:0/789975251 >> 172.21.15.27:0/789975251 conn(0x558c7df0f8a0 msgr2=0x558c7df29340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.532+0000 7f307f7fe640 0 client.25813 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:18:05.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.538+0000 7f30a9368740 1 -- 172.21.15.27:0/789975251 shutdown_connections 2024-04-25T00:18:05.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.539+0000 7f30a9368740 1 -- 172.21.15.27:0/789975251 wait complete. 2024-04-25T00:18:05.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.539+0000 7f30a9368740 20 client.25813 trim_cache size 0 max 16384 2024-04-25T00:18:05.542 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:18:05.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.542+0000 7f30a6584640 1 -- 172.21.15.27:0/3383529480 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f30880a2d80 msgr2=0x7f30880a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.542+0000 7f30a6584640 1 --2- 172.21.15.27:0/3383529480 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f30880a2d80 0x7f30880a5240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f309800af00 tx=0x7f309800a4d0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.542+0000 7f30a6584640 1 -- 172.21.15.27:0/3383529480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30a01050c0 msgr2=0x7f30a01a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.542+0000 7f30a6584640 1 --2- 172.21.15.27:0/3383529480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30a01050c0 0x7f30a01a4760 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f3094005940 tx=0x7f3094004270 comp rx=0 tx=0).stop 2024-04-25T00:18:05.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.545+0000 7f30a6584640 1 -- 172.21.15.27:0/3383529480 shutdown_connections 2024-04-25T00:18:05.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.545+0000 7f30a6584640 1 --2- 172.21.15.27:0/3383529480 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f30880a2d80 0x7f30880a5240 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.545+0000 7f30a6584640 1 --2- 172.21.15.27:0/3383529480 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30a01a2f80 0x7f30a01a3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.545+0000 7f30a6584640 1 --2- 172.21.15.27:0/3383529480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30a01050c0 0x7f30a01a4760 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.545+0000 7f30a6584640 1 --2- 172.21.15.27:0/3383529480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30a00ffff0 0x7f30a01a4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.545+0000 7f30a6584640 1 -- 172.21.15.27:0/3383529480 >> 172.21.15.27:0/3383529480 conn(0x7f30a0076be0 msgr2=0x7f30a0102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.546+0000 7f30a6584640 1 -- 172.21.15.27:0/3383529480 shutdown_connections 2024-04-25T00:18:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.546+0000 7f30a6584640 1 -- 172.21.15.27:0/3383529480 wait complete. 2024-04-25T00:18:05.568 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T00:18:05.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.737+0000 7f1a331a2640 1 -- 172.21.15.27:0/2958092264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a2c1066b0 msgr2=0x7f1a2c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.737+0000 7f1a331a2640 1 --2- 172.21.15.27:0/2958092264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a2c1066b0 0x7f1a2c106a90 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009930 tx=0x7f1a2002a420 comp rx=0 tx=0).stop 2024-04-25T00:18:05.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.737+0000 7f1a331a2640 1 -- 172.21.15.27:0/2958092264 shutdown_connections 2024-04-25T00:18:05.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.737+0000 7f1a331a2640 1 --2- 172.21.15.27:0/2958092264 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1a2c100fd0 0x7f1a2c10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.737+0000 7f1a331a2640 1 --2- 172.21.15.27:0/2958092264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a2c100610 0x7f1a2c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.737+0000 7f1a331a2640 1 --2- 172.21.15.27:0/2958092264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a2c1066b0 0x7f1a2c106a90 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.738+0000 7f1a331a2640 1 -- 172.21.15.27:0/2958092264 >> 172.21.15.27:0/2958092264 conn(0x7f1a2c077740 msgr2=0x7f1a2c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.738+0000 7f1a331a2640 1 -- 172.21.15.27:0/2958092264 shutdown_connections 2024-04-25T00:18:05.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.738+0000 7f1a331a2640 1 -- 172.21.15.27:0/2958092264 wait complete. 2024-04-25T00:18:05.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.738+0000 7f1a331a2640 1 Processor -- start 2024-04-25T00:18:05.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.738+0000 7f1a331a2640 1 -- start start 2024-04-25T00:18:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.738+0000 7f1a331a2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a2c100610 0x7f1a2c198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.738+0000 7f1a331a2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1a2c100fd0 0x7f1a2c1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.738+0000 7f1a331a2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a2c1066b0 0x7f1a2c19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.739+0000 7f1a331a2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c111b30 con 0x7f1a2c1066b0 2024-04-25T00:18:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.739+0000 7f1a331a2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c1119b0 con 0x7f1a2c100fd0 2024-04-25T00:18:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.739+0000 7f1a331a2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c111cb0 con 0x7f1a2c100610 2024-04-25T00:18:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.739+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1a2c100fd0 0x7f1a2c1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.739+0000 7f1a3173f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a2c1066b0 0x7f1a2c19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.739+0000 7f1a30f3e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a2c100610 0x7f1a2c198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.739+0000 7f1a30f3e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a2c100610 0x7f1a2c198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:46958/0 (socket says 172.21.15.27:46958) 2024-04-25T00:18:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.739+0000 7f1a3173f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a2c1066b0 0x7f1a2c19c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51846/0 (socket says 172.21.15.27:51846) 2024-04-25T00:18:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.739+0000 7f1a30f3e640 1 -- 172.21.15.27:0/3674655151 learned_addr learned my addr 172.21.15.27:0/3674655151 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.739+0000 7f1a3173f640 1 -- 172.21.15.27:0/3674655151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a2c100610 msgr2=0x7f1a2c198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.739+0000 7f1a3173f640 1 --2- 172.21.15.27:0/3674655151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a2c100610 0x7f1a2c198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.739+0000 7f1a3173f640 1 -- 172.21.15.27:0/3674655151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1a2c100fd0 msgr2=0x7f1a2c1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.739+0000 7f1a3173f640 1 --2- 172.21.15.27:0/3674655151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1a2c100fd0 0x7f1a2c1985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.739+0000 7f1a3173f640 1 -- 172.21.15.27:0/3674655151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a2c19d060 con 0x7f1a2c1066b0 2024-04-25T00:18:05.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.739+0000 7f1a3173f640 1 --2- 172.21.15.27:0/3674655151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a2c1066b0 0x7f1a2c19c930 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009fd0 tx=0x7f1a2400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.740+0000 7f1a29ffb640 1 -- 172.21.15.27:0/3674655151 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2400ce20 con 0x7f1a2c1066b0 2024-04-25T00:18:05.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.740+0000 7f1a331a2640 1 -- 172.21.15.27:0/3674655151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a2c19d350 con 0x7f1a2c1066b0 2024-04-25T00:18:05.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.740+0000 7f1a29ffb640 1 -- 172.21.15.27:0/3674655151 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a24010430 con 0x7f1a2c1066b0 2024-04-25T00:18:05.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.740+0000 7f1a29ffb640 1 -- 172.21.15.27:0/3674655151 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2400f620 con 0x7f1a2c1066b0 2024-04-25T00:18:05.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.740+0000 7f1a331a2640 1 -- 172.21.15.27:0/3674655151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a2c10dea0 con 0x7f1a2c1066b0 2024-04-25T00:18:05.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.743+0000 7f1a29ffb640 1 -- 172.21.15.27:0/3674655151 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1a24010950 con 0x7f1a2c1066b0 2024-04-25T00:18:05.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.743+0000 7f1a35f91740 1 Processor -- start 2024-04-25T00:18:05.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.743+0000 7f1a35f91740 1 -- start start 2024-04-25T00:18:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a35f91740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1ac580 0x55618e1ac960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a35f91740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1acea0 0x55618e1b7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a35f91740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1b7570 0x55618e1b9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a35f91740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55618e111670 con 0x55618e1acea0 2024-04-25T00:18:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a35f91740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55618e0b9970 con 0x55618e1b7570 2024-04-25T00:18:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a35f91740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55618e10d630 con 0x55618e1ac580 2024-04-25T00:18:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a29ffb640 1 --2- 172.21.15.27:0/3674655151 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1a1c0a2d00 0x7f1a1c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a30f3e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1acea0 0x55618e1b7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1ac580 0x55618e1ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a30f3e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1acea0 0x55618e1b7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51852/0 (socket says 172.21.15.27:51852) 2024-04-25T00:18:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a30f3e640 1 -- 172.21.15.27:0/3533194490 learned_addr learned my addr 172.21.15.27:0/3533194490 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a2bfff640 1 --2- 172.21.15.27:0/3533194490 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1b7570 0x55618e1b9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a3173f640 1 --2- 172.21.15.27:0/3674655151 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1a1c0a2d00 0x7f1a1c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a29ffb640 1 -- 172.21.15.27:0/3674655151 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f1a240cfa80 con 0x7f1a2c1066b0 2024-04-25T00:18:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a30f3e640 1 -- 172.21.15.27:0/3533194490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1ac580 msgr2=0x55618e1ac960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a30f3e640 1 --2- 172.21.15.27:0/3533194490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1ac580 0x55618e1ac960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a30f3e640 1 -- 172.21.15.27:0/3533194490 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1b7570 msgr2=0x55618e1b9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a30f3e640 1 --2- 172.21.15.27:0/3533194490 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1b7570 0x55618e1b9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a30f3e640 1 -- 172.21.15.27:0/3533194490 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55618e0dd400 con 0x55618e1acea0 2024-04-25T00:18:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.744+0000 7f1a2bfff640 1 --2- 172.21.15.27:0/3533194490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1ac580 0x55618e1ac960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.745+0000 7f1a30f3e640 1 --2- 172.21.15.27:0/3533194490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1acea0 0x55618e1b7030 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1a20006010 tx=0x7f1a20005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbad92fc12668031 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.745+0000 7f1a3173f640 1 --2- 172.21.15.27:0/3674655151 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1a1c0a2d00 0x7f1a1c0a51c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400c760 tx=0x7f1a24010cf0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.745+0000 7f1a16ffd640 1 -- 172.21.15.27:0/3533194490 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a20040070 con 0x55618e1acea0 2024-04-25T00:18:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.745+0000 7f1a16ffd640 1 -- 172.21.15.27:0/3533194490 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a20031970 con 0x55618e1acea0 2024-04-25T00:18:05.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.745+0000 7f1a16ffd640 1 -- 172.21.15.27:0/3533194490 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a20035040 con 0x55618e1acea0 2024-04-25T00:18:05.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.745+0000 7f1a35f91740 1 -- 172.21.15.27:0/3533194490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1acea0 msgr2=0x55618e1b7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.745+0000 7f1a35f91740 1 --2- 172.21.15.27:0/3533194490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1acea0 0x55618e1b7030 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1a20006010 tx=0x7f1a20005e80 comp rx=0 tx=0).stop 2024-04-25T00:18:05.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.746+0000 7f1a35f91740 1 -- 172.21.15.27:0/3533194490 shutdown_connections 2024-04-25T00:18:05.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.746+0000 7f1a35f91740 1 --2- 172.21.15.27:0/3533194490 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1b7570 0x55618e1b9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.746+0000 7f1a35f91740 1 --2- 172.21.15.27:0/3533194490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1acea0 0x55618e1b7030 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.746+0000 7f1a35f91740 1 --2- 172.21.15.27:0/3533194490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1ac580 0x55618e1ac960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.746+0000 7f1a35f91740 1 -- 172.21.15.27:0/3533194490 >> 172.21.15.27:0/3533194490 conn(0x55618e1a88a0 msgr2=0x55618e1a8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.747+0000 7f1a35f91740 1 -- 172.21.15.27:0/3533194490 shutdown_connections 2024-04-25T00:18:05.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.748+0000 7f1a35f91740 1 -- 172.21.15.27:0/3533194490 wait complete. 2024-04-25T00:18:05.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.748+0000 7f1a35f91740 1 -- start start 2024-04-25T00:18:05.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.748+0000 7f1a35f91740 1 Processor -- start 2024-04-25T00:18:05.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.749+0000 7f1a35f91740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 0x55618e1b6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.749+0000 7f1a35f91740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1acea0 0x55618e1b4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.749+0000 7f1a35f91740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1b7570 0x55618e1b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.749+0000 7f1a35f91740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55618e1ba300 con 0x55618e1ac580 2024-04-25T00:18:05.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.749+0000 7f1a35f91740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55618e0cd050 con 0x55618e1b7570 2024-04-25T00:18:05.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.749+0000 7f1a35f91740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55618e1ba480 con 0x55618e1acea0 2024-04-25T00:18:05.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.749+0000 7f1a35f91740 10 client.0 _ll_register_callbacks cb 0x55618de24be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:05.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.749+0000 7f1a30f3e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1acea0 0x55618e1b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.749+0000 7f1a30f3e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1acea0 0x55618e1b4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:46980/0 (socket says 172.21.15.27:46980) 2024-04-25T00:18:05.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.749+0000 7f1a30f3e640 1 -- 172.21.15.27:0/3613157482 learned_addr learned my addr 172.21.15.27:0/3613157482 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:05.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.750+0000 7f1a30f3e640 1 -- 172.21.15.27:0/3613157482 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1b7570 msgr2=0x55618e1b4830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:18:05.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.750+0000 7f1a30f3e640 1 --2- 172.21.15.27:0/3613157482 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1b7570 0x55618e1b4830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.750+0000 7f1a30f3e640 1 -- 172.21.15.27:0/3613157482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 msgr2=0x55618e1b6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.750+0000 7f1a30f3e640 1 --2- 172.21.15.27:0/3613157482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 0x55618e1b6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.750+0000 7f1a30f3e640 1 -- 172.21.15.27:0/3613157482 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55618e09edc0 con 0x55618e1acea0 2024-04-25T00:18:05.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.750+0000 7f1a30f3e640 1 --2- 172.21.15.27:0/3613157482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1acea0 0x55618e1b4220 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009900 tx=0x7f1a200052a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17e44a773a190734 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.750+0000 7f1a0bfff640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:18:05.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.750+0000 7f1a0bfff640 1 -- 172.21.15.27:0/3613157482 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2004c070 con 0x55618e1acea0 2024-04-25T00:18:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.750+0000 7f1a0bfff640 1 -- 172.21.15.27:0/3613157482 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f1a20004960 con 0x55618e1acea0 2024-04-25T00:18:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.750+0000 7f1a35f91740 1 -- 172.21.15.27:0/3613157482 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a20031970 con 0x55618e1acea0 2024-04-25T00:18:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.750+0000 7f1a0bfff640 1 -- 172.21.15.27:0/3613157482 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1a20058050 con 0x55618e1acea0 2024-04-25T00:18:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.750+0000 7f1a35f91740 10 client.25825 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.750+0000 7f1a35f91740 1 -- 172.21.15.27:0/3613157482 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55618de253f0 con 0x55618e1acea0 2024-04-25T00:18:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.751+0000 7f1a0bfff640 1 -- 172.21.15.27:0/3613157482 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a20031970 con 0x55618e1acea0 2024-04-25T00:18:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.751+0000 7f1a0bfff640 1 -- 172.21.15.27:0/3613157482 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f1a2003c990 con 0x55618e1acea0 2024-04-25T00:18:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.751+0000 7f1a35f91740 10 client.25825 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.751+0000 7f1a35f91740 10 client.25825 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:18:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.751+0000 7f1a35f91740 20 client.25825 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.751+0000 7f1a35f91740 1 --2- 172.21.15.27:0/3613157482 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55618e1bd100 0x55618e1bf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.751+0000 7f1a35f91740 4 client.25825 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:18:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.751+0000 7f1a35f91740 1 -- 172.21.15.27:0/3613157482 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55618e11ebb0 con 0x55618e1bd100 2024-04-25T00:18:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.751+0000 7f1a2bfff640 1 --2- 172.21.15.27:0/3613157482 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55618e1bd100 0x55618e1bf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.752+0000 7f1a2bfff640 1 --2- 172.21.15.27:0/3613157482 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55618e1bd100 0x55618e1bf530 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c9ca82eaf58b7868 server_cookie=241a1deeccde5ac5 in_seq=0 out_seq=0 2024-04-25T00:18:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.752+0000 7f1a0bfff640 10 client.25825 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:18:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.752+0000 7f1a0bfff640 1 -- 172.21.15.27:0/3613157482 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55618e11ebb0 con 0x55618e1bd100 2024-04-25T00:18:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.752+0000 7f1a0bfff640 10 client.25825 handle_command_reply: tid=0 2024-04-25T00:18:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.752+0000 7f1a35f91740 1 client.25825 shutdown 2024-04-25T00:18:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.753+0000 7f1a35f91740 1 -- 172.21.15.27:0/3613157482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1acea0 msgr2=0x55618e1b4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.753+0000 7f1a35f91740 1 --2- 172.21.15.27:0/3613157482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1acea0 0x55618e1b4220 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009900 tx=0x7f1a200052a0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.753+0000 7f1a35f91740 1 -- 172.21.15.27:0/3613157482 shutdown_connections 2024-04-25T00:18:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.753+0000 7f1a35f91740 1 --2- 172.21.15.27:0/3613157482 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55618e1bd100 0x55618e1bf530 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.753+0000 7f1a35f91740 1 --2- 172.21.15.27:0/3613157482 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1b7570 0x55618e1b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.753+0000 7f1a0bfff640 0 client.25825 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:18:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.753+0000 7f1a35f91740 1 --2- 172.21.15.27:0/3613157482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1acea0 0x55618e1b4220 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.753+0000 7f1a35f91740 1 --2- 172.21.15.27:0/3613157482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 0x55618e1b6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.753+0000 7f1a35f91740 1 -- 172.21.15.27:0/3613157482 >> 172.21.15.27:0/3613157482 conn(0x55618e1a88a0 msgr2=0x55618e1ae6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.753+0000 7f1a35f91740 1 -- 172.21.15.27:0/3613157482 shutdown_connections 2024-04-25T00:18:05.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.753+0000 7f1a35f91740 1 -- 172.21.15.27:0/3613157482 wait complete. 2024-04-25T00:18:05.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.753+0000 7f1a35f91740 20 client.25825 trim_cache size 0 max 16384 2024-04-25T00:18:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.767+0000 7f1a35f91740 1 Processor -- start 2024-04-25T00:18:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.767+0000 7f1a35f91740 1 -- start start 2024-04-25T00:18:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.768+0000 7f1a35f91740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 0x55618e1b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.768+0000 7f1a35f91740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1acea0 0x55618e1a7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.768+0000 7f1a35f91740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1b7570 0x55618e1b6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.768+0000 7f1a35f91740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c072f40 con 0x55618e1ac580 2024-04-25T00:18:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.768+0000 7f1a35f91740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c1aa1c0 con 0x55618e1acea0 2024-04-25T00:18:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.768+0000 7f1a35f91740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c073610 con 0x55618e1b7570 2024-04-25T00:18:05.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.768+0000 7f1a30f3e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1acea0 0x55618e1a7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.768+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 0x55618e1b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.769+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 0x55618e1b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51870/0 (socket says 172.21.15.27:51870) 2024-04-25T00:18:05.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.769+0000 7f1a2bfff640 1 -- 172.21.15.27:0/559937255 learned_addr learned my addr 172.21.15.27:0/559937255 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:05.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.769+0000 7f1a2bfff640 1 --2- 172.21.15.27:0/559937255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1b7570 0x55618e1b6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.769+0000 7f1a30f3e640 1 -- 172.21.15.27:0/559937255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1b7570 msgr2=0x55618e1b6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.769+0000 7f1a30f3e640 1 --2- 172.21.15.27:0/559937255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1b7570 0x55618e1b6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.769+0000 7f1a30f3e640 1 -- 172.21.15.27:0/559937255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 msgr2=0x55618e1b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.769+0000 7f1a30f3e640 1 --2- 172.21.15.27:0/559937255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 0x55618e1b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.769+0000 7f1a30f3e640 1 -- 172.21.15.27:0/559937255 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55618deef7d0 con 0x55618e1acea0 2024-04-25T00:18:05.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.769+0000 7f1a2bfff640 1 --2- 172.21.15.27:0/559937255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 0x55618e1b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:05.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.769+0000 7f1a30f3e640 1 --2- 172.21.15.27:0/559937255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1acea0 0x55618e1a7b40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1a2002a460 tx=0x7f1a20005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78b03d60125ba1ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.770+0000 7f1a167fc640 1 -- 172.21.15.27:0/559937255 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a20055070 con 0x55618e1acea0 2024-04-25T00:18:05.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.770+0000 7f1a167fc640 1 -- 172.21.15.27:0/559937255 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a20005850 con 0x55618e1acea0 2024-04-25T00:18:05.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.770+0000 7f1a167fc640 1 -- 172.21.15.27:0/559937255 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2004c040 con 0x55618e1acea0 2024-04-25T00:18:05.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.771+0000 7f1a35f91740 1 -- 172.21.15.27:0/559937255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1acea0 msgr2=0x55618e1a7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.771+0000 7f1a35f91740 1 --2- 172.21.15.27:0/559937255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1acea0 0x55618e1a7b40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1a2002a460 tx=0x7f1a20005c00 comp rx=0 tx=0).stop 2024-04-25T00:18:05.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.772+0000 7f1a35f91740 1 -- 172.21.15.27:0/559937255 shutdown_connections 2024-04-25T00:18:05.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.772+0000 7f1a35f91740 1 --2- 172.21.15.27:0/559937255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1b7570 0x55618e1b6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.772+0000 7f1a35f91740 1 --2- 172.21.15.27:0/559937255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1acea0 0x55618e1a7b40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.772+0000 7f1a35f91740 1 --2- 172.21.15.27:0/559937255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 0x55618e1b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.772+0000 7f1a35f91740 1 -- 172.21.15.27:0/559937255 >> 172.21.15.27:0/559937255 conn(0x55618e1a88a0 msgr2=0x55618e1b3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.772+0000 7f1a35f91740 1 -- 172.21.15.27:0/559937255 shutdown_connections 2024-04-25T00:18:05.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.772+0000 7f1a35f91740 1 -- 172.21.15.27:0/559937255 wait complete. 2024-04-25T00:18:05.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.772+0000 7f1a35f91740 1 -- start start 2024-04-25T00:18:05.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.772+0000 7f1a35f91740 1 Processor -- start 2024-04-25T00:18:05.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a35f91740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 0x55618e1cd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a35f91740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1acea0 0x55618e1cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a35f91740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1b7570 0x55618e247d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a35f91740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c1a9a30 con 0x55618e1ac580 2024-04-25T00:18:05.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a35f91740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c1a9360 con 0x55618e1acea0 2024-04-25T00:18:05.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a35f91740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c070f00 con 0x55618e1b7570 2024-04-25T00:18:05.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a35f91740 10 client.0 _ll_register_callbacks cb 0x7f1a1800be40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:05.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 0x55618e1cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 0x55618e1cd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51880/0 (socket says 172.21.15.27:51880) 2024-04-25T00:18:05.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a2bfff640 1 -- 172.21.15.27:0/3664598210 learned_addr learned my addr 172.21.15.27:0/3664598210 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:05.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a30f3e640 1 --2- 172.21.15.27:0/3664598210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1acea0 0x55618e1cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a2bfff640 1 --2- 172.21.15.27:0/3664598210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1b7570 0x55618e247d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a2bfff640 1 -- 172.21.15.27:0/3664598210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1b7570 msgr2=0x55618e247d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a2bfff640 1 --2- 172.21.15.27:0/3664598210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1b7570 0x55618e247d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a2bfff640 1 -- 172.21.15.27:0/3664598210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1acea0 msgr2=0x55618e1cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a2bfff640 1 --2- 172.21.15.27:0/3664598210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1acea0 0x55618e1cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a2bfff640 1 -- 172.21.15.27:0/3664598210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55618e248300 con 0x55618e1ac580 2024-04-25T00:18:05.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a2bfff640 1 --2- 172.21.15.27:0/3664598210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 0x55618e1cd370 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1a18005fd0 tx=0x7f1a1800b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e704d973c2b0b4fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.773+0000 7f1a0bfff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:18:05.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.774+0000 7f1a0bfff640 1 -- 172.21.15.27:0/3664598210 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a18027e60 con 0x55618e1ac580 2024-04-25T00:18:05.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.774+0000 7f1a35f91740 1 -- 172.21.15.27:0/3664598210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a20005850 con 0x55618e1ac580 2024-04-25T00:18:05.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.774+0000 7f1a0bfff640 1 -- 172.21.15.27:0/3664598210 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f1a1801e890 con 0x55618e1ac580 2024-04-25T00:18:05.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.774+0000 7f1a0bfff640 1 -- 172.21.15.27:0/3664598210 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1a18037050 con 0x55618e1ac580 2024-04-25T00:18:05.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.774+0000 7f1a35f91740 10 client.16323 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:05.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.774+0000 7f1a35f91740 1 -- 172.21.15.27:0/3664598210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55618e24f730 con 0x55618e1ac580 2024-04-25T00:18:05.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.774+0000 7f1a0bfff640 1 -- 172.21.15.27:0/3664598210 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a20005850 con 0x55618e1ac580 2024-04-25T00:18:05.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.774+0000 7f1a0bfff640 1 -- 172.21.15.27:0/3664598210 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f1a18004bb0 con 0x55618e1ac580 2024-04-25T00:18:05.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.774+0000 7f1a35f91740 10 client.16323 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:05.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.774+0000 7f1a35f91740 10 client.16323 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:18:05.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.774+0000 7f1a35f91740 20 client.16323 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:05.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.774+0000 7f1a35f91740 1 --2- 172.21.15.27:0/3664598210 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55618e1bd100 0x55618e251ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.774+0000 7f1a35f91740 4 client.16323 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:18:05.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.774+0000 7f1a35f91740 1 -- 172.21.15.27:0/3664598210 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55618e118f00 con 0x55618e1bd100 2024-04-25T00:18:05.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.774+0000 7f1a30f3e640 1 --2- 172.21.15.27:0/3664598210 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55618e1bd100 0x55618e251ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.775+0000 7f1a0bfff640 10 client.16323 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:18:05.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.775+0000 7f1a30f3e640 1 --2- 172.21.15.27:0/3664598210 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55618e1bd100 0x55618e251ea0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=eeb1d9e601ed7576 server_cookie=f7bc2e86e8385846 in_seq=0 out_seq=0 2024-04-25T00:18:05.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.775+0000 7f1a0bfff640 1 -- 172.21.15.27:0/3664598210 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55618e118f00 con 0x55618e1bd100 2024-04-25T00:18:05.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.775+0000 7f1a0bfff640 10 client.16323 handle_command_reply: tid=0 2024-04-25T00:18:05.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.775+0000 7f1a35f91740 1 client.16323 shutdown 2024-04-25T00:18:05.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.775+0000 7f1a35f91740 1 -- 172.21.15.27:0/3664598210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 msgr2=0x55618e1cd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.775+0000 7f1a35f91740 1 --2- 172.21.15.27:0/3664598210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 0x55618e1cd370 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1a18005fd0 tx=0x7f1a1800b390 comp rx=0 tx=0).stop 2024-04-25T00:18:05.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.775+0000 7f1a35f91740 1 -- 172.21.15.27:0/3664598210 shutdown_connections 2024-04-25T00:18:05.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.775+0000 7f1a35f91740 1 --2- 172.21.15.27:0/3664598210 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55618e1bd100 0x55618e251ea0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.775+0000 7f1a35f91740 1 --2- 172.21.15.27:0/3664598210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55618e1b7570 0x55618e247d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.775+0000 7f1a35f91740 1 --2- 172.21.15.27:0/3664598210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55618e1acea0 0x55618e1cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.775+0000 7f1a35f91740 1 --2- 172.21.15.27:0/3664598210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55618e1ac580 0x55618e1cd370 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.775+0000 7f1a0bfff640 0 client.16323 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:18:05.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.775+0000 7f1a35f91740 1 -- 172.21.15.27:0/3664598210 >> 172.21.15.27:0/3664598210 conn(0x55618e1a88a0 msgr2=0x55618e1a7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.775+0000 7f1a35f91740 1 -- 172.21.15.27:0/3664598210 shutdown_connections 2024-04-25T00:18:05.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.775+0000 7f1a35f91740 1 -- 172.21.15.27:0/3664598210 wait complete. 2024-04-25T00:18:05.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.775+0000 7f1a35f91740 20 client.16323 trim_cache size 0 max 16384 2024-04-25T00:18:05.783 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:18:05.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.776+0000 7f1a331a2640 1 -- 172.21.15.27:0/3674655151 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1a1c0a2d00 msgr2=0x7f1a1c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.776+0000 7f1a331a2640 1 --2- 172.21.15.27:0/3674655151 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1a1c0a2d00 0x7f1a1c0a51c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400c760 tx=0x7f1a24010cf0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.776+0000 7f1a331a2640 1 -- 172.21.15.27:0/3674655151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a2c1066b0 msgr2=0x7f1a2c19c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.776+0000 7f1a331a2640 1 --2- 172.21.15.27:0/3674655151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a2c1066b0 0x7f1a2c19c930 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009fd0 tx=0x7f1a2400ef10 comp rx=0 tx=0).stop 2024-04-25T00:18:05.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.776+0000 7f1a331a2640 1 -- 172.21.15.27:0/3674655151 shutdown_connections 2024-04-25T00:18:05.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.776+0000 7f1a331a2640 1 --2- 172.21.15.27:0/3674655151 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1a1c0a2d00 0x7f1a1c0a51c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.776+0000 7f1a331a2640 1 --2- 172.21.15.27:0/3674655151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a2c1066b0 0x7f1a2c19c930 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.776+0000 7f1a331a2640 1 --2- 172.21.15.27:0/3674655151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1a2c100fd0 0x7f1a2c1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.776+0000 7f1a331a2640 1 --2- 172.21.15.27:0/3674655151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a2c100610 0x7f1a2c198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.776+0000 7f1a331a2640 1 -- 172.21.15.27:0/3674655151 >> 172.21.15.27:0/3674655151 conn(0x7f1a2c077740 msgr2=0x7f1a2c0ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.776+0000 7f1a331a2640 1 -- 172.21.15.27:0/3674655151 shutdown_connections 2024-04-25T00:18:05.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.776+0000 7f1a331a2640 1 -- 172.21.15.27:0/3674655151 wait complete. 2024-04-25T00:18:05.788 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T00:18:05.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.908+0000 7f3439c58640 1 -- 172.21.15.27:0/477058233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3434102100 msgr2=0x7f343410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.908+0000 7f3439c58640 1 --2- 172.21.15.27:0/477058233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3434102100 0x7f343410e610 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f342c009cd0 tx=0x7f342c02a8e0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.908+0000 7f3439c58640 1 -- 172.21.15.27:0/477058233 shutdown_connections 2024-04-25T00:18:05.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.908+0000 7f3439c58640 1 --2- 172.21.15.27:0/477058233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3434102100 0x7f343410e610 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.908+0000 7f3439c58640 1 --2- 172.21.15.27:0/477058233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3434101740 0x7f3434101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.908+0000 7f3439c58640 1 --2- 172.21.15.27:0/477058233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3434107750 0x7f3434107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.908+0000 7f3439c58640 1 -- 172.21.15.27:0/477058233 >> 172.21.15.27:0/477058233 conn(0x7f3434078bd0 msgr2=0x7f3434078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.908+0000 7f3439c58640 1 -- 172.21.15.27:0/477058233 shutdown_connections 2024-04-25T00:18:05.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.908+0000 7f3439c58640 1 -- 172.21.15.27:0/477058233 wait complete. 2024-04-25T00:18:05.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.909+0000 7f3439c58640 1 Processor -- start 2024-04-25T00:18:05.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.909+0000 7f3439c58640 1 -- start start 2024-04-25T00:18:05.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.909+0000 7f3439c58640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3434101740 0x7f3434198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.909+0000 7f3439c58640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3434102100 0x7f3434198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.909+0000 7f3439c58640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3434107750 0x7f343419c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.909+0000 7f3439c58640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3434111bb0 con 0x7f3434101740 2024-04-25T00:18:05.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.909+0000 7f3439c58640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3434111a30 con 0x7f3434107750 2024-04-25T00:18:05.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.909+0000 7f3439c58640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3434111d30 con 0x7f3434102100 2024-04-25T00:18:05.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.909+0000 7f34337fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3434101740 0x7f3434198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.909+0000 7f34337fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3434101740 0x7f3434198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51898/0 (socket says 172.21.15.27:51898) 2024-04-25T00:18:05.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.909+0000 7f34337fe640 1 -- 172.21.15.27:0/1467885650 learned_addr learned my addr 172.21.15.27:0/1467885650 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:05.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.909+0000 7f3432ffd640 1 --2- 172.21.15.27:0/1467885650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3434102100 0x7f3434198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.909+0000 7f3433fff640 1 --2- 172.21.15.27:0/1467885650 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3434107750 0x7f343419c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.910+0000 7f34337fe640 1 -- 172.21.15.27:0/1467885650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3434102100 msgr2=0x7f3434198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.910+0000 7f34337fe640 1 --2- 172.21.15.27:0/1467885650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3434102100 0x7f3434198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.910+0000 7f34337fe640 1 -- 172.21.15.27:0/1467885650 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3434107750 msgr2=0x7f343419c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.910+0000 7f34337fe640 1 --2- 172.21.15.27:0/1467885650 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3434107750 0x7f343419c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.910+0000 7f34337fe640 1 -- 172.21.15.27:0/1467885650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343419cff0 con 0x7f3434101740 2024-04-25T00:18:05.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.910+0000 7f3432ffd640 1 --2- 172.21.15.27:0/1467885650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3434102100 0x7f3434198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:05.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.910+0000 7f3433fff640 1 --2- 172.21.15.27:0/1467885650 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3434107750 0x7f343419c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:05.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.910+0000 7f34337fe640 1 --2- 172.21.15.27:0/1467885650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3434101740 0x7f3434198050 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f342400d770 tx=0x7f342400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.910+0000 7f3430ff9640 1 -- 172.21.15.27:0/1467885650 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3424017070 con 0x7f3434101740 2024-04-25T00:18:05.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.910+0000 7f3430ff9640 1 -- 172.21.15.27:0/1467885650 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3424005500 con 0x7f3434101740 2024-04-25T00:18:05.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.910+0000 7f3439c58640 1 -- 172.21.15.27:0/1467885650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343419d2e0 con 0x7f3434101740 2024-04-25T00:18:05.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.910+0000 7f3430ff9640 1 -- 172.21.15.27:0/1467885650 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3424004e00 con 0x7f3434101740 2024-04-25T00:18:05.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.910+0000 7f3439c58640 1 -- 172.21.15.27:0/1467885650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34341a4b70 con 0x7f3434101740 2024-04-25T00:18:05.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.913+0000 7f3430ff9640 1 -- 172.21.15.27:0/1467885650 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3424004fa0 con 0x7f3434101740 2024-04-25T00:18:05.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.913+0000 7f343c967740 1 Processor -- start 2024-04-25T00:18:05.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.913+0000 7f343c967740 1 -- start start 2024-04-25T00:18:05.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.913+0000 7f343c967740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 0x56109de0c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.913+0000 7f343c967740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de0cea0 0x56109de17030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.913+0000 7f343c967740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de17570 0x56109de19960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.913+0000 7f343c967740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56109dd71670 con 0x56109de0cea0 2024-04-25T00:18:05.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.913+0000 7f343c967740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56109dd19970 con 0x56109de0c580 2024-04-25T00:18:05.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.913+0000 7f343c967740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56109dd6d630 con 0x56109de17570 2024-04-25T00:18:05.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.914+0000 7f34337fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de0cea0 0x56109de17030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.914+0000 7f3432ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de17570 0x56109de19960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.914+0000 7f3432ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de17570 0x56109de19960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:47018/0 (socket says 172.21.15.27:47018) 2024-04-25T00:18:05.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.914+0000 7f3432ffd640 1 -- 172.21.15.27:0/3864727629 learned_addr learned my addr 172.21.15.27:0/3864727629 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:05.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.914+0000 7f3433fff640 1 --2- 172.21.15.27:0/3864727629 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 0x56109de0c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.914+0000 7f3430ff9640 1 --2- 172.21.15.27:0/1467885650 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f341c0a2d00 0x7f341c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.914+0000 7f3432ffd640 1 -- 172.21.15.27:0/3864727629 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 msgr2=0x56109de0c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.914+0000 7f3432ffd640 1 --2- 172.21.15.27:0/3864727629 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 0x56109de0c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.914+0000 7f3432ffd640 1 -- 172.21.15.27:0/3864727629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de0cea0 msgr2=0x56109de17030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.914+0000 7f3432ffd640 1 --2- 172.21.15.27:0/3864727629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de0cea0 0x56109de17030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.914+0000 7f3432ffd640 1 -- 172.21.15.27:0/3864727629 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56109dd3d400 con 0x56109de17570 2024-04-25T00:18:05.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.914+0000 7f3433fff640 1 --2- 172.21.15.27:0/1467885650 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f341c0a2d00 0x7f341c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.914+0000 7f3430ff9640 1 -- 172.21.15.27:0/1467885650 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f34240cf0e0 con 0x7f3434101740 2024-04-25T00:18:05.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.914+0000 7f3432ffd640 1 --2- 172.21.15.27:0/3864727629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de17570 0x56109de19960 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f34280114f0 tx=0x7f34280119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b11d19ff16de2db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.914+0000 7f3419ffb640 1 -- 172.21.15.27:0/3864727629 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342801e070 con 0x56109de17570 2024-04-25T00:18:05.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.914+0000 7f3419ffb640 1 -- 172.21.15.27:0/3864727629 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34280035f0 con 0x56109de17570 2024-04-25T00:18:05.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.914+0000 7f3419ffb640 1 -- 172.21.15.27:0/3864727629 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342800f820 con 0x56109de17570 2024-04-25T00:18:05.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.915+0000 7f343c967740 1 -- 172.21.15.27:0/3864727629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de17570 msgr2=0x56109de19960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.915+0000 7f3433fff640 1 --2- 172.21.15.27:0/1467885650 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f341c0a2d00 0x7f341c0a51c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f342c005e30 tx=0x7f342c0095c0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.915+0000 7f343c967740 1 --2- 172.21.15.27:0/3864727629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de17570 0x56109de19960 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f34280114f0 tx=0x7f34280119b0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.915+0000 7f343c967740 1 -- 172.21.15.27:0/3864727629 shutdown_connections 2024-04-25T00:18:05.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.915+0000 7f343c967740 1 --2- 172.21.15.27:0/3864727629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de17570 0x56109de19960 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.915+0000 7f343c967740 1 --2- 172.21.15.27:0/3864727629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de0cea0 0x56109de17030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.915+0000 7f343c967740 1 --2- 172.21.15.27:0/3864727629 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 0x56109de0c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.915+0000 7f343c967740 1 -- 172.21.15.27:0/3864727629 >> 172.21.15.27:0/3864727629 conn(0x56109de088a0 msgr2=0x56109de08c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.915+0000 7f343c967740 1 -- 172.21.15.27:0/3864727629 shutdown_connections 2024-04-25T00:18:05.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.915+0000 7f343c967740 1 -- 172.21.15.27:0/3864727629 wait complete. 2024-04-25T00:18:05.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.915+0000 7f343c967740 1 -- start start 2024-04-25T00:18:05.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.916+0000 7f343c967740 1 Processor -- start 2024-04-25T00:18:05.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.916+0000 7f343c967740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 0x56109de16ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.916+0000 7f343c967740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 0x56109de14190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.916+0000 7f343c967740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de146d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.916+0000 7f343c967740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56109de1a360 con 0x56109de17570 2024-04-25T00:18:05.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.916+0000 7f343c967740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56109dd2d050 con 0x56109de0c580 2024-04-25T00:18:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.916+0000 7f343c967740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56109de1a4e0 con 0x56109de0cea0 2024-04-25T00:18:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.916+0000 7f343c967740 10 client.0 _ll_register_callbacks cb 0x56109da84be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.916+0000 7f3433fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de146d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.916+0000 7f3433fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de146d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51908/0 (socket says 172.21.15.27:51908) 2024-04-25T00:18:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.916+0000 7f3433fff640 1 -- 172.21.15.27:0/610251626 learned_addr learned my addr 172.21.15.27:0/610251626 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.916+0000 7f34337fe640 1 --2- 172.21.15.27:0/610251626 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 0x56109de16ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.916+0000 7f3432ffd640 1 --2- 172.21.15.27:0/610251626 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 0x56109de14190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.917+0000 7f34337fe640 1 -- 172.21.15.27:0/610251626 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 msgr2=0x56109de14190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.917+0000 7f34337fe640 1 --2- 172.21.15.27:0/610251626 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 0x56109de14190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.917+0000 7f34337fe640 1 -- 172.21.15.27:0/610251626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 msgr2=0x56109de146d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.917+0000 7f34337fe640 1 --2- 172.21.15.27:0/610251626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de146d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.917+0000 7f34337fe640 1 -- 172.21.15.27:0/610251626 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56109dcfedc0 con 0x56109de0c580 2024-04-25T00:18:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.917+0000 7f3433fff640 1 --2- 172.21.15.27:0/610251626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de146d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:05.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.918+0000 7f34337fe640 1 --2- 172.21.15.27:0/610251626 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 0x56109de16ea0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f34240cc4c0 tx=0x7f34240d27c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e15de0362ef5e9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.918+0000 7f3412ffd640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T00:18:05.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.918+0000 7f3412ffd640 1 -- 172.21.15.27:0/610251626 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3424020040 con 0x56109de0c580 2024-04-25T00:18:05.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.918+0000 7f3412ffd640 1 -- 172.21.15.27:0/610251626 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f34240979b0 con 0x56109de0c580 2024-04-25T00:18:05.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.918+0000 7f343c967740 1 -- 172.21.15.27:0/610251626 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34280035f0 con 0x56109de0c580 2024-04-25T00:18:05.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.918+0000 7f3412ffd640 1 -- 172.21.15.27:0/610251626 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f342409aca0 con 0x56109de0c580 2024-04-25T00:18:05.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.918+0000 7f343c967740 10 client.25877 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:05.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.918+0000 7f343c967740 1 -- 172.21.15.27:0/610251626 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56109da853f0 con 0x56109de0c580 2024-04-25T00:18:05.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.919+0000 7f3412ffd640 1 -- 172.21.15.27:0/610251626 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34280035f0 con 0x56109de0c580 2024-04-25T00:18:05.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.919+0000 7f3412ffd640 1 -- 172.21.15.27:0/610251626 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f34240cc740 con 0x56109de0c580 2024-04-25T00:18:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.919+0000 7f343c967740 10 client.25877 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.919+0000 7f343c967740 10 client.25877 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:18:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.919+0000 7f343c967740 20 client.25877 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.919+0000 7f343c967740 1 --2- 172.21.15.27:0/610251626 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x56109de27520 0x56109de299b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.919+0000 7f343c967740 4 client.25877 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:18:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.919+0000 7f343c967740 1 -- 172.21.15.27:0/610251626 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56109dd7ebb0 con 0x56109de27520 2024-04-25T00:18:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.919+0000 7f3432ffd640 1 --2- 172.21.15.27:0/610251626 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x56109de27520 0x56109de299b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.920+0000 7f3432ffd640 1 --2- 172.21.15.27:0/610251626 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x56109de27520 0x56109de299b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=803a36765ab46f2e server_cookie=174a6a374fdd7416 in_seq=0 out_seq=0 2024-04-25T00:18:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.920+0000 7f3412ffd640 10 client.25877 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:18:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.920+0000 7f3412ffd640 1 -- 172.21.15.27:0/610251626 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56109dd7ebb0 con 0x56109de27520 2024-04-25T00:18:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.920+0000 7f3412ffd640 10 client.25877 handle_command_reply: tid=0 2024-04-25T00:18:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.921+0000 7f343c967740 1 client.25877 shutdown 2024-04-25T00:18:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.921+0000 7f343c967740 1 -- 172.21.15.27:0/610251626 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 msgr2=0x56109de16ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.921+0000 7f343c967740 1 --2- 172.21.15.27:0/610251626 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 0x56109de16ea0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f34240cc4c0 tx=0x7f34240d27c0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.921+0000 7f343c967740 1 -- 172.21.15.27:0/610251626 shutdown_connections 2024-04-25T00:18:05.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.921+0000 7f343c967740 1 --2- 172.21.15.27:0/610251626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de146d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.921+0000 7f343c967740 1 --2- 172.21.15.27:0/610251626 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 0x56109de14190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.921+0000 7f343c967740 1 --2- 172.21.15.27:0/610251626 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x56109de27520 0x56109de299b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.921+0000 7f343c967740 1 --2- 172.21.15.27:0/610251626 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 0x56109de16ea0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.921+0000 7f3412ffd640 0 client.25877 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:18:05.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.921+0000 7f343c967740 1 -- 172.21.15.27:0/610251626 >> 172.21.15.27:0/610251626 conn(0x56109de088a0 msgr2=0x56109de07b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.921+0000 7f343c967740 1 -- 172.21.15.27:0/610251626 shutdown_connections 2024-04-25T00:18:05.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.921+0000 7f343c967740 1 -- 172.21.15.27:0/610251626 wait complete. 2024-04-25T00:18:05.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.921+0000 7f343c967740 20 client.25877 trim_cache size 0 max 16384 2024-04-25T00:18:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.936+0000 7f343c967740 1 Processor -- start 2024-04-25T00:18:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.936+0000 7f343c967740 1 -- start start 2024-04-25T00:18:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f343c967740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 0x56109de23250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f343c967740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 0x56109de0efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f343c967740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de169d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f343c967740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f34341aec60 con 0x56109de17570 2024-04-25T00:18:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f343c967740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f34341b0860 con 0x56109de0c580 2024-04-25T00:18:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f343c967740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f34341af330 con 0x56109de0cea0 2024-04-25T00:18:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f3433fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de169d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f34337fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 0x56109de23250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f3433fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de169d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51914/0 (socket says 172.21.15.27:51914) 2024-04-25T00:18:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f3432ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 0x56109de0efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f3433fff640 1 -- 172.21.15.27:0/321463457 learned_addr learned my addr 172.21.15.27:0/321463457 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f3432ffd640 1 -- 172.21.15.27:0/321463457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 msgr2=0x56109de23250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f3432ffd640 1 --2- 172.21.15.27:0/321463457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 0x56109de23250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f3432ffd640 1 -- 172.21.15.27:0/321463457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 msgr2=0x56109de169d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f3432ffd640 1 --2- 172.21.15.27:0/321463457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de169d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f3432ffd640 1 -- 172.21.15.27:0/321463457 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56109db4f7d0 con 0x56109de0cea0 2024-04-25T00:18:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f3433fff640 1 --2- 172.21.15.27:0/321463457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de169d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.937+0000 7f3432ffd640 1 --2- 172.21.15.27:0/321463457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 0x56109de0efd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3428011eb0 tx=0x7f3428009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecca9fb5a24fcffd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.938+0000 7f34197fa640 1 -- 172.21.15.27:0/321463457 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342801e040 con 0x56109de0cea0 2024-04-25T00:18:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.938+0000 7f34197fa640 1 -- 172.21.15.27:0/321463457 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f342802a070 con 0x56109de0cea0 2024-04-25T00:18:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.938+0000 7f34197fa640 1 -- 172.21.15.27:0/321463457 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3428005590 con 0x56109de0cea0 2024-04-25T00:18:05.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.938+0000 7f343c967740 1 -- 172.21.15.27:0/321463457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 msgr2=0x56109de0efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.938+0000 7f343c967740 1 --2- 172.21.15.27:0/321463457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 0x56109de0efd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3428011eb0 tx=0x7f3428009b50 comp rx=0 tx=0).stop 2024-04-25T00:18:05.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.938+0000 7f343c967740 1 -- 172.21.15.27:0/321463457 shutdown_connections 2024-04-25T00:18:05.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.938+0000 7f343c967740 1 --2- 172.21.15.27:0/321463457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de169d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.938+0000 7f343c967740 1 --2- 172.21.15.27:0/321463457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 0x56109de0efd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.938+0000 7f343c967740 1 --2- 172.21.15.27:0/321463457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 0x56109de23250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.938+0000 7f343c967740 1 -- 172.21.15.27:0/321463457 >> 172.21.15.27:0/321463457 conn(0x56109de088a0 msgr2=0x56109de0e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.938+0000 7f343c967740 1 -- 172.21.15.27:0/321463457 shutdown_connections 2024-04-25T00:18:05.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.938+0000 7f343c967740 1 -- 172.21.15.27:0/321463457 wait complete. 2024-04-25T00:18:05.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.939+0000 7f343c967740 1 -- start start 2024-04-25T00:18:05.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.939+0000 7f343c967740 1 Processor -- start 2024-04-25T00:18:05.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.939+0000 7f343c967740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 0x56109de114d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.939+0000 7f343c967740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 0x56109de11a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.939+0000 7f343c967740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de8caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.939+0000 7f343c967740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f34341b00d0 con 0x56109de17570 2024-04-25T00:18:05.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.939+0000 7f343c967740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f34341afa00 con 0x56109de0c580 2024-04-25T00:18:05.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.939+0000 7f343c967740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f34341acc20 con 0x56109de0cea0 2024-04-25T00:18:05.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.939+0000 7f343c967740 10 client.0 _ll_register_callbacks cb 0x7f34240056b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:05.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.939+0000 7f3433fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de8caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.939+0000 7f3432ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 0x56109de11a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.939+0000 7f34337fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 0x56109de114d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.939+0000 7f3432ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 0x56109de11a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:47040/0 (socket says 172.21.15.27:47040) 2024-04-25T00:18:05.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.939+0000 7f3433fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de8caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51926/0 (socket says 172.21.15.27:51926) 2024-04-25T00:18:05.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.939+0000 7f3432ffd640 1 -- 172.21.15.27:0/429424385 learned_addr learned my addr 172.21.15.27:0/429424385 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:05.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.940+0000 7f3432ffd640 1 -- 172.21.15.27:0/429424385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 msgr2=0x56109de114d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.940+0000 7f3432ffd640 1 --2- 172.21.15.27:0/429424385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 0x56109de114d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.940+0000 7f3432ffd640 1 -- 172.21.15.27:0/429424385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 msgr2=0x56109de8caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.940+0000 7f3432ffd640 1 --2- 172.21.15.27:0/429424385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de8caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.940+0000 7f3432ffd640 1 -- 172.21.15.27:0/429424385 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56109de8d030 con 0x56109de0cea0 2024-04-25T00:18:05.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.940+0000 7f3433fff640 1 --2- 172.21.15.27:0/429424385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de8caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:05.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.940+0000 7f3432ffd640 1 --2- 172.21.15.27:0/429424385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 0x56109de11a10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f342800a010 tx=0x7f3428005a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=708d0940b48269d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.940+0000 7f3412ffd640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:18:05.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.940+0000 7f3412ffd640 1 -- 172.21.15.27:0/429424385 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3428032070 con 0x56109de0cea0 2024-04-25T00:18:05.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.940+0000 7f343c967740 1 -- 172.21.15.27:0/429424385 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f342802a070 con 0x56109de0cea0 2024-04-25T00:18:05.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.940+0000 7f3412ffd640 1 -- 172.21.15.27:0/429424385 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f3428011050 con 0x56109de0cea0 2024-04-25T00:18:05.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.940+0000 7f3412ffd640 1 -- 172.21.15.27:0/429424385 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f342803e050 con 0x56109de0cea0 2024-04-25T00:18:05.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.940+0000 7f343c967740 10 client.25849 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:05.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.940+0000 7f343c967740 1 -- 172.21.15.27:0/429424385 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56109de8e260 con 0x56109de0cea0 2024-04-25T00:18:05.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.940+0000 7f3412ffd640 1 -- 172.21.15.27:0/429424385 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f342802a070 con 0x56109de0cea0 2024-04-25T00:18:05.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.941+0000 7f3412ffd640 1 -- 172.21.15.27:0/429424385 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f342802b190 con 0x56109de0cea0 2024-04-25T00:18:05.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.941+0000 7f343c967740 10 client.25849 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:05.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.941+0000 7f343c967740 10 client.25849 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:18:05.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.941+0000 7f343c967740 20 client.25849 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:05.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.941+0000 7f343c967740 1 --2- 172.21.15.27:0/429424385 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x56109de8e850 0x56109de90c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.941+0000 7f343c967740 4 client.25849 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:18:05.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.941+0000 7f343c967740 1 -- 172.21.15.27:0/429424385 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56109dd78f00 con 0x56109de8e850 2024-04-25T00:18:05.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.941+0000 7f34337fe640 1 --2- 172.21.15.27:0/429424385 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x56109de8e850 0x56109de90c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:05 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1758759482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:18:05.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.941+0000 7f3412ffd640 10 client.25849 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:18:05.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.941+0000 7f34337fe640 1 --2- 172.21.15.27:0/429424385 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x56109de8e850 0x56109de90c70 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7b9e0f599d8d4510 server_cookie=7d12e4f5d2a1d9a6 in_seq=0 out_seq=0 2024-04-25T00:18:05.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.941+0000 7f3412ffd640 1 -- 172.21.15.27:0/429424385 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56109dd78f00 con 0x56109de8e850 2024-04-25T00:18:05.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.941+0000 7f3412ffd640 10 client.25849 handle_command_reply: tid=0 2024-04-25T00:18:05.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.941+0000 7f343c967740 1 client.25849 shutdown 2024-04-25T00:18:05.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f343c967740 1 -- 172.21.15.27:0/429424385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 msgr2=0x56109de11a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f343c967740 1 --2- 172.21.15.27:0/429424385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 0x56109de11a10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f342800a010 tx=0x7f3428005a70 comp rx=0 tx=0).stop 2024-04-25T00:18:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f343c967740 1 -- 172.21.15.27:0/429424385 shutdown_connections 2024-04-25T00:18:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f343c967740 1 --2- 172.21.15.27:0/429424385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56109de17570 0x56109de8caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f343c967740 1 --2- 172.21.15.27:0/429424385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56109de0cea0 0x56109de11a10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f343c967740 1 --2- 172.21.15.27:0/429424385 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x56109de8e850 0x56109de90c70 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f343c967740 1 --2- 172.21.15.27:0/429424385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56109de0c580 0x56109de114d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f3412ffd640 0 client.25849 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:18:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f343c967740 1 -- 172.21.15.27:0/429424385 >> 172.21.15.27:0/429424385 conn(0x56109de088a0 msgr2=0x56109de22240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f343c967740 1 -- 172.21.15.27:0/429424385 shutdown_connections 2024-04-25T00:18:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f343c967740 1 -- 172.21.15.27:0/429424385 wait complete. 2024-04-25T00:18:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f343c967740 20 client.25849 trim_cache size 0 max 16384 2024-04-25T00:18:05.950 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:18:05.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f3439c58640 1 -- 172.21.15.27:0/1467885650 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f341c0a2d00 msgr2=0x7f341c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f3439c58640 1 --2- 172.21.15.27:0/1467885650 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f341c0a2d00 0x7f341c0a51c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f342c005e30 tx=0x7f342c0095c0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f3439c58640 1 -- 172.21.15.27:0/1467885650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3434101740 msgr2=0x7f3434198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f3439c58640 1 --2- 172.21.15.27:0/1467885650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3434101740 0x7f3434198050 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f342400d770 tx=0x7f342400dc30 comp rx=0 tx=0).stop 2024-04-25T00:18:05.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f3439c58640 1 -- 172.21.15.27:0/1467885650 shutdown_connections 2024-04-25T00:18:05.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f3439c58640 1 --2- 172.21.15.27:0/1467885650 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f341c0a2d00 0x7f341c0a51c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f3439c58640 1 --2- 172.21.15.27:0/1467885650 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3434107750 0x7f343419c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f3439c58640 1 --2- 172.21.15.27:0/1467885650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3434102100 0x7f3434198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f3439c58640 1 --2- 172.21.15.27:0/1467885650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3434101740 0x7f3434198050 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.942+0000 7f3439c58640 1 -- 172.21.15.27:0/1467885650 >> 172.21.15.27:0/1467885650 conn(0x7f3434078bd0 msgr2=0x7f34340ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.943+0000 7f3439c58640 1 -- 172.21.15.27:0/1467885650 shutdown_connections 2024-04-25T00:18:05.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:05.943+0000 7f3439c58640 1 -- 172.21.15.27:0/1467885650 wait complete. 2024-04-25T00:18:05.956 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T00:18:06.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.074+0000 7f55855c8640 1 -- 172.21.15.27:0/634371551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5580100fd0 msgr2=0x7f558010d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.074+0000 7f55855c8640 1 --2- 172.21.15.27:0/634371551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5580100fd0 0x7f558010d4e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5578009cd0 tx=0x7f557802a8e0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.074+0000 7f55855c8640 1 -- 172.21.15.27:0/634371551 shutdown_connections 2024-04-25T00:18:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.074+0000 7f55855c8640 1 --2- 172.21.15.27:0/634371551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5580100fd0 0x7f558010d4e0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.074+0000 7f55855c8640 1 --2- 172.21.15.27:0/634371551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5580100610 0x7f5580100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.074+0000 7f55855c8640 1 --2- 172.21.15.27:0/634371551 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f55801066b0 0x7f5580106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.074+0000 7f55855c8640 1 -- 172.21.15.27:0/634371551 >> 172.21.15.27:0/634371551 conn(0x7f5580077740 msgr2=0x7f5580077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.074+0000 7f55855c8640 1 -- 172.21.15.27:0/634371551 shutdown_connections 2024-04-25T00:18:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.074+0000 7f55855c8640 1 -- 172.21.15.27:0/634371551 wait complete. 2024-04-25T00:18:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.075+0000 7f55855c8640 1 Processor -- start 2024-04-25T00:18:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.075+0000 7f55855c8640 1 -- start start 2024-04-25T00:18:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.075+0000 7f55855c8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5580100610 0x7f558019e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.075+0000 7f55855c8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5580100fd0 0x7f558019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.075+0000 7f55855c8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55801066b0 0x7f55801a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.075+0000 7f55855c8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5580118180 con 0x7f5580100610 2024-04-25T00:18:06.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.075+0000 7f55855c8640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5580118000 con 0x7f5580100fd0 2024-04-25T00:18:06.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.075+0000 7f55855c8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5580118300 con 0x7f55801066b0 2024-04-25T00:18:06.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.075+0000 7f557effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5580100610 0x7f558019e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.075+0000 7f557effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5580100610 0x7f558019e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51946/0 (socket says 172.21.15.27:51946) 2024-04-25T00:18:06.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.075+0000 7f557effd640 1 -- 172.21.15.27:0/3615798000 learned_addr learned my addr 172.21.15.27:0/3615798000 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:06.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.075+0000 7f5577fff640 1 --2- 172.21.15.27:0/3615798000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5580100fd0 0x7f558019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.075+0000 7f557f7fe640 1 --2- 172.21.15.27:0/3615798000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55801066b0 0x7f55801a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.076+0000 7f5577fff640 1 -- 172.21.15.27:0/3615798000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55801066b0 msgr2=0x7f55801a2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.076+0000 7f5577fff640 1 --2- 172.21.15.27:0/3615798000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55801066b0 0x7f55801a2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.076+0000 7f5577fff640 1 -- 172.21.15.27:0/3615798000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5580100610 msgr2=0x7f558019e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.076+0000 7f5577fff640 1 --2- 172.21.15.27:0/3615798000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5580100610 0x7f558019e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.076+0000 7f5577fff640 1 -- 172.21.15.27:0/3615798000 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55801a3500 con 0x7f5580100fd0 2024-04-25T00:18:06.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.076+0000 7f5577fff640 1 --2- 172.21.15.27:0/3615798000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5580100fd0 0x7f558019eaa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f556800e9f0 tx=0x7f556800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:06.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.076+0000 7f557cff9640 1 -- 172.21.15.27:0/3615798000 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556800ccc0 con 0x7f5580100fd0 2024-04-25T00:18:06.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.076+0000 7f55855c8640 1 -- 172.21.15.27:0/3615798000 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55801a37f0 con 0x7f5580100fd0 2024-04-25T00:18:06.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.076+0000 7f55855c8640 1 -- 172.21.15.27:0/3615798000 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55801ab080 con 0x7f5580100fd0 2024-04-25T00:18:06.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.076+0000 7f557cff9640 1 -- 172.21.15.27:0/3615798000 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5568005020 con 0x7f5580100fd0 2024-04-25T00:18:06.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.076+0000 7f557cff9640 1 -- 172.21.15.27:0/3615798000 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556801c070 con 0x7f5580100fd0 2024-04-25T00:18:06.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.079+0000 7f557cff9640 1 -- 172.21.15.27:0/3615798000 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5568004020 con 0x7f5580100fd0 2024-04-25T00:18:06.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.079+0000 7f55883b2740 1 Processor -- start 2024-04-25T00:18:06.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.079+0000 7f55883b2740 1 -- start start 2024-04-25T00:18:06.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.079+0000 7f557cff9640 1 --2- 172.21.15.27:0/3615798000 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f556c0a2d00 0x7f556c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f557f7fe640 1 --2- 172.21.15.27:0/3615798000 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f556c0a2d00 0x7f556c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f557cff9640 1 -- 172.21.15.27:0/3615798000 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f55680cf750 con 0x7f5580100fd0 2024-04-25T00:18:06.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f55883b2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209580 0x5580f8209960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f55883b2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8209ea0 0x5580f8214030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f55883b2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8214570 0x5580f8216960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f55883b2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5580f816e670 con 0x5580f8209580 2024-04-25T00:18:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f55883b2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5580f8116970 con 0x5580f8214570 2024-04-25T00:18:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f55883b2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5580f816a630 con 0x5580f8209ea0 2024-04-25T00:18:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f557effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209580 0x5580f8209960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f557effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209580 0x5580f8209960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51954/0 (socket says 172.21.15.27:51954) 2024-04-25T00:18:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f557effd640 1 -- 172.21.15.27:0/2357929611 learned_addr learned my addr 172.21.15.27:0/2357929611 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f5577fff640 1 --2- 172.21.15.27:0/2357929611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8209ea0 0x5580f8214030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f557f7fe640 1 --2- 172.21.15.27:0/3615798000 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f556c0a2d00 0x7f556c0a51c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5578005e30 tx=0x7f5578032fd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f557f7fe640 1 --2- 172.21.15.27:0/2357929611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8214570 0x5580f8216960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f557effd640 1 -- 172.21.15.27:0/2357929611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8209ea0 msgr2=0x5580f8214030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f557effd640 1 --2- 172.21.15.27:0/2357929611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8209ea0 0x5580f8214030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f557effd640 1 -- 172.21.15.27:0/2357929611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8214570 msgr2=0x5580f8216960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f557effd640 1 --2- 172.21.15.27:0/2357929611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8214570 0x5580f8216960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f557effd640 1 -- 172.21.15.27:0/2357929611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580f813a400 con 0x5580f8209580 2024-04-25T00:18:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f557f7fe640 1 --2- 172.21.15.27:0/2357929611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8214570 0x5580f8216960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:18:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.080+0000 7f5577fff640 1 --2- 172.21.15.27:0/2357929611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8209ea0 0x5580f8214030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.081+0000 7f557effd640 1 --2- 172.21.15.27:0/2357929611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209580 0x5580f8209960 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5570009d60 tx=0x7f5570011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c3c8a5ce48c244 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.081+0000 7f55757fa640 1 -- 172.21.15.27:0/2357929611 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5570004230 con 0x5580f8209580 2024-04-25T00:18:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.081+0000 7f55757fa640 1 -- 172.21.15.27:0/2357929611 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55700043d0 con 0x5580f8209580 2024-04-25T00:18:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.081+0000 7f55757fa640 1 -- 172.21.15.27:0/2357929611 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5570019840 con 0x5580f8209580 2024-04-25T00:18:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.081+0000 7f55883b2740 1 -- 172.21.15.27:0/2357929611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209580 msgr2=0x5580f8209960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.081+0000 7f55883b2740 1 --2- 172.21.15.27:0/2357929611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209580 0x5580f8209960 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5570009d60 tx=0x7f5570011810 comp rx=0 tx=0).stop 2024-04-25T00:18:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f55883b2740 1 -- 172.21.15.27:0/2357929611 shutdown_connections 2024-04-25T00:18:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f55883b2740 1 --2- 172.21.15.27:0/2357929611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8214570 0x5580f8216960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f55883b2740 1 --2- 172.21.15.27:0/2357929611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8209ea0 0x5580f8214030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f55883b2740 1 --2- 172.21.15.27:0/2357929611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209580 0x5580f8209960 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f55883b2740 1 -- 172.21.15.27:0/2357929611 >> 172.21.15.27:0/2357929611 conn(0x5580f82058a0 msgr2=0x5580f8205c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f55883b2740 1 -- 172.21.15.27:0/2357929611 shutdown_connections 2024-04-25T00:18:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f55883b2740 1 -- 172.21.15.27:0/2357929611 wait complete. 2024-04-25T00:18:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f55883b2740 1 -- start start 2024-04-25T00:18:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f55883b2740 1 Processor -- start 2024-04-25T00:18:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f55883b2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209580 0x5580f8213eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f55883b2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8209ea0 0x5580f82111a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f55883b2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8214570 0x5580f82116e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f55883b2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5580f8217360 con 0x5580f8209580 2024-04-25T00:18:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f55883b2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5580f812a050 con 0x5580f8209ea0 2024-04-25T00:18:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f55883b2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5580f82174e0 con 0x5580f8214570 2024-04-25T00:18:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f55883b2740 10 client.0 _ll_register_callbacks cb 0x5580f7e81be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f557effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209580 0x5580f8213eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f557effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209580 0x5580f8213eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51968/0 (socket says 172.21.15.27:51968) 2024-04-25T00:18:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f557effd640 1 -- 172.21.15.27:0/1979236305 learned_addr learned my addr 172.21.15.27:0/1979236305 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.082+0000 7f5577fff640 1 --2- 172.21.15.27:0/1979236305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8209ea0 0x5580f82111a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.083+0000 7f557f7fe640 1 --2- 172.21.15.27:0/1979236305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8214570 0x5580f82116e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.083+0000 7f557effd640 1 -- 172.21.15.27:0/1979236305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8214570 msgr2=0x5580f82116e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.083+0000 7f557effd640 1 --2- 172.21.15.27:0/1979236305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8214570 0x5580f82116e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.083+0000 7f557effd640 1 -- 172.21.15.27:0/1979236305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8209ea0 msgr2=0x5580f82111a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.083+0000 7f557effd640 1 --2- 172.21.15.27:0/1979236305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8209ea0 0x5580f82111a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.083+0000 7f557effd640 1 -- 172.21.15.27:0/1979236305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580f80fbdc0 con 0x5580f8209580 2024-04-25T00:18:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.083+0000 7f5577fff640 1 --2- 172.21.15.27:0/1979236305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8209ea0 0x5580f82111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.083+0000 7f557effd640 1 --2- 172.21.15.27:0/1979236305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209580 0x5580f8213eb0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f5570009620 tx=0x7f5570009a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=361be98a6c941c56 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.083+0000 7f55667fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:18:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.083+0000 7f55667fc640 1 -- 172.21.15.27:0/1979236305 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5570028070 con 0x5580f8209580 2024-04-25T00:18:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.083+0000 7f55883b2740 1 -- 172.21.15.27:0/1979236305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55700043d0 con 0x5580f8209580 2024-04-25T00:18:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.083+0000 7f55667fc640 1 -- 172.21.15.27:0/1979236305 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f55700053d0 con 0x5580f8209580 2024-04-25T00:18:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.083+0000 7f55667fc640 1 -- 172.21.15.27:0/1979236305 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5570034050 con 0x5580f8209580 2024-04-25T00:18:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.083+0000 7f55883b2740 10 client.16353 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.083+0000 7f55883b2740 1 -- 172.21.15.27:0/1979236305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5580f7e823f0 con 0x5580f8209580 2024-04-25T00:18:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.084+0000 7f55667fc640 1 -- 172.21.15.27:0/1979236305 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55700043d0 con 0x5580f8209580 2024-04-25T00:18:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.084+0000 7f55667fc640 1 -- 172.21.15.27:0/1979236305 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f5570004c20 con 0x5580f8209580 2024-04-25T00:18:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.084+0000 7f55883b2740 10 client.16353 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.084+0000 7f55883b2740 10 client.16353 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:18:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.084+0000 7f55883b2740 20 client.16353 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.084+0000 7f55883b2740 1 --2- 172.21.15.27:0/1979236305 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5580f8224540 0x5580f8226930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.084+0000 7f55883b2740 4 client.16353 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:18:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.084+0000 7f55883b2740 1 -- 172.21.15.27:0/1979236305 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580f817bbb0 con 0x5580f8224540 2024-04-25T00:18:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.084+0000 7f5577fff640 1 --2- 172.21.15.27:0/1979236305 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5580f8224540 0x5580f8226930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.084+0000 7f55667fc640 10 client.16353 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:18:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.084+0000 7f5577fff640 1 --2- 172.21.15.27:0/1979236305 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5580f8224540 0x5580f8226930 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=87b520fe6a401bd0 server_cookie=3704121b7c720a2e in_seq=0 out_seq=0 2024-04-25T00:18:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.085+0000 7f55667fc640 1 -- 172.21.15.27:0/1979236305 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5580f817bbb0 con 0x5580f8224540 2024-04-25T00:18:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.085+0000 7f55667fc640 10 client.16353 handle_command_reply: tid=0 2024-04-25T00:18:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.085+0000 7f55883b2740 1 client.16353 shutdown 2024-04-25T00:18:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.085+0000 7f55883b2740 1 -- 172.21.15.27:0/1979236305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209580 msgr2=0x5580f8213eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.085+0000 7f55883b2740 1 --2- 172.21.15.27:0/1979236305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209580 0x5580f8213eb0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f5570009620 tx=0x7f5570009a90 comp rx=0 tx=0).stop 2024-04-25T00:18:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.085+0000 7f55883b2740 1 -- 172.21.15.27:0/1979236305 shutdown_connections 2024-04-25T00:18:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.085+0000 7f55883b2740 1 --2- 172.21.15.27:0/1979236305 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5580f8224540 0x5580f8226930 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.085+0000 7f55883b2740 1 --2- 172.21.15.27:0/1979236305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8214570 0x5580f82116e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.085+0000 7f55883b2740 1 --2- 172.21.15.27:0/1979236305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8209ea0 0x5580f82111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.085+0000 7f55667fc640 0 client.16353 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:18:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.085+0000 7f55883b2740 1 --2- 172.21.15.27:0/1979236305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209580 0x5580f8213eb0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.085+0000 7f55883b2740 1 -- 172.21.15.27:0/1979236305 >> 172.21.15.27:0/1979236305 conn(0x5580f82058a0 msgr2=0x5580f820b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.085+0000 7f55883b2740 1 -- 172.21.15.27:0/1979236305 shutdown_connections 2024-04-25T00:18:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.085+0000 7f55883b2740 1 -- 172.21.15.27:0/1979236305 wait complete. 2024-04-25T00:18:06.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.085+0000 7f55883b2740 20 client.16353 trim_cache size 0 max 16384 2024-04-25T00:18:06.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.102+0000 7f55883b2740 1 Processor -- start 2024-04-25T00:18:06.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.102+0000 7f55883b2740 1 -- start start 2024-04-25T00:18:06.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.102+0000 7f55883b2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8209580 0x5580f8210ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.102+0000 7f55883b2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209ea0 0x5580f820c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.102+0000 7f55883b2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8214570 0x5580f82139e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.102+0000 7f55883b2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f55801b5140 con 0x5580f8209ea0 2024-04-25T00:18:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.102+0000 7f55883b2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f55801b6d40 con 0x5580f8214570 2024-04-25T00:18:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.102+0000 7f55883b2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55801b5810 con 0x5580f8209580 2024-04-25T00:18:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.103+0000 7f5577fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209ea0 0x5580f820c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.103+0000 7f5577fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209ea0 0x5580f820c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51982/0 (socket says 172.21.15.27:51982) 2024-04-25T00:18:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.103+0000 7f5577fff640 1 -- 172.21.15.27:0/1300413498 learned_addr learned my addr 172.21.15.27:0/1300413498 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.103+0000 7f557effd640 1 --2- 172.21.15.27:0/1300413498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8209580 0x5580f8210ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.103+0000 7f557f7fe640 1 --2- 172.21.15.27:0/1300413498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8214570 0x5580f82139e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.103+0000 7f5577fff640 1 -- 172.21.15.27:0/1300413498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8209580 msgr2=0x5580f8210ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.103+0000 7f5577fff640 1 --2- 172.21.15.27:0/1300413498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8209580 0x5580f8210ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.103+0000 7f5577fff640 1 -- 172.21.15.27:0/1300413498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8214570 msgr2=0x5580f82139e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.103+0000 7f5577fff640 1 --2- 172.21.15.27:0/1300413498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8214570 0x5580f82139e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.103+0000 7f5577fff640 1 -- 172.21.15.27:0/1300413498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580f7f4c7d0 con 0x5580f8209ea0 2024-04-25T00:18:06.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.103+0000 7f5577fff640 1 --2- 172.21.15.27:0/1300413498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209ea0 0x5580f820c740 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f5568098290 tx=0x7f556809e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f606139e39d2a9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:06.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.103+0000 7f5574ff9640 1 -- 172.21.15.27:0/1300413498 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5568099bc0 con 0x5580f8209ea0 2024-04-25T00:18:06.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.103+0000 7f5574ff9640 1 -- 172.21.15.27:0/1300413498 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5568097f30 con 0x5580f8209ea0 2024-04-25T00:18:06.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.104+0000 7f5574ff9640 1 -- 172.21.15.27:0/1300413498 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55680860b0 con 0x5580f8209ea0 2024-04-25T00:18:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.104+0000 7f55883b2740 1 -- 172.21.15.27:0/1300413498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209ea0 msgr2=0x5580f820c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.104+0000 7f55883b2740 1 --2- 172.21.15.27:0/1300413498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209ea0 0x5580f820c740 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f5568098290 tx=0x7f556809e040 comp rx=0 tx=0).stop 2024-04-25T00:18:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.105+0000 7f55883b2740 1 -- 172.21.15.27:0/1300413498 shutdown_connections 2024-04-25T00:18:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.105+0000 7f55883b2740 1 --2- 172.21.15.27:0/1300413498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8214570 0x5580f82139e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.105+0000 7f55883b2740 1 --2- 172.21.15.27:0/1300413498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8209ea0 0x5580f820c740 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.105+0000 7f55883b2740 1 --2- 172.21.15.27:0/1300413498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8209580 0x5580f8210ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.105+0000 7f55883b2740 1 -- 172.21.15.27:0/1300413498 >> 172.21.15.27:0/1300413498 conn(0x5580f82058a0 msgr2=0x5580f8204b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.105+0000 7f55883b2740 1 -- 172.21.15.27:0/1300413498 shutdown_connections 2024-04-25T00:18:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.105+0000 7f55883b2740 1 -- 172.21.15.27:0/1300413498 wait complete. 2024-04-25T00:18:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.105+0000 7f55883b2740 1 -- start start 2024-04-25T00:18:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.105+0000 7f55883b2740 1 Processor -- start 2024-04-25T00:18:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.105+0000 7f55883b2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8209580 0x5580f82176f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.106+0000 7f55883b2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8209ea0 0x5580f8217c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.106+0000 7f557effd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8209580 0x5580f82176f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.106+0000 7f557effd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8209580 0x5580f82176f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:60098/0 (socket says 172.21.15.27:60098) 2024-04-25T00:18:06.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.106+0000 7f557effd640 1 -- 172.21.15.27:0/1308737233 learned_addr learned my addr 172.21.15.27:0/1308737233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:06.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.106+0000 7f55883b2740 1 --2- 172.21.15.27:0/1308737233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8214570 0x5580f82181b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.106+0000 7f55883b2740 1 -- 172.21.15.27:0/1308737233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f55801b65b0 con 0x5580f8214570 2024-04-25T00:18:06.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.106+0000 7f55883b2740 1 -- 172.21.15.27:0/1308737233 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f55801b5ee0 con 0x5580f8209580 2024-04-25T00:18:06.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.106+0000 7f55883b2740 1 -- 172.21.15.27:0/1308737233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55801b3100 con 0x5580f8209ea0 2024-04-25T00:18:06.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.106+0000 7f55883b2740 10 client.0 _ll_register_callbacks cb 0x7f5568097be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:06.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.106+0000 7f5577fff640 1 --2- 172.21.15.27:0/1308737233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8209ea0 0x5580f8217c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.106+0000 7f557f7fe640 1 --2- 172.21.15.27:0/1308737233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8214570 0x5580f82181b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.107+0000 7f557f7fe640 1 -- 172.21.15.27:0/1308737233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8209ea0 msgr2=0x5580f8217c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:06.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.107+0000 7f557f7fe640 1 --2- 172.21.15.27:0/1308737233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8209ea0 0x5580f8217c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.107+0000 7f557f7fe640 1 -- 172.21.15.27:0/1308737233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8209580 msgr2=0x5580f82176f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:06.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.107+0000 7f557f7fe640 1 --2- 172.21.15.27:0/1308737233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8209580 0x5580f82176f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.107+0000 7f557f7fe640 1 -- 172.21.15.27:0/1308737233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580f8288bf0 con 0x5580f8214570 2024-04-25T00:18:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.107+0000 7f5577fff640 1 --2- 172.21.15.27:0/1308737233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8209ea0 0x5580f8217c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.107+0000 7f557effd640 1 --2- 172.21.15.27:0/1308737233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8209580 0x5580f82176f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.107+0000 7f557f7fe640 1 --2- 172.21.15.27:0/1308737233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8214570 0x5580f82181b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f5578009ca0 tx=0x7f557802a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a467b8fb99a6e107 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.107+0000 7f55667fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:18:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.108+0000 7f55667fc640 1 -- 172.21.15.27:0/1308737233 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5578004450 con 0x5580f8214570 2024-04-25T00:18:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.108+0000 7f55667fc640 1 -- 172.21.15.27:0/1308737233 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f55780045f0 con 0x5580f8214570 2024-04-25T00:18:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.108+0000 7f55883b2740 1 -- 172.21.15.27:0/1308737233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5568097f30 con 0x5580f8214570 2024-04-25T00:18:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.108+0000 7f55667fc640 1 -- 172.21.15.27:0/1308737233 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f55780047f0 con 0x5580f8214570 2024-04-25T00:18:06.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.108+0000 7f55883b2740 10 client.16359 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:06.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.108+0000 7f55883b2740 1 -- 172.21.15.27:0/1308737233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5580f8289ef0 con 0x5580f8214570 2024-04-25T00:18:06.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.109+0000 7f55667fc640 1 -- 172.21.15.27:0/1308737233 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5568097f30 con 0x5580f8214570 2024-04-25T00:18:06.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.109+0000 7f55667fc640 1 -- 172.21.15.27:0/1308737233 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f5578030070 con 0x5580f8214570 2024-04-25T00:18:06.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.109+0000 7f55883b2740 10 client.16359 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:06.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.109+0000 7f55883b2740 10 client.16359 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:18:06.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.109+0000 7f55883b2740 20 client.16359 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:06.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.109+0000 7f55883b2740 1 --2- 172.21.15.27:0/1308737233 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5580f828a660 0x5580f828ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.109+0000 7f55883b2740 4 client.16359 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:18:06.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.109+0000 7f55883b2740 1 -- 172.21.15.27:0/1308737233 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5580f8175f00 con 0x5580f828a660 2024-04-25T00:18:06.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.109+0000 7f557effd640 1 --2- 172.21.15.27:0/1308737233 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5580f828a660 0x5580f828ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.109+0000 7f55667fc640 10 client.16359 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:18:06.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.110+0000 7f557effd640 1 --2- 172.21.15.27:0/1308737233 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5580f828a660 0x5580f828ca50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9ef8de444dafbbe9 server_cookie=ea8fffd37d8cf060 in_seq=0 out_seq=0 2024-04-25T00:18:06.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.110+0000 7f55667fc640 1 -- 172.21.15.27:0/1308737233 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5580f8175f00 con 0x5580f828a660 2024-04-25T00:18:06.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.110+0000 7f55667fc640 10 client.16359 handle_command_reply: tid=0 2024-04-25T00:18:06.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.110+0000 7f55883b2740 1 client.16359 shutdown 2024-04-25T00:18:06.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.110+0000 7f55883b2740 1 -- 172.21.15.27:0/1308737233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8214570 msgr2=0x5580f82181b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.110+0000 7f55883b2740 1 --2- 172.21.15.27:0/1308737233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8214570 0x5580f82181b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f5578009ca0 tx=0x7f557802a8e0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55883b2740 1 -- 172.21.15.27:0/1308737233 shutdown_connections 2024-04-25T00:18:06.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55883b2740 1 --2- 172.21.15.27:0/1308737233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f8214570 0x5580f82181b0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55883b2740 1 --2- 172.21.15.27:0/1308737233 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5580f828a660 0x5580f828ca50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55883b2740 1 --2- 172.21.15.27:0/1308737233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5580f8209ea0 0x5580f8217c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55883b2740 1 --2- 172.21.15.27:0/1308737233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5580f8209580 0x5580f82176f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55667fc640 0 client.16359 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:18:06.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55883b2740 1 -- 172.21.15.27:0/1308737233 >> 172.21.15.27:0/1308737233 conn(0x5580f82058a0 msgr2=0x5580f821faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:06.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55883b2740 1 -- 172.21.15.27:0/1308737233 shutdown_connections 2024-04-25T00:18:06.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55883b2740 1 -- 172.21.15.27:0/1308737233 wait complete. 2024-04-25T00:18:06.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55883b2740 20 client.16359 trim_cache size 0 max 16384 2024-04-25T00:18:06.115 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:18:06.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55855c8640 1 -- 172.21.15.27:0/3615798000 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f556c0a2d00 msgr2=0x7f556c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55855c8640 1 --2- 172.21.15.27:0/3615798000 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f556c0a2d00 0x7f556c0a51c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5578005e30 tx=0x7f5578032fd0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55855c8640 1 -- 172.21.15.27:0/3615798000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5580100fd0 msgr2=0x7f558019eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55855c8640 1 --2- 172.21.15.27:0/3615798000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5580100fd0 0x7f558019eaa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f556800e9f0 tx=0x7f556800eeb0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55855c8640 1 -- 172.21.15.27:0/3615798000 shutdown_connections 2024-04-25T00:18:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55855c8640 1 --2- 172.21.15.27:0/3615798000 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f556c0a2d00 0x7f556c0a51c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55855c8640 1 --2- 172.21.15.27:0/3615798000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55801066b0 0x7f55801a2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55855c8640 1 --2- 172.21.15.27:0/3615798000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5580100fd0 0x7f558019eaa0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55855c8640 1 --2- 172.21.15.27:0/3615798000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5580100610 0x7f558019e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55855c8640 1 -- 172.21.15.27:0/3615798000 >> 172.21.15.27:0/3615798000 conn(0x7f5580077740 msgr2=0x7f55800ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55855c8640 1 -- 172.21.15.27:0/3615798000 shutdown_connections 2024-04-25T00:18:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.111+0000 7f55855c8640 1 -- 172.21.15.27:0/3615798000 wait complete. 2024-04-25T00:18:06.124 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T00:18:06.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.240+0000 7fcaf64a5640 1 -- 172.21.15.27:0/3268327766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf010bd00 msgr2=0x7fcaf010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.240+0000 7fcaf64a5640 1 --2- 172.21.15.27:0/3268327766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf010bd00 0x7fcaf010e1a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fcae8009ca0 tx=0x7fcae802a9e0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.240+0000 7fcaf64a5640 1 -- 172.21.15.27:0/3268327766 shutdown_connections 2024-04-25T00:18:06.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.240+0000 7fcaf64a5640 1 --2- 172.21.15.27:0/3268327766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf010bd00 0x7fcaf010e1a0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.240+0000 7fcaf64a5640 1 --2- 172.21.15.27:0/3268327766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcaf01032e0 0x7fcaf010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.240+0000 7fcaf64a5640 1 --2- 172.21.15.27:0/3268327766 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcaf01029c0 0x7fcaf0102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.240+0000 7fcaf64a5640 1 -- 172.21.15.27:0/3268327766 >> 172.21.15.27:0/3268327766 conn(0x7fcaf0077740 msgr2=0x7fcaf0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:06.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.240+0000 7fcaf64a5640 1 -- 172.21.15.27:0/3268327766 shutdown_connections 2024-04-25T00:18:06.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.240+0000 7fcaf64a5640 1 -- 172.21.15.27:0/3268327766 wait complete. 2024-04-25T00:18:06.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.240+0000 7fcaf64a5640 1 Processor -- start 2024-04-25T00:18:06.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.240+0000 7fcaf64a5640 1 -- start start 2024-04-25T00:18:06.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaf64a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf01029c0 0x7fcaf0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaf64a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcaf01032e0 0x7fcaf0198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaf64a5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcaf010bd00 0x7fcaf019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaf64a5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf0111b00 con 0x7fcaf01029c0 2024-04-25T00:18:06.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaf64a5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf0111980 con 0x7fcaf010bd00 2024-04-25T00:18:06.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaf64a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf0111c80 con 0x7fcaf01032e0 2024-04-25T00:18:06.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaeffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf01029c0 0x7fcaf0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaeffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf01029c0 0x7fcaf0198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52012/0 (socket says 172.21.15.27:52012) 2024-04-25T00:18:06.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaeffff640 1 -- 172.21.15.27:0/860254464 learned_addr learned my addr 172.21.15.27:0/860254464 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:06.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaf4a42640 1 --2- 172.21.15.27:0/860254464 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcaf010bd00 0x7fcaf019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaef7fe640 1 --2- 172.21.15.27:0/860254464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcaf01032e0 0x7fcaf0198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaeffff640 1 -- 172.21.15.27:0/860254464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcaf01032e0 msgr2=0x7fcaf0198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaeffff640 1 --2- 172.21.15.27:0/860254464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcaf01032e0 0x7fcaf0198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaeffff640 1 -- 172.21.15.27:0/860254464 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcaf010bd00 msgr2=0x7fcaf019c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaeffff640 1 --2- 172.21.15.27:0/860254464 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcaf010bd00 0x7fcaf019c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaeffff640 1 -- 172.21.15.27:0/860254464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf019cfa0 con 0x7fcaf01029c0 2024-04-25T00:18:06.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaef7fe640 1 --2- 172.21.15.27:0/860254464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcaf01032e0 0x7fcaf0198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:06.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaf4a42640 1 --2- 172.21.15.27:0/860254464 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcaf010bd00 0x7fcaf019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:18:06.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaeffff640 1 --2- 172.21.15.27:0/860254464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf01029c0 0x7fcaf0198000 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fcae000d950 tx=0x7fcae000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:06.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.241+0000 7fcaed7fa640 1 -- 172.21.15.27:0/860254464 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae0017070 con 0x7fcaf01029c0 2024-04-25T00:18:06.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.242+0000 7fcaf64a5640 1 -- 172.21.15.27:0/860254464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaf019d290 con 0x7fcaf01029c0 2024-04-25T00:18:06.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.242+0000 7fcaf64a5640 1 -- 172.21.15.27:0/860254464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaf01a4b20 con 0x7fcaf01029c0 2024-04-25T00:18:06.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.242+0000 7fcaed7fa640 1 -- 172.21.15.27:0/860254464 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcae0009c30 con 0x7fcaf01029c0 2024-04-25T00:18:06.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.242+0000 7fcaed7fa640 1 -- 172.21.15.27:0/860254464 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae0020060 con 0x7fcaf01029c0 2024-04-25T00:18:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.245+0000 7fcaed7fa640 1 -- 172.21.15.27:0/860254464 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fcae0010460 con 0x7fcaf01029c0 2024-04-25T00:18:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.245+0000 7fcaf91b1740 1 Processor -- start 2024-04-25T00:18:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.245+0000 7fcaf91b1740 1 -- start start 2024-04-25T00:18:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.246+0000 7fcaf91b1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c085640 0x560e4c085a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.246+0000 7fcaf91b1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c080ea0 0x560e4c081320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.246+0000 7fcaf91b1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c081860 0x560e4c08d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.246+0000 7fcaf91b1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560e4bfdc670 con 0x560e4c085640 2024-04-25T00:18:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.246+0000 7fcaf91b1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560e4bf84970 con 0x560e4c080ea0 2024-04-25T00:18:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.246+0000 7fcaf91b1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560e4bfd8630 con 0x560e4c081860 2024-04-25T00:18:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.246+0000 7fcaf4a42640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c085640 0x560e4c085a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.246+0000 7fcaf4a42640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c085640 0x560e4c085a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52018/0 (socket says 172.21.15.27:52018) 2024-04-25T00:18:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.246+0000 7fcaf4a42640 1 -- 172.21.15.27:0/1014044119 learned_addr learned my addr 172.21.15.27:0/1014044119 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.246+0000 7fcaef7fe640 1 --2- 172.21.15.27:0/1014044119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c081860 0x560e4c08d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.246+0000 7fcaed7fa640 1 --2- 172.21.15.27:0/860254464 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fcad80a2d80 0x7fcad80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.246+0000 7fcaf4a42640 1 --2- 172.21.15.27:0/860254464 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fcad80a2d80 0x7fcad80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.246+0000 7fcaef7fe640 1 -- 172.21.15.27:0/1014044119 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c080ea0 msgr2=0x560e4c081320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:06.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.246+0000 7fcaed7fa640 1 -- 172.21.15.27:0/860254464 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fcae00cf0e0 con 0x7fcaf01029c0 2024-04-25T00:18:06.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.247+0000 7fcaeffff640 1 --2- 172.21.15.27:0/1014044119 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c080ea0 0x560e4c081320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.247+0000 7fcaef7fe640 1 --2- 172.21.15.27:0/1014044119 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c080ea0 0x560e4c081320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.247+0000 7fcaef7fe640 1 -- 172.21.15.27:0/1014044119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c085640 msgr2=0x560e4c085a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:06.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.247+0000 7fcaef7fe640 1 --2- 172.21.15.27:0/1014044119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c085640 0x560e4c085a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.247+0000 7fcaef7fe640 1 -- 172.21.15.27:0/1014044119 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e4bfa8400 con 0x560e4c081860 2024-04-25T00:18:06.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.247+0000 7fcaf4a42640 1 --2- 172.21.15.27:0/1014044119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c085640 0x560e4c085a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:06.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.247+0000 7fcaef7fe640 1 --2- 172.21.15.27:0/1014044119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c081860 0x560e4c08d590 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcae4009ac0 tx=0x7fcae4009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13bd9000357c378c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:06.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.247+0000 7fcaf4a42640 1 --2- 172.21.15.27:0/860254464 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fcad80a2d80 0x7fcad80a5240 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcae8005fa0 tx=0x7fcae802a9e0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:06.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.247+0000 7fcad67fc640 1 -- 172.21.15.27:0/1014044119 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae4016070 con 0x560e4c081860 2024-04-25T00:18:06.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.247+0000 7fcad67fc640 1 -- 172.21.15.27:0/1014044119 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcae4005500 con 0x560e4c081860 2024-04-25T00:18:06.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.247+0000 7fcad67fc640 1 -- 172.21.15.27:0/1014044119 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae40116d0 con 0x560e4c081860 2024-04-25T00:18:06.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.248+0000 7fcaf91b1740 1 -- 172.21.15.27:0/1014044119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c081860 msgr2=0x560e4c08d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.248+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/1014044119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c081860 0x560e4c08d590 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcae4009ac0 tx=0x7fcae4009f80 comp rx=0 tx=0).stop 2024-04-25T00:18:06.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.248+0000 7fcaf91b1740 1 -- 172.21.15.27:0/1014044119 shutdown_connections 2024-04-25T00:18:06.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.248+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/1014044119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c081860 0x560e4c08d590 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.248+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/1014044119 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c080ea0 0x560e4c081320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.248+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/1014044119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c085640 0x560e4c085a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.248+0000 7fcaf91b1740 1 -- 172.21.15.27:0/1014044119 >> 172.21.15.27:0/1014044119 conn(0x560e4c073350 msgr2=0x560e4c073730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:06.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.249+0000 7fcaf91b1740 1 -- 172.21.15.27:0/1014044119 shutdown_connections 2024-04-25T00:18:06.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.249+0000 7fcaf91b1740 1 -- 172.21.15.27:0/1014044119 wait complete. 2024-04-25T00:18:06.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.249+0000 7fcaf91b1740 1 -- start start 2024-04-25T00:18:06.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.250+0000 7fcaf91b1740 1 Processor -- start 2024-04-25T00:18:06.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.250+0000 7fcaf91b1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c080ea0 0x560e4c08d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.250+0000 7fcaf91b1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c081860 0x560e4c08a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.250+0000 7fcaf91b1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c085640 0x560e4c08aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.250+0000 7fcaeffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c080ea0 0x560e4c08d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.250+0000 7fcaeffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c080ea0 0x560e4c08d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52032/0 (socket says 172.21.15.27:52032) 2024-04-25T00:18:06.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.250+0000 7fcaeffff640 1 -- 172.21.15.27:0/1553364448 learned_addr learned my addr 172.21.15.27:0/1553364448 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:06.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.250+0000 7fcaf91b1740 1 -- 172.21.15.27:0/1553364448 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560e4c08dbe0 con 0x560e4c080ea0 2024-04-25T00:18:06.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.250+0000 7fcaf91b1740 1 -- 172.21.15.27:0/1553364448 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560e4bf98050 con 0x560e4c081860 2024-04-25T00:18:06.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.250+0000 7fcaf91b1740 1 -- 172.21.15.27:0/1553364448 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560e4c08dd60 con 0x560e4c085640 2024-04-25T00:18:06.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.250+0000 7fcaf91b1740 10 client.0 _ll_register_callbacks cb 0x560e4bcefbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:06.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.250+0000 7fcaf4a42640 1 --2- 172.21.15.27:0/1553364448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c085640 0x560e4c08aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.250+0000 7fcaef7fe640 1 --2- 172.21.15.27:0/1553364448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c081860 0x560e4c08a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:05 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1758759482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:18:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.251+0000 7fcaeffff640 1 -- 172.21.15.27:0/1553364448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c085640 msgr2=0x560e4c08aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.251+0000 7fcaeffff640 1 --2- 172.21.15.27:0/1553364448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c085640 0x560e4c08aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.251+0000 7fcaeffff640 1 -- 172.21.15.27:0/1553364448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c081860 msgr2=0x560e4c08a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.251+0000 7fcaeffff640 1 --2- 172.21.15.27:0/1553364448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c081860 0x560e4c08a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.251+0000 7fcaeffff640 1 -- 172.21.15.27:0/1553364448 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e4bf69dc0 con 0x560e4c080ea0 2024-04-25T00:18:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.251+0000 7fcaf4a42640 1 --2- 172.21.15.27:0/1553364448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c085640 0x560e4c08aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.251+0000 7fcaeffff640 1 --2- 172.21.15.27:0/1553364448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c080ea0 0x560e4c08d2a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fcae0010e50 tx=0x7fcae009ba80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb8c5679872379 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.251+0000 7fcacf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:18:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.251+0000 7fcacf7fe640 1 -- 172.21.15.27:0/1553364448 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae009f600 con 0x560e4c080ea0 2024-04-25T00:18:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.251+0000 7fcacf7fe640 1 -- 172.21.15.27:0/1553364448 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fcae00bb990 con 0x560e4c080ea0 2024-04-25T00:18:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.251+0000 7fcaf91b1740 1 -- 172.21.15.27:0/1553364448 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcae4005500 con 0x560e4c080ea0 2024-04-25T00:18:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.251+0000 7fcacf7fe640 1 -- 172.21.15.27:0/1553364448 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcae00a3ec0 con 0x560e4c080ea0 2024-04-25T00:18:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.251+0000 7fcaf91b1740 10 client.16371 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.251+0000 7fcaf91b1740 1 -- 172.21.15.27:0/1553364448 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560e4bcf03f0 con 0x560e4c080ea0 2024-04-25T00:18:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.252+0000 7fcacf7fe640 1 -- 172.21.15.27:0/1553364448 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcae4005500 con 0x560e4c080ea0 2024-04-25T00:18:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.252+0000 7fcacf7fe640 1 -- 172.21.15.27:0/1553364448 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fcae0023060 con 0x560e4c080ea0 2024-04-25T00:18:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.252+0000 7fcaf91b1740 10 client.16371 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.252+0000 7fcaf91b1740 10 client.16371 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:18:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.252+0000 7fcaf91b1740 20 client.16371 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.252+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/1553364448 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560e4c09add0 0x560e4c09d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.252+0000 7fcaf91b1740 4 client.16371 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:18:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.252+0000 7fcaf91b1740 1 -- 172.21.15.27:0/1553364448 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e4bfe9bb0 con 0x560e4c09add0 2024-04-25T00:18:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.252+0000 7fcaef7fe640 1 --2- 172.21.15.27:0/1553364448 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560e4c09add0 0x560e4c09d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.252+0000 7fcacf7fe640 10 client.16371 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.252+0000 7fcaef7fe640 1 --2- 172.21.15.27:0/1553364448 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560e4c09add0 0x560e4c09d210 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=305bdbb6fbfcdccd server_cookie=b85d53c68cdb952e in_seq=0 out_seq=0 2024-04-25T00:18:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.253+0000 7fcacf7fe640 1 -- 172.21.15.27:0/1553364448 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e4bfe9bb0 con 0x560e4c09add0 2024-04-25T00:18:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.253+0000 7fcacf7fe640 10 client.16371 handle_command_reply: tid=0 2024-04-25T00:18:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.253+0000 7fcaf91b1740 1 client.16371 shutdown 2024-04-25T00:18:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.253+0000 7fcaf91b1740 1 -- 172.21.15.27:0/1553364448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c080ea0 msgr2=0x560e4c08d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.253+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/1553364448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c080ea0 0x560e4c08d2a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fcae0010e50 tx=0x7fcae009ba80 comp rx=0 tx=0).stop 2024-04-25T00:18:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.253+0000 7fcaf91b1740 1 -- 172.21.15.27:0/1553364448 shutdown_connections 2024-04-25T00:18:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.253+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/1553364448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c085640 0x560e4c08aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.253+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/1553364448 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560e4c09add0 0x560e4c09d210 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.253+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/1553364448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c081860 0x560e4c08a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.253+0000 7fcacf7fe640 0 client.16371 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.253+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/1553364448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c080ea0 0x560e4c08d2a0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.253+0000 7fcaf91b1740 1 -- 172.21.15.27:0/1553364448 >> 172.21.15.27:0/1553364448 conn(0x560e4c073350 msgr2=0x560e4c080720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.253+0000 7fcaf91b1740 1 -- 172.21.15.27:0/1553364448 shutdown_connections 2024-04-25T00:18:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.253+0000 7fcaf91b1740 1 -- 172.21.15.27:0/1553364448 wait complete. 2024-04-25T00:18:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.254+0000 7fcaf91b1740 20 client.16371 trim_cache size 0 max 16384 2024-04-25T00:18:06.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.269+0000 7fcaf91b1740 1 Processor -- start 2024-04-25T00:18:06.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.269+0000 7fcaf91b1740 1 -- start start 2024-04-25T00:18:06.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.270+0000 7fcaf91b1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c080ea0 0x560e4c080960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.270+0000 7fcaf91b1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c081860 0x560e4c096b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.270+0000 7fcaf91b1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c085640 0x560e4c08cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.270+0000 7fcaf91b1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcadc00fe40 con 0x560e4c080ea0 2024-04-25T00:18:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.270+0000 7fcaf91b1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcadc011a40 con 0x560e4c085640 2024-04-25T00:18:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.270+0000 7fcaf91b1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcadc010510 con 0x560e4c081860 2024-04-25T00:18:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.270+0000 7fcaeffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c080ea0 0x560e4c080960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.270+0000 7fcaef7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c081860 0x560e4c096b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.270+0000 7fcaef7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c081860 0x560e4c096b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:47134/0 (socket says 172.21.15.27:47134) 2024-04-25T00:18:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.270+0000 7fcaeffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c080ea0 0x560e4c080960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52034/0 (socket says 172.21.15.27:52034) 2024-04-25T00:18:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.270+0000 7fcaef7fe640 1 -- 172.21.15.27:0/4044291763 learned_addr learned my addr 172.21.15.27:0/4044291763 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.270+0000 7fcaef7fe640 1 -- 172.21.15.27:0/4044291763 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c085640 msgr2=0x560e4c08cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.270+0000 7fcaef7fe640 1 --2- 172.21.15.27:0/4044291763 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c085640 0x560e4c08cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.270+0000 7fcaef7fe640 1 -- 172.21.15.27:0/4044291763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c080ea0 msgr2=0x560e4c080960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.270+0000 7fcaef7fe640 1 --2- 172.21.15.27:0/4044291763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c080ea0 0x560e4c080960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.270+0000 7fcaef7fe640 1 -- 172.21.15.27:0/4044291763 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e4bdba7d0 con 0x560e4c081860 2024-04-25T00:18:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.271+0000 7fcaef7fe640 1 --2- 172.21.15.27:0/4044291763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c081860 0x560e4c096b40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcae4000c00 tx=0x7fcae4004be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11c506c5f56b9b1d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.271+0000 7fcad5ffb640 1 -- 172.21.15.27:0/4044291763 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae4016040 con 0x560e4c081860 2024-04-25T00:18:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.271+0000 7fcad5ffb640 1 -- 172.21.15.27:0/4044291763 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcae4004930 con 0x560e4c081860 2024-04-25T00:18:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.271+0000 7fcad5ffb640 1 -- 172.21.15.27:0/4044291763 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae4005000 con 0x560e4c081860 2024-04-25T00:18:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.272+0000 7fcaf91b1740 1 -- 172.21.15.27:0/4044291763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c081860 msgr2=0x560e4c096b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.272+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/4044291763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c081860 0x560e4c096b40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcae4000c00 tx=0x7fcae4004be0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.272+0000 7fcaf91b1740 1 -- 172.21.15.27:0/4044291763 shutdown_connections 2024-04-25T00:18:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.272+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/4044291763 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c085640 0x560e4c08cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.272+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/4044291763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c081860 0x560e4c096b40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.272+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/4044291763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c080ea0 0x560e4c080960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.272+0000 7fcaf91b1740 1 -- 172.21.15.27:0/4044291763 >> 172.21.15.27:0/4044291763 conn(0x560e4c073350 msgr2=0x560e4c084a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.272+0000 7fcaf91b1740 1 -- 172.21.15.27:0/4044291763 shutdown_connections 2024-04-25T00:18:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.272+0000 7fcaf91b1740 1 -- 172.21.15.27:0/4044291763 wait complete. 2024-04-25T00:18:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.273+0000 7fcaf91b1740 1 -- start start 2024-04-25T00:18:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.273+0000 7fcaf91b1740 1 Processor -- start 2024-04-25T00:18:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.273+0000 7fcaf91b1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c080ea0 0x560e4c093240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.274+0000 7fcaf91b1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c081860 0x560e4c086e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.274+0000 7fcaf91b1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c085640 0x560e4c087380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.274+0000 7fcaf91b1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcadc0112b0 con 0x560e4c081860 2024-04-25T00:18:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.274+0000 7fcaf91b1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcadc010be0 con 0x560e4c085640 2024-04-25T00:18:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.274+0000 7fcaf91b1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcadc00de00 con 0x560e4c080ea0 2024-04-25T00:18:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.274+0000 7fcaf91b1740 10 client.0 _ll_register_callbacks cb 0x7fcae009ac20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.274+0000 7fcaeffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c080ea0 0x560e4c093240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.274+0000 7fcaeffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c080ea0 0x560e4c093240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:47142/0 (socket says 172.21.15.27:47142) 2024-04-25T00:18:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.274+0000 7fcaeffff640 1 -- 172.21.15.27:0/688645215 learned_addr learned my addr 172.21.15.27:0/688645215 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.274+0000 7fcaef7fe640 1 --2- 172.21.15.27:0/688645215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c081860 0x560e4c086e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.274+0000 7fcaf4a42640 1 --2- 172.21.15.27:0/688645215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c085640 0x560e4c087380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.274+0000 7fcaeffff640 1 -- 172.21.15.27:0/688645215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c085640 msgr2=0x560e4c087380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.274+0000 7fcaeffff640 1 --2- 172.21.15.27:0/688645215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c085640 0x560e4c087380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.274+0000 7fcaeffff640 1 -- 172.21.15.27:0/688645215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c081860 msgr2=0x560e4c086e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.274+0000 7fcaeffff640 1 --2- 172.21.15.27:0/688645215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c081860 0x560e4c086e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.274+0000 7fcaeffff640 1 -- 172.21.15.27:0/688645215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e4c0878f0 con 0x560e4c080ea0 2024-04-25T00:18:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.275+0000 7fcaeffff640 1 --2- 172.21.15.27:0/688645215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c080ea0 0x560e4c093240 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcae00058c0 tx=0x7fcae00cc930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=476793b92489ae51 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.275+0000 7fcacf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:18:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.275+0000 7fcacf7fe640 1 -- 172.21.15.27:0/688645215 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae009f600 con 0x560e4c080ea0 2024-04-25T00:18:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.275+0000 7fcacf7fe640 1 -- 172.21.15.27:0/688645215 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fcae00ccdd0 con 0x560e4c080ea0 2024-04-25T00:18:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.275+0000 7fcaf91b1740 1 -- 172.21.15.27:0/688645215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcae4004930 con 0x560e4c080ea0 2024-04-25T00:18:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.276+0000 7fcacf7fe640 1 -- 172.21.15.27:0/688645215 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcae00bb330 con 0x560e4c080ea0 2024-04-25T00:18:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.277+0000 7fcaf91b1740 10 client.25882 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.277+0000 7fcaf91b1740 1 -- 172.21.15.27:0/688645215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560e4c1010d0 con 0x560e4c080ea0 2024-04-25T00:18:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.277+0000 7fcacf7fe640 1 -- 172.21.15.27:0/688645215 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcae00bb610 con 0x560e4c080ea0 2024-04-25T00:18:06.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:05 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1758759482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:18:06.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.277+0000 7fcacf7fe640 1 -- 172.21.15.27:0/688645215 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fcae009a6a0 con 0x560e4c080ea0 2024-04-25T00:18:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.277+0000 7fcaf91b1740 10 client.25882 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.277+0000 7fcaf91b1740 10 client.25882 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:18:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.277+0000 7fcaf91b1740 20 client.25882 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.277+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/688645215 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560e4c09add0 0x560e4c103890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.277+0000 7fcaf91b1740 4 client.25882 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:18:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.277+0000 7fcaf91b1740 1 -- 172.21.15.27:0/688645215 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fcae4004930 con 0x560e4c09add0 2024-04-25T00:18:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.278+0000 7fcaef7fe640 1 --2- 172.21.15.27:0/688645215 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560e4c09add0 0x560e4c103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.278+0000 7fcacf7fe640 10 client.25882 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.278+0000 7fcaef7fe640 1 --2- 172.21.15.27:0/688645215 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560e4c09add0 0x560e4c103890 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc3d61247277240a server_cookie=1c7821e2341111d4 in_seq=0 out_seq=0 2024-04-25T00:18:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.278+0000 7fcacf7fe640 1 -- 172.21.15.27:0/688645215 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fcae4004930 con 0x560e4c09add0 2024-04-25T00:18:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.278+0000 7fcacf7fe640 10 client.25882 handle_command_reply: tid=0 2024-04-25T00:18:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.278+0000 7fcaf91b1740 1 client.25882 shutdown 2024-04-25T00:18:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.279+0000 7fcaf91b1740 1 -- 172.21.15.27:0/688645215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c080ea0 msgr2=0x560e4c093240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.279+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/688645215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c080ea0 0x560e4c093240 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcae00058c0 tx=0x7fcae00cc930 comp rx=0 tx=0).stop 2024-04-25T00:18:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.279+0000 7fcaf91b1740 1 -- 172.21.15.27:0/688645215 shutdown_connections 2024-04-25T00:18:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.279+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/688645215 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560e4c09add0 0x560e4c103890 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.279+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/688645215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4c085640 0x560e4c087380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.279+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/688645215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e4c081860 0x560e4c086e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.279+0000 7fcacf7fe640 0 client.25882 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.279+0000 7fcaf91b1740 1 --2- 172.21.15.27:0/688645215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4c080ea0 0x560e4c093240 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.279+0000 7fcaf91b1740 1 -- 172.21.15.27:0/688645215 >> 172.21.15.27:0/688645215 conn(0x560e4c073350 msgr2=0x560e4c084200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.279+0000 7fcaf91b1740 1 -- 172.21.15.27:0/688645215 shutdown_connections 2024-04-25T00:18:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.279+0000 7fcaf91b1740 1 -- 172.21.15.27:0/688645215 wait complete. 2024-04-25T00:18:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.279+0000 7fcaf91b1740 20 client.25882 trim_cache size 0 max 16384 2024-04-25T00:18:06.284 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:18:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.280+0000 7fcaf64a5640 1 -- 172.21.15.27:0/860254464 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fcad80a2d80 msgr2=0x7fcad80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.280+0000 7fcaf64a5640 1 --2- 172.21.15.27:0/860254464 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fcad80a2d80 0x7fcad80a5240 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcae8005fa0 tx=0x7fcae802a9e0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.280+0000 7fcaf64a5640 1 -- 172.21.15.27:0/860254464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf01029c0 msgr2=0x7fcaf0198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.280+0000 7fcaf64a5640 1 --2- 172.21.15.27:0/860254464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf01029c0 0x7fcaf0198000 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fcae000d950 tx=0x7fcae000de10 comp rx=0 tx=0).stop 2024-04-25T00:18:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.280+0000 7fcaf64a5640 1 -- 172.21.15.27:0/860254464 shutdown_connections 2024-04-25T00:18:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.280+0000 7fcaf64a5640 1 --2- 172.21.15.27:0/860254464 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fcad80a2d80 0x7fcad80a5240 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.280+0000 7fcaf64a5640 1 --2- 172.21.15.27:0/860254464 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcaf010bd00 0x7fcaf019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.280+0000 7fcaf64a5640 1 --2- 172.21.15.27:0/860254464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcaf01032e0 0x7fcaf0198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.280+0000 7fcaf64a5640 1 --2- 172.21.15.27:0/860254464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf01029c0 0x7fcaf0198000 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.280+0000 7fcaf64a5640 1 -- 172.21.15.27:0/860254464 >> 172.21.15.27:0/860254464 conn(0x7fcaf0077740 msgr2=0x7fcaf00fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.280+0000 7fcaf64a5640 1 -- 172.21.15.27:0/860254464 shutdown_connections 2024-04-25T00:18:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:06.280+0000 7fcaf64a5640 1 -- 172.21.15.27:0/860254464 wait complete. 2024-04-25T00:18:07.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:06 smithi027 ceph-mon[29208]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 108 MiB/s wr, 13.46k op/s 2024-04-25T00:18:07.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:06 smithi139 ceph-mon[32833]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 108 MiB/s wr, 13.46k op/s 2024-04-25T00:18:07.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:06 smithi153 ceph-mon[32896]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 108 MiB/s wr, 13.46k op/s 2024-04-25T00:18:07.298 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T00:18:07.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:18:07.298 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T00:18:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.423+0000 7ff4c7d7e640 1 -- 172.21.15.27:0/1859074212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c0101790 msgr2=0x7ff4c0101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.423+0000 7ff4c7d7e640 1 --2- 172.21.15.27:0/1859074212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c0101790 0x7ff4c0101c10 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0009930 tx=0x7ff4b002a420 comp rx=0 tx=0).stop 2024-04-25T00:18:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.423+0000 7ff4c7d7e640 1 -- 172.21.15.27:0/1859074212 shutdown_connections 2024-04-25T00:18:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.423+0000 7ff4c7d7e640 1 --2- 172.21.15.27:0/1859074212 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff4c0102150 0x7ff4c010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.423+0000 7ff4c7d7e640 1 --2- 172.21.15.27:0/1859074212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c0101790 0x7ff4c0101c10 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.423+0000 7ff4c7d7e640 1 --2- 172.21.15.27:0/1859074212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c01077a0 0x7ff4c0107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.423+0000 7ff4c7d7e640 1 -- 172.21.15.27:0/1859074212 >> 172.21.15.27:0/1859074212 conn(0x7ff4c00fd2a0 msgr2=0x7ff4c00ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.423+0000 7ff4c7d7e640 1 -- 172.21.15.27:0/1859074212 shutdown_connections 2024-04-25T00:18:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.423+0000 7ff4c7d7e640 1 -- 172.21.15.27:0/1859074212 wait complete. 2024-04-25T00:18:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.424+0000 7ff4c7d7e640 1 Processor -- start 2024-04-25T00:18:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.424+0000 7ff4c7d7e640 1 -- start start 2024-04-25T00:18:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c7d7e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c0101790 0x7ff4c0198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c7d7e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff4c0102150 0x7ff4c0198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c7d7e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c01077a0 0x7ff4c019c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c7d7e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c0111bb0 con 0x7ff4c0101790 2024-04-25T00:18:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c7d7e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c0111a30 con 0x7ff4c0102150 2024-04-25T00:18:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c7d7e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c0111d30 con 0x7ff4c01077a0 2024-04-25T00:18:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c5b1a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c0101790 0x7ff4c0198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c5b1a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c0101790 0x7ff4c0198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48158/0 (socket says 172.21.15.27:48158) 2024-04-25T00:18:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c5b1a640 1 -- 172.21.15.27:0/782248834 learned_addr learned my addr 172.21.15.27:0/782248834 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c5319640 1 --2- 172.21.15.27:0/782248834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff4c0102150 0x7ff4c0198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c631b640 1 --2- 172.21.15.27:0/782248834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c01077a0 0x7ff4c019c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c5b1a640 1 -- 172.21.15.27:0/782248834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c01077a0 msgr2=0x7ff4c019c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c5b1a640 1 --2- 172.21.15.27:0/782248834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c01077a0 0x7ff4c019c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c5b1a640 1 -- 172.21.15.27:0/782248834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff4c0102150 msgr2=0x7ff4c0198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c5b1a640 1 --2- 172.21.15.27:0/782248834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff4c0102150 0x7ff4c0198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c5b1a640 1 -- 172.21.15.27:0/782248834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c019d100 con 0x7ff4c0101790 2024-04-25T00:18:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c5319640 1 --2- 172.21.15.27:0/782248834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff4c0102150 0x7ff4c0198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.425+0000 7ff4c5b1a640 1 --2- 172.21.15.27:0/782248834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c0101790 0x7ff4c0198110 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00e9f0 tx=0x7ff4bc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.426+0000 7ff4b6ffd640 1 -- 172.21.15.27:0/782248834 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc00cd80 con 0x7ff4c0101790 2024-04-25T00:18:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.426+0000 7ff4b6ffd640 1 -- 172.21.15.27:0/782248834 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4bc005500 con 0x7ff4c0101790 2024-04-25T00:18:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.426+0000 7ff4c7d7e640 1 -- 172.21.15.27:0/782248834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c019d3f0 con 0x7ff4c0101790 2024-04-25T00:18:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.426+0000 7ff4b6ffd640 1 -- 172.21.15.27:0/782248834 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc010690 con 0x7ff4c0101790 2024-04-25T00:18:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.426+0000 7ff4c7d7e640 1 -- 172.21.15.27:0/782248834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c01a4c80 con 0x7ff4c0101790 2024-04-25T00:18:07.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.428+0000 7ff4b6ffd640 1 -- 172.21.15.27:0/782248834 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff4bc0046c0 con 0x7ff4c0101790 2024-04-25T00:18:07.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.429+0000 7ff4cab6c740 1 Processor -- start 2024-04-25T00:18:07.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.429+0000 7ff4cab6c740 1 -- start start 2024-04-25T00:18:07.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.429+0000 7ff4cab6c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5e4980 0x55c2aa5e2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.429+0000 7ff4cab6c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e3290 0x55c2aa5e3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.429+0000 7ff4cab6c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5ee010 0x55c2aa5f0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.429+0000 7ff4cab6c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c2aa5472c0 con 0x55c2aa5ee010 2024-04-25T00:18:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.429+0000 7ff4cab6c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c2aa4ef910 con 0x55c2aa5e4980 2024-04-25T00:18:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.429+0000 7ff4cab6c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c2aa543280 con 0x55c2aa5e3290 2024-04-25T00:18:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.429+0000 7ff4c631b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5e4980 0x55c2aa5e2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.429+0000 7ff4c5319640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5ee010 0x55c2aa5f0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.429+0000 7ff4b6ffd640 1 --2- 172.21.15.27:0/782248834 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff4ac0a2d00 0x7ff4ac0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.429+0000 7ff4c631b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5e4980 0x55c2aa5e2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:54478/0 (socket says 172.21.15.27:54478) 2024-04-25T00:18:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.429+0000 7ff4c631b640 1 -- 172.21.15.27:0/918910385 learned_addr learned my addr 172.21.15.27:0/918910385 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.429+0000 7ff4c5b1a640 1 --2- 172.21.15.27:0/918910385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e3290 0x55c2aa5e3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.430+0000 7ff4c631b640 1 --2- 172.21.15.27:0/782248834 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff4ac0a2d00 0x7ff4ac0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.430+0000 7ff4b6ffd640 1 -- 172.21.15.27:0/782248834 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7ff4bc0cfb70 con 0x7ff4c0101790 2024-04-25T00:18:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.430+0000 7ff4c5319640 1 -- 172.21.15.27:0/918910385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e3290 msgr2=0x55c2aa5e3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.430+0000 7ff4c5319640 1 --2- 172.21.15.27:0/918910385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e3290 0x55c2aa5e3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.430+0000 7ff4c5319640 1 -- 172.21.15.27:0/918910385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5e4980 msgr2=0x55c2aa5e2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.430+0000 7ff4c5319640 1 --2- 172.21.15.27:0/918910385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5e4980 0x55c2aa5e2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.430+0000 7ff4c5319640 1 -- 172.21.15.27:0/918910385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2aa500fb0 con 0x55c2aa5ee010 2024-04-25T00:18:07.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.430+0000 7ff4c5b1a640 1 --2- 172.21.15.27:0/918910385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e3290 0x55c2aa5e3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:18:07.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.430+0000 7ff4c5319640 1 --2- 172.21.15.27:0/918910385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5ee010 0x55c2aa5f0400 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0005940 tx=0x7ff4b0005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af4ee6520cc3dbd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:07.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.430+0000 7ff4c631b640 1 --2- 172.21.15.27:0/782248834 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff4ac0a2d00 0x7ff4ac0a51c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8009fd0 tx=0x7ff4b8004ab0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:07.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.430+0000 7ff4abfff640 1 -- 172.21.15.27:0/918910385 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b0035040 con 0x55c2aa5ee010 2024-04-25T00:18:07.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.430+0000 7ff4abfff640 1 -- 172.21.15.27:0/918910385 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4b0031870 con 0x55c2aa5ee010 2024-04-25T00:18:07.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.430+0000 7ff4abfff640 1 -- 172.21.15.27:0/918910385 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b003f7e0 con 0x55c2aa5ee010 2024-04-25T00:18:07.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.431+0000 7ff4cab6c740 1 -- 172.21.15.27:0/918910385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5ee010 msgr2=0x55c2aa5f0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.431+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/918910385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5ee010 0x55c2aa5f0400 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0005940 tx=0x7ff4b0005c00 comp rx=0 tx=0).stop 2024-04-25T00:18:07.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.431+0000 7ff4cab6c740 1 -- 172.21.15.27:0/918910385 shutdown_connections 2024-04-25T00:18:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.431+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/918910385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5ee010 0x55c2aa5f0400 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.431+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/918910385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e3290 0x55c2aa5e3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.431+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/918910385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5e4980 0x55c2aa5e2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.431+0000 7ff4cab6c740 1 -- 172.21.15.27:0/918910385 >> 172.21.15.27:0/918910385 conn(0x55c2aa5ab950 msgr2=0x55c2aa5df7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.431+0000 7ff4cab6c740 1 -- 172.21.15.27:0/918910385 shutdown_connections 2024-04-25T00:18:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.431+0000 7ff4cab6c740 1 -- 172.21.15.27:0/918910385 wait complete. 2024-04-25T00:18:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.431+0000 7ff4cab6c740 1 -- start start 2024-04-25T00:18:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4cab6c740 1 Processor -- start 2024-04-25T00:18:07.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4cab6c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e3290 0x55c2aa5eaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4cab6c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5e4980 0x55c2aa5eb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4cab6c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5ee010 0x55c2aa5eb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4cab6c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c2aa5f0e40 con 0x55c2aa5e4980 2024-04-25T00:18:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4cab6c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c2aa502e90 con 0x55c2aa5ee010 2024-04-25T00:18:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4cab6c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c2aa5f0fc0 con 0x55c2aa5e3290 2024-04-25T00:18:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4cab6c740 10 client.0 _ll_register_callbacks cb 0x55c2aa53e970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4c5319640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5e4980 0x55c2aa5eb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4c5b1a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e3290 0x55c2aa5eaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4c631b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5ee010 0x55c2aa5eb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4c5319640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5e4980 0x55c2aa5eb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48182/0 (socket says 172.21.15.27:48182) 2024-04-25T00:18:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4c5319640 1 -- 172.21.15.27:0/2083561246 learned_addr learned my addr 172.21.15.27:0/2083561246 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4c631b640 1 -- 172.21.15.27:0/2083561246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e3290 msgr2=0x55c2aa5eaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4c631b640 1 --2- 172.21.15.27:0/2083561246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e3290 0x55c2aa5eaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4c631b640 1 -- 172.21.15.27:0/2083561246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5e4980 msgr2=0x55c2aa5eb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4c631b640 1 --2- 172.21.15.27:0/2083561246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5e4980 0x55c2aa5eb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4c631b640 1 -- 172.21.15.27:0/2083561246 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2aa3237e0 con 0x55c2aa5ee010 2024-04-25T00:18:07.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.432+0000 7ff4c5319640 1 --2- 172.21.15.27:0/2083561246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5e4980 0x55c2aa5eb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:07.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.433+0000 7ff4c631b640 1 --2- 172.21.15.27:0/2083561246 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5ee010 0x55c2aa5eb970 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8013fd0 tx=0x7ff4b8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f196f3059401da09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:07.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.433+0000 7ff4a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T00:18:07.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.433+0000 7ff4a97fa640 1 -- 172.21.15.27:0/2083561246 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8026070 con 0x55c2aa5ee010 2024-04-25T00:18:07.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.433+0000 7ff4cab6c740 1 -- 172.21.15.27:0/2083561246 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4b0031870 con 0x55c2aa5ee010 2024-04-25T00:18:07.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.433+0000 7ff4a97fa640 1 -- 172.21.15.27:0/2083561246 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7ff4b8010dc0 con 0x55c2aa5ee010 2024-04-25T00:18:07.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.433+0000 7ff4a97fa640 1 -- 172.21.15.27:0/2083561246 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff4b8032050 con 0x55c2aa5ee010 2024-04-25T00:18:07.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.433+0000 7ff4cab6c740 10 client.25925 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:07.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.433+0000 7ff4cab6c740 1 -- 172.21.15.27:0/2083561246 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c2aa383f70 con 0x55c2aa5ee010 2024-04-25T00:18:07.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.433+0000 7ff4a97fa640 1 -- 172.21.15.27:0/2083561246 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4b0031870 con 0x55c2aa5ee010 2024-04-25T00:18:07.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.433+0000 7ff4a97fa640 1 -- 172.21.15.27:0/2083561246 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7ff4b800f3c0 con 0x55c2aa5ee010 2024-04-25T00:18:07.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.433+0000 7ff4cab6c740 10 client.25925 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:07.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.433+0000 7ff4cab6c740 10 client.25925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:18:07.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.434+0000 7ff4cab6c740 20 client.25925 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:07.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.434+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/2083561246 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55c2aa5fe140 0x55c2aa600550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.434+0000 7ff4cab6c740 4 client.25925 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:18:07.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.434+0000 7ff4cab6c740 1 -- 172.21.15.27:0/2083561246 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2aa51b5a0 con 0x55c2aa5fe140 2024-04-25T00:18:07.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.434+0000 7ff4c5b1a640 1 --2- 172.21.15.27:0/2083561246 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55c2aa5fe140 0x55c2aa600550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.434+0000 7ff4c5b1a640 1 --2- 172.21.15.27:0/2083561246 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55c2aa5fe140 0x55c2aa600550 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=157d286fc1df5812 server_cookie=76131075e2f0a3e in_seq=0 out_seq=0 2024-04-25T00:18:07.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.434+0000 7ff4a97fa640 10 client.25925 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:07.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.435+0000 7ff4a97fa640 1 -- 172.21.15.27:0/2083561246 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c2aa51b5a0 con 0x55c2aa5fe140 2024-04-25T00:18:07.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.435+0000 7ff4a97fa640 10 client.25925 handle_command_reply: tid=0 2024-04-25T00:18:07.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.435+0000 7ff4cab6c740 1 client.25925 shutdown 2024-04-25T00:18:07.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.435+0000 7ff4cab6c740 1 -- 172.21.15.27:0/2083561246 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5ee010 msgr2=0x55c2aa5eb970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.435+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/2083561246 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5ee010 0x55c2aa5eb970 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8013fd0 tx=0x7ff4b8004a40 comp rx=0 tx=0).stop 2024-04-25T00:18:07.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.435+0000 7ff4cab6c740 1 -- 172.21.15.27:0/2083561246 shutdown_connections 2024-04-25T00:18:07.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.435+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/2083561246 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55c2aa5fe140 0x55c2aa600550 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.435+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/2083561246 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5ee010 0x55c2aa5eb970 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.435+0000 7ff4a97fa640 0 client.25925 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:07.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.435+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/2083561246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5e4980 0x55c2aa5eb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.435+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/2083561246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e3290 0x55c2aa5eaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.435+0000 7ff4cab6c740 1 -- 172.21.15.27:0/2083561246 >> 172.21.15.27:0/2083561246 conn(0x55c2aa5ab950 msgr2=0x55c2aa5e55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:07.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.435+0000 7ff4cab6c740 1 -- 172.21.15.27:0/2083561246 shutdown_connections 2024-04-25T00:18:07.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.435+0000 7ff4cab6c740 1 -- 172.21.15.27:0/2083561246 wait complete. 2024-04-25T00:18:07.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.435+0000 7ff4cab6c740 20 client.25925 trim_cache size 0 max 16384 2024-04-25T00:18:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.454+0000 7ff4cab6c740 1 Processor -- start 2024-04-25T00:18:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.454+0000 7ff4cab6c740 1 -- start start 2024-04-25T00:18:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.454+0000 7ff4cab6c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5e3290 0x55c2aa5f9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.454+0000 7ff4cab6c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e4980 0x55c2aa605e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.454+0000 7ff4cab6c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5ee010 0x55c2aa6063a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.454+0000 7ff4cab6c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c01aca20 con 0x55c2aa5ee010 2024-04-25T00:18:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.454+0000 7ff4cab6c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c01ae620 con 0x55c2aa5e3290 2024-04-25T00:18:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.454+0000 7ff4cab6c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c01ad0f0 con 0x55c2aa5e4980 2024-04-25T00:18:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.454+0000 7ff4c631b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5ee010 0x55c2aa6063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.454+0000 7ff4c631b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5ee010 0x55c2aa6063a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48190/0 (socket says 172.21.15.27:48190) 2024-04-25T00:18:07.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.454+0000 7ff4c631b640 1 -- 172.21.15.27:0/39600414 learned_addr learned my addr 172.21.15.27:0/39600414 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:07.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.454+0000 7ff4c631b640 1 -- 172.21.15.27:0/39600414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e4980 msgr2=0x55c2aa605e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.454+0000 7ff4c631b640 1 --2- 172.21.15.27:0/39600414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e4980 0x55c2aa605e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.454+0000 7ff4c631b640 1 -- 172.21.15.27:0/39600414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5e3290 msgr2=0x55c2aa5f9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.454+0000 7ff4c631b640 1 --2- 172.21.15.27:0/39600414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5e3290 0x55c2aa5f9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.454+0000 7ff4c631b640 1 -- 172.21.15.27:0/39600414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2aa21b170 con 0x55c2aa5ee010 2024-04-25T00:18:07.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.455+0000 7ff4c631b640 1 --2- 172.21.15.27:0/39600414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5ee010 0x55c2aa6063a0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8013fa0 tx=0x7ff4b80136c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=240d33c3ad4ea94a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:07.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.455+0000 7ff4ab7fe640 1 -- 172.21.15.27:0/39600414 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8026040 con 0x55c2aa5ee010 2024-04-25T00:18:07.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.455+0000 7ff4ab7fe640 1 -- 172.21.15.27:0/39600414 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4b8004830 con 0x55c2aa5ee010 2024-04-25T00:18:07.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.455+0000 7ff4ab7fe640 1 -- 172.21.15.27:0/39600414 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8010bc0 con 0x55c2aa5ee010 2024-04-25T00:18:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.456+0000 7ff4cab6c740 1 -- 172.21.15.27:0/39600414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5ee010 msgr2=0x55c2aa6063a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.456+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/39600414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5ee010 0x55c2aa6063a0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8013fa0 tx=0x7ff4b80136c0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.456+0000 7ff4cab6c740 1 -- 172.21.15.27:0/39600414 shutdown_connections 2024-04-25T00:18:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.456+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/39600414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5ee010 0x55c2aa6063a0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.456+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/39600414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e4980 0x55c2aa605e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.456+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/39600414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5e3290 0x55c2aa5f9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.456+0000 7ff4cab6c740 1 -- 172.21.15.27:0/39600414 >> 172.21.15.27:0/39600414 conn(0x55c2aa5ab950 msgr2=0x55c2aa5de950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.456+0000 7ff4cab6c740 1 -- 172.21.15.27:0/39600414 shutdown_connections 2024-04-25T00:18:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.456+0000 7ff4cab6c740 1 -- 172.21.15.27:0/39600414 wait complete. 2024-04-25T00:18:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.456+0000 7ff4cab6c740 1 -- start start 2024-04-25T00:18:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.456+0000 7ff4cab6c740 1 Processor -- start 2024-04-25T00:18:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4cab6c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e3290 0x55c2aa5f43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4cab6c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5e4980 0x55c2aa5f48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4cab6c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5ee010 0x55c2aa6636e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4cab6c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c01ade90 con 0x55c2aa5e4980 2024-04-25T00:18:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4cab6c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c01ad7c0 con 0x55c2aa5ee010 2024-04-25T00:18:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4cab6c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c01aa9e0 con 0x55c2aa5e3290 2024-04-25T00:18:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4cab6c740 10 client.0 _ll_register_callbacks cb 0x7ff4b8013e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4c5319640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5e4980 0x55c2aa5f48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4c5319640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5e4980 0x55c2aa5f48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48204/0 (socket says 172.21.15.27:48204) 2024-04-25T00:18:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4c5319640 1 -- 172.21.15.27:0/1802782076 learned_addr learned my addr 172.21.15.27:0/1802782076 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4c631b640 1 --2- 172.21.15.27:0/1802782076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5ee010 0x55c2aa6636e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4c5b1a640 1 --2- 172.21.15.27:0/1802782076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e3290 0x55c2aa5f43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4c5319640 1 -- 172.21.15.27:0/1802782076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e3290 msgr2=0x55c2aa5f43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4c5319640 1 --2- 172.21.15.27:0/1802782076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e3290 0x55c2aa5f43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4c5319640 1 -- 172.21.15.27:0/1802782076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5ee010 msgr2=0x55c2aa6636e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4c5319640 1 --2- 172.21.15.27:0/1802782076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5ee010 0x55c2aa6636e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4c5319640 1 -- 172.21.15.27:0/1802782076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2aa663c20 con 0x55c2aa5e4980 2024-04-25T00:18:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4c5319640 1 --2- 172.21.15.27:0/1802782076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5e4980 0x55c2aa5f48e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0031470 tx=0x7ff4b00318a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e74451e2d219fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:18:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4a97fa640 1 -- 172.21.15.27:0/1802782076 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b0035070 con 0x55c2aa5e4980 2024-04-25T00:18:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4a97fa640 1 -- 172.21.15.27:0/1802782076 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7ff4b0004250 con 0x55c2aa5e4980 2024-04-25T00:18:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.457+0000 7ff4cab6c740 1 -- 172.21.15.27:0/1802782076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4b8004830 con 0x55c2aa5e4980 2024-04-25T00:18:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.458+0000 7ff4a97fa640 1 -- 172.21.15.27:0/1802782076 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff4b0004450 con 0x55c2aa5e4980 2024-04-25T00:18:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.458+0000 7ff4cab6c740 10 client.16395 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.458+0000 7ff4cab6c740 1 -- 172.21.15.27:0/1802782076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c2aa664f10 con 0x55c2aa5e4980 2024-04-25T00:18:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.458+0000 7ff4a97fa640 1 -- 172.21.15.27:0/1802782076 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4b8004830 con 0x55c2aa5e4980 2024-04-25T00:18:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.458+0000 7ff4a97fa640 1 -- 172.21.15.27:0/1802782076 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7ff4b00038d0 con 0x55c2aa5e4980 2024-04-25T00:18:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.458+0000 7ff4cab6c740 10 client.16395 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.458+0000 7ff4cab6c740 10 client.16395 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:18:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.458+0000 7ff4cab6c740 20 client.16395 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.458+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/1802782076 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55c2aa6652a0 0x55c2aa6676c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.458+0000 7ff4cab6c740 4 client.16395 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T00:18:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.458+0000 7ff4cab6c740 1 -- 172.21.15.27:0/1802782076 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c2aa554800 con 0x55c2aa6652a0 2024-04-25T00:18:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.458+0000 7ff4c5b1a640 1 --2- 172.21.15.27:0/1802782076 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55c2aa6652a0 0x55c2aa6676c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.459+0000 7ff4c5b1a640 1 --2- 172.21.15.27:0/1802782076 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55c2aa6652a0 0x55c2aa6676c0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6eb119da9fac116 server_cookie=f043950508ce6b08 in_seq=0 out_seq=0 2024-04-25T00:18:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.459+0000 7ff4a97fa640 10 client.16395 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.460+0000 7ff4a97fa640 1 -- 172.21.15.27:0/1802782076 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c2aa554800 con 0x55c2aa6652a0 2024-04-25T00:18:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.460+0000 7ff4a97fa640 10 client.16395 handle_command_reply: tid=0 2024-04-25T00:18:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.460+0000 7ff4cab6c740 1 client.16395 shutdown 2024-04-25T00:18:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.460+0000 7ff4cab6c740 1 -- 172.21.15.27:0/1802782076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5e4980 msgr2=0x55c2aa5f48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.460+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/1802782076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5e4980 0x55c2aa5f48e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0031470 tx=0x7ff4b00318a0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.460+0000 7ff4cab6c740 1 -- 172.21.15.27:0/1802782076 shutdown_connections 2024-04-25T00:18:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.460+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/1802782076 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55c2aa6652a0 0x55c2aa6676c0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.460+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/1802782076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c2aa5ee010 0x55c2aa6636e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.460+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/1802782076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c2aa5e4980 0x55c2aa5f48e0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.460+0000 7ff4cab6c740 1 --2- 172.21.15.27:0/1802782076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2aa5e3290 0x55c2aa5f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.460+0000 7ff4cab6c740 1 -- 172.21.15.27:0/1802782076 >> 172.21.15.27:0/1802782076 conn(0x55c2aa5ab950 msgr2=0x55c2aa5e4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.460+0000 7ff4a97fa640 0 client.16395 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.461+0000 7ff4cab6c740 1 -- 172.21.15.27:0/1802782076 shutdown_connections 2024-04-25T00:18:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.461+0000 7ff4cab6c740 1 -- 172.21.15.27:0/1802782076 wait complete. 2024-04-25T00:18:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.461+0000 7ff4cab6c740 20 client.16395 trim_cache size 0 max 16384 2024-04-25T00:18:07.466 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:18:07.466 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T00:18:07.466 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "a40f0131-2b37-4eb9-a2bd-6fee5124aebc", 2024-04-25T00:18:07.467 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T00:18:07.467 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:18:07.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.461+0000 7ff4c7d7e640 1 -- 172.21.15.27:0/782248834 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff4ac0a2d00 msgr2=0x7ff4ac0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.461+0000 7ff4c7d7e640 1 --2- 172.21.15.27:0/782248834 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff4ac0a2d00 0x7ff4ac0a51c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8009fd0 tx=0x7ff4b8004ab0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.461+0000 7ff4c7d7e640 1 -- 172.21.15.27:0/782248834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c0101790 msgr2=0x7ff4c0198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.461+0000 7ff4c7d7e640 1 --2- 172.21.15.27:0/782248834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c0101790 0x7ff4c0198110 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00e9f0 tx=0x7ff4bc00edf0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.461+0000 7ff4c7d7e640 1 -- 172.21.15.27:0/782248834 shutdown_connections 2024-04-25T00:18:07.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.461+0000 7ff4c7d7e640 1 --2- 172.21.15.27:0/782248834 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff4ac0a2d00 0x7ff4ac0a51c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.461+0000 7ff4c7d7e640 1 --2- 172.21.15.27:0/782248834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c01077a0 0x7ff4c019c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.461+0000 7ff4c7d7e640 1 --2- 172.21.15.27:0/782248834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff4c0102150 0x7ff4c0198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.461+0000 7ff4c7d7e640 1 --2- 172.21.15.27:0/782248834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c0101790 0x7ff4c0198110 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.461+0000 7ff4c7d7e640 1 -- 172.21.15.27:0/782248834 >> 172.21.15.27:0/782248834 conn(0x7ff4c00fd2a0 msgr2=0x7ff4c00fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:07.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.461+0000 7ff4c7d7e640 1 -- 172.21.15.27:0/782248834 shutdown_connections 2024-04-25T00:18:07.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.461+0000 7ff4c7d7e640 1 -- 172.21.15.27:0/782248834 wait complete. 2024-04-25T00:18:07.474 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:18:07.474 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:18:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.652+0000 7f4e8b310640 1 -- 172.21.15.27:0/3817649250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e841007b0 msgr2=0x7f4e84100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.652+0000 7f4e8b310640 1 --2- 172.21.15.27:0/3817649250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e841007b0 0x7f4e84100b90 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c009ca0 tx=0x7f4e7c02a9e0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.652+0000 7f4e8b310640 1 -- 172.21.15.27:0/3817649250 shutdown_connections 2024-04-25T00:18:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.652+0000 7f4e8b310640 1 --2- 172.21.15.27:0/3817649250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e84105ad0 0x7f4e84105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.652+0000 7f4e8b310640 1 --2- 172.21.15.27:0/3817649250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e841010d0 0x7f4e84105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.652+0000 7f4e8b310640 1 --2- 172.21.15.27:0/3817649250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e841007b0 0x7f4e84100b90 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.652+0000 7f4e8b310640 1 -- 172.21.15.27:0/3817649250 >> 172.21.15.27:0/3817649250 conn(0x7f4e84077740 msgr2=0x7f4e84077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.653+0000 7f4e8b310640 1 -- 172.21.15.27:0/3817649250 shutdown_connections 2024-04-25T00:18:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.653+0000 7f4e8b310640 1 -- 172.21.15.27:0/3817649250 wait complete. 2024-04-25T00:18:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.654+0000 7f4e8b310640 1 Processor -- start 2024-04-25T00:18:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.654+0000 7f4e8b310640 1 -- start start 2024-04-25T00:18:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.654+0000 7f4e8b310640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e841007b0 0x7f4e84114760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.654+0000 7f4e8b310640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e841010d0 0x7f4e84114ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.654+0000 7f4e8b310640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e84105ad0 0x7f4e8410d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.654+0000 7f4e8b310640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4e84118120 con 0x7f4e84105ad0 2024-04-25T00:18:07.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.654+0000 7f4e8b310640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4e84117fa0 con 0x7f4e841010d0 2024-04-25T00:18:07.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.654+0000 7f4e8b310640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e841182a0 con 0x7f4e841007b0 2024-04-25T00:18:07.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.654+0000 7f4e890ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e841007b0 0x7f4e84114760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.654+0000 7f4e890ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e841007b0 0x7f4e84114760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:42566/0 (socket says 172.21.15.27:42566) 2024-04-25T00:18:07.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.654+0000 7f4e890ac640 1 -- 172.21.15.27:0/2710593405 learned_addr learned my addr 172.21.15.27:0/2710593405 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:07.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.655+0000 7f4e898ad640 1 --2- 172.21.15.27:0/2710593405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e84105ad0 0x7f4e8410d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.655+0000 7f4e890ac640 1 -- 172.21.15.27:0/2710593405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e841010d0 msgr2=0x7f4e84114ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.655+0000 7f4e890ac640 1 --2- 172.21.15.27:0/2710593405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e841010d0 0x7f4e84114ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.655+0000 7f4e890ac640 1 -- 172.21.15.27:0/2710593405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e84105ad0 msgr2=0x7f4e8410d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.655+0000 7f4e890ac640 1 --2- 172.21.15.27:0/2710593405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e84105ad0 0x7f4e8410d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.655+0000 7f4e890ac640 1 -- 172.21.15.27:0/2710593405 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e8410e0c0 con 0x7f4e841007b0 2024-04-25T00:18:07.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.656+0000 7f4e890ac640 1 --2- 172.21.15.27:0/2710593405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e841007b0 0x7f4e84114760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c02f870 tx=0x7f4e7c02f8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:07.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.657+0000 7f4e7a7fc640 1 -- 172.21.15.27:0/2710593405 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c046070 con 0x7f4e841007b0 2024-04-25T00:18:07.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.657+0000 7f4e7a7fc640 1 -- 172.21.15.27:0/2710593405 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e7c02e870 con 0x7f4e841007b0 2024-04-25T00:18:07.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.657+0000 7f4e8b310640 1 -- 172.21.15.27:0/2710593405 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e8410e350 con 0x7f4e841007b0 2024-04-25T00:18:07.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.657+0000 7f4e8b310640 1 -- 172.21.15.27:0/2710593405 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e84112280 con 0x7f4e841007b0 2024-04-25T00:18:07.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.657+0000 7f4e7a7fc640 1 -- 172.21.15.27:0/2710593405 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c033070 con 0x7f4e841007b0 2024-04-25T00:18:07.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.663+0000 7f4e8e01e740 1 Processor -- start 2024-04-25T00:18:07.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.664+0000 7f4e7a7fc640 1 -- 172.21.15.27:0/2710593405 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4e7c049070 con 0x7f4e841007b0 2024-04-25T00:18:07.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.664+0000 7f4e8e01e740 1 -- start start 2024-04-25T00:18:07.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.664+0000 7f4e8e01e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 0x55728c39ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.664+0000 7f4e8e01e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c39ef50 0x55728c3a90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.664+0000 7f4e8e01e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 0x55728c3aba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.664+0000 7f4e8e01e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55728c3036b0 con 0x55728c39ef50 2024-04-25T00:18:07.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.664+0000 7f4e8e01e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55728c2ab880 con 0x55728c3a9620 2024-04-25T00:18:07.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.664+0000 7f4e8e01e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55728c2ff670 con 0x55728c39e630 2024-04-25T00:18:07.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.664+0000 7f4e888ab640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 0x55728c3aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.664+0000 7f4e898ad640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 0x55728c39ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.664+0000 7f4e898ad640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 0x55728c39ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:42568/0 (socket says 172.21.15.27:42568) 2024-04-25T00:18:07.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.664+0000 7f4e888ab640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 0x55728c3aba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:54536/0 (socket says 172.21.15.27:54536) 2024-04-25T00:18:07.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.664+0000 7f4e898ad640 1 -- 172.21.15.27:0/4201367674 learned_addr learned my addr 172.21.15.27:0/4201367674 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:07.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.665+0000 7f4e890ac640 1 --2- 172.21.15.27:0/4201367674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c39ef50 0x55728c3a90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.665+0000 7f4e888ab640 1 -- 172.21.15.27:0/4201367674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 msgr2=0x55728c39ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.665+0000 7f4e888ab640 1 --2- 172.21.15.27:0/4201367674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 0x55728c39ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.665+0000 7f4e888ab640 1 -- 172.21.15.27:0/4201367674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c39ef50 msgr2=0x55728c3a90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.665+0000 7f4e888ab640 1 --2- 172.21.15.27:0/4201367674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c39ef50 0x55728c3a90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.665+0000 7f4e888ab640 1 -- 172.21.15.27:0/4201367674 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55728c0e17d0 con 0x55728c3a9620 2024-04-25T00:18:07.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.665+0000 7f4e888ab640 1 --2- 172.21.15.27:0/4201367674 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 0x55728c3aba10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4e8410f030 tx=0x7f4e7400e350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5279804c03c7552a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:07.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.666+0000 7f4e7a7fc640 1 --2- 172.21.15.27:0/2710593405 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4e700a2ed0 0x7f4e700a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.666+0000 7f4e7a7fc640 1 -- 172.21.15.27:0/2710593405 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f4e7c0f0180 con 0x7f4e841007b0 2024-04-25T00:18:07.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.666+0000 7f4e6f7fe640 1 -- 172.21.15.27:0/4201367674 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7400ed20 con 0x55728c3a9620 2024-04-25T00:18:07.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.666+0000 7f4e6f7fe640 1 -- 172.21.15.27:0/4201367674 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e74005020 con 0x55728c3a9620 2024-04-25T00:18:07.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.666+0000 7f4e6f7fe640 1 -- 172.21.15.27:0/4201367674 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e74004af0 con 0x55728c3a9620 2024-04-25T00:18:07.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.667+0000 7f4e898ad640 1 --2- 172.21.15.27:0/2710593405 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4e700a2ed0 0x7f4e700a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.667+0000 7f4e8e01e740 1 -- 172.21.15.27:0/4201367674 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 msgr2=0x55728c3aba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.667+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/4201367674 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 0x55728c3aba10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4e8410f030 tx=0x7f4e7400e350 comp rx=0 tx=0).stop 2024-04-25T00:18:07.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.668+0000 7f4e898ad640 1 --2- 172.21.15.27:0/2710593405 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4e700a2ed0 0x7f4e700a5390 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4e80009e60 tx=0x7f4e80005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:07.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.668+0000 7f4e8e01e740 1 -- 172.21.15.27:0/4201367674 shutdown_connections 2024-04-25T00:18:07.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.668+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/4201367674 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 0x55728c3aba10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.668+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/4201367674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c39ef50 0x55728c3a90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.668+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/4201367674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 0x55728c39ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.668+0000 7f4e8e01e740 1 -- 172.21.15.27:0/4201367674 >> 172.21.15.27:0/4201367674 conn(0x55728c39a950 msgr2=0x55728c39ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:07.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.668+0000 7f4e8e01e740 1 -- 172.21.15.27:0/4201367674 shutdown_connections 2024-04-25T00:18:07.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.668+0000 7f4e8e01e740 1 -- 172.21.15.27:0/4201367674 wait complete. 2024-04-25T00:18:07.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.669+0000 7f4e8e01e740 1 -- start start 2024-04-25T00:18:07.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.669+0000 7f4e8e01e740 1 Processor -- start 2024-04-25T00:18:07.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.669+0000 7f4e8e01e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 0x55728c3a6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.669+0000 7f4e8e01e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c39ef50 0x55728c3a6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.669+0000 7f4e8e01e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c3a9620 0x55728c3a6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.669+0000 7f4e8e01e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55728c3ac410 con 0x55728c3a9620 2024-04-25T00:18:07.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.670+0000 7f4e8e01e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55728c2bef50 con 0x55728c39ef50 2024-04-25T00:18:07.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.670+0000 7f4e8e01e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55728c3ac590 con 0x55728c39e630 2024-04-25T00:18:07.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.670+0000 7f4e8e01e740 10 client.0 _ll_register_callbacks cb 0x55728c2fa000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:07.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.670+0000 7f4e898ad640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c3a9620 0x55728c3a6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.670+0000 7f4e898ad640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c3a9620 0x55728c3a6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48238/0 (socket says 172.21.15.27:48238) 2024-04-25T00:18:07.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.670+0000 7f4e898ad640 1 -- 172.21.15.27:0/4077431996 learned_addr learned my addr 172.21.15.27:0/4077431996 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:07.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.670+0000 7f4e888ab640 1 --2- 172.21.15.27:0/4077431996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c39ef50 0x55728c3a6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.670+0000 7f4e890ac640 1 --2- 172.21.15.27:0/4077431996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 0x55728c3a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.672+0000 7f4e888ab640 1 -- 172.21.15.27:0/4077431996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 msgr2=0x55728c3a6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.672+0000 7f4e888ab640 1 --2- 172.21.15.27:0/4077431996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 0x55728c3a6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.672+0000 7f4e888ab640 1 -- 172.21.15.27:0/4077431996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c3a9620 msgr2=0x55728c3a6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.672+0000 7f4e888ab640 1 --2- 172.21.15.27:0/4077431996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c3a9620 0x55728c3a6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.672+0000 7f4e888ab640 1 -- 172.21.15.27:0/4077431996 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55728c0173f0 con 0x55728c39ef50 2024-04-25T00:18:07.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.673+0000 7f4e888ab640 1 --2- 172.21.15.27:0/4077431996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c39ef50 0x55728c3a6a90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4e74005770 tx=0x7f4e74005400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c211a7a4a1e6fa9b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:07.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.673+0000 7f4e6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T00:18:07.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.674+0000 7f4e6cff9640 1 -- 172.21.15.27:0/4077431996 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e74004450 con 0x55728c39ef50 2024-04-25T00:18:07.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.674+0000 7f4e6cff9640 1 -- 172.21.15.27:0/4077431996 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f4e74004810 con 0x55728c39ef50 2024-04-25T00:18:07.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.674+0000 7f4e8e01e740 1 -- 172.21.15.27:0/4077431996 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e74005020 con 0x55728c39ef50 2024-04-25T00:18:07.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.675+0000 7f4e6cff9640 1 -- 172.21.15.27:0/4077431996 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4e74004a10 con 0x55728c39ef50 2024-04-25T00:18:07.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.675+0000 7f4e8e01e740 10 client.25937 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:07.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.675+0000 7f4e8e01e740 1 -- 172.21.15.27:0/4077431996 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55728bfd70f0 con 0x55728c39ef50 2024-04-25T00:18:07.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.676+0000 7f4e6cff9640 1 -- 172.21.15.27:0/4077431996 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e74005020 con 0x55728c39ef50 2024-04-25T00:18:07.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.676+0000 7f4e6cff9640 1 -- 172.21.15.27:0/4077431996 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f4e7402d480 con 0x55728c39ef50 2024-04-25T00:18:07.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.676+0000 7f4e8e01e740 10 client.25937 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:07.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.676+0000 7f4e8e01e740 10 client.25937 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:18:07.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.676+0000 7f4e8e01e740 20 client.25937 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:07.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.676+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/4077431996 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55728c3b9850 0x55728c3bbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.676+0000 7f4e8e01e740 4 client.25937 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:18:07.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.676+0000 7f4e8e01e740 1 -- 172.21.15.27:0/4077431996 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55728c310bf0 con 0x55728c3b9850 2024-04-25T00:18:07.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.677+0000 7f4e890ac640 1 --2- 172.21.15.27:0/4077431996 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55728c3b9850 0x55728c3bbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.678+0000 7f4e6cff9640 10 client.25937 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:07.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.678+0000 7f4e890ac640 1 --2- 172.21.15.27:0/4077431996 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55728c3b9850 0x55728c3bbc40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=926d8946e0eb6544 server_cookie=817958bc857e6c57 in_seq=0 out_seq=0 2024-04-25T00:18:07.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.679+0000 7f4e6cff9640 1 -- 172.21.15.27:0/4077431996 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55728c310bf0 con 0x55728c3b9850 2024-04-25T00:18:07.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.679+0000 7f4e6cff9640 10 client.25937 handle_command_reply: tid=0 2024-04-25T00:18:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.679+0000 7f4e8e01e740 1 client.25937 shutdown 2024-04-25T00:18:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.679+0000 7f4e8e01e740 1 -- 172.21.15.27:0/4077431996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c39ef50 msgr2=0x55728c3a6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.679+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/4077431996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c39ef50 0x55728c3a6a90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4e74005770 tx=0x7f4e74005400 comp rx=0 tx=0).stop 2024-04-25T00:18:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.679+0000 7f4e8e01e740 1 -- 172.21.15.27:0/4077431996 shutdown_connections 2024-04-25T00:18:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.679+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/4077431996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c3a9620 0x55728c3a6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.679+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/4077431996 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55728c3b9850 0x55728c3bbc40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.679+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/4077431996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c39ef50 0x55728c3a6a90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.679+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/4077431996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 0x55728c3a6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.679+0000 7f4e8e01e740 1 -- 172.21.15.27:0/4077431996 >> 172.21.15.27:0/4077431996 conn(0x55728c39a950 msgr2=0x55728c399af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.679+0000 7f4e6cff9640 0 client.25937 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.680+0000 7f4e8e01e740 1 -- 172.21.15.27:0/4077431996 shutdown_connections 2024-04-25T00:18:07.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.680+0000 7f4e8e01e740 1 -- 172.21.15.27:0/4077431996 wait complete. 2024-04-25T00:18:07.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.680+0000 7f4e8e01e740 20 client.25937 trim_cache size 0 max 16384 2024-04-25T00:18:07.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.717+0000 7f4e8e01e740 1 Processor -- start 2024-04-25T00:18:07.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.718+0000 7f4e8e01e740 1 -- start start 2024-04-25T00:18:07.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.719+0000 7f4e8e01e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 0x55728c3a8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.719+0000 7f4e8e01e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c39ef50 0x55728c3b5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.720+0000 7f4e890ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 0x55728c3a8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.720+0000 7f4e890ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 0x55728c3a8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:42586/0 (socket says 172.21.15.27:42586) 2024-04-25T00:18:07.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.720+0000 7f4e890ac640 1 -- 172.21.15.27:0/1619758632 learned_addr learned my addr 172.21.15.27:0/1619758632 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:07.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.720+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/1619758632 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 0x55728c3a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.720+0000 7f4e8e01e740 1 -- 172.21.15.27:0/1619758632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6800ffa0 con 0x55728c39ef50 2024-04-25T00:18:07.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.720+0000 7f4e898ad640 1 --2- 172.21.15.27:0/1619758632 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 0x55728c3a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.720+0000 7f4e8e01e740 1 -- 172.21.15.27:0/1619758632 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4e68011ba0 con 0x55728c3a9620 2024-04-25T00:18:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.720+0000 7f4e8e01e740 1 -- 172.21.15.27:0/1619758632 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e68010670 con 0x55728c39e630 2024-04-25T00:18:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.721+0000 7f4e890ac640 1 -- 172.21.15.27:0/1619758632 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 msgr2=0x55728c3a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.721+0000 7f4e890ac640 1 --2- 172.21.15.27:0/1619758632 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 0x55728c3a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.721+0000 7f4e890ac640 1 -- 172.21.15.27:0/1619758632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c39ef50 msgr2=0x55728c3b5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.721+0000 7f4e890ac640 1 --2- 172.21.15.27:0/1619758632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c39ef50 0x55728c3b5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.721+0000 7f4e890ac640 1 -- 172.21.15.27:0/1619758632 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55728c38c090 con 0x55728c39e630 2024-04-25T00:18:07.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.722+0000 7f4e890ac640 1 --2- 172.21.15.27:0/1619758632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 0x55728c3a8c80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0bbff0 tx=0x7f4e7c0dc040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a4b80a4dca84d61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:07.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.722+0000 7f4e6effd640 1 -- 172.21.15.27:0/1619758632 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c0b6e60 con 0x55728c39e630 2024-04-25T00:18:07.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.722+0000 7f4e6effd640 1 -- 172.21.15.27:0/1619758632 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e7c0bc9f0 con 0x55728c39e630 2024-04-25T00:18:07.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.722+0000 7f4e6effd640 1 -- 172.21.15.27:0/1619758632 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c0ba680 con 0x55728c39e630 2024-04-25T00:18:07.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.723+0000 7f4e8e01e740 1 -- 172.21.15.27:0/1619758632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 msgr2=0x55728c3a8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.723+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/1619758632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 0x55728c3a8c80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0bbff0 tx=0x7f4e7c0dc040 comp rx=0 tx=0).stop 2024-04-25T00:18:07.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.724+0000 7f4e8e01e740 1 -- 172.21.15.27:0/1619758632 shutdown_connections 2024-04-25T00:18:07.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.724+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/1619758632 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 0x55728c3a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.724+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/1619758632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c39ef50 0x55728c3b5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.724+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/1619758632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39e630 0x55728c3a8c80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.724+0000 7f4e8e01e740 1 -- 172.21.15.27:0/1619758632 >> 172.21.15.27:0/1619758632 conn(0x55728c39a950 msgr2=0x55728c399260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.727+0000 7f4e8e01e740 1 -- 172.21.15.27:0/1619758632 shutdown_connections 2024-04-25T00:18:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.727+0000 7f4e8e01e740 1 -- 172.21.15.27:0/1619758632 wait complete. 2024-04-25T00:18:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.727+0000 7f4e8e01e740 1 -- start start 2024-04-25T00:18:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.727+0000 7f4e8e01e740 1 Processor -- start 2024-04-25T00:18:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.728+0000 7f4e8e01e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c39e630 0x55728c3b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.728+0000 7f4e8e01e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39ef50 0x55728c3acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.728+0000 7f4e8e01e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 0x55728c3ad330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.728+0000 7f4e8e01e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4e68011410 con 0x55728c39e630 2024-04-25T00:18:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.728+0000 7f4e8e01e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4e68010d40 con 0x55728c3a9620 2024-04-25T00:18:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.728+0000 7f4e8e01e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6800df60 con 0x55728c39ef50 2024-04-25T00:18:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.728+0000 7f4e8e01e740 10 client.0 _ll_register_callbacks cb 0x7f4e80009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:07.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.729+0000 7f4e898ad640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 0x55728c3ad330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.729+0000 7f4e898ad640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 0x55728c3ad330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:54556/0 (socket says 172.21.15.27:54556) 2024-04-25T00:18:07.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.729+0000 7f4e898ad640 1 -- 172.21.15.27:0/131871009 learned_addr learned my addr 172.21.15.27:0/131871009 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:07.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.729+0000 7f4e888ab640 1 --2- 172.21.15.27:0/131871009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39ef50 0x55728c3acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.730+0000 7f4e890ac640 1 --2- 172.21.15.27:0/131871009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c39e630 0x55728c3b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.730+0000 7f4e898ad640 1 -- 172.21.15.27:0/131871009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39ef50 msgr2=0x55728c3acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.730+0000 7f4e898ad640 1 --2- 172.21.15.27:0/131871009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39ef50 0x55728c3acdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.730+0000 7f4e898ad640 1 -- 172.21.15.27:0/131871009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c39e630 msgr2=0x55728c3b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.730+0000 7f4e898ad640 1 --2- 172.21.15.27:0/131871009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c39e630 0x55728c3b1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.730+0000 7f4e898ad640 1 -- 172.21.15.27:0/131871009 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55728c3ad8f0 con 0x55728c3a9620 2024-04-25T00:18:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.731+0000 7f4e898ad640 1 --2- 172.21.15.27:0/131871009 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 0x55728c3ad330 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4e80004550 tx=0x7f4e80004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c19b79ac863c248 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.731+0000 7f4e6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T00:18:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.732+0000 7f4e6cff9640 1 -- 172.21.15.27:0/131871009 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e80005020 con 0x55728c3a9620 2024-04-25T00:18:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.732+0000 7f4e6cff9640 1 -- 172.21.15.27:0/131871009 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f4e800051c0 con 0x55728c3a9620 2024-04-25T00:18:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.733+0000 7f4e8e01e740 1 -- 172.21.15.27:0/131871009 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e7c0bc9f0 con 0x55728c3a9620 2024-04-25T00:18:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.733+0000 7f4e6cff9640 1 -- 172.21.15.27:0/131871009 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4e800053c0 con 0x55728c3a9620 2024-04-25T00:18:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.733+0000 7f4e8e01e740 10 client.25943 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.733+0000 7f4e8e01e740 1 -- 172.21.15.27:0/131871009 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55728c41fac0 con 0x55728c3a9620 2024-04-25T00:18:07.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.734+0000 7f4e6cff9640 1 -- 172.21.15.27:0/131871009 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e80005560 con 0x55728c3a9620 2024-04-25T00:18:07.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.734+0000 7f4e6cff9640 1 -- 172.21.15.27:0/131871009 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f4e80029070 con 0x55728c3a9620 2024-04-25T00:18:07.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.735+0000 7f4e8e01e740 10 client.25943 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.735+0000 7f4e8e01e740 10 client.25943 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:18:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.735+0000 7f4e8e01e740 20 client.25943 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.735+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/131871009 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55728c420330 0x55728c422750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.735+0000 7f4e8e01e740 4 client.25943 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:18:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.735+0000 7f4e8e01e740 1 -- 172.21.15.27:0/131871009 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f4e7c0bc9f0 con 0x55728c420330 2024-04-25T00:18:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.735+0000 7f4e890ac640 1 --2- 172.21.15.27:0/131871009 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55728c420330 0x55728c422750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.736+0000 7f4e890ac640 1 --2- 172.21.15.27:0/131871009 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55728c420330 0x55728c422750 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab0ee63c92223fa4 server_cookie=9a92079f681f3cb3 in_seq=0 out_seq=0 2024-04-25T00:18:07.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.736+0000 7f4e6cff9640 10 client.25943 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:07.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.736+0000 7f4e6cff9640 1 -- 172.21.15.27:0/131871009 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x7f4e7c0bc9f0 con 0x55728c420330 2024-04-25T00:18:07.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.736+0000 7f4e6cff9640 10 client.25943 handle_command_reply: tid=0 2024-04-25T00:18:07.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.736+0000 7f4e8e01e740 1 client.25943 shutdown 2024-04-25T00:18:07.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.737+0000 7f4e8e01e740 1 -- 172.21.15.27:0/131871009 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 msgr2=0x55728c3ad330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.737+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/131871009 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 0x55728c3ad330 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4e80004550 tx=0x7f4e80004730 comp rx=0 tx=0).stop 2024-04-25T00:18:07.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.738+0000 7f4e8e01e740 1 -- 172.21.15.27:0/131871009 shutdown_connections 2024-04-25T00:18:07.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.738+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/131871009 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55728c420330 0x55728c422750 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.738+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/131871009 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55728c3a9620 0x55728c3ad330 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.738+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/131871009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55728c39ef50 0x55728c3acdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.738+0000 7f4e8e01e740 1 --2- 172.21.15.27:0/131871009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55728c39e630 0x55728c3b1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.738+0000 7f4e8e01e740 1 -- 172.21.15.27:0/131871009 >> 172.21.15.27:0/131871009 conn(0x55728c39a950 msgr2=0x55728c399af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:07.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.739+0000 7f4e6cff9640 0 client.25943 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.740+0000 7f4e8e01e740 1 -- 172.21.15.27:0/131871009 shutdown_connections 2024-04-25T00:18:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.740+0000 7f4e8e01e740 1 -- 172.21.15.27:0/131871009 wait complete. 2024-04-25T00:18:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.740+0000 7f4e8e01e740 20 client.25943 trim_cache size 0 max 16384 2024-04-25T00:18:07.742 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:18:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "scrub active (201 inodes in the stack)", 2024-04-25T00:18:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:18:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: "a40f0131-2b37-4eb9-a2bd-6fee5124aebc": { 2024-04-25T00:18:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:18:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "a40f0131-2b37-4eb9-a2bd-6fee5124aebc", 2024-04-25T00:18:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:18:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:18:07.743 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:18:07.743 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:18:07.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.743+0000 7f4e8b310640 1 -- 172.21.15.27:0/2710593405 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4e700a2ed0 msgr2=0x7f4e700a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.743+0000 7f4e8b310640 1 --2- 172.21.15.27:0/2710593405 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4e700a2ed0 0x7f4e700a5390 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4e80009e60 tx=0x7f4e80005f90 comp rx=0 tx=0).stop 2024-04-25T00:18:07.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.743+0000 7f4e8b310640 1 -- 172.21.15.27:0/2710593405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e841007b0 msgr2=0x7f4e84114760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.743+0000 7f4e8b310640 1 --2- 172.21.15.27:0/2710593405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e841007b0 0x7f4e84114760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c02f870 tx=0x7f4e7c02f8a0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.743+0000 7f4e8b310640 1 -- 172.21.15.27:0/2710593405 shutdown_connections 2024-04-25T00:18:07.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.743+0000 7f4e8b310640 1 --2- 172.21.15.27:0/2710593405 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4e700a2ed0 0x7f4e700a5390 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.743+0000 7f4e8b310640 1 --2- 172.21.15.27:0/2710593405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e84105ad0 0x7f4e8410d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.743+0000 7f4e8b310640 1 --2- 172.21.15.27:0/2710593405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e841010d0 0x7f4e84114ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.743+0000 7f4e8b310640 1 --2- 172.21.15.27:0/2710593405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e841007b0 0x7f4e84114760 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.743+0000 7f4e8b310640 1 -- 172.21.15.27:0/2710593405 >> 172.21.15.27:0/2710593405 conn(0x7f4e84077740 msgr2=0x7f4e840fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:07.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.744+0000 7f4e8b310640 1 -- 172.21.15.27:0/2710593405 shutdown_connections 2024-04-25T00:18:07.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:07.744+0000 7f4e8b310640 1 -- 172.21.15.27:0/2710593405 wait complete. 2024-04-25T00:18:07.766 INFO:tasks.cephfs.filesystem:scrub status for tag:a40f0131-2b37-4eb9-a2bd-6fee5124aebc - {'path': '/', 'tag': 'a40f0131-2b37-4eb9-a2bd-6fee5124aebc', 'options': 'recursive,force'} 2024-04-25T00:18:09.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:08 smithi027 ceph-mon[29208]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 156 MiB/s wr, 18.16k op/s 2024-04-25T00:18:09.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:08 smithi027 ceph-mon[29208]: scrub queued for path: / 2024-04-25T00:18:09.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:08 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:18:09.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:08 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:18:09.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:08 smithi139 ceph-mon[32833]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 156 MiB/s wr, 18.16k op/s 2024-04-25T00:18:09.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:08 smithi139 ceph-mon[32833]: scrub queued for path: / 2024-04-25T00:18:09.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:08 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:18:09.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:08 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:18:09.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:08 smithi153 ceph-mon[32896]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 156 MiB/s wr, 18.16k op/s 2024-04-25T00:18:09.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:08 smithi153 ceph-mon[32896]: scrub queued for path: / 2024-04-25T00:18:09.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:08 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:18:09.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:08 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:18:11.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:10 smithi027 ceph-mon[29208]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 118 MiB/s wr, 13.17k op/s 2024-04-25T00:18:11.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:10 smithi139 ceph-mon[32833]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 118 MiB/s wr, 13.17k op/s 2024-04-25T00:18:11.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:10 smithi153 ceph-mon[32896]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 118 MiB/s wr, 13.17k op/s 2024-04-25T00:18:13.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:12 smithi027 ceph-mon[29208]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 113 MiB/s wr, 12.57k op/s 2024-04-25T00:18:13.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:12 smithi139 ceph-mon[32833]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 113 MiB/s wr, 12.57k op/s 2024-04-25T00:18:13.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:12 smithi153 ceph-mon[32896]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 113 MiB/s wr, 12.57k op/s 2024-04-25T00:18:15.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:14 smithi027 ceph-mon[29208]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 153 MiB/s wr, 17.05k op/s 2024-04-25T00:18:15.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:14 smithi139 ceph-mon[32833]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 153 MiB/s wr, 17.05k op/s 2024-04-25T00:18:15.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:14 smithi153 ceph-mon[32896]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 153 MiB/s wr, 17.05k op/s 2024-04-25T00:18:17.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:16 smithi027 ceph-mon[29208]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 100 MiB/s wr, 10.65k op/s 2024-04-25T00:18:17.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:16 smithi139 ceph-mon[32833]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 100 MiB/s wr, 10.65k op/s 2024-04-25T00:18:17.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:16 smithi153 ceph-mon[32896]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 100 MiB/s wr, 10.65k op/s 2024-04-25T00:18:18.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:17 smithi027 ceph-mon[29208]: pgmap v79: 129 pgs: 129 active+clean; 13 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 150 MiB/s wr, 15.76k op/s 2024-04-25T00:18:18.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:17 smithi139 ceph-mon[32833]: pgmap v79: 129 pgs: 129 active+clean; 13 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 150 MiB/s wr, 15.76k op/s 2024-04-25T00:18:18.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:17 smithi153 ceph-mon[32896]: pgmap v79: 129 pgs: 129 active+clean; 13 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 150 MiB/s wr, 15.76k op/s 2024-04-25T00:18:20.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:20 smithi027 ceph-mon[29208]: pgmap v80: 129 pgs: 129 active+clean; 13 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 110 MiB/s wr, 11.94k op/s 2024-04-25T00:18:20.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:20 smithi139 ceph-mon[32833]: pgmap v80: 129 pgs: 129 active+clean; 13 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 110 MiB/s wr, 11.94k op/s 2024-04-25T00:18:20.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:20 smithi153 ceph-mon[32896]: pgmap v80: 129 pgs: 129 active+clean; 13 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 110 MiB/s wr, 11.94k op/s 2024-04-25T00:18:22.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:22 smithi027 ceph-mon[29208]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 105 MiB/s wr, 11.32k op/s 2024-04-25T00:18:22.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:22 smithi139 ceph-mon[32833]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 105 MiB/s wr, 11.32k op/s 2024-04-25T00:18:22.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:22 smithi153 ceph-mon[32896]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 105 MiB/s wr, 11.32k op/s 2024-04-25T00:18:24.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:24 smithi027 ceph-mon[29208]: pgmap v82: 129 pgs: 129 active+clean; 13 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 148 MiB/s wr, 15.93k op/s 2024-04-25T00:18:24.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:24 smithi139 ceph-mon[32833]: pgmap v82: 129 pgs: 129 active+clean; 13 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 148 MiB/s wr, 15.93k op/s 2024-04-25T00:18:24.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:24 smithi153 ceph-mon[32896]: pgmap v82: 129 pgs: 129 active+clean; 13 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 148 MiB/s wr, 15.93k op/s 2024-04-25T00:18:26.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:26 smithi027 ceph-mon[29208]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 106 MiB/s wr, 11.06k op/s 2024-04-25T00:18:26.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:26 smithi139 ceph-mon[32833]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 106 MiB/s wr, 11.06k op/s 2024-04-25T00:18:26.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:26 smithi153 ceph-mon[32896]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 106 MiB/s wr, 11.06k op/s 2024-04-25T00:18:28.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:28 smithi027 ceph-mon[29208]: pgmap v84: 129 pgs: 129 active+clean; 13 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 155 MiB/s wr, 16.32k op/s 2024-04-25T00:18:28.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:28 smithi139 ceph-mon[32833]: pgmap v84: 129 pgs: 129 active+clean; 13 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 155 MiB/s wr, 16.32k op/s 2024-04-25T00:18:28.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:28 smithi153 ceph-mon[32896]: pgmap v84: 129 pgs: 129 active+clean; 13 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 155 MiB/s wr, 16.32k op/s 2024-04-25T00:18:30.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:30 smithi027 ceph-mon[29208]: pgmap v85: 129 pgs: 129 active+clean; 13 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 111 MiB/s wr, 11.89k op/s 2024-04-25T00:18:30.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:30 smithi139 ceph-mon[32833]: pgmap v85: 129 pgs: 129 active+clean; 13 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 111 MiB/s wr, 11.89k op/s 2024-04-25T00:18:30.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:30 smithi153 ceph-mon[32896]: pgmap v85: 129 pgs: 129 active+clean; 13 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 111 MiB/s wr, 11.89k op/s 2024-04-25T00:18:32.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:32 smithi139 ceph-mon[32833]: pgmap v86: 129 pgs: 129 active+clean; 13 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 104 MiB/s wr, 11.18k op/s 2024-04-25T00:18:32.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:32 smithi153 ceph-mon[32896]: pgmap v86: 129 pgs: 129 active+clean; 13 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 104 MiB/s wr, 11.18k op/s 2024-04-25T00:18:32.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:32 smithi027 ceph-mon[29208]: pgmap v86: 129 pgs: 129 active+clean; 13 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 104 MiB/s wr, 11.18k op/s 2024-04-25T00:18:34.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:34 smithi139 ceph-mon[32833]: pgmap v87: 129 pgs: 129 active+clean; 13 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 142 MiB/s wr, 15.81k op/s 2024-04-25T00:18:34.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:34 smithi153 ceph-mon[32896]: pgmap v87: 129 pgs: 129 active+clean; 13 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 142 MiB/s wr, 15.81k op/s 2024-04-25T00:18:34.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:34 smithi027 ceph-mon[29208]: pgmap v87: 129 pgs: 129 active+clean; 13 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 142 MiB/s wr, 15.81k op/s 2024-04-25T00:18:36.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:36 smithi139 ceph-mon[32833]: pgmap v88: 129 pgs: 129 active+clean; 13 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 96 MiB/s wr, 10.88k op/s 2024-04-25T00:18:36.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:36 smithi153 ceph-mon[32896]: pgmap v88: 129 pgs: 129 active+clean; 13 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 96 MiB/s wr, 10.88k op/s 2024-04-25T00:18:36.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:36 smithi027 ceph-mon[29208]: pgmap v88: 129 pgs: 129 active+clean; 13 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 96 MiB/s wr, 10.88k op/s 2024-04-25T00:18:37.767 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:18:37.767 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:18:38.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:38 smithi027 ceph-mon[29208]: pgmap v89: 129 pgs: 129 active+clean; 13 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 120 MiB/s wr, 14.22k op/s 2024-04-25T00:18:38.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:38 smithi139 ceph-mon[32833]: pgmap v89: 129 pgs: 129 active+clean; 13 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 120 MiB/s wr, 14.22k op/s 2024-04-25T00:18:38.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:38 smithi153 ceph-mon[32896]: pgmap v89: 129 pgs: 129 active+clean; 13 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 120 MiB/s wr, 14.22k op/s 2024-04-25T00:18:40.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:40 smithi139 ceph-mon[32833]: pgmap v90: 129 pgs: 129 active+clean; 13 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 82 MiB/s wr, 10.42k op/s 2024-04-25T00:18:40.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:40 smithi153 ceph-mon[32896]: pgmap v90: 129 pgs: 129 active+clean; 13 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 82 MiB/s wr, 10.42k op/s 2024-04-25T00:18:40.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:40 smithi027 ceph-mon[29208]: pgmap v90: 129 pgs: 129 active+clean; 13 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 82 MiB/s wr, 10.42k op/s 2024-04-25T00:18:42.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:42 smithi139 ceph-mon[32833]: pgmap v91: 129 pgs: 129 active+clean; 13 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 76 MiB/s wr, 9.80k op/s 2024-04-25T00:18:42.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:42 smithi153 ceph-mon[32896]: pgmap v91: 129 pgs: 129 active+clean; 13 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 76 MiB/s wr, 9.80k op/s 2024-04-25T00:18:42.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:42 smithi027 ceph-mon[29208]: pgmap v91: 129 pgs: 129 active+clean; 13 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 76 MiB/s wr, 9.80k op/s 2024-04-25T00:18:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.292+0000 7efe6ce38640 1 -- 172.21.15.27:0/3963875148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe681066b0 msgr2=0x7efe68106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.292+0000 7efe6ce38640 1 --2- 172.21.15.27:0/3963875148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe681066b0 0x7efe68106a90 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7efe58009930 tx=0x7efe5802a420 comp rx=0 tx=0).stop 2024-04-25T00:18:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.293+0000 7efe6ce38640 1 -- 172.21.15.27:0/3963875148 shutdown_connections 2024-04-25T00:18:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.293+0000 7efe6ce38640 1 --2- 172.21.15.27:0/3963875148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe68100fd0 0x7efe6810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.293+0000 7efe6ce38640 1 --2- 172.21.15.27:0/3963875148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe68100610 0x7efe68100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.293+0000 7efe6ce38640 1 --2- 172.21.15.27:0/3963875148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe681066b0 0x7efe68106a90 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.293+0000 7efe6ce38640 1 -- 172.21.15.27:0/3963875148 >> 172.21.15.27:0/3963875148 conn(0x7efe68077740 msgr2=0x7efe68077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:43.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.293+0000 7efe6ce38640 1 -- 172.21.15.27:0/3963875148 shutdown_connections 2024-04-25T00:18:43.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.293+0000 7efe6ce38640 1 -- 172.21.15.27:0/3963875148 wait complete. 2024-04-25T00:18:43.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.294+0000 7efe6ce38640 1 Processor -- start 2024-04-25T00:18:43.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.294+0000 7efe6ce38640 1 -- start start 2024-04-25T00:18:43.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.294+0000 7efe6ce38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe68100610 0x7efe68198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.294+0000 7efe6ce38640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe68100fd0 0x7efe68199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.294+0000 7efe6ce38640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe681066b0 0x7efe681997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.294+0000 7efe6ce38640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efe68110a80 con 0x7efe681066b0 2024-04-25T00:18:43.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.294+0000 7efe6ce38640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efe68110900 con 0x7efe68100fd0 2024-04-25T00:18:43.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.294+0000 7efe6ce38640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe68110c00 con 0x7efe68100610 2024-04-25T00:18:43.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.294+0000 7efe6659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe68100610 0x7efe68198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:43.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.294+0000 7efe66d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe681066b0 0x7efe681997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:43.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.294+0000 7efe6659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe68100610 0x7efe68198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:52188/0 (socket says 172.21.15.27:52188) 2024-04-25T00:18:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.294+0000 7efe6659c640 1 -- 172.21.15.27:0/2171847249 learned_addr learned my addr 172.21.15.27:0/2171847249 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.294+0000 7efe5ffff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe68100fd0 0x7efe68199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.295+0000 7efe6659c640 1 -- 172.21.15.27:0/2171847249 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe68100fd0 msgr2=0x7efe68199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.295+0000 7efe6659c640 1 --2- 172.21.15.27:0/2171847249 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe68100fd0 0x7efe68199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.295+0000 7efe6659c640 1 -- 172.21.15.27:0/2171847249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe681066b0 msgr2=0x7efe681997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.295+0000 7efe6659c640 1 --2- 172.21.15.27:0/2171847249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe681066b0 0x7efe681997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.295+0000 7efe6659c640 1 -- 172.21.15.27:0/2171847249 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe681a64d0 con 0x7efe68100610 2024-04-25T00:18:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.295+0000 7efe66d9d640 1 --2- 172.21.15.27:0/2171847249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe681066b0 0x7efe681997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.295+0000 7efe6659c640 1 --2- 172.21.15.27:0/2171847249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe68100610 0x7efe68198c40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efe5802a940 tx=0x7efe58003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.295+0000 7efe5f7fe640 1 -- 172.21.15.27:0/2171847249 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5803f070 con 0x7efe68100610 2024-04-25T00:18:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.295+0000 7efe5f7fe640 1 -- 172.21.15.27:0/2171847249 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe58031c90 con 0x7efe68100610 2024-04-25T00:18:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.295+0000 7efe6ce38640 1 -- 172.21.15.27:0/2171847249 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe681a6760 con 0x7efe68100610 2024-04-25T00:18:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.295+0000 7efe5f7fe640 1 -- 172.21.15.27:0/2171847249 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe58035030 con 0x7efe68100610 2024-04-25T00:18:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.295+0000 7efe6ce38640 1 -- 172.21.15.27:0/2171847249 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe681a6c50 con 0x7efe68100610 2024-04-25T00:18:43.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.297+0000 7efe5f7fe640 1 -- 172.21.15.27:0/2171847249 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7efe58042070 con 0x7efe68100610 2024-04-25T00:18:43.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.298+0000 7efe5f7fe640 1 --2- 172.21.15.27:0/2171847249 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7efe540a2d30 0x7efe540a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.298+0000 7efe5ffff640 1 --2- 172.21.15.27:0/2171847249 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7efe540a2d30 0x7efe540a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:43.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.298+0000 7efe5f7fe640 1 -- 172.21.15.27:0/2171847249 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7efe580efe20 con 0x7efe68100610 2024-04-25T00:18:43.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.299+0000 7efe5ffff640 1 --2- 172.21.15.27:0/2171847249 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7efe540a2d30 0x7efe540a51f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efe68069420 tx=0x7efe5000b390 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:43.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.721+0000 7efe6fc25740 1 Processor -- start 2024-04-25T00:18:43.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.721+0000 7efe6fc25740 1 -- start start 2024-04-25T00:18:43.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.721+0000 7efe6fc25740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981827630 0x560981827a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.721+0000 7efe6fc25740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x5609818320e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.721+0000 7efe6fc25740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981832620 0x560981834a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.721+0000 7efe6fc25740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56098178c6b0 con 0x560981827f50 2024-04-25T00:18:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.721+0000 7efe6fc25740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560981734880 con 0x560981827630 2024-04-25T00:18:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.721+0000 7efe6fc25740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560981788670 con 0x560981832620 2024-04-25T00:18:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.722+0000 7efe5ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x5609818320e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.722+0000 7efe5ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x5609818320e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50594/0 (socket says 172.21.15.27:50594) 2024-04-25T00:18:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.722+0000 7efe5ffff640 1 -- 172.21.15.27:0/4294694809 learned_addr learned my addr 172.21.15.27:0/4294694809 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.722+0000 7efe66d9d640 1 --2- 172.21.15.27:0/4294694809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981832620 0x560981834a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.722+0000 7efe6659c640 1 --2- 172.21.15.27:0/4294694809 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981827630 0x560981827a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.722+0000 7efe5ffff640 1 -- 172.21.15.27:0/4294694809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981832620 msgr2=0x560981834a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.722+0000 7efe5ffff640 1 --2- 172.21.15.27:0/4294694809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981832620 0x560981834a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.722+0000 7efe5ffff640 1 -- 172.21.15.27:0/4294694809 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981827630 msgr2=0x560981827a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.722+0000 7efe5ffff640 1 --2- 172.21.15.27:0/4294694809 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981827630 0x560981827a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.722+0000 7efe5ffff640 1 -- 172.21.15.27:0/4294694809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56098156a7d0 con 0x560981827f50 2024-04-25T00:18:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.722+0000 7efe6659c640 1 --2- 172.21.15.27:0/4294694809 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981827630 0x560981827a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.722+0000 7efe5ffff640 1 --2- 172.21.15.27:0/4294694809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x5609818320e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7efe50003c70 tx=0x7efe50013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd59ce597090e5e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.722+0000 7efe5cff9640 1 -- 172.21.15.27:0/4294694809 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe50003ea0 con 0x560981827f50 2024-04-25T00:18:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.722+0000 7efe5cff9640 1 -- 172.21.15.27:0/4294694809 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe5001e900 con 0x560981827f50 2024-04-25T00:18:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.722+0000 7efe5cff9640 1 -- 172.21.15.27:0/4294694809 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe50026a60 con 0x560981827f50 2024-04-25T00:18:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.723+0000 7efe6fc25740 1 -- 172.21.15.27:0/4294694809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 msgr2=0x5609818320e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.723+0000 7efe6fc25740 1 --2- 172.21.15.27:0/4294694809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x5609818320e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7efe50003c70 tx=0x7efe50013000 comp rx=0 tx=0).stop 2024-04-25T00:18:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.723+0000 7efe6fc25740 1 -- 172.21.15.27:0/4294694809 shutdown_connections 2024-04-25T00:18:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.723+0000 7efe6fc25740 1 --2- 172.21.15.27:0/4294694809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981832620 0x560981834a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.723+0000 7efe6fc25740 1 --2- 172.21.15.27:0/4294694809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x5609818320e0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.723+0000 7efe6fc25740 1 --2- 172.21.15.27:0/4294694809 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981827630 0x560981827a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.723+0000 7efe6fc25740 1 -- 172.21.15.27:0/4294694809 >> 172.21.15.27:0/4294694809 conn(0x560981823950 msgr2=0x560981823d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.723+0000 7efe6fc25740 1 -- 172.21.15.27:0/4294694809 shutdown_connections 2024-04-25T00:18:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.723+0000 7efe6fc25740 1 -- 172.21.15.27:0/4294694809 wait complete. 2024-04-25T00:18:43.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.723+0000 7efe6fc25740 1 -- start start 2024-04-25T00:18:43.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe6fc25740 1 Processor -- start 2024-04-25T00:18:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe6fc25740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981827630 0x56098182f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe6fc25740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x56098182fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe6fc25740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981832620 0x560981830020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe6fc25740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560981835410 con 0x560981827f50 2024-04-25T00:18:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe6fc25740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560981747f50 con 0x560981832620 2024-04-25T00:18:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe6fc25740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560981835590 con 0x560981827630 2024-04-25T00:18:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe6fc25740 10 client.0 _ll_register_callbacks cb 0x560981783000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe5ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x56098182fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe5ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x56098182fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50602/0 (socket says 172.21.15.27:50602) 2024-04-25T00:18:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe66d9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981832620 0x560981830020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe5ffff640 1 -- 172.21.15.27:0/1037697181 learned_addr learned my addr 172.21.15.27:0/1037697181 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe6659c640 1 --2- 172.21.15.27:0/1037697181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981827630 0x56098182f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe5ffff640 1 -- 172.21.15.27:0/1037697181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981827630 msgr2=0x56098182f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe5ffff640 1 --2- 172.21.15.27:0/1037697181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981827630 0x56098182f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe5ffff640 1 -- 172.21.15.27:0/1037697181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981832620 msgr2=0x560981830020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe5ffff640 1 --2- 172.21.15.27:0/1037697181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981832620 0x560981830020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe5ffff640 1 -- 172.21.15.27:0/1037697181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609814a03f0 con 0x560981827f50 2024-04-25T00:18:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe5ffff640 1 --2- 172.21.15.27:0/1037697181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x56098182fae0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7efe50003ed0 tx=0x7efe50003f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c494327f28b0efd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.724+0000 7efe4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:18:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.725+0000 7efe4dffb640 1 -- 172.21.15.27:0/1037697181 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe50036060 con 0x560981827f50 2024-04-25T00:18:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.725+0000 7efe4dffb640 1 -- 172.21.15.27:0/1037697181 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7efe5001e630 con 0x560981827f50 2024-04-25T00:18:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.725+0000 7efe6fc25740 1 -- 172.21.15.27:0/1037697181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe5001e900 con 0x560981827f50 2024-04-25T00:18:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.725+0000 7efe4dffb640 1 -- 172.21.15.27:0/1037697181 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efe5003d050 con 0x560981827f50 2024-04-25T00:18:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.725+0000 7efe6fc25740 10 client.16422 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.725+0000 7efe6fc25740 1 -- 172.21.15.27:0/1037697181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5609814600f0 con 0x560981827f50 2024-04-25T00:18:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.725+0000 7efe4dffb640 1 -- 172.21.15.27:0/1037697181 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe5001e900 con 0x560981827f50 2024-04-25T00:18:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.725+0000 7efe4dffb640 1 -- 172.21.15.27:0/1037697181 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7efe50031470 con 0x560981827f50 2024-04-25T00:18:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.725+0000 7efe6fc25740 10 client.16422 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.725+0000 7efe6fc25740 10 client.16422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:18:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.725+0000 7efe6fc25740 20 client.16422 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.725+0000 7efe6fc25740 1 --2- 172.21.15.27:0/1037697181 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560981842840 0x560981844c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.725+0000 7efe6fc25740 4 client.16422 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:18:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.725+0000 7efe6fc25740 1 -- 172.21.15.27:0/1037697181 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560981799bf0 con 0x560981842840 2024-04-25T00:18:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.725+0000 7efe6659c640 1 --2- 172.21.15.27:0/1037697181 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560981842840 0x560981844c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.726+0000 7efe6659c640 1 --2- 172.21.15.27:0/1037697181 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560981842840 0x560981844c30 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=157b1c45b50e55ec server_cookie=762a035227d9eecd in_seq=0 out_seq=0 2024-04-25T00:18:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.726+0000 7efe4dffb640 10 client.16422 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.727+0000 7efe4dffb640 1 -- 172.21.15.27:0/1037697181 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560981799bf0 con 0x560981842840 2024-04-25T00:18:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.727+0000 7efe4dffb640 10 client.16422 handle_command_reply: tid=0 2024-04-25T00:18:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.727+0000 7efe6fc25740 1 client.16422 shutdown 2024-04-25T00:18:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.727+0000 7efe6fc25740 1 -- 172.21.15.27:0/1037697181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 msgr2=0x56098182fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.727+0000 7efe6fc25740 1 --2- 172.21.15.27:0/1037697181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x56098182fae0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7efe50003ed0 tx=0x7efe50003f00 comp rx=0 tx=0).stop 2024-04-25T00:18:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.727+0000 7efe6fc25740 1 -- 172.21.15.27:0/1037697181 shutdown_connections 2024-04-25T00:18:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.727+0000 7efe6fc25740 1 --2- 172.21.15.27:0/1037697181 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560981842840 0x560981844c30 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.727+0000 7efe6fc25740 1 --2- 172.21.15.27:0/1037697181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981832620 0x560981830020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.727+0000 7efe6fc25740 1 --2- 172.21.15.27:0/1037697181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x56098182fae0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.727+0000 7efe6fc25740 1 --2- 172.21.15.27:0/1037697181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981827630 0x56098182f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.727+0000 7efe6fc25740 1 -- 172.21.15.27:0/1037697181 >> 172.21.15.27:0/1037697181 conn(0x560981823950 msgr2=0x560981829630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.727+0000 7efe4dffb640 0 client.16422 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.727+0000 7efe6fc25740 1 -- 172.21.15.27:0/1037697181 shutdown_connections 2024-04-25T00:18:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.727+0000 7efe6fc25740 1 -- 172.21.15.27:0/1037697181 wait complete. 2024-04-25T00:18:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.727+0000 7efe6fc25740 20 client.16422 trim_cache size 0 max 16384 2024-04-25T00:18:43.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.751+0000 7efe6fc25740 1 Processor -- start 2024-04-25T00:18:43.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.751+0000 7efe6fc25740 1 -- start start 2024-04-25T00:18:43.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.751+0000 7efe6fc25740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981827630 0x560981831cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.751+0000 7efe6fc25740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x5609818298d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.751+0000 7efe6fc25740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981832620 0x56098182fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.751+0000 7efe6fc25740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efe681b0d80 con 0x560981827f50 2024-04-25T00:18:43.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.751+0000 7efe6fc25740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efe681b2980 con 0x560981832620 2024-04-25T00:18:43.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.751+0000 7efe6fc25740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe681b1450 con 0x560981827630 2024-04-25T00:18:43.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.752+0000 7efe5ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x5609818298d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:43.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.752+0000 7efe66d9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981832620 0x56098182fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:43.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.752+0000 7efe5ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x5609818298d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50610/0 (socket says 172.21.15.27:50610) 2024-04-25T00:18:43.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.752+0000 7efe5ffff640 1 -- 172.21.15.27:0/4273352472 learned_addr learned my addr 172.21.15.27:0/4273352472 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:43.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.752+0000 7efe6659c640 1 --2- 172.21.15.27:0/4273352472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981827630 0x560981831cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:43.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.752+0000 7efe5ffff640 1 -- 172.21.15.27:0/4273352472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981827630 msgr2=0x560981831cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:43.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.752+0000 7efe5ffff640 1 --2- 172.21.15.27:0/4273352472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981827630 0x560981831cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.752+0000 7efe5ffff640 1 -- 172.21.15.27:0/4273352472 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981832620 msgr2=0x56098182fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:43.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.752+0000 7efe5ffff640 1 --2- 172.21.15.27:0/4273352472 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981832620 0x56098182fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.752+0000 7efe5ffff640 1 -- 172.21.15.27:0/4273352472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560981845190 con 0x560981827f50 2024-04-25T00:18:43.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.752+0000 7efe66d9d640 1 --2- 172.21.15.27:0/4273352472 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981832620 0x56098182fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:43.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.752+0000 7efe6659c640 1 --2- 172.21.15.27:0/4273352472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981827630 0x560981831cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:18:43.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.752+0000 7efe5ffff640 1 --2- 172.21.15.27:0/4273352472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x5609818298d0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7efe500319e0 tx=0x7efe500036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce58155e3868ab51 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:43.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.752+0000 7efe4ffff640 1 -- 172.21.15.27:0/4273352472 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5002a020 con 0x560981827f50 2024-04-25T00:18:43.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.752+0000 7efe4ffff640 1 -- 172.21.15.27:0/4273352472 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe50011ca0 con 0x560981827f50 2024-04-25T00:18:43.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.752+0000 7efe4ffff640 1 -- 172.21.15.27:0/4273352472 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe50036070 con 0x560981827f50 2024-04-25T00:18:43.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.753+0000 7efe6fc25740 1 -- 172.21.15.27:0/4273352472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 msgr2=0x5609818298d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:43.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.753+0000 7efe6fc25740 1 --2- 172.21.15.27:0/4273352472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x5609818298d0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7efe500319e0 tx=0x7efe500036f0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.753+0000 7efe6fc25740 1 -- 172.21.15.27:0/4273352472 shutdown_connections 2024-04-25T00:18:43.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.753+0000 7efe6fc25740 1 --2- 172.21.15.27:0/4273352472 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981832620 0x56098182fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.753+0000 7efe6fc25740 1 --2- 172.21.15.27:0/4273352472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x5609818298d0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.753+0000 7efe6fc25740 1 --2- 172.21.15.27:0/4273352472 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981827630 0x560981831cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.753+0000 7efe6fc25740 1 -- 172.21.15.27:0/4273352472 >> 172.21.15.27:0/4273352472 conn(0x560981823950 msgr2=0x560981822af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:43.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.753+0000 7efe6fc25740 1 -- 172.21.15.27:0/4273352472 shutdown_connections 2024-04-25T00:18:43.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.753+0000 7efe6fc25740 1 -- 172.21.15.27:0/4273352472 wait complete. 2024-04-25T00:18:43.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.753+0000 7efe6fc25740 1 -- start start 2024-04-25T00:18:43.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.754+0000 7efe6fc25740 1 Processor -- start 2024-04-25T00:18:43.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.754+0000 7efe6fc25740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981827630 0x56098183aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.754+0000 7efe6fc25740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x56098182be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.754+0000 7efe6fc25740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981832620 0x56098182c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.754+0000 7efe6fc25740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efe681b21f0 con 0x560981827f50 2024-04-25T00:18:43.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.754+0000 7efe6fc25740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efe681b1b20 con 0x560981827630 2024-04-25T00:18:43.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.754+0000 7efe6fc25740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe681aed40 con 0x560981832620 2024-04-25T00:18:43.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.754+0000 7efe6fc25740 10 client.0 _ll_register_callbacks cb 0x7efe60009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:18:43.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.754+0000 7efe5ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x56098182be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.754+0000 7efe5ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x56098182be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50624/0 (socket says 172.21.15.27:50624) 2024-04-25T00:18:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.754+0000 7efe5ffff640 1 -- 172.21.15.27:0/1857219611 learned_addr learned my addr 172.21.15.27:0/1857219611 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:18:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.754+0000 7efe5ffff640 1 -- 172.21.15.27:0/1857219611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981832620 msgr2=0x56098182c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.754+0000 7efe5ffff640 1 --2- 172.21.15.27:0/1857219611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981832620 0x56098182c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.754+0000 7efe5ffff640 1 -- 172.21.15.27:0/1857219611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981827630 msgr2=0x56098183aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.754+0000 7efe5ffff640 1 --2- 172.21.15.27:0/1857219611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981827630 0x56098183aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.754+0000 7efe5ffff640 1 -- 172.21.15.27:0/1857219611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56098182c9f0 con 0x560981827f50 2024-04-25T00:18:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.755+0000 7efe5ffff640 1 --2- 172.21.15.27:0/1857219611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x56098182be10 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7efe50031c40 tx=0x7efe500316a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788754d41788c528 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.755+0000 7efe4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:18:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.755+0000 7efe4dffb640 1 -- 172.21.15.27:0/1857219611 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5002a030 con 0x560981827f50 2024-04-25T00:18:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.755+0000 7efe4dffb640 1 -- 172.21.15.27:0/1857219611 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7efe50005250 con 0x560981827f50 2024-04-25T00:18:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.756+0000 7efe6fc25740 1 -- 172.21.15.27:0/1857219611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe50011ca0 con 0x560981827f50 2024-04-25T00:18:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.756+0000 7efe4dffb640 1 -- 172.21.15.27:0/1857219611 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efe50036070 con 0x560981827f50 2024-04-25T00:18:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.756+0000 7efe6fc25740 10 client.16428 fetch_fsmap learned FSMap version 30 2024-04-25T00:18:43.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.756+0000 7efe6fc25740 1 -- 172.21.15.27:0/1857219611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5609818a8ad0 con 0x560981827f50 2024-04-25T00:18:43.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.757+0000 7efe4dffb640 1 -- 172.21.15.27:0/1857219611 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe50011ca0 con 0x560981827f50 2024-04-25T00:18:43.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.757+0000 7efe4dffb640 1 -- 172.21.15.27:0/1857219611 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7efe50048030 con 0x560981827f50 2024-04-25T00:18:43.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.757+0000 7efe6fc25740 10 client.16428 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T00:18:43.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.757+0000 7efe6fc25740 10 client.16428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:18:43.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.757+0000 7efe6fc25740 20 client.16428 populate_metadata read hostname 'smithi027' 2024-04-25T00:18:43.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.757+0000 7efe6fc25740 1 --2- 172.21.15.27:0/1857219611 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5609818a9240 0x5609818ab680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:43.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.757+0000 7efe6fc25740 4 client.16428 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:18:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.757+0000 7efe6fc25740 1 -- 172.21.15.27:0/1857219611 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560981793f40 con 0x5609818a9240 2024-04-25T00:18:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.758+0000 7efe6659c640 1 --2- 172.21.15.27:0/1857219611 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5609818a9240 0x5609818ab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.760+0000 7efe4dffb640 10 client.16428 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:43.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.760+0000 7efe6659c640 1 --2- 172.21.15.27:0/1857219611 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5609818a9240 0x5609818ab680 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2fafdf48c854123 server_cookie=fee6012761176384 in_seq=0 out_seq=0 2024-04-25T00:18:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.761+0000 7efe4dffb640 1 -- 172.21.15.27:0/1857219611 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x560981793f40 con 0x5609818a9240 2024-04-25T00:18:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.761+0000 7efe4dffb640 10 client.16428 handle_command_reply: tid=0 2024-04-25T00:18:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.761+0000 7efe6fc25740 1 client.16428 shutdown 2024-04-25T00:18:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.761+0000 7efe6fc25740 1 -- 172.21.15.27:0/1857219611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 msgr2=0x56098182be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.761+0000 7efe6fc25740 1 --2- 172.21.15.27:0/1857219611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x56098182be10 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7efe50031c40 tx=0x7efe500316a0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.761+0000 7efe6fc25740 1 -- 172.21.15.27:0/1857219611 shutdown_connections 2024-04-25T00:18:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.761+0000 7efe6fc25740 1 --2- 172.21.15.27:0/1857219611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560981832620 0x56098182c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.761+0000 7efe6fc25740 1 --2- 172.21.15.27:0/1857219611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560981827f50 0x56098182be10 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.761+0000 7efe6fc25740 1 --2- 172.21.15.27:0/1857219611 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5609818a9240 0x5609818ab680 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.761+0000 7efe6fc25740 1 --2- 172.21.15.27:0/1857219611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560981827630 0x56098183aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.761+0000 7efe4dffb640 0 client.16428 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:18:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.761+0000 7efe6fc25740 1 -- 172.21.15.27:0/1857219611 >> 172.21.15.27:0/1857219611 conn(0x560981823950 msgr2=0x56098183d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.761+0000 7efe6fc25740 1 -- 172.21.15.27:0/1857219611 shutdown_connections 2024-04-25T00:18:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.761+0000 7efe6fc25740 1 -- 172.21.15.27:0/1857219611 wait complete. 2024-04-25T00:18:43.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.762+0000 7efe6fc25740 20 client.16428 trim_cache size 0 max 16384 2024-04-25T00:18:43.764 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:18:43.764 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "scrub active (201 inodes in the stack)", 2024-04-25T00:18:43.764 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:18:43.764 INFO:teuthology.orchestra.run.smithi027.stdout: "a40f0131-2b37-4eb9-a2bd-6fee5124aebc": { 2024-04-25T00:18:43.764 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:18:43.764 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "a40f0131-2b37-4eb9-a2bd-6fee5124aebc", 2024-04-25T00:18:43.764 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:18:43.764 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:18:43.765 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:18:43.765 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:18:43.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.762+0000 7efe6ce38640 1 -- 172.21.15.27:0/2171847249 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7efe540a2d30 msgr2=0x7efe540a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:43.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.762+0000 7efe6ce38640 1 --2- 172.21.15.27:0/2171847249 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7efe540a2d30 0x7efe540a51f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efe68069420 tx=0x7efe5000b390 comp rx=0 tx=0).stop 2024-04-25T00:18:43.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.762+0000 7efe6ce38640 1 -- 172.21.15.27:0/2171847249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe68100610 msgr2=0x7efe68198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:43.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.762+0000 7efe6ce38640 1 --2- 172.21.15.27:0/2171847249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe68100610 0x7efe68198c40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efe5802a940 tx=0x7efe58003b90 comp rx=0 tx=0).stop 2024-04-25T00:18:43.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.762+0000 7efe6ce38640 1 -- 172.21.15.27:0/2171847249 shutdown_connections 2024-04-25T00:18:43.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.762+0000 7efe6ce38640 1 --2- 172.21.15.27:0/2171847249 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7efe540a2d30 0x7efe540a51f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.762+0000 7efe6ce38640 1 --2- 172.21.15.27:0/2171847249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe681066b0 0x7efe681997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.762+0000 7efe6ce38640 1 --2- 172.21.15.27:0/2171847249 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe68100fd0 0x7efe68199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.762+0000 7efe6ce38640 1 --2- 172.21.15.27:0/2171847249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe68100610 0x7efe68198c40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:43.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.762+0000 7efe6ce38640 1 -- 172.21.15.27:0/2171847249 >> 172.21.15.27:0/2171847249 conn(0x7efe68077740 msgr2=0x7efe680ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:43.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.762+0000 7efe6ce38640 1 -- 172.21.15.27:0/2171847249 shutdown_connections 2024-04-25T00:18:43.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:18:43.762+0000 7efe6ce38640 1 -- 172.21.15.27:0/2171847249 wait complete. 2024-04-25T00:18:43.780 INFO:tasks.cephfs.filesystem:scrub status for tag:a40f0131-2b37-4eb9-a2bd-6fee5124aebc - {'path': '/', 'tag': 'a40f0131-2b37-4eb9-a2bd-6fee5124aebc', 'options': 'recursive,force'} 2024-04-25T00:18:44.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:44 smithi139 ceph-mon[32833]: pgmap v92: 129 pgs: 129 active+clean; 13 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 100 MiB/s wr, 13.35k op/s 2024-04-25T00:18:44.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:44 smithi153 ceph-mon[32896]: pgmap v92: 129 pgs: 129 active+clean; 13 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 100 MiB/s wr, 13.35k op/s 2024-04-25T00:18:44.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:44 smithi027 ceph-mon[29208]: pgmap v92: 129 pgs: 129 active+clean; 13 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 100 MiB/s wr, 13.35k op/s 2024-04-25T00:18:46.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:46 smithi139 ceph-mon[32833]: pgmap v93: 129 pgs: 129 active+clean; 13 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 64 MiB/s wr, 8.95k op/s 2024-04-25T00:18:46.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:46 smithi153 ceph-mon[32896]: pgmap v93: 129 pgs: 129 active+clean; 13 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 64 MiB/s wr, 8.95k op/s 2024-04-25T00:18:46.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:46 smithi027 ceph-mon[29208]: pgmap v93: 129 pgs: 129 active+clean; 13 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 64 MiB/s wr, 8.95k op/s 2024-04-25T00:18:48.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:48 smithi139 ceph-mon[32833]: pgmap v94: 129 pgs: 129 active+clean; 13 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 77 MiB/s wr, 11.01k op/s 2024-04-25T00:18:48.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:48 smithi153 ceph-mon[32896]: pgmap v94: 129 pgs: 129 active+clean; 13 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 77 MiB/s wr, 11.01k op/s 2024-04-25T00:18:48.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:48 smithi027 ceph-mon[29208]: pgmap v94: 129 pgs: 129 active+clean; 13 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 77 MiB/s wr, 11.01k op/s 2024-04-25T00:18:50.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:50 smithi139 ceph-mon[32833]: pgmap v95: 129 pgs: 129 active+clean; 13 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 61 MiB/s wr, 8.83k op/s 2024-04-25T00:18:50.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:50 smithi153 ceph-mon[32896]: pgmap v95: 129 pgs: 129 active+clean; 13 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 61 MiB/s wr, 8.83k op/s 2024-04-25T00:18:50.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:50 smithi027 ceph-mon[29208]: pgmap v95: 129 pgs: 129 active+clean; 13 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 61 MiB/s wr, 8.83k op/s 2024-04-25T00:18:52.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:52 smithi139 ceph-mon[32833]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 50 MiB/s wr, 7.37k op/s 2024-04-25T00:18:52.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:52 smithi153 ceph-mon[32896]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 50 MiB/s wr, 7.37k op/s 2024-04-25T00:18:52.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:52 smithi027 ceph-mon[29208]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 50 MiB/s wr, 7.37k op/s 2024-04-25T00:18:53.504 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:53 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:18:53.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:53 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:18:53.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:53 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:18:54.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:54 smithi153 ceph-mon[32896]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 69 MiB/s wr, 10.49k op/s 2024-04-25T00:18:54.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:54 smithi027 ceph-mon[29208]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 69 MiB/s wr, 10.49k op/s 2024-04-25T00:18:54.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:54 smithi139 ceph-mon[32833]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 69 MiB/s wr, 10.49k op/s 2024-04-25T00:18:56.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:56 smithi153 ceph-mon[32896]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 48 MiB/s wr, 7.32k op/s 2024-04-25T00:18:56.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:56 smithi027 ceph-mon[29208]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 48 MiB/s wr, 7.32k op/s 2024-04-25T00:18:56.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:56 smithi139 ceph-mon[32833]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 48 MiB/s wr, 7.32k op/s 2024-04-25T00:18:58.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:18:58 smithi153 ceph-mon[32896]: pgmap v99: 129 pgs: 129 active+clean; 13 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 63 MiB/s wr, 9.99k op/s 2024-04-25T00:18:58.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:18:58 smithi027 ceph-mon[29208]: pgmap v99: 129 pgs: 129 active+clean; 13 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 63 MiB/s wr, 9.99k op/s 2024-04-25T00:18:58.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:18:58 smithi139 ceph-mon[32833]: pgmap v99: 129 pgs: 129 active+clean; 13 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 63 MiB/s wr, 9.99k op/s 2024-04-25T00:19:00.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:00 smithi153 ceph-mon[32896]: pgmap v100: 129 pgs: 129 active+clean; 13 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 55 MiB/s wr, 8.77k op/s 2024-04-25T00:19:00.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:00 smithi027 ceph-mon[29208]: pgmap v100: 129 pgs: 129 active+clean; 13 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 55 MiB/s wr, 8.77k op/s 2024-04-25T00:19:00.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:00 smithi139 ceph-mon[32833]: pgmap v100: 129 pgs: 129 active+clean; 13 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 55 MiB/s wr, 8.77k op/s 2024-04-25T00:19:02.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:02 smithi027 ceph-mon[29208]: pgmap v101: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 49 MiB/s wr, 7.93k op/s 2024-04-25T00:19:02.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:02 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:19:02.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:02 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:19:02.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:02 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:19:02.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:02 smithi139 ceph-mon[32833]: pgmap v101: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 49 MiB/s wr, 7.93k op/s 2024-04-25T00:19:02.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:02 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:19:02.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:02 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:19:02.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:02 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:19:02.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:02 smithi153 ceph-mon[32896]: pgmap v101: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 49 MiB/s wr, 7.93k op/s 2024-04-25T00:19:02.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:02 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:19:02.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:02 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:19:02.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:02 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:19:04.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:04 smithi027 ceph-mon[29208]: pgmap v102: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 67 MiB/s wr, 10.82k op/s 2024-04-25T00:19:04.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:04 smithi139 ceph-mon[32833]: pgmap v102: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 67 MiB/s wr, 10.82k op/s 2024-04-25T00:19:04.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:04 smithi153 ceph-mon[32896]: pgmap v102: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 67 MiB/s wr, 10.82k op/s 2024-04-25T00:19:06.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:06 smithi027 ceph-mon[29208]: pgmap v103: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 48 MiB/s wr, 7.80k op/s 2024-04-25T00:19:06.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:06 smithi139 ceph-mon[32833]: pgmap v103: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 48 MiB/s wr, 7.80k op/s 2024-04-25T00:19:06.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:06 smithi153 ceph-mon[32896]: pgmap v103: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 48 MiB/s wr, 7.80k op/s 2024-04-25T00:19:08.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:08 smithi027 ceph-mon[29208]: pgmap v104: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 58 MiB/s wr, 9.37k op/s 2024-04-25T00:19:08.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:08 smithi139 ceph-mon[32833]: pgmap v104: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 58 MiB/s wr, 9.37k op/s 2024-04-25T00:19:08.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:08 smithi153 ceph-mon[32896]: pgmap v104: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 58 MiB/s wr, 9.37k op/s 2024-04-25T00:19:10.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:10 smithi027 ceph-mon[29208]: pgmap v105: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 44 MiB/s wr, 7.10k op/s 2024-04-25T00:19:10.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:10 smithi139 ceph-mon[32833]: pgmap v105: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 44 MiB/s wr, 7.10k op/s 2024-04-25T00:19:10.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:10 smithi153 ceph-mon[32896]: pgmap v105: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 44 MiB/s wr, 7.10k op/s 2024-04-25T00:19:12.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:12 smithi027 ceph-mon[29208]: pgmap v106: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 43 MiB/s wr, 6.82k op/s 2024-04-25T00:19:12.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:12 smithi139 ceph-mon[32833]: pgmap v106: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 43 MiB/s wr, 6.82k op/s 2024-04-25T00:19:12.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:12 smithi153 ceph-mon[32896]: pgmap v106: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 43 MiB/s wr, 6.82k op/s 2024-04-25T00:19:13.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:19:13.781 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:19:14.665 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:14 smithi027 ceph-mon[29208]: pgmap v107: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 63 MiB/s wr, 10.29k op/s 2024-04-25T00:19:14.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:14 smithi139 ceph-mon[32833]: pgmap v107: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 63 MiB/s wr, 10.29k op/s 2024-04-25T00:19:14.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:14 smithi153 ceph-mon[32896]: pgmap v107: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 63 MiB/s wr, 10.29k op/s 2024-04-25T00:19:15.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:15 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:19:15.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:15 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:19:15.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:15 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:19:15.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.834+0000 7ff1c0264640 1 -- 172.21.15.27:0/1504272052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b81029c0 msgr2=0x7ff1b8102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:15.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.834+0000 7ff1c0264640 1 --2- 172.21.15.27:0/1504272052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b81029c0 0x7ff1b8102da0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4009900 tx=0x7ff1b402a510 comp rx=0 tx=0).stop 2024-04-25T00:19:15.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.834+0000 7ff1c0264640 1 -- 172.21.15.27:0/1504272052 shutdown_connections 2024-04-25T00:19:15.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.834+0000 7ff1c0264640 1 --2- 172.21.15.27:0/1504272052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b810bd00 0x7ff1b810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.834+0000 7ff1c0264640 1 --2- 172.21.15.27:0/1504272052 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1b81032e0 0x7ff1b810b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.834+0000 7ff1c0264640 1 --2- 172.21.15.27:0/1504272052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b81029c0 0x7ff1b8102da0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.834+0000 7ff1c0264640 1 -- 172.21.15.27:0/1504272052 >> 172.21.15.27:0/1504272052 conn(0x7ff1b8077740 msgr2=0x7ff1b8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:15.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.834+0000 7ff1c0264640 1 -- 172.21.15.27:0/1504272052 shutdown_connections 2024-04-25T00:19:15.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.835+0000 7ff1c0264640 1 -- 172.21.15.27:0/1504272052 wait complete. 2024-04-25T00:19:15.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.835+0000 7ff1c0264640 1 Processor -- start 2024-04-25T00:19:15.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.835+0000 7ff1c0264640 1 -- start start 2024-04-25T00:19:15.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.836+0000 7ff1c0264640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b81029c0 0x7ff1b8102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.836+0000 7ff1c0264640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1b81032e0 0x7ff1b8100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.836+0000 7ff1c0264640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b810bd00 0x7ff1b8100c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.836+0000 7ff1c0264640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b8111b50 con 0x7ff1b810bd00 2024-04-25T00:19:15.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.836+0000 7ff1c0264640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b81119d0 con 0x7ff1b81032e0 2024-04-25T00:19:15.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.836+0000 7ff1c0264640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b8111cd0 con 0x7ff1b81029c0 2024-04-25T00:19:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.836+0000 7ff1be000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b81029c0 0x7ff1b8102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.836+0000 7ff1bd7ff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1b81032e0 0x7ff1b8100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.836+0000 7ff1bd7ff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1b81032e0 0x7ff1b8100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:38002/0 (socket says 172.21.15.27:38002) 2024-04-25T00:19:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.836+0000 7ff1bd7ff640 1 -- 172.21.15.27:0/2119162303 learned_addr learned my addr 172.21.15.27:0/2119162303 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:19:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.836+0000 7ff1be801640 1 --2- 172.21.15.27:0/2119162303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b810bd00 0x7ff1b8100c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.837+0000 7ff1bd7ff640 1 -- 172.21.15.27:0/2119162303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b81029c0 msgr2=0x7ff1b8102210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.837+0000 7ff1bd7ff640 1 --2- 172.21.15.27:0/2119162303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b81029c0 0x7ff1b8102210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.837+0000 7ff1bd7ff640 1 -- 172.21.15.27:0/2119162303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b810bd00 msgr2=0x7ff1b8100c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.837+0000 7ff1bd7ff640 1 --2- 172.21.15.27:0/2119162303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b810bd00 0x7ff1b8100c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.837+0000 7ff1bd7ff640 1 -- 172.21.15.27:0/2119162303 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b81aaa10 con 0x7ff1b81032e0 2024-04-25T00:19:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.837+0000 7ff1be000640 1 --2- 172.21.15.27:0/2119162303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b81029c0 0x7ff1b8102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:19:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.837+0000 7ff1bd7ff640 1 --2- 172.21.15.27:0/2119162303 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1b81032e0 0x7ff1b8100610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff1a800d950 tx=0x7ff1a800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:15.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.837+0000 7ff1aeffd640 1 -- 172.21.15.27:0/2119162303 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8017070 con 0x7ff1b81032e0 2024-04-25T00:19:15.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.837+0000 7ff1c0264640 1 -- 172.21.15.27:0/2119162303 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1b81aac40 con 0x7ff1b81032e0 2024-04-25T00:19:15.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.837+0000 7ff1aeffd640 1 -- 172.21.15.27:0/2119162303 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1a8009c30 con 0x7ff1b81032e0 2024-04-25T00:19:15.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.838+0000 7ff1aeffd640 1 -- 172.21.15.27:0/2119162303 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a800f6b0 con 0x7ff1b81032e0 2024-04-25T00:19:15.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.838+0000 7ff1c0264640 1 -- 172.21.15.27:0/2119162303 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1b81ab190 con 0x7ff1b81032e0 2024-04-25T00:19:15.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.840+0000 7ff1aeffd640 1 -- 172.21.15.27:0/2119162303 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff1a80109d0 con 0x7ff1b81032e0 2024-04-25T00:19:15.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.841+0000 7ff1aeffd640 1 --2- 172.21.15.27:0/2119162303 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff1a40a2d00 0x7ff1a40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.841+0000 7ff1be000640 1 --2- 172.21.15.27:0/2119162303 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff1a40a2d00 0x7ff1a40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.841+0000 7ff1aeffd640 1 -- 172.21.15.27:0/2119162303 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7ff1a80cf790 con 0x7ff1b81032e0 2024-04-25T00:19:15.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.841+0000 7ff1be000640 1 --2- 172.21.15.27:0/2119162303 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff1a40a2d00 0x7ff1a40a51c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4000c00 tx=0x7ff1b4005a40 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:15.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.878+0000 7ff1c2f72740 1 Processor -- start 2024-04-25T00:19:15.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.878+0000 7ff1c2f72740 1 -- start start 2024-04-25T00:19:15.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.878+0000 7ff1c2f72740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605e630 0x55b23605ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.878+0000 7ff1c2f72740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b23605ef50 0x55b2360690e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.878+0000 7ff1c2f72740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b236069620 0x55b23606ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.878+0000 7ff1c2f72740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b235fc36b0 con 0x55b23605e630 2024-04-25T00:19:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.878+0000 7ff1c2f72740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b235f6b880 con 0x55b23605ef50 2024-04-25T00:19:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.878+0000 7ff1c2f72740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b235fbf670 con 0x55b236069620 2024-04-25T00:19:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.878+0000 7ff1be000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605e630 0x55b23605ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.878+0000 7ff1be000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605e630 0x55b23605ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59930/0 (socket says 172.21.15.27:59930) 2024-04-25T00:19:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.878+0000 7ff1be000640 1 -- 172.21.15.27:0/2714041188 learned_addr learned my addr 172.21.15.27:0/2714041188 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:19:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.878+0000 7ff1be801640 1 --2- 172.21.15.27:0/2714041188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b236069620 0x55b23606ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.878+0000 7ff1bd7ff640 1 --2- 172.21.15.27:0/2714041188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b23605ef50 0x55b2360690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.879+0000 7ff1be000640 1 -- 172.21.15.27:0/2714041188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b236069620 msgr2=0x55b23606ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.879+0000 7ff1be000640 1 --2- 172.21.15.27:0/2714041188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b236069620 0x55b23606ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.879+0000 7ff1be000640 1 -- 172.21.15.27:0/2714041188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b23605ef50 msgr2=0x55b2360690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.879+0000 7ff1be000640 1 --2- 172.21.15.27:0/2714041188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b23605ef50 0x55b2360690e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.879+0000 7ff1be000640 1 -- 172.21.15.27:0/2714041188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b235da17d0 con 0x55b23605e630 2024-04-25T00:19:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.879+0000 7ff1bd7ff640 1 --2- 172.21.15.27:0/2714041188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b23605ef50 0x55b2360690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:19:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.879+0000 7ff1be000640 1 --2- 172.21.15.27:0/2714041188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605e630 0x55b23605ea10 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4033000 tx=0x7ff1b403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8baabb4b3d33df7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:15.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.879+0000 7ff1a3fff640 1 -- 172.21.15.27:0/2714041188 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4035050 con 0x55b23605e630 2024-04-25T00:19:15.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.879+0000 7ff1a3fff640 1 -- 172.21.15.27:0/2714041188 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1b404d070 con 0x55b23605e630 2024-04-25T00:19:15.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.879+0000 7ff1a3fff640 1 -- 172.21.15.27:0/2714041188 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4032b70 con 0x55b23605e630 2024-04-25T00:19:15.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.880+0000 7ff1c2f72740 1 -- 172.21.15.27:0/2714041188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605e630 msgr2=0x55b23605ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:15.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.880+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/2714041188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605e630 0x55b23605ea10 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4033000 tx=0x7ff1b403c000 comp rx=0 tx=0).stop 2024-04-25T00:19:15.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.880+0000 7ff1c2f72740 1 -- 172.21.15.27:0/2714041188 shutdown_connections 2024-04-25T00:19:15.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.880+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/2714041188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b236069620 0x55b23606ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.880+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/2714041188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b23605ef50 0x55b2360690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.880+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/2714041188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605e630 0x55b23605ea10 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.880+0000 7ff1c2f72740 1 -- 172.21.15.27:0/2714041188 >> 172.21.15.27:0/2714041188 conn(0x55b23605a950 msgr2=0x55b23605ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.880+0000 7ff1c2f72740 1 -- 172.21.15.27:0/2714041188 shutdown_connections 2024-04-25T00:19:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.880+0000 7ff1c2f72740 1 -- 172.21.15.27:0/2714041188 wait complete. 2024-04-25T00:19:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.880+0000 7ff1c2f72740 1 -- start start 2024-04-25T00:19:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.880+0000 7ff1c2f72740 1 Processor -- start 2024-04-25T00:19:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1c2f72740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23605e630 0x55b236066580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1c2f72740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605ef50 0x55b236066ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1c2f72740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b236069620 0x55b236067000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1c2f72740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b23606c3b0 con 0x55b23605ef50 2024-04-25T00:19:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1c2f72740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b235f7ef50 con 0x55b236069620 2024-04-25T00:19:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1c2f72740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b23606c530 con 0x55b23605e630 2024-04-25T00:19:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1c2f72740 10 client.0 _ll_register_callbacks cb 0x55b235fba000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:19:15.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1bd7ff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605ef50 0x55b236066ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1be000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23605e630 0x55b236066580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1be000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23605e630 0x55b236066580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:44398/0 (socket says 172.21.15.27:44398) 2024-04-25T00:19:15.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1bd7ff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605ef50 0x55b236066ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59938/0 (socket says 172.21.15.27:59938) 2024-04-25T00:19:15.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1be000640 1 -- 172.21.15.27:0/481532706 learned_addr learned my addr 172.21.15.27:0/481532706 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:19:15.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1be801640 1 --2- 172.21.15.27:0/481532706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b236069620 0x55b236067000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1be000640 1 -- 172.21.15.27:0/481532706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b236069620 msgr2=0x55b236067000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:15.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1be000640 1 --2- 172.21.15.27:0/481532706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b236069620 0x55b236067000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1be000640 1 -- 172.21.15.27:0/481532706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605ef50 msgr2=0x55b236066ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:15.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1be000640 1 --2- 172.21.15.27:0/481532706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605ef50 0x55b236066ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.881+0000 7ff1be000640 1 -- 172.21.15.27:0/481532706 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b235cd73f0 con 0x55b23605e630 2024-04-25T00:19:15.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.882+0000 7ff1be000640 1 --2- 172.21.15.27:0/481532706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23605e630 0x55b236066580 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff1b402a550 tx=0x7ff1b4032ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f04ba93e2d912d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:15.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.882+0000 7ff1a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:19:15.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.882+0000 7ff1a17fa640 1 -- 172.21.15.27:0/481532706 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4035070 con 0x55b23605e630 2024-04-25T00:19:15.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.882+0000 7ff1a17fa640 1 -- 172.21.15.27:0/481532706 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7ff1b4005580 con 0x55b23605e630 2024-04-25T00:19:15.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.882+0000 7ff1c2f72740 1 -- 172.21.15.27:0/481532706 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1b404d070 con 0x55b23605e630 2024-04-25T00:19:15.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.882+0000 7ff1a17fa640 1 -- 172.21.15.27:0/481532706 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff1b40046c0 con 0x55b23605e630 2024-04-25T00:19:15.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.882+0000 7ff1c2f72740 10 client.25942 fetch_fsmap learned FSMap version 31 2024-04-25T00:19:15.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.882+0000 7ff1c2f72740 1 -- 172.21.15.27:0/481532706 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b235c970f0 con 0x55b23605e630 2024-04-25T00:19:15.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.882+0000 7ff1a17fa640 1 -- 172.21.15.27:0/481532706 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1b404d070 con 0x55b23605e630 2024-04-25T00:19:15.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.882+0000 7ff1a17fa640 1 -- 172.21.15.27:0/481532706 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7ff1b4003a60 con 0x55b23605e630 2024-04-25T00:19:15.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.882+0000 7ff1c2f72740 10 client.25942 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T00:19:15.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.882+0000 7ff1c2f72740 10 client.25942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:19:15.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.883+0000 7ff1c2f72740 20 client.25942 populate_metadata read hostname 'smithi027' 2024-04-25T00:19:15.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.883+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/481532706 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55b236079730 0x55b23607bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.883+0000 7ff1c2f72740 4 client.25942 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:19:15.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.883+0000 7ff1c2f72740 1 -- 172.21.15.27:0/481532706 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b235fd0bf0 con 0x55b236079730 2024-04-25T00:19:15.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.883+0000 7ff1bd7ff640 1 --2- 172.21.15.27:0/481532706 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55b236079730 0x55b23607bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.883+0000 7ff1bd7ff640 1 --2- 172.21.15.27:0/481532706 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55b236079730 0x55b23607bb20 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54946c206640127 server_cookie=510df5f492e68fea in_seq=0 out_seq=0 2024-04-25T00:19:15.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.883+0000 7ff1a17fa640 10 client.25942 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:19:15.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.884+0000 7ff1a17fa640 1 -- 172.21.15.27:0/481532706 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b235fd0bf0 con 0x55b236079730 2024-04-25T00:19:15.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.884+0000 7ff1a17fa640 10 client.25942 handle_command_reply: tid=0 2024-04-25T00:19:15.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.884+0000 7ff1c2f72740 1 client.25942 shutdown 2024-04-25T00:19:15.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.885+0000 7ff1c2f72740 1 -- 172.21.15.27:0/481532706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23605e630 msgr2=0x55b236066580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:15.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.885+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/481532706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23605e630 0x55b236066580 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff1b402a550 tx=0x7ff1b4032ed0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.885+0000 7ff1c2f72740 1 -- 172.21.15.27:0/481532706 shutdown_connections 2024-04-25T00:19:15.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.885+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/481532706 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55b236079730 0x55b23607bb20 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.885+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/481532706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b236069620 0x55b236067000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.885+0000 7ff1a17fa640 0 client.25942 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:19:15.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.885+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/481532706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605ef50 0x55b236066ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.885+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/481532706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23605e630 0x55b236066580 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.885+0000 7ff1c2f72740 1 -- 172.21.15.27:0/481532706 >> 172.21.15.27:0/481532706 conn(0x55b23605a950 msgr2=0x55b236060630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:15.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.885+0000 7ff1c2f72740 1 -- 172.21.15.27:0/481532706 shutdown_connections 2024-04-25T00:19:15.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.885+0000 7ff1c2f72740 1 -- 172.21.15.27:0/481532706 wait complete. 2024-04-25T00:19:15.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.885+0000 7ff1c2f72740 20 client.25942 trim_cache size 0 max 16384 2024-04-25T00:19:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.906+0000 7ff1c2f72740 1 Processor -- start 2024-04-25T00:19:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.906+0000 7ff1c2f72740 1 -- start start 2024-04-25T00:19:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.906+0000 7ff1c2f72740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b23605e630 0x55b236068cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.906+0000 7ff1c2f72740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23605ef50 0x55b2360752a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.906+0000 7ff1c2f72740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b236069620 0x55b2360654e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.906+0000 7ff1c2f72740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b81b5130 con 0x55b236069620 2024-04-25T00:19:15.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.906+0000 7ff1c2f72740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b81b6cd0 con 0x55b23605e630 2024-04-25T00:19:15.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.906+0000 7ff1c2f72740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b81b5800 con 0x55b23605ef50 2024-04-25T00:19:15.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.907+0000 7ff1be801640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b236069620 0x55b2360654e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.907+0000 7ff1be801640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b236069620 0x55b2360654e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59944/0 (socket says 172.21.15.27:59944) 2024-04-25T00:19:15.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.907+0000 7ff1be801640 1 -- 172.21.15.27:0/368409214 learned_addr learned my addr 172.21.15.27:0/368409214 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:19:15.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.907+0000 7ff1bd7ff640 1 --2- 172.21.15.27:0/368409214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23605ef50 0x55b2360752a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.907+0000 7ff1be000640 1 --2- 172.21.15.27:0/368409214 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b23605e630 0x55b236068cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.907+0000 7ff1be801640 1 -- 172.21.15.27:0/368409214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23605ef50 msgr2=0x55b2360752a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:15.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.907+0000 7ff1be801640 1 --2- 172.21.15.27:0/368409214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23605ef50 0x55b2360752a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.907+0000 7ff1be801640 1 -- 172.21.15.27:0/368409214 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b23605e630 msgr2=0x55b236068cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:15.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.907+0000 7ff1be801640 1 --2- 172.21.15.27:0/368409214 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b23605e630 0x55b236068cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.907+0000 7ff1be801640 1 -- 172.21.15.27:0/368409214 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b23604c090 con 0x55b236069620 2024-04-25T00:19:15.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.907+0000 7ff1be801640 1 --2- 172.21.15.27:0/368409214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b236069620 0x55b2360654e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff1b00135c0 tx=0x7ff1b0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d940525cb7dbf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:15.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.907+0000 7ff1a37fe640 1 -- 172.21.15.27:0/368409214 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b0021070 con 0x55b236069620 2024-04-25T00:19:15.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.907+0000 7ff1a37fe640 1 -- 172.21.15.27:0/368409214 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1b0011a40 con 0x55b236069620 2024-04-25T00:19:15.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.907+0000 7ff1a37fe640 1 -- 172.21.15.27:0/368409214 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b001cc70 con 0x55b236069620 2024-04-25T00:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.908+0000 7ff1c2f72740 1 -- 172.21.15.27:0/368409214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b236069620 msgr2=0x55b2360654e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.908+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/368409214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b236069620 0x55b2360654e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff1b00135c0 tx=0x7ff1b0009f90 comp rx=0 tx=0).stop 2024-04-25T00:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.908+0000 7ff1c2f72740 1 -- 172.21.15.27:0/368409214 shutdown_connections 2024-04-25T00:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.908+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/368409214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b236069620 0x55b2360654e0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.908+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/368409214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23605ef50 0x55b2360752a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.908+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/368409214 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b23605e630 0x55b236068cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.908+0000 7ff1c2f72740 1 -- 172.21.15.27:0/368409214 >> 172.21.15.27:0/368409214 conn(0x55b23605a950 msgr2=0x55b236059af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.908+0000 7ff1c2f72740 1 -- 172.21.15.27:0/368409214 shutdown_connections 2024-04-25T00:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.908+0000 7ff1c2f72740 1 -- 172.21.15.27:0/368409214 wait complete. 2024-04-25T00:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.908+0000 7ff1c2f72740 1 -- start start 2024-04-25T00:19:15.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.908+0000 7ff1c2f72740 1 Processor -- start 2024-04-25T00:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.909+0000 7ff1c2f72740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605e630 0x55b2360632a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.909+0000 7ff1c2f72740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23605ef50 0x55b2360637e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.909+0000 7ff1be000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605e630 0x55b2360632a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.909+0000 7ff1c2f72740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b236069620 0x55b236063d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.909+0000 7ff1c2f72740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b81b65a0 con 0x55b23605e630 2024-04-25T00:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.909+0000 7ff1be000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605e630 0x55b2360632a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59960/0 (socket says 172.21.15.27:59960) 2024-04-25T00:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.909+0000 7ff1c2f72740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b81b5ed0 con 0x55b236069620 2024-04-25T00:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.909+0000 7ff1be000640 1 -- 172.21.15.27:0/2862334370 learned_addr learned my addr 172.21.15.27:0/2862334370 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.909+0000 7ff1c2f72740 1 -- 172.21.15.27:0/2862334370 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b81b30f0 con 0x55b23605ef50 2024-04-25T00:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.909+0000 7ff1c2f72740 10 client.0 _ll_register_callbacks cb 0x7ff1b0009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.909+0000 7ff1bd7ff640 1 --2- 172.21.15.27:0/2862334370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23605ef50 0x55b2360637e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.909+0000 7ff1be801640 1 --2- 172.21.15.27:0/2862334370 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b236069620 0x55b236063d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.910+0000 7ff1be000640 1 -- 172.21.15.27:0/2862334370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23605ef50 msgr2=0x55b2360637e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:15.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.910+0000 7ff1be000640 1 --2- 172.21.15.27:0/2862334370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23605ef50 0x55b2360637e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.910+0000 7ff1be000640 1 -- 172.21.15.27:0/2862334370 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b236069620 msgr2=0x55b236063d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.910+0000 7ff1be000640 1 --2- 172.21.15.27:0/2862334370 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b236069620 0x55b236063d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.910+0000 7ff1be000640 1 -- 172.21.15.27:0/2862334370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2360deae0 con 0x55b23605e630 2024-04-25T00:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.910+0000 7ff1be000640 1 --2- 172.21.15.27:0/2862334370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605e630 0x55b2360632a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4032920 tx=0x7ff1b4009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cec6e9358edd075 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.910+0000 7ff1a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.910+0000 7ff1a17fa640 1 -- 172.21.15.27:0/2862334370 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4035040 con 0x55b23605e630 2024-04-25T00:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.910+0000 7ff1a17fa640 1 -- 172.21.15.27:0/2862334370 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7ff1b40484d0 con 0x55b23605e630 2024-04-25T00:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.910+0000 7ff1c2f72740 1 -- 172.21.15.27:0/2862334370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1b0011a40 con 0x55b23605e630 2024-04-25T00:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.910+0000 7ff1a17fa640 1 -- 172.21.15.27:0/2862334370 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff1b403f040 con 0x55b23605e630 2024-04-25T00:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.910+0000 7ff1c2f72740 10 client.16446 fetch_fsmap learned FSMap version 31 2024-04-25T00:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.910+0000 7ff1c2f72740 1 -- 172.21.15.27:0/2862334370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b2360dfdf0 con 0x55b23605e630 2024-04-25T00:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.911+0000 7ff1a17fa640 1 -- 172.21.15.27:0/2862334370 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1b0011a40 con 0x55b23605e630 2024-04-25T00:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.911+0000 7ff1a17fa640 1 -- 172.21.15.27:0/2862334370 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7ff1b4003950 con 0x55b23605e630 2024-04-25T00:19:15.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.911+0000 7ff1c2f72740 10 client.16446 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T00:19:15.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.911+0000 7ff1c2f72740 10 client.16446 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:19:15.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.911+0000 7ff1c2f72740 20 client.16446 populate_metadata read hostname 'smithi027' 2024-04-25T00:19:15.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.911+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/2862334370 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55b2360e03a0 0x55b2360e27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:15.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.911+0000 7ff1c2f72740 4 client.16446 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:19:15.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.911+0000 7ff1c2f72740 1 -- 172.21.15.27:0/2862334370 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b235fcaf40 con 0x55b2360e03a0 2024-04-25T00:19:15.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.911+0000 7ff1bd7ff640 1 --2- 172.21.15.27:0/2862334370 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55b2360e03a0 0x55b2360e27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:15.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.912+0000 7ff1a17fa640 10 client.16446 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:19:15.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.912+0000 7ff1bd7ff640 1 --2- 172.21.15.27:0/2862334370 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55b2360e03a0 0x55b2360e27c0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d1965f44885b0ce server_cookie=168b63d4d8613359 in_seq=0 out_seq=0 2024-04-25T00:19:15.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.912+0000 7ff1a17fa640 1 -- 172.21.15.27:0/2862334370 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55b235fcaf40 con 0x55b2360e03a0 2024-04-25T00:19:15.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.912+0000 7ff1a17fa640 10 client.16446 handle_command_reply: tid=0 2024-04-25T00:19:15.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.912+0000 7ff1c2f72740 1 client.16446 shutdown 2024-04-25T00:19:15.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.912+0000 7ff1c2f72740 1 -- 172.21.15.27:0/2862334370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605e630 msgr2=0x55b2360632a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:15.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.912+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/2862334370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605e630 0x55b2360632a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4032920 tx=0x7ff1b4009230 comp rx=0 tx=0).stop 2024-04-25T00:19:15.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.912+0000 7ff1c2f72740 1 -- 172.21.15.27:0/2862334370 shutdown_connections 2024-04-25T00:19:15.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.912+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/2862334370 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55b2360e03a0 0x55b2360e27c0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.912+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/2862334370 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b236069620 0x55b236063d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.912+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/2862334370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b23605ef50 0x55b2360637e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.912+0000 7ff1c2f72740 1 --2- 172.21.15.27:0/2862334370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b23605e630 0x55b2360632a0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.912+0000 7ff1c2f72740 1 -- 172.21.15.27:0/2862334370 >> 172.21.15.27:0/2862334370 conn(0x55b23605a950 msgr2=0x55b236066370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:15.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.912+0000 7ff1a17fa640 0 client.16446 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:19:15.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.912+0000 7ff1c2f72740 1 -- 172.21.15.27:0/2862334370 shutdown_connections 2024-04-25T00:19:15.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.912+0000 7ff1c2f72740 1 -- 172.21.15.27:0/2862334370 wait complete. 2024-04-25T00:19:15.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.913+0000 7ff1c2f72740 20 client.16446 trim_cache size 0 max 16384 2024-04-25T00:19:15.918 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:19:15.919 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "scrub active (201 inodes in the stack)", 2024-04-25T00:19:15.919 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:19:15.919 INFO:teuthology.orchestra.run.smithi027.stdout: "a40f0131-2b37-4eb9-a2bd-6fee5124aebc": { 2024-04-25T00:19:15.919 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:19:15.919 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "a40f0131-2b37-4eb9-a2bd-6fee5124aebc", 2024-04-25T00:19:15.919 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:19:15.919 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:19:15.919 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:19:15.919 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:19:15.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.913+0000 7ff1c0264640 1 -- 172.21.15.27:0/2119162303 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff1a40a2d00 msgr2=0x7ff1a40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:15.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.913+0000 7ff1c0264640 1 --2- 172.21.15.27:0/2119162303 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff1a40a2d00 0x7ff1a40a51c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4000c00 tx=0x7ff1b4005a40 comp rx=0 tx=0).stop 2024-04-25T00:19:15.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.913+0000 7ff1c0264640 1 -- 172.21.15.27:0/2119162303 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1b81032e0 msgr2=0x7ff1b8100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:15.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.913+0000 7ff1c0264640 1 --2- 172.21.15.27:0/2119162303 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1b81032e0 0x7ff1b8100610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff1a800d950 tx=0x7ff1a800de10 comp rx=0 tx=0).stop 2024-04-25T00:19:15.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.913+0000 7ff1c0264640 1 -- 172.21.15.27:0/2119162303 shutdown_connections 2024-04-25T00:19:15.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.913+0000 7ff1c0264640 1 --2- 172.21.15.27:0/2119162303 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff1a40a2d00 0x7ff1a40a51c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.913+0000 7ff1c0264640 1 --2- 172.21.15.27:0/2119162303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b810bd00 0x7ff1b8100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.913+0000 7ff1c0264640 1 --2- 172.21.15.27:0/2119162303 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1b81032e0 0x7ff1b8100610 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.913+0000 7ff1c0264640 1 --2- 172.21.15.27:0/2119162303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b81029c0 0x7ff1b8102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:15.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.913+0000 7ff1c0264640 1 -- 172.21.15.27:0/2119162303 >> 172.21.15.27:0/2119162303 conn(0x7ff1b8077740 msgr2=0x7ff1b80fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:15.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.913+0000 7ff1c0264640 1 -- 172.21.15.27:0/2119162303 shutdown_connections 2024-04-25T00:19:15.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:15.913+0000 7ff1c0264640 1 -- 172.21.15.27:0/2119162303 wait complete. 2024-04-25T00:19:15.949 INFO:tasks.cephfs.filesystem:scrub status for tag:a40f0131-2b37-4eb9-a2bd-6fee5124aebc - {'path': '/', 'tag': 'a40f0131-2b37-4eb9-a2bd-6fee5124aebc', 'options': 'recursive,force'} 2024-04-25T00:19:16.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:16 smithi027 ceph-mon[29208]: pgmap v108: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 49 MiB/s wr, 7.91k op/s 2024-04-25T00:19:16.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:16 smithi139 ceph-mon[32833]: pgmap v108: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 49 MiB/s wr, 7.91k op/s 2024-04-25T00:19:16.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:16 smithi153 ceph-mon[32896]: pgmap v108: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 49 MiB/s wr, 7.91k op/s 2024-04-25T00:19:18.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:18 smithi027 ceph-mon[29208]: pgmap v109: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 61 MiB/s wr, 10.04k op/s 2024-04-25T00:19:18.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:18 smithi139 ceph-mon[32833]: pgmap v109: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 61 MiB/s wr, 10.04k op/s 2024-04-25T00:19:18.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:18 smithi153 ceph-mon[32896]: pgmap v109: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 61 MiB/s wr, 10.04k op/s 2024-04-25T00:19:20.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:20 smithi027 ceph-mon[29208]: pgmap v110: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 52 MiB/s wr, 8.59k op/s 2024-04-25T00:19:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:20 smithi139 ceph-mon[32833]: pgmap v110: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 52 MiB/s wr, 8.59k op/s 2024-04-25T00:19:20.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:20 smithi153 ceph-mon[32896]: pgmap v110: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 52 MiB/s wr, 8.59k op/s 2024-04-25T00:19:22.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:22 smithi027 ceph-mon[29208]: pgmap v111: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 51 MiB/s wr, 8.39k op/s 2024-04-25T00:19:22.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:22 smithi139 ceph-mon[32833]: pgmap v111: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 51 MiB/s wr, 8.39k op/s 2024-04-25T00:19:22.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:22 smithi153 ceph-mon[32896]: pgmap v111: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 51 MiB/s wr, 8.39k op/s 2024-04-25T00:19:24.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:24 smithi027 ceph-mon[29208]: pgmap v112: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 62 MiB/s wr, 10.36k op/s 2024-04-25T00:19:24.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:24 smithi139 ceph-mon[32833]: pgmap v112: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 62 MiB/s wr, 10.36k op/s 2024-04-25T00:19:24.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:24 smithi153 ceph-mon[32896]: pgmap v112: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 62 MiB/s wr, 10.36k op/s 2024-04-25T00:19:26.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:26 smithi027 ceph-mon[29208]: pgmap v113: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 39 MiB/s wr, 6.58k op/s 2024-04-25T00:19:26.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:26 smithi139 ceph-mon[32833]: pgmap v113: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 39 MiB/s wr, 6.58k op/s 2024-04-25T00:19:26.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:26 smithi153 ceph-mon[32896]: pgmap v113: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 39 MiB/s wr, 6.58k op/s 2024-04-25T00:19:28.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:28 smithi027 ceph-mon[29208]: pgmap v114: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 50 MiB/s wr, 8.73k op/s 2024-04-25T00:19:28.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:28 smithi139 ceph-mon[32833]: pgmap v114: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 50 MiB/s wr, 8.73k op/s 2024-04-25T00:19:28.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:28 smithi153 ceph-mon[32896]: pgmap v114: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 50 MiB/s wr, 8.73k op/s 2024-04-25T00:19:30.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:30 smithi027 ceph-mon[29208]: pgmap v115: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 42 MiB/s wr, 7.62k op/s 2024-04-25T00:19:30.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:30 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:19:30.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:30 smithi139 ceph-mon[32833]: pgmap v115: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 42 MiB/s wr, 7.62k op/s 2024-04-25T00:19:30.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:30 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:19:30.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:30 smithi153 ceph-mon[32896]: pgmap v115: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 42 MiB/s wr, 7.62k op/s 2024-04-25T00:19:30.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:30 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:19:32.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:32 smithi027 ceph-mon[29208]: pgmap v116: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 38 MiB/s wr, 6.96k op/s 2024-04-25T00:19:32.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:32 smithi139 ceph-mon[32833]: pgmap v116: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 38 MiB/s wr, 6.96k op/s 2024-04-25T00:19:32.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:32 smithi153 ceph-mon[32896]: pgmap v116: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 38 MiB/s wr, 6.96k op/s 2024-04-25T00:19:34.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:34 smithi027 ceph-mon[29208]: pgmap v117: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 47 MiB/s wr, 8.82k op/s 2024-04-25T00:19:34.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:34 smithi139 ceph-mon[32833]: pgmap v117: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 47 MiB/s wr, 8.82k op/s 2024-04-25T00:19:34.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:34 smithi153 ceph-mon[32896]: pgmap v117: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 47 MiB/s wr, 8.82k op/s 2024-04-25T00:19:35.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:35 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:19:35.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:35 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:19:35.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:35 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:19:36.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:36 smithi027 ceph-mon[29208]: pgmap v118: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 32 MiB/s wr, 6.29k op/s 2024-04-25T00:19:36.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:36 smithi139 ceph-mon[32833]: pgmap v118: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 32 MiB/s wr, 6.29k op/s 2024-04-25T00:19:36.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:36 smithi153 ceph-mon[32896]: pgmap v118: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 32 MiB/s wr, 6.29k op/s 2024-04-25T00:19:38.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:38 smithi027 ceph-mon[29208]: pgmap v119: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 42 MiB/s wr, 8.52k op/s 2024-04-25T00:19:38.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:38 smithi139 ceph-mon[32833]: pgmap v119: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 42 MiB/s wr, 8.52k op/s 2024-04-25T00:19:38.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:38 smithi153 ceph-mon[32896]: pgmap v119: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 42 MiB/s wr, 8.52k op/s 2024-04-25T00:19:40.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:40 smithi027 ceph-mon[29208]: pgmap v120: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 33 MiB/s wr, 6.86k op/s 2024-04-25T00:19:40.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:40 smithi139 ceph-mon[32833]: pgmap v120: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 33 MiB/s wr, 6.86k op/s 2024-04-25T00:19:40.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:40 smithi153 ceph-mon[32896]: pgmap v120: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 33 MiB/s wr, 6.86k op/s 2024-04-25T00:19:42.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:42 smithi027 ceph-mon[29208]: pgmap v121: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 27 MiB/s wr, 5.67k op/s 2024-04-25T00:19:42.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:42 smithi139 ceph-mon[32833]: pgmap v121: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 27 MiB/s wr, 5.67k op/s 2024-04-25T00:19:42.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:42 smithi153 ceph-mon[32896]: pgmap v121: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 27 MiB/s wr, 5.67k op/s 2024-04-25T00:19:44.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:44 smithi027 ceph-mon[29208]: pgmap v122: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 50 MiB/s wr, 9.54k op/s 2024-04-25T00:19:44.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:44 smithi139 ceph-mon[32833]: pgmap v122: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 50 MiB/s wr, 9.54k op/s 2024-04-25T00:19:44.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:44 smithi153 ceph-mon[32896]: pgmap v122: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 50 MiB/s wr, 9.54k op/s 2024-04-25T00:19:45.950 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:19:45.951 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:19:46.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.145+0000 7f9909e38640 1 -- 172.21.15.27:0/2248311892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f990406a4d0 msgr2=0x7f9904105260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:46.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.145+0000 7f9909e38640 1 --2- 172.21.15.27:0/2248311892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f990406a4d0 0x7f9904105260 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f98fc009ca0 tx=0x7f98fc02a8f0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.147+0000 7f9909e38640 1 -- 172.21.15.27:0/2248311892 shutdown_connections 2024-04-25T00:19:46.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.147+0000 7f9909e38640 1 --2- 172.21.15.27:0/2248311892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f990406a4d0 0x7f9904105260 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.147+0000 7f9909e38640 1 --2- 172.21.15.27:0/2248311892 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9904069a50 0x7f9904069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.147+0000 7f9909e38640 1 --2- 172.21.15.27:0/2248311892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9904106670 0x7f9904106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.147+0000 7f9909e38640 1 -- 172.21.15.27:0/2248311892 >> 172.21.15.27:0/2248311892 conn(0x7f99040761c0 msgr2=0x7f99040765d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:46.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.147+0000 7f9909e38640 1 -- 172.21.15.27:0/2248311892 shutdown_connections 2024-04-25T00:19:46.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.147+0000 7f9909e38640 1 -- 172.21.15.27:0/2248311892 wait complete. 2024-04-25T00:19:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.148+0000 7f9909e38640 1 Processor -- start 2024-04-25T00:19:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.148+0000 7f9909e38640 1 -- start start 2024-04-25T00:19:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.148+0000 7f9909e38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9904069a50 0x7f9904114c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.148+0000 7f9909e38640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f990406a4d0 0x7f99041151d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.148+0000 7f9909e38640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9904106670 0x7f990410ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.148+0000 7f9909e38640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9904107750 con 0x7f9904106670 2024-04-25T00:19:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.148+0000 7f9909e38640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f99041075d0 con 0x7f990406a4d0 2024-04-25T00:19:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.148+0000 7f9909e38640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f99041078d0 con 0x7f9904069a50 2024-04-25T00:19:46.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.149+0000 7f9903fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9904106670 0x7f990410ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:46.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.149+0000 7f9903fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9904106670 0x7f990410ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54056/0 (socket says 172.21.15.27:54056) 2024-04-25T00:19:46.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.149+0000 7f9903fff640 1 -- 172.21.15.27:0/2181742070 learned_addr learned my addr 172.21.15.27:0/2181742070 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:19:46.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.150+0000 7f99037fe640 1 --2- 172.21.15.27:0/2181742070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9904069a50 0x7f9904114c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:46.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.151+0000 7f99037fe640 1 -- 172.21.15.27:0/2181742070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f990406a4d0 msgr2=0x7f99041151d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:19:46.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.151+0000 7f99037fe640 1 --2- 172.21.15.27:0/2181742070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f990406a4d0 0x7f99041151d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.151+0000 7f99037fe640 1 -- 172.21.15.27:0/2181742070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9904106670 msgr2=0x7f990410ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:46.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.151+0000 7f99037fe640 1 --2- 172.21.15.27:0/2181742070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9904106670 0x7f990410ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.151+0000 7f99037fe640 1 -- 172.21.15.27:0/2181742070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990410f610 con 0x7f9904069a50 2024-04-25T00:19:46.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.152+0000 7f99037fe640 1 --2- 172.21.15.27:0/2181742070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9904069a50 0x7f9904114c90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f98f400b3f0 tx=0x7f98f400b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:46.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.153+0000 7f9900ff9640 1 -- 172.21.15.27:0/2181742070 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4015020 con 0x7f9904069a50 2024-04-25T00:19:46.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.153+0000 7f9900ff9640 1 -- 172.21.15.27:0/2181742070 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98f4011d10 con 0x7f9904069a50 2024-04-25T00:19:46.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.153+0000 7f9900ff9640 1 -- 172.21.15.27:0/2181742070 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f400f2e0 con 0x7f9904069a50 2024-04-25T00:19:46.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.153+0000 7f9909e38640 1 -- 172.21.15.27:0/2181742070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f990410f900 con 0x7f9904069a50 2024-04-25T00:19:46.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.154+0000 7f9909e38640 1 -- 172.21.15.27:0/2181742070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f990411de10 con 0x7f9904069a50 2024-04-25T00:19:46.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.159+0000 7f990cc26740 1 Processor -- start 2024-04-25T00:19:46.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.159+0000 7f990cc26740 1 -- start start 2024-04-25T00:19:46.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.160+0000 7f990cc26740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b392510 0x56222b3928f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.160+0000 7f990cc26740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392e30 0x56222b3ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.160+0000 7f990cc26740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b3ae630 0x56222b3b0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.160+0000 7f990cc26740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56222b2f76b0 con 0x56222b3ae630 2024-04-25T00:19:46.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.160+0000 7f990cc26740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56222b29f880 con 0x56222b392510 2024-04-25T00:19:46.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.160+0000 7f990cc26740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56222b2f3670 con 0x56222b392e30 2024-04-25T00:19:46.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.160+0000 7f9903fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b392510 0x56222b3928f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:46.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.160+0000 7f9903fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b392510 0x56222b3928f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:47602/0 (socket says 172.21.15.27:47602) 2024-04-25T00:19:46.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.160+0000 7f9903fff640 1 -- 172.21.15.27:0/1966661209 learned_addr learned my addr 172.21.15.27:0/1966661209 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:19:46.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.160+0000 7f9902ffd640 1 --2- 172.21.15.27:0/1966661209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b3ae630 0x56222b3b0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:46.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.161+0000 7f9903fff640 1 -- 172.21.15.27:0/1966661209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392e30 msgr2=0x56222b3ae0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:19:46.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.161+0000 7f9903fff640 1 --2- 172.21.15.27:0/1966661209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392e30 0x56222b3ae0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.161+0000 7f9903fff640 1 -- 172.21.15.27:0/1966661209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b3ae630 msgr2=0x56222b3b0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:46.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.161+0000 7f9903fff640 1 --2- 172.21.15.27:0/1966661209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b3ae630 0x56222b3b0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.161+0000 7f9903fff640 1 -- 172.21.15.27:0/1966661209 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56222b0d57d0 con 0x56222b392510 2024-04-25T00:19:46.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.161+0000 7f9900ff9640 1 -- 172.21.15.27:0/2181742070 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f98f4018020 con 0x7f9904069a50 2024-04-25T00:19:46.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.162+0000 7f9900ff9640 1 --2- 172.21.15.27:0/2181742070 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f98ec0a2f00 0x7f98ec0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.162+0000 7f99037fe640 1 --2- 172.21.15.27:0/2181742070 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f98ec0a2f00 0x7f98ec0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:46.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.163+0000 7f9903fff640 1 --2- 172.21.15.27:0/1966661209 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b392510 0x56222b3928f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f98fc031d10 tx=0x7f98fc003970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=103f2f450775939 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:46.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.163+0000 7f9900ff9640 1 -- 172.21.15.27:0/2181742070 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f98f40cee20 con 0x7f9904069a50 2024-04-25T00:19:46.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.164+0000 7f99037fe640 1 --2- 172.21.15.27:0/2181742070 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f98ec0a2f00 0x7f98ec0a53c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f98f409ada0 tx=0x7f98f4097340 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:46.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.164+0000 7f98e9ffb640 1 -- 172.21.15.27:0/1966661209 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc003d90 con 0x56222b392510 2024-04-25T00:19:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.166+0000 7f98e9ffb640 1 -- 172.21.15.27:0/1966661209 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98fc036030 con 0x56222b392510 2024-04-25T00:19:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.166+0000 7f98e9ffb640 1 -- 172.21.15.27:0/1966661209 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc0305f0 con 0x56222b392510 2024-04-25T00:19:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.167+0000 7f990cc26740 1 -- 172.21.15.27:0/1966661209 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b392510 msgr2=0x56222b3928f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.167+0000 7f990cc26740 1 --2- 172.21.15.27:0/1966661209 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b392510 0x56222b3928f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f98fc031d10 tx=0x7f98fc003970 comp rx=0 tx=0).stop 2024-04-25T00:19:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.168+0000 7f990cc26740 1 -- 172.21.15.27:0/1966661209 shutdown_connections 2024-04-25T00:19:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.168+0000 7f990cc26740 1 --2- 172.21.15.27:0/1966661209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b3ae630 0x56222b3b0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.168+0000 7f990cc26740 1 --2- 172.21.15.27:0/1966661209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392e30 0x56222b3ae0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.168+0000 7f990cc26740 1 --2- 172.21.15.27:0/1966661209 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b392510 0x56222b3928f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.168+0000 7f990cc26740 1 -- 172.21.15.27:0/1966661209 >> 172.21.15.27:0/1966661209 conn(0x56222b38e950 msgr2=0x56222b38ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.171+0000 7f990cc26740 1 -- 172.21.15.27:0/1966661209 shutdown_connections 2024-04-25T00:19:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.172+0000 7f990cc26740 1 -- 172.21.15.27:0/1966661209 wait complete. 2024-04-25T00:19:46.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.172+0000 7f990cc26740 1 -- start start 2024-04-25T00:19:46.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.172+0000 7f990cc26740 1 Processor -- start 2024-04-25T00:19:46.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.174+0000 7f990cc26740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b392510 0x56222b39f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.174+0000 7f990cc26740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392e30 0x56222b39fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.174+0000 7f990cc26740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b3ae630 0x56222b39aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.174+0000 7f990cc26740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56222b395260 con 0x56222b392510 2024-04-25T00:19:46.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.174+0000 7f990cc26740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56222b2b2f50 con 0x56222b3ae630 2024-04-25T00:19:46.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.174+0000 7f990cc26740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56222b3953e0 con 0x56222b392e30 2024-04-25T00:19:46.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.174+0000 7f990cc26740 10 client.0 _ll_register_callbacks cb 0x56222b2ee000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:19:46.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.175+0000 7f9903fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b392510 0x56222b39f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:46.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.175+0000 7f9903fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b392510 0x56222b39f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54080/0 (socket says 172.21.15.27:54080) 2024-04-25T00:19:46.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.175+0000 7f9903fff640 1 -- 172.21.15.27:0/1925098995 learned_addr learned my addr 172.21.15.27:0/1925098995 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:19:46.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.175+0000 7f9903fff640 1 -- 172.21.15.27:0/1925098995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392e30 msgr2=0x56222b39fc20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:19:46.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.175+0000 7f9903fff640 1 --2- 172.21.15.27:0/1925098995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392e30 0x56222b39fc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.175+0000 7f9903fff640 1 -- 172.21.15.27:0/1925098995 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b3ae630 msgr2=0x56222b39aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:46.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.175+0000 7f9903fff640 1 --2- 172.21.15.27:0/1925098995 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b3ae630 0x56222b39aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.175+0000 7f9903fff640 1 -- 172.21.15.27:0/1925098995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56222b00b3f0 con 0x56222b392510 2024-04-25T00:19:46.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.176+0000 7f9903fff640 1 --2- 172.21.15.27:0/1925098995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b392510 0x56222b39f6e0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f98fc009620 tx=0x7f98fc004980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ddc9f7787609f53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:46.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.176+0000 7f98e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:19:46.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.176+0000 7f98e2ffd640 1 -- 172.21.15.27:0/1925098995 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc003da0 con 0x56222b392510 2024-04-25T00:19:46.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.177+0000 7f98e2ffd640 1 -- 172.21.15.27:0/1925098995 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f98fc003f40 con 0x56222b392510 2024-04-25T00:19:46.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.178+0000 7f990cc26740 1 -- 172.21.15.27:0/1925098995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98fc036030 con 0x56222b392510 2024-04-25T00:19:46.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.180+0000 7f98e2ffd640 1 -- 172.21.15.27:0/1925098995 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f98fc004140 con 0x56222b392510 2024-04-25T00:19:46.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.182+0000 7f990cc26740 10 client.16455 fetch_fsmap learned FSMap version 33 2024-04-25T00:19:46.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.182+0000 7f990cc26740 1 -- 172.21.15.27:0/1925098995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56222afcb0f0 con 0x56222b392510 2024-04-25T00:19:46.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.183+0000 7f98e2ffd640 1 -- 172.21.15.27:0/1925098995 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98fc036030 con 0x56222b392510 2024-04-25T00:19:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.183+0000 7f98e2ffd640 1 -- 172.21.15.27:0/1925098995 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f98fc065070 con 0x56222b392510 2024-04-25T00:19:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.183+0000 7f990cc26740 10 client.16455 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T00:19:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.183+0000 7f990cc26740 10 client.16455 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:19:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.183+0000 7f990cc26740 20 client.16455 populate_metadata read hostname 'smithi027' 2024-04-25T00:19:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.183+0000 7f990cc26740 1 --2- 172.21.15.27:0/1925098995 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56222b3aab30 0x56222b3acf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.183+0000 7f990cc26740 4 client.16455 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:19:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.183+0000 7f990cc26740 1 -- 172.21.15.27:0/1925098995 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56222b304bf0 con 0x56222b3aab30 2024-04-25T00:19:46.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.187+0000 7f9902ffd640 1 --2- 172.21.15.27:0/1925098995 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56222b3aab30 0x56222b3acf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:46.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.188+0000 7f9902ffd640 1 --2- 172.21.15.27:0/1925098995 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56222b3aab30 0x56222b3acf40 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca2dbe8359b9bc63 server_cookie=81e2768fee2404c0 in_seq=0 out_seq=0 2024-04-25T00:19:46.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.194+0000 7f98e2ffd640 10 client.16455 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:19:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.194+0000 7f98e2ffd640 1 -- 172.21.15.27:0/1925098995 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56222b304bf0 con 0x56222b3aab30 2024-04-25T00:19:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.194+0000 7f98e2ffd640 10 client.16455 handle_command_reply: tid=0 2024-04-25T00:19:46.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.196+0000 7f990cc26740 1 client.16455 shutdown 2024-04-25T00:19:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.196+0000 7f990cc26740 1 -- 172.21.15.27:0/1925098995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b392510 msgr2=0x56222b39f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.196+0000 7f990cc26740 1 --2- 172.21.15.27:0/1925098995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b392510 0x56222b39f6e0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f98fc009620 tx=0x7f98fc004980 comp rx=0 tx=0).stop 2024-04-25T00:19:46.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.198+0000 7f990cc26740 1 -- 172.21.15.27:0/1925098995 shutdown_connections 2024-04-25T00:19:46.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.198+0000 7f990cc26740 1 --2- 172.21.15.27:0/1925098995 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56222b3aab30 0x56222b3acf40 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.198+0000 7f990cc26740 1 --2- 172.21.15.27:0/1925098995 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b3ae630 0x56222b39aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.198+0000 7f990cc26740 1 --2- 172.21.15.27:0/1925098995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392e30 0x56222b39fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.198+0000 7f990cc26740 1 --2- 172.21.15.27:0/1925098995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b392510 0x56222b39f6e0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.198+0000 7f990cc26740 1 -- 172.21.15.27:0/1925098995 >> 172.21.15.27:0/1925098995 conn(0x56222b38e950 msgr2=0x56222b394690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:46.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.198+0000 7f98e2ffd640 0 client.16455 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:19:46.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.201+0000 7f990cc26740 1 -- 172.21.15.27:0/1925098995 shutdown_connections 2024-04-25T00:19:46.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.203+0000 7f990cc26740 1 -- 172.21.15.27:0/1925098995 wait complete. 2024-04-25T00:19:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.207+0000 7f990cc26740 20 client.16455 trim_cache size 0 max 16384 2024-04-25T00:19:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.248+0000 7f990cc26740 1 Processor -- start 2024-04-25T00:19:46.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.249+0000 7f990cc26740 1 -- start start 2024-04-25T00:19:46.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.249+0000 7f990cc26740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392510 0x56222b39a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.249+0000 7f990cc26740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b392e30 0x56222b394510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.249+0000 7f990cc26740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b3ae630 0x56222b394a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.249+0000 7f990cc26740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f990411f550 con 0x56222b3ae630 2024-04-25T00:19:46.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.249+0000 7f990cc26740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9904121090 con 0x56222b392e30 2024-04-25T00:19:46.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.249+0000 7f990cc26740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f990411fc20 con 0x56222b392510 2024-04-25T00:19:46.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.252+0000 7f9903fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392510 0x56222b39a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:46.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.252+0000 7f9903fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392510 0x56222b39a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:34834/0 (socket says 172.21.15.27:34834) 2024-04-25T00:19:46.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.252+0000 7f9903fff640 1 -- 172.21.15.27:0/2233876545 learned_addr learned my addr 172.21.15.27:0/2233876545 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:19:46.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.252+0000 7f9903fff640 1 --2- 172.21.15.27:0/2233876545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b3ae630 0x56222b394a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:46.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.254+0000 7f9902ffd640 1 --2- 172.21.15.27:0/2233876545 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b392e30 0x56222b394510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:46.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.254+0000 7f9903fff640 1 -- 172.21.15.27:0/2233876545 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b392e30 msgr2=0x56222b394510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:46.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.254+0000 7f9903fff640 1 --2- 172.21.15.27:0/2233876545 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b392e30 0x56222b394510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.254+0000 7f9903fff640 1 -- 172.21.15.27:0/2233876545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b3ae630 msgr2=0x56222b394a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:46.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.254+0000 7f9903fff640 1 --2- 172.21.15.27:0/2233876545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b3ae630 0x56222b394a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.254+0000 7f9903fff640 1 -- 172.21.15.27:0/2233876545 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56222b38e780 con 0x56222b392510 2024-04-25T00:19:46.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.254+0000 7f9903fff640 1 --2- 172.21.15.27:0/2233876545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392510 0x56222b39a5a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f98fc031b40 tx=0x7f98fc009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d535bb63a06f8073 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:46.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.255+0000 7f98e97fa640 1 -- 172.21.15.27:0/2233876545 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc0042e0 con 0x56222b392510 2024-04-25T00:19:46.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.255+0000 7f98e97fa640 1 -- 172.21.15.27:0/2233876545 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98fc036040 con 0x56222b392510 2024-04-25T00:19:46.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.255+0000 7f98e97fa640 1 -- 172.21.15.27:0/2233876545 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc030ad0 con 0x56222b392510 2024-04-25T00:19:46.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.257+0000 7f990cc26740 1 -- 172.21.15.27:0/2233876545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392510 msgr2=0x56222b39a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:46.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.257+0000 7f990cc26740 1 --2- 172.21.15.27:0/2233876545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392510 0x56222b39a5a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f98fc031b40 tx=0x7f98fc009800 comp rx=0 tx=0).stop 2024-04-25T00:19:46.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.261+0000 7f990cc26740 1 -- 172.21.15.27:0/2233876545 shutdown_connections 2024-04-25T00:19:46.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.261+0000 7f990cc26740 1 --2- 172.21.15.27:0/2233876545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b3ae630 0x56222b394a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.261+0000 7f990cc26740 1 --2- 172.21.15.27:0/2233876545 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b392e30 0x56222b394510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.261+0000 7f990cc26740 1 --2- 172.21.15.27:0/2233876545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392510 0x56222b39a5a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.261+0000 7f990cc26740 1 -- 172.21.15.27:0/2233876545 >> 172.21.15.27:0/2233876545 conn(0x56222b38e950 msgr2=0x56222b393e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.266+0000 7f990cc26740 1 -- 172.21.15.27:0/2233876545 shutdown_connections 2024-04-25T00:19:46.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.267+0000 7f990cc26740 1 -- 172.21.15.27:0/2233876545 wait complete. 2024-04-25T00:19:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.268+0000 7f990cc26740 1 -- start start 2024-04-25T00:19:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.268+0000 7f990cc26740 1 Processor -- start 2024-04-25T00:19:46.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.272+0000 7f990cc26740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b392510 0x56222b3b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.272+0000 7f990cc26740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392e30 0x56222b3b6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.272+0000 7f990cc26740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b3ae630 0x56222b3b43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.272+0000 7f990cc26740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f99041209c0 con 0x56222b392510 2024-04-25T00:19:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.272+0000 7f990cc26740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f99041202f0 con 0x56222b3ae630 2024-04-25T00:19:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.272+0000 7f990cc26740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9904067cd0 con 0x56222b392e30 2024-04-25T00:19:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.272+0000 7f990cc26740 10 client.0 _ll_register_callbacks cb 0x7f98fc0318a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:19:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.273+0000 7f9902ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392e30 0x56222b3b6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.273+0000 7f9902ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392e30 0x56222b3b6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:34848/0 (socket says 172.21.15.27:34848) 2024-04-25T00:19:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.273+0000 7f9902ffd640 1 -- 172.21.15.27:0/562289992 learned_addr learned my addr 172.21.15.27:0/562289992 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:19:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.273+0000 7f9903fff640 1 --2- 172.21.15.27:0/562289992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b3ae630 0x56222b3b43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.273+0000 7f9903fff640 1 --2- 172.21.15.27:0/562289992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b392510 0x56222b3b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.273+0000 7f9902ffd640 1 -- 172.21.15.27:0/562289992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b3ae630 msgr2=0x56222b3b43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.273+0000 7f9902ffd640 1 --2- 172.21.15.27:0/562289992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b3ae630 0x56222b3b43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.273+0000 7f9902ffd640 1 -- 172.21.15.27:0/562289992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b392510 msgr2=0x56222b3b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.273+0000 7f9902ffd640 1 --2- 172.21.15.27:0/562289992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b392510 0x56222b3b5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.273+0000 7f9902ffd640 1 -- 172.21.15.27:0/562289992 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56222b3b4ac0 con 0x56222b392e30 2024-04-25T00:19:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.274+0000 7f9902ffd640 1 --2- 172.21.15.27:0/562289992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392e30 0x56222b3b6400 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f98f800b4d0 tx=0x7f98f800bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3974ed07711818e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.276+0000 7f98e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:19:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.276+0000 7f98e2ffd640 1 -- 172.21.15.27:0/562289992 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8015da0 con 0x56222b392e30 2024-04-25T00:19:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.276+0000 7f990cc26740 1 -- 172.21.15.27:0/562289992 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98fc036040 con 0x56222b392e30 2024-04-25T00:19:46.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.276+0000 7f98e2ffd640 1 -- 172.21.15.27:0/562289992 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f98f8017040 con 0x56222b392e30 2024-04-25T00:19:46.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.280+0000 7f98e2ffd640 1 -- 172.21.15.27:0/562289992 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f98f8017240 con 0x56222b392e30 2024-04-25T00:19:46.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.281+0000 7f990cc26740 10 client.25960 fetch_fsmap learned FSMap version 33 2024-04-25T00:19:46.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.281+0000 7f990cc26740 1 -- 172.21.15.27:0/562289992 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56222b4135b0 con 0x56222b392e30 2024-04-25T00:19:46.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.284+0000 7f98e2ffd640 1 -- 172.21.15.27:0/562289992 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98fc036040 con 0x56222b392e30 2024-04-25T00:19:46.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.284+0000 7f98e2ffd640 1 -- 172.21.15.27:0/562289992 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f98f8029b80 con 0x56222b392e30 2024-04-25T00:19:46.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.284+0000 7f990cc26740 10 client.25960 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T00:19:46.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.284+0000 7f990cc26740 10 client.25960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:19:46.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.284+0000 7f990cc26740 20 client.25960 populate_metadata read hostname 'smithi027' 2024-04-25T00:19:46.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.284+0000 7f990cc26740 1 --2- 172.21.15.27:0/562289992 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56222b413d10 0x56222b416130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:46.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.284+0000 7f990cc26740 4 client.25960 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:19:46.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.284+0000 7f990cc26740 1 -- 172.21.15.27:0/562289992 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56222b2fef40 con 0x56222b413d10 2024-04-25T00:19:46.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.284+0000 7f9903fff640 1 --2- 172.21.15.27:0/562289992 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56222b413d10 0x56222b416130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:46.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.285+0000 7f98e2ffd640 10 client.25960 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:19:46.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.285+0000 7f9903fff640 1 --2- 172.21.15.27:0/562289992 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56222b413d10 0x56222b416130 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd43d0d8a97cdd2e server_cookie=1b190d8aa72c68a1 in_seq=0 out_seq=0 2024-04-25T00:19:46.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.287+0000 7f98e2ffd640 1 -- 172.21.15.27:0/562289992 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x56222b2fef40 con 0x56222b413d10 2024-04-25T00:19:46.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.287+0000 7f98e2ffd640 10 client.25960 handle_command_reply: tid=0 2024-04-25T00:19:46.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.287+0000 7f990cc26740 1 client.25960 shutdown 2024-04-25T00:19:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.291+0000 7f990cc26740 1 -- 172.21.15.27:0/562289992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392e30 msgr2=0x56222b3b6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.291+0000 7f990cc26740 1 --2- 172.21.15.27:0/562289992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392e30 0x56222b3b6400 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f98f800b4d0 tx=0x7f98f800bf50 comp rx=0 tx=0).stop 2024-04-25T00:19:46.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.294+0000 7f990cc26740 1 -- 172.21.15.27:0/562289992 shutdown_connections 2024-04-25T00:19:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.294+0000 7f990cc26740 1 --2- 172.21.15.27:0/562289992 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56222b413d10 0x56222b416130 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.294+0000 7f990cc26740 1 --2- 172.21.15.27:0/562289992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56222b3ae630 0x56222b3b43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.294+0000 7f990cc26740 1 --2- 172.21.15.27:0/562289992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56222b392e30 0x56222b3b6400 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.294+0000 7f990cc26740 1 --2- 172.21.15.27:0/562289992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56222b392510 0x56222b3b5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.294+0000 7f990cc26740 1 -- 172.21.15.27:0/562289992 >> 172.21.15.27:0/562289992 conn(0x56222b38e950 msgr2=0x56222b394690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.294+0000 7f98e2ffd640 0 client.25960 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:19:46.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.297+0000 7f990cc26740 1 -- 172.21.15.27:0/562289992 shutdown_connections 2024-04-25T00:19:46.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.297+0000 7f990cc26740 1 -- 172.21.15.27:0/562289992 wait complete. 2024-04-25T00:19:46.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.298+0000 7f990cc26740 20 client.25960 trim_cache size 0 max 16384 2024-04-25T00:19:46.298 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:19:46.298 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "scrub active (201 inodes in the stack)", 2024-04-25T00:19:46.298 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:19:46.298 INFO:teuthology.orchestra.run.smithi027.stdout: "a40f0131-2b37-4eb9-a2bd-6fee5124aebc": { 2024-04-25T00:19:46.299 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:19:46.299 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "a40f0131-2b37-4eb9-a2bd-6fee5124aebc", 2024-04-25T00:19:46.299 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:19:46.299 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:19:46.299 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:19:46.299 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:19:46.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.298+0000 7f9909e38640 1 -- 172.21.15.27:0/2181742070 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f98ec0a2f00 msgr2=0x7f98ec0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:46.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.298+0000 7f9909e38640 1 --2- 172.21.15.27:0/2181742070 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f98ec0a2f00 0x7f98ec0a53c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f98f409ada0 tx=0x7f98f4097340 comp rx=0 tx=0).stop 2024-04-25T00:19:46.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.298+0000 7f9909e38640 1 -- 172.21.15.27:0/2181742070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9904069a50 msgr2=0x7f9904114c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:46.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.298+0000 7f9909e38640 1 --2- 172.21.15.27:0/2181742070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9904069a50 0x7f9904114c90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f98f400b3f0 tx=0x7f98f400b8b0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.299+0000 7f9909e38640 1 -- 172.21.15.27:0/2181742070 shutdown_connections 2024-04-25T00:19:46.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.299+0000 7f9909e38640 1 --2- 172.21.15.27:0/2181742070 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f98ec0a2f00 0x7f98ec0a53c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.299+0000 7f9909e38640 1 --2- 172.21.15.27:0/2181742070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9904106670 0x7f990410ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.299+0000 7f9909e38640 1 --2- 172.21.15.27:0/2181742070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f990406a4d0 0x7f99041151d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.299+0000 7f9909e38640 1 --2- 172.21.15.27:0/2181742070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9904069a50 0x7f9904114c90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.299+0000 7f9909e38640 1 -- 172.21.15.27:0/2181742070 >> 172.21.15.27:0/2181742070 conn(0x7f99040761c0 msgr2=0x7f9904103230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.299+0000 7f9909e38640 1 -- 172.21.15.27:0/2181742070 shutdown_connections 2024-04-25T00:19:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:19:46.299+0000 7f9909e38640 1 -- 172.21.15.27:0/2181742070 wait complete. 2024-04-25T00:19:46.310 INFO:tasks.cephfs.filesystem:scrub status for tag:a40f0131-2b37-4eb9-a2bd-6fee5124aebc - {'path': '/', 'tag': 'a40f0131-2b37-4eb9-a2bd-6fee5124aebc', 'options': 'recursive,force'} 2024-04-25T00:19:46.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:46 smithi027 ceph-mon[29208]: pgmap v123: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 38 MiB/s wr, 7.09k op/s 2024-04-25T00:19:46.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:46 smithi139 ceph-mon[32833]: pgmap v123: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 38 MiB/s wr, 7.09k op/s 2024-04-25T00:19:46.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:46 smithi153 ceph-mon[32896]: pgmap v123: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 38 MiB/s wr, 7.09k op/s 2024-04-25T00:19:48.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:48 smithi027 ceph-mon[29208]: pgmap v124: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 52 MiB/s wr, 9.58k op/s 2024-04-25T00:19:48.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:48 smithi139 ceph-mon[32833]: pgmap v124: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 52 MiB/s wr, 9.58k op/s 2024-04-25T00:19:48.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:48 smithi153 ceph-mon[32896]: pgmap v124: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 52 MiB/s wr, 9.58k op/s 2024-04-25T00:19:50.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:50 smithi027 ceph-mon[29208]: pgmap v125: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 51 MiB/s wr, 9.07k op/s 2024-04-25T00:19:50.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:50 smithi139 ceph-mon[32833]: pgmap v125: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 51 MiB/s wr, 9.07k op/s 2024-04-25T00:19:50.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:50 smithi153 ceph-mon[32896]: pgmap v125: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 51 MiB/s wr, 9.07k op/s 2024-04-25T00:19:52.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:52 smithi027 ceph-mon[29208]: pgmap v126: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 47 MiB/s wr, 8.07k op/s 2024-04-25T00:19:52.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:52 smithi139 ceph-mon[32833]: pgmap v126: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 47 MiB/s wr, 8.07k op/s 2024-04-25T00:19:52.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:52 smithi153 ceph-mon[32896]: pgmap v126: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 47 MiB/s wr, 8.07k op/s 2024-04-25T00:19:54.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:54 smithi027 ceph-mon[29208]: pgmap v127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 68 MiB/s wr, 12.07k op/s 2024-04-25T00:19:54.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:54 smithi139 ceph-mon[32833]: pgmap v127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 68 MiB/s wr, 12.07k op/s 2024-04-25T00:19:54.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:54 smithi153 ceph-mon[32896]: pgmap v127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 68 MiB/s wr, 12.07k op/s 2024-04-25T00:19:56.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:56 smithi027 ceph-mon[29208]: pgmap v128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 45 MiB/s wr, 8.20k op/s 2024-04-25T00:19:56.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:56 smithi139 ceph-mon[32833]: pgmap v128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 45 MiB/s wr, 8.20k op/s 2024-04-25T00:19:56.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:56 smithi153 ceph-mon[32896]: pgmap v128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 45 MiB/s wr, 8.20k op/s 2024-04-25T00:19:58.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:19:58 smithi027 ceph-mon[29208]: pgmap v129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 51 MiB/s wr, 9.10k op/s 2024-04-25T00:19:58.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:19:58 smithi139 ceph-mon[32833]: pgmap v129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 51 MiB/s wr, 9.10k op/s 2024-04-25T00:19:58.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:19:58 smithi153 ceph-mon[32896]: pgmap v129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 51 MiB/s wr, 9.10k op/s 2024-04-25T00:20:00.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:00 smithi027 ceph-mon[29208]: pgmap v130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 51 MiB/s wr, 9.18k op/s 2024-04-25T00:20:00.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:00 smithi027 ceph-mon[29208]: overall HEALTH_OK 2024-04-25T00:20:00.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:00 smithi139 ceph-mon[32833]: pgmap v130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 51 MiB/s wr, 9.18k op/s 2024-04-25T00:20:00.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:00 smithi139 ceph-mon[32833]: overall HEALTH_OK 2024-04-25T00:20:00.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:00 smithi153 ceph-mon[32896]: pgmap v130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 51 MiB/s wr, 9.18k op/s 2024-04-25T00:20:00.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:00 smithi153 ceph-mon[32896]: overall HEALTH_OK 2024-04-25T00:20:02.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:02 smithi027 ceph-mon[29208]: pgmap v131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 42 MiB/s wr, 7.46k op/s 2024-04-25T00:20:02.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:02 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:20:02.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:02 smithi139 ceph-mon[32833]: pgmap v131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 42 MiB/s wr, 7.46k op/s 2024-04-25T00:20:02.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:02 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:20:02.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:02 smithi153 ceph-mon[32896]: pgmap v131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 42 MiB/s wr, 7.46k op/s 2024-04-25T00:20:02.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:02 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:20:03.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:03 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:20:03.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:03 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:20:03.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:03 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:20:03.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:03 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:20:03.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:03 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:20:03.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:03 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:20:03.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:03 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:20:03.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:03 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:20:03.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:03 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:20:04.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:04 smithi139 ceph-mon[32833]: pgmap v132: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 60 MiB/s wr, 10.83k op/s 2024-04-25T00:20:04.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:04 smithi153 ceph-mon[32896]: pgmap v132: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 60 MiB/s wr, 10.83k op/s 2024-04-25T00:20:04.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:04 smithi027 ceph-mon[29208]: pgmap v132: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 60 MiB/s wr, 10.83k op/s 2024-04-25T00:20:06.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:06 smithi153 ceph-mon[32896]: pgmap v133: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 38 MiB/s wr, 6.84k op/s 2024-04-25T00:20:06.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:06 smithi027 ceph-mon[29208]: pgmap v133: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 38 MiB/s wr, 6.84k op/s 2024-04-25T00:20:07.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:06 smithi139 ceph-mon[32833]: pgmap v133: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 38 MiB/s wr, 6.84k op/s 2024-04-25T00:20:08.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:08 smithi153 ceph-mon[32896]: pgmap v134: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 42 MiB/s wr, 7.69k op/s 2024-04-25T00:20:08.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:08 smithi027 ceph-mon[29208]: pgmap v134: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 42 MiB/s wr, 7.69k op/s 2024-04-25T00:20:09.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:08 smithi139 ceph-mon[32833]: pgmap v134: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 42 MiB/s wr, 7.69k op/s 2024-04-25T00:20:10.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:10 smithi027 ceph-mon[29208]: pgmap v135: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 47 MiB/s wr, 8.86k op/s 2024-04-25T00:20:11.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:10 smithi139 ceph-mon[32833]: pgmap v135: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 47 MiB/s wr, 8.86k op/s 2024-04-25T00:20:11.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:10 smithi153 ceph-mon[32896]: pgmap v135: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 47 MiB/s wr, 8.86k op/s 2024-04-25T00:20:12.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:12 smithi027 ceph-mon[29208]: pgmap v136: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 33 MiB/s wr, 6.31k op/s 2024-04-25T00:20:13.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:12 smithi139 ceph-mon[32833]: pgmap v136: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 33 MiB/s wr, 6.31k op/s 2024-04-25T00:20:13.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:12 smithi153 ceph-mon[32896]: pgmap v136: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 33 MiB/s wr, 6.31k op/s 2024-04-25T00:20:14.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:14 smithi027 ceph-mon[29208]: pgmap v137: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 42 MiB/s wr, 8.22k op/s 2024-04-25T00:20:15.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:14 smithi139 ceph-mon[32833]: pgmap v137: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 42 MiB/s wr, 8.22k op/s 2024-04-25T00:20:15.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:14 smithi153 ceph-mon[32896]: pgmap v137: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 42 MiB/s wr, 8.22k op/s 2024-04-25T00:20:16.312 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:20:16.312 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:20:16.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.497+0000 7f5668d32640 1 -- 172.21.15.27:0/1944867194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5664074330 msgr2=0x7f56641122b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.497+0000 7f5668d32640 1 --2- 172.21.15.27:0/1944867194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5664074330 0x7f56641122b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f565c009ca0 tx=0x7f565c02a970 comp rx=0 tx=0).stop 2024-04-25T00:20:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.497+0000 7f5668d32640 1 -- 172.21.15.27:0/1944867194 shutdown_connections 2024-04-25T00:20:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.497+0000 7f5668d32640 1 --2- 172.21.15.27:0/1944867194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56641127f0 0x7f5664114be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.497+0000 7f5668d32640 1 --2- 172.21.15.27:0/1944867194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5664074330 0x7f56641122b0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f565c009ca0 tx=0x7f565c02a970 comp rx=0 tx=0).stop 2024-04-25T00:20:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.497+0000 7f5668d32640 1 --2- 172.21.15.27:0/1944867194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5664073a10 0x7f5664073df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.497+0000 7f5668d32640 1 -- 172.21.15.27:0/1944867194 >> 172.21.15.27:0/1944867194 conn(0x7f5664078b90 msgr2=0x7f5664078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:16.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.499+0000 7f5668d32640 1 -- 172.21.15.27:0/1944867194 shutdown_connections 2024-04-25T00:20:16.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.499+0000 7f5668d32640 1 -- 172.21.15.27:0/1944867194 wait complete. 2024-04-25T00:20:16.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.500+0000 7f5668d32640 1 Processor -- start 2024-04-25T00:20:16.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.500+0000 7f5668d32640 1 -- start start 2024-04-25T00:20:16.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.500+0000 7f5668d32640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5664073a10 0x7f56641a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.500+0000 7f5668d32640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56641127f0 0x7f56641a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.500+0000 7f5668d32640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566419e740 0x7f566419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.500+0000 7f5668d32640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f56641181e0 con 0x7f56641127f0 2024-04-25T00:20:16.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.500+0000 7f5668d32640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5664118060 con 0x7f5664073a10 2024-04-25T00:20:16.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.500+0000 7f5668d32640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5664118360 con 0x7f566419e740 2024-04-25T00:20:16.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.501+0000 7f5661d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56641127f0 0x7f56641a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:16.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.501+0000 7f5661d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56641127f0 0x7f56641a4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41840/0 (socket says 172.21.15.27:41840) 2024-04-25T00:20:16.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.501+0000 7f5661d9b640 1 -- 172.21.15.27:0/3483639503 learned_addr learned my addr 172.21.15.27:0/3483639503 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:20:16.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.501+0000 7f566259c640 1 --2- 172.21.15.27:0/3483639503 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5664073a10 0x7f56641a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:16.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.502+0000 7f5662d9d640 1 --2- 172.21.15.27:0/3483639503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566419e740 0x7f566419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:16.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.502+0000 7f5661d9b640 1 -- 172.21.15.27:0/3483639503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566419e740 msgr2=0x7f566419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:16.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.502+0000 7f5661d9b640 1 --2- 172.21.15.27:0/3483639503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566419e740 0x7f566419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.502+0000 7f5661d9b640 1 -- 172.21.15.27:0/3483639503 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5664073a10 msgr2=0x7f56641a4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:16.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.502+0000 7f5661d9b640 1 --2- 172.21.15.27:0/3483639503 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5664073a10 0x7f56641a4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.502+0000 7f5661d9b640 1 -- 172.21.15.27:0/3483639503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566419f340 con 0x7f56641127f0 2024-04-25T00:20:16.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.502+0000 7f5662d9d640 1 --2- 172.21.15.27:0/3483639503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566419e740 0x7f566419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:20:16.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.502+0000 7f5661d9b640 1 --2- 172.21.15.27:0/3483639503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56641127f0 0x7f56641a4b70 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f565c005e30 tx=0x7f565c003ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:16.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.503+0000 7f56537fe640 1 -- 172.21.15.27:0/3483639503 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565c034070 con 0x7f56641127f0 2024-04-25T00:20:16.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.503+0000 7f56537fe640 1 -- 172.21.15.27:0/3483639503 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f565c030b80 con 0x7f56641127f0 2024-04-25T00:20:16.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.503+0000 7f56537fe640 1 -- 172.21.15.27:0/3483639503 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565c004850 con 0x7f56641127f0 2024-04-25T00:20:16.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.503+0000 7f5668d32640 1 -- 172.21.15.27:0/3483639503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f566419f5d0 con 0x7f56641127f0 2024-04-25T00:20:16.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.503+0000 7f5668d32640 1 -- 172.21.15.27:0/3483639503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56641ab670 con 0x7f56641127f0 2024-04-25T00:20:16.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.507+0000 7f566ba3d740 1 Processor -- start 2024-04-25T00:20:16.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.508+0000 7f56537fe640 1 -- 172.21.15.27:0/3483639503 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f565c037020 con 0x7f56641127f0 2024-04-25T00:20:16.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.509+0000 7f566ba3d740 1 -- start start 2024-04-25T00:20:16.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.509+0000 7f566ba3d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedef630 0x5638aedefa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.509+0000 7f566ba3d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedeff50 0x5638aedfa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.509+0000 7f566ba3d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedfa620 0x5638aedfca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.509+0000 7f566ba3d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5638aed546b0 con 0x5638aedeff50 2024-04-25T00:20:16.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.509+0000 7f566ba3d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5638aecfc880 con 0x5638aedef630 2024-04-25T00:20:16.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.509+0000 7f566ba3d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5638aed50670 con 0x5638aedfa620 2024-04-25T00:20:16.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.509+0000 7f566259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedeff50 0x5638aedfa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:16.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.510+0000 7f5662d9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedef630 0x5638aedefa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:16.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.510+0000 7f566259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedeff50 0x5638aedfa0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41850/0 (socket says 172.21.15.27:41850) 2024-04-25T00:20:16.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.510+0000 7f566259c640 1 -- 172.21.15.27:0/1034348046 learned_addr learned my addr 172.21.15.27:0/1034348046 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:20:16.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.510+0000 7f5661d9b640 1 --2- 172.21.15.27:0/1034348046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedfa620 0x5638aedfca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:16.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.510+0000 7f56537fe640 1 --2- 172.21.15.27:0/3483639503 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f564c0a2f20 0x7f564c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.510+0000 7f5662d9d640 1 -- 172.21.15.27:0/1034348046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedfa620 msgr2=0x5638aedfca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:16.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.510+0000 7f5662d9d640 1 --2- 172.21.15.27:0/1034348046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedfa620 0x5638aedfca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.510+0000 7f5662d9d640 1 -- 172.21.15.27:0/1034348046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedeff50 msgr2=0x5638aedfa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:16.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.510+0000 7f5662d9d640 1 --2- 172.21.15.27:0/1034348046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedeff50 0x5638aedfa0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.510+0000 7f5662d9d640 1 -- 172.21.15.27:0/1034348046 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638aeb327d0 con 0x5638aedef630 2024-04-25T00:20:16.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.510+0000 7f56537fe640 1 -- 172.21.15.27:0/3483639503 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f565c0f03e0 con 0x7f56641127f0 2024-04-25T00:20:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.512+0000 7f5662d9d640 1 --2- 172.21.15.27:0/3483639503 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f564c0a2f20 0x7f564c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.512+0000 7f5662d9d640 1 --2- 172.21.15.27:0/1034348046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedef630 0x5638aedefa10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5658009d00 tx=0x7f56580117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58754d56987cffd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.512+0000 7f5650ff9640 1 -- 172.21.15.27:0/1034348046 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5658023070 con 0x5638aedef630 2024-04-25T00:20:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.512+0000 7f5650ff9640 1 -- 172.21.15.27:0/1034348046 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f565801d960 con 0x5638aedef630 2024-04-25T00:20:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.512+0000 7f5650ff9640 1 -- 172.21.15.27:0/1034348046 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565800f860 con 0x5638aedef630 2024-04-25T00:20:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.513+0000 7f566ba3d740 1 -- 172.21.15.27:0/1034348046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedef630 msgr2=0x5638aedefa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:16.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.513+0000 7f566ba3d740 1 --2- 172.21.15.27:0/1034348046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedef630 0x5638aedefa10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5658009d00 tx=0x7f56580117b0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.513+0000 7f5662d9d640 1 --2- 172.21.15.27:0/3483639503 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f564c0a2f20 0x7f564c0a53e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5658020000 tx=0x7f565801d000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:16.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.513+0000 7f566ba3d740 1 -- 172.21.15.27:0/1034348046 shutdown_connections 2024-04-25T00:20:16.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.513+0000 7f566ba3d740 1 --2- 172.21.15.27:0/1034348046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedfa620 0x5638aedfca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.513+0000 7f566ba3d740 1 --2- 172.21.15.27:0/1034348046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedeff50 0x5638aedfa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.513+0000 7f566ba3d740 1 --2- 172.21.15.27:0/1034348046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedef630 0x5638aedefa10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.513+0000 7f566ba3d740 1 -- 172.21.15.27:0/1034348046 >> 172.21.15.27:0/1034348046 conn(0x5638aedeb950 msgr2=0x5638aedebd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:16.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.514+0000 7f566ba3d740 1 -- 172.21.15.27:0/1034348046 shutdown_connections 2024-04-25T00:20:16.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.514+0000 7f566ba3d740 1 -- 172.21.15.27:0/1034348046 wait complete. 2024-04-25T00:20:16.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.514+0000 7f566ba3d740 1 -- start start 2024-04-25T00:20:16.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.514+0000 7f566ba3d740 1 Processor -- start 2024-04-25T00:20:16.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.514+0000 7f566ba3d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedef630 0x5638aedf7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.514+0000 7f566ba3d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedeff50 0x5638aedf7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.515+0000 7f566ba3d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedfa620 0x5638aedf8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.515+0000 7f566ba3d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5638aee0e420 con 0x5638aedef630 2024-04-25T00:20:16.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.515+0000 7f566ba3d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5638aed0ff50 con 0x5638aedfa620 2024-04-25T00:20:16.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.515+0000 7f566ba3d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5638aee0e5a0 con 0x5638aedeff50 2024-04-25T00:20:16.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.515+0000 7f566ba3d740 10 client.0 _ll_register_callbacks cb 0x5638aed4b000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:20:16.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.515+0000 7f5661d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedeff50 0x5638aedf7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:16.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.515+0000 7f5662d9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedfa620 0x5638aedf8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:16.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.515+0000 7f5661d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedeff50 0x5638aedf7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:36914/0 (socket says 172.21.15.27:36914) 2024-04-25T00:20:16.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.515+0000 7f5661d9b640 1 -- 172.21.15.27:0/3851494545 learned_addr learned my addr 172.21.15.27:0/3851494545 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:20:16.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.516+0000 7f566259c640 1 --2- 172.21.15.27:0/3851494545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedef630 0x5638aedf7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:16.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.516+0000 7f566259c640 1 -- 172.21.15.27:0/3851494545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedeff50 msgr2=0x5638aedf7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:16.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.516+0000 7f566259c640 1 --2- 172.21.15.27:0/3851494545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedeff50 0x5638aedf7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.516+0000 7f566259c640 1 -- 172.21.15.27:0/3851494545 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedfa620 msgr2=0x5638aedf8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:16.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.516+0000 7f566259c640 1 --2- 172.21.15.27:0/3851494545 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedfa620 0x5638aedf8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.516+0000 7f566259c640 1 -- 172.21.15.27:0/3851494545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638aea683f0 con 0x5638aedef630 2024-04-25T00:20:16.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.516+0000 7f566259c640 1 --2- 172.21.15.27:0/3851494545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedef630 0x5638aedf7580 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f56540104d0 tx=0x7f56540108d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af82beeea39b60e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:16.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.516+0000 7f5649ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:20:16.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.517+0000 7f5649ffb640 1 -- 172.21.15.27:0/3851494545 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5654018070 con 0x5638aedef630 2024-04-25T00:20:16.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.517+0000 7f5649ffb640 1 -- 172.21.15.27:0/3851494545 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f5654014640 con 0x5638aedef630 2024-04-25T00:20:16.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.517+0000 7f566ba3d740 1 -- 172.21.15.27:0/3851494545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f565801d960 con 0x5638aedef630 2024-04-25T00:20:16.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.518+0000 7f5649ffb640 1 -- 172.21.15.27:0/3851494545 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5654022050 con 0x5638aedef630 2024-04-25T00:20:16.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.518+0000 7f566ba3d740 10 client.16473 fetch_fsmap learned FSMap version 33 2024-04-25T00:20:16.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.518+0000 7f566ba3d740 1 -- 172.21.15.27:0/3851494545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5638aea280f0 con 0x5638aedef630 2024-04-25T00:20:16.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.519+0000 7f5649ffb640 1 -- 172.21.15.27:0/3851494545 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f565801d960 con 0x5638aedef630 2024-04-25T00:20:16.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.519+0000 7f5649ffb640 1 -- 172.21.15.27:0/3851494545 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f565401db40 con 0x5638aedef630 2024-04-25T00:20:16.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.519+0000 7f566ba3d740 10 client.16473 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T00:20:16.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.519+0000 7f566ba3d740 10 client.16473 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:20:16.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.519+0000 7f566ba3d740 20 client.16473 populate_metadata read hostname 'smithi027' 2024-04-25T00:20:16.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.519+0000 7f566ba3d740 1 --2- 172.21.15.27:0/3851494545 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5638aee1b990 0x5638aee1dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.519+0000 7f566ba3d740 4 client.16473 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:20:16.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.519+0000 7f566ba3d740 1 -- 172.21.15.27:0/3851494545 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638aed61bf0 con 0x5638aee1b990 2024-04-25T00:20:16.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.520+0000 7f5661d9b640 1 --2- 172.21.15.27:0/3851494545 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5638aee1b990 0x5638aee1dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:16.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.522+0000 7f5661d9b640 1 --2- 172.21.15.27:0/3851494545 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5638aee1b990 0x5638aee1dda0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fd6381a52a436a8 server_cookie=9f8aeef4b672e104 in_seq=0 out_seq=0 2024-04-25T00:20:16.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.522+0000 7f5649ffb640 10 client.16473 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:20:16.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.524+0000 7f5649ffb640 1 -- 172.21.15.27:0/3851494545 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5638aed61bf0 con 0x5638aee1b990 2024-04-25T00:20:16.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.524+0000 7f5649ffb640 10 client.16473 handle_command_reply: tid=0 2024-04-25T00:20:16.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.524+0000 7f566ba3d740 1 client.16473 shutdown 2024-04-25T00:20:16.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.525+0000 7f566ba3d740 1 -- 172.21.15.27:0/3851494545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedef630 msgr2=0x5638aedf7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:16.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.525+0000 7f566ba3d740 1 --2- 172.21.15.27:0/3851494545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedef630 0x5638aedf7580 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f56540104d0 tx=0x7f56540108d0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.528+0000 7f566ba3d740 1 -- 172.21.15.27:0/3851494545 shutdown_connections 2024-04-25T00:20:16.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.528+0000 7f566ba3d740 1 --2- 172.21.15.27:0/3851494545 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5638aee1b990 0x5638aee1dda0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.528+0000 7f566ba3d740 1 --2- 172.21.15.27:0/3851494545 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedfa620 0x5638aedf8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.528+0000 7f566ba3d740 1 --2- 172.21.15.27:0/3851494545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedeff50 0x5638aedf7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.528+0000 7f566ba3d740 1 --2- 172.21.15.27:0/3851494545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedef630 0x5638aedf7580 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.528+0000 7f566ba3d740 1 -- 172.21.15.27:0/3851494545 >> 172.21.15.27:0/3851494545 conn(0x5638aedeb950 msgr2=0x5638aedf1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.528+0000 7f5649ffb640 0 client.16473 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:20:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.528+0000 7f566ba3d740 1 -- 172.21.15.27:0/3851494545 shutdown_connections 2024-04-25T00:20:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.528+0000 7f566ba3d740 1 -- 172.21.15.27:0/3851494545 wait complete. 2024-04-25T00:20:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.528+0000 7f566ba3d740 20 client.16473 trim_cache size 0 max 16384 2024-04-25T00:20:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.564+0000 7f566ba3d740 1 Processor -- start 2024-04-25T00:20:16.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.564+0000 7f566ba3d740 1 -- start start 2024-04-25T00:20:16.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.564+0000 7f566ba3d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedef630 0x5638aee172e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.564+0000 7f566ba3d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedeff50 0x5638aedf5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.564+0000 7f566ba3d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedfa620 0x5638aedf7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.564+0000 7f566ba3d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f56641ad000 con 0x5638aedef630 2024-04-25T00:20:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.564+0000 7f566ba3d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f56641aeb40 con 0x5638aedeff50 2024-04-25T00:20:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.564+0000 7f566ba3d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f56641ad6d0 con 0x5638aedfa620 2024-04-25T00:20:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.565+0000 7f5661d9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedeff50 0x5638aedf5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.565+0000 7f5661d9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedeff50 0x5638aedf5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:43476/0 (socket says 172.21.15.27:43476) 2024-04-25T00:20:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.565+0000 7f5661d9b640 1 -- 172.21.15.27:0/2830170095 learned_addr learned my addr 172.21.15.27:0/2830170095 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:20:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.565+0000 7f5662d9d640 1 --2- 172.21.15.27:0/2830170095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedfa620 0x5638aedf7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.567+0000 7f5662d9d640 1 -- 172.21.15.27:0/2830170095 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedeff50 msgr2=0x5638aedf5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.567+0000 7f5662d9d640 1 --2- 172.21.15.27:0/2830170095 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedeff50 0x5638aedf5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.567+0000 7f5662d9d640 1 -- 172.21.15.27:0/2830170095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedef630 msgr2=0x5638aee172e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.567+0000 7f5662d9d640 1 --2- 172.21.15.27:0/2830170095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedef630 0x5638aee172e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.567+0000 7f5662d9d640 1 -- 172.21.15.27:0/2830170095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638aedef430 con 0x5638aedfa620 2024-04-25T00:20:16.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.571+0000 7f5662d9d640 1 --2- 172.21.15.27:0/2830170095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedfa620 0x5638aedf7e20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5658021040 tx=0x7f5658027000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11fedcfb1ec0a5aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.573+0000 7f564bfff640 1 -- 172.21.15.27:0/2830170095 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5658023040 con 0x5638aedfa620 2024-04-25T00:20:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.573+0000 7f564bfff640 1 -- 172.21.15.27:0/2830170095 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56580146b0 con 0x5638aedfa620 2024-04-25T00:20:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.573+0000 7f564bfff640 1 -- 172.21.15.27:0/2830170095 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565800f860 con 0x5638aedfa620 2024-04-25T00:20:16.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.574+0000 7f566ba3d740 1 -- 172.21.15.27:0/2830170095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedfa620 msgr2=0x5638aedf7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:16.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.574+0000 7f566ba3d740 1 --2- 172.21.15.27:0/2830170095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedfa620 0x5638aedf7e20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5658021040 tx=0x7f5658027000 comp rx=0 tx=0).stop 2024-04-25T00:20:16.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.575+0000 7f566ba3d740 1 -- 172.21.15.27:0/2830170095 shutdown_connections 2024-04-25T00:20:16.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.575+0000 7f566ba3d740 1 --2- 172.21.15.27:0/2830170095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedfa620 0x5638aedf7e20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.575+0000 7f566ba3d740 1 --2- 172.21.15.27:0/2830170095 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedeff50 0x5638aedf5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.575+0000 7f566ba3d740 1 --2- 172.21.15.27:0/2830170095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedef630 0x5638aee172e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.575+0000 7f566ba3d740 1 -- 172.21.15.27:0/2830170095 >> 172.21.15.27:0/2830170095 conn(0x5638aedeb950 msgr2=0x5638aedeaaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:16.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.578+0000 7f566ba3d740 1 -- 172.21.15.27:0/2830170095 shutdown_connections 2024-04-25T00:20:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.579+0000 7f566ba3d740 1 -- 172.21.15.27:0/2830170095 wait complete. 2024-04-25T00:20:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.579+0000 7f566ba3d740 1 -- start start 2024-04-25T00:20:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.579+0000 7f566ba3d740 1 Processor -- start 2024-04-25T00:20:16.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.580+0000 7f566ba3d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedef630 0x5638aee127f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.580+0000 7f566ba3d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedeff50 0x5638aee12d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.580+0000 7f566ba3d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedfa620 0x5638aee13290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.580+0000 7f566ba3d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f56641ae470 con 0x5638aedeff50 2024-04-25T00:20:16.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.580+0000 7f566ba3d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f56641adda0 con 0x5638aedef630 2024-04-25T00:20:16.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.580+0000 7f566ba3d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f566410d8c0 con 0x5638aedfa620 2024-04-25T00:20:16.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.580+0000 7f566ba3d740 10 client.0 _ll_register_callbacks cb 0x7f565400b500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:20:16.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.580+0000 7f566259c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedef630 0x5638aee127f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:16.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.580+0000 7f566259c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedef630 0x5638aee127f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:43478/0 (socket says 172.21.15.27:43478) 2024-04-25T00:20:16.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.580+0000 7f566259c640 1 -- 172.21.15.27:0/402791167 learned_addr learned my addr 172.21.15.27:0/402791167 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:20:16.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.581+0000 7f5662d9d640 1 --2- 172.21.15.27:0/402791167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedfa620 0x5638aee13290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:16.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.581+0000 7f566259c640 1 -- 172.21.15.27:0/402791167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedfa620 msgr2=0x5638aee13290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:16.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.581+0000 7f566259c640 1 --2- 172.21.15.27:0/402791167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedfa620 0x5638aee13290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.581+0000 7f566259c640 1 -- 172.21.15.27:0/402791167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedeff50 msgr2=0x5638aee12d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:16.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.581+0000 7f5661d9b640 1 --2- 172.21.15.27:0/402791167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedeff50 0x5638aee12d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.581+0000 7f566259c640 1 --2- 172.21.15.27:0/402791167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedeff50 0x5638aee12d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.581+0000 7f566259c640 1 -- 172.21.15.27:0/402791167 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638aee13ac0 con 0x5638aedef630 2024-04-25T00:20:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.581+0000 7f5661d9b640 1 --2- 172.21.15.27:0/402791167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedeff50 0x5638aee12d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:20:16.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.582+0000 7f566259c640 1 --2- 172.21.15.27:0/402791167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedef630 0x5638aee127f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5654020040 tx=0x7f5654013c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3062ffbfa57da0c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:16.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.582+0000 7f5649ffb640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T00:20:16.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.582+0000 7f5649ffb640 1 -- 172.21.15.27:0/402791167 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5654018070 con 0x5638aedef630 2024-04-25T00:20:16.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.582+0000 7f566ba3d740 1 -- 172.21.15.27:0/402791167 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56580146b0 con 0x5638aedef630 2024-04-25T00:20:16.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.582+0000 7f5649ffb640 1 -- 172.21.15.27:0/402791167 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f565400f040 con 0x5638aedef630 2024-04-25T00:20:16.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.583+0000 7f5649ffb640 1 -- 172.21.15.27:0/402791167 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5654036050 con 0x5638aedef630 2024-04-25T00:20:16.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.585+0000 7f566ba3d740 10 client.25997 fetch_fsmap learned FSMap version 33 2024-04-25T00:20:16.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.585+0000 7f566ba3d740 1 -- 172.21.15.27:0/402791167 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5638aee93a00 con 0x5638aedef630 2024-04-25T00:20:16.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.586+0000 7f5649ffb640 1 -- 172.21.15.27:0/402791167 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56580146b0 con 0x5638aedef630 2024-04-25T00:20:16.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.586+0000 7f5649ffb640 1 -- 172.21.15.27:0/402791167 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f5654004480 con 0x5638aedef630 2024-04-25T00:20:16.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.586+0000 7f566ba3d740 10 client.25997 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T00:20:16.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.586+0000 7f566ba3d740 10 client.25997 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:20:16.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.586+0000 7f566ba3d740 20 client.25997 populate_metadata read hostname 'smithi027' 2024-04-25T00:20:16.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.586+0000 7f566ba3d740 1 --2- 172.21.15.27:0/402791167 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5638aee940c0 0x5638aee964e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:16.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.586+0000 7f566ba3d740 4 client.25997 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:20:16.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.586+0000 7f566ba3d740 1 -- 172.21.15.27:0/402791167 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5638aed5bf40 con 0x5638aee940c0 2024-04-25T00:20:16.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.588+0000 7f5661d9b640 1 --2- 172.21.15.27:0/402791167 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5638aee940c0 0x5638aee964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:16.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.590+0000 7f5661d9b640 1 --2- 172.21.15.27:0/402791167 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5638aee940c0 0x5638aee964e0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46d8cef16c92b594 server_cookie=118429f231ab4992 in_seq=0 out_seq=0 2024-04-25T00:20:16.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.590+0000 7f5649ffb640 10 client.25997 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:20:16.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.590+0000 7f5649ffb640 1 -- 172.21.15.27:0/402791167 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5638aed5bf40 con 0x5638aee940c0 2024-04-25T00:20:16.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.590+0000 7f5649ffb640 10 client.25997 handle_command_reply: tid=0 2024-04-25T00:20:16.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.590+0000 7f566ba3d740 1 client.25997 shutdown 2024-04-25T00:20:16.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.592+0000 7f566ba3d740 1 -- 172.21.15.27:0/402791167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedef630 msgr2=0x5638aee127f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:16.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.592+0000 7f566ba3d740 1 --2- 172.21.15.27:0/402791167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedef630 0x5638aee127f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5654020040 tx=0x7f5654013c10 comp rx=0 tx=0).stop 2024-04-25T00:20:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.598+0000 7f566ba3d740 1 -- 172.21.15.27:0/402791167 shutdown_connections 2024-04-25T00:20:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.598+0000 7f566ba3d740 1 --2- 172.21.15.27:0/402791167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638aedfa620 0x5638aee13290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.598+0000 7f566ba3d740 1 --2- 172.21.15.27:0/402791167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638aedeff50 0x5638aee12d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.598+0000 7f566ba3d740 1 --2- 172.21.15.27:0/402791167 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5638aee940c0 0x5638aee964e0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.598+0000 7f566ba3d740 1 --2- 172.21.15.27:0/402791167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638aedef630 0x5638aee127f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.598+0000 7f566ba3d740 1 -- 172.21.15.27:0/402791167 >> 172.21.15.27:0/402791167 conn(0x5638aedeb950 msgr2=0x5638aedeaaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:16.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.599+0000 7f5649ffb640 0 client.25997 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:20:16.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.600+0000 7f566ba3d740 1 -- 172.21.15.27:0/402791167 shutdown_connections 2024-04-25T00:20:16.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.602+0000 7f566ba3d740 1 -- 172.21.15.27:0/402791167 wait complete. 2024-04-25T00:20:16.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.602+0000 7f566ba3d740 20 client.25997 trim_cache size 0 max 16384 2024-04-25T00:20:16.603 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:20:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "scrub active (201 inodes in the stack)", 2024-04-25T00:20:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:20:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "a40f0131-2b37-4eb9-a2bd-6fee5124aebc": { 2024-04-25T00:20:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:20:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "a40f0131-2b37-4eb9-a2bd-6fee5124aebc", 2024-04-25T00:20:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:20:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:20:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:20:16.604 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:20:16.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.606+0000 7f5668d32640 1 -- 172.21.15.27:0/3483639503 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f564c0a2f20 msgr2=0x7f564c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:16.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.606+0000 7f5668d32640 1 --2- 172.21.15.27:0/3483639503 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f564c0a2f20 0x7f564c0a53e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5658020000 tx=0x7f565801d000 comp rx=0 tx=0).stop 2024-04-25T00:20:16.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.606+0000 7f5668d32640 1 -- 172.21.15.27:0/3483639503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56641127f0 msgr2=0x7f56641a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:16.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.606+0000 7f5668d32640 1 --2- 172.21.15.27:0/3483639503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56641127f0 0x7f56641a4b70 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f565c005e30 tx=0x7f565c003ee0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.609+0000 7f5668d32640 1 -- 172.21.15.27:0/3483639503 shutdown_connections 2024-04-25T00:20:16.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.609+0000 7f5668d32640 1 --2- 172.21.15.27:0/3483639503 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f564c0a2f20 0x7f564c0a53e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.609+0000 7f5668d32640 1 --2- 172.21.15.27:0/3483639503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566419e740 0x7f566419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.609+0000 7f5668d32640 1 --2- 172.21.15.27:0/3483639503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56641127f0 0x7f56641a4b70 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.609+0000 7f5668d32640 1 --2- 172.21.15.27:0/3483639503 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5664073a10 0x7f56641a4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:16.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.609+0000 7f5668d32640 1 -- 172.21.15.27:0/3483639503 >> 172.21.15.27:0/3483639503 conn(0x7f5664078b90 msgr2=0x7f56640fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:16.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.613+0000 7f5668d32640 1 -- 172.21.15.27:0/3483639503 shutdown_connections 2024-04-25T00:20:16.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:16.614+0000 7f5668d32640 1 -- 172.21.15.27:0/3483639503 wait complete. 2024-04-25T00:20:16.633 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:16 smithi027 ceph-mon[29208]: pgmap v138: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 29 MiB/s wr, 5.86k op/s 2024-04-25T00:20:16.634 INFO:tasks.cephfs.filesystem:scrub status for tag:a40f0131-2b37-4eb9-a2bd-6fee5124aebc - {'path': '/', 'tag': 'a40f0131-2b37-4eb9-a2bd-6fee5124aebc', 'options': 'recursive,force'} 2024-04-25T00:20:16.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:16 smithi153 ceph-mon[32896]: pgmap v138: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 29 MiB/s wr, 5.86k op/s 2024-04-25T00:20:17.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:16 smithi139 ceph-mon[32833]: pgmap v138: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 29 MiB/s wr, 5.86k op/s 2024-04-25T00:20:18.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:18 smithi027 ceph-mon[29208]: pgmap v139: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 31 MiB/s wr, 6.40k op/s 2024-04-25T00:20:19.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:18 smithi139 ceph-mon[32833]: pgmap v139: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 31 MiB/s wr, 6.40k op/s 2024-04-25T00:20:19.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:18 smithi153 ceph-mon[32896]: pgmap v139: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 31 MiB/s wr, 6.40k op/s 2024-04-25T00:20:20.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:20 smithi027 ceph-mon[29208]: pgmap v140: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 37 MiB/s wr, 7.72k op/s 2024-04-25T00:20:21.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:20 smithi139 ceph-mon[32833]: pgmap v140: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 37 MiB/s wr, 7.72k op/s 2024-04-25T00:20:21.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:20 smithi153 ceph-mon[32896]: pgmap v140: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 37 MiB/s wr, 7.72k op/s 2024-04-25T00:20:22.805 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:22 smithi139 ceph-mon[32833]: pgmap v141: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 27 MiB/s wr, 5.64k op/s 2024-04-25T00:20:22.828 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:22 smithi153 ceph-mon[32896]: pgmap v141: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 27 MiB/s wr, 5.64k op/s 2024-04-25T00:20:22.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:22 smithi027 ceph-mon[29208]: pgmap v141: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 27 MiB/s wr, 5.64k op/s 2024-04-25T00:20:24.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:24 smithi027 ceph-mon[29208]: pgmap v142: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 35 MiB/s wr, 7.35k op/s 2024-04-25T00:20:25.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:24 smithi139 ceph-mon[32833]: pgmap v142: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 35 MiB/s wr, 7.35k op/s 2024-04-25T00:20:25.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:24 smithi153 ceph-mon[32896]: pgmap v142: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 35 MiB/s wr, 7.35k op/s 2024-04-25T00:20:26.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:26 smithi027 ceph-mon[29208]: pgmap v143: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 33 MiB/s wr, 6.94k op/s 2024-04-25T00:20:27.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:26 smithi139 ceph-mon[32833]: pgmap v143: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 33 MiB/s wr, 6.94k op/s 2024-04-25T00:20:27.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:26 smithi153 ceph-mon[32896]: pgmap v143: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 33 MiB/s wr, 6.94k op/s 2024-04-25T00:20:28.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:28 smithi027 ceph-mon[29208]: pgmap v144: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 31 MiB/s wr, 6.48k op/s 2024-04-25T00:20:29.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:28 smithi139 ceph-mon[32833]: pgmap v144: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 31 MiB/s wr, 6.48k op/s 2024-04-25T00:20:29.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:28 smithi153 ceph-mon[32896]: pgmap v144: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 31 MiB/s wr, 6.48k op/s 2024-04-25T00:20:30.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:30 smithi027 ceph-mon[29208]: pgmap v145: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 37 MiB/s wr, 7.94k op/s 2024-04-25T00:20:31.008 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:30 smithi139 ceph-mon[32833]: pgmap v145: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 37 MiB/s wr, 7.94k op/s 2024-04-25T00:20:31.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:30 smithi153 ceph-mon[32896]: pgmap v145: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 37 MiB/s wr, 7.94k op/s 2024-04-25T00:20:32.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:32 smithi027 ceph-mon[29208]: pgmap v146: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 29 MiB/s wr, 6.22k op/s 2024-04-25T00:20:33.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:32 smithi139 ceph-mon[32833]: pgmap v146: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 29 MiB/s wr, 6.22k op/s 2024-04-25T00:20:33.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:32 smithi153 ceph-mon[32896]: pgmap v146: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 29 MiB/s wr, 6.22k op/s 2024-04-25T00:20:34.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:34 smithi027 ceph-mon[29208]: pgmap v147: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 35 MiB/s wr, 7.42k op/s 2024-04-25T00:20:35.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:34 smithi139 ceph-mon[32833]: pgmap v147: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 35 MiB/s wr, 7.42k op/s 2024-04-25T00:20:35.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:34 smithi153 ceph-mon[32896]: pgmap v147: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 35 MiB/s wr, 7.42k op/s 2024-04-25T00:20:36.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:36 smithi027 ceph-mon[29208]: pgmap v148: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 33 MiB/s wr, 7.05k op/s 2024-04-25T00:20:37.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:36 smithi139 ceph-mon[32833]: pgmap v148: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 33 MiB/s wr, 7.05k op/s 2024-04-25T00:20:37.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:36 smithi153 ceph-mon[32896]: pgmap v148: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 33 MiB/s wr, 7.05k op/s 2024-04-25T00:20:38.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:38 smithi027 ceph-mon[29208]: pgmap v149: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 30 MiB/s wr, 6.47k op/s 2024-04-25T00:20:39.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:38 smithi139 ceph-mon[32833]: pgmap v149: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 30 MiB/s wr, 6.47k op/s 2024-04-25T00:20:39.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:38 smithi153 ceph-mon[32896]: pgmap v149: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 30 MiB/s wr, 6.47k op/s 2024-04-25T00:20:39.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:39 smithi027 ceph-mon[29208]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-25T00:20:39.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:39 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:20:39.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:39 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:40.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:39 smithi139 ceph-mon[32833]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-25T00:20:40.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:39 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:20:40.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:39 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:40.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:39 smithi153 ceph-mon[32896]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-25T00:20:40.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:39 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:20:40.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:39 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:40.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:40 smithi027 ceph-mon[29208]: pgmap v150: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 36 MiB/s wr, 8.01k op/s 2024-04-25T00:20:40.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:40 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:20:40.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:40 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:41.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:40 smithi139 ceph-mon[32833]: pgmap v150: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 36 MiB/s wr, 8.01k op/s 2024-04-25T00:20:41.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:40 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:20:41.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:40 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:41.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:40 smithi153 ceph-mon[32896]: pgmap v150: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 36 MiB/s wr, 8.01k op/s 2024-04-25T00:20:41.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:40 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:20:41.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:40 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:41.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:41 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:20:41.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:41 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:42.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:41 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:20:42.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:41 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:42.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:41 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:20:42.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:41 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:42.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:42 smithi027 ceph-mon[29208]: pgmap v151: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 28 MiB/s wr, 6.20k op/s 2024-04-25T00:20:42.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:42 smithi027 ceph-mon[29208]: Health check failed: 0 slow ops, oldest one blocked for 184 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:20:43.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:42 smithi139 ceph-mon[32833]: pgmap v151: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 28 MiB/s wr, 6.20k op/s 2024-04-25T00:20:43.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:42 smithi139 ceph-mon[32833]: Health check failed: 0 slow ops, oldest one blocked for 184 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:20:43.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:42 smithi153 ceph-mon[32896]: pgmap v151: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 28 MiB/s wr, 6.20k op/s 2024-04-25T00:20:43.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:42 smithi153 ceph-mon[32896]: Health check failed: 0 slow ops, oldest one blocked for 184 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:20:43.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:43 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:20:43.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:43 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:44.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:43 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:20:44.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:43 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:44.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:43 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:20:44.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:43 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:44.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:44 smithi027 ceph-mon[29208]: pgmap v152: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 32 MiB/s wr, 7.17k op/s 2024-04-25T00:20:44.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:44 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:20:44.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:44 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:20:44.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:44 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:44 smithi139 ceph-mon[32833]: pgmap v152: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 32 MiB/s wr, 7.17k op/s 2024-04-25T00:20:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:44 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:20:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:44 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:20:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:44 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:45.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:44 smithi153 ceph-mon[32896]: pgmap v152: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 32 MiB/s wr, 7.17k op/s 2024-04-25T00:20:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:44 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:20:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:44 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:20:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:44 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:46.636 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:20:46.636 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:20:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.756+0000 7f3922aee640 1 -- 172.21.15.27:0/3838266431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f391c10bd00 msgr2=0x7f391c10e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.756+0000 7f3922aee640 1 --2- 172.21.15.27:0/3838266431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f391c10bd00 0x7f391c10e1a0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f3914009ca0 tx=0x7f391402a9e0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.756+0000 7f3922aee640 1 -- 172.21.15.27:0/3838266431 shutdown_connections 2024-04-25T00:20:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.756+0000 7f3922aee640 1 --2- 172.21.15.27:0/3838266431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f391c10bd00 0x7f391c10e1a0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.756+0000 7f3922aee640 1 --2- 172.21.15.27:0/3838266431 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f391c1032e0 0x7f391c10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.756+0000 7f3922aee640 1 --2- 172.21.15.27:0/3838266431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f391c1029c0 0x7f391c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.756+0000 7f3922aee640 1 -- 172.21.15.27:0/3838266431 >> 172.21.15.27:0/3838266431 conn(0x7f391c077740 msgr2=0x7f391c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.757+0000 7f3922aee640 1 -- 172.21.15.27:0/3838266431 shutdown_connections 2024-04-25T00:20:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.757+0000 7f3922aee640 1 -- 172.21.15.27:0/3838266431 wait complete. 2024-04-25T00:20:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.757+0000 7f3922aee640 1 Processor -- start 2024-04-25T00:20:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.757+0000 7f3922aee640 1 -- start start 2024-04-25T00:20:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.757+0000 7f3922aee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f391c1029c0 0x7f391c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.757+0000 7f3922aee640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f391c1032e0 0x7f391c198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.757+0000 7f3922aee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f391c10bd00 0x7f391c19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.757+0000 7f3922aee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f391c111b00 con 0x7f391c10bd00 2024-04-25T00:20:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.757+0000 7f3922aee640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f391c111980 con 0x7f391c1032e0 2024-04-25T00:20:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.757+0000 7f3922aee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f391c111c80 con 0x7f391c1029c0 2024-04-25T00:20:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.757+0000 7f392108b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f391c10bd00 0x7f391c19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.757+0000 7f392108b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f391c10bd00 0x7f391c19c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33416/0 (socket says 172.21.15.27:33416) 2024-04-25T00:20:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.757+0000 7f392088a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f391c1029c0 0x7f391c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.758+0000 7f392108b640 1 -- 172.21.15.27:0/3226657721 learned_addr learned my addr 172.21.15.27:0/3226657721 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:20:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.758+0000 7f391bfff640 1 --2- 172.21.15.27:0/3226657721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f391c1032e0 0x7f391c198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.758+0000 7f392108b640 1 -- 172.21.15.27:0/3226657721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f391c1029c0 msgr2=0x7f391c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.758+0000 7f392108b640 1 --2- 172.21.15.27:0/3226657721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f391c1029c0 0x7f391c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.758+0000 7f392108b640 1 -- 172.21.15.27:0/3226657721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f391c1032e0 msgr2=0x7f391c198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.758+0000 7f392108b640 1 --2- 172.21.15.27:0/3226657721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f391c1032e0 0x7f391c198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.758+0000 7f392108b640 1 -- 172.21.15.27:0/3226657721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f391c19cfb0 con 0x7f391c10bd00 2024-04-25T00:20:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.758+0000 7f392108b640 1 --2- 172.21.15.27:0/3226657721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f391c10bd00 0x7f391c19c880 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3914006010 tx=0x7f3914003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.758+0000 7f3919ffb640 1 -- 172.21.15.27:0/3226657721 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3914047070 con 0x7f391c10bd00 2024-04-25T00:20:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.758+0000 7f3922aee640 1 -- 172.21.15.27:0/3226657721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f391c19d240 con 0x7f391c10bd00 2024-04-25T00:20:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.758+0000 7f3919ffb640 1 -- 172.21.15.27:0/3226657721 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f391402fb20 con 0x7f391c10bd00 2024-04-25T00:20:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.758+0000 7f3919ffb640 1 -- 172.21.15.27:0/3226657721 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3914033050 con 0x7f391c10bd00 2024-04-25T00:20:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.758+0000 7f3922aee640 1 -- 172.21.15.27:0/3226657721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f391c1a4b30 con 0x7f391c10bd00 2024-04-25T00:20:46.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.761+0000 7f3919ffb640 1 -- 172.21.15.27:0/3226657721 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f391404a070 con 0x7f391c10bd00 2024-04-25T00:20:46.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.761+0000 7f39257fa740 1 Processor -- start 2024-04-25T00:20:46.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f39257fa740 1 -- start start 2024-04-25T00:20:46.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f39257fa740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4066530 0x556de4066910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f39257fa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4066e50 0x556de40754f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f39257fa740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4075a30 0x556de4077e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f39257fa740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556de3fcb6b0 con 0x556de4075a30 2024-04-25T00:20:46.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f39257fa740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556de3f73880 con 0x556de4066530 2024-04-25T00:20:46.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f39257fa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556de3fc7670 con 0x556de4066e50 2024-04-25T00:20:46.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f391bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4075a30 0x556de4077e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f391bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4075a30 0x556de4077e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33430/0 (socket says 172.21.15.27:33430) 2024-04-25T00:20:46.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f392088a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4066e50 0x556de40754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f391bfff640 1 -- 172.21.15.27:0/4027552756 learned_addr learned my addr 172.21.15.27:0/4027552756 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:20:46.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f391bfff640 1 --2- 172.21.15.27:0/4027552756 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4066530 0x556de4066910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f3919ffb640 1 --2- 172.21.15.27:0/3226657721 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f390c0a2d80 0x7f390c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f391bfff640 1 -- 172.21.15.27:0/4027552756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4066e50 msgr2=0x556de40754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:46.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f391bfff640 1 --2- 172.21.15.27:0/4027552756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4066e50 0x556de40754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f391bfff640 1 -- 172.21.15.27:0/4027552756 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4066530 msgr2=0x556de4066910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:46.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f391bfff640 1 --2- 172.21.15.27:0/4027552756 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4066530 0x556de4066910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f392108b640 1 --2- 172.21.15.27:0/3226657721 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f390c0a2d80 0x7f390c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f391bfff640 1 -- 172.21.15.27:0/4027552756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556de3da97d0 con 0x556de4075a30 2024-04-25T00:20:46.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f392088a640 1 --2- 172.21.15.27:0/4027552756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4066e50 0x556de40754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:20:46.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f3919ffb640 1 -- 172.21.15.27:0/3226657721 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f39140f0270 con 0x7f391c10bd00 2024-04-25T00:20:46.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.762+0000 7f391bfff640 1 --2- 172.21.15.27:0/4027552756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4075a30 0x556de4077e20 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f3908011780 tx=0x7f3908004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436f366fe702167f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:46.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.763+0000 7f3906ffd640 1 -- 172.21.15.27:0/4027552756 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3908004050 con 0x556de4075a30 2024-04-25T00:20:46.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.763+0000 7f3906ffd640 1 -- 172.21.15.27:0/4027552756 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f390800fca0 con 0x556de4075a30 2024-04-25T00:20:46.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.763+0000 7f3906ffd640 1 -- 172.21.15.27:0/4027552756 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3908003b10 con 0x556de4075a30 2024-04-25T00:20:46.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.763+0000 7f39257fa740 1 -- 172.21.15.27:0/4027552756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4075a30 msgr2=0x556de4077e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:46.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.763+0000 7f392108b640 1 --2- 172.21.15.27:0/3226657721 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f390c0a2d80 0x7f390c0a5240 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f39140bc1d0 tx=0x7f391408e490 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:46.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.763+0000 7f39257fa740 1 --2- 172.21.15.27:0/4027552756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4075a30 0x556de4077e20 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f3908011780 tx=0x7f3908004730 comp rx=0 tx=0).stop 2024-04-25T00:20:46.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.763+0000 7f39257fa740 1 -- 172.21.15.27:0/4027552756 shutdown_connections 2024-04-25T00:20:46.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.763+0000 7f39257fa740 1 --2- 172.21.15.27:0/4027552756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4075a30 0x556de4077e20 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.763+0000 7f39257fa740 1 --2- 172.21.15.27:0/4027552756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4066e50 0x556de40754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.763+0000 7f39257fa740 1 --2- 172.21.15.27:0/4027552756 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4066530 0x556de4066910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.763+0000 7f39257fa740 1 -- 172.21.15.27:0/4027552756 >> 172.21.15.27:0/4027552756 conn(0x556de4062950 msgr2=0x556de4062d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:46.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.763+0000 7f39257fa740 1 -- 172.21.15.27:0/4027552756 shutdown_connections 2024-04-25T00:20:46.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.763+0000 7f39257fa740 1 -- 172.21.15.27:0/4027552756 wait complete. 2024-04-25T00:20:46.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.764+0000 7f39257fa740 1 -- start start 2024-04-25T00:20:46.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.764+0000 7f39257fa740 1 Processor -- start 2024-04-25T00:20:46.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.764+0000 7f39257fa740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066530 0x556de40729a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.764+0000 7f39257fa740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4066e50 0x556de4072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.764+0000 7f39257fa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4075a30 0x556de4073420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.764+0000 7f39257fa740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556de40787f0 con 0x556de4066530 2024-04-25T00:20:46.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.764+0000 7f39257fa740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556de3f86f50 con 0x556de4066e50 2024-04-25T00:20:46.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.764+0000 7f39257fa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556de4078970 con 0x556de4075a30 2024-04-25T00:20:46.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.764+0000 7f39257fa740 10 client.0 _ll_register_callbacks cb 0x556de3fc2000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:20:46.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.764+0000 7f391bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066530 0x556de40729a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.764+0000 7f391bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066530 0x556de40729a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33442/0 (socket says 172.21.15.27:33442) 2024-04-25T00:20:46.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.764+0000 7f391bfff640 1 -- 172.21.15.27:0/3189977106 learned_addr learned my addr 172.21.15.27:0/3189977106 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:20:46.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.764+0000 7f392088a640 1 --2- 172.21.15.27:0/3189977106 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4066e50 0x556de4072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.764+0000 7f391bfff640 1 --2- 172.21.15.27:0/3189977106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4075a30 0x556de4073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.765+0000 7f391bfff640 1 -- 172.21.15.27:0/3189977106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4075a30 msgr2=0x556de4073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:46.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.765+0000 7f391bfff640 1 --2- 172.21.15.27:0/3189977106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4075a30 0x556de4073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.765+0000 7f391bfff640 1 -- 172.21.15.27:0/3189977106 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4066e50 msgr2=0x556de4072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:46.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.765+0000 7f391bfff640 1 --2- 172.21.15.27:0/3189977106 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4066e50 0x556de4072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.765+0000 7f391bfff640 1 -- 172.21.15.27:0/3189977106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556de3cdf3f0 con 0x556de4066530 2024-04-25T00:20:46.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.765+0000 7f391bfff640 1 --2- 172.21.15.27:0/3189977106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066530 0x556de40729a0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3908005bc0 tx=0x7f3908003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5390e4e9cd0745f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:46.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.765+0000 7f38fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:20:46.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.765+0000 7f38fbfff640 1 -- 172.21.15.27:0/3189977106 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3908023020 con 0x556de4066530 2024-04-25T00:20:46.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.765+0000 7f38fbfff640 1 -- 172.21.15.27:0/3189977106 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f3908003ce0 con 0x556de4066530 2024-04-25T00:20:46.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.765+0000 7f39257fa740 1 -- 172.21.15.27:0/3189977106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f390800fca0 con 0x556de4066530 2024-04-25T00:20:46.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.765+0000 7f38fbfff640 1 -- 172.21.15.27:0/3189977106 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3908033050 con 0x556de4066530 2024-04-25T00:20:46.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.765+0000 7f39257fa740 10 client.16485 fetch_fsmap learned FSMap version 38 2024-04-25T00:20:46.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.765+0000 7f39257fa740 1 -- 172.21.15.27:0/3189977106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556de3c9f0f0 con 0x556de4066530 2024-04-25T00:20:46.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.765+0000 7f38fbfff640 1 -- 172.21.15.27:0/3189977106 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f390800fca0 con 0x556de4066530 2024-04-25T00:20:46.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.765+0000 7f38fbfff640 1 -- 172.21.15.27:0/3189977106 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f390801f890 con 0x556de4066530 2024-04-25T00:20:46.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.766+0000 7f39257fa740 10 client.16485 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T00:20:46.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.766+0000 7f39257fa740 10 client.16485 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:20:46.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.766+0000 7f39257fa740 20 client.16485 populate_metadata read hostname 'smithi027' 2024-04-25T00:20:46.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.766+0000 7f39257fa740 1 --2- 172.21.15.27:0/3189977106 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556de4085cc0 0x556de40880b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.766+0000 7f39257fa740 4 client.16485 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:20:46.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.766+0000 7f39257fa740 1 -- 172.21.15.27:0/3189977106 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556de3fd8bf0 con 0x556de4085cc0 2024-04-25T00:20:46.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.766+0000 7f392088a640 1 --2- 172.21.15.27:0/3189977106 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556de4085cc0 0x556de40880b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.766+0000 7f38fbfff640 10 client.16485 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:20:46.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.766+0000 7f392088a640 1 --2- 172.21.15.27:0/3189977106 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556de4085cc0 0x556de40880b0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b753b21cd2796c6 server_cookie=decc3e98eb15d840 in_seq=0 out_seq=0 2024-04-25T00:20:46.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.767+0000 7f38fbfff640 1 -- 172.21.15.27:0/3189977106 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556de3fd8bf0 con 0x556de4085cc0 2024-04-25T00:20:46.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.767+0000 7f38fbfff640 10 client.16485 handle_command_reply: tid=0 2024-04-25T00:20:46.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.767+0000 7f39257fa740 1 client.16485 shutdown 2024-04-25T00:20:46.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.767+0000 7f39257fa740 1 -- 172.21.15.27:0/3189977106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066530 msgr2=0x556de40729a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:46.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.767+0000 7f39257fa740 1 --2- 172.21.15.27:0/3189977106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066530 0x556de40729a0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3908005bc0 tx=0x7f3908003730 comp rx=0 tx=0).stop 2024-04-25T00:20:46.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.767+0000 7f39257fa740 1 -- 172.21.15.27:0/3189977106 shutdown_connections 2024-04-25T00:20:46.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.767+0000 7f39257fa740 1 --2- 172.21.15.27:0/3189977106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4075a30 0x556de4073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.767+0000 7f39257fa740 1 --2- 172.21.15.27:0/3189977106 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556de4085cc0 0x556de40880b0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.767+0000 7f39257fa740 1 --2- 172.21.15.27:0/3189977106 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4066e50 0x556de4072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.767+0000 7f39257fa740 1 --2- 172.21.15.27:0/3189977106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066530 0x556de40729a0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.767+0000 7f38fbfff640 0 client.16485 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:20:46.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.767+0000 7f39257fa740 1 -- 172.21.15.27:0/3189977106 >> 172.21.15.27:0/3189977106 conn(0x556de4062950 msgr2=0x556de4068780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:46.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.767+0000 7f39257fa740 1 -- 172.21.15.27:0/3189977106 shutdown_connections 2024-04-25T00:20:46.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.767+0000 7f39257fa740 1 -- 172.21.15.27:0/3189977106 wait complete. 2024-04-25T00:20:46.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.767+0000 7f39257fa740 20 client.16485 trim_cache size 0 max 16384 2024-04-25T00:20:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.785+0000 7f39257fa740 1 Processor -- start 2024-04-25T00:20:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.785+0000 7f39257fa740 1 -- start start 2024-04-25T00:20:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.785+0000 7f39257fa740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066530 0x556de4081720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.785+0000 7f39257fa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4066e50 0x556de408d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.785+0000 7f39257fa740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4075a30 0x556de408ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.785+0000 7f39257fa740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f391c1ac960 con 0x556de4066530 2024-04-25T00:20:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.785+0000 7f39257fa740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f391c1ae4a0 con 0x556de4075a30 2024-04-25T00:20:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.785+0000 7f39257fa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f391c1ad030 con 0x556de4066e50 2024-04-25T00:20:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.785+0000 7f391bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066530 0x556de4081720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.785+0000 7f391bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066530 0x556de4081720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33450/0 (socket says 172.21.15.27:33450) 2024-04-25T00:20:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.785+0000 7f391bfff640 1 -- 172.21.15.27:0/3819255969 learned_addr learned my addr 172.21.15.27:0/3819255969 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:20:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.785+0000 7f392088a640 1 --2- 172.21.15.27:0/3819255969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4066e50 0x556de408d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.785+0000 7f391bfff640 1 -- 172.21.15.27:0/3819255969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4066e50 msgr2=0x556de408d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.785+0000 7f391bfff640 1 --2- 172.21.15.27:0/3819255969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4066e50 0x556de408d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.785+0000 7f391bfff640 1 -- 172.21.15.27:0/3819255969 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4075a30 msgr2=0x556de408ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.785+0000 7f391bfff640 1 --2- 172.21.15.27:0/3819255969 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4075a30 0x556de408ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.785+0000 7f391bfff640 1 -- 172.21.15.27:0/3819255969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556de4066c80 con 0x556de4066530 2024-04-25T00:20:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.786+0000 7f391bfff640 1 --2- 172.21.15.27:0/3819255969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066530 0x556de4081720 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f390800a010 tx=0x7f3908005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf7fc0c9bad054b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.786+0000 7f39067fc640 1 -- 172.21.15.27:0/3819255969 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3908023040 con 0x556de4066530 2024-04-25T00:20:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.786+0000 7f39067fc640 1 -- 172.21.15.27:0/3819255969 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3908009a10 con 0x556de4066530 2024-04-25T00:20:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.786+0000 7f39067fc640 1 -- 172.21.15.27:0/3819255969 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39080052b0 con 0x556de4066530 2024-04-25T00:20:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.786+0000 7f39257fa740 1 -- 172.21.15.27:0/3819255969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066530 msgr2=0x556de4081720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.786+0000 7f39257fa740 1 --2- 172.21.15.27:0/3819255969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066530 0x556de4081720 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f390800a010 tx=0x7f3908005d30 comp rx=0 tx=0).stop 2024-04-25T00:20:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.786+0000 7f39257fa740 1 -- 172.21.15.27:0/3819255969 shutdown_connections 2024-04-25T00:20:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.786+0000 7f39257fa740 1 --2- 172.21.15.27:0/3819255969 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4075a30 0x556de408ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.786+0000 7f39257fa740 1 --2- 172.21.15.27:0/3819255969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4066e50 0x556de408d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.786+0000 7f39257fa740 1 --2- 172.21.15.27:0/3819255969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066530 0x556de4081720 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.786+0000 7f39257fa740 1 -- 172.21.15.27:0/3819255969 >> 172.21.15.27:0/3819255969 conn(0x556de4062950 msgr2=0x556de405da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.786+0000 7f39257fa740 1 -- 172.21.15.27:0/3819255969 shutdown_connections 2024-04-25T00:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.786+0000 7f39257fa740 1 -- 172.21.15.27:0/3819255969 wait complete. 2024-04-25T00:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.787+0000 7f39257fa740 1 -- start start 2024-04-25T00:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.787+0000 7f39257fa740 1 Processor -- start 2024-04-25T00:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.787+0000 7f39257fa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4066530 0x556de407dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.787+0000 7f39257fa740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066e50 0x556de4079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.787+0000 7f39257fa740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4075a30 0x556de4079750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.787+0000 7f39257fa740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f391c1addd0 con 0x556de4066e50 2024-04-25T00:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.787+0000 7f39257fa740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f391c1ad700 con 0x556de4075a30 2024-04-25T00:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.787+0000 7f39257fa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f391c1aa920 con 0x556de4066530 2024-04-25T00:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.787+0000 7f39257fa740 10 client.0 _ll_register_callbacks cb 0x7f3908009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.787+0000 7f392088a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066e50 0x556de4079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.787+0000 7f392088a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066e50 0x556de4079210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33466/0 (socket says 172.21.15.27:33466) 2024-04-25T00:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.787+0000 7f392088a640 1 -- 172.21.15.27:0/1577826822 learned_addr learned my addr 172.21.15.27:0/1577826822 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.787+0000 7f391bfff640 1 --2- 172.21.15.27:0/1577826822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4075a30 0x556de4079750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.788+0000 7f391bfff640 1 --2- 172.21.15.27:0/1577826822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4066530 0x556de407dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.788+0000 7f392088a640 1 -- 172.21.15.27:0/1577826822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4066530 msgr2=0x556de407dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.788+0000 7f392088a640 1 --2- 172.21.15.27:0/1577826822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4066530 0x556de407dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.788+0000 7f392088a640 1 -- 172.21.15.27:0/1577826822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4075a30 msgr2=0x556de4079750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.788+0000 7f392088a640 1 --2- 172.21.15.27:0/1577826822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4075a30 0x556de4079750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.788+0000 7f392088a640 1 -- 172.21.15.27:0/1577826822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556de4088420 con 0x556de4066e50 2024-04-25T00:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.788+0000 7f392088a640 1 --2- 172.21.15.27:0/1577826822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066e50 0x556de4079210 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f391000b8a0 tx=0x7f391000be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555ad3b22696a59a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.788+0000 7f38fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.788+0000 7f38fbfff640 1 -- 172.21.15.27:0/1577826822 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3910020360 con 0x556de4066e50 2024-04-25T00:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.788+0000 7f38fbfff640 1 -- 172.21.15.27:0/1577826822 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f3910020500 con 0x556de4066e50 2024-04-25T00:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.788+0000 7f39257fa740 1 -- 172.21.15.27:0/1577826822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3908009a10 con 0x556de4066e50 2024-04-25T00:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.788+0000 7f38fbfff640 1 -- 172.21.15.27:0/1577826822 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3910030050 con 0x556de4066e50 2024-04-25T00:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.788+0000 7f39257fa740 10 client.16491 fetch_fsmap learned FSMap version 38 2024-04-25T00:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.788+0000 7f39257fa740 1 -- 172.21.15.27:0/1577826822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556de40ebc70 con 0x556de4066e50 2024-04-25T00:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.788+0000 7f38fbfff640 1 -- 172.21.15.27:0/1577826822 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3908009a10 con 0x556de4066e50 2024-04-25T00:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.788+0000 7f38fbfff640 1 -- 172.21.15.27:0/1577826822 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f3910011c20 con 0x556de4066e50 2024-04-25T00:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f39257fa740 10 client.16491 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T00:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f39257fa740 10 client.16491 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f39257fa740 20 client.16491 populate_metadata read hostname 'smithi027' 2024-04-25T00:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f39257fa740 1 --2- 172.21.15.27:0/1577826822 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556de40ec3a0 0x556de40ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f39257fa740 4 client.16491 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f39257fa740 1 -- 172.21.15.27:0/1577826822 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556de3fd2f40 con 0x556de40ec3a0 2024-04-25T00:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f391bfff640 1 --2- 172.21.15.27:0/1577826822 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556de40ec3a0 0x556de40ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f38fbfff640 10 client.16491 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f391bfff640 1 --2- 172.21.15.27:0/1577826822 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556de40ec3a0 0x556de40ee7c0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a7b24222b18837a server_cookie=807e1fbf2eb6398e in_seq=0 out_seq=0 2024-04-25T00:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f38fbfff640 1 -- 172.21.15.27:0/1577826822 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x556de3fd2f40 con 0x556de40ec3a0 2024-04-25T00:20:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f38fbfff640 10 client.16491 handle_command_reply: tid=0 2024-04-25T00:20:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f39257fa740 1 client.16491 shutdown 2024-04-25T00:20:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f39257fa740 1 -- 172.21.15.27:0/1577826822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066e50 msgr2=0x556de4079210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f39257fa740 1 --2- 172.21.15.27:0/1577826822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066e50 0x556de4079210 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f391000b8a0 tx=0x7f391000be40 comp rx=0 tx=0).stop 2024-04-25T00:20:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f39257fa740 1 -- 172.21.15.27:0/1577826822 shutdown_connections 2024-04-25T00:20:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f39257fa740 1 --2- 172.21.15.27:0/1577826822 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556de40ec3a0 0x556de40ee7c0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f39257fa740 1 --2- 172.21.15.27:0/1577826822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556de4075a30 0x556de4079750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f39257fa740 1 --2- 172.21.15.27:0/1577826822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556de4066e50 0x556de4079210 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f38fbfff640 0 client.16491 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:20:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f39257fa740 1 --2- 172.21.15.27:0/1577826822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556de4066530 0x556de407dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.789+0000 7f39257fa740 1 -- 172.21.15.27:0/1577826822 >> 172.21.15.27:0/1577826822 conn(0x556de4062950 msgr2=0x556de4061260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.790+0000 7f39257fa740 1 -- 172.21.15.27:0/1577826822 shutdown_connections 2024-04-25T00:20:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.790+0000 7f39257fa740 1 -- 172.21.15.27:0/1577826822 wait complete. 2024-04-25T00:20:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.790+0000 7f39257fa740 20 client.16491 trim_cache size 0 max 16384 2024-04-25T00:20:46.799 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:20:46.799 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "scrub active (201 inodes in the stack)", 2024-04-25T00:20:46.799 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:20:46.799 INFO:teuthology.orchestra.run.smithi027.stdout: "a40f0131-2b37-4eb9-a2bd-6fee5124aebc": { 2024-04-25T00:20:46.799 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:20:46.799 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "a40f0131-2b37-4eb9-a2bd-6fee5124aebc", 2024-04-25T00:20:46.800 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:20:46.800 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:20:46.800 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:20:46.800 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:20:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.790+0000 7f3922aee640 1 -- 172.21.15.27:0/3226657721 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f390c0a2d80 msgr2=0x7f390c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.790+0000 7f3922aee640 1 --2- 172.21.15.27:0/3226657721 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f390c0a2d80 0x7f390c0a5240 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f39140bc1d0 tx=0x7f391408e490 comp rx=0 tx=0).stop 2024-04-25T00:20:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.790+0000 7f3922aee640 1 -- 172.21.15.27:0/3226657721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f391c10bd00 msgr2=0x7f391c19c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.790+0000 7f3922aee640 1 --2- 172.21.15.27:0/3226657721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f391c10bd00 0x7f391c19c880 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3914006010 tx=0x7f3914003ba0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.790+0000 7f3922aee640 1 -- 172.21.15.27:0/3226657721 shutdown_connections 2024-04-25T00:20:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.790+0000 7f3922aee640 1 --2- 172.21.15.27:0/3226657721 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f390c0a2d80 0x7f390c0a5240 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.790+0000 7f3922aee640 1 --2- 172.21.15.27:0/3226657721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f391c10bd00 0x7f391c19c880 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.790+0000 7f3922aee640 1 --2- 172.21.15.27:0/3226657721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f391c1032e0 0x7f391c198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.790+0000 7f3922aee640 1 --2- 172.21.15.27:0/3226657721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f391c1029c0 0x7f391c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.790+0000 7f3922aee640 1 -- 172.21.15.27:0/3226657721 >> 172.21.15.27:0/3226657721 conn(0x7f391c077740 msgr2=0x7f391c0fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.790+0000 7f3922aee640 1 -- 172.21.15.27:0/3226657721 shutdown_connections 2024-04-25T00:20:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:20:46.790+0000 7f3922aee640 1 -- 172.21.15.27:0/3226657721 wait complete. 2024-04-25T00:20:46.803 INFO:tasks.cephfs.filesystem:scrub status for tag:a40f0131-2b37-4eb9-a2bd-6fee5124aebc - {'path': '/', 'tag': 'a40f0131-2b37-4eb9-a2bd-6fee5124aebc', 'options': 'recursive,force'} 2024-04-25T00:20:46.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:46 smithi027 ceph-mon[29208]: pgmap v153: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 31 MiB/s wr, 6.96k op/s 2024-04-25T00:20:46.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:46 smithi027 ceph-mon[29208]: Health check update: 3 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-25T00:20:47.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:46 smithi139 ceph-mon[32833]: pgmap v153: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 31 MiB/s wr, 6.96k op/s 2024-04-25T00:20:47.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:46 smithi139 ceph-mon[32833]: Health check update: 3 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-25T00:20:47.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:46 smithi153 ceph-mon[32896]: pgmap v153: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 31 MiB/s wr, 6.96k op/s 2024-04-25T00:20:47.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:46 smithi153 ceph-mon[32896]: Health check update: 3 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-25T00:20:48.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:47 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 189 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:20:48.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:47 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:20:48.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:47 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:48.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:47 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 189 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:20:48.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:47 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:20:48.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:47 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:48.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:47 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 189 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:20:48.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:47 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:20:48.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:47 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:49.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:48 smithi139 ceph-mon[32833]: pgmap v154: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 28 MiB/s wr, 6.29k op/s 2024-04-25T00:20:49.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:48 smithi153 ceph-mon[32896]: pgmap v154: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 28 MiB/s wr, 6.29k op/s 2024-04-25T00:20:49.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:48 smithi027 ceph-mon[29208]: pgmap v154: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 28 MiB/s wr, 6.29k op/s 2024-04-25T00:20:51.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:50 smithi139 ceph-mon[32833]: pgmap v155: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 25 MiB/s wr, 5.87k op/s 2024-04-25T00:20:51.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:50 smithi153 ceph-mon[32896]: pgmap v155: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 25 MiB/s wr, 5.87k op/s 2024-04-25T00:20:51.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:50 smithi027 ceph-mon[29208]: pgmap v155: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 25 MiB/s wr, 5.87k op/s 2024-04-25T00:20:52.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:51 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:20:52.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:51 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:52.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:51 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:20:52.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:51 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:52.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:51 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:20:52.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:51 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:53.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:52 smithi139 ceph-mon[32833]: pgmap v156: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 21 MiB/s wr, 4.82k op/s 2024-04-25T00:20:53.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:52 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:20:53.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:52 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:53.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:52 smithi153 ceph-mon[32896]: pgmap v156: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 21 MiB/s wr, 4.82k op/s 2024-04-25T00:20:53.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:52 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:20:53.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:52 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:53.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:52 smithi027 ceph-mon[29208]: pgmap v156: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 21 MiB/s wr, 4.82k op/s 2024-04-25T00:20:53.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:52 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:20:53.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:52 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:55.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:54 smithi139 ceph-mon[32833]: pgmap v157: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 21 MiB/s wr, 4.96k op/s 2024-04-25T00:20:55.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:54 smithi153 ceph-mon[32896]: pgmap v157: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 21 MiB/s wr, 4.96k op/s 2024-04-25T00:20:55.043 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:54 smithi027 ceph-mon[29208]: pgmap v157: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 21 MiB/s wr, 4.96k op/s 2024-04-25T00:20:56.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:55 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:20:56.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:55 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:56.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:55 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 194 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:20:56.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:55 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:20:56.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:55 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:56.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:55 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 194 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:20:56.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:55 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:20:56.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:55 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:56.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:55 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 194 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:20:57.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:56 smithi139 ceph-mon[32833]: pgmap v158: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 16 MiB/s wr, 3.69k op/s 2024-04-25T00:20:57.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:56 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:20:57.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:56 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:20:57.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:56 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:57.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:56 smithi153 ceph-mon[32896]: pgmap v158: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 16 MiB/s wr, 3.69k op/s 2024-04-25T00:20:57.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:56 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:20:57.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:56 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:20:57.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:56 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:57.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:56 smithi027 ceph-mon[29208]: pgmap v158: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 16 MiB/s wr, 3.69k op/s 2024-04-25T00:20:57.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:56 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:20:57.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:56 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:20:57.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:56 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:20:59.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:58 smithi153 ceph-mon[32896]: pgmap v159: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 16 MiB/s wr, 3.68k op/s 2024-04-25T00:20:59.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:58 smithi027 ceph-mon[29208]: pgmap v159: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 16 MiB/s wr, 3.68k op/s 2024-04-25T00:20:59.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:58 smithi139 ceph-mon[32833]: pgmap v159: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 16 MiB/s wr, 3.68k op/s 2024-04-25T00:21:00.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:59 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:00.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:20:59 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:00.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:59 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:00.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:20:59 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:00.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:59 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:00.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:20:59 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:01.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:00 smithi153 ceph-mon[32896]: pgmap v160: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 14 MiB/s wr, 3.29k op/s 2024-04-25T00:21:01.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:00 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:01.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:00 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:01.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:00 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:01.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:00 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 199 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:01.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:00 smithi027 ceph-mon[29208]: pgmap v160: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 14 MiB/s wr, 3.29k op/s 2024-04-25T00:21:01.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:00 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:01.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:00 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:01.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:00 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:01.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:00 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 199 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:01.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:00 smithi139 ceph-mon[32833]: pgmap v160: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 14 MiB/s wr, 3.29k op/s 2024-04-25T00:21:01.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:00 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:01.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:00 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:01.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:00 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:01.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:00 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 199 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:03.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:02 smithi153 ceph-mon[32896]: pgmap v161: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 15 MiB/s wr, 3.53k op/s 2024-04-25T00:21:03.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:02 smithi027 ceph-mon[29208]: pgmap v161: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 15 MiB/s wr, 3.53k op/s 2024-04-25T00:21:03.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:02 smithi139 ceph-mon[32833]: pgmap v161: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 15 MiB/s wr, 3.53k op/s 2024-04-25T00:21:04.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:03 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:04.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:03 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:04.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:03 smithi027 ceph-mon[29208]: 1 slow requests, 1 included below; oldest blocked for > 183.214530 secs 2024-04-25T00:21:04.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:03 smithi027 ceph-mon[29208]: slow request 183.214530 seconds old, received at 2024-04-25T00:18:00.173433+0000: client_request(client.15834:16 mksnap #0x1//scheduled-2024-04-25-00_18_00_UTC 2024-04-25T00:18:00.173209+0000 caller_uid=0, caller_gid=0{}) currently acquired locks 2024-04-25T00:21:04.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:03 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:21:04.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:03 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:04.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:03 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:04.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:03 smithi139 ceph-mon[32833]: 1 slow requests, 1 included below; oldest blocked for > 183.214530 secs 2024-04-25T00:21:04.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:03 smithi139 ceph-mon[32833]: slow request 183.214530 seconds old, received at 2024-04-25T00:18:00.173433+0000: client_request(client.15834:16 mksnap #0x1//scheduled-2024-04-25-00_18_00_UTC 2024-04-25T00:18:00.173209+0000 caller_uid=0, caller_gid=0{}) currently acquired locks 2024-04-25T00:21:04.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:03 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:21:04.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:03 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:04.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:03 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:04.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:03 smithi153 ceph-mon[32896]: 1 slow requests, 1 included below; oldest blocked for > 183.214530 secs 2024-04-25T00:21:04.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:03 smithi153 ceph-mon[32896]: slow request 183.214530 seconds old, received at 2024-04-25T00:18:00.173433+0000: client_request(client.15834:16 mksnap #0x1//scheduled-2024-04-25-00_18_00_UTC 2024-04-25T00:18:00.173209+0000 caller_uid=0, caller_gid=0{}) currently acquired locks 2024-04-25T00:21:04.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:03 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:21:05.182 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:04 smithi027 ceph-mon[29208]: pgmap v162: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 14 MiB/s wr, 3.38k op/s 2024-04-25T00:21:05.183 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:04 smithi027 ceph-mon[29208]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-04-25T00:21:05.183 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:04 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:05.183 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:04 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:05.183 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:04 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:05.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:04 smithi139 ceph-mon[32833]: pgmap v162: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 14 MiB/s wr, 3.38k op/s 2024-04-25T00:21:05.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:04 smithi139 ceph-mon[32833]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-04-25T00:21:05.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:04 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:05.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:04 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:05.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:04 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:05.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:04 smithi153 ceph-mon[32896]: pgmap v162: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 14 MiB/s wr, 3.38k op/s 2024-04-25T00:21:05.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:04 smithi153 ceph-mon[32896]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-04-25T00:21:05.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:04 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:05.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:04 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:05.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:04 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:06.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:05 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 204 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:06.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:05 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 204 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:06.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:05 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 204 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:07.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:06 smithi027 ceph-mon[29208]: pgmap v163: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 16 MiB/s wr, 3.88k op/s 2024-04-25T00:21:07.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:06 smithi139 ceph-mon[32833]: pgmap v163: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 16 MiB/s wr, 3.88k op/s 2024-04-25T00:21:07.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:06 smithi153 ceph-mon[32896]: pgmap v163: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 16 MiB/s wr, 3.88k op/s 2024-04-25T00:21:09.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:08 smithi027 ceph-mon[29208]: pgmap v164: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 4.31k op/s 2024-04-25T00:21:09.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:08 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:09.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:08 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:09.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:08 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 188.214588 secs 2024-04-25T00:21:09.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:08 smithi139 ceph-mon[32833]: pgmap v164: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 4.31k op/s 2024-04-25T00:21:09.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:08 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:09.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:08 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:09.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:08 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 188.214588 secs 2024-04-25T00:21:09.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:08 smithi153 ceph-mon[32896]: pgmap v164: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 4.31k op/s 2024-04-25T00:21:09.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:08 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:09.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:08 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:09.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:08 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 188.214588 secs 2024-04-25T00:21:11.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:10 smithi027 ceph-mon[29208]: pgmap v165: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 15 MiB/s wr, 3.83k op/s 2024-04-25T00:21:11.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:10 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:21:11.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:10 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:21:11.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:10 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:21:11.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:10 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 209 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:11.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:10 smithi139 ceph-mon[32833]: pgmap v165: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 15 MiB/s wr, 3.83k op/s 2024-04-25T00:21:11.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:10 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:21:11.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:10 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:21:11.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:10 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:21:11.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:10 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 209 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:11.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:10 smithi153 ceph-mon[32896]: pgmap v165: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 15 MiB/s wr, 3.83k op/s 2024-04-25T00:21:11.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:10 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:21:11.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:10 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:21:11.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:10 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:21:11.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:10 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 209 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:12.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:11 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:12.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:11 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:12.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:11 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:12.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:11 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:12.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:11 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:12.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:11 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:13.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:12 smithi027 ceph-mon[29208]: pgmap v166: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 18 MiB/s wr, 4.37k op/s 2024-04-25T00:21:13.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:12 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:13.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:12 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:13.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:12 smithi139 ceph-mon[32833]: pgmap v166: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 18 MiB/s wr, 4.37k op/s 2024-04-25T00:21:13.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:12 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:13.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:12 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:13.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:12 smithi153 ceph-mon[32896]: pgmap v166: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 18 MiB/s wr, 4.37k op/s 2024-04-25T00:21:13.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:12 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:13.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:12 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:14.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:13 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 193.214662 secs 2024-04-25T00:21:14.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:13 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 193.214662 secs 2024-04-25T00:21:14.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:13 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 193.214662 secs 2024-04-25T00:21:15.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:14 smithi027 ceph-mon[29208]: pgmap v167: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 19 MiB/s wr, 4.76k op/s 2024-04-25T00:21:15.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:14 smithi139 ceph-mon[32833]: pgmap v167: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 19 MiB/s wr, 4.76k op/s 2024-04-25T00:21:15.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:14 smithi153 ceph-mon[32896]: pgmap v167: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 19 MiB/s wr, 4.76k op/s 2024-04-25T00:21:16.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:15 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:16.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:15 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:16.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:15 smithi027 ceph-mon[29208]: pgmap v168: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 19 MiB/s wr, 4.65k op/s 2024-04-25T00:21:16.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:15 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 214 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:16.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:15 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:16.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:15 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:16.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:15 smithi139 ceph-mon[32833]: pgmap v168: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 19 MiB/s wr, 4.65k op/s 2024-04-25T00:21:16.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:15 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 214 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:16.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:15 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:16.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:15 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:16.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:15 smithi153 ceph-mon[32896]: pgmap v168: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 19 MiB/s wr, 4.65k op/s 2024-04-25T00:21:16.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:15 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 214 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:16.804 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:21:16.804 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:21:17.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.004+0000 7f53aa9f9640 1 -- 172.21.15.27:0/2284806802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a410ff90 msgr2=0x7f53a410da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:17.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.004+0000 7f53aa9f9640 1 --2- 172.21.15.27:0/2284806802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a410ff90 0x7f53a410da80 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f539401a940 tx=0x7f539403b490 comp rx=0 tx=0).stop 2024-04-25T00:21:17.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.004+0000 7f53aa9f9640 1 -- 172.21.15.27:0/2284806802 shutdown_connections 2024-04-25T00:21:17.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.004+0000 7f53aa9f9640 1 --2- 172.21.15.27:0/2284806802 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53a411af20 0x7f53a411d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.004+0000 7f53aa9f9640 1 --2- 172.21.15.27:0/2284806802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a410dfc0 0x7f53a410e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.004+0000 7f53aa9f9640 1 --2- 172.21.15.27:0/2284806802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a410ff90 0x7f53a410da80 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.004+0000 7f53aa9f9640 1 -- 172.21.15.27:0/2284806802 >> 172.21.15.27:0/2284806802 conn(0x7f53a406c850 msgr2=0x7f53a406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:17.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.006+0000 7f53aa9f9640 1 -- 172.21.15.27:0/2284806802 shutdown_connections 2024-04-25T00:21:17.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.007+0000 7f53aa9f9640 1 -- 172.21.15.27:0/2284806802 wait complete. 2024-04-25T00:21:17.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.008+0000 7f53aa9f9640 1 Processor -- start 2024-04-25T00:21:17.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.009+0000 7f53aa9f9640 1 -- start start 2024-04-25T00:21:17.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.009+0000 7f53aa9f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a410dfc0 0x7f53a411a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.009+0000 7f53aa9f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a410ff90 0x7f53a4116890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.009+0000 7f53aa9f9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53a411af20 0x7f53a4116dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.009+0000 7f53aa9f9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53a41131a0 con 0x7f53a410ff90 2024-04-25T00:21:17.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.009+0000 7f53aa9f9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f53a4113020 con 0x7f53a411af20 2024-04-25T00:21:17.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.009+0000 7f53aa9f9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53a4113320 con 0x7f53a410dfc0 2024-04-25T00:21:17.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.010+0000 7f53a37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a410ff90 0x7f53a4116890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:17.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.010+0000 7f53a37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a410ff90 0x7f53a4116890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34026/0 (socket says 172.21.15.27:34026) 2024-04-25T00:21:17.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.010+0000 7f53a37fe640 1 -- 172.21.15.27:0/1998372159 learned_addr learned my addr 172.21.15.27:0/1998372159 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:21:17.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.010+0000 7f53a3fff640 1 --2- 172.21.15.27:0/1998372159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a410dfc0 0x7f53a411a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:17.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.012+0000 7f53a37fe640 1 -- 172.21.15.27:0/1998372159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a410dfc0 msgr2=0x7f53a411a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:17.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.012+0000 7f53a37fe640 1 --2- 172.21.15.27:0/1998372159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a410dfc0 0x7f53a411a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.012+0000 7f53a37fe640 1 -- 172.21.15.27:0/1998372159 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53a411af20 msgr2=0x7f53a4116dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:17.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.012+0000 7f53a37fe640 1 --2- 172.21.15.27:0/1998372159 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53a411af20 0x7f53a4116dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.012+0000 7f53a37fe640 1 -- 172.21.15.27:0/1998372159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a4117690 con 0x7f53a410ff90 2024-04-25T00:21:17.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.012+0000 7f53a37fe640 1 --2- 172.21.15.27:0/1998372159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a410ff90 0x7f53a4116890 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f539800b450 tx=0x7f539800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:17.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.014+0000 7f53a17fa640 1 -- 172.21.15.27:0/1998372159 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398015020 con 0x7f53a410ff90 2024-04-25T00:21:17.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.014+0000 7f53a17fa640 1 -- 172.21.15.27:0/1998372159 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5398011d70 con 0x7f53a410ff90 2024-04-25T00:21:17.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.014+0000 7f53a17fa640 1 -- 172.21.15.27:0/1998372159 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539800f340 con 0x7f53a410ff90 2024-04-25T00:21:17.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.014+0000 7f53aa9f9640 1 -- 172.21.15.27:0/1998372159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a4079a60 con 0x7f53a410ff90 2024-04-25T00:21:17.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.014+0000 7f53aa9f9640 1 -- 172.21.15.27:0/1998372159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a4079f60 con 0x7f53a410ff90 2024-04-25T00:21:17.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.018+0000 7f53ad7e6740 1 Processor -- start 2024-04-25T00:21:17.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.020+0000 7f53a17fa640 1 -- 172.21.15.27:0/1998372159 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5398018020 con 0x7f53a410ff90 2024-04-25T00:21:17.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.022+0000 7f53ad7e6740 1 -- start start 2024-04-25T00:21:17.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.022+0000 7f53ad7e6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc7f630 0x56292cc7fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.022+0000 7f53ad7e6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc7ff50 0x56292cc8a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.022+0000 7f53ad7e6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc8a620 0x56292cc8ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.022+0000 7f53ad7e6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56292cbe46b0 con 0x56292cc8a620 2024-04-25T00:21:17.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.022+0000 7f53ad7e6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56292cb8c880 con 0x56292cc7ff50 2024-04-25T00:21:17.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.022+0000 7f53ad7e6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56292cbe0670 con 0x56292cc7f630 2024-04-25T00:21:17.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.023+0000 7f53a3fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc7ff50 0x56292cc8a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.023+0000 7f53a3fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc7ff50 0x56292cc8a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:51888/0 (socket says 172.21.15.27:51888) 2024-04-25T00:21:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.023+0000 7f53a3fff640 1 -- 172.21.15.27:0/4070001145 learned_addr learned my addr 172.21.15.27:0/4070001145 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:21:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.024+0000 7f53a17fa640 1 --2- 172.21.15.27:0/1998372159 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f538c0a2f00 0x7f538c0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.024+0000 7f53a17fa640 1 -- 172.21.15.27:0/1998372159 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f53980cf130 con 0x7f53a410ff90 2024-04-25T00:21:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.025+0000 7f53a37fe640 1 --2- 172.21.15.27:0/4070001145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc8a620 0x56292cc8ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.025+0000 7f53a3fff640 1 -- 172.21.15.27:0/4070001145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc7f630 msgr2=0x56292cc7fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.025+0000 7f53a3fff640 1 --2- 172.21.15.27:0/4070001145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc7f630 0x56292cc7fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.026+0000 7f53a3fff640 1 -- 172.21.15.27:0/4070001145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc8a620 msgr2=0x56292cc8ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.026+0000 7f53a3fff640 1 --2- 172.21.15.27:0/4070001145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc8a620 0x56292cc8ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.026+0000 7f53a3fff640 1 -- 172.21.15.27:0/4070001145 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56292c9c27d0 con 0x56292cc7ff50 2024-04-25T00:21:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.027+0000 7f53a8f96640 1 --2- 172.21.15.27:0/1998372159 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f538c0a2f00 0x7f538c0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.027+0000 7f53a3fff640 1 --2- 172.21.15.27:0/4070001145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc7ff50 0x56292cc8a0e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f539401a940 tx=0x7f5394005200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b03aa43cf317806 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.028+0000 7f538a7fc640 1 -- 172.21.15.27:0/4070001145 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5394046030 con 0x56292cc7ff50 2024-04-25T00:21:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.028+0000 7f538a7fc640 1 -- 172.21.15.27:0/4070001145 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5394004ba0 con 0x56292cc7ff50 2024-04-25T00:21:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.028+0000 7f538a7fc640 1 -- 172.21.15.27:0/4070001145 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5394040c60 con 0x56292cc7ff50 2024-04-25T00:21:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.030+0000 7f53a8f96640 1 --2- 172.21.15.27:0/1998372159 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f538c0a2f00 0x7f538c0a53c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f53a4072910 tx=0x7f539c001710 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:17.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.031+0000 7f53ad7e6740 1 -- 172.21.15.27:0/4070001145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc7ff50 msgr2=0x56292cc8a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:17.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.031+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/4070001145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc7ff50 0x56292cc8a0e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f539401a940 tx=0x7f5394005200 comp rx=0 tx=0).stop 2024-04-25T00:21:17.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.034+0000 7f53ad7e6740 1 -- 172.21.15.27:0/4070001145 shutdown_connections 2024-04-25T00:21:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.034+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/4070001145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc8a620 0x56292cc8ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.034+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/4070001145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc7ff50 0x56292cc8a0e0 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f539401a940 tx=0x7f5394005200 comp rx=0 tx=0).stop 2024-04-25T00:21:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.034+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/4070001145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc7f630 0x56292cc7fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.034+0000 7f53ad7e6740 1 -- 172.21.15.27:0/4070001145 >> 172.21.15.27:0/4070001145 conn(0x56292cc7b950 msgr2=0x56292cc7bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:17.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.036+0000 7f53ad7e6740 1 -- 172.21.15.27:0/4070001145 shutdown_connections 2024-04-25T00:21:17.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.039+0000 7f53ad7e6740 1 -- 172.21.15.27:0/4070001145 wait complete. 2024-04-25T00:21:17.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.040+0000 7f53ad7e6740 1 -- start start 2024-04-25T00:21:17.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.040+0000 7f53ad7e6740 1 Processor -- start 2024-04-25T00:21:17.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.041+0000 7f53ad7e6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc7f630 0x56292cc876c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.041+0000 7f53ad7e6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc8a620 0x56292cc87c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.041+0000 7f53ad7e6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc88140 0x56292cc968b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.041+0000 7f53ad7e6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56292cc8d3b0 con 0x56292cc7f630 2024-04-25T00:21:17.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.041+0000 7f53ad7e6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56292cb9ff50 con 0x56292cc88140 2024-04-25T00:21:17.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.041+0000 7f53ad7e6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56292cc8d530 con 0x56292cc8a620 2024-04-25T00:21:17.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.041+0000 7f53ad7e6740 10 client.0 _ll_register_callbacks cb 0x56292cbdb000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:21:17.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.042+0000 7f53a3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc7f630 0x56292cc876c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:17.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.042+0000 7f53a3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc7f630 0x56292cc876c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34044/0 (socket says 172.21.15.27:34044) 2024-04-25T00:21:17.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.042+0000 7f53a3fff640 1 -- 172.21.15.27:0/456644994 learned_addr learned my addr 172.21.15.27:0/456644994 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:21:17.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.042+0000 7f53a3fff640 1 -- 172.21.15.27:0/456644994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc8a620 msgr2=0x56292cc87c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:17.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.042+0000 7f53a37fe640 1 --2- 172.21.15.27:0/456644994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc8a620 0x56292cc87c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:17.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.042+0000 7f53a3fff640 1 --2- 172.21.15.27:0/456644994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc8a620 0x56292cc87c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.042+0000 7f53a3fff640 1 -- 172.21.15.27:0/456644994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc88140 msgr2=0x56292cc968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:17.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.042+0000 7f53a3fff640 1 --2- 172.21.15.27:0/456644994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc88140 0x56292cc968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.042+0000 7f53a3fff640 1 -- 172.21.15.27:0/456644994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56292c8f83f0 con 0x56292cc7f630 2024-04-25T00:21:17.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.043+0000 7f53a3fff640 1 --2- 172.21.15.27:0/456644994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc7f630 0x56292cc876c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f5394005ab0 tx=0x7f5394005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ef2aff4425f531 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:17.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.044+0000 7f53837fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:21:17.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.044+0000 7f53837fe640 1 -- 172.21.15.27:0/456644994 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5394004d50 con 0x56292cc7f630 2024-04-25T00:21:17.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.045+0000 7f53837fe640 1 -- 172.21.15.27:0/456644994 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f5394019040 con 0x56292cc7f630 2024-04-25T00:21:17.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.045+0000 7f53ad7e6740 1 -- 172.21.15.27:0/456644994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5394004ba0 con 0x56292cc7f630 2024-04-25T00:21:17.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.047+0000 7f53837fe640 1 -- 172.21.15.27:0/456644994 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f53940046c0 con 0x56292cc7f630 2024-04-25T00:21:17.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.047+0000 7f53ad7e6740 10 client.16500 fetch_fsmap learned FSMap version 52 2024-04-25T00:21:17.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.047+0000 7f53ad7e6740 1 -- 172.21.15.27:0/456644994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x56292c8b80f0 con 0x56292cc7f630 2024-04-25T00:21:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.048+0000 7f53837fe640 1 -- 172.21.15.27:0/456644994 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5394004ba0 con 0x56292cc7f630 2024-04-25T00:21:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.048+0000 7f53837fe640 1 -- 172.21.15.27:0/456644994 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 52) ==== 8362+0+0 (secure 0 0 0) 0x7f53940035f0 con 0x56292cc7f630 2024-04-25T00:21:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.048+0000 7f53ad7e6740 10 client.16500 fetch_fsmap finished waiting for FSMap version 52 2024-04-25T00:21:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.048+0000 7f53ad7e6740 10 client.16500 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:21:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.048+0000 7f53ad7e6740 20 client.16500 populate_metadata read hostname 'smithi027' 2024-04-25T00:21:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.048+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/456644994 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56292cc9aef0 0x56292cc9d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.048+0000 7f53ad7e6740 4 client.16500 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:21:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.048+0000 7f53ad7e6740 1 -- 172.21.15.27:0/456644994 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56292cbf1bf0 con 0x56292cc9aef0 2024-04-25T00:21:17.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.049+0000 7f53a37fe640 1 --2- 172.21.15.27:0/456644994 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56292cc9aef0 0x56292cc9d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:17.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.052+0000 7f53a37fe640 1 --2- 172.21.15.27:0/456644994 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56292cc9aef0 0x56292cc9d370 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5d94a9981ff88b7 server_cookie=6d601382c7c323a8 in_seq=0 out_seq=0 2024-04-25T00:21:17.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.052+0000 7f53837fe640 10 client.16500 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:21:17.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.053+0000 7f53837fe640 1 -- 172.21.15.27:0/456644994 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56292cbf1bf0 con 0x56292cc9aef0 2024-04-25T00:21:17.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.053+0000 7f53837fe640 10 client.16500 handle_command_reply: tid=0 2024-04-25T00:21:17.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.053+0000 7f53ad7e6740 1 client.16500 shutdown 2024-04-25T00:21:17.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.055+0000 7f53ad7e6740 1 -- 172.21.15.27:0/456644994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc7f630 msgr2=0x56292cc876c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:17.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.055+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/456644994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc7f630 0x56292cc876c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f5394005ab0 tx=0x7f5394005020 comp rx=0 tx=0).stop 2024-04-25T00:21:17.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.058+0000 7f53ad7e6740 1 -- 172.21.15.27:0/456644994 shutdown_connections 2024-04-25T00:21:17.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.058+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/456644994 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56292cc9aef0 0x56292cc9d370 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.058+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/456644994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc88140 0x56292cc968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.058+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/456644994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc8a620 0x56292cc87c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.058+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/456644994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc7f630 0x56292cc876c0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.058+0000 7f53ad7e6740 1 -- 172.21.15.27:0/456644994 >> 172.21.15.27:0/456644994 conn(0x56292cc7b950 msgr2=0x56292cc81650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:17.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.060+0000 7f53837fe640 0 client.16500 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:21:17.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.064+0000 7f53ad7e6740 1 -- 172.21.15.27:0/456644994 shutdown_connections 2024-04-25T00:21:17.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.068+0000 7f53ad7e6740 1 -- 172.21.15.27:0/456644994 wait complete. 2024-04-25T00:21:17.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.072+0000 7f53ad7e6740 20 client.16500 trim_cache size 0 max 16384 2024-04-25T00:21:17.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.111+0000 7f53ad7e6740 1 Processor -- start 2024-04-25T00:21:17.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.113+0000 7f53ad7e6740 1 -- start start 2024-04-25T00:21:17.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.114+0000 7f53ad7e6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc7b950 0x56292cc85fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.114+0000 7f53ad7e6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc7f630 0x56292cca2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.114+0000 7f53ad7e6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc88140 0x56292cca31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.114+0000 7f53ad7e6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f539000ffb0 con 0x56292cc88140 2024-04-25T00:21:17.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.114+0000 7f53ad7e6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5390011af0 con 0x56292cc7b950 2024-04-25T00:21:17.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.114+0000 7f53ad7e6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5390010680 con 0x56292cc7f630 2024-04-25T00:21:17.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.115+0000 7f53a37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc7f630 0x56292cca2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:17.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.115+0000 7f53a37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc7f630 0x56292cca2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:49738/0 (socket says 172.21.15.27:49738) 2024-04-25T00:21:17.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.115+0000 7f53a37fe640 1 -- 172.21.15.27:0/1140059899 learned_addr learned my addr 172.21.15.27:0/1140059899 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:21:17.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.115+0000 7f53a8f96640 1 --2- 172.21.15.27:0/1140059899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc88140 0x56292cca31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:17.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.115+0000 7f53a3fff640 1 --2- 172.21.15.27:0/1140059899 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc7b950 0x56292cc85fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:17.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.116+0000 7f53a3fff640 1 -- 172.21.15.27:0/1140059899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc7f630 msgr2=0x56292cca2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:17.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.116+0000 7f53a3fff640 1 --2- 172.21.15.27:0/1140059899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc7f630 0x56292cca2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.116+0000 7f53a3fff640 1 -- 172.21.15.27:0/1140059899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc88140 msgr2=0x56292cca31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:17.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.116+0000 7f53a3fff640 1 --2- 172.21.15.27:0/1140059899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc88140 0x56292cca31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.116+0000 7f53a3fff640 1 -- 172.21.15.27:0/1140059899 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56292cc6d090 con 0x56292cc7b950 2024-04-25T00:21:17.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.121+0000 7f53a3fff640 1 --2- 172.21.15.27:0/1140059899 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc7b950 0x56292cc85fb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f539401a860 tx=0x7f5394000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e19db3016a32fad9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:17.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.124+0000 7f5389ffb640 1 -- 172.21.15.27:0/1140059899 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5394046070 con 0x56292cc7b950 2024-04-25T00:21:17.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.124+0000 7f5389ffb640 1 -- 172.21.15.27:0/1140059899 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5394003ca0 con 0x56292cc7b950 2024-04-25T00:21:17.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.124+0000 7f5389ffb640 1 -- 172.21.15.27:0/1140059899 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53940049b0 con 0x56292cc7b950 2024-04-25T00:21:17.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.125+0000 7f53ad7e6740 1 -- 172.21.15.27:0/1140059899 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc7b950 msgr2=0x56292cc85fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:17.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.125+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/1140059899 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc7b950 0x56292cc85fb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f539401a860 tx=0x7f5394000bc0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.126+0000 7f53ad7e6740 1 -- 172.21.15.27:0/1140059899 shutdown_connections 2024-04-25T00:21:17.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.126+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/1140059899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc88140 0x56292cca31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.126+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/1140059899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc7f630 0x56292cca2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.126+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/1140059899 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc7b950 0x56292cc85fb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.126+0000 7f53ad7e6740 1 -- 172.21.15.27:0/1140059899 >> 172.21.15.27:0/1140059899 conn(0x56292cc87d60 msgr2=0x56292cc7a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:17.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.131+0000 7f53ad7e6740 1 -- 172.21.15.27:0/1140059899 shutdown_connections 2024-04-25T00:21:17.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.132+0000 7f53ad7e6740 1 -- 172.21.15.27:0/1140059899 wait complete. 2024-04-25T00:21:17.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.132+0000 7f53ad7e6740 1 -- start start 2024-04-25T00:21:17.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.133+0000 7f53ad7e6740 1 Processor -- start 2024-04-25T00:21:17.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.133+0000 7f53ad7e6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc7b950 0x56292cc843f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.133+0000 7f53ad7e6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc7f630 0x56292cc84930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.133+0000 7f53ad7e6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc88140 0x56292cc84e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.133+0000 7f53ad7e6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5390011420 con 0x56292cc88140 2024-04-25T00:21:17.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.133+0000 7f53ad7e6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5390010d50 con 0x56292cc7f630 2024-04-25T00:21:17.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.133+0000 7f53ad7e6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f539000df70 con 0x56292cc7b950 2024-04-25T00:21:17.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.133+0000 7f53ad7e6740 10 client.0 _ll_register_callbacks cb 0x7f5394005150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:21:17.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.133+0000 7f53a8f96640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc88140 0x56292cc84e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:17.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.134+0000 7f53a8f96640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc88140 0x56292cc84e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34060/0 (socket says 172.21.15.27:34060) 2024-04-25T00:21:17.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.134+0000 7f53a8f96640 1 -- 172.21.15.27:0/1705792215 learned_addr learned my addr 172.21.15.27:0/1705792215 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:21:17.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.134+0000 7f53a8f96640 1 -- 172.21.15.27:0/1705792215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc7b950 msgr2=0x56292cc843f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:21:17.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.134+0000 7f53a8f96640 1 --2- 172.21.15.27:0/1705792215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc7b950 0x56292cc843f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.134+0000 7f53a8f96640 1 -- 172.21.15.27:0/1705792215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc7f630 msgr2=0x56292cc84930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:17.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.134+0000 7f53a8f96640 1 --2- 172.21.15.27:0/1705792215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc7f630 0x56292cc84930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.134+0000 7f53a8f96640 1 -- 172.21.15.27:0/1705792215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56292cd00540 con 0x56292cc88140 2024-04-25T00:21:17.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.135+0000 7f53a8f96640 1 --2- 172.21.15.27:0/1705792215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc88140 0x56292cc84e70 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f539c015990 tx=0x7f539c0012e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3377806507408af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:17.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.137+0000 7f53837fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:21:17.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.137+0000 7f53837fe640 1 -- 172.21.15.27:0/1705792215 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539c026060 con 0x56292cc88140 2024-04-25T00:21:17.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.137+0000 7f53ad7e6740 1 -- 172.21.15.27:0/1705792215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5394003ca0 con 0x56292cc88140 2024-04-25T00:21:17.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.137+0000 7f53837fe640 1 -- 172.21.15.27:0/1705792215 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f539c014930 con 0x56292cc88140 2024-04-25T00:21:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.139+0000 7f53837fe640 1 -- 172.21.15.27:0/1705792215 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f539c014b30 con 0x56292cc88140 2024-04-25T00:21:17.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.144+0000 7f53ad7e6740 10 client.16506 fetch_fsmap learned FSMap version 52 2024-04-25T00:21:17.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.144+0000 7f53ad7e6740 1 -- 172.21.15.27:0/1705792215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x56292cd018b0 con 0x56292cc88140 2024-04-25T00:21:17.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.145+0000 7f53837fe640 1 -- 172.21.15.27:0/1705792215 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5394003ca0 con 0x56292cc88140 2024-04-25T00:21:17.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.146+0000 7f53837fe640 1 -- 172.21.15.27:0/1705792215 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 52) ==== 8362+0+0 (secure 0 0 0) 0x7f539c029350 con 0x56292cc88140 2024-04-25T00:21:17.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.147+0000 7f53ad7e6740 10 client.16506 fetch_fsmap finished waiting for FSMap version 52 2024-04-25T00:21:17.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.147+0000 7f53ad7e6740 10 client.16506 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:21:17.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.147+0000 7f53ad7e6740 20 client.16506 populate_metadata read hostname 'smithi027' 2024-04-25T00:21:17.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.147+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/1705792215 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56292cc8a620 0x56292cd040f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:17.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.147+0000 7f53ad7e6740 4 client.16506 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:21:17.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.147+0000 7f53ad7e6740 1 -- 172.21.15.27:0/1705792215 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56292cbebf40 con 0x56292cc8a620 2024-04-25T00:21:17.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.148+0000 7f53a3fff640 1 --2- 172.21.15.27:0/1705792215 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56292cc8a620 0x56292cd040f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.152+0000 7f53a3fff640 1 --2- 172.21.15.27:0/1705792215 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56292cc8a620 0x56292cd040f0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4714a4e69c5aeda3 server_cookie=ea45e9687ea9b426 in_seq=0 out_seq=0 2024-04-25T00:21:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.152+0000 7f53837fe640 10 client.16506 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:21:17.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.156+0000 7f53837fe640 1 -- 172.21.15.27:0/1705792215 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x56292cbebf40 con 0x56292cc8a620 2024-04-25T00:21:17.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.156+0000 7f53837fe640 10 client.16506 handle_command_reply: tid=0 2024-04-25T00:21:17.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.157+0000 7f53ad7e6740 1 client.16506 shutdown 2024-04-25T00:21:17.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.164+0000 7f53ad7e6740 1 -- 172.21.15.27:0/1705792215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc88140 msgr2=0x56292cc84e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:17.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.164+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/1705792215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc88140 0x56292cc84e70 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f539c015990 tx=0x7f539c0012e0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.165+0000 7f53ad7e6740 1 -- 172.21.15.27:0/1705792215 shutdown_connections 2024-04-25T00:21:17.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.165+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/1705792215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56292cc88140 0x56292cc84e70 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.165+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/1705792215 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56292cc8a620 0x56292cd040f0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.165+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/1705792215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56292cc7f630 0x56292cc84930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.165+0000 7f53ad7e6740 1 --2- 172.21.15.27:0/1705792215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56292cc7b950 0x56292cc843f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.165+0000 7f53ad7e6740 1 -- 172.21.15.27:0/1705792215 >> 172.21.15.27:0/1705792215 conn(0x56292cc87d60 msgr2=0x56292cc7a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:17.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.166+0000 7f53837fe640 0 client.16506 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:21:17.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.170+0000 7f53ad7e6740 1 -- 172.21.15.27:0/1705792215 shutdown_connections 2024-04-25T00:21:17.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.170+0000 7f53ad7e6740 1 -- 172.21.15.27:0/1705792215 wait complete. 2024-04-25T00:21:17.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.171+0000 7f53ad7e6740 20 client.16506 trim_cache size 0 max 16384 2024-04-25T00:21:17.173 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:21:17.173 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "scrub active (201 inodes in the stack)", 2024-04-25T00:21:17.173 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:21:17.173 INFO:teuthology.orchestra.run.smithi027.stdout: "a40f0131-2b37-4eb9-a2bd-6fee5124aebc": { 2024-04-25T00:21:17.173 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:21:17.173 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "a40f0131-2b37-4eb9-a2bd-6fee5124aebc", 2024-04-25T00:21:17.173 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:21:17.173 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:21:17.173 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:21:17.173 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:21:17.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.173+0000 7f53aa9f9640 1 -- 172.21.15.27:0/1998372159 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f538c0a2f00 msgr2=0x7f538c0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:17.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.173+0000 7f53aa9f9640 1 --2- 172.21.15.27:0/1998372159 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f538c0a2f00 0x7f538c0a53c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f53a4072910 tx=0x7f539c001710 comp rx=0 tx=0).stop 2024-04-25T00:21:17.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.173+0000 7f53aa9f9640 1 -- 172.21.15.27:0/1998372159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a410ff90 msgr2=0x7f53a4116890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:17.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.174+0000 7f53aa9f9640 1 --2- 172.21.15.27:0/1998372159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a410ff90 0x7f53a4116890 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f539800b450 tx=0x7f539800b910 comp rx=0 tx=0).stop 2024-04-25T00:21:17.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.174+0000 7f53aa9f9640 1 -- 172.21.15.27:0/1998372159 shutdown_connections 2024-04-25T00:21:17.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.174+0000 7f53aa9f9640 1 --2- 172.21.15.27:0/1998372159 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f538c0a2f00 0x7f538c0a53c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.174+0000 7f53aa9f9640 1 --2- 172.21.15.27:0/1998372159 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53a411af20 0x7f53a4116dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.174+0000 7f53aa9f9640 1 --2- 172.21.15.27:0/1998372159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a410ff90 0x7f53a4116890 secure :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f539800b450 tx=0x7f539800b910 comp rx=0 tx=0).stop 2024-04-25T00:21:17.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.174+0000 7f53aa9f9640 1 --2- 172.21.15.27:0/1998372159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a410dfc0 0x7f53a411a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:17.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.174+0000 7f53aa9f9640 1 -- 172.21.15.27:0/1998372159 >> 172.21.15.27:0/1998372159 conn(0x7f53a406c850 msgr2=0x7f53a411be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:17.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.176+0000 7f53aa9f9640 1 -- 172.21.15.27:0/1998372159 shutdown_connections 2024-04-25T00:21:17.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:17.176+0000 7f53aa9f9640 1 -- 172.21.15.27:0/1998372159 wait complete. 2024-04-25T00:21:17.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:16 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:17.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:16 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:17.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:16 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:17.199 INFO:tasks.cephfs.filesystem:scrub status for tag:a40f0131-2b37-4eb9-a2bd-6fee5124aebc - {'path': '/', 'tag': 'a40f0131-2b37-4eb9-a2bd-6fee5124aebc', 'options': 'recursive,force'} 2024-04-25T00:21:17.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:16 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:17.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:16 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:17.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:16 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:17.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:16 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:17.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:16 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:17.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:16 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:18.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:17 smithi027 ceph-mon[29208]: pgmap v169: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 18 MiB/s wr, 4.39k op/s 2024-04-25T00:21:18.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:17 smithi139 ceph-mon[32833]: pgmap v169: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 18 MiB/s wr, 4.39k op/s 2024-04-25T00:21:18.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:17 smithi153 ceph-mon[32896]: pgmap v169: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 18 MiB/s wr, 4.39k op/s 2024-04-25T00:21:19.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:18 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:19.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:18 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:19.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:18 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 198.214729 secs 2024-04-25T00:21:19.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:18 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:19.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:18 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:19.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:18 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 198.214729 secs 2024-04-25T00:21:19.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:18 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:19.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:18 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:19.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:18 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 198.214729 secs 2024-04-25T00:21:20.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:19 smithi027 ceph-mon[29208]: pgmap v170: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 17 MiB/s wr, 4.32k op/s 2024-04-25T00:21:20.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:19 smithi139 ceph-mon[32833]: pgmap v170: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 17 MiB/s wr, 4.32k op/s 2024-04-25T00:21:20.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:19 smithi153 ceph-mon[32896]: pgmap v170: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 17 MiB/s wr, 4.32k op/s 2024-04-25T00:21:21.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:20 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:21.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:20 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:21.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:20 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:21.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:20 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 219 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:21.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:20 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:21.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:20 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:21.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:20 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:21.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:20 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 219 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:21.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:20 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:21.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:20 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:21.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:20 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:21.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:20 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 219 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:22.177 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:21 smithi153 ceph-mon[32896]: pgmap v171: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 18 MiB/s wr, 4.53k op/s 2024-04-25T00:21:22.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:21 smithi027 ceph-mon[29208]: pgmap v171: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 18 MiB/s wr, 4.53k op/s 2024-04-25T00:21:22.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:21 smithi139 ceph-mon[32833]: pgmap v171: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 18 MiB/s wr, 4.53k op/s 2024-04-25T00:21:23.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:22 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:23.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:22 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:23.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:22 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:23.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:22 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:23.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:22 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:23.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:22 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:24.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:23 smithi027 ceph-mon[29208]: pgmap v172: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 19 MiB/s wr, 4.74k op/s 2024-04-25T00:21:24.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:23 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 203.214802 secs 2024-04-25T00:21:24.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:23 smithi139 ceph-mon[32833]: pgmap v172: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 19 MiB/s wr, 4.74k op/s 2024-04-25T00:21:24.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:23 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 203.214802 secs 2024-04-25T00:21:24.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:23 smithi153 ceph-mon[32896]: pgmap v172: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 19 MiB/s wr, 4.74k op/s 2024-04-25T00:21:24.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:23 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 203.214802 secs 2024-04-25T00:21:25.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:24 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:25.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:24 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:25.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:24 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:25.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:24 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:25.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:24 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:25.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:24 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:25.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:24 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:25.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:24 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:25.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:24 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:25 smithi139 ceph-mon[32833]: pgmap v173: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 19 MiB/s wr, 4.65k op/s 2024-04-25T00:21:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:25 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 224 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:26.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:25 smithi153 ceph-mon[32896]: pgmap v173: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 19 MiB/s wr, 4.65k op/s 2024-04-25T00:21:26.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:25 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 224 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:26.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:25 smithi027 ceph-mon[29208]: pgmap v173: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 19 MiB/s wr, 4.65k op/s 2024-04-25T00:21:26.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:25 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 224 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:28.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:28 smithi139 ceph-mon[32833]: pgmap v174: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 18 MiB/s wr, 4.53k op/s 2024-04-25T00:21:28.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:28 smithi153 ceph-mon[32896]: pgmap v174: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 18 MiB/s wr, 4.53k op/s 2024-04-25T00:21:28.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:28 smithi027 ceph-mon[29208]: pgmap v174: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 18 MiB/s wr, 4.53k op/s 2024-04-25T00:21:29.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:29 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:29.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:29 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:29.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:29 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 208.214857 secs 2024-04-25T00:21:29.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:29 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:29.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:29 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:29.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:29 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 208.214857 secs 2024-04-25T00:21:29.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:29 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:29.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:29 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:29.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:29 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 208.214857 secs 2024-04-25T00:21:30.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:30 smithi139 ceph-mon[32833]: pgmap v175: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 18 MiB/s wr, 4.40k op/s 2024-04-25T00:21:30.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:30 smithi153 ceph-mon[32896]: pgmap v175: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 18 MiB/s wr, 4.40k op/s 2024-04-25T00:21:30.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:30 smithi027 ceph-mon[29208]: pgmap v175: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 18 MiB/s wr, 4.40k op/s 2024-04-25T00:21:31.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:31 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:31.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:31 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:31.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:31 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 230 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:31.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:31 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:31.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:31 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:31.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:31 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 230 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:31.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:31 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:31.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:31 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:31.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:31 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 230 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:32.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:32 smithi153 ceph-mon[32896]: pgmap v176: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 19 MiB/s wr, 4.63k op/s 2024-04-25T00:21:32.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:32 smithi027 ceph-mon[29208]: pgmap v176: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 19 MiB/s wr, 4.63k op/s 2024-04-25T00:21:32.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:32 smithi139 ceph-mon[32833]: pgmap v176: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 19 MiB/s wr, 4.63k op/s 2024-04-25T00:21:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:33 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:33.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:33 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:33.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:33 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:33.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:33 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:33.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:33 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:33.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:33 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:34.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:34 smithi027 ceph-mon[29208]: pgmap v177: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 19 MiB/s wr, 4.77k op/s 2024-04-25T00:21:34.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:34 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 213.214927 secs 2024-04-25T00:21:34.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:34 smithi139 ceph-mon[32833]: pgmap v177: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 19 MiB/s wr, 4.77k op/s 2024-04-25T00:21:34.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:34 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 213.214927 secs 2024-04-25T00:21:34.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:34 smithi153 ceph-mon[32896]: pgmap v177: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 19 MiB/s wr, 4.77k op/s 2024-04-25T00:21:34.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:34 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 213.214927 secs 2024-04-25T00:21:35.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:35 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:35.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:35 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:35.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:35 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:35.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:35 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:35.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:35 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:35.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:35 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:36.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:36 smithi027 ceph-mon[29208]: pgmap v178: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 18 MiB/s wr, 4.56k op/s 2024-04-25T00:21:36.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:36 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 235 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:36.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:36 smithi139 ceph-mon[32833]: pgmap v178: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 18 MiB/s wr, 4.56k op/s 2024-04-25T00:21:36.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:36 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 235 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:36.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:36 smithi153 ceph-mon[32896]: pgmap v178: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 18 MiB/s wr, 4.56k op/s 2024-04-25T00:21:36.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:36 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 235 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:37.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:37 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:37.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:37 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:37.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:37 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:37.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:37 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:37.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:37 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:37.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:37 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:37.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:37 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:37.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:37 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:37.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:37 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:38.593 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:38 smithi027 ceph-mon[29208]: pgmap v179: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 15 MiB/s wr, 3.87k op/s 2024-04-25T00:21:38.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:38 smithi139 ceph-mon[32833]: pgmap v179: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 15 MiB/s wr, 3.87k op/s 2024-04-25T00:21:38.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:38 smithi153 ceph-mon[32896]: pgmap v179: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 15 MiB/s wr, 3.87k op/s 2024-04-25T00:21:39.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:39 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:39.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:39 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:39.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:39 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 218.214994 secs 2024-04-25T00:21:39.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:39 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:39.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:39 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:39.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:39 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 218.214994 secs 2024-04-25T00:21:39.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:39 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:39.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:39 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:39.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:39 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 218.214994 secs 2024-04-25T00:21:40.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:40 smithi027 ceph-mon[29208]: pgmap v180: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 15 MiB/s wr, 3.81k op/s 2024-04-25T00:21:40.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:40 smithi139 ceph-mon[32833]: pgmap v180: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 15 MiB/s wr, 3.81k op/s 2024-04-25T00:21:40.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:40 smithi153 ceph-mon[32896]: pgmap v180: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 15 MiB/s wr, 3.81k op/s 2024-04-25T00:21:41.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:41 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:41.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:41 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:41.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:41 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:41.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:41 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 240 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:41.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:41 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:41.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:41 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:41.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:41 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:41.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:41 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 240 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:41.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:41 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:41.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:41 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:41.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:41 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:41.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:41 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 240 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:42.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:42 smithi027 ceph-mon[29208]: pgmap v181: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 17 MiB/s wr, 4.16k op/s 2024-04-25T00:21:42.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:42 smithi139 ceph-mon[32833]: pgmap v181: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 17 MiB/s wr, 4.16k op/s 2024-04-25T00:21:42.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:42 smithi153 ceph-mon[32896]: pgmap v181: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 17 MiB/s wr, 4.16k op/s 2024-04-25T00:21:43.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:43 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:43.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:43 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:43.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:43 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:43.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:43 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:43.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:43 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:43.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:43 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:44.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:44 smithi027 ceph-mon[29208]: pgmap v182: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 17 MiB/s wr, 4.35k op/s 2024-04-25T00:21:44.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:44 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 223.215057 secs 2024-04-25T00:21:44.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:44 smithi139 ceph-mon[32833]: pgmap v182: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 17 MiB/s wr, 4.35k op/s 2024-04-25T00:21:44.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:44 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 223.215057 secs 2024-04-25T00:21:44.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:44 smithi153 ceph-mon[32896]: pgmap v182: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 17 MiB/s wr, 4.35k op/s 2024-04-25T00:21:44.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:44 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 223.215057 secs 2024-04-25T00:21:45.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:45 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:45.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:45 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:45.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:45 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:45.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:45 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:45.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:45 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:45.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:45 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:45.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:45 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:45.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:45 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:45.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:45 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:46.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:46 smithi027 ceph-mon[29208]: pgmap v183: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 17 MiB/s wr, 4.22k op/s 2024-04-25T00:21:46.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:46 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 245 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:46.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:46 smithi139 ceph-mon[32833]: pgmap v183: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 17 MiB/s wr, 4.22k op/s 2024-04-25T00:21:46.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:46 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 245 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:46.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:46 smithi153 ceph-mon[32896]: pgmap v183: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 17 MiB/s wr, 4.22k op/s 2024-04-25T00:21:46.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:46 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 245 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:47.201 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:21:47.202 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:21:47.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.392+0000 7f5d7fcd4640 1 -- 172.21.15.27:0/3982472163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d78100fd0 msgr2=0x7f5d7810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:47.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.392+0000 7f5d7fcd4640 1 --2- 172.21.15.27:0/3982472163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d78100fd0 0x7f5d7810d4e0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5d70009cd0 tx=0x7f5d7002a910 comp rx=0 tx=0).stop 2024-04-25T00:21:47.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.394+0000 7f5d7fcd4640 1 -- 172.21.15.27:0/3982472163 shutdown_connections 2024-04-25T00:21:47.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.394+0000 7f5d7fcd4640 1 --2- 172.21.15.27:0/3982472163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d78100fd0 0x7f5d7810d4e0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.394+0000 7f5d7fcd4640 1 --2- 172.21.15.27:0/3982472163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d78100610 0x7f5d78100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.394+0000 7f5d7fcd4640 1 --2- 172.21.15.27:0/3982472163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5d781066b0 0x7f5d78106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.394+0000 7f5d7fcd4640 1 -- 172.21.15.27:0/3982472163 >> 172.21.15.27:0/3982472163 conn(0x7f5d78077740 msgr2=0x7f5d78077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:47.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.397+0000 7f5d7fcd4640 1 -- 172.21.15.27:0/3982472163 shutdown_connections 2024-04-25T00:21:47.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.400+0000 7f5d7fcd4640 1 -- 172.21.15.27:0/3982472163 wait complete. 2024-04-25T00:21:47.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.401+0000 7f5d7fcd4640 1 Processor -- start 2024-04-25T00:21:47.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.402+0000 7f5d7fcd4640 1 -- start start 2024-04-25T00:21:47.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.402+0000 7f5d7fcd4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5d78100610 0x7f5d78196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.403+0000 7f5d7fcd4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d78100fd0 0x7f5d78197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.403+0000 7f5d7fcd4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d781066b0 0x7f5d7819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.403+0000 7f5d7fcd4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5d78110b20 con 0x7f5d78100fd0 2024-04-25T00:21:47.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.403+0000 7f5d7fcd4640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5d781109a0 con 0x7f5d78100610 2024-04-25T00:21:47.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.403+0000 7f5d7fcd4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5d78110ca0 con 0x7f5d781066b0 2024-04-25T00:21:47.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.404+0000 7f5d7d26f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d78100fd0 0x7f5d78197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:47.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.404+0000 7f5d7d26f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d78100fd0 0x7f5d78197520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59396/0 (socket says 172.21.15.27:59396) 2024-04-25T00:21:47.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.404+0000 7f5d7d26f640 1 -- 172.21.15.27:0/1169750107 learned_addr learned my addr 172.21.15.27:0/1169750107 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:21:47.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.405+0000 7f5d7e271640 1 --2- 172.21.15.27:0/1169750107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d781066b0 0x7f5d7819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:47.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.406+0000 7f5d7e271640 1 -- 172.21.15.27:0/1169750107 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5d78100610 msgr2=0x7f5d78196fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:21:47.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.406+0000 7f5d7e271640 1 --2- 172.21.15.27:0/1169750107 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5d78100610 0x7f5d78196fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.406+0000 7f5d7e271640 1 -- 172.21.15.27:0/1169750107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d78100fd0 msgr2=0x7f5d78197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:47.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.406+0000 7f5d7e271640 1 --2- 172.21.15.27:0/1169750107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d78100fd0 0x7f5d78197520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.406+0000 7f5d7e271640 1 -- 172.21.15.27:0/1169750107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d7819bfd0 con 0x7f5d781066b0 2024-04-25T00:21:47.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.406+0000 7f5d7e271640 1 --2- 172.21.15.27:0/1169750107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d781066b0 0x7f5d7819b8a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5d7002ae30 tx=0x7f5d70030fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:47.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.407+0000 7f5d6effd640 1 -- 172.21.15.27:0/1169750107 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d70003cb0 con 0x7f5d781066b0 2024-04-25T00:21:47.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.407+0000 7f5d6effd640 1 -- 172.21.15.27:0/1169750107 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d70003e50 con 0x7f5d781066b0 2024-04-25T00:21:47.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.407+0000 7f5d6effd640 1 -- 172.21.15.27:0/1169750107 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7003b410 con 0x7f5d781066b0 2024-04-25T00:21:47.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.407+0000 7f5d7fcd4640 1 -- 172.21.15.27:0/1169750107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d7819c260 con 0x7f5d781066b0 2024-04-25T00:21:47.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.407+0000 7f5d7fcd4640 1 -- 172.21.15.27:0/1169750107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d780688d0 con 0x7f5d781066b0 2024-04-25T00:21:47.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.411+0000 7f5d829e3740 1 Processor -- start 2024-04-25T00:21:47.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.413+0000 7f5d6effd640 1 -- 172.21.15.27:0/1169750107 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5d700040d0 con 0x7f5d781066b0 2024-04-25T00:21:47.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.414+0000 7f5d6effd640 1 --2- 172.21.15.27:0/1169750107 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5d640a2d80 0x7f5d640a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.414+0000 7f5d6effd640 1 -- 172.21.15.27:0/1169750107 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f5d700f0e30 con 0x7f5d781066b0 2024-04-25T00:21:47.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.419+0000 7f5d829e3740 1 -- start start 2024-04-25T00:21:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.420+0000 7f5d829e3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52031300 0x560f520316e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.420+0000 7f5d829e3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031c20 0x560f52044cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.420+0000 7f5d829e3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52045200 0x560f520475f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.420+0000 7f5d829e3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560f51f966b0 con 0x560f52031c20 2024-04-25T00:21:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.420+0000 7f5d829e3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560f51f3e880 con 0x560f52045200 2024-04-25T00:21:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.420+0000 7f5d829e3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560f51f92670 con 0x560f52031300 2024-04-25T00:21:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.420+0000 7f5d7da70640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52031300 0x560f520316e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.420+0000 7f5d7da70640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52031300 0x560f520316e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:52910/0 (socket says 172.21.15.27:52910) 2024-04-25T00:21:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.420+0000 7f5d7da70640 1 -- 172.21.15.27:0/2098956686 learned_addr learned my addr 172.21.15.27:0/2098956686 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:21:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.421+0000 7f5d7d26f640 1 --2- 172.21.15.27:0/1169750107 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5d640a2d80 0x7f5d640a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.421+0000 7f5d7da70640 1 -- 172.21.15.27:0/2098956686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52045200 msgr2=0x560f520475f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:21:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.421+0000 7f5d7da70640 1 --2- 172.21.15.27:0/2098956686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52045200 0x560f520475f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.421+0000 7f5d7da70640 1 -- 172.21.15.27:0/2098956686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031c20 msgr2=0x560f52044cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.421+0000 7f5d7da70640 1 --2- 172.21.15.27:0/2098956686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031c20 0x560f52044cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.421+0000 7f5d7da70640 1 -- 172.21.15.27:0/2098956686 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f51d747d0 con 0x560f52031300 2024-04-25T00:21:47.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.426+0000 7f5d7da70640 1 --2- 172.21.15.27:0/2098956686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52031300 0x560f520316e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5d68009740 tx=0x7f5d68009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f187a4103a44394 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:47.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.428+0000 7f5d63fff640 1 -- 172.21.15.27:0/2098956686 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d68015020 con 0x560f52031300 2024-04-25T00:21:47.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.428+0000 7f5d63fff640 1 -- 172.21.15.27:0/2098956686 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d68009e50 con 0x560f52031300 2024-04-25T00:21:47.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.428+0000 7f5d7d26f640 1 --2- 172.21.15.27:0/1169750107 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5d640a2d80 0x7f5d640a5240 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x560f52034030 tx=0x7f5d74007400 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:47.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.428+0000 7f5d63fff640 1 -- 172.21.15.27:0/2098956686 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d680105e0 con 0x560f52031300 2024-04-25T00:21:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.429+0000 7f5d829e3740 1 -- 172.21.15.27:0/2098956686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52031300 msgr2=0x560f520316e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.430+0000 7f5d829e3740 1 --2- 172.21.15.27:0/2098956686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52031300 0x560f520316e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5d68009740 tx=0x7f5d68009c00 comp rx=0 tx=0).stop 2024-04-25T00:21:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.430+0000 7f5d829e3740 1 -- 172.21.15.27:0/2098956686 shutdown_connections 2024-04-25T00:21:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.430+0000 7f5d829e3740 1 --2- 172.21.15.27:0/2098956686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52045200 0x560f520475f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.430+0000 7f5d829e3740 1 --2- 172.21.15.27:0/2098956686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031c20 0x560f52044cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.430+0000 7f5d829e3740 1 --2- 172.21.15.27:0/2098956686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52031300 0x560f520316e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.430+0000 7f5d829e3740 1 -- 172.21.15.27:0/2098956686 >> 172.21.15.27:0/2098956686 conn(0x560f5202d950 msgr2=0x560f5202dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.431+0000 7f5d829e3740 1 -- 172.21.15.27:0/2098956686 shutdown_connections 2024-04-25T00:21:47.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.432+0000 7f5d829e3740 1 -- 172.21.15.27:0/2098956686 wait complete. 2024-04-25T00:21:47.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.433+0000 7f5d829e3740 1 -- start start 2024-04-25T00:21:47.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.433+0000 7f5d829e3740 1 Processor -- start 2024-04-25T00:21:47.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.433+0000 7f5d829e3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52031300 0x560f520421e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.433+0000 7f5d829e3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031c20 0x560f52042720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.434+0000 7f5d829e3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52045200 0x560f52042c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.434+0000 7f5d829e3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560f52047c20 con 0x560f52031c20 2024-04-25T00:21:47.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.434+0000 7f5d829e3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560f51f51f50 con 0x560f52031300 2024-04-25T00:21:47.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.434+0000 7f5d829e3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560f52047da0 con 0x560f52045200 2024-04-25T00:21:47.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.434+0000 7f5d829e3740 10 client.0 _ll_register_callbacks cb 0x560f51f8d000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:21:47.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.434+0000 7f5d7da70640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52031300 0x560f520421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:47.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.434+0000 7f5d7da70640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52031300 0x560f520421e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:50136/0 (socket says 172.21.15.27:50136) 2024-04-25T00:21:47.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.434+0000 7f5d7da70640 1 -- 172.21.15.27:0/3670504677 learned_addr learned my addr 172.21.15.27:0/3670504677 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:21:47.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.435+0000 7f5d7e271640 1 --2- 172.21.15.27:0/3670504677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52045200 0x560f52042c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:47.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.435+0000 7f5d7d26f640 1 --2- 172.21.15.27:0/3670504677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031c20 0x560f52042720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:47.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.435+0000 7f5d7da70640 1 -- 172.21.15.27:0/3670504677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52045200 msgr2=0x560f52042c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:47.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.435+0000 7f5d7da70640 1 --2- 172.21.15.27:0/3670504677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52045200 0x560f52042c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.435+0000 7f5d7da70640 1 -- 172.21.15.27:0/3670504677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031c20 msgr2=0x560f52042720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:47.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.435+0000 7f5d7da70640 1 --2- 172.21.15.27:0/3670504677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031c20 0x560f52042720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.435+0000 7f5d7da70640 1 -- 172.21.15.27:0/3670504677 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f51caa3f0 con 0x560f52031300 2024-04-25T00:21:47.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.436+0000 7f5d7da70640 1 --2- 172.21.15.27:0/3670504677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52031300 0x560f520421e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5d68011c30 tx=0x7f5d68011420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77a472273bf4f8e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:47.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.436+0000 7f5d617fa640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T00:21:47.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.436+0000 7f5d617fa640 1 -- 172.21.15.27:0/3670504677 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6800bce0 con 0x560f52031300 2024-04-25T00:21:47.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.436+0000 7f5d829e3740 1 -- 172.21.15.27:0/3670504677 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d68009e50 con 0x560f52031300 2024-04-25T00:21:47.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.436+0000 7f5d617fa640 1 -- 172.21.15.27:0/3670504677 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f5d68004040 con 0x560f52031300 2024-04-25T00:21:47.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.436+0000 7f5d617fa640 1 -- 172.21.15.27:0/3670504677 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f5d6800be80 con 0x560f52031300 2024-04-25T00:21:47.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.436+0000 7f5d829e3740 10 client.26021 fetch_fsmap learned FSMap version 65 2024-04-25T00:21:47.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.436+0000 7f5d829e3740 1 -- 172.21.15.27:0/3670504677 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x560f51c6a0f0 con 0x560f52031300 2024-04-25T00:21:47.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.437+0000 7f5d617fa640 1 -- 172.21.15.27:0/3670504677 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d68009e50 con 0x560f52031300 2024-04-25T00:21:47.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.437+0000 7f5d617fa640 1 -- 172.21.15.27:0/3670504677 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 65) ==== 8362+0+0 (secure 0 0 0) 0x7f5d6801a420 con 0x560f52031300 2024-04-25T00:21:47.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.437+0000 7f5d829e3740 10 client.26021 fetch_fsmap finished waiting for FSMap version 65 2024-04-25T00:21:47.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.437+0000 7f5d829e3740 10 client.26021 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:21:47.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.437+0000 7f5d829e3740 20 client.26021 populate_metadata read hostname 'smithi027' 2024-04-25T00:21:47.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.437+0000 7f5d829e3740 1 --2- 172.21.15.27:0/3670504677 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560f5204afd0 0x560f5204d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.437+0000 7f5d829e3740 4 client.26021 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:21:47.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.437+0000 7f5d829e3740 1 -- 172.21.15.27:0/3670504677 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f51fa3bf0 con 0x560f5204afd0 2024-04-25T00:21:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.438+0000 7f5d7d26f640 1 --2- 172.21.15.27:0/3670504677 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560f5204afd0 0x560f5204d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.438+0000 7f5d617fa640 10 client.26021 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:21:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.438+0000 7f5d7d26f640 1 --2- 172.21.15.27:0/3670504677 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560f5204afd0 0x560f5204d3f0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6354feb2b4e8a53b server_cookie=1cbf619fbe477764 in_seq=0 out_seq=0 2024-04-25T00:21:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.439+0000 7f5d617fa640 1 -- 172.21.15.27:0/3670504677 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560f51fa3bf0 con 0x560f5204afd0 2024-04-25T00:21:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.439+0000 7f5d617fa640 10 client.26021 handle_command_reply: tid=0 2024-04-25T00:21:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.439+0000 7f5d829e3740 1 client.26021 shutdown 2024-04-25T00:21:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.439+0000 7f5d829e3740 1 -- 172.21.15.27:0/3670504677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52031300 msgr2=0x560f520421e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.439+0000 7f5d829e3740 1 --2- 172.21.15.27:0/3670504677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52031300 0x560f520421e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5d68011c30 tx=0x7f5d68011420 comp rx=0 tx=0).stop 2024-04-25T00:21:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.439+0000 7f5d829e3740 1 -- 172.21.15.27:0/3670504677 shutdown_connections 2024-04-25T00:21:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.439+0000 7f5d829e3740 1 --2- 172.21.15.27:0/3670504677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52045200 0x560f52042c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.439+0000 7f5d829e3740 1 --2- 172.21.15.27:0/3670504677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031c20 0x560f52042720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.439+0000 7f5d829e3740 1 --2- 172.21.15.27:0/3670504677 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560f5204afd0 0x560f5204d3f0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.439+0000 7f5d829e3740 1 --2- 172.21.15.27:0/3670504677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52031300 0x560f520421e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.439+0000 7f5d617fa640 0 client.26021 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:21:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.439+0000 7f5d829e3740 1 -- 172.21.15.27:0/3670504677 >> 172.21.15.27:0/3670504677 conn(0x560f5202d950 msgr2=0x560f520337c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.439+0000 7f5d829e3740 1 -- 172.21.15.27:0/3670504677 shutdown_connections 2024-04-25T00:21:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.439+0000 7f5d829e3740 1 -- 172.21.15.27:0/3670504677 wait complete. 2024-04-25T00:21:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.440+0000 7f5d829e3740 20 client.26021 trim_cache size 0 max 16384 2024-04-25T00:21:47.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.474+0000 7f5d829e3740 1 Processor -- start 2024-04-25T00:21:47.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.476+0000 7f5d829e3740 1 -- start start 2024-04-25T00:21:47.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.476+0000 7f5d829e3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52031300 0x560f52044910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.476+0000 7f5d829e3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031c20 0x560f52042ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.476+0000 7f5d829e3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52045200 0x560f52052d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.476+0000 7f5d829e3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5d781a42b0 con 0x560f52031c20 2024-04-25T00:21:47.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.476+0000 7f5d829e3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5d781a5df0 con 0x560f52031300 2024-04-25T00:21:47.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.476+0000 7f5d829e3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5d781a4980 con 0x560f52045200 2024-04-25T00:21:47.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.476+0000 7f5d7d26f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031c20 0x560f52042ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:47.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.476+0000 7f5d7d26f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031c20 0x560f52042ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59426/0 (socket says 172.21.15.27:59426) 2024-04-25T00:21:47.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.476+0000 7f5d7d26f640 1 -- 172.21.15.27:0/1227825340 learned_addr learned my addr 172.21.15.27:0/1227825340 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:21:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.479+0000 7f5d7d26f640 1 -- 172.21.15.27:0/1227825340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52045200 msgr2=0x560f52052d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:21:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.479+0000 7f5d7d26f640 1 --2- 172.21.15.27:0/1227825340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52045200 0x560f52052d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.479+0000 7f5d7d26f640 1 -- 172.21.15.27:0/1227825340 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52031300 msgr2=0x560f52044910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:21:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.479+0000 7f5d7d26f640 1 --2- 172.21.15.27:0/1227825340 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52031300 0x560f52044910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.479+0000 7f5d7d26f640 1 -- 172.21.15.27:0/1227825340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f52031a50 con 0x560f52031c20 2024-04-25T00:21:47.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.479+0000 7f5d7d26f640 1 --2- 172.21.15.27:0/1227825340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031c20 0x560f52042ac0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x560f52041a80 tx=0x7f5d74015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a0ed819011b8ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.484+0000 7f5d637fe640 1 -- 172.21.15.27:0/1227825340 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d74026d50 con 0x560f52031c20 2024-04-25T00:21:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.484+0000 7f5d637fe640 1 -- 172.21.15.27:0/1227825340 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d7402b070 con 0x560f52031c20 2024-04-25T00:21:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.484+0000 7f5d637fe640 1 -- 172.21.15.27:0/1227825340 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d74011420 con 0x560f52031c20 2024-04-25T00:21:47.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.485+0000 7f5d829e3740 1 -- 172.21.15.27:0/1227825340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031c20 msgr2=0x560f52042ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:47.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.485+0000 7f5d829e3740 1 --2- 172.21.15.27:0/1227825340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031c20 0x560f52042ac0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x560f52041a80 tx=0x7f5d74015000 comp rx=0 tx=0).stop 2024-04-25T00:21:47.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.488+0000 7f5d829e3740 1 -- 172.21.15.27:0/1227825340 shutdown_connections 2024-04-25T00:21:47.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.488+0000 7f5d829e3740 1 --2- 172.21.15.27:0/1227825340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52045200 0x560f52052d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.488+0000 7f5d829e3740 1 --2- 172.21.15.27:0/1227825340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031c20 0x560f52042ac0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.488+0000 7f5d829e3740 1 --2- 172.21.15.27:0/1227825340 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52031300 0x560f52044910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.488+0000 7f5d829e3740 1 -- 172.21.15.27:0/1227825340 >> 172.21.15.27:0/1227825340 conn(0x560f5202d950 msgr2=0x560f52040ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:47.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.494+0000 7f5d829e3740 1 -- 172.21.15.27:0/1227825340 shutdown_connections 2024-04-25T00:21:47.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.494+0000 7f5d829e3740 1 -- 172.21.15.27:0/1227825340 wait complete. 2024-04-25T00:21:47.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.495+0000 7f5d829e3740 1 -- start start 2024-04-25T00:21:47.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.495+0000 7f5d829e3740 1 Processor -- start 2024-04-25T00:21:47.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.496+0000 7f5d829e3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031300 0x560f520d5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.496+0000 7f5d829e3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52031c20 0x560f520d5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.496+0000 7f5d829e3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52045200 0x560f52050060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.496+0000 7f5d829e3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5d781a5720 con 0x560f52031300 2024-04-25T00:21:47.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.496+0000 7f5d829e3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5d781a5050 con 0x560f52031c20 2024-04-25T00:21:47.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.496+0000 7f5d829e3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5d78072090 con 0x560f52045200 2024-04-25T00:21:47.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.496+0000 7f5d829e3740 10 client.0 _ll_register_callbacks cb 0x7f5d68009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:21:47.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.497+0000 7f5d7e271640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52045200 0x560f52050060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:47.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.497+0000 7f5d7da70640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031300 0x560f520d5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:47.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.497+0000 7f5d7da70640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031300 0x560f520d5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59442/0 (socket says 172.21.15.27:59442) 2024-04-25T00:21:47.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.497+0000 7f5d7da70640 1 -- 172.21.15.27:0/2257975358 learned_addr learned my addr 172.21.15.27:0/2257975358 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:21:47.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.498+0000 7f5d7da70640 1 -- 172.21.15.27:0/2257975358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52045200 msgr2=0x560f52050060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:47.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.498+0000 7f5d7da70640 1 --2- 172.21.15.27:0/2257975358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52045200 0x560f52050060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.498+0000 7f5d7da70640 1 -- 172.21.15.27:0/2257975358 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52031c20 msgr2=0x560f520d5cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:21:47.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.498+0000 7f5d7da70640 1 --2- 172.21.15.27:0/2257975358 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52031c20 0x560f520d5cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.498+0000 7f5d7da70640 1 -- 172.21.15.27:0/2257975358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f52050750 con 0x560f52031300 2024-04-25T00:21:47.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.498+0000 7f5d7da70640 1 --2- 172.21.15.27:0/2257975358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031300 0x560f520d5790 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f5d6801aa30 tx=0x7f5d6800b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60df55fac98fb585 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:47.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.498+0000 7f5d617fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:21:47.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.499+0000 7f5d617fa640 1 -- 172.21.15.27:0/2257975358 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d68015040 con 0x560f52031300 2024-04-25T00:21:47.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.499+0000 7f5d617fa640 1 -- 172.21.15.27:0/2257975358 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f5d6801aab0 con 0x560f52031300 2024-04-25T00:21:47.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.499+0000 7f5d829e3740 1 -- 172.21.15.27:0/2257975358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d7402b070 con 0x560f52031300 2024-04-25T00:21:47.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.499+0000 7f5d617fa640 1 -- 172.21.15.27:0/2257975358 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f5d6800bce0 con 0x560f52031300 2024-04-25T00:21:47.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.501+0000 7f5d829e3740 10 client.16521 fetch_fsmap learned FSMap version 65 2024-04-25T00:21:47.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.501+0000 7f5d829e3740 1 -- 172.21.15.27:0/2257975358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x560f520dcf20 con 0x560f52031300 2024-04-25T00:21:47.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.504+0000 7f5d617fa640 1 -- 172.21.15.27:0/2257975358 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d7402b070 con 0x560f52031300 2024-04-25T00:21:47.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.504+0000 7f5d617fa640 1 -- 172.21.15.27:0/2257975358 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 65) ==== 8362+0+0 (secure 0 0 0) 0x7f5d680037c0 con 0x560f52031300 2024-04-25T00:21:47.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.504+0000 7f5d829e3740 10 client.16521 fetch_fsmap finished waiting for FSMap version 65 2024-04-25T00:21:47.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.504+0000 7f5d829e3740 10 client.16521 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:21:47.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.504+0000 7f5d829e3740 20 client.16521 populate_metadata read hostname 'smithi027' 2024-04-25T00:21:47.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.504+0000 7f5d829e3740 1 --2- 172.21.15.27:0/2257975358 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560f520dd650 0x560f520dfa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:47.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.504+0000 7f5d829e3740 4 client.16521 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:21:47.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.504+0000 7f5d829e3740 1 -- 172.21.15.27:0/2257975358 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560f51f9df40 con 0x560f520dd650 2024-04-25T00:21:47.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.505+0000 7f5d7d26f640 1 --2- 172.21.15.27:0/2257975358 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560f520dd650 0x560f520dfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:47.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.507+0000 7f5d7d26f640 1 --2- 172.21.15.27:0/2257975358 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560f520dd650 0x560f520dfa70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a058c5fb46fb5d38 server_cookie=4345e0b2342f550e in_seq=0 out_seq=0 2024-04-25T00:21:47.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.507+0000 7f5d617fa640 10 client.16521 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:21:47.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.508+0000 7f5d617fa640 1 -- 172.21.15.27:0/2257975358 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x560f51f9df40 con 0x560f520dd650 2024-04-25T00:21:47.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.508+0000 7f5d617fa640 10 client.16521 handle_command_reply: tid=0 2024-04-25T00:21:47.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.508+0000 7f5d829e3740 1 client.16521 shutdown 2024-04-25T00:21:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.510+0000 7f5d829e3740 1 -- 172.21.15.27:0/2257975358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031300 msgr2=0x560f520d5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.510+0000 7f5d829e3740 1 --2- 172.21.15.27:0/2257975358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031300 0x560f520d5790 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f5d6801aa30 tx=0x7f5d6800b5a0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.510+0000 7f5d829e3740 1 -- 172.21.15.27:0/2257975358 shutdown_connections 2024-04-25T00:21:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.510+0000 7f5d829e3740 1 --2- 172.21.15.27:0/2257975358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f52045200 0x560f52050060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.510+0000 7f5d829e3740 1 --2- 172.21.15.27:0/2257975358 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x560f520dd650 0x560f520dfa70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.510+0000 7f5d829e3740 1 --2- 172.21.15.27:0/2257975358 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f52031c20 0x560f520d5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.510+0000 7f5d829e3740 1 --2- 172.21.15.27:0/2257975358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f52031300 0x560f520d5790 secure :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f5d6801aa30 tx=0x7f5d6800b5a0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.510+0000 7f5d829e3740 1 -- 172.21.15.27:0/2257975358 >> 172.21.15.27:0/2257975358 conn(0x560f5202d950 msgr2=0x560f5202c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.510+0000 7f5d617fa640 0 client.16521 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:21:47.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.512+0000 7f5d829e3740 1 -- 172.21.15.27:0/2257975358 shutdown_connections 2024-04-25T00:21:47.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.513+0000 7f5d829e3740 1 -- 172.21.15.27:0/2257975358 wait complete. 2024-04-25T00:21:47.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.513+0000 7f5d829e3740 20 client.16521 trim_cache size 0 max 16384 2024-04-25T00:21:47.514 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:21:47.514 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "scrub active (201 inodes in the stack)", 2024-04-25T00:21:47.514 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:21:47.514 INFO:teuthology.orchestra.run.smithi027.stdout: "a40f0131-2b37-4eb9-a2bd-6fee5124aebc": { 2024-04-25T00:21:47.514 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:21:47.515 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "a40f0131-2b37-4eb9-a2bd-6fee5124aebc", 2024-04-25T00:21:47.515 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:21:47.515 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:21:47.515 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:21:47.515 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:21:47.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.515+0000 7f5d7fcd4640 1 -- 172.21.15.27:0/1169750107 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5d640a2d80 msgr2=0x7f5d640a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:47.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.515+0000 7f5d7fcd4640 1 --2- 172.21.15.27:0/1169750107 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5d640a2d80 0x7f5d640a5240 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x560f52034030 tx=0x7f5d74007400 comp rx=0 tx=0).stop 2024-04-25T00:21:47.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.515+0000 7f5d7fcd4640 1 -- 172.21.15.27:0/1169750107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d781066b0 msgr2=0x7f5d7819b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:47.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.515+0000 7f5d7fcd4640 1 --2- 172.21.15.27:0/1169750107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d781066b0 0x7f5d7819b8a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5d7002ae30 tx=0x7f5d70030fb0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.515+0000 7f5d7fcd4640 1 -- 172.21.15.27:0/1169750107 shutdown_connections 2024-04-25T00:21:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.515+0000 7f5d7fcd4640 1 --2- 172.21.15.27:0/1169750107 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5d640a2d80 0x7f5d640a5240 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.515+0000 7f5d7fcd4640 1 --2- 172.21.15.27:0/1169750107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d781066b0 0x7f5d7819b8a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.515+0000 7f5d7fcd4640 1 --2- 172.21.15.27:0/1169750107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d78100fd0 0x7f5d78197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.515+0000 7f5d7fcd4640 1 --2- 172.21.15.27:0/1169750107 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5d78100610 0x7f5d78196fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.515+0000 7f5d7fcd4640 1 -- 172.21.15.27:0/1169750107 >> 172.21.15.27:0/1169750107 conn(0x7f5d78077740 msgr2=0x7f5d780ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.516+0000 7f5d7fcd4640 1 -- 172.21.15.27:0/1169750107 shutdown_connections 2024-04-25T00:21:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:21:47.517+0000 7f5d7fcd4640 1 -- 172.21.15.27:0/1169750107 wait complete. 2024-04-25T00:21:47.533 INFO:tasks.cephfs.filesystem:scrub status for tag:a40f0131-2b37-4eb9-a2bd-6fee5124aebc - {'path': '/', 'tag': 'a40f0131-2b37-4eb9-a2bd-6fee5124aebc', 'options': 'recursive,force'} 2024-04-25T00:21:48.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:48 smithi027 ceph-mon[29208]: pgmap v184: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 15 MiB/s wr, 3.79k op/s 2024-04-25T00:21:48.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:48 smithi139 ceph-mon[32833]: pgmap v184: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 15 MiB/s wr, 3.79k op/s 2024-04-25T00:21:48.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:48 smithi153 ceph-mon[32896]: pgmap v184: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 15 MiB/s wr, 3.79k op/s 2024-04-25T00:21:49.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:49 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 228.215128 secs 2024-04-25T00:21:49.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:49 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:49.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:49 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:49.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:49 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 228.215128 secs 2024-04-25T00:21:49.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:49 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:49.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:49 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:49.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:49 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 228.215128 secs 2024-04-25T00:21:49.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:49 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:49.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:49 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:50.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:50 smithi139 ceph-mon[32833]: pgmap v185: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 16 MiB/s wr, 4.03k op/s 2024-04-25T00:21:50.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:50 smithi153 ceph-mon[32896]: pgmap v185: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 16 MiB/s wr, 4.03k op/s 2024-04-25T00:21:50.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:50 smithi027 ceph-mon[29208]: pgmap v185: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 16 MiB/s wr, 4.03k op/s 2024-04-25T00:21:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:51 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:51 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:51 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 250 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:51.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:51 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:51.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:51 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:51.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:51 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 250 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:51.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:51 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:51.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:51 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:51.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:51 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 250 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:52.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:52 smithi139 ceph-mon[32833]: pgmap v186: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 17 MiB/s wr, 4.22k op/s 2024-04-25T00:21:52.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:52 smithi153 ceph-mon[32896]: pgmap v186: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 17 MiB/s wr, 4.22k op/s 2024-04-25T00:21:52.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:52 smithi027 ceph-mon[29208]: pgmap v186: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 17 MiB/s wr, 4.22k op/s 2024-04-25T00:21:53.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:53 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:53.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:53 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:53.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:53 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 233.215214 secs 2024-04-25T00:21:53.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:53 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:53.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:53 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:53.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:53 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 233.215214 secs 2024-04-25T00:21:53.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:53 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:53.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:53 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:53.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:53 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 233.215214 secs 2024-04-25T00:21:54.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:54 smithi139 ceph-mon[32833]: pgmap v187: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 18 MiB/s wr, 4.37k op/s 2024-04-25T00:21:54.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:54 smithi153 ceph-mon[32896]: pgmap v187: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 18 MiB/s wr, 4.37k op/s 2024-04-25T00:21:54.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:54 smithi027 ceph-mon[29208]: pgmap v187: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 18 MiB/s wr, 4.37k op/s 2024-04-25T00:21:55.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:55 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:55.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:55 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:55.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:55 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:55.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:55 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:56.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:55 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:56.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:55 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:56.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:56 smithi153 ceph-mon[32896]: pgmap v188: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 18 MiB/s wr, 4.38k op/s 2024-04-25T00:21:56.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:56 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 255 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:56.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:56 smithi027 ceph-mon[29208]: pgmap v188: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 18 MiB/s wr, 4.38k op/s 2024-04-25T00:21:56.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:56 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 255 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:57.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:56 smithi139 ceph-mon[32833]: pgmap v188: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 18 MiB/s wr, 4.38k op/s 2024-04-25T00:21:57.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:56 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 255 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:21:57.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:57 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:57.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:57 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:57.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:57 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:58.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:57 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:58.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:57 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:58.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:57 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:58.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:57 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:21:58.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:57 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:21:58.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:57 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:21:58.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:58 smithi027 ceph-mon[29208]: pgmap v189: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 15 MiB/s wr, 3.77k op/s 2024-04-25T00:21:58.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:58 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 238.215668 secs 2024-04-25T00:21:59.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:58 smithi139 ceph-mon[32833]: pgmap v189: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 15 MiB/s wr, 3.77k op/s 2024-04-25T00:21:59.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:58 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 238.215668 secs 2024-04-25T00:21:59.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:58 smithi153 ceph-mon[32896]: pgmap v189: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 15 MiB/s wr, 3.77k op/s 2024-04-25T00:21:59.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:58 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 238.215668 secs 2024-04-25T00:21:59.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:59 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:21:59.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:21:59 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:00.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:59 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:00.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:21:59 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:00.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:59 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:00.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:21:59 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:00.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:00 smithi027 ceph-mon[29208]: pgmap v190: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 17 MiB/s wr, 4.34k op/s 2024-04-25T00:22:01.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:00 smithi139 ceph-mon[32833]: pgmap v190: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 17 MiB/s wr, 4.34k op/s 2024-04-25T00:22:01.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:00 smithi153 ceph-mon[32896]: pgmap v190: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 17 MiB/s wr, 4.34k op/s 2024-04-25T00:22:01.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:01 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:01.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:01 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:01.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:01 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:01.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:01 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 260 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:02.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:01 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:02.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:01 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:02.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:01 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:02.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:01 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 260 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:02.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:01 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:02.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:01 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:02.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:01 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:02.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:01 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 260 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:02.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:02 smithi027 ceph-mon[29208]: pgmap v191: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 18 MiB/s wr, 4.45k op/s 2024-04-25T00:22:03.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:02 smithi139 ceph-mon[32833]: pgmap v191: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 18 MiB/s wr, 4.45k op/s 2024-04-25T00:22:03.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:02 smithi153 ceph-mon[32896]: pgmap v191: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 18 MiB/s wr, 4.45k op/s 2024-04-25T00:22:03.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:03 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:03.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:03 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:03.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:03 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 243.215730 secs 2024-04-25T00:22:04.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:03 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:04.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:03 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:04.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:03 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 243.215730 secs 2024-04-25T00:22:04.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:03 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:04.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:03 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:04.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:03 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 243.215730 secs 2024-04-25T00:22:04.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:04 smithi027 ceph-mon[29208]: pgmap v192: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 4.53k op/s 2024-04-25T00:22:04.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:04 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:04.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:04 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:05.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:04 smithi139 ceph-mon[32833]: pgmap v192: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 4.53k op/s 2024-04-25T00:22:05.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:04 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:05.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:04 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:05.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:04 smithi153 ceph-mon[32896]: pgmap v192: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 4.53k op/s 2024-04-25T00:22:05.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:04 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:05.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:04 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:05.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:05 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:05.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:05 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:06.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:05 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:06.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:05 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:06.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:05 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:06.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:05 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:06.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:06 smithi027 ceph-mon[29208]: pgmap v193: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 17 MiB/s wr, 4.33k op/s 2024-04-25T00:22:06.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:06 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 265 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:07.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:06 smithi139 ceph-mon[32833]: pgmap v193: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 17 MiB/s wr, 4.33k op/s 2024-04-25T00:22:07.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:06 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 265 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:07.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:06 smithi153 ceph-mon[32896]: pgmap v193: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 17 MiB/s wr, 4.33k op/s 2024-04-25T00:22:07.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:06 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 265 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:08.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:08 smithi027 ceph-mon[29208]: pgmap v194: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 14 MiB/s wr, 3.52k op/s 2024-04-25T00:22:08.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:08 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:08.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:08 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:08.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:08 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 248.215801 secs 2024-04-25T00:22:09.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:08 smithi139 ceph-mon[32833]: pgmap v194: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 14 MiB/s wr, 3.52k op/s 2024-04-25T00:22:09.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:08 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:09.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:08 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:09.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:08 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 248.215801 secs 2024-04-25T00:22:09.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:08 smithi153 ceph-mon[32896]: pgmap v194: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 14 MiB/s wr, 3.52k op/s 2024-04-25T00:22:09.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:08 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:09.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:08 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:09.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:08 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 248.215801 secs 2024-04-25T00:22:10.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:10 smithi027 ceph-mon[29208]: pgmap v195: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 18 MiB/s wr, 4.37k op/s 2024-04-25T00:22:10.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:10 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:22:11.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:10 smithi139 ceph-mon[32833]: pgmap v195: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 18 MiB/s wr, 4.37k op/s 2024-04-25T00:22:11.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:10 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:22:11.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:10 smithi153 ceph-mon[32896]: pgmap v195: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 18 MiB/s wr, 4.37k op/s 2024-04-25T00:22:11.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:10 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:22:11.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:11 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:11.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:11 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:11.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:11 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 271 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:11.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:11 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:22:11.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:11 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:22:11.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:11 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:22:12.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:11 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:12.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:11 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:12.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:11 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 271 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:12.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:11 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:22:12.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:11 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:22:12.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:11 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:22:12.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:11 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:12.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:11 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:12.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:11 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 271 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:12.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:11 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:22:12.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:11 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:22:12.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:11 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:22:12.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:12 smithi027 ceph-mon[29208]: pgmap v196: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 17 MiB/s wr, 4.12k op/s 2024-04-25T00:22:12.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:12 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:12.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:12 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:13.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:12 smithi139 ceph-mon[32833]: pgmap v196: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 17 MiB/s wr, 4.12k op/s 2024-04-25T00:22:13.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:12 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:13.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:12 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:13.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:12 smithi153 ceph-mon[32896]: pgmap v196: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 17 MiB/s wr, 4.12k op/s 2024-04-25T00:22:13.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:12 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:13.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:12 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:13.946 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:13 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 253.215853 secs 2024-04-25T00:22:14.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:13 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 253.215853 secs 2024-04-25T00:22:14.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:13 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 253.215853 secs 2024-04-25T00:22:15.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:14 smithi139 ceph-mon[32833]: pgmap v197: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 17 MiB/s wr, 4.21k op/s 2024-04-25T00:22:15.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:14 smithi153 ceph-mon[32896]: pgmap v197: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 17 MiB/s wr, 4.21k op/s 2024-04-25T00:22:15.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:14 smithi027 ceph-mon[29208]: pgmap v197: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 17 MiB/s wr, 4.21k op/s 2024-04-25T00:22:16.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:15 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:16.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:15 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:16.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:15 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:16.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:15 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:16.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:15 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:16.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:15 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:17.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:16 smithi139 ceph-mon[32833]: pgmap v198: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 18 MiB/s wr, 4.51k op/s 2024-04-25T00:22:17.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:16 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 276 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:17.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:16 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:17.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:16 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:17.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:16 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:17.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:16 smithi153 ceph-mon[32896]: pgmap v198: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 18 MiB/s wr, 4.51k op/s 2024-04-25T00:22:17.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:16 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 276 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:17.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:16 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:17.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:16 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:17.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:16 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:17.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:16 smithi027 ceph-mon[29208]: pgmap v198: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 18 MiB/s wr, 4.51k op/s 2024-04-25T00:22:17.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:16 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 276 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:17.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:16 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:17.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:16 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:17.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:16 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:17.534 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:22:17.534 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:22:17.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.716+0000 7f8901727640 1 -- 172.21.15.27:0/394064251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88fc1091c0 msgr2=0x7f88fc11cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:17.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.716+0000 7f8901727640 1 --2- 172.21.15.27:0/394064251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88fc1091c0 0x7f88fc11cf30 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f88e801a910 tx=0x7f88e803b500 comp rx=0 tx=0).stop 2024-04-25T00:22:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.719+0000 7f8901727640 1 -- 172.21.15.27:0/394064251 shutdown_connections 2024-04-25T00:22:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.719+0000 7f8901727640 1 --2- 172.21.15.27:0/394064251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88fc1091c0 0x7f88fc11cf30 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.719+0000 7f8901727640 1 --2- 172.21.15.27:0/394064251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88fc108740 0x7f88fc108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.719+0000 7f8901727640 1 --2- 172.21.15.27:0/394064251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88fc10fde0 0x7f88fc1101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.719+0000 7f8901727640 1 -- 172.21.15.27:0/394064251 >> 172.21.15.27:0/394064251 conn(0x7f88fc06c6c0 msgr2=0x7f88fc06cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.727+0000 7f8901727640 1 -- 172.21.15.27:0/394064251 shutdown_connections 2024-04-25T00:22:17.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.730+0000 7f8901727640 1 -- 172.21.15.27:0/394064251 wait complete. 2024-04-25T00:22:17.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.731+0000 7f8901727640 1 Processor -- start 2024-04-25T00:22:17.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.732+0000 7f8901727640 1 -- start start 2024-04-25T00:22:17.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.733+0000 7f8901727640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88fc108740 0x7f88fc11cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.733+0000 7f8901727640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88fc1091c0 0x7f88fc118e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.733+0000 7f8901727640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88fc10fde0 0x7f88fc119360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.733+0000 7f8901727640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc113170 con 0x7f88fc108740 2024-04-25T00:22:17.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.733+0000 7f8901727640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc112ff0 con 0x7f88fc1091c0 2024-04-25T00:22:17.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.733+0000 7f8901727640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc1132f0 con 0x7f88fc10fde0 2024-04-25T00:22:17.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.734+0000 7f88fb7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88fc1091c0 0x7f88fc118e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:17.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.734+0000 7f88fb7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88fc1091c0 0x7f88fc118e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:52450/0 (socket says 172.21.15.27:52450) 2024-04-25T00:22:17.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.734+0000 7f88fb7fe640 1 -- 172.21.15.27:0/3932448007 learned_addr learned my addr 172.21.15.27:0/3932448007 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:22:17.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.735+0000 7f8900f26640 1 --2- 172.21.15.27:0/3932448007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88fc10fde0 0x7f88fc119360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:17.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.735+0000 7f88fb7fe640 1 -- 172.21.15.27:0/3932448007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88fc10fde0 msgr2=0x7f88fc119360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:17.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.735+0000 7f88fb7fe640 1 --2- 172.21.15.27:0/3932448007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88fc10fde0 0x7f88fc119360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.735+0000 7f88fb7fe640 1 -- 172.21.15.27:0/3932448007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88fc108740 msgr2=0x7f88fc11cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:17.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.735+0000 7f88fb7fe640 1 --2- 172.21.15.27:0/3932448007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88fc108740 0x7f88fc11cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.735+0000 7f88fb7fe640 1 -- 172.21.15.27:0/3932448007 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88fc119c20 con 0x7f88fc1091c0 2024-04-25T00:22:17.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.738+0000 7f88fb7fe640 1 --2- 172.21.15.27:0/3932448007 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88fc1091c0 0x7f88fc118e20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00b450 tx=0x7f88ec00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:17.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.739+0000 7f88f97fa640 1 -- 172.21.15.27:0/3932448007 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec015020 con 0x7f88fc1091c0 2024-04-25T00:22:17.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.739+0000 7f8901727640 1 -- 172.21.15.27:0/3932448007 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88fc1aac00 con 0x7f88fc1091c0 2024-04-25T00:22:17.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.739+0000 7f8901727640 1 -- 172.21.15.27:0/3932448007 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88fc1ab120 con 0x7f88fc1091c0 2024-04-25T00:22:17.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.739+0000 7f88f97fa640 1 -- 172.21.15.27:0/3932448007 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88ec011d70 con 0x7f88fc1091c0 2024-04-25T00:22:17.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.739+0000 7f88f97fa640 1 -- 172.21.15.27:0/3932448007 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec00f340 con 0x7f88fc1091c0 2024-04-25T00:22:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.743+0000 7f8905772740 1 Processor -- start 2024-04-25T00:22:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.743+0000 7f8905772740 1 -- start start 2024-04-25T00:22:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.743+0000 7f8905772740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5aa800 0x564e9a5aabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.743+0000 7f8905772740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5ab120 0x564e9a5b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.743+0000 7f8905772740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5b57f0 0x564e9a5b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.743+0000 7f8905772740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564e9a50f6b0 con 0x564e9a5ab120 2024-04-25T00:22:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.743+0000 7f8905772740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564e9a4b7880 con 0x564e9a5aa800 2024-04-25T00:22:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.743+0000 7f8905772740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564e9a50b670 con 0x564e9a5b57f0 2024-04-25T00:22:17.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.744+0000 7f88fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5ab120 0x564e9a5b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:17.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.744+0000 7f88fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5ab120 0x564e9a5b52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37050/0 (socket says 172.21.15.27:37050) 2024-04-25T00:22:17.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.744+0000 7f88fbfff640 1 -- 172.21.15.27:0/901558836 learned_addr learned my addr 172.21.15.27:0/901558836 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:22:17.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.745+0000 7f88fbfff640 1 -- 172.21.15.27:0/901558836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5b57f0 msgr2=0x564e9a5b7be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:22:17.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.745+0000 7f88fbfff640 1 --2- 172.21.15.27:0/901558836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5b57f0 0x564e9a5b7be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.745+0000 7f88fbfff640 1 -- 172.21.15.27:0/901558836 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5aa800 msgr2=0x564e9a5aabe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:22:17.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.745+0000 7f88fbfff640 1 --2- 172.21.15.27:0/901558836 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5aa800 0x564e9a5aabe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.745+0000 7f88fbfff640 1 -- 172.21.15.27:0/901558836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e9a2ed7d0 con 0x564e9a5ab120 2024-04-25T00:22:17.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.745+0000 7f88fbfff640 1 --2- 172.21.15.27:0/901558836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5ab120 0x564e9a5b52b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f88f4009d80 tx=0x7f88f400b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928c5a20aeb310c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:17.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.746+0000 7f88f27fc640 1 -- 172.21.15.27:0/901558836 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f400bb30 con 0x564e9a5ab120 2024-04-25T00:22:17.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.746+0000 7f88f27fc640 1 -- 172.21.15.27:0/901558836 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88f4017070 con 0x564e9a5ab120 2024-04-25T00:22:17.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.746+0000 7f88f97fa640 1 -- 172.21.15.27:0/3932448007 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f88ec018020 con 0x7f88fc1091c0 2024-04-25T00:22:17.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.747+0000 7f8905772740 1 -- 172.21.15.27:0/901558836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5ab120 msgr2=0x564e9a5b52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:17.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.747+0000 7f8905772740 1 --2- 172.21.15.27:0/901558836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5ab120 0x564e9a5b52b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f88f4009d80 tx=0x7f88f400b420 comp rx=0 tx=0).stop 2024-04-25T00:22:17.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.747+0000 7f88f97fa640 1 --2- 172.21.15.27:0/3932448007 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f88e00a2f00 0x7f88e00a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.747+0000 7f88f97fa640 1 -- 172.21.15.27:0/3932448007 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f88ec0cf650 con 0x7f88fc1091c0 2024-04-25T00:22:17.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.749+0000 7f8905772740 1 -- 172.21.15.27:0/901558836 shutdown_connections 2024-04-25T00:22:17.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.749+0000 7f8905772740 1 --2- 172.21.15.27:0/901558836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5b57f0 0x564e9a5b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.749+0000 7f8905772740 1 --2- 172.21.15.27:0/901558836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5ab120 0x564e9a5b52b0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.749+0000 7f8905772740 1 --2- 172.21.15.27:0/901558836 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5aa800 0x564e9a5aabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.749+0000 7f8905772740 1 -- 172.21.15.27:0/901558836 >> 172.21.15.27:0/901558836 conn(0x564e9a5a6b20 msgr2=0x564e9a5a6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:17.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.749+0000 7f8900f26640 1 --2- 172.21.15.27:0/3932448007 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f88e00a2f00 0x7f88e00a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:17.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.752+0000 7f8905772740 1 -- 172.21.15.27:0/901558836 shutdown_connections 2024-04-25T00:22:17.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.754+0000 7f8905772740 1 -- 172.21.15.27:0/901558836 wait complete. 2024-04-25T00:22:17.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.755+0000 7f8905772740 1 -- start start 2024-04-25T00:22:17.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.755+0000 7f8905772740 1 Processor -- start 2024-04-25T00:22:17.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.756+0000 7f8905772740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5aa800 0x564e9a5b5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.756+0000 7f8905772740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5ab120 0x564e9a5b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.756+0000 7f8905772740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5b57f0 0x564e9a5b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.756+0000 7f8905772740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564e9a5b8630 con 0x564e9a5b57f0 2024-04-25T00:22:17.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.756+0000 7f8905772740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564e9a4caf50 con 0x564e9a5aa800 2024-04-25T00:22:17.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.756+0000 7f8905772740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564e9a5b87b0 con 0x564e9a5ab120 2024-04-25T00:22:17.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.756+0000 7f8900f26640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5b57f0 0x564e9a5b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:17.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.756+0000 7f8900f26640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5b57f0 0x564e9a5b2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37062/0 (socket says 172.21.15.27:37062) 2024-04-25T00:22:17.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.756+0000 7f8900f26640 1 -- 172.21.15.27:0/2619773698 learned_addr learned my addr 172.21.15.27:0/2619773698 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:22:17.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.756+0000 7f8900f26640 1 --2- 172.21.15.27:0/3932448007 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f88e00a2f00 0x7f88e00a53c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f88e8005dc0 tx=0x7f88e8005fd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:17.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.756+0000 7f8905772740 10 client.0 _ll_register_callbacks cb 0x564e9a506000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:22:17.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.756+0000 7f8900f26640 1 -- 172.21.15.27:0/2619773698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5ab120 msgr2=0x564e9a5b25c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:22:17.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.756+0000 7f88fbfff640 1 --2- 172.21.15.27:0/2619773698 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5aa800 0x564e9a5b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:17.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.757+0000 7f88fb7fe640 1 --2- 172.21.15.27:0/2619773698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5ab120 0x564e9a5b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:17.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.757+0000 7f8900f26640 1 --2- 172.21.15.27:0/2619773698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5ab120 0x564e9a5b25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.757+0000 7f8900f26640 1 -- 172.21.15.27:0/2619773698 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5aa800 msgr2=0x564e9a5b5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:17.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.757+0000 7f8900f26640 1 --2- 172.21.15.27:0/2619773698 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5aa800 0x564e9a5b5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.757+0000 7f8900f26640 1 -- 172.21.15.27:0/2619773698 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e9a2233f0 con 0x564e9a5b57f0 2024-04-25T00:22:17.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.757+0000 7f88fbfff640 1 --2- 172.21.15.27:0/2619773698 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5aa800 0x564e9a5b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:22:17.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.757+0000 7f88fb7fe640 1 --2- 172.21.15.27:0/2619773698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5ab120 0x564e9a5b25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:22:17.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.757+0000 7f8900f26640 1 --2- 172.21.15.27:0/2619773698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5b57f0 0x564e9a5b2d00 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f88e80045a0 tx=0x7f88e801a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fcd149fa2d63886 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:17.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.757+0000 7f88df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:22:17.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.758+0000 7f88df7fe640 1 -- 172.21.15.27:0/2619773698 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8053070 con 0x564e9a5b57f0 2024-04-25T00:22:17.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.758+0000 7f8905772740 1 -- 172.21.15.27:0/2619773698 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88f4017070 con 0x564e9a5b57f0 2024-04-25T00:22:17.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.758+0000 7f88df7fe640 1 -- 172.21.15.27:0/2619773698 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f88e8005420 con 0x564e9a5b57f0 2024-04-25T00:22:17.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.758+0000 7f88df7fe640 1 -- 172.21.15.27:0/2619773698 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f88e8067050 con 0x564e9a5b57f0 2024-04-25T00:22:17.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.759+0000 7f8905772740 10 client.16530 fetch_fsmap learned FSMap version 80 2024-04-25T00:22:17.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.759+0000 7f8905772740 1 -- 172.21.15.27:0/2619773698 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=4+}) -- 0x564e9a1e30f0 con 0x564e9a5b57f0 2024-04-25T00:22:17.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.759+0000 7f88df7fe640 1 -- 172.21.15.27:0/2619773698 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88f4017070 con 0x564e9a5b57f0 2024-04-25T00:22:17.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.759+0000 7f88df7fe640 1 -- 172.21.15.27:0/2619773698 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 80) ==== 8362+0+0 (secure 0 0 0) 0x7f88e8062030 con 0x564e9a5b57f0 2024-04-25T00:22:17.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.759+0000 7f8905772740 10 client.16530 fetch_fsmap finished waiting for FSMap version 80 2024-04-25T00:22:17.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.759+0000 7f8905772740 10 client.16530 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:22:17.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.759+0000 7f8905772740 20 client.16530 populate_metadata read hostname 'smithi027' 2024-04-25T00:22:17.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.759+0000 7f8905772740 1 --2- 172.21.15.27:0/2619773698 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564e9a5c5ee0 0x564e9a5c8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.759+0000 7f8905772740 4 client.16530 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:22:17.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.759+0000 7f8905772740 1 -- 172.21.15.27:0/2619773698 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e9a51cbf0 con 0x564e9a5c5ee0 2024-04-25T00:22:17.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.760+0000 7f88fbfff640 1 --2- 172.21.15.27:0/2619773698 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564e9a5c5ee0 0x564e9a5c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:17.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.761+0000 7f88fbfff640 1 --2- 172.21.15.27:0/2619773698 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564e9a5c5ee0 0x564e9a5c8360 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e40026a67bd61ba3 server_cookie=cb559d9f966d5b4 in_seq=0 out_seq=0 2024-04-25T00:22:17.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.761+0000 7f88df7fe640 10 client.16530 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:22:17.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.763+0000 7f88df7fe640 1 -- 172.21.15.27:0/2619773698 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e9a51cbf0 con 0x564e9a5c5ee0 2024-04-25T00:22:17.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.763+0000 7f88df7fe640 10 client.16530 handle_command_reply: tid=0 2024-04-25T00:22:17.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.763+0000 7f8905772740 1 client.16530 shutdown 2024-04-25T00:22:17.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.769+0000 7f8905772740 1 -- 172.21.15.27:0/2619773698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5b57f0 msgr2=0x564e9a5b2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:17.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.769+0000 7f8905772740 1 --2- 172.21.15.27:0/2619773698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5b57f0 0x564e9a5b2d00 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f88e80045a0 tx=0x7f88e801a270 comp rx=0 tx=0).stop 2024-04-25T00:22:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.770+0000 7f8905772740 1 -- 172.21.15.27:0/2619773698 shutdown_connections 2024-04-25T00:22:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.770+0000 7f8905772740 1 --2- 172.21.15.27:0/2619773698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5b57f0 0x564e9a5b2d00 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.770+0000 7f8905772740 1 --2- 172.21.15.27:0/2619773698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5ab120 0x564e9a5b25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.770+0000 7f8905772740 1 --2- 172.21.15.27:0/2619773698 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564e9a5c5ee0 0x564e9a5c8360 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.770+0000 7f8905772740 1 --2- 172.21.15.27:0/2619773698 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5aa800 0x564e9a5b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.770+0000 7f8905772740 1 -- 172.21.15.27:0/2619773698 >> 172.21.15.27:0/2619773698 conn(0x564e9a5a6b20 msgr2=0x564e9a5ac820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:17.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.771+0000 7f88df7fe640 0 client.16530 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:22:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.776+0000 7f8905772740 1 -- 172.21.15.27:0/2619773698 shutdown_connections 2024-04-25T00:22:17.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.780+0000 7f8905772740 1 -- 172.21.15.27:0/2619773698 wait complete. 2024-04-25T00:22:17.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.780+0000 7f8905772740 20 client.16530 trim_cache size 0 max 16384 2024-04-25T00:22:17.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.819+0000 7f8905772740 1 Processor -- start 2024-04-25T00:22:17.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.819+0000 7f8905772740 1 -- start start 2024-04-25T00:22:17.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.819+0000 7f8905772740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5aa800 0x564e9a5acfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.819+0000 7f8905772740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5ab120 0x564e9a5cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.819+0000 7f8905772740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5b57f0 0x564e9a5ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.819+0000 7f8905772740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f88e400fef0 con 0x564e9a5aa800 2024-04-25T00:22:17.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.819+0000 7f8905772740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f88e4011a30 con 0x564e9a5b57f0 2024-04-25T00:22:17.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.819+0000 7f8905772740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f88e40105c0 con 0x564e9a5ab120 2024-04-25T00:22:17.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.820+0000 7f88fb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5ab120 0x564e9a5cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:17.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.820+0000 7f88fb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5ab120 0x564e9a5cdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:37126/0 (socket says 172.21.15.27:37126) 2024-04-25T00:22:17.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.820+0000 7f88fb7fe640 1 -- 172.21.15.27:0/1811842981 learned_addr learned my addr 172.21.15.27:0/1811842981 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:22:17.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.823+0000 7f8900f26640 1 --2- 172.21.15.27:0/1811842981 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5b57f0 0x564e9a5ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:17.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.824+0000 7f8900f26640 1 -- 172.21.15.27:0/1811842981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5ab120 msgr2=0x564e9a5cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:17.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.824+0000 7f8900f26640 1 --2- 172.21.15.27:0/1811842981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5ab120 0x564e9a5cdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.824+0000 7f8900f26640 1 -- 172.21.15.27:0/1811842981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5aa800 msgr2=0x564e9a5acfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:17.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.824+0000 7f8900f26640 1 --2- 172.21.15.27:0/1811842981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5aa800 0x564e9a5acfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.824+0000 7f8900f26640 1 -- 172.21.15.27:0/1811842981 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e9a5aaf50 con 0x564e9a5b57f0 2024-04-25T00:22:17.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.824+0000 7f8900f26640 1 --2- 172.21.15.27:0/1811842981 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5b57f0 0x564e9a5ce1b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f88e8004570 tx=0x7f88e8004d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c825ada522e6e55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:17.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.825+0000 7f88f1ffb640 1 -- 172.21.15.27:0/1811842981 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8053040 con 0x564e9a5b57f0 2024-04-25T00:22:17.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.832+0000 7f88f1ffb640 1 -- 172.21.15.27:0/1811842981 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88e8040590 con 0x564e9a5b57f0 2024-04-25T00:22:17.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.832+0000 7f88f1ffb640 1 -- 172.21.15.27:0/1811842981 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8046040 con 0x564e9a5b57f0 2024-04-25T00:22:17.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.834+0000 7f8905772740 1 -- 172.21.15.27:0/1811842981 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5b57f0 msgr2=0x564e9a5ce1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:17.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.834+0000 7f8905772740 1 --2- 172.21.15.27:0/1811842981 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5b57f0 0x564e9a5ce1b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f88e8004570 tx=0x7f88e8004d50 comp rx=0 tx=0).stop 2024-04-25T00:22:17.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.838+0000 7f8905772740 1 -- 172.21.15.27:0/1811842981 shutdown_connections 2024-04-25T00:22:17.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.838+0000 7f8905772740 1 --2- 172.21.15.27:0/1811842981 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5b57f0 0x564e9a5ce1b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.838+0000 7f8905772740 1 --2- 172.21.15.27:0/1811842981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5ab120 0x564e9a5cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.838+0000 7f8905772740 1 --2- 172.21.15.27:0/1811842981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5aa800 0x564e9a5acfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.838+0000 7f8905772740 1 -- 172.21.15.27:0/1811842981 >> 172.21.15.27:0/1811842981 conn(0x564e9a5a6b20 msgr2=0x564e9a5b0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:17.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.843+0000 7f8905772740 1 -- 172.21.15.27:0/1811842981 shutdown_connections 2024-04-25T00:22:17.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.844+0000 7f8905772740 1 -- 172.21.15.27:0/1811842981 wait complete. 2024-04-25T00:22:17.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.845+0000 7f8905772740 1 -- start start 2024-04-25T00:22:17.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.845+0000 7f8905772740 1 Processor -- start 2024-04-25T00:22:17.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.849+0000 7f8905772740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5aa800 0x564e9a5bdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.849+0000 7f8905772740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5ab120 0x564e9a5af110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.849+0000 7f8905772740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5b57f0 0x564e9a5af690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.849+0000 7f8905772740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f88e4011360 con 0x564e9a5ab120 2024-04-25T00:22:17.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.849+0000 7f8905772740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f88e4010c90 con 0x564e9a5aa800 2024-04-25T00:22:17.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.849+0000 7f8905772740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f88e400deb0 con 0x564e9a5b57f0 2024-04-25T00:22:17.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.850+0000 7f8905772740 10 client.0 _ll_register_callbacks cb 0x7f88e801a790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:22:17.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.850+0000 7f8900f26640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5b57f0 0x564e9a5af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:17.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.851+0000 7f8900f26640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5b57f0 0x564e9a5af690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:37140/0 (socket says 172.21.15.27:37140) 2024-04-25T00:22:17.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.851+0000 7f8900f26640 1 -- 172.21.15.27:0/4014207968 learned_addr learned my addr 172.21.15.27:0/4014207968 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:22:17.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.851+0000 7f88fbfff640 1 --2- 172.21.15.27:0/4014207968 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5aa800 0x564e9a5bdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:17.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.853+0000 7f8900f26640 1 -- 172.21.15.27:0/4014207968 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5aa800 msgr2=0x564e9a5bdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:17.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.853+0000 7f8900f26640 1 --2- 172.21.15.27:0/4014207968 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5aa800 0x564e9a5bdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.853+0000 7f8900f26640 1 -- 172.21.15.27:0/4014207968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5ab120 msgr2=0x564e9a5af110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:17.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.853+0000 7f8900f26640 1 --2- 172.21.15.27:0/4014207968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5ab120 0x564e9a5af110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.853+0000 7f8900f26640 1 -- 172.21.15.27:0/4014207968 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e9a5afec0 con 0x564e9a5b57f0 2024-04-25T00:22:17.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.861+0000 7f8900f26640 1 --2- 172.21.15.27:0/4014207968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5b57f0 0x564e9a5af690 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f88e80407c0 tx=0x7f88e8005220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f82e08f6cbe8cd1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:17.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.862+0000 7f88df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:22:17.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.868+0000 7f88df7fe640 1 -- 172.21.15.27:0/4014207968 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8046070 con 0x564e9a5b57f0 2024-04-25T00:22:17.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.868+0000 7f88df7fe640 1 -- 172.21.15.27:0/4014207968 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f88e8040c30 con 0x564e9a5b57f0 2024-04-25T00:22:17.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.869+0000 7f8905772740 1 -- 172.21.15.27:0/4014207968 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88e8040590 con 0x564e9a5b57f0 2024-04-25T00:22:17.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.875+0000 7f88df7fe640 1 -- 172.21.15.27:0/4014207968 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f88e8053070 con 0x564e9a5b57f0 2024-04-25T00:22:17.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.876+0000 7f8905772740 10 client.26011 fetch_fsmap learned FSMap version 80 2024-04-25T00:22:17.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.877+0000 7f8905772740 1 -- 172.21.15.27:0/4014207968 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=4+}) -- 0x564e9a62c5a0 con 0x564e9a5b57f0 2024-04-25T00:22:17.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.881+0000 7f88df7fe640 1 -- 172.21.15.27:0/4014207968 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88e8040590 con 0x564e9a5b57f0 2024-04-25T00:22:17.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.881+0000 7f88df7fe640 1 -- 172.21.15.27:0/4014207968 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 80) ==== 8362+0+0 (secure 0 0 0) 0x7f88e8062030 con 0x564e9a5b57f0 2024-04-25T00:22:17.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.881+0000 7f8905772740 10 client.26011 fetch_fsmap finished waiting for FSMap version 80 2024-04-25T00:22:17.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.881+0000 7f8905772740 10 client.26011 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:22:17.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.881+0000 7f8905772740 20 client.26011 populate_metadata read hostname 'smithi027' 2024-04-25T00:22:17.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.881+0000 7f8905772740 1 --2- 172.21.15.27:0/4014207968 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564e9a62c960 0x564e9a62ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:17.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.881+0000 7f8905772740 4 client.26011 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:22:17.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.881+0000 7f8905772740 1 -- 172.21.15.27:0/4014207968 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564e9a516f40 con 0x564e9a62c960 2024-04-25T00:22:17.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.884+0000 7f88fbfff640 1 --2- 172.21.15.27:0/4014207968 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564e9a62c960 0x564e9a62ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:17.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.892+0000 7f88fbfff640 1 --2- 172.21.15.27:0/4014207968 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564e9a62c960 0x564e9a62ed80 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34267eb4c5f7f81d server_cookie=52e3b8077d00644a in_seq=0 out_seq=0 2024-04-25T00:22:17.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.892+0000 7f88df7fe640 10 client.26011 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:22:17.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.893+0000 7f88df7fe640 1 -- 172.21.15.27:0/4014207968 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x564e9a516f40 con 0x564e9a62c960 2024-04-25T00:22:17.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.893+0000 7f88df7fe640 10 client.26011 handle_command_reply: tid=0 2024-04-25T00:22:17.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.895+0000 7f8905772740 1 client.26011 shutdown 2024-04-25T00:22:17.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.900+0000 7f8905772740 1 -- 172.21.15.27:0/4014207968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5b57f0 msgr2=0x564e9a5af690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:17.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.900+0000 7f8905772740 1 --2- 172.21.15.27:0/4014207968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5b57f0 0x564e9a5af690 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f88e80407c0 tx=0x7f88e8005220 comp rx=0 tx=0).stop 2024-04-25T00:22:17.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.900+0000 7f8905772740 1 -- 172.21.15.27:0/4014207968 shutdown_connections 2024-04-25T00:22:17.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.900+0000 7f8905772740 1 --2- 172.21.15.27:0/4014207968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e9a5b57f0 0x564e9a5af690 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.900+0000 7f8905772740 1 --2- 172.21.15.27:0/4014207968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e9a5ab120 0x564e9a5af110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.900+0000 7f8905772740 1 --2- 172.21.15.27:0/4014207968 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564e9a62c960 0x564e9a62ed80 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.900+0000 7f8905772740 1 --2- 172.21.15.27:0/4014207968 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e9a5aa800 0x564e9a5bdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.900+0000 7f88df7fe640 0 client.26011 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:22:17.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.900+0000 7f8905772740 1 -- 172.21.15.27:0/4014207968 >> 172.21.15.27:0/4014207968 conn(0x564e9a5a6b20 msgr2=0x564e9a5a5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:17.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.902+0000 7f8905772740 1 -- 172.21.15.27:0/4014207968 shutdown_connections 2024-04-25T00:22:17.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.902+0000 7f8905772740 1 -- 172.21.15.27:0/4014207968 wait complete. 2024-04-25T00:22:17.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.903+0000 7f8905772740 20 client.26011 trim_cache size 0 max 16384 2024-04-25T00:22:17.906 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:22:17.906 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "scrub active (201 inodes in the stack)", 2024-04-25T00:22:17.907 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:22:17.907 INFO:teuthology.orchestra.run.smithi027.stdout: "a40f0131-2b37-4eb9-a2bd-6fee5124aebc": { 2024-04-25T00:22:17.907 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:22:17.907 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "a40f0131-2b37-4eb9-a2bd-6fee5124aebc", 2024-04-25T00:22:17.907 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:22:17.907 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:22:17.907 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:22:17.907 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:22:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.906+0000 7f8901727640 1 -- 172.21.15.27:0/3932448007 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f88e00a2f00 msgr2=0x7f88e00a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.906+0000 7f8901727640 1 --2- 172.21.15.27:0/3932448007 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f88e00a2f00 0x7f88e00a53c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f88e8005dc0 tx=0x7f88e8005fd0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.906+0000 7f8901727640 1 -- 172.21.15.27:0/3932448007 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88fc1091c0 msgr2=0x7f88fc118e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.906+0000 7f8901727640 1 --2- 172.21.15.27:0/3932448007 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88fc1091c0 0x7f88fc118e20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00b450 tx=0x7f88ec00b910 comp rx=0 tx=0).stop 2024-04-25T00:22:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.906+0000 7f8901727640 1 -- 172.21.15.27:0/3932448007 shutdown_connections 2024-04-25T00:22:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.906+0000 7f8901727640 1 --2- 172.21.15.27:0/3932448007 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f88e00a2f00 0x7f88e00a53c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.906+0000 7f8901727640 1 --2- 172.21.15.27:0/3932448007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88fc10fde0 0x7f88fc119360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.906+0000 7f8901727640 1 --2- 172.21.15.27:0/3932448007 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88fc1091c0 0x7f88fc118e20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.906+0000 7f8901727640 1 --2- 172.21.15.27:0/3932448007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88fc108740 0x7f88fc11cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.907+0000 7f8901727640 1 -- 172.21.15.27:0/3932448007 >> 172.21.15.27:0/3932448007 conn(0x7f88fc06c6c0 msgr2=0x7f88fc070400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:17.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.909+0000 7f8901727640 1 -- 172.21.15.27:0/3932448007 shutdown_connections 2024-04-25T00:22:17.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:17.909+0000 7f8901727640 1 -- 172.21.15.27:0/3932448007 wait complete. 2024-04-25T00:22:17.929 INFO:tasks.cephfs.filesystem:scrub status for tag:a40f0131-2b37-4eb9-a2bd-6fee5124aebc - {'path': '/', 'tag': 'a40f0131-2b37-4eb9-a2bd-6fee5124aebc', 'options': 'recursive,force'} 2024-04-25T00:22:19.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:18 smithi139 ceph-mon[32833]: pgmap v199: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 16 MiB/s wr, 3.89k op/s 2024-04-25T00:22:19.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:18 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 258.215924 secs 2024-04-25T00:22:19.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:18 smithi153 ceph-mon[32896]: pgmap v199: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 16 MiB/s wr, 3.89k op/s 2024-04-25T00:22:19.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:18 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 258.215924 secs 2024-04-25T00:22:19.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:18 smithi027 ceph-mon[29208]: pgmap v199: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 16 MiB/s wr, 3.89k op/s 2024-04-25T00:22:19.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:18 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 258.215924 secs 2024-04-25T00:22:20.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:19 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:20.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:19 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:20.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:19 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:20.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:19 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:20.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:19 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:20.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:19 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:21.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:20 smithi139 ceph-mon[32833]: pgmap v200: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 19 MiB/s wr, 4.63k op/s 2024-04-25T00:22:21.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:20 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:21.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:20 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:21.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:20 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:21.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:20 smithi153 ceph-mon[32896]: pgmap v200: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 19 MiB/s wr, 4.63k op/s 2024-04-25T00:22:21.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:20 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:21.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:20 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:21.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:20 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:21.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:20 smithi027 ceph-mon[29208]: pgmap v200: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 19 MiB/s wr, 4.63k op/s 2024-04-25T00:22:21.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:20 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:21.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:20 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:21.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:20 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:23.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:22 smithi153 ceph-mon[32896]: pgmap v201: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 17 MiB/s wr, 4.33k op/s 2024-04-25T00:22:23.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:22 smithi027 ceph-mon[29208]: pgmap v201: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 17 MiB/s wr, 4.33k op/s 2024-04-25T00:22:23.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:22 smithi139 ceph-mon[32833]: pgmap v201: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 17 MiB/s wr, 4.33k op/s 2024-04-25T00:22:24.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:23 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:24.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:23 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:24.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:23 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 263.215993 secs 2024-04-25T00:22:24.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:23 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:24.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:23 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:24.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:23 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 263.215993 secs 2024-04-25T00:22:24.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:23 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:24.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:23 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:24.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:23 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 263.215993 secs 2024-04-25T00:22:25.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:24 smithi153 ceph-mon[32896]: pgmap v202: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 17 MiB/s wr, 4.19k op/s 2024-04-25T00:22:25.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:24 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:25.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:24 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:25.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:24 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:25.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:24 smithi027 ceph-mon[29208]: pgmap v202: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 17 MiB/s wr, 4.19k op/s 2024-04-25T00:22:25.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:24 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:25.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:24 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:25.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:24 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:25.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:24 smithi139 ceph-mon[32833]: pgmap v202: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 17 MiB/s wr, 4.19k op/s 2024-04-25T00:22:25.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:24 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:25.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:24 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:25.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:24 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:26.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:25 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 281 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:25 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 281 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:26.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:25 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 281 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:27.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:26 smithi027 ceph-mon[29208]: pgmap v203: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 17 MiB/s wr, 4.34k op/s 2024-04-25T00:22:27.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:26 smithi139 ceph-mon[32833]: pgmap v203: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 17 MiB/s wr, 4.34k op/s 2024-04-25T00:22:27.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:26 smithi153 ceph-mon[32896]: pgmap v203: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 17 MiB/s wr, 4.34k op/s 2024-04-25T00:22:29.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:28 smithi027 ceph-mon[29208]: pgmap v204: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 13 MiB/s wr, 3.22k op/s 2024-04-25T00:22:29.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:28 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 268.216062 secs 2024-04-25T00:22:29.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:28 smithi139 ceph-mon[32833]: pgmap v204: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 13 MiB/s wr, 3.22k op/s 2024-04-25T00:22:29.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:28 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 268.216062 secs 2024-04-25T00:22:29.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:28 smithi153 ceph-mon[32896]: pgmap v204: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 13 MiB/s wr, 3.22k op/s 2024-04-25T00:22:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:28 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 268.216062 secs 2024-04-25T00:22:31.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:30 smithi027 ceph-mon[29208]: pgmap v205: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 15 MiB/s wr, 3.84k op/s 2024-04-25T00:22:31.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:30 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 291 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:31.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:30 smithi139 ceph-mon[32833]: pgmap v205: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 15 MiB/s wr, 3.84k op/s 2024-04-25T00:22:31.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:30 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 291 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:31.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:30 smithi153 ceph-mon[32896]: pgmap v205: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 15 MiB/s wr, 3.84k op/s 2024-04-25T00:22:31.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:30 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 291 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:32.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:31 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:32.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:31 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:32.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:31 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:32.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:31 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:32.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:31 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:32.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:31 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:33.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:32 smithi027 ceph-mon[29208]: pgmap v206: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 16 MiB/s wr, 3.96k op/s 2024-04-25T00:22:33.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:32 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:33.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:32 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:33.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:32 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:33.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:32 smithi139 ceph-mon[32833]: pgmap v206: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 16 MiB/s wr, 3.96k op/s 2024-04-25T00:22:33.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:32 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:33.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:32 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:33.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:32 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:33.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:32 smithi153 ceph-mon[32896]: pgmap v206: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 16 MiB/s wr, 3.96k op/s 2024-04-25T00:22:33.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:32 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:33.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:32 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:33.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:32 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:34.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:33 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 273.216125 secs 2024-04-25T00:22:34.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:33 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 273.216125 secs 2024-04-25T00:22:34.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:33 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 273.216125 secs 2024-04-25T00:22:35.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:34 smithi027 ceph-mon[29208]: pgmap v207: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 16 MiB/s wr, 4.06k op/s 2024-04-25T00:22:35.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:34 smithi139 ceph-mon[32833]: pgmap v207: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 16 MiB/s wr, 4.06k op/s 2024-04-25T00:22:35.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:34 smithi153 ceph-mon[32896]: pgmap v207: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 16 MiB/s wr, 4.06k op/s 2024-04-25T00:22:36.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:35 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:36.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:35 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:36.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:35 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:36.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:35 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:36.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:35 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:36.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:35 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:37.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:36 smithi027 ceph-mon[29208]: pgmap v208: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 17 MiB/s wr, 4.34k op/s 2024-04-25T00:22:37.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:36 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 297 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:37.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:36 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:37.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:36 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:37.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:36 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:37.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:36 smithi139 ceph-mon[32833]: pgmap v208: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 17 MiB/s wr, 4.34k op/s 2024-04-25T00:22:37.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:36 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 297 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:37.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:36 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:37.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:36 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:37.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:36 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:37.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:36 smithi153 ceph-mon[32896]: pgmap v208: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 17 MiB/s wr, 4.34k op/s 2024-04-25T00:22:37.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:36 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 297 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:37.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:36 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:37.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:36 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:37.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:36 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:38.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:37 smithi027 ceph-mon[29208]: pgmap v209: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 15 MiB/s wr, 3.68k op/s 2024-04-25T00:22:38.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:37 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:38.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:37 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:38.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:37 smithi139 ceph-mon[32833]: pgmap v209: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 15 MiB/s wr, 3.68k op/s 2024-04-25T00:22:38.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:37 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:38.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:37 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:38.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:37 smithi153 ceph-mon[32896]: pgmap v209: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 15 MiB/s wr, 3.68k op/s 2024-04-25T00:22:38.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:37 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:38.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:37 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:39.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:38 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 278.216190 secs 2024-04-25T00:22:39.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:38 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 278.216190 secs 2024-04-25T00:22:39.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:38 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 278.216190 secs 2024-04-25T00:22:40.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:39 smithi027 ceph-mon[29208]: pgmap v210: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 17 MiB/s wr, 4.30k op/s 2024-04-25T00:22:40.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:39 smithi139 ceph-mon[32833]: pgmap v210: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 17 MiB/s wr, 4.30k op/s 2024-04-25T00:22:40.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:39 smithi153 ceph-mon[32896]: pgmap v210: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 17 MiB/s wr, 4.30k op/s 2024-04-25T00:22:41.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:40 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:41.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:40 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:41.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:40 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:41.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:40 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:41.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:40 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:41.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:40 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:41.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:40 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:41.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:40 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:41.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:40 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:42.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:41 smithi139 ceph-mon[32833]: pgmap v211: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 16 MiB/s wr, 4.03k op/s 2024-04-25T00:22:42.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:41 smithi153 ceph-mon[32896]: pgmap v211: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 16 MiB/s wr, 4.03k op/s 2024-04-25T00:22:42.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:41 smithi027 ceph-mon[29208]: pgmap v211: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 16 MiB/s wr, 4.03k op/s 2024-04-25T00:22:43.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:42 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:43.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:42 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:43.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:42 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:43.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:42 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:43.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:42 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:43.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:42 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:44.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:43 smithi139 ceph-mon[32833]: pgmap v212: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 13 MiB/s wr, 3.33k op/s 2024-04-25T00:22:44.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:43 smithi139 ceph-mon[32833]: 1 slow requests, 0 included below; oldest blocked for > 283.216293 secs 2024-04-25T00:22:44.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:43 smithi153 ceph-mon[32896]: pgmap v212: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 13 MiB/s wr, 3.33k op/s 2024-04-25T00:22:44.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:43 smithi153 ceph-mon[32896]: 1 slow requests, 0 included below; oldest blocked for > 283.216293 secs 2024-04-25T00:22:44.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:43 smithi027 ceph-mon[29208]: pgmap v212: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 13 MiB/s wr, 3.33k op/s 2024-04-25T00:22:44.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:43 smithi027 ceph-mon[29208]: 1 slow requests, 0 included below; oldest blocked for > 283.216293 secs 2024-04-25T00:22:45.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:44 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:45.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:44 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:45.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:44 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:45.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:44 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:45.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:44 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:45.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:44 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:45.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:44 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:45.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:44 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:45.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:44 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:46.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:45 smithi139 ceph-mon[32833]: pgmap v213: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 13 MiB/s wr, 3.22k op/s 2024-04-25T00:22:46.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:45 smithi139 ceph-mon[32833]: Health check update: 0 slow ops, oldest one blocked for 302 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:46.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:45 smithi153 ceph-mon[32896]: pgmap v213: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 13 MiB/s wr, 3.22k op/s 2024-04-25T00:22:46.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:45 smithi153 ceph-mon[32896]: Health check update: 0 slow ops, oldest one blocked for 302 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:46.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:45 smithi027 ceph-mon[29208]: pgmap v213: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 13 MiB/s wr, 3.22k op/s 2024-04-25T00:22:46.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:45 smithi027 ceph-mon[29208]: Health check update: 0 slow ops, oldest one blocked for 302 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-25T00:22:47.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:46 smithi139 ceph-mon[32833]: Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 302 sec, osd.9 has slow ops) 2024-04-25T00:22:47.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:46 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:22:47.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:46 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:22:47.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:46 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:46 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:46 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:22:47.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:46 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:46 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:22:47.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:46 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:46 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:22:47.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:46 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:46 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:22:47.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:46 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:46 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:22:47.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:46 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:46 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:46 smithi153 ceph-mon[32896]: Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 302 sec, osd.9 has slow ops) 2024-04-25T00:22:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:46 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:22:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:46 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:22:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:46 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:46 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:46 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:22:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:46 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:46 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:22:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:46 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:46 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:22:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:46 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:46 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:22:47.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:46 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:46 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:22:47.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:46 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:46 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:46 smithi027 ceph-mon[29208]: Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 302 sec, osd.9 has slow ops) 2024-04-25T00:22:47.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:46 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:22:47.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:46 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:22:47.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:46 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:46 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:46 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:22:47.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:46 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:46 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:22:47.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:46 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:46 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:22:47.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:46 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:46 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:22:47.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:46 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:46 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:22:47.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:46 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:46 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:47.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:22:47.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:22:48.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.051+0000 7f65fda9a640 1 -- 172.21.15.27:0/4116170992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65f8100fd0 msgr2=0x7f65f810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.051+0000 7f65fda9a640 1 --2- 172.21.15.27:0/4116170992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65f8100fd0 0x7f65f810d4e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009930 tx=0x7f65f002a420 comp rx=0 tx=0).stop 2024-04-25T00:22:48.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.051+0000 7f65fda9a640 1 -- 172.21.15.27:0/4116170992 shutdown_connections 2024-04-25T00:22:48.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.051+0000 7f65fda9a640 1 --2- 172.21.15.27:0/4116170992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65f8100fd0 0x7f65f810d4e0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.051+0000 7f65fda9a640 1 --2- 172.21.15.27:0/4116170992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65f8100610 0x7f65f8100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.051+0000 7f65fda9a640 1 --2- 172.21.15.27:0/4116170992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65f81066b0 0x7f65f8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.051+0000 7f65fda9a640 1 -- 172.21.15.27:0/4116170992 >> 172.21.15.27:0/4116170992 conn(0x7f65f8077740 msgr2=0x7f65f8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:48.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.051+0000 7f65fda9a640 1 -- 172.21.15.27:0/4116170992 shutdown_connections 2024-04-25T00:22:48.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.051+0000 7f65fda9a640 1 -- 172.21.15.27:0/4116170992 wait complete. 2024-04-25T00:22:48.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.051+0000 7f65fda9a640 1 Processor -- start 2024-04-25T00:22:48.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65fda9a640 1 -- start start 2024-04-25T00:22:48.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65fda9a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65f8100610 0x7f65f8197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65fda9a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65f8100fd0 0x7f65f81984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65fda9a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65f81066b0 0x7f65f819c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65fda9a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f65f81109f0 con 0x7f65f81066b0 2024-04-25T00:22:48.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65fda9a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f65f8110870 con 0x7f65f8100610 2024-04-25T00:22:48.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65fda9a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f65f8110b70 con 0x7f65f8100fd0 2024-04-25T00:22:48.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65f81066b0 0x7f65f819c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65f81066b0 0x7f65f819c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34480/0 (socket says 172.21.15.27:34480) 2024-04-25T00:22:48.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65f7fff640 1 -- 172.21.15.27:0/3382645292 learned_addr learned my addr 172.21.15.27:0/3382645292 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:22:48.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65f77fe640 1 --2- 172.21.15.27:0/3382645292 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65f8100610 0x7f65f8197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65f6ffd640 1 --2- 172.21.15.27:0/3382645292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65f8100fd0 0x7f65f81984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65f7fff640 1 -- 172.21.15.27:0/3382645292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65f8100fd0 msgr2=0x7f65f81984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65f7fff640 1 --2- 172.21.15.27:0/3382645292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65f8100fd0 0x7f65f81984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65f7fff640 1 -- 172.21.15.27:0/3382645292 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65f8100610 msgr2=0x7f65f8197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65f7fff640 1 --2- 172.21.15.27:0/3382645292 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65f8100610 0x7f65f8197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65f7fff640 1 -- 172.21.15.27:0/3382645292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f819cf30 con 0x7f65f81066b0 2024-04-25T00:22:48.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.052+0000 7f65f7fff640 1 --2- 172.21.15.27:0/3382645292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65f81066b0 0x7f65f819c800 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f65f0005ab0 tx=0x7f65f00040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:48.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.053+0000 7f65f4ff9640 1 -- 172.21.15.27:0/3382645292 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f003f030 con 0x7f65f81066b0 2024-04-25T00:22:48.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.053+0000 7f65f4ff9640 1 -- 172.21.15.27:0/3382645292 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f00044c0 con 0x7f65f81066b0 2024-04-25T00:22:48.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.053+0000 7f65fda9a640 1 -- 172.21.15.27:0/3382645292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f819d1c0 con 0x7f65f81066b0 2024-04-25T00:22:48.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.053+0000 7f65f4ff9640 1 -- 172.21.15.27:0/3382645292 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f0035070 con 0x7f65f81066b0 2024-04-25T00:22:48.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.053+0000 7f65fda9a640 1 -- 172.21.15.27:0/3382645292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f81a4ab0 con 0x7f65f81066b0 2024-04-25T00:22:48.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.055+0000 7f65f4ff9640 1 -- 172.21.15.27:0/3382645292 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f65f0042030 con 0x7f65f81066b0 2024-04-25T00:22:48.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.055+0000 7f66007a7740 1 Processor -- start 2024-04-25T00:22:48.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.055+0000 7f66007a7740 1 -- start start 2024-04-25T00:22:48.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f66007a7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b0c530 0x55a7d8b0c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f66007a7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0ce50 0x55a7d8b1b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f66007a7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b1de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f65f4ff9640 1 --2- 172.21.15.27:0/3382645292 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f65e40a2d00 0x7f65e40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f66007a7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a7d8a716b0 con 0x55a7d8b0c530 2024-04-25T00:22:48.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f66007a7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a7d8a19880 con 0x55a7d8b0ce50 2024-04-25T00:22:48.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f66007a7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a7d8a6d670 con 0x55a7d8b1ba30 2024-04-25T00:22:48.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f65f6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b0c530 0x55a7d8b0c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f65f6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b0c530 0x55a7d8b0c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34496/0 (socket says 172.21.15.27:34496) 2024-04-25T00:22:48.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f65f6ffd640 1 -- 172.21.15.27:0/3079345747 learned_addr learned my addr 172.21.15.27:0/3079345747 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:22:48.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f65f6ffd640 1 --2- 172.21.15.27:0/3382645292 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f65e40a2d00 0x7f65e40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f65f4ff9640 1 -- 172.21.15.27:0/3382645292 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f65f00efbf0 con 0x7f65f81066b0 2024-04-25T00:22:48.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f65f77fe640 1 --2- 172.21.15.27:0/3079345747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0ce50 0x55a7d8b1b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f65f7fff640 1 --2- 172.21.15.27:0/3079345747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b1de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f65f6ffd640 1 -- 172.21.15.27:0/3079345747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b1ba30 msgr2=0x55a7d8b1de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:48.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f65f6ffd640 1 --2- 172.21.15.27:0/3079345747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b1de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f65f6ffd640 1 -- 172.21.15.27:0/3079345747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0ce50 msgr2=0x55a7d8b1b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:48.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f65f6ffd640 1 --2- 172.21.15.27:0/3079345747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0ce50 0x55a7d8b1b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f65f6ffd640 1 -- 172.21.15.27:0/3079345747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7d884f7d0 con 0x55a7d8b0c530 2024-04-25T00:22:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f65f77fe640 1 --2- 172.21.15.27:0/3079345747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0ce50 0x55a7d8b1b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:22:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f65f6ffd640 1 --2- 172.21.15.27:0/3079345747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b0c530 0x55a7d8b0c910 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f65e0014cf0 tx=0x7f65e00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f961c3ed1f4ee3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f65f6ffd640 1 --2- 172.21.15.27:0/3382645292 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f65e40a2d00 0x7f65e40a51c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f65e0011cb0 tx=0x7f65e0016000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.056+0000 7f65edffb640 1 -- 172.21.15.27:0/3079345747 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e0022070 con 0x55a7d8b0c530 2024-04-25T00:22:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.057+0000 7f65edffb640 1 -- 172.21.15.27:0/3079345747 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65e0005500 con 0x55a7d8b0c530 2024-04-25T00:22:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.057+0000 7f65edffb640 1 -- 172.21.15.27:0/3079345747 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e0004ba0 con 0x55a7d8b0c530 2024-04-25T00:22:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.057+0000 7f66007a7740 1 -- 172.21.15.27:0/3079345747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b0c530 msgr2=0x55a7d8b0c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.057+0000 7f66007a7740 1 --2- 172.21.15.27:0/3079345747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b0c530 0x55a7d8b0c910 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f65e0014cf0 tx=0x7f65e00117b0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.057+0000 7f66007a7740 1 -- 172.21.15.27:0/3079345747 shutdown_connections 2024-04-25T00:22:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.057+0000 7f66007a7740 1 --2- 172.21.15.27:0/3079345747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b1de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.057+0000 7f66007a7740 1 --2- 172.21.15.27:0/3079345747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0ce50 0x55a7d8b1b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.057+0000 7f66007a7740 1 --2- 172.21.15.27:0/3079345747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b0c530 0x55a7d8b0c910 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.057+0000 7f66007a7740 1 -- 172.21.15.27:0/3079345747 >> 172.21.15.27:0/3079345747 conn(0x55a7d8b08950 msgr2=0x55a7d8b08d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.057+0000 7f66007a7740 1 -- 172.21.15.27:0/3079345747 shutdown_connections 2024-04-25T00:22:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.057+0000 7f66007a7740 1 -- 172.21.15.27:0/3079345747 wait complete. 2024-04-25T00:22:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.058+0000 7f66007a7740 1 -- start start 2024-04-25T00:22:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.058+0000 7f66007a7740 1 Processor -- start 2024-04-25T00:22:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.058+0000 7f66007a7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0c530 0x55a7d8b18a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.058+0000 7f66007a7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b0ce50 0x55a7d8b18fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.058+0000 7f66007a7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b19510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.058+0000 7f66007a7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a7d8b1e790 con 0x55a7d8b1ba30 2024-04-25T00:22:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.058+0000 7f66007a7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a7d8a2cf50 con 0x55a7d8b0c530 2024-04-25T00:22:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.058+0000 7f66007a7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a7d8b1e910 con 0x55a7d8b0ce50 2024-04-25T00:22:48.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.058+0000 7f66007a7740 10 client.0 _ll_register_callbacks cb 0x55a7d8a68000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:22:48.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.058+0000 7f65f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b19510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.058+0000 7f65f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b19510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34508/0 (socket says 172.21.15.27:34508) 2024-04-25T00:22:48.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.058+0000 7f65f7fff640 1 -- 172.21.15.27:0/304570694 learned_addr learned my addr 172.21.15.27:0/304570694 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:22:48.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.058+0000 7f65f6ffd640 1 --2- 172.21.15.27:0/304570694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b0ce50 0x55a7d8b18fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.058+0000 7f65f77fe640 1 --2- 172.21.15.27:0/304570694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0c530 0x55a7d8b18a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.059+0000 7f65f7fff640 1 -- 172.21.15.27:0/304570694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b0ce50 msgr2=0x55a7d8b18fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:48.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.059+0000 7f65f7fff640 1 --2- 172.21.15.27:0/304570694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b0ce50 0x55a7d8b18fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.059+0000 7f65f7fff640 1 -- 172.21.15.27:0/304570694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0c530 msgr2=0x55a7d8b18a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:48.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.059+0000 7f65f7fff640 1 --2- 172.21.15.27:0/304570694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0c530 0x55a7d8b18a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.059+0000 7f65f7fff640 1 -- 172.21.15.27:0/304570694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7d87853f0 con 0x55a7d8b1ba30 2024-04-25T00:22:48.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.059+0000 7f65f7fff640 1 --2- 172.21.15.27:0/304570694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b19510 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f65f00b8770 tx=0x7f65f00b8170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed94da9b58e88949 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:48.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.059+0000 7f65d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:22:48.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.059+0000 7f65d6ffd640 1 -- 172.21.15.27:0/304570694 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f0049040 con 0x55a7d8b1ba30 2024-04-25T00:22:48.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.059+0000 7f65d6ffd640 1 -- 172.21.15.27:0/304570694 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f65f00ed250 con 0x55a7d8b1ba30 2024-04-25T00:22:48.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.059+0000 7f66007a7740 1 -- 172.21.15.27:0/304570694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65e0005500 con 0x55a7d8b1ba30 2024-04-25T00:22:48.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.059+0000 7f65d6ffd640 1 -- 172.21.15.27:0/304570694 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f65f00ed450 con 0x55a7d8b1ba30 2024-04-25T00:22:48.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.059+0000 7f66007a7740 10 client.16542 fetch_fsmap learned FSMap version 92 2024-04-25T00:22:48.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.059+0000 7f66007a7740 1 -- 172.21.15.27:0/304570694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=4+}) -- 0x55a7d87450f0 con 0x55a7d8b1ba30 2024-04-25T00:22:48.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.059+0000 7f65d6ffd640 1 -- 172.21.15.27:0/304570694 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65e0005500 con 0x55a7d8b1ba30 2024-04-25T00:22:48.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.060+0000 7f65d6ffd640 1 -- 172.21.15.27:0/304570694 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 92) ==== 8362+0+0 (secure 0 0 0) 0x7f65f00bb1f0 con 0x55a7d8b1ba30 2024-04-25T00:22:48.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.060+0000 7f66007a7740 10 client.16542 fetch_fsmap finished waiting for FSMap version 92 2024-04-25T00:22:48.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.060+0000 7f66007a7740 10 client.16542 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:22:48.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.060+0000 7f66007a7740 20 client.16542 populate_metadata read hostname 'smithi027' 2024-04-25T00:22:48.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.060+0000 7f66007a7740 1 --2- 172.21.15.27:0/304570694 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55a7d8b2bcf0 0x55a7d8b2e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.060+0000 7f66007a7740 4 client.16542 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:22:48.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.060+0000 7f66007a7740 1 -- 172.21.15.27:0/304570694 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7d8a7ebf0 con 0x55a7d8b2bcf0 2024-04-25T00:22:48.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.060+0000 7f65f77fe640 1 --2- 172.21.15.27:0/304570694 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55a7d8b2bcf0 0x55a7d8b2e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.060+0000 7f65d6ffd640 10 client.16542 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:22:48.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.060+0000 7f65f77fe640 1 --2- 172.21.15.27:0/304570694 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55a7d8b2bcf0 0x55a7d8b2e130 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0f0349ab05d9a8e server_cookie=a0199d8fe30aab64 in_seq=0 out_seq=0 2024-04-25T00:22:48.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.061+0000 7f65d6ffd640 1 -- 172.21.15.27:0/304570694 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a7d8a7ebf0 con 0x55a7d8b2bcf0 2024-04-25T00:22:48.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.061+0000 7f65d6ffd640 10 client.16542 handle_command_reply: tid=0 2024-04-25T00:22:48.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.061+0000 7f66007a7740 1 client.16542 shutdown 2024-04-25T00:22:48.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.061+0000 7f66007a7740 1 -- 172.21.15.27:0/304570694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 msgr2=0x55a7d8b19510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.061+0000 7f66007a7740 1 --2- 172.21.15.27:0/304570694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b19510 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f65f00b8770 tx=0x7f65f00b8170 comp rx=0 tx=0).stop 2024-04-25T00:22:48.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.061+0000 7f66007a7740 1 -- 172.21.15.27:0/304570694 shutdown_connections 2024-04-25T00:22:48.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.061+0000 7f66007a7740 1 --2- 172.21.15.27:0/304570694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b19510 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.061+0000 7f66007a7740 1 --2- 172.21.15.27:0/304570694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b0ce50 0x55a7d8b18fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.061+0000 7f66007a7740 1 --2- 172.21.15.27:0/304570694 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55a7d8b2bcf0 0x55a7d8b2e130 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.061+0000 7f66007a7740 1 --2- 172.21.15.27:0/304570694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0c530 0x55a7d8b18a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.061+0000 7f66007a7740 1 -- 172.21.15.27:0/304570694 >> 172.21.15.27:0/304570694 conn(0x55a7d8b08950 msgr2=0x55a7d8b0e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:48.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.061+0000 7f65d6ffd640 0 client.16542 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:22:48.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.062+0000 7f66007a7740 1 -- 172.21.15.27:0/304570694 shutdown_connections 2024-04-25T00:22:48.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.062+0000 7f66007a7740 1 -- 172.21.15.27:0/304570694 wait complete. 2024-04-25T00:22:48.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.062+0000 7f66007a7740 20 client.16542 trim_cache size 0 max 16384 2024-04-25T00:22:48.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.079+0000 7f66007a7740 1 Processor -- start 2024-04-25T00:22:48.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.079+0000 7f66007a7740 1 -- start start 2024-04-25T00:22:48.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f66007a7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0c530 0x55a7d8b27650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f66007a7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b0ce50 0x55a7d8b19330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f66007a7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b0e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f66007a7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f65f81ac890 con 0x55a7d8b1ba30 2024-04-25T00:22:48.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f66007a7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f65f81ae3d0 con 0x55a7d8b0c530 2024-04-25T00:22:48.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f66007a7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f65f81acf60 con 0x55a7d8b0ce50 2024-04-25T00:22:48.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f65f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b0e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f65f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b0e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34524/0 (socket says 172.21.15.27:34524) 2024-04-25T00:22:48.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f65f7fff640 1 -- 172.21.15.27:0/1750159775 learned_addr learned my addr 172.21.15.27:0/1750159775 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:22:48.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f65f6ffd640 1 --2- 172.21.15.27:0/1750159775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b0ce50 0x55a7d8b19330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f65f7fff640 1 -- 172.21.15.27:0/1750159775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b0ce50 msgr2=0x55a7d8b19330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:48.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f65f7fff640 1 --2- 172.21.15.27:0/1750159775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b0ce50 0x55a7d8b19330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f65f7fff640 1 -- 172.21.15.27:0/1750159775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0c530 msgr2=0x55a7d8b27650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:48.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f65f7fff640 1 --2- 172.21.15.27:0/1750159775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0c530 0x55a7d8b27650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f65f7fff640 1 -- 172.21.15.27:0/1750159775 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7d8b0cc80 con 0x55a7d8b1ba30 2024-04-25T00:22:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f65f7fff640 1 --2- 172.21.15.27:0/1750159775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b0e600 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f65f00b8200 tx=0x7f65f0009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4330a401535f78 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f65ed7fa640 1 -- 172.21.15.27:0/1750159775 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f00c4a10 con 0x55a7d8b1ba30 2024-04-25T00:22:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f65ed7fa640 1 -- 172.21.15.27:0/1750159775 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f003ba70 con 0x55a7d8b1ba30 2024-04-25T00:22:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.080+0000 7f65ed7fa640 1 -- 172.21.15.27:0/1750159775 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f00b8410 con 0x55a7d8b1ba30 2024-04-25T00:22:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.081+0000 7f66007a7740 1 -- 172.21.15.27:0/1750159775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 msgr2=0x55a7d8b0e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.081+0000 7f66007a7740 1 --2- 172.21.15.27:0/1750159775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b0e600 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f65f00b8200 tx=0x7f65f0009560 comp rx=0 tx=0).stop 2024-04-25T00:22:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.081+0000 7f66007a7740 1 -- 172.21.15.27:0/1750159775 shutdown_connections 2024-04-25T00:22:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.081+0000 7f66007a7740 1 --2- 172.21.15.27:0/1750159775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b0e600 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.081+0000 7f66007a7740 1 --2- 172.21.15.27:0/1750159775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b0ce50 0x55a7d8b19330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.081+0000 7f66007a7740 1 --2- 172.21.15.27:0/1750159775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0c530 0x55a7d8b27650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.081+0000 7f66007a7740 1 -- 172.21.15.27:0/1750159775 >> 172.21.15.27:0/1750159775 conn(0x55a7d8b08950 msgr2=0x55a7d8b0def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.081+0000 7f66007a7740 1 -- 172.21.15.27:0/1750159775 shutdown_connections 2024-04-25T00:22:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.081+0000 7f66007a7740 1 -- 172.21.15.27:0/1750159775 wait complete. 2024-04-25T00:22:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.081+0000 7f66007a7740 1 -- start start 2024-04-25T00:22:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f66007a7740 1 Processor -- start 2024-04-25T00:22:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f66007a7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b0c530 0x55a7d8b23b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f66007a7740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0ce50 0x55a7d8b240a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f66007a7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b15290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f66007a7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f65f81add00 con 0x55a7d8b1ba30 2024-04-25T00:22:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f66007a7740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f65f81ad630 con 0x55a7d8b0ce50 2024-04-25T00:22:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f66007a7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f65f81aa850 con 0x55a7d8b0c530 2024-04-25T00:22:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f66007a7740 10 client.0 _ll_register_callbacks cb 0x7f65f0005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:22:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f65f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b15290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f65f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b15290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34532/0 (socket says 172.21.15.27:34532) 2024-04-25T00:22:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f65f7fff640 1 -- 172.21.15.27:0/317891851 learned_addr learned my addr 172.21.15.27:0/317891851 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:22:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f65f6ffd640 1 --2- 172.21.15.27:0/317891851 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0ce50 0x55a7d8b240a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f65f7fff640 1 -- 172.21.15.27:0/317891851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b0c530 msgr2=0x55a7d8b23b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f65f7fff640 1 --2- 172.21.15.27:0/317891851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b0c530 0x55a7d8b23b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f65f7fff640 1 -- 172.21.15.27:0/317891851 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0ce50 msgr2=0x55a7d8b240a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f65f7fff640 1 --2- 172.21.15.27:0/317891851 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0ce50 0x55a7d8b240a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f65f7fff640 1 -- 172.21.15.27:0/317891851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7d8b15ac0 con 0x55a7d8b1ba30 2024-04-25T00:22:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f65f7fff640 1 --2- 172.21.15.27:0/317891851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b15290 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f65f00c44e0 tx=0x7f65f00092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c9371aeb5cbc88d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.082+0000 7f65d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:22:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.083+0000 7f65d6ffd640 1 -- 172.21.15.27:0/317891851 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f009f060 con 0x55a7d8b1ba30 2024-04-25T00:22:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.083+0000 7f65d6ffd640 1 -- 172.21.15.27:0/317891851 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f65f003bc60 con 0x55a7d8b1ba30 2024-04-25T00:22:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.083+0000 7f66007a7740 1 -- 172.21.15.27:0/317891851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65f003ba70 con 0x55a7d8b1ba30 2024-04-25T00:22:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.083+0000 7f65d6ffd640 1 -- 172.21.15.27:0/317891851 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f65f003be60 con 0x55a7d8b1ba30 2024-04-25T00:22:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.083+0000 7f66007a7740 10 client.16548 fetch_fsmap learned FSMap version 92 2024-04-25T00:22:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.083+0000 7f66007a7740 1 -- 172.21.15.27:0/317891851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=4+}) -- 0x55a7d8b15ac0 con 0x55a7d8b1ba30 2024-04-25T00:22:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.083+0000 7f65d6ffd640 1 -- 172.21.15.27:0/317891851 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f003ba70 con 0x55a7d8b1ba30 2024-04-25T00:22:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.083+0000 7f65d6ffd640 1 -- 172.21.15.27:0/317891851 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 92) ==== 8362+0+0 (secure 0 0 0) 0x7f65f009abc0 con 0x55a7d8b1ba30 2024-04-25T00:22:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.083+0000 7f66007a7740 10 client.16548 fetch_fsmap finished waiting for FSMap version 92 2024-04-25T00:22:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.083+0000 7f66007a7740 10 client.16548 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.083+0000 7f66007a7740 20 client.16548 populate_metadata read hostname 'smithi027' 2024-04-25T00:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.083+0000 7f66007a7740 1 --2- 172.21.15.27:0/317891851 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55a7d8b920b0 0x55a7d8b944a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.083+0000 7f66007a7740 4 client.16548 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.083+0000 7f66007a7740 1 -- 172.21.15.27:0/317891851 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a7d8a78f40 con 0x55a7d8b920b0 2024-04-25T00:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.084+0000 7f65f77fe640 1 --2- 172.21.15.27:0/317891851 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55a7d8b920b0 0x55a7d8b944a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.084+0000 7f65d6ffd640 10 client.16548 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.084+0000 7f65f77fe640 1 --2- 172.21.15.27:0/317891851 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55a7d8b920b0 0x55a7d8b944a0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aec5489bf931918e server_cookie=18ff1f0746a029f6 in_seq=0 out_seq=0 2024-04-25T00:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.084+0000 7f65d6ffd640 1 -- 172.21.15.27:0/317891851 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a7d8a78f40 con 0x55a7d8b920b0 2024-04-25T00:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.084+0000 7f65d6ffd640 10 client.16548 handle_command_reply: tid=0 2024-04-25T00:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.084+0000 7f66007a7740 1 client.16548 shutdown 2024-04-25T00:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.084+0000 7f66007a7740 1 -- 172.21.15.27:0/317891851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 msgr2=0x55a7d8b15290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.084+0000 7f66007a7740 1 --2- 172.21.15.27:0/317891851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b15290 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f65f00c44e0 tx=0x7f65f00092e0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.084+0000 7f66007a7740 1 -- 172.21.15.27:0/317891851 shutdown_connections 2024-04-25T00:22:48.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.084+0000 7f66007a7740 1 --2- 172.21.15.27:0/317891851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a7d8b1ba30 0x55a7d8b15290 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.084+0000 7f66007a7740 1 --2- 172.21.15.27:0/317891851 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55a7d8b920b0 0x55a7d8b944a0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.084+0000 7f66007a7740 1 --2- 172.21.15.27:0/317891851 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a7d8b0ce50 0x55a7d8b240a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.084+0000 7f66007a7740 1 --2- 172.21.15.27:0/317891851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a7d8b0c530 0x55a7d8b23b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.084+0000 7f65d6ffd640 0 client.16548 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:22:48.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.084+0000 7f66007a7740 1 -- 172.21.15.27:0/317891851 >> 172.21.15.27:0/317891851 conn(0x55a7d8b08950 msgr2=0x55a7d8b07af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:48.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.084+0000 7f66007a7740 1 -- 172.21.15.27:0/317891851 shutdown_connections 2024-04-25T00:22:48.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.084+0000 7f66007a7740 1 -- 172.21.15.27:0/317891851 wait complete. 2024-04-25T00:22:48.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.085+0000 7f66007a7740 20 client.16548 trim_cache size 0 max 16384 2024-04-25T00:22:48.095 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:22:48.096 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T00:22:48.096 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:22:48.096 INFO:teuthology.orchestra.run.smithi027.stdout: "a40f0131-2b37-4eb9-a2bd-6fee5124aebc": { 2024-04-25T00:22:48.096 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:22:48.096 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "a40f0131-2b37-4eb9-a2bd-6fee5124aebc", 2024-04-25T00:22:48.096 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:22:48.096 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:22:48.096 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:22:48.096 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:22:48.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.085+0000 7f65fda9a640 1 -- 172.21.15.27:0/3382645292 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f65e40a2d00 msgr2=0x7f65e40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.085+0000 7f65fda9a640 1 --2- 172.21.15.27:0/3382645292 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f65e40a2d00 0x7f65e40a51c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f65e0011cb0 tx=0x7f65e0016000 comp rx=0 tx=0).stop 2024-04-25T00:22:48.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.085+0000 7f65fda9a640 1 -- 172.21.15.27:0/3382645292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65f81066b0 msgr2=0x7f65f819c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.085+0000 7f65fda9a640 1 --2- 172.21.15.27:0/3382645292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65f81066b0 0x7f65f819c800 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f65f0005ab0 tx=0x7f65f00040c0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.085+0000 7f65fda9a640 1 -- 172.21.15.27:0/3382645292 shutdown_connections 2024-04-25T00:22:48.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.085+0000 7f65fda9a640 1 --2- 172.21.15.27:0/3382645292 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f65e40a2d00 0x7f65e40a51c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.085+0000 7f65fda9a640 1 --2- 172.21.15.27:0/3382645292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65f81066b0 0x7f65f819c800 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.085+0000 7f65fda9a640 1 --2- 172.21.15.27:0/3382645292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65f8100fd0 0x7f65f81984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.085+0000 7f65fda9a640 1 --2- 172.21.15.27:0/3382645292 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65f8100610 0x7f65f8197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.085+0000 7f65fda9a640 1 -- 172.21.15.27:0/3382645292 >> 172.21.15.27:0/3382645292 conn(0x7f65f8077740 msgr2=0x7f65f80ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:48.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.085+0000 7f65fda9a640 1 -- 172.21.15.27:0/3382645292 shutdown_connections 2024-04-25T00:22:48.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:22:48.085+0000 7f65fda9a640 1 -- 172.21.15.27:0/3382645292 wait complete. 2024-04-25T00:22:48.099 INFO:tasks.cephfs.filesystem:scrub status for tag:a40f0131-2b37-4eb9-a2bd-6fee5124aebc - {'path': '/', 'tag': 'a40f0131-2b37-4eb9-a2bd-6fee5124aebc', 'options': 'recursive,force'} 2024-04-25T00:22:48.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:47 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:22:48.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:47 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:48.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:47 smithi139 ceph-mon[32833]: pgmap v214: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 10 MiB/s wr, 2.43k op/s 2024-04-25T00:22:48.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:47 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:22:48.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:47 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:48.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:47 smithi153 ceph-mon[32896]: pgmap v214: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 10 MiB/s wr, 2.43k op/s 2024-04-25T00:22:48.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:47 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:22:48.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:47 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:22:48.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:47 smithi027 ceph-mon[29208]: pgmap v214: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 10 MiB/s wr, 2.43k op/s 2024-04-25T00:22:50.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:50 smithi153 ceph-mon[32896]: pgmap v215: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 9.0 MiB/s wr, 2.19k op/s 2024-04-25T00:22:50.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:50 smithi153 ceph-mon[32896]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 306 secs 2024-04-25T00:22:50.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:50 smithi027 ceph-mon[29208]: pgmap v215: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 9.0 MiB/s wr, 2.19k op/s 2024-04-25T00:22:50.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:50 smithi027 ceph-mon[29208]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 306 secs 2024-04-25T00:22:50.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:50 smithi139 ceph-mon[32833]: pgmap v215: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 9.0 MiB/s wr, 2.19k op/s 2024-04-25T00:22:50.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:50 smithi139 ceph-mon[32833]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 306 secs 2024-04-25T00:22:51.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:51 smithi153 ceph-mon[32896]: Health check update: 2 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-25T00:22:51.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:51 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:51.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:51 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:51.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:51 smithi027 ceph-mon[29208]: Health check update: 2 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-25T00:22:51.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:51 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:51.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:51 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:51 smithi139 ceph-mon[32833]: Health check update: 2 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-25T00:22:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:51 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6836/2477304919,v1:172.21.15.139:6837/2477304919] up:standby-replay 2024-04-25T00:22:51.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:51 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:52.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:52 smithi153 ceph-mon[32896]: pgmap v216: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 6.2 MiB/s wr, 1.49k op/s 2024-04-25T00:22:52.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:52 smithi153 ceph-mon[32896]: MDS health message cleared (mds.?): 1 slow requests are blocked > 180 secs 2024-04-25T00:22:52.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:52 smithi153 ceph-mon[32896]: MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 180 secs, oldest blocked for 303 secs 2024-04-25T00:22:52.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:52 smithi153 ceph-mon[32896]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 306 secs 2024-04-25T00:22:52.654 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:52 smithi139 ceph-mon[32833]: pgmap v216: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 6.2 MiB/s wr, 1.49k op/s 2024-04-25T00:22:52.655 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:52 smithi139 ceph-mon[32833]: MDS health message cleared (mds.?): 1 slow requests are blocked > 180 secs 2024-04-25T00:22:52.655 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:52 smithi139 ceph-mon[32833]: MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 180 secs, oldest blocked for 303 secs 2024-04-25T00:22:52.655 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:52 smithi139 ceph-mon[32833]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 306 secs 2024-04-25T00:22:52.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:52 smithi027 ceph-mon[29208]: pgmap v216: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 6.2 MiB/s wr, 1.49k op/s 2024-04-25T00:22:52.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:52 smithi027 ceph-mon[29208]: MDS health message cleared (mds.?): 1 slow requests are blocked > 180 secs 2024-04-25T00:22:52.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:52 smithi027 ceph-mon[29208]: MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 180 secs, oldest blocked for 303 secs 2024-04-25T00:22:52.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:52 smithi027 ceph-mon[29208]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 306 secs 2024-04-25T00:22:53.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:53 smithi153 ceph-mon[32896]: Health check cleared: MDS_SLOW_METADATA_IO (was: 2 MDSs report slow metadata IOs) 2024-04-25T00:22:53.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:53 smithi153 ceph-mon[32896]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-04-25T00:22:53.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:53 smithi153 ceph-mon[32896]: Cluster is now healthy 2024-04-25T00:22:53.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:53 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:53.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:53 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:53.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:53 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:53.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:53 smithi027 ceph-mon[29208]: Health check cleared: MDS_SLOW_METADATA_IO (was: 2 MDSs report slow metadata IOs) 2024-04-25T00:22:53.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:53 smithi027 ceph-mon[29208]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-04-25T00:22:53.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:53 smithi027 ceph-mon[29208]: Cluster is now healthy 2024-04-25T00:22:53.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:53 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:53.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:53 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:53.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:53 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:53.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:53 smithi139 ceph-mon[32833]: Health check cleared: MDS_SLOW_METADATA_IO (was: 2 MDSs report slow metadata IOs) 2024-04-25T00:22:53.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:53 smithi139 ceph-mon[32833]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-04-25T00:22:53.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:53 smithi139 ceph-mon[32833]: Cluster is now healthy 2024-04-25T00:22:53.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:53 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] up:active 2024-04-25T00:22:53.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:53 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/3286639641,v1:172.21.15.27:6839/3286639641] up:standby-replay 2024-04-25T00:22:53.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:53 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:22:54.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:54 smithi153 ceph-mon[32896]: pgmap v217: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 1.02k op/s 2024-04-25T00:22:54.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:54 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:22:54.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:54 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:22:54.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:54 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:22:54.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:54 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:22:54.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:54 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:22:54.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:54 smithi027 ceph-mon[29208]: pgmap v217: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 1.02k op/s 2024-04-25T00:22:54.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:54 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:22:54.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:54 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:22:54.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:54 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:22:54.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:54 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:22:54.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:54 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:22:54.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:54 smithi139 ceph-mon[32833]: pgmap v217: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 1.02k op/s 2024-04-25T00:22:54.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:54 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:22:54.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:54 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:22:54.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:54 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:22:54.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:54 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:22:54.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:54 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:22:56.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:56 smithi027 ceph-mon[29208]: pgmap v218: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 749 op/s 2024-04-25T00:22:56.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:56 smithi139 ceph-mon[32833]: pgmap v218: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 749 op/s 2024-04-25T00:22:56.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:56 smithi153 ceph-mon[32896]: pgmap v218: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 749 op/s 2024-04-25T00:22:58.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:22:58 smithi027 ceph-mon[29208]: pgmap v219: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 319 KiB/s wr, 97 op/s 2024-04-25T00:22:58.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:22:58 smithi139 ceph-mon[32833]: pgmap v219: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 319 KiB/s wr, 97 op/s 2024-04-25T00:22:58.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:22:58 smithi153 ceph-mon[32896]: pgmap v219: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 319 KiB/s wr, 97 op/s 2024-04-25T00:23:00.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:00 smithi027 ceph-mon[29208]: pgmap v220: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 30 KiB/s wr, 25 op/s 2024-04-25T00:23:00.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:00 smithi139 ceph-mon[32833]: pgmap v220: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 30 KiB/s wr, 25 op/s 2024-04-25T00:23:00.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:00 smithi153 ceph-mon[32896]: pgmap v220: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 30 KiB/s wr, 25 op/s 2024-04-25T00:23:02.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:02 smithi027 ceph-mon[29208]: pgmap v221: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-25T00:23:02.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:02 smithi139 ceph-mon[32833]: pgmap v221: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-25T00:23:02.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:02 smithi153 ceph-mon[32896]: pgmap v221: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-25T00:23:04.636 INFO:tasks.workunit.client.0.smithi027.stdout:Syncing()...21 sec 2024-04-25T00:23:04.637 INFO:tasks.workunit.client.0.smithi027.stdout:FFSB benchmark finished at: Thu Apr 25 00:23:04 2024 2024-04-25T00:23:04.637 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:23:04.637 INFO:tasks.workunit.client.0.smithi027.stdout:Results: 2024-04-25T00:23:04.637 INFO:tasks.workunit.client.0.smithi027.stdout:Benchmark took 329.85 sec 2024-04-25T00:23:04.637 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:23:04.637 INFO:tasks.workunit.client.0.smithi027.stdout:Total Results 2024-04-25T00:23:04.637 INFO:tasks.workunit.client.0.smithi027.stdout:=============== 2024-04-25T00:23:04.637 INFO:tasks.workunit.client.0.smithi027.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-04-25T00:23:04.637 INFO:tasks.workunit.client.0.smithi027.stdout: ======= ============ ========= ======= =========== ========== 2024-04-25T00:23:04.637 INFO:tasks.workunit.client.0.smithi027.stdout: write : 4990720 15130.08 100.000% 100.000% 59.1MB/sec 2024-04-25T00:23:04.638 INFO:tasks.workunit.client.0.smithi027.stdout:- 2024-04-25T00:23:04.638 INFO:tasks.workunit.client.0.smithi027.stdout:15130.08 Transactions per Second 2024-04-25T00:23:04.638 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:23:04.638 INFO:tasks.workunit.client.0.smithi027.stdout:Throughput Results 2024-04-25T00:23:04.638 INFO:tasks.workunit.client.0.smithi027.stdout:=================== 2024-04-25T00:23:04.638 INFO:tasks.workunit.client.0.smithi027.stdout:Write Throughput: 59.1MB/sec 2024-04-25T00:23:04.662 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:23:04.662 INFO:tasks.workunit.client.0.smithi027.stdout:System Call Latency statistics in millisecs 2024-04-25T00:23:04.662 INFO:tasks.workunit.client.0.smithi027.stdout:===== 2024-04-25T00:23:04.662 INFO:tasks.workunit.client.0.smithi027.stdout: Min Avg Max Total Calls 2024-04-25T00:23:04.663 INFO:tasks.workunit.client.0.smithi027.stdout: ======== ======== ======== ============ 2024-04-25T00:23:04.663 INFO:tasks.workunit.client.0.smithi027.stdout:[ open] 0.005000 0.085316 29.016001 3899 2024-04-25T00:23:04.663 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-25T00:23:04.663 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-25T00:23:04.663 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-25T00:23:04.663 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-25T00:23:04.663 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-25T00:23:04.663 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-04-25T00:23:04.663 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-04-25T00:23:04.663 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[7] 20.000000 - 50.000000 : 1 2024-04-25T00:23:04.663 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[8] 10.000000 - 20.000000 : 7 2024-04-25T00:23:04.663 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[9] 5.000000 - 10.000000 : 11 2024-04-25T00:23:04.663 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[10] 2.000000 - 5.000000 : 13 2024-04-25T00:23:04.663 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[11] 1.000000 - 2.000000 : 6 2024-04-25T00:23:04.664 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[12] 0.500000 - 1.000000 : 3 2024-04-25T00:23:04.664 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[13] 0.200000 - 0.500000 : 2 2024-04-25T00:23:04.664 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[14] 0.100000 - 0.200000 : 10 2024-04-25T00:23:04.664 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[15] 0.050000 - 0.100000 : 45 2024-04-25T00:23:04.664 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[16] 0.020000 - 0.050000 : 1741 2024-04-25T00:23:04.664 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[17] 0.010000 - 0.020000 : 1708 2024-04-25T00:23:04.664 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[18] 0.000000 - 0.010000 : 352 2024-04-25T00:23:04.664 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:23:04.664 INFO:tasks.workunit.client.0.smithi027.stdout:[ write] 0.001000 1.952603 34322.289062 4990720 2024-04-25T00:23:04.664 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[0] 5000.000000 - 10000.000000 : 481 2024-04-25T00:23:04.664 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[1] 2000.000000 - 5000.000000 : 313 2024-04-25T00:23:04.664 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[2] 1000.000000 - 2000.000000 : 179 2024-04-25T00:23:04.664 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[3] 500.000000 - 1000.000000 : 182 2024-04-25T00:23:04.664 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[4] 200.000000 - 500.000000 : 786 2024-04-25T00:23:04.665 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[5] 100.000000 - 200.000000 : 2462 2024-04-25T00:23:04.665 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[6] 50.000000 - 100.000000 : 5126 2024-04-25T00:23:04.665 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[7] 20.000000 - 50.000000 : 6911 2024-04-25T00:23:04.665 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[8] 10.000000 - 20.000000 : 733 2024-04-25T00:23:04.665 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[9] 5.000000 - 10.000000 : 540 2024-04-25T00:23:04.665 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[10] 2.000000 - 5.000000 : 683 2024-04-25T00:23:04.665 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[11] 1.000000 - 2.000000 : 547 2024-04-25T00:23:04.665 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[12] 0.500000 - 1.000000 : 326 2024-04-25T00:23:04.666 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[13] 0.200000 - 0.500000 : 281 2024-04-25T00:23:04.666 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[14] 0.100000 - 0.200000 : 342 2024-04-25T00:23:04.666 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[15] 0.050000 - 0.100000 : 1123 2024-04-25T00:23:04.666 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[16] 0.020000 - 0.050000 : 7746 2024-04-25T00:23:04.666 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[17] 0.010000 - 0.020000 : 41553 2024-04-25T00:23:04.666 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[18] 0.000000 - 0.010000 : 4920186 2024-04-25T00:23:04.666 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:23:04.666 INFO:tasks.workunit.client.0.smithi027.stdout:[ lseek] 0.000000 0.001153 58.011002 4990720 2024-04-25T00:23:04.666 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-25T00:23:04.666 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-25T00:23:04.666 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-25T00:23:04.666 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-25T00:23:04.667 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-25T00:23:04.667 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-04-25T00:23:04.667 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[6] 50.000000 - 100.000000 : 1 2024-04-25T00:23:04.667 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[7] 20.000000 - 50.000000 : 38 2024-04-25T00:23:04.667 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[8] 10.000000 - 20.000000 : 66 2024-04-25T00:23:04.667 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[9] 5.000000 - 10.000000 : 44 2024-04-25T00:23:04.667 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[10] 2.000000 - 5.000000 : 33 2024-04-25T00:23:04.667 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[11] 1.000000 - 2.000000 : 29 2024-04-25T00:23:04.667 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[12] 0.500000 - 1.000000 : 18 2024-04-25T00:23:04.667 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[13] 0.200000 - 0.500000 : 29 2024-04-25T00:23:04.667 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[14] 0.100000 - 0.200000 : 42 2024-04-25T00:23:04.667 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[15] 0.050000 - 0.100000 : 91 2024-04-25T00:23:04.667 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[16] 0.020000 - 0.050000 : 590 2024-04-25T00:23:04.667 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[17] 0.010000 - 0.020000 : 2020 2024-04-25T00:23:04.668 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[18] 0.000000 - 0.010000 : 4987719 2024-04-25T00:23:04.668 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:23:04.668 INFO:tasks.workunit.client.0.smithi027.stdout:[ close] 0.001000 0.013514 21.990000 3899 2024-04-25T00:23:04.668 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-25T00:23:04.668 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-25T00:23:04.668 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-25T00:23:04.668 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-25T00:23:04.668 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-25T00:23:04.668 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-04-25T00:23:04.668 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-04-25T00:23:04.668 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[7] 20.000000 - 50.000000 : 1 2024-04-25T00:23:04.668 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-04-25T00:23:04.668 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[9] 5.000000 - 10.000000 : 1 2024-04-25T00:23:04.669 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[10] 2.000000 - 5.000000 : 0 2024-04-25T00:23:04.669 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[11] 1.000000 - 2.000000 : 0 2024-04-25T00:23:04.669 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[12] 0.500000 - 1.000000 : 0 2024-04-25T00:23:04.669 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[13] 0.200000 - 0.500000 : 0 2024-04-25T00:23:04.669 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-04-25T00:23:04.669 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[15] 0.050000 - 0.100000 : 1 2024-04-25T00:23:04.669 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[16] 0.020000 - 0.050000 : 11 2024-04-25T00:23:04.669 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[17] 0.010000 - 0.020000 : 36 2024-04-25T00:23:04.669 INFO:tasks.workunit.client.0.smithi027.stdout: msec_range[18] 0.000000 - 0.010000 : 3849 2024-04-25T00:23:04.669 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:23:04.669 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:23:04.669 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:23:04.669 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-04-25T00:23:04.670 INFO:tasks.workunit.client.0.smithi027.stdout:1.3% User Time 2024-04-25T00:23:04.670 INFO:tasks.workunit.client.0.smithi027.stdout:5.1% System Time 2024-04-25T00:23:04.670 INFO:tasks.workunit.client.0.smithi027.stdout:6.4% CPU Utilization 2024-04-25T00:23:04.671 INFO:tasks.workunit.client.0.smithi027.stderr:+ cd .. 2024-04-25T00:23:04.671 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm -r tmp ffsb 2024-04-25T00:23:04.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:04 smithi027 ceph-mon[29208]: pgmap v222: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-25T00:23:04.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:04 smithi139 ceph-mon[32833]: pgmap v222: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-25T00:23:04.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:04 smithi153 ceph-mon[32896]: pgmap v222: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-25T00:23:06.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:06 smithi027 ceph-mon[29208]: pgmap v223: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 85 KiB/s wr, 25 op/s 2024-04-25T00:23:06.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:06 smithi139 ceph-mon[32833]: pgmap v223: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 85 KiB/s wr, 25 op/s 2024-04-25T00:23:06.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:06 smithi153 ceph-mon[32896]: pgmap v223: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 85 KiB/s wr, 25 op/s 2024-04-25T00:23:06.992 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T00:23:06.993 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T00:23:07.047 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-04-25T00:23:07.047 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T00:23:07.660 DEBUG:teuthology.parallel:result is None 2024-04-25T00:23:07.661 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T00:23:07.727 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T00:23:07.727 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-25T00:23:07.743 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-25T00:23:07.743 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T00:23:07.743 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T00:23:07.743 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-25T00:23:07.797 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-25T00:23:08.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:08 smithi027 ceph-mon[29208]: pgmap v224: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 84 KiB/s wr, 25 op/s 2024-04-25T00:23:08.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:08 smithi139 ceph-mon[32833]: pgmap v224: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 84 KiB/s wr, 25 op/s 2024-04-25T00:23:08.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:08 smithi153 ceph-mon[32896]: pgmap v224: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 84 KiB/s wr, 25 op/s 2024-04-25T00:23:10.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:10 smithi027 ceph-mon[29208]: pgmap v225: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 89 KiB/s wr, 37 op/s 2024-04-25T00:23:10.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:10 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:23:10.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:10 smithi139 ceph-mon[32833]: pgmap v225: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 89 KiB/s wr, 37 op/s 2024-04-25T00:23:10.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:10 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:23:10.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:10 smithi153 ceph-mon[32896]: pgmap v225: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 89 KiB/s wr, 37 op/s 2024-04-25T00:23:10.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:10 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:23:11.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:11 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:23:11.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:11 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:23:11.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:11 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:23:12.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:12 smithi027 ceph-mon[29208]: pgmap v226: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 234 KiB/s wr, 69 op/s 2024-04-25T00:23:12.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:12 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:23:12.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:12 smithi139 ceph-mon[32833]: pgmap v226: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 234 KiB/s wr, 69 op/s 2024-04-25T00:23:12.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:12 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:23:12.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:12 smithi153 ceph-mon[32896]: pgmap v226: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 234 KiB/s wr, 69 op/s 2024-04-25T00:23:12.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:12 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:23:13.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:13 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:23:13.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:13 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:23:13.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:13 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:23:13.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:13 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:23:13.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:13 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:23:13.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:13 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:23:13.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:13 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:23:13.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:13 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:23:13.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:13 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:23:13.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:13 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:23:13.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:13 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:23:13.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:13 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:23:14.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:14 smithi027 ceph-mon[29208]: pgmap v227: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 236 KiB/s wr, 84 op/s 2024-04-25T00:23:14.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:14 smithi139 ceph-mon[32833]: pgmap v227: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 236 KiB/s wr, 84 op/s 2024-04-25T00:23:14.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:14 smithi153 ceph-mon[32896]: pgmap v227: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 236 KiB/s wr, 84 op/s 2024-04-25T00:23:16.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:16 smithi027 ceph-mon[29208]: pgmap v228: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 245 KiB/s wr, 121 op/s 2024-04-25T00:23:16.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:16 smithi139 ceph-mon[32833]: pgmap v228: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 245 KiB/s wr, 121 op/s 2024-04-25T00:23:16.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:16 smithi153 ceph-mon[32896]: pgmap v228: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 245 KiB/s wr, 121 op/s 2024-04-25T00:23:18.100 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:23:18.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:23:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.205+0000 7fa22c837640 1 -- 172.21.15.27:0/2397131090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa228069dc0 msgr2=0x7fa22806a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.205+0000 7fa22c837640 1 --2- 172.21.15.27:0/2397131090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa228069dc0 0x7fa22806a240 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa210009930 tx=0x7fa21002a420 comp rx=0 tx=0).stop 2024-04-25T00:23:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.205+0000 7fa22c837640 1 -- 172.21.15.27:0/2397131090 shutdown_connections 2024-04-25T00:23:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.205+0000 7fa22c837640 1 --2- 172.21.15.27:0/2397131090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa22806a780 0x7fa22810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.205+0000 7fa22c837640 1 --2- 172.21.15.27:0/2397131090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa228069dc0 0x7fa22806a240 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.205+0000 7fa22c837640 1 --2- 172.21.15.27:0/2397131090 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa228107730 0x7fa228107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.205+0000 7fa22c837640 1 -- 172.21.15.27:0/2397131090 >> 172.21.15.27:0/2397131090 conn(0x7fa2280776a0 msgr2=0x7fa228077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.205+0000 7fa22c837640 1 -- 172.21.15.27:0/2397131090 shutdown_connections 2024-04-25T00:23:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.205+0000 7fa22c837640 1 -- 172.21.15.27:0/2397131090 wait complete. 2024-04-25T00:23:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.206+0000 7fa22c837640 1 Processor -- start 2024-04-25T00:23:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.206+0000 7fa22c837640 1 -- start start 2024-04-25T00:23:18.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.206+0000 7fa22c837640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa228069dc0 0x7fa228198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.206+0000 7fa22c837640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa22806a780 0x7fa2281985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.206+0000 7fa22c837640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa228107730 0x7fa22819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.206+0000 7fa22c837640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa228111b40 con 0x7fa22806a780 2024-04-25T00:23:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.206+0000 7fa22c837640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa2281119c0 con 0x7fa228107730 2024-04-25T00:23:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.206+0000 7fa22c837640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa228111cc0 con 0x7fa228069dc0 2024-04-25T00:23:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.206+0000 7fa21ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa22806a780 0x7fa2281985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.206+0000 7fa21ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa22806a780 0x7fa2281985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49158/0 (socket says 172.21.15.27:49158) 2024-04-25T00:23:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.206+0000 7fa21ffff640 1 -- 172.21.15.27:0/101206552 learned_addr learned my addr 172.21.15.27:0/101206552 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.206+0000 7fa226d9d640 1 --2- 172.21.15.27:0/101206552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa228107730 0x7fa22819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.206+0000 7fa22659c640 1 --2- 172.21.15.27:0/101206552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa228069dc0 0x7fa228198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.207+0000 7fa226d9d640 1 -- 172.21.15.27:0/101206552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa228069dc0 msgr2=0x7fa228198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.207+0000 7fa226d9d640 1 --2- 172.21.15.27:0/101206552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa228069dc0 0x7fa228198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.207+0000 7fa226d9d640 1 -- 172.21.15.27:0/101206552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa22806a780 msgr2=0x7fa2281985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.207+0000 7fa226d9d640 1 --2- 172.21.15.27:0/101206552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa22806a780 0x7fa2281985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.207+0000 7fa226d9d640 1 -- 172.21.15.27:0/101206552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa22819d000 con 0x7fa228107730 2024-04-25T00:23:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.207+0000 7fa226d9d640 1 --2- 172.21.15.27:0/101206552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa228107730 0x7fa22819c8d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa220009fd0 tx=0x7fa22000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.207+0000 7fa21f7fe640 1 -- 172.21.15.27:0/101206552 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22000ce20 con 0x7fa228107730 2024-04-25T00:23:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.207+0000 7fa22659c640 1 --2- 172.21.15.27:0/101206552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa228069dc0 0x7fa228198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:23:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.207+0000 7fa21ffff640 1 --2- 172.21.15.27:0/101206552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa22806a780 0x7fa2281985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.208+0000 7fa21f7fe640 1 -- 172.21.15.27:0/101206552 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa220005020 con 0x7fa228107730 2024-04-25T00:23:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.208+0000 7fa21f7fe640 1 -- 172.21.15.27:0/101206552 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa220010720 con 0x7fa228107730 2024-04-25T00:23:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.210+0000 7fa22c837640 1 -- 172.21.15.27:0/101206552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa22819d2f0 con 0x7fa228107730 2024-04-25T00:23:18.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.211+0000 7fa22c837640 1 -- 172.21.15.27:0/101206552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2281005a0 con 0x7fa228107730 2024-04-25T00:23:18.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.214+0000 7fa21f7fe640 1 -- 172.21.15.27:0/101206552 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa2200051d0 con 0x7fa228107730 2024-04-25T00:23:18.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.215+0000 7fa21f7fe640 1 --2- 172.21.15.27:0/101206552 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa2140a7210 0x7fa2140a96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.215+0000 7fa21f7fe640 1 -- 172.21.15.27:0/101206552 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fa2200cfdc0 con 0x7fa228107730 2024-04-25T00:23:18.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.216+0000 7fa22659c640 1 --2- 172.21.15.27:0/101206552 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa2140a7210 0x7fa2140a96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.217+0000 7fa22f617740 1 Processor -- start 2024-04-25T00:23:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.220+0000 7fa22f617740 1 -- start start 2024-04-25T00:23:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.220+0000 7fa22f617740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559338687380 0x559338687760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.220+0000 7fa22f617740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559338687d30 0x55933869a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.220+0000 7fa22f617740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55933869ae30 0x55933869d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.220+0000 7fa22f617740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5593385ec6b0 con 0x55933869ae30 2024-04-25T00:23:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.220+0000 7fa22f617740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x559338594880 con 0x559338687d30 2024-04-25T00:23:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.220+0000 7fa22f617740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5593385e8670 con 0x559338687380 2024-04-25T00:23:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.221+0000 7fa226d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55933869ae30 0x55933869d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.221+0000 7fa226d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55933869ae30 0x55933869d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49160/0 (socket says 172.21.15.27:49160) 2024-04-25T00:23:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.221+0000 7fa226d9d640 1 -- 172.21.15.27:0/3482218829 learned_addr learned my addr 172.21.15.27:0/3482218829 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.221+0000 7fa22659c640 1 --2- 172.21.15.27:0/3482218829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559338687380 0x559338687760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.221+0000 7fa22659c640 1 --2- 172.21.15.27:0/101206552 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa2140a7210 0x7fa2140a96d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa218009ad0 tx=0x7fa218005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.221+0000 7fa226d9d640 1 -- 172.21.15.27:0/3482218829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559338687380 msgr2=0x559338687760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.221+0000 7fa226d9d640 1 --2- 172.21.15.27:0/3482218829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559338687380 0x559338687760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.221+0000 7fa226d9d640 1 -- 172.21.15.27:0/3482218829 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559338687d30 msgr2=0x55933869a8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:23:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.221+0000 7fa226d9d640 1 --2- 172.21.15.27:0/3482218829 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559338687d30 0x55933869a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.221+0000 7fa226d9d640 1 -- 172.21.15.27:0/3482218829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593383ca7d0 con 0x55933869ae30 2024-04-25T00:23:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.222+0000 7fa226d9d640 1 --2- 172.21.15.27:0/3482218829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55933869ae30 0x55933869d220 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa220098900 tx=0x7fa22009c630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e107a7d3771b34f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.223+0000 7fa21cff9640 1 -- 172.21.15.27:0/3482218829 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2200a02e0 con 0x55933869ae30 2024-04-25T00:23:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.223+0000 7fa21cff9640 1 -- 172.21.15.27:0/3482218829 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2200b7070 con 0x55933869ae30 2024-04-25T00:23:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.224+0000 7fa22f617740 1 -- 172.21.15.27:0/3482218829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55933869ae30 msgr2=0x55933869d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.224+0000 7fa22f617740 1 --2- 172.21.15.27:0/3482218829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55933869ae30 0x55933869d220 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa220098900 tx=0x7fa22009c630 comp rx=0 tx=0).stop 2024-04-25T00:23:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.226+0000 7fa22f617740 1 -- 172.21.15.27:0/3482218829 shutdown_connections 2024-04-25T00:23:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.226+0000 7fa22f617740 1 --2- 172.21.15.27:0/3482218829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55933869ae30 0x55933869d220 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.226+0000 7fa22f617740 1 --2- 172.21.15.27:0/3482218829 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559338687d30 0x55933869a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.226+0000 7fa22f617740 1 --2- 172.21.15.27:0/3482218829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559338687380 0x559338687760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.226+0000 7fa22f617740 1 -- 172.21.15.27:0/3482218829 >> 172.21.15.27:0/3482218829 conn(0x559338683950 msgr2=0x559338683d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.227+0000 7fa22f617740 1 -- 172.21.15.27:0/3482218829 shutdown_connections 2024-04-25T00:23:18.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.230+0000 7fa22f617740 1 -- 172.21.15.27:0/3482218829 wait complete. 2024-04-25T00:23:18.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.230+0000 7fa22f617740 1 -- start start 2024-04-25T00:23:18.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.230+0000 7fa22f617740 1 Processor -- start 2024-04-25T00:23:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.231+0000 7fa22f617740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559338687380 0x5593386947c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.231+0000 7fa22f617740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559338687d30 0x559338694d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.231+0000 7fa22f617740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55933869ae30 0x55933869a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.231+0000 7fa22f617740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55933868a2b0 con 0x559338687d30 2024-04-25T00:23:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.231+0000 7fa22f617740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5593385a7f50 con 0x559338687380 2024-04-25T00:23:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.231+0000 7fa22f617740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55933868a430 con 0x55933869ae30 2024-04-25T00:23:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.231+0000 7fa22f617740 10 client.0 _ll_register_callbacks cb 0x5593385e3000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:18.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.233+0000 7fa226d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55933869ae30 0x55933869a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.233+0000 7fa226d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55933869ae30 0x55933869a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:55906/0 (socket says 172.21.15.27:55906) 2024-04-25T00:23:18.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.233+0000 7fa226d9d640 1 -- 172.21.15.27:0/4279220201 learned_addr learned my addr 172.21.15.27:0/4279220201 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:18.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.233+0000 7fa22659c640 1 --2- 172.21.15.27:0/4279220201 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559338687380 0x5593386947c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.233+0000 7fa226d9d640 1 -- 172.21.15.27:0/4279220201 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559338687380 msgr2=0x5593386947c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.233+0000 7fa226d9d640 1 --2- 172.21.15.27:0/4279220201 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559338687380 0x5593386947c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.233+0000 7fa226d9d640 1 -- 172.21.15.27:0/4279220201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559338687d30 msgr2=0x559338694d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.233+0000 7fa226d9d640 1 --2- 172.21.15.27:0/4279220201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559338687d30 0x559338694d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.233+0000 7fa226d9d640 1 -- 172.21.15.27:0/4279220201 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593383003f0 con 0x55933869ae30 2024-04-25T00:23:18.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.235+0000 7fa226d9d640 1 --2- 172.21.15.27:0/4279220201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55933869ae30 0x55933869a4e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa220009230 tx=0x7fa220005770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=614237d4ffb7ce94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.237+0000 7fa20dffb640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:23:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.237+0000 7fa20dffb640 1 -- 172.21.15.27:0/4279220201 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2200cd5e0 con 0x55933869ae30 2024-04-25T00:23:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.237+0000 7fa22f617740 1 -- 172.21.15.27:0/4279220201 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2200b7070 con 0x55933869ae30 2024-04-25T00:23:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.237+0000 7fa20dffb640 1 -- 172.21.15.27:0/4279220201 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fa22009b380 con 0x55933869ae30 2024-04-25T00:23:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.237+0000 7fa20dffb640 1 -- 172.21.15.27:0/4279220201 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fa22009b580 con 0x55933869ae30 2024-04-25T00:23:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.238+0000 7fa22f617740 10 client.26035 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.238+0000 7fa22f617740 1 -- 172.21.15.27:0/4279220201 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5593382c00f0 con 0x55933869ae30 2024-04-25T00:23:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.238+0000 7fa20dffb640 1 -- 172.21.15.27:0/4279220201 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2200b7070 con 0x55933869ae30 2024-04-25T00:23:18.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.240+0000 7fa20dffb640 1 -- 172.21.15.27:0/4279220201 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7fa220097c80 con 0x55933869ae30 2024-04-25T00:23:18.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.240+0000 7fa22f617740 10 client.26035 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:18.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.240+0000 7fa22f617740 10 client.26035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:23:18.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.240+0000 7fa22f617740 20 client.26035 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:18.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.240+0000 7fa22f617740 1 --2- 172.21.15.27:0/4279220201 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5593386a2660 0x5593386a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.240+0000 7fa22f617740 4 client.26035 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:23:18.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.240+0000 7fa22f617740 1 -- 172.21.15.27:0/4279220201 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593385f9bf0 con 0x5593386a2660 2024-04-25T00:23:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.241+0000 7fa22659c640 1 --2- 172.21.15.27:0/4279220201 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5593386a2660 0x5593386a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.246+0000 7fa22659c640 1 --2- 172.21.15.27:0/4279220201 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5593386a2660 0x5593386a4a50 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7482f5acb1cbf08 server_cookie=1ab5e38ccc054acf in_seq=0 out_seq=0 2024-04-25T00:23:18.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.246+0000 7fa20dffb640 10 client.26035 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:18.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.248+0000 7fa20dffb640 1 -- 172.21.15.27:0/4279220201 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5593385f9bf0 con 0x5593386a2660 2024-04-25T00:23:18.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.248+0000 7fa20dffb640 10 client.26035 handle_command_reply: tid=0 2024-04-25T00:23:18.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.248+0000 7fa22f617740 1 client.26035 shutdown 2024-04-25T00:23:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.248+0000 7fa22f617740 1 -- 172.21.15.27:0/4279220201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55933869ae30 msgr2=0x55933869a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.248+0000 7fa22f617740 1 --2- 172.21.15.27:0/4279220201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55933869ae30 0x55933869a4e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa220009230 tx=0x7fa220005770 comp rx=0 tx=0).stop 2024-04-25T00:23:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.248+0000 7fa22f617740 1 -- 172.21.15.27:0/4279220201 shutdown_connections 2024-04-25T00:23:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.248+0000 7fa22f617740 1 --2- 172.21.15.27:0/4279220201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55933869ae30 0x55933869a4e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.248+0000 7fa22f617740 1 --2- 172.21.15.27:0/4279220201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559338687d30 0x559338694d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.248+0000 7fa22f617740 1 --2- 172.21.15.27:0/4279220201 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5593386a2660 0x5593386a4a50 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.248+0000 7fa22f617740 1 --2- 172.21.15.27:0/4279220201 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559338687380 0x5593386947c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.249+0000 7fa22f617740 1 -- 172.21.15.27:0/4279220201 >> 172.21.15.27:0/4279220201 conn(0x559338683950 msgr2=0x5593386897c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.249+0000 7fa20dffb640 0 client.26035 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.249+0000 7fa22f617740 1 -- 172.21.15.27:0/4279220201 shutdown_connections 2024-04-25T00:23:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.249+0000 7fa22f617740 1 -- 172.21.15.27:0/4279220201 wait complete. 2024-04-25T00:23:18.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.252+0000 7fa22f617740 20 client.26035 trim_cache size 0 max 16384 2024-04-25T00:23:18.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.291+0000 7fa22f617740 1 Processor -- start 2024-04-25T00:23:18.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.291+0000 7fa22f617740 1 -- start start 2024-04-25T00:23:18.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.291+0000 7fa22f617740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559338687380 0x559338682970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.291+0000 7fa22f617740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559338687d30 0x5593386aa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.292+0000 7fa22f617740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55933869ae30 0x5593386aa830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.292+0000 7fa22659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559338687380 0x559338682970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.292+0000 7fa22659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559338687380 0x559338682970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49170/0 (socket says 172.21.15.27:49170) 2024-04-25T00:23:18.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.292+0000 7fa22659c640 1 -- 172.21.15.27:0/828971671 learned_addr learned my addr 172.21.15.27:0/828971671 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:18.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.292+0000 7fa22f617740 1 -- 172.21.15.27:0/828971671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa2281acc50 con 0x559338687380 2024-04-25T00:23:18.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.292+0000 7fa22f617740 1 -- 172.21.15.27:0/828971671 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa2281ae7f0 con 0x55933869ae30 2024-04-25T00:23:18.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.292+0000 7fa22f617740 1 -- 172.21.15.27:0/828971671 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa2281ad320 con 0x559338687d30 2024-04-25T00:23:18.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.292+0000 7fa226d9d640 1 --2- 172.21.15.27:0/828971671 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55933869ae30 0x5593386aa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.293+0000 7fa226d9d640 1 -- 172.21.15.27:0/828971671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559338687d30 msgr2=0x5593386aa2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:23:18.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.293+0000 7fa226d9d640 1 --2- 172.21.15.27:0/828971671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559338687d30 0x5593386aa2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.293+0000 7fa226d9d640 1 -- 172.21.15.27:0/828971671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559338687380 msgr2=0x559338682970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:18.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.293+0000 7fa226d9d640 1 --2- 172.21.15.27:0/828971671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559338687380 0x559338682970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.293+0000 7fa226d9d640 1 -- 172.21.15.27:0/828971671 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559338683780 con 0x55933869ae30 2024-04-25T00:23:18.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.295+0000 7fa226d9d640 1 --2- 172.21.15.27:0/828971671 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55933869ae30 0x5593386aa830 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa22001f000 tx=0x7fa2200bc160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95a1d4e610bba183 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:18.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.296+0000 7fa20ffff640 1 -- 172.21.15.27:0/828971671 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa220085250 con 0x55933869ae30 2024-04-25T00:23:18.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.296+0000 7fa20ffff640 1 -- 172.21.15.27:0/828971671 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2200b7070 con 0x55933869ae30 2024-04-25T00:23:18.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.296+0000 7fa20ffff640 1 -- 172.21.15.27:0/828971671 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa220096aa0 con 0x55933869ae30 2024-04-25T00:23:18.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.298+0000 7fa22f617740 1 -- 172.21.15.27:0/828971671 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55933869ae30 msgr2=0x5593386aa830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:18.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.298+0000 7fa22f617740 1 --2- 172.21.15.27:0/828971671 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55933869ae30 0x5593386aa830 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa22001f000 tx=0x7fa2200bc160 comp rx=0 tx=0).stop 2024-04-25T00:23:18.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.299+0000 7fa22f617740 1 -- 172.21.15.27:0/828971671 shutdown_connections 2024-04-25T00:23:18.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.299+0000 7fa22f617740 1 --2- 172.21.15.27:0/828971671 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55933869ae30 0x5593386aa830 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.299+0000 7fa22f617740 1 --2- 172.21.15.27:0/828971671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559338687d30 0x5593386aa2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.299+0000 7fa22f617740 1 --2- 172.21.15.27:0/828971671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559338687380 0x559338682970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.299+0000 7fa22f617740 1 -- 172.21.15.27:0/828971671 >> 172.21.15.27:0/828971671 conn(0x559338683950 msgr2=0x559338682260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:18.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.302+0000 7fa22f617740 1 -- 172.21.15.27:0/828971671 shutdown_connections 2024-04-25T00:23:18.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.302+0000 7fa22f617740 1 -- 172.21.15.27:0/828971671 wait complete. 2024-04-25T00:23:18.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.302+0000 7fa22f617740 1 -- start start 2024-04-25T00:23:18.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.303+0000 7fa22f617740 1 Processor -- start 2024-04-25T00:23:18.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.303+0000 7fa22f617740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559338687380 0x5593386b5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.303+0000 7fa22f617740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559338687d30 0x5593386b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.304+0000 7fa22659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559338687380 0x5593386b5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.304+0000 7fa22659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559338687380 0x5593386b5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:55936/0 (socket says 172.21.15.27:55936) 2024-04-25T00:23:18.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.304+0000 7fa22659c640 1 -- 172.21.15.27:0/2117601765 learned_addr learned my addr 172.21.15.27:0/2117601765 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:18.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.304+0000 7fa22f617740 1 --2- 172.21.15.27:0/2117601765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55933869ae30 0x5593386b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.304+0000 7fa21ffff640 1 --2- 172.21.15.27:0/2117601765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559338687d30 0x5593386b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.304+0000 7fa22f617740 1 -- 172.21.15.27:0/2117601765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa2281ae120 con 0x559338687d30 2024-04-25T00:23:18.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.304+0000 7fa22f617740 1 -- 172.21.15.27:0/2117601765 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa2281ada20 con 0x55933869ae30 2024-04-25T00:23:18.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.304+0000 7fa22f617740 1 -- 172.21.15.27:0/2117601765 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa2281aabe0 con 0x559338687380 2024-04-25T00:23:18.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.304+0000 7fa226d9d640 1 --2- 172.21.15.27:0/2117601765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55933869ae30 0x5593386b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.305+0000 7fa22659c640 1 -- 172.21.15.27:0/2117601765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55933869ae30 msgr2=0x5593386b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:18.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.305+0000 7fa22659c640 1 --2- 172.21.15.27:0/2117601765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55933869ae30 0x5593386b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.305+0000 7fa22659c640 1 -- 172.21.15.27:0/2117601765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559338687d30 msgr2=0x5593386b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:18.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.305+0000 7fa22659c640 1 --2- 172.21.15.27:0/2117601765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559338687d30 0x5593386b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.305+0000 7fa22659c640 1 -- 172.21.15.27:0/2117601765 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559338695070 con 0x559338687380 2024-04-25T00:23:18.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.305+0000 7fa22f617740 10 client.0 _ll_register_callbacks cb 0x7fa218016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:18.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.305+0000 7fa22659c640 1 --2- 172.21.15.27:0/2117601765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559338687380 0x5593386b5b70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa218010df0 tx=0x7fa218013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dd2ee640ef6b6cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:18.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.305+0000 7fa20dffb640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:23:18.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.305+0000 7fa20dffb640 1 -- 172.21.15.27:0/2117601765 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21802e070 con 0x559338687380 2024-04-25T00:23:18.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.305+0000 7fa22f617740 1 -- 172.21.15.27:0/2117601765 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2200b7070 con 0x559338687380 2024-04-25T00:23:18.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.306+0000 7fa20dffb640 1 -- 172.21.15.27:0/2117601765 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fa218004030 con 0x559338687380 2024-04-25T00:23:18.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.306+0000 7fa20dffb640 1 -- 172.21.15.27:0/2117601765 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fa218037050 con 0x559338687380 2024-04-25T00:23:18.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.308+0000 7fa22f617740 10 client.26038 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:18.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.308+0000 7fa22f617740 1 -- 172.21.15.27:0/2117601765 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({fsmap=97}) -- 0x5593387192a0 con 0x559338687380 2024-04-25T00:23:18.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.312+0000 7fa20dffb640 1 -- 172.21.15.27:0/2117601765 <== mon.1 v2:172.21.15.153:3300/0 4 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7fa218031020 con 0x559338687380 2024-04-25T00:23:18.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.312+0000 7fa22f617740 10 client.26038 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:18.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.312+0000 7fa22f617740 10 client.26038 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:23:18.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.312+0000 7fa22f617740 20 client.26038 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:18.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.312+0000 7fa22f617740 1 --2- 172.21.15.27:0/2117601765 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x559338719870 0x55933871bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.312+0000 7fa22f617740 4 client.26038 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:23:18.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.312+0000 7fa22f617740 1 -- 172.21.15.27:0/2117601765 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fa2200b7070 con 0x559338719870 2024-04-25T00:23:18.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.313+0000 7fa21ffff640 1 --2- 172.21.15.27:0/2117601765 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x559338719870 0x55933871bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.318+0000 7fa21ffff640 1 --2- 172.21.15.27:0/2117601765 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x559338719870 0x55933871bc90 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adb763188ebaedf6 server_cookie=e76633b1b002854d in_seq=0 out_seq=0 2024-04-25T00:23:18.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.318+0000 7fa20dffb640 10 client.26038 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.321+0000 7fa20dffb640 1 -- 172.21.15.27:0/2117601765 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fa2200b7070 con 0x559338719870 2024-04-25T00:23:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.321+0000 7fa20dffb640 10 client.26038 handle_command_reply: tid=0 2024-04-25T00:23:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.321+0000 7fa22f617740 1 client.26038 shutdown 2024-04-25T00:23:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.326+0000 7fa22f617740 1 -- 172.21.15.27:0/2117601765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559338687380 msgr2=0x5593386b5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.326+0000 7fa22f617740 1 --2- 172.21.15.27:0/2117601765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559338687380 0x5593386b5b70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa218010df0 tx=0x7fa218013000 comp rx=0 tx=0).stop 2024-04-25T00:23:18.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.328+0000 7fa22f617740 1 -- 172.21.15.27:0/2117601765 shutdown_connections 2024-04-25T00:23:18.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.328+0000 7fa22f617740 1 --2- 172.21.15.27:0/2117601765 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x559338719870 0x55933871bc90 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.328+0000 7fa22f617740 1 --2- 172.21.15.27:0/2117601765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55933869ae30 0x5593386b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.328+0000 7fa22f617740 1 --2- 172.21.15.27:0/2117601765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559338687d30 0x5593386b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.328+0000 7fa20dffb640 0 client.26038 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:18.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.328+0000 7fa22f617740 1 --2- 172.21.15.27:0/2117601765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559338687380 0x5593386b5b70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.328+0000 7fa22f617740 1 -- 172.21.15.27:0/2117601765 >> 172.21.15.27:0/2117601765 conn(0x559338683950 msgr2=0x55933868e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.329+0000 7fa22f617740 1 -- 172.21.15.27:0/2117601765 shutdown_connections 2024-04-25T00:23:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.329+0000 7fa22f617740 1 -- 172.21.15.27:0/2117601765 wait complete. 2024-04-25T00:23:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.329+0000 7fa22f617740 20 client.26038 trim_cache size 0 max 16384 2024-04-25T00:23:18.334 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:23:18.334 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T00:23:18.334 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T00:23:18.335 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:23:18.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.333+0000 7fa22c837640 1 -- 172.21.15.27:0/101206552 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa2140a7210 msgr2=0x7fa2140a96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:18.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.333+0000 7fa22c837640 1 --2- 172.21.15.27:0/101206552 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa2140a7210 0x7fa2140a96d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa218009ad0 tx=0x7fa218005f90 comp rx=0 tx=0).stop 2024-04-25T00:23:18.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.333+0000 7fa22c837640 1 -- 172.21.15.27:0/101206552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa228107730 msgr2=0x7fa22819c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:18.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.333+0000 7fa22c837640 1 --2- 172.21.15.27:0/101206552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa228107730 0x7fa22819c8d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa220009fd0 tx=0x7fa22000ef10 comp rx=0 tx=0).stop 2024-04-25T00:23:18.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.334+0000 7fa22c837640 1 -- 172.21.15.27:0/101206552 shutdown_connections 2024-04-25T00:23:18.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.334+0000 7fa22c837640 1 --2- 172.21.15.27:0/101206552 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa2140a7210 0x7fa2140a96d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.334+0000 7fa22c837640 1 --2- 172.21.15.27:0/101206552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa228107730 0x7fa22819c8d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.334+0000 7fa22c837640 1 --2- 172.21.15.27:0/101206552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa22806a780 0x7fa2281985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.334+0000 7fa22c837640 1 --2- 172.21.15.27:0/101206552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa228069dc0 0x7fa228198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.334+0000 7fa22c837640 1 -- 172.21.15.27:0/101206552 >> 172.21.15.27:0/101206552 conn(0x7fa2280776a0 msgr2=0x7fa2280fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:18.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.335+0000 7fa22c837640 1 -- 172.21.15.27:0/101206552 shutdown_connections 2024-04-25T00:23:18.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.336+0000 7fa22c837640 1 -- 172.21.15.27:0/101206552 wait complete. 2024-04-25T00:23:18.359 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T00:23:18.359 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:23:18.397 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:18 smithi027 ceph-mon[29208]: pgmap v229: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 161 KiB/s wr, 114 op/s 2024-04-25T00:23:18.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.536+0000 7f4e5c563640 1 -- 172.21.15.27:0/3869049972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e54100520 msgr2=0x7f4e54100900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:18.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.536+0000 7f4e5c563640 1 --2- 172.21.15.27:0/3869049972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e54100520 0x7f4e54100900 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4e44009860 tx=0x7f4e4402a380 comp rx=0 tx=0).stop 2024-04-25T00:23:18.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.536+0000 7f4e5c563640 1 -- 172.21.15.27:0/3869049972 shutdown_connections 2024-04-25T00:23:18.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.536+0000 7f4e5c563640 1 --2- 172.21.15.27:0/3869049972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5410e0d0 0x7f4e541104c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.536+0000 7f4e5c563640 1 --2- 172.21.15.27:0/3869049972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e54100e40 0x7f4e5410db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.536+0000 7f4e5c563640 1 --2- 172.21.15.27:0/3869049972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e54100520 0x7f4e54100900 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.536+0000 7f4e5c563640 1 -- 172.21.15.27:0/3869049972 >> 172.21.15.27:0/3869049972 conn(0x7f4e540769c0 msgr2=0x7f4e54076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:18.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.538+0000 7f4e5c563640 1 -- 172.21.15.27:0/3869049972 shutdown_connections 2024-04-25T00:23:18.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.539+0000 7f4e5c563640 1 -- 172.21.15.27:0/3869049972 wait complete. 2024-04-25T00:23:18.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.540+0000 7f4e5c563640 1 Processor -- start 2024-04-25T00:23:18.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.540+0000 7f4e5c563640 1 -- start start 2024-04-25T00:23:18.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.541+0000 7f4e5c563640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e54100520 0x7f4e5419a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.541+0000 7f4e5c563640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e54100e40 0x7f4e5419a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.541+0000 7f4e5c563640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5410e0d0 0x7f4e5419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.541+0000 7f4e5c563640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4e54113d80 con 0x7f4e5410e0d0 2024-04-25T00:23:18.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.541+0000 7f4e5c563640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4e54113c00 con 0x7f4e54100e40 2024-04-25T00:23:18.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.541+0000 7f4e5c563640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e54113f00 con 0x7f4e54100520 2024-04-25T00:23:18.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.542+0000 7f4e5a2ff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e54100520 0x7f4e5419a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.542+0000 7f4e5a2ff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e54100520 0x7f4e5419a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:55940/0 (socket says 172.21.15.27:55940) 2024-04-25T00:23:18.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.542+0000 7f4e5a2ff640 1 -- 172.21.15.27:0/2369253807 learned_addr learned my addr 172.21.15.27:0/2369253807 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:18.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.542+0000 7f4e5ab00640 1 --2- 172.21.15.27:0/2369253807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5410e0d0 0x7f4e5419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.543+0000 7f4e5a2ff640 1 -- 172.21.15.27:0/2369253807 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e54100e40 msgr2=0x7f4e5419a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:18.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.543+0000 7f4e5a2ff640 1 --2- 172.21.15.27:0/2369253807 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e54100e40 0x7f4e5419a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.543+0000 7f4e5a2ff640 1 -- 172.21.15.27:0/2369253807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5410e0d0 msgr2=0x7f4e5419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:18.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.543+0000 7f4e5a2ff640 1 --2- 172.21.15.27:0/2369253807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5410e0d0 0x7f4e5419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.543+0000 7f4e5a2ff640 1 -- 172.21.15.27:0/2369253807 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e5419f290 con 0x7f4e54100520 2024-04-25T00:23:18.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.543+0000 7f4e5ab00640 1 --2- 172.21.15.27:0/2369253807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5410e0d0 0x7f4e5419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:18.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.543+0000 7f4e5a2ff640 1 --2- 172.21.15.27:0/2369253807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e54100520 0x7f4e5419a2a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4e4402a8f0 tx=0x7f4e44033d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:18.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.544+0000 7f4e4b7fe640 1 -- 172.21.15.27:0/2369253807 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e44037040 con 0x7f4e54100520 2024-04-25T00:23:18.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.544+0000 7f4e4b7fe640 1 -- 172.21.15.27:0/2369253807 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e44040070 con 0x7f4e54100520 2024-04-25T00:23:18.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.544+0000 7f4e4b7fe640 1 -- 172.21.15.27:0/2369253807 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e44005520 con 0x7f4e54100520 2024-04-25T00:23:18.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.545+0000 7f4e5c563640 1 -- 172.21.15.27:0/2369253807 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e5419f520 con 0x7f4e54100520 2024-04-25T00:23:18.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.545+0000 7f4e5c563640 1 -- 172.21.15.27:0/2369253807 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e5419f990 con 0x7f4e54100520 2024-04-25T00:23:18.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.547+0000 7f4e5c563640 1 -- 172.21.15.27:0/2369253807 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e5406a560 con 0x7f4e54100520 2024-04-25T00:23:18.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.549+0000 7f4e4b7fe640 1 -- 172.21.15.27:0/2369253807 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4e44004240 con 0x7f4e54100520 2024-04-25T00:23:18.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.550+0000 7f4e4b7fe640 1 --2- 172.21.15.27:0/2369253807 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4e400a2e40 0x7f4e400a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:18.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.550+0000 7f4e4b7fe640 1 -- 172.21.15.27:0/2369253807 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f4e440f0040 con 0x7f4e54100520 2024-04-25T00:23:18.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.551+0000 7f4e59afe640 1 --2- 172.21.15.27:0/2369253807 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4e400a2e40 0x7f4e400a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:18.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.552+0000 7f4e59afe640 1 --2- 172.21.15.27:0/2369253807 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4e400a2e40 0x7f4e400a5300 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4e5410eec0 tx=0x7f4e50009540 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:18.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.554+0000 7f4e4b7fe640 1 -- 172.21.15.27:0/2369253807 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4e5419f990 con 0x7f4e54100520 2024-04-25T00:23:18.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:18 smithi139 ceph-mon[32833]: pgmap v229: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 161 KiB/s wr, 114 op/s 2024-04-25T00:23:18.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:18 smithi153 ceph-mon[32896]: pgmap v229: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 161 KiB/s wr, 114 op/s 2024-04-25T00:23:18.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.992+0000 7f4e5c563640 1 -- 172.21.15.27:0/2369253807 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e5419b5f0 con 0x7f4e54100520 2024-04-25T00:23:18.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.994+0000 7f4e4b7fe640 1 -- 172.21.15.27:0/2369253807 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+11954 (secure 0 0 0) 0x7f4e440bc9c0 con 0x7f4e54100520 2024-04-25T00:23:18.995 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:23:18.995 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":97,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:23:11.564498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":147,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":148,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":146,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24514,24430,24521,24428,24496]},"id":1}]} 2024-04-25T00:23:18.996 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 97 2024-04-25T00:23:18.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.997+0000 7f4e497fa640 1 -- 172.21.15.27:0/2369253807 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4e400a2e40 msgr2=0x7f4e400a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:18.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.997+0000 7f4e497fa640 1 --2- 172.21.15.27:0/2369253807 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4e400a2e40 0x7f4e400a5300 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4e5410eec0 tx=0x7f4e50009540 comp rx=0 tx=0).stop 2024-04-25T00:23:18.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.997+0000 7f4e497fa640 1 -- 172.21.15.27:0/2369253807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e54100520 msgr2=0x7f4e5419a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:18.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.997+0000 7f4e497fa640 1 --2- 172.21.15.27:0/2369253807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e54100520 0x7f4e5419a2a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4e4402a8f0 tx=0x7f4e44033d10 comp rx=0 tx=0).stop 2024-04-25T00:23:19.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.998+0000 7f4e497fa640 1 -- 172.21.15.27:0/2369253807 shutdown_connections 2024-04-25T00:23:19.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.998+0000 7f4e497fa640 1 --2- 172.21.15.27:0/2369253807 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4e400a2e40 0x7f4e400a5300 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.998+0000 7f4e497fa640 1 --2- 172.21.15.27:0/2369253807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5410e0d0 0x7f4e5419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.998+0000 7f4e497fa640 1 --2- 172.21.15.27:0/2369253807 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e54100e40 0x7f4e5419a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.998+0000 7f4e497fa640 1 --2- 172.21.15.27:0/2369253807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e54100520 0x7f4e5419a2a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.998+0000 7f4e497fa640 1 -- 172.21.15.27:0/2369253807 >> 172.21.15.27:0/2369253807 conn(0x7f4e540769c0 msgr2=0x7f4e5410f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.999+0000 7f4e497fa640 1 -- 172.21.15.27:0/2369253807 shutdown_connections 2024-04-25T00:23:19.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:18.999+0000 7f4e497fa640 1 -- 172.21.15.27:0/2369253807 wait complete. 2024-04-25T00:23:19.017 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T00:23:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.222+0000 7f724e212640 1 -- 172.21.15.27:0/2311794969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f724811af20 msgr2=0x7f724811d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.222+0000 7f724e212640 1 --2- 172.21.15.27:0/2311794969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f724811af20 0x7f724811d310 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7238009900 tx=0x7f723802a410 comp rx=0 tx=0).stop 2024-04-25T00:23:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.222+0000 7f724e212640 1 -- 172.21.15.27:0/2311794969 shutdown_connections 2024-04-25T00:23:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.222+0000 7f724e212640 1 --2- 172.21.15.27:0/2311794969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f724811af20 0x7f724811d310 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.222+0000 7f724e212640 1 --2- 172.21.15.27:0/2311794969 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f724810dfc0 0x7f724810e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.222+0000 7f724e212640 1 --2- 172.21.15.27:0/2311794969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724810ff90 0x7f724810da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.222+0000 7f724e212640 1 -- 172.21.15.27:0/2311794969 >> 172.21.15.27:0/2311794969 conn(0x7f724806c850 msgr2=0x7f724806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.224+0000 7f724e212640 1 -- 172.21.15.27:0/2311794969 shutdown_connections 2024-04-25T00:23:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.224+0000 7f724e212640 1 -- 172.21.15.27:0/2311794969 wait complete. 2024-04-25T00:23:19.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.225+0000 7f724e212640 1 Processor -- start 2024-04-25T00:23:19.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.227+0000 7f724e212640 1 -- start start 2024-04-25T00:23:19.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.227+0000 7f724e212640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724810dfc0 0x7f72481168b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.227+0000 7f724e212640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f724810ff90 0x7f7248116df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.227+0000 7f724e212640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f724811b350 0x7f724811b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.227+0000 7f724e212640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7248110d10 con 0x7f724810dfc0 2024-04-25T00:23:19.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.227+0000 7f724e212640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7248110b90 con 0x7f724810ff90 2024-04-25T00:23:19.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.227+0000 7f724e212640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7248110e90 con 0x7f724811b350 2024-04-25T00:23:19.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.229+0000 7f72477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724810dfc0 0x7f72481168b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.229+0000 7f72477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724810dfc0 0x7f72481168b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49206/0 (socket says 172.21.15.27:49206) 2024-04-25T00:23:19.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.229+0000 7f72477fe640 1 -- 172.21.15.27:0/3911914940 learned_addr learned my addr 172.21.15.27:0/3911914940 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:19.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.229+0000 7f7246ffd640 1 --2- 172.21.15.27:0/3911914940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f724810ff90 0x7f7248116df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.229+0000 7f7247fff640 1 --2- 172.21.15.27:0/3911914940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f724811b350 0x7f724811b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.234+0000 7f7247fff640 1 -- 172.21.15.27:0/3911914940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f724810ff90 msgr2=0x7f7248116df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.234+0000 7f7247fff640 1 --2- 172.21.15.27:0/3911914940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f724810ff90 0x7f7248116df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.234+0000 7f7247fff640 1 -- 172.21.15.27:0/3911914940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724810dfc0 msgr2=0x7f72481168b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.234+0000 7f7247fff640 1 --2- 172.21.15.27:0/3911914940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724810dfc0 0x7f72481168b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.234+0000 7f7247fff640 1 -- 172.21.15.27:0/3911914940 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f724811bd40 con 0x7f724811b350 2024-04-25T00:23:19.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.239+0000 7f7247fff640 1 --2- 172.21.15.27:0/3911914940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f724811b350 0x7f724811b800 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f723802aa00 tx=0x7f72380045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.242+0000 7f7244ff9640 1 -- 172.21.15.27:0/3911914940 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238040070 con 0x7f724811b350 2024-04-25T00:23:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.242+0000 7f7244ff9640 1 -- 172.21.15.27:0/3911914940 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7238003880 con 0x7f724811b350 2024-04-25T00:23:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.242+0000 7f7244ff9640 1 -- 172.21.15.27:0/3911914940 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238035040 con 0x7f724811b350 2024-04-25T00:23:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.243+0000 7f724e212640 1 -- 172.21.15.27:0/3911914940 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f724811bfd0 con 0x7f724811b350 2024-04-25T00:23:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.243+0000 7f724e212640 1 -- 172.21.15.27:0/3911914940 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72481bc2f0 con 0x7f724811b350 2024-04-25T00:23:19.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.249+0000 7f7250ffb740 1 Processor -- start 2024-04-25T00:23:19.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.250+0000 7f7244ff9640 1 -- 172.21.15.27:0/3911914940 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7238004d00 con 0x7f724811b350 2024-04-25T00:23:19.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.251+0000 7f7244ff9640 1 --2- 172.21.15.27:0/3911914940 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f72300a2e90 0x7f72300a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.251+0000 7f7244ff9640 1 -- 172.21.15.27:0/3911914940 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f72380316e0 con 0x7f724811b350 2024-04-25T00:23:19.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.251+0000 7f7250ffb740 1 -- start start 2024-04-25T00:23:19.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.252+0000 7f7250ffb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d14f580 0x55dd7d14f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.252+0000 7f7250ffb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d14fea0 0x55dd7d15a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.252+0000 7f7250ffb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d15a570 0x55dd7d15c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.252+0000 7f7250ffb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55dd7d0b4670 con 0x55dd7d14fea0 2024-04-25T00:23:19.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.252+0000 7f7250ffb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55dd7d05c970 con 0x55dd7d14f580 2024-04-25T00:23:19.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.252+0000 7f7250ffb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dd7d0b0630 con 0x55dd7d15a570 2024-04-25T00:23:19.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.252+0000 7f7247fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d15a570 0x55dd7d15c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.253+0000 7f7247fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d15a570 0x55dd7d15c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:55976/0 (socket says 172.21.15.27:55976) 2024-04-25T00:23:19.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.253+0000 7f7247fff640 1 -- 172.21.15.27:0/1317216819 learned_addr learned my addr 172.21.15.27:0/1317216819 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:19.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.253+0000 7f72477fe640 1 --2- 172.21.15.27:0/1317216819 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d14f580 0x55dd7d14f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.253+0000 7f7246ffd640 1 --2- 172.21.15.27:0/3911914940 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f72300a2e90 0x7f72300a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.253+0000 7f7246ffd640 1 --2- 172.21.15.27:0/1317216819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d14fea0 0x55dd7d15a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.254+0000 7f7246ffd640 1 --2- 172.21.15.27:0/3911914940 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f72300a2e90 0x7f72300a5350 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7240009fd0 tx=0x7f7240009f40 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.254+0000 7f7246ffd640 1 -- 172.21.15.27:0/1317216819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d15a570 msgr2=0x55dd7d15c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.254+0000 7f7246ffd640 1 --2- 172.21.15.27:0/1317216819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d15a570 0x55dd7d15c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.254+0000 7f7246ffd640 1 -- 172.21.15.27:0/1317216819 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d14f580 msgr2=0x55dd7d14f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.254+0000 7f7246ffd640 1 --2- 172.21.15.27:0/1317216819 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d14f580 0x55dd7d14f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.254+0000 7f7246ffd640 1 -- 172.21.15.27:0/1317216819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd7d080400 con 0x55dd7d14fea0 2024-04-25T00:23:19.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.254+0000 7f7246ffd640 1 --2- 172.21.15.27:0/1317216819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d14fea0 0x55dd7d15a030 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f7248118070 tx=0x7f724001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d0ba0c358c2bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.255+0000 7f722dffb640 1 -- 172.21.15.27:0/1317216819 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7240028070 con 0x55dd7d14fea0 2024-04-25T00:23:19.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.255+0000 7f722dffb640 1 -- 172.21.15.27:0/1317216819 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f724001f070 con 0x55dd7d14fea0 2024-04-25T00:23:19.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.256+0000 7f7250ffb740 1 -- 172.21.15.27:0/1317216819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d14fea0 msgr2=0x55dd7d15a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.256+0000 7f7250ffb740 1 --2- 172.21.15.27:0/1317216819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d14fea0 0x55dd7d15a030 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f7248118070 tx=0x7f724001c000 comp rx=0 tx=0).stop 2024-04-25T00:23:19.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.256+0000 7f7250ffb740 1 -- 172.21.15.27:0/1317216819 shutdown_connections 2024-04-25T00:23:19.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.256+0000 7f7250ffb740 1 --2- 172.21.15.27:0/1317216819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d15a570 0x55dd7d15c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.256+0000 7f7250ffb740 1 --2- 172.21.15.27:0/1317216819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d14fea0 0x55dd7d15a030 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.256+0000 7f7250ffb740 1 --2- 172.21.15.27:0/1317216819 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d14f580 0x55dd7d14f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.256+0000 7f7250ffb740 1 -- 172.21.15.27:0/1317216819 >> 172.21.15.27:0/1317216819 conn(0x55dd7d14b8a0 msgr2=0x55dd7d14bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.256+0000 7f7250ffb740 1 -- 172.21.15.27:0/1317216819 shutdown_connections 2024-04-25T00:23:19.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.257+0000 7f7250ffb740 1 -- 172.21.15.27:0/1317216819 wait complete. 2024-04-25T00:23:19.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.257+0000 7f7250ffb740 1 -- start start 2024-04-25T00:23:19.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.257+0000 7f7250ffb740 1 Processor -- start 2024-04-25T00:23:19.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.258+0000 7f7250ffb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d14fea0 0x55dd7d159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.258+0000 7f7250ffb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d15a570 0x55dd7d1571d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.258+0000 7f7250ffb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d157710 0x55dd7d157b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.258+0000 7f7250ffb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55dd7d17f360 con 0x55dd7d15a570 2024-04-25T00:23:19.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.258+0000 7f7250ffb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55dd7d070050 con 0x55dd7d157710 2024-04-25T00:23:19.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.258+0000 7f7250ffb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dd7d17f4e0 con 0x55dd7d14fea0 2024-04-25T00:23:19.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.258+0000 7f7250ffb740 10 client.0 _ll_register_callbacks cb 0x55dd7cdc7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:19.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.259+0000 7f72477fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d14fea0 0x55dd7d159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.259+0000 7f72477fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d14fea0 0x55dd7d159ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:55980/0 (socket says 172.21.15.27:55980) 2024-04-25T00:23:19.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.259+0000 7f72477fe640 1 -- 172.21.15.27:0/2931655651 learned_addr learned my addr 172.21.15.27:0/2931655651 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:19.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.261+0000 7f7247fff640 1 --2- 172.21.15.27:0/2931655651 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d157710 0x55dd7d157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.264+0000 7f72477fe640 1 -- 172.21.15.27:0/2931655651 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d157710 msgr2=0x55dd7d157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.264+0000 7f72477fe640 1 --2- 172.21.15.27:0/2931655651 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d157710 0x55dd7d157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.264+0000 7f72477fe640 1 -- 172.21.15.27:0/2931655651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d15a570 msgr2=0x55dd7d1571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.264+0000 7f72477fe640 1 --2- 172.21.15.27:0/2931655651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d15a570 0x55dd7d1571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.264+0000 7f72477fe640 1 -- 172.21.15.27:0/2931655651 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd7d041dc0 con 0x55dd7d14fea0 2024-04-25T00:23:19.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.264+0000 7f72477fe640 1 --2- 172.21.15.27:0/2931655651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d14fea0 0x55dd7d159ee0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f723c011560 tx=0x7f723c011a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0bcd21e361cd926 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.264+0000 7f7226ffd640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:23:19.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.265+0000 7f7226ffd640 1 -- 172.21.15.27:0/2931655651 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723c013670 con 0x55dd7d14fea0 2024-04-25T00:23:19.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.265+0000 7f7250ffb740 1 -- 172.21.15.27:0/2931655651 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f724001f070 con 0x55dd7d14fea0 2024-04-25T00:23:19.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.265+0000 7f7226ffd640 1 -- 172.21.15.27:0/2931655651 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f723c013810 con 0x55dd7d14fea0 2024-04-25T00:23:19.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.265+0000 7f7226ffd640 1 -- 172.21.15.27:0/2931655651 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f723c026050 con 0x55dd7d14fea0 2024-04-25T00:23:19.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.267+0000 7f7250ffb740 10 client.26056 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:19.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.267+0000 7f7250ffb740 1 -- 172.21.15.27:0/2931655651 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55dd7cdc83f0 con 0x55dd7d14fea0 2024-04-25T00:23:19.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.271+0000 7f7226ffd640 1 -- 172.21.15.27:0/2931655651 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f724001f070 con 0x55dd7d14fea0 2024-04-25T00:23:19.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.272+0000 7f7226ffd640 1 -- 172.21.15.27:0/2931655651 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f723c01d400 con 0x55dd7d14fea0 2024-04-25T00:23:19.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.272+0000 7f7250ffb740 10 client.26056 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:19.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.272+0000 7f7250ffb740 10 client.26056 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:23:19.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.272+0000 7f7250ffb740 20 client.26056 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:19.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.272+0000 7f7250ffb740 1 --2- 172.21.15.27:0/2931655651 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55dd7d1af030 0x55dd7d1b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.272+0000 7f7250ffb740 4 client.26056 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:23:19.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.272+0000 7f7250ffb740 1 -- 172.21.15.27:0/2931655651 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd7d0c1bb0 con 0x55dd7d1af030 2024-04-25T00:23:19.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.279+0000 7f7246ffd640 1 --2- 172.21.15.27:0/2931655651 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55dd7d1af030 0x55dd7d1b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.280+0000 7f7246ffd640 1 --2- 172.21.15.27:0/2931655651 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55dd7d1af030 0x55dd7d1b14d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=19e047ebbfa2ea6a server_cookie=8e5ebba4a169c8a1 in_seq=0 out_seq=0 2024-04-25T00:23:19.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.280+0000 7f7226ffd640 10 client.26056 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:23:19.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.282+0000 7f7226ffd640 1 -- 172.21.15.27:0/2931655651 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dd7d0c1bb0 con 0x55dd7d1af030 2024-04-25T00:23:19.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.282+0000 7f7226ffd640 10 client.26056 handle_command_reply: tid=0 2024-04-25T00:23:19.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.282+0000 7f7250ffb740 1 client.26056 shutdown 2024-04-25T00:23:19.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.282+0000 7f7250ffb740 1 -- 172.21.15.27:0/2931655651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d14fea0 msgr2=0x55dd7d159ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.282+0000 7f7250ffb740 1 --2- 172.21.15.27:0/2931655651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d14fea0 0x55dd7d159ee0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f723c011560 tx=0x7f723c011a20 comp rx=0 tx=0).stop 2024-04-25T00:23:19.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.283+0000 7f7250ffb740 1 -- 172.21.15.27:0/2931655651 shutdown_connections 2024-04-25T00:23:19.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.283+0000 7f7250ffb740 1 --2- 172.21.15.27:0/2931655651 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55dd7d1af030 0x55dd7d1b14d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.283+0000 7f7250ffb740 1 --2- 172.21.15.27:0/2931655651 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d157710 0x55dd7d157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.283+0000 7f7250ffb740 1 --2- 172.21.15.27:0/2931655651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d15a570 0x55dd7d1571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.283+0000 7f7250ffb740 1 --2- 172.21.15.27:0/2931655651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d14fea0 0x55dd7d159ee0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.283+0000 7f7250ffb740 1 -- 172.21.15.27:0/2931655651 >> 172.21.15.27:0/2931655651 conn(0x55dd7d14b8a0 msgr2=0x55dd7d14ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.283+0000 7f7226ffd640 0 client.26056 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:23:19.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.284+0000 7f7250ffb740 1 -- 172.21.15.27:0/2931655651 shutdown_connections 2024-04-25T00:23:19.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.284+0000 7f7250ffb740 1 -- 172.21.15.27:0/2931655651 wait complete. 2024-04-25T00:23:19.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.284+0000 7f7250ffb740 20 client.26056 trim_cache size 0 max 16384 2024-04-25T00:23:19.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.310+0000 7f7250ffb740 1 Processor -- start 2024-04-25T00:23:19.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.310+0000 7f7250ffb740 1 -- start start 2024-04-25T00:23:19.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.310+0000 7f7250ffb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d14fea0 0x55dd7d152860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.310+0000 7f7250ffb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d157710 0x55dd7d159a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.310+0000 7f7250ffb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d15a570 0x55dd7d151420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.310+0000 7f7250ffb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f723400fea0 con 0x55dd7d15a570 2024-04-25T00:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.310+0000 7f7250ffb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7234011a40 con 0x55dd7d157710 2024-04-25T00:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.310+0000 7f7250ffb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7234010570 con 0x55dd7d14fea0 2024-04-25T00:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.311+0000 7f7247fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d15a570 0x55dd7d151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.311+0000 7f7247fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d15a570 0x55dd7d151420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49224/0 (socket says 172.21.15.27:49224) 2024-04-25T00:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.311+0000 7f7247fff640 1 -- 172.21.15.27:0/3685776866 learned_addr learned my addr 172.21.15.27:0/3685776866 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:19.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.311+0000 7f72477fe640 1 --2- 172.21.15.27:0/3685776866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d14fea0 0x55dd7d152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.311+0000 7f72477fe640 1 -- 172.21.15.27:0/3685776866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d157710 msgr2=0x55dd7d159a10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:23:19.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.311+0000 7f72477fe640 1 --2- 172.21.15.27:0/3685776866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d157710 0x55dd7d159a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.311+0000 7f72477fe640 1 -- 172.21.15.27:0/3685776866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d15a570 msgr2=0x55dd7d151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.311+0000 7f72477fe640 1 --2- 172.21.15.27:0/3685776866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d15a570 0x55dd7d151420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.311+0000 7f72477fe640 1 -- 172.21.15.27:0/3685776866 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd7ce927d0 con 0x55dd7d14fea0 2024-04-25T00:23:19.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.312+0000 7f72477fe640 1 --2- 172.21.15.27:0/3685776866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d14fea0 0x55dd7d152860 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f723c01d940 tx=0x7f723c009590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69f3e373ede5d075 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.312+0000 7f722d7fa640 1 -- 172.21.15.27:0/3685776866 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723c02f070 con 0x55dd7d14fea0 2024-04-25T00:23:19.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.312+0000 7f722d7fa640 1 -- 172.21.15.27:0/3685776866 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f723c01d9c0 con 0x55dd7d14fea0 2024-04-25T00:23:19.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.312+0000 7f722d7fa640 1 -- 172.21.15.27:0/3685776866 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723c0146a0 con 0x55dd7d14fea0 2024-04-25T00:23:19.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.313+0000 7f7250ffb740 1 -- 172.21.15.27:0/3685776866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d14fea0 msgr2=0x55dd7d152860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.313+0000 7f7250ffb740 1 --2- 172.21.15.27:0/3685776866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d14fea0 0x55dd7d152860 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f723c01d940 tx=0x7f723c009590 comp rx=0 tx=0).stop 2024-04-25T00:23:19.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.314+0000 7f7250ffb740 1 -- 172.21.15.27:0/3685776866 shutdown_connections 2024-04-25T00:23:19.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.314+0000 7f7250ffb740 1 --2- 172.21.15.27:0/3685776866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d15a570 0x55dd7d151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.314+0000 7f7250ffb740 1 --2- 172.21.15.27:0/3685776866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d157710 0x55dd7d159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.314+0000 7f7250ffb740 1 --2- 172.21.15.27:0/3685776866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d14fea0 0x55dd7d152860 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.314+0000 7f7250ffb740 1 -- 172.21.15.27:0/3685776866 >> 172.21.15.27:0/3685776866 conn(0x55dd7d14b8a0 msgr2=0x55dd7d150d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.315+0000 7f7250ffb740 1 -- 172.21.15.27:0/3685776866 shutdown_connections 2024-04-25T00:23:19.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.315+0000 7f7250ffb740 1 -- 172.21.15.27:0/3685776866 wait complete. 2024-04-25T00:23:19.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.315+0000 7f7250ffb740 1 -- start start 2024-04-25T00:23:19.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.315+0000 7f7250ffb740 1 Processor -- start 2024-04-25T00:23:19.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.316+0000 7f7250ffb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d14fea0 0x55dd7d153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.316+0000 7f7250ffb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d15a570 0x55dd7d154410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.316+0000 7f7250ffb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d154950 0x55dd7d214650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.316+0000 7f7250ffb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7234011370 con 0x55dd7d15a570 2024-04-25T00:23:19.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.316+0000 7f7250ffb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7234010c70 con 0x55dd7d14fea0 2024-04-25T00:23:19.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.316+0000 7f7250ffb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f723400df00 con 0x55dd7d154950 2024-04-25T00:23:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.316+0000 7f7250ffb740 10 client.0 _ll_register_callbacks cb 0x7f723c009600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.316+0000 7f7247fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d154950 0x55dd7d214650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.316+0000 7f72477fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d14fea0 0x55dd7d153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.316+0000 7f7247fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d154950 0x55dd7d214650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:55990/0 (socket says 172.21.15.27:55990) 2024-04-25T00:23:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.316+0000 7f7247fff640 1 -- 172.21.15.27:0/1279278951 learned_addr learned my addr 172.21.15.27:0/1279278951 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.317+0000 7f7246ffd640 1 --2- 172.21.15.27:0/1279278951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d15a570 0x55dd7d154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.317+0000 7f7246ffd640 1 -- 172.21.15.27:0/1279278951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d154950 msgr2=0x55dd7d214650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.317+0000 7f7246ffd640 1 --2- 172.21.15.27:0/1279278951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d154950 0x55dd7d214650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.317+0000 7f7246ffd640 1 -- 172.21.15.27:0/1279278951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d14fea0 msgr2=0x55dd7d153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.317+0000 7f7246ffd640 1 --2- 172.21.15.27:0/1279278951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d14fea0 0x55dd7d153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.317+0000 7f7246ffd640 1 -- 172.21.15.27:0/1279278951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd7d214b90 con 0x55dd7d15a570 2024-04-25T00:23:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.317+0000 7f7246ffd640 1 --2- 172.21.15.27:0/1279278951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d15a570 0x55dd7d154410 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x55dd7d1587a0 tx=0x7f72400096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87ad0839433be230 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.318+0000 7f7226ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.318+0000 7f7226ffd640 1 -- 172.21.15.27:0/1279278951 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7240028040 con 0x55dd7d15a570 2024-04-25T00:23:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.318+0000 7f7250ffb740 1 -- 172.21.15.27:0/1279278951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f723c01d9c0 con 0x55dd7d15a570 2024-04-25T00:23:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.318+0000 7f7226ffd640 1 -- 172.21.15.27:0/1279278951 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f724000b240 con 0x55dd7d15a570 2024-04-25T00:23:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.319+0000 7f7226ffd640 1 -- 172.21.15.27:0/1279278951 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f724001ab60 con 0x55dd7d15a570 2024-04-25T00:23:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.319+0000 7f7250ffb740 10 client.16581 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.319+0000 7f7250ffb740 1 -- 172.21.15.27:0/1279278951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55dd7d215e30 con 0x55dd7d15a570 2024-04-25T00:23:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.319+0000 7f7226ffd640 1 -- 172.21.15.27:0/1279278951 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f723c01d9c0 con 0x55dd7d15a570 2024-04-25T00:23:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.320+0000 7f7226ffd640 1 -- 172.21.15.27:0/1279278951 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f7240023460 con 0x55dd7d15a570 2024-04-25T00:23:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.321+0000 7f7250ffb740 10 client.16581 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.321+0000 7f7250ffb740 10 client.16581 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:23:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.321+0000 7f7250ffb740 20 client.16581 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.321+0000 7f7250ffb740 1 --2- 172.21.15.27:0/1279278951 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55dd7d216510 0x55dd7d218930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.321+0000 7f7250ffb740 4 client.16581 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:23:19.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.321+0000 7f7250ffb740 1 -- 172.21.15.27:0/1279278951 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dd7d0bbf00 con 0x55dd7d216510 2024-04-25T00:23:19.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.321+0000 7f72477fe640 1 --2- 172.21.15.27:0/1279278951 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55dd7d216510 0x55dd7d218930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.322+0000 7f72477fe640 1 --2- 172.21.15.27:0/1279278951 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55dd7d216510 0x55dd7d218930 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9836a3d8f5a5cf67 server_cookie=e50d49d082a0c4db in_seq=0 out_seq=0 2024-04-25T00:23:19.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.322+0000 7f7226ffd640 10 client.16581 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:23:19.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.322+0000 7f7226ffd640 1 -- 172.21.15.27:0/1279278951 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dd7d0bbf00 con 0x55dd7d216510 2024-04-25T00:23:19.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.322+0000 7f7226ffd640 10 client.16581 handle_command_reply: tid=0 2024-04-25T00:23:19.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.322+0000 7f7250ffb740 1 client.16581 shutdown 2024-04-25T00:23:19.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.322+0000 7f7250ffb740 1 -- 172.21.15.27:0/1279278951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d15a570 msgr2=0x55dd7d154410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.322+0000 7f7250ffb740 1 --2- 172.21.15.27:0/1279278951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d15a570 0x55dd7d154410 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x55dd7d1587a0 tx=0x7f72400096d0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.322+0000 7f7250ffb740 1 -- 172.21.15.27:0/1279278951 shutdown_connections 2024-04-25T00:23:19.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.322+0000 7f7250ffb740 1 --2- 172.21.15.27:0/1279278951 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55dd7d216510 0x55dd7d218930 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.322+0000 7f7250ffb740 1 --2- 172.21.15.27:0/1279278951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd7d154950 0x55dd7d214650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.322+0000 7f7250ffb740 1 --2- 172.21.15.27:0/1279278951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd7d15a570 0x55dd7d154410 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.322+0000 7f7250ffb740 1 --2- 172.21.15.27:0/1279278951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd7d14fea0 0x55dd7d153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.322+0000 7f7250ffb740 1 -- 172.21.15.27:0/1279278951 >> 172.21.15.27:0/1279278951 conn(0x55dd7d14b8a0 msgr2=0x55dd7d1515a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.322+0000 7f7226ffd640 0 client.16581 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:23:19.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.322+0000 7f7250ffb740 1 -- 172.21.15.27:0/1279278951 shutdown_connections 2024-04-25T00:23:19.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.322+0000 7f7250ffb740 1 -- 172.21.15.27:0/1279278951 wait complete. 2024-04-25T00:23:19.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.323+0000 7f7250ffb740 20 client.16581 trim_cache size 0 max 16384 2024-04-25T00:23:19.337 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:23:19.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.324+0000 7f724e212640 1 -- 172.21.15.27:0/3911914940 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f72300a2e90 msgr2=0x7f72300a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.324+0000 7f724e212640 1 --2- 172.21.15.27:0/3911914940 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f72300a2e90 0x7f72300a5350 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7240009fd0 tx=0x7f7240009f40 comp rx=0 tx=0).stop 2024-04-25T00:23:19.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.324+0000 7f724e212640 1 -- 172.21.15.27:0/3911914940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f724811b350 msgr2=0x7f724811b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.324+0000 7f724e212640 1 --2- 172.21.15.27:0/3911914940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f724811b350 0x7f724811b800 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f723802aa00 tx=0x7f72380045e0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.324+0000 7f724e212640 1 -- 172.21.15.27:0/3911914940 shutdown_connections 2024-04-25T00:23:19.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.324+0000 7f724e212640 1 --2- 172.21.15.27:0/3911914940 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f72300a2e90 0x7f72300a5350 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.324+0000 7f724e212640 1 --2- 172.21.15.27:0/3911914940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f724811b350 0x7f724811b800 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.324+0000 7f724e212640 1 --2- 172.21.15.27:0/3911914940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f724810ff90 0x7f7248116df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.324+0000 7f724e212640 1 --2- 172.21.15.27:0/3911914940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724810dfc0 0x7f72481168b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.324+0000 7f724e212640 1 -- 172.21.15.27:0/3911914940 >> 172.21.15.27:0/3911914940 conn(0x7f724806c850 msgr2=0x7f7248072270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.324+0000 7f724e212640 1 -- 172.21.15.27:0/3911914940 shutdown_connections 2024-04-25T00:23:19.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.324+0000 7f724e212640 1 -- 172.21.15.27:0/3911914940 wait complete. 2024-04-25T00:23:19.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T00:23:19.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.509+0000 7f8e01e5d640 1 -- 172.21.15.27:0/2600088196 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8dfc10ff90 msgr2=0x7f8dfc10da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.509+0000 7f8e01e5d640 1 --2- 172.21.15.27:0/2600088196 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8dfc10ff90 0x7f8dfc10da80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8de401a940 tx=0x7f8de403b490 comp rx=0 tx=0).stop 2024-04-25T00:23:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.510+0000 7f8e01e5d640 1 -- 172.21.15.27:0/2600088196 shutdown_connections 2024-04-25T00:23:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.510+0000 7f8e01e5d640 1 --2- 172.21.15.27:0/2600088196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dfc11af20 0x7f8dfc11d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.510+0000 7f8e01e5d640 1 --2- 172.21.15.27:0/2600088196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dfc10dfc0 0x7f8dfc10e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.510+0000 7f8e01e5d640 1 --2- 172.21.15.27:0/2600088196 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8dfc10ff90 0x7f8dfc10da80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.510+0000 7f8e01e5d640 1 -- 172.21.15.27:0/2600088196 >> 172.21.15.27:0/2600088196 conn(0x7f8dfc06c850 msgr2=0x7f8dfc06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.512+0000 7f8e01e5d640 1 -- 172.21.15.27:0/2600088196 shutdown_connections 2024-04-25T00:23:19.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.512+0000 7f8e01e5d640 1 -- 172.21.15.27:0/2600088196 wait complete. 2024-04-25T00:23:19.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.513+0000 7f8e01e5d640 1 Processor -- start 2024-04-25T00:23:19.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.513+0000 7f8e01e5d640 1 -- start start 2024-04-25T00:23:19.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.513+0000 7f8e01e5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dfc10dfc0 0x7f8dfc11a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.513+0000 7f8e01e5d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8dfc10ff90 0x7f8dfc116700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.513+0000 7f8e01e5d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dfc11af20 0x7f8dfc116c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.513+0000 7f8e01e5d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc113120 con 0x7f8dfc11af20 2024-04-25T00:23:19.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.513+0000 7f8e01e5d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc112fa0 con 0x7f8dfc10ff90 2024-04-25T00:23:19.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.513+0000 7f8e01e5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc1132a0 con 0x7f8dfc10dfc0 2024-04-25T00:23:19.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.514+0000 7f8dfbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dfc11af20 0x7f8dfc116c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.514+0000 7f8dfbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dfc11af20 0x7f8dfc116c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49236/0 (socket says 172.21.15.27:49236) 2024-04-25T00:23:19.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.514+0000 7f8dfbfff640 1 -- 172.21.15.27:0/1685442781 learned_addr learned my addr 172.21.15.27:0/1685442781 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:19.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.514+0000 7f8dfaffd640 1 --2- 172.21.15.27:0/1685442781 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8dfc10ff90 0x7f8dfc116700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.515+0000 7f8dfbfff640 1 -- 172.21.15.27:0/1685442781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dfc10dfc0 msgr2=0x7f8dfc11a620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:23:19.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.515+0000 7f8dfbfff640 1 --2- 172.21.15.27:0/1685442781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dfc10dfc0 0x7f8dfc11a620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.515+0000 7f8dfbfff640 1 -- 172.21.15.27:0/1685442781 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8dfc10ff90 msgr2=0x7f8dfc116700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.515+0000 7f8dfbfff640 1 --2- 172.21.15.27:0/1685442781 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8dfc10ff90 0x7f8dfc116700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.515+0000 7f8dfbfff640 1 -- 172.21.15.27:0/1685442781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dfc117370 con 0x7f8dfc11af20 2024-04-25T00:23:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.516+0000 7f8dfbfff640 1 --2- 172.21.15.27:0/1685442781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dfc11af20 0x7f8dfc116c40 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f8df4030980 tx=0x7f8df4030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.516+0000 7f8df8ff9640 1 -- 172.21.15.27:0/1685442781 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df402ecd0 con 0x7f8dfc11af20 2024-04-25T00:23:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.516+0000 7f8df8ff9640 1 -- 172.21.15.27:0/1685442781 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8df402ee70 con 0x7f8dfc11af20 2024-04-25T00:23:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.516+0000 7f8df8ff9640 1 -- 172.21.15.27:0/1685442781 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df40326a0 con 0x7f8dfc11af20 2024-04-25T00:23:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.517+0000 7f8e01e5d640 1 -- 172.21.15.27:0/1685442781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dfc1aacb0 con 0x7f8dfc11af20 2024-04-25T00:23:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.517+0000 7f8e01e5d640 1 -- 172.21.15.27:0/1685442781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dfc1ab060 con 0x7f8dfc11af20 2024-04-25T00:23:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.521+0000 7f8df8ff9640 1 -- 172.21.15.27:0/1685442781 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8df40046c0 con 0x7f8dfc11af20 2024-04-25T00:23:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.522+0000 7f8df8ff9640 1 --2- 172.21.15.27:0/1685442781 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8de80a2f00 0x7f8de80a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.522+0000 7f8df8ff9640 1 -- 172.21.15.27:0/1685442781 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8df40f1c70 con 0x7f8dfc11af20 2024-04-25T00:23:19.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.523+0000 7f8dfb7fe640 1 --2- 172.21.15.27:0/1685442781 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8de80a2f00 0x7f8de80a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.523+0000 7f8dfb7fe640 1 --2- 172.21.15.27:0/1685442781 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8de80a2f00 0x7f8de80a53c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8de4044040 tx=0x7f8de4005fd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.525+0000 7f8e04b6f740 1 Processor -- start 2024-04-25T00:23:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.525+0000 7f8e04b6f740 1 -- start start 2024-04-25T00:23:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.525+0000 7f8e04b6f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5ee580 0x55732d5ee960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.525+0000 7f8e04b6f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55732d5eeea0 0x55732d5f9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.525+0000 7f8e04b6f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5f9570 0x55732d5fb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.525+0000 7f8e04b6f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55732d553670 con 0x55732d5eeea0 2024-04-25T00:23:19.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.525+0000 7f8e04b6f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55732d4fb970 con 0x55732d5ee580 2024-04-25T00:23:19.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.525+0000 7f8e04b6f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55732d54f630 con 0x55732d5f9570 2024-04-25T00:23:19.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.526+0000 7f8dfb7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5ee580 0x55732d5ee960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.526+0000 7f8dfb7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5ee580 0x55732d5ee960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:60754/0 (socket says 172.21.15.27:60754) 2024-04-25T00:23:19.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.526+0000 7f8dfb7fe640 1 -- 172.21.15.27:0/2147761337 learned_addr learned my addr 172.21.15.27:0/2147761337 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:19.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.526+0000 7f8dfb7fe640 1 -- 172.21.15.27:0/2147761337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5f9570 msgr2=0x55732d5fb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.526+0000 7f8dfb7fe640 1 --2- 172.21.15.27:0/2147761337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5f9570 0x55732d5fb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.526+0000 7f8dfb7fe640 1 -- 172.21.15.27:0/2147761337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55732d5eeea0 msgr2=0x55732d5f9030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:23:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.526+0000 7f8dfb7fe640 1 --2- 172.21.15.27:0/2147761337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55732d5eeea0 0x55732d5f9030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.526+0000 7f8dfb7fe640 1 -- 172.21.15.27:0/2147761337 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55732d51f400 con 0x55732d5ee580 2024-04-25T00:23:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.526+0000 7f8dfb7fe640 1 --2- 172.21.15.27:0/2147761337 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5ee580 0x55732d5ee960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8de4004930 tx=0x7f8de400f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a68a37385e385634 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.527+0000 7f8df1ffb640 1 -- 172.21.15.27:0/2147761337 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de4040d50 con 0x55732d5ee580 2024-04-25T00:23:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.527+0000 7f8df1ffb640 1 -- 172.21.15.27:0/2147761337 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8de404f070 con 0x55732d5ee580 2024-04-25T00:23:19.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.527+0000 7f8df1ffb640 1 -- 172.21.15.27:0/2147761337 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de40427a0 con 0x55732d5ee580 2024-04-25T00:23:19.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.528+0000 7f8e04b6f740 1 -- 172.21.15.27:0/2147761337 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5ee580 msgr2=0x55732d5ee960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.528+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/2147761337 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5ee580 0x55732d5ee960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8de4004930 tx=0x7f8de400f000 comp rx=0 tx=0).stop 2024-04-25T00:23:19.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.528+0000 7f8e04b6f740 1 -- 172.21.15.27:0/2147761337 shutdown_connections 2024-04-25T00:23:19.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.528+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/2147761337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5f9570 0x55732d5fb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.528+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/2147761337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55732d5eeea0 0x55732d5f9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.528+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/2147761337 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5ee580 0x55732d5ee960 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.528+0000 7f8e04b6f740 1 -- 172.21.15.27:0/2147761337 >> 172.21.15.27:0/2147761337 conn(0x55732d5ea8a0 msgr2=0x55732d5eac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.530+0000 7f8e04b6f740 1 -- 172.21.15.27:0/2147761337 shutdown_connections 2024-04-25T00:23:19.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.532+0000 7f8e04b6f740 1 -- 172.21.15.27:0/2147761337 wait complete. 2024-04-25T00:23:19.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.532+0000 7f8e04b6f740 1 -- start start 2024-04-25T00:23:19.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.532+0000 7f8e04b6f740 1 Processor -- start 2024-04-25T00:23:19.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.533+0000 7f8e04b6f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5ee580 0x55732d5f9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.533+0000 7f8e04b6f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5eeea0 0x55732d5f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.533+0000 7f8e04b6f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55732d5f9570 0x55732d5f69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.533+0000 7f8e04b6f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55732d5fc300 con 0x55732d5f9570 2024-04-25T00:23:19.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.533+0000 7f8e04b6f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55732d50f050 con 0x55732d5ee580 2024-04-25T00:23:19.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.533+0000 7f8e04b6f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55732d5fc480 con 0x55732d5eeea0 2024-04-25T00:23:19.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.533+0000 7f8e04b6f740 10 client.0 _ll_register_callbacks cb 0x55732d266be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:19.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.535+0000 7f8dfb7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5ee580 0x55732d5f9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.535+0000 7f8dfb7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5ee580 0x55732d5f9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:60758/0 (socket says 172.21.15.27:60758) 2024-04-25T00:23:19.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.535+0000 7f8dfb7fe640 1 -- 172.21.15.27:0/995242338 learned_addr learned my addr 172.21.15.27:0/995242338 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:19.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.535+0000 7f8dfb7fe640 1 -- 172.21.15.27:0/995242338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5eeea0 msgr2=0x55732d5f62f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:23:19.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.535+0000 7f8dfb7fe640 1 --2- 172.21.15.27:0/995242338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5eeea0 0x55732d5f62f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.535+0000 7f8dfb7fe640 1 -- 172.21.15.27:0/995242338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55732d5f9570 msgr2=0x55732d5f69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.535+0000 7f8dfb7fe640 1 --2- 172.21.15.27:0/995242338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55732d5f9570 0x55732d5f69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.535+0000 7f8dfb7fe640 1 -- 172.21.15.27:0/995242338 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55732d4e0dc0 con 0x55732d5ee580 2024-04-25T00:23:19.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.536+0000 7f8dfb7fe640 1 --2- 172.21.15.27:0/995242338 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5ee580 0x55732d5f9000 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8de400fe40 tx=0x7f8de400f680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0674fc67bb2b714 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.537+0000 7f8ddaffd640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T00:23:19.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.537+0000 7f8ddaffd640 1 -- 172.21.15.27:0/995242338 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de4042440 con 0x55732d5ee580 2024-04-25T00:23:19.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.537+0000 7f8e04b6f740 1 -- 172.21.15.27:0/995242338 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8de404f070 con 0x55732d5ee580 2024-04-25T00:23:19.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.537+0000 7f8ddaffd640 1 -- 172.21.15.27:0/995242338 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8de40425e0 con 0x55732d5ee580 2024-04-25T00:23:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.538+0000 7f8ddaffd640 1 -- 172.21.15.27:0/995242338 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f8de4064050 con 0x55732d5ee580 2024-04-25T00:23:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.538+0000 7f8e04b6f740 10 client.26087 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.538+0000 7f8e04b6f740 1 -- 172.21.15.27:0/995242338 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55732d2673f0 con 0x55732d5ee580 2024-04-25T00:23:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.540+0000 7f8ddaffd640 1 -- 172.21.15.27:0/995242338 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8de404f070 con 0x55732d5ee580 2024-04-25T00:23:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.540+0000 7f8ddaffd640 1 -- 172.21.15.27:0/995242338 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f8de404ab10 con 0x55732d5ee580 2024-04-25T00:23:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.540+0000 7f8e04b6f740 10 client.26087 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.540+0000 7f8e04b6f740 10 client.26087 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:23:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.540+0000 7f8e04b6f740 20 client.26087 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.540+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/995242338 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55732d609660 0x55732d60bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.540+0000 7f8e04b6f740 4 client.26087 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:23:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.540+0000 7f8e04b6f740 1 -- 172.21.15.27:0/995242338 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55732d560bb0 con 0x55732d609660 2024-04-25T00:23:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.544+0000 7f8dfaffd640 1 --2- 172.21.15.27:0/995242338 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55732d609660 0x55732d60bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.546+0000 7f8dfaffd640 1 --2- 172.21.15.27:0/995242338 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55732d609660 0x55732d60bac0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7f1d52ba3fd1240b server_cookie=df69fb8fd060dd48 in_seq=0 out_seq=0 2024-04-25T00:23:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.546+0000 7f8ddaffd640 10 client.26087 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:23:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.547+0000 7f8ddaffd640 1 -- 172.21.15.27:0/995242338 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55732d560bb0 con 0x55732d609660 2024-04-25T00:23:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.547+0000 7f8ddaffd640 10 client.26087 handle_command_reply: tid=0 2024-04-25T00:23:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.547+0000 7f8e04b6f740 1 client.26087 shutdown 2024-04-25T00:23:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.547+0000 7f8e04b6f740 1 -- 172.21.15.27:0/995242338 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5ee580 msgr2=0x55732d5f9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.547+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/995242338 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5ee580 0x55732d5f9000 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8de400fe40 tx=0x7f8de400f680 comp rx=0 tx=0).stop 2024-04-25T00:23:19.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.548+0000 7f8e04b6f740 1 -- 172.21.15.27:0/995242338 shutdown_connections 2024-04-25T00:23:19.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.548+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/995242338 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55732d609660 0x55732d60bac0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.548+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/995242338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55732d5f9570 0x55732d5f69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.548+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/995242338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5eeea0 0x55732d5f62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.548+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/995242338 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5ee580 0x55732d5f9000 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.548+0000 7f8e04b6f740 1 -- 172.21.15.27:0/995242338 >> 172.21.15.27:0/995242338 conn(0x55732d5ea8a0 msgr2=0x55732d5e9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.548+0000 7f8ddaffd640 0 client.26087 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:23:19.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.548+0000 7f8e04b6f740 1 -- 172.21.15.27:0/995242338 shutdown_connections 2024-04-25T00:23:19.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.548+0000 7f8e04b6f740 1 -- 172.21.15.27:0/995242338 wait complete. 2024-04-25T00:23:19.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.548+0000 7f8e04b6f740 20 client.26087 trim_cache size 0 max 16384 2024-04-25T00:23:19.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.581+0000 7f8e04b6f740 1 Processor -- start 2024-04-25T00:23:19.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.581+0000 7f8e04b6f740 1 -- start start 2024-04-25T00:23:19.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.581+0000 7f8e04b6f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55732d5ee580 0x55732d5ea2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.581+0000 7f8e04b6f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5eeea0 0x55732d5f8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.581+0000 7f8e04b6f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5f9570 0x55732d611390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.582+0000 7f8e04b6f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc1b5300 con 0x55732d5ee580 2024-04-25T00:23:19.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.582+0000 7f8e04b6f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc1b6ea0 con 0x55732d5f9570 2024-04-25T00:23:19.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.582+0000 7f8e04b6f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc1b59d0 con 0x55732d5eeea0 2024-04-25T00:23:19.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.582+0000 7f8dfaffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5eeea0 0x55732d5f8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.582+0000 7f8dfaffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5eeea0 0x55732d5f8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:56026/0 (socket says 172.21.15.27:56026) 2024-04-25T00:23:19.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.582+0000 7f8dfaffd640 1 -- 172.21.15.27:0/1526909934 learned_addr learned my addr 172.21.15.27:0/1526909934 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:19.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.582+0000 7f8dfbfff640 1 --2- 172.21.15.27:0/1526909934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5f9570 0x55732d611390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.583+0000 7f8dfaffd640 1 -- 172.21.15.27:0/1526909934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5f9570 msgr2=0x55732d611390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.583+0000 7f8dfaffd640 1 --2- 172.21.15.27:0/1526909934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5f9570 0x55732d611390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.583+0000 7f8dfaffd640 1 -- 172.21.15.27:0/1526909934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55732d5ee580 msgr2=0x55732d5ea2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.583+0000 7f8dfaffd640 1 --2- 172.21.15.27:0/1526909934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55732d5ee580 0x55732d5ea2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.583+0000 7f8dfaffd640 1 -- 172.21.15.27:0/1526909934 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55732d5fbcd0 con 0x55732d5eeea0 2024-04-25T00:23:19.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.586+0000 7f8dfaffd640 1 --2- 172.21.15.27:0/1526909934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5eeea0 0x55732d5f8b30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc117e70 tx=0x7f8dec00c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fa0ce622e19db22 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.588+0000 7f8df17fa640 1 -- 172.21.15.27:0/1526909934 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec017630 con 0x55732d5eeea0 2024-04-25T00:23:19.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.588+0000 7f8df17fa640 1 -- 172.21.15.27:0/1526909934 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dec0177d0 con 0x55732d5eeea0 2024-04-25T00:23:19.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.589+0000 7f8e04b6f740 1 -- 172.21.15.27:0/1526909934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5eeea0 msgr2=0x55732d5f8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.589+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/1526909934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5eeea0 0x55732d5f8b30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc117e70 tx=0x7f8dec00c830 comp rx=0 tx=0).stop 2024-04-25T00:23:19.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.589+0000 7f8e04b6f740 1 -- 172.21.15.27:0/1526909934 shutdown_connections 2024-04-25T00:23:19.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.589+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/1526909934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5f9570 0x55732d611390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.589+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/1526909934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5eeea0 0x55732d5f8b30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.589+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/1526909934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55732d5ee580 0x55732d5ea2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.589+0000 7f8e04b6f740 1 -- 172.21.15.27:0/1526909934 >> 172.21.15.27:0/1526909934 conn(0x55732d5ea8a0 msgr2=0x55732d5f05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.591+0000 7f8e04b6f740 1 -- 172.21.15.27:0/1526909934 shutdown_connections 2024-04-25T00:23:19.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.591+0000 7f8e04b6f740 1 -- 172.21.15.27:0/1526909934 wait complete. 2024-04-25T00:23:19.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.591+0000 7f8e04b6f740 1 -- start start 2024-04-25T00:23:19.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.591+0000 7f8e04b6f740 1 Processor -- start 2024-04-25T00:23:19.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.592+0000 7f8e04b6f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55732d5ee580 0x55732d601ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.592+0000 7f8e04b6f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5f9570 0x55732d5f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.592+0000 7f8e04b6f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5f3f50 0x55732d5f2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.592+0000 7f8e04b6f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc1b67d0 con 0x55732d5ee580 2024-04-25T00:23:19.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.592+0000 7f8e04b6f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc1b60d0 con 0x55732d5f3f50 2024-04-25T00:23:19.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.592+0000 7f8e04b6f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc1b3290 con 0x55732d5f9570 2024-04-25T00:23:19.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.592+0000 7f8e04b6f740 10 client.0 _ll_register_callbacks cb 0x7f8de401a790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:19.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.592+0000 7f8dfbfff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5f3f50 0x55732d5f2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.592+0000 7f8dfbfff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5f3f50 0x55732d5f2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:60782/0 (socket says 172.21.15.27:60782) 2024-04-25T00:23:19.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.592+0000 7f8dfb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55732d5ee580 0x55732d601ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.592+0000 7f8dfbfff640 1 -- 172.21.15.27:0/2810598878 learned_addr learned my addr 172.21.15.27:0/2810598878 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.592+0000 7f8dfaffd640 1 --2- 172.21.15.27:0/2810598878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5f9570 0x55732d5f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.592+0000 7f8dfbfff640 1 -- 172.21.15.27:0/2810598878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5f9570 msgr2=0x55732d5f39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.592+0000 7f8dfbfff640 1 --2- 172.21.15.27:0/2810598878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5f9570 0x55732d5f39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.592+0000 7f8dfbfff640 1 -- 172.21.15.27:0/2810598878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55732d5ee580 msgr2=0x55732d601ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.592+0000 7f8dfbfff640 1 --2- 172.21.15.27:0/2810598878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55732d5ee580 0x55732d601ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.592+0000 7f8dfbfff640 1 -- 172.21.15.27:0/2810598878 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55732d3317d0 con 0x55732d5f3f50 2024-04-25T00:23:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.593+0000 7f8dfbfff640 1 --2- 172.21.15.27:0/2810598878 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5f3f50 0x55732d5f2560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8df40bdd70 tx=0x7f8df40be4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db84bb70604c31ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.594+0000 7f8ddaffd640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T00:23:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.594+0000 7f8ddaffd640 1 -- 172.21.15.27:0/2810598878 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df40db060 con 0x55732d5f3f50 2024-04-25T00:23:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.594+0000 7f8ddaffd640 1 -- 172.21.15.27:0/2810598878 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8df40ef290 con 0x55732d5f3f50 2024-04-25T00:23:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.594+0000 7f8e04b6f740 1 -- 172.21.15.27:0/2810598878 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55732d55af00 con 0x55732d5f3f50 2024-04-25T00:23:19.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.594+0000 7f8ddaffd640 1 -- 172.21.15.27:0/2810598878 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f8df40ca030 con 0x55732d5f3f50 2024-04-25T00:23:19.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.596+0000 7f8e04b6f740 10 client.26093 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:19.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.596+0000 7f8e04b6f740 1 -- 172.21.15.27:0/2810598878 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55732d691420 con 0x55732d5f3f50 2024-04-25T00:23:19.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.598+0000 7f8ddaffd640 1 -- 172.21.15.27:0/2810598878 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55732d55af00 con 0x55732d5f3f50 2024-04-25T00:23:19.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.598+0000 7f8ddaffd640 1 -- 172.21.15.27:0/2810598878 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f8df40bcca0 con 0x55732d5f3f50 2024-04-25T00:23:19.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.598+0000 7f8e04b6f740 10 client.26093 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:19.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.598+0000 7f8e04b6f740 10 client.26093 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:23:19.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.598+0000 7f8e04b6f740 20 client.26093 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:19.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.598+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/2810598878 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55732d691ba0 0x55732d693fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.598+0000 7f8e04b6f740 4 client.26093 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:23:19.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.598+0000 7f8e04b6f740 1 -- 172.21.15.27:0/2810598878 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55732d4d8820 con 0x55732d691ba0 2024-04-25T00:23:19.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.600+0000 7f8dfb7fe640 1 --2- 172.21.15.27:0/2810598878 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55732d691ba0 0x55732d693fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.601+0000 7f8dfb7fe640 1 --2- 172.21.15.27:0/2810598878 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55732d691ba0 0x55732d693fc0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3cad9a6fe036b5ea server_cookie=763171317738ce4a in_seq=0 out_seq=0 2024-04-25T00:23:19.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.601+0000 7f8ddaffd640 10 client.26093 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:23:19.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.601+0000 7f8ddaffd640 1 -- 172.21.15.27:0/2810598878 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55732d4d8820 con 0x55732d691ba0 2024-04-25T00:23:19.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.601+0000 7f8ddaffd640 10 client.26093 handle_command_reply: tid=0 2024-04-25T00:23:19.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.601+0000 7f8e04b6f740 1 client.26093 shutdown 2024-04-25T00:23:19.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.602+0000 7f8e04b6f740 1 -- 172.21.15.27:0/2810598878 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5f3f50 msgr2=0x55732d5f2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.602+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/2810598878 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5f3f50 0x55732d5f2560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8df40bdd70 tx=0x7f8df40be4e0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.604+0000 7f8e04b6f740 1 -- 172.21.15.27:0/2810598878 shutdown_connections 2024-04-25T00:23:19.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.604+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/2810598878 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55732d691ba0 0x55732d693fc0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.604+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/2810598878 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55732d5f3f50 0x55732d5f2560 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.604+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/2810598878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55732d5f9570 0x55732d5f39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.604+0000 7f8e04b6f740 1 --2- 172.21.15.27:0/2810598878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55732d5ee580 0x55732d601ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.604+0000 7f8e04b6f740 1 -- 172.21.15.27:0/2810598878 >> 172.21.15.27:0/2810598878 conn(0x55732d5ea8a0 msgr2=0x55732d5f0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.604+0000 7f8ddaffd640 0 client.26093 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:23:19.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.608+0000 7f8e04b6f740 1 -- 172.21.15.27:0/2810598878 shutdown_connections 2024-04-25T00:23:19.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.610+0000 7f8e04b6f740 1 -- 172.21.15.27:0/2810598878 wait complete. 2024-04-25T00:23:19.613 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:23:19.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.611+0000 7f8e04b6f740 20 client.26093 trim_cache size 0 max 16384 2024-04-25T00:23:19.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.613+0000 7f8e01e5d640 1 -- 172.21.15.27:0/1685442781 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8de80a2f00 msgr2=0x7f8de80a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.613+0000 7f8e01e5d640 1 --2- 172.21.15.27:0/1685442781 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8de80a2f00 0x7f8de80a53c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8de4044040 tx=0x7f8de4005fd0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.613+0000 7f8e01e5d640 1 -- 172.21.15.27:0/1685442781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dfc11af20 msgr2=0x7f8dfc116c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.613+0000 7f8e01e5d640 1 --2- 172.21.15.27:0/1685442781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dfc11af20 0x7f8dfc116c40 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f8df4030980 tx=0x7f8df4030d80 comp rx=0 tx=0).stop 2024-04-25T00:23:19.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.615+0000 7f8e01e5d640 1 -- 172.21.15.27:0/1685442781 shutdown_connections 2024-04-25T00:23:19.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.615+0000 7f8e01e5d640 1 --2- 172.21.15.27:0/1685442781 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8de80a2f00 0x7f8de80a53c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.615+0000 7f8e01e5d640 1 --2- 172.21.15.27:0/1685442781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dfc11af20 0x7f8dfc116c40 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.615+0000 7f8e01e5d640 1 --2- 172.21.15.27:0/1685442781 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8dfc10ff90 0x7f8dfc116700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.615+0000 7f8e01e5d640 1 --2- 172.21.15.27:0/1685442781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dfc10dfc0 0x7f8dfc11a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.615+0000 7f8e01e5d640 1 -- 172.21.15.27:0/1685442781 >> 172.21.15.27:0/1685442781 conn(0x7f8dfc06c850 msgr2=0x7f8dfc11c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.617+0000 7f8e01e5d640 1 -- 172.21.15.27:0/1685442781 shutdown_connections 2024-04-25T00:23:19.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.619+0000 7f8e01e5d640 1 -- 172.21.15.27:0/1685442781 wait complete. 2024-04-25T00:23:19.638 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T00:23:19.639 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:19 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2369253807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:23:19.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:19 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2369253807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:23:19.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:19 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2369253807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:23:19.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.805+0000 7fb82ae02640 1 -- 172.21.15.27:0/3915374844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb824100600 msgr2=0x7fb824100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.805+0000 7fb82ae02640 1 --2- 172.21.15.27:0/3915374844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb824100600 0x7fb824100a80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb814009cd0 tx=0x7fb81402a8e0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.805+0000 7fb82ae02640 1 -- 172.21.15.27:0/3915374844 shutdown_connections 2024-04-25T00:23:19.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.805+0000 7fb82ae02640 1 --2- 172.21.15.27:0/3915374844 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb824100fc0 0x7fb82410d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.805+0000 7fb82ae02640 1 --2- 172.21.15.27:0/3915374844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb824100600 0x7fb824100a80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.805+0000 7fb82ae02640 1 --2- 172.21.15.27:0/3915374844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8241066a0 0x7fb824106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.805+0000 7fb82ae02640 1 -- 172.21.15.27:0/3915374844 >> 172.21.15.27:0/3915374844 conn(0x7fb8240776e0 msgr2=0x7fb824077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.806+0000 7fb82ae02640 1 -- 172.21.15.27:0/3915374844 shutdown_connections 2024-04-25T00:23:19.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.806+0000 7fb82ae02640 1 -- 172.21.15.27:0/3915374844 wait complete. 2024-04-25T00:23:19.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.807+0000 7fb82ae02640 1 Processor -- start 2024-04-25T00:23:19.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.807+0000 7fb82ae02640 1 -- start start 2024-04-25T00:23:19.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.808+0000 7fb82ae02640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb824100600 0x7fb824196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.808+0000 7fb82ae02640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb824100fc0 0x7fb824197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.808+0000 7fb82ae02640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8241066a0 0x7fb82419b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.808+0000 7fb82ae02640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb824110a70 con 0x7fb8241066a0 2024-04-25T00:23:19.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.808+0000 7fb82ae02640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb8241108f0 con 0x7fb824100600 2024-04-25T00:23:19.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.808+0000 7fb82ae02640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb824110bf0 con 0x7fb824100fc0 2024-04-25T00:23:19.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.809+0000 7fb828b9e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb824100600 0x7fb824196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.809+0000 7fb82939f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8241066a0 0x7fb82419b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.809+0000 7fb82939f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8241066a0 0x7fb82419b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49280/0 (socket says 172.21.15.27:49280) 2024-04-25T00:23:19.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.809+0000 7fb82939f640 1 -- 172.21.15.27:0/2632193704 learned_addr learned my addr 172.21.15.27:0/2632193704 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:19.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.809+0000 7fb82939f640 1 -- 172.21.15.27:0/2632193704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb824100fc0 msgr2=0x7fb824197430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:23:19.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.809+0000 7fb82939f640 1 --2- 172.21.15.27:0/2632193704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb824100fc0 0x7fb824197430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.809+0000 7fb82939f640 1 -- 172.21.15.27:0/2632193704 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb824100600 msgr2=0x7fb824196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.809+0000 7fb82939f640 1 --2- 172.21.15.27:0/2632193704 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb824100600 0x7fb824196ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.809+0000 7fb82939f640 1 -- 172.21.15.27:0/2632193704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb82419bee0 con 0x7fb8241066a0 2024-04-25T00:23:19.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.810+0000 7fb82939f640 1 --2- 172.21.15.27:0/2632193704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8241066a0 0x7fb82419b7b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb80c00e970 tx=0x7fb80c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.816+0000 7fb821ffb640 1 -- 172.21.15.27:0/2632193704 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c00ccc0 con 0x7fb8241066a0 2024-04-25T00:23:19.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.816+0000 7fb821ffb640 1 -- 172.21.15.27:0/2632193704 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb80c00ce60 con 0x7fb8241066a0 2024-04-25T00:23:19.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.817+0000 7fb821ffb640 1 -- 172.21.15.27:0/2632193704 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c010490 con 0x7fb8241066a0 2024-04-25T00:23:19.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.818+0000 7fb82ae02640 1 -- 172.21.15.27:0/2632193704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb82419c1d0 con 0x7fb8241066a0 2024-04-25T00:23:19.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.818+0000 7fb82ae02640 1 -- 172.21.15.27:0/2632193704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8241a3a60 con 0x7fb8241066a0 2024-04-25T00:23:19.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.823+0000 7fb82dbea740 1 Processor -- start 2024-04-25T00:23:19.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.824+0000 7fb821ffb640 1 -- 172.21.15.27:0/2632193704 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb80c004680 con 0x7fb8241066a0 2024-04-25T00:23:19.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.824+0000 7fb821ffb640 1 --2- 172.21.15.27:0/2632193704 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb8100a2be0 0x7fb8100a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.825+0000 7fb82dbea740 1 -- start start 2024-04-25T00:23:19.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.825+0000 7fb821ffb640 1 -- 172.21.15.27:0/2632193704 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fb80c0cfce0 con 0x7fb8241066a0 2024-04-25T00:23:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.825+0000 7fb82dbea740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed89f6d0 0x5565ed89fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.825+0000 7fb82dbea740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 0x5565ed892b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.825+0000 7fb82dbea740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed8930a0 0x5565ed8a75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.825+0000 7fb82dbea740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5565ed7f6670 con 0x5565ed89f6d0 2024-04-25T00:23:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.825+0000 7fb82dbea740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5565ed79e970 con 0x5565ed8930a0 2024-04-25T00:23:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.825+0000 7fb82dbea740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5565ed7f2630 con 0x5565ed892700 2024-04-25T00:23:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.826+0000 7fb823fff640 1 --2- 172.21.15.27:0/2632193704 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb8100a2be0 0x7fb8100a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.826+0000 7fb823fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 0x5565ed892b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.826+0000 7fb82939f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed8930a0 0x5565ed8a75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.826+0000 7fb82939f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed8930a0 0x5565ed8a75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:60804/0 (socket says 172.21.15.27:60804) 2024-04-25T00:23:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.826+0000 7fb82939f640 1 -- 172.21.15.27:0/1138301530 learned_addr learned my addr 172.21.15.27:0/1138301530 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:19.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.827+0000 7fb823fff640 1 -- 172.21.15.27:0/1138301530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed8930a0 msgr2=0x5565ed8a75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.827+0000 7fb823fff640 1 --2- 172.21.15.27:0/1138301530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed8930a0 0x5565ed8a75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.827+0000 7fb823fff640 1 -- 172.21.15.27:0/1138301530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed89f6d0 msgr2=0x5565ed89fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.827+0000 7fb823fff640 1 --2- 172.21.15.27:0/1138301530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed89f6d0 0x5565ed89fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.827+0000 7fb823fff640 1 -- 172.21.15.27:0/1138301530 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565ed7c2400 con 0x5565ed892700 2024-04-25T00:23:19.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.827+0000 7fb823fff640 1 --2- 172.21.15.27:0/1138301530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 0x5565ed892b60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb814009430 tx=0x7fb814045000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c014f8004114271 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.827+0000 7fb823fff640 1 --2- 172.21.15.27:0/2632193704 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb8100a2be0 0x7fb8100a50a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb814004af0 tx=0x7fb814005fd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.828+0000 7fb82ae02640 1 -- 172.21.15.27:0/1138301530 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb814003e00 con 0x5565ed892700 2024-04-25T00:23:19.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.828+0000 7fb82ae02640 1 -- 172.21.15.27:0/1138301530 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb814036040 con 0x5565ed892700 2024-04-25T00:23:19.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.828+0000 7fb82ae02640 1 -- 172.21.15.27:0/1138301530 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb814030cf0 con 0x5565ed892700 2024-04-25T00:23:19.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.829+0000 7fb82dbea740 1 -- 172.21.15.27:0/1138301530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 msgr2=0x5565ed892b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.829+0000 7fb82dbea740 1 --2- 172.21.15.27:0/1138301530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 0x5565ed892b60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb814009430 tx=0x7fb814045000 comp rx=0 tx=0).stop 2024-04-25T00:23:19.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.832+0000 7fb82dbea740 1 -- 172.21.15.27:0/1138301530 shutdown_connections 2024-04-25T00:23:19.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.832+0000 7fb82dbea740 1 --2- 172.21.15.27:0/1138301530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed8930a0 0x5565ed8a75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.832+0000 7fb82dbea740 1 --2- 172.21.15.27:0/1138301530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 0x5565ed892b60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.832+0000 7fb82dbea740 1 --2- 172.21.15.27:0/1138301530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed89f6d0 0x5565ed89fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.832+0000 7fb82dbea740 1 -- 172.21.15.27:0/1138301530 >> 172.21.15.27:0/1138301530 conn(0x5565ed88d5f0 msgr2=0x5565ed88d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.835+0000 7fb82dbea740 1 -- 172.21.15.27:0/1138301530 shutdown_connections 2024-04-25T00:23:19.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.835+0000 7fb82dbea740 1 -- 172.21.15.27:0/1138301530 wait complete. 2024-04-25T00:23:19.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.835+0000 7fb82dbea740 1 -- start start 2024-04-25T00:23:19.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.835+0000 7fb82dbea740 1 Processor -- start 2024-04-25T00:23:19.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.836+0000 7fb82dbea740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 0x5565ed8a7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.836+0000 7fb82dbea740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed8930a0 0x5565ed8a4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.836+0000 7fb82dbea740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed89f6d0 0x5565ed8a4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.836+0000 7fb82dbea740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5565ed893bb0 con 0x5565ed8930a0 2024-04-25T00:23:19.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.836+0000 7fb82dbea740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5565ed7b2050 con 0x5565ed89f6d0 2024-04-25T00:23:19.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.836+0000 7fb82dbea740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5565ed893d30 con 0x5565ed892700 2024-04-25T00:23:19.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.836+0000 7fb82dbea740 10 client.0 _ll_register_callbacks cb 0x5565ed509be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:19.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.837+0000 7fb823fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed8930a0 0x5565ed8a4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.837+0000 7fb823fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed8930a0 0x5565ed8a4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49302/0 (socket says 172.21.15.27:49302) 2024-04-25T00:23:19.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.837+0000 7fb823fff640 1 -- 172.21.15.27:0/3286297194 learned_addr learned my addr 172.21.15.27:0/3286297194 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:19.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.837+0000 7fb828b9e640 1 --2- 172.21.15.27:0/3286297194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 0x5565ed8a7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.837+0000 7fb82939f640 1 --2- 172.21.15.27:0/3286297194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed89f6d0 0x5565ed8a4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.838+0000 7fb828b9e640 1 -- 172.21.15.27:0/3286297194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed89f6d0 msgr2=0x5565ed8a4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.838+0000 7fb828b9e640 1 --2- 172.21.15.27:0/3286297194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed89f6d0 0x5565ed8a4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.838+0000 7fb828b9e640 1 -- 172.21.15.27:0/3286297194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed8930a0 msgr2=0x5565ed8a4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.838+0000 7fb828b9e640 1 --2- 172.21.15.27:0/3286297194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed8930a0 0x5565ed8a4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.838+0000 7fb828b9e640 1 -- 172.21.15.27:0/3286297194 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565ed783dc0 con 0x5565ed892700 2024-04-25T00:23:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.839+0000 7fb823fff640 1 --2- 172.21.15.27:0/3286297194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed8930a0 0x5565ed8a4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:23:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.839+0000 7fb828b9e640 1 --2- 172.21.15.27:0/3286297194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 0x5565ed8a7410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009ee0 tx=0x7fb81c00e6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=357d5e151d62ab3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.841+0000 7fb803fff640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:23:19.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.841+0000 7fb803fff640 1 -- 172.21.15.27:0/3286297194 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c016070 con 0x5565ed892700 2024-04-25T00:23:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.841+0000 7fb803fff640 1 -- 172.21.15.27:0/3286297194 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fb81c005220 con 0x5565ed892700 2024-04-25T00:23:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.842+0000 7fb82dbea740 1 -- 172.21.15.27:0/3286297194 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb814036040 con 0x5565ed892700 2024-04-25T00:23:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.843+0000 7fb803fff640 1 -- 172.21.15.27:0/3286297194 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fb81c01f050 con 0x5565ed892700 2024-04-25T00:23:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.844+0000 7fb82dbea740 10 client.26080 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.844+0000 7fb82dbea740 1 -- 172.21.15.27:0/3286297194 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5565ed50a3f0 con 0x5565ed892700 2024-04-25T00:23:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.844+0000 7fb803fff640 1 -- 172.21.15.27:0/3286297194 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb814036040 con 0x5565ed892700 2024-04-25T00:23:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.845+0000 7fb803fff640 1 -- 172.21.15.27:0/3286297194 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7fb81c00f040 con 0x5565ed892700 2024-04-25T00:23:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.845+0000 7fb82dbea740 10 client.26080 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.846+0000 7fb82dbea740 10 client.26080 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:23:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.846+0000 7fb82dbea740 20 client.26080 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.846+0000 7fb82dbea740 1 --2- 172.21.15.27:0/3286297194 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5565ed8bda90 0x5565ed8bfef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.846+0000 7fb82dbea740 4 client.26080 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:23:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.846+0000 7fb82dbea740 1 -- 172.21.15.27:0/3286297194 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565ed803bb0 con 0x5565ed8bda90 2024-04-25T00:23:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.847+0000 7fb823fff640 1 --2- 172.21.15.27:0/3286297194 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5565ed8bda90 0x5565ed8bfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.852+0000 7fb823fff640 1 --2- 172.21.15.27:0/3286297194 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5565ed8bda90 0x5565ed8bfef0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cc8c0644c7b23f9a server_cookie=1540385301631871 in_seq=0 out_seq=0 2024-04-25T00:23:19.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.852+0000 7fb803fff640 10 client.26080 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:23:19.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.853+0000 7fb803fff640 1 -- 172.21.15.27:0/3286297194 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5565ed803bb0 con 0x5565ed8bda90 2024-04-25T00:23:19.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.853+0000 7fb803fff640 10 client.26080 handle_command_reply: tid=0 2024-04-25T00:23:19.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.853+0000 7fb82dbea740 1 client.26080 shutdown 2024-04-25T00:23:19.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.859+0000 7fb82dbea740 1 -- 172.21.15.27:0/3286297194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 msgr2=0x5565ed8a7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.859+0000 7fb82dbea740 1 --2- 172.21.15.27:0/3286297194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 0x5565ed8a7410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009ee0 tx=0x7fb81c00e6c0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.860+0000 7fb82dbea740 1 -- 172.21.15.27:0/3286297194 shutdown_connections 2024-04-25T00:23:19.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.860+0000 7fb82dbea740 1 --2- 172.21.15.27:0/3286297194 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5565ed8bda90 0x5565ed8bfef0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.860+0000 7fb82dbea740 1 --2- 172.21.15.27:0/3286297194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed89f6d0 0x5565ed8a4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.860+0000 7fb82dbea740 1 --2- 172.21.15.27:0/3286297194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed8930a0 0x5565ed8a4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.860+0000 7fb82dbea740 1 --2- 172.21.15.27:0/3286297194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 0x5565ed8a7410 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.860+0000 7fb82dbea740 1 -- 172.21.15.27:0/3286297194 >> 172.21.15.27:0/3286297194 conn(0x5565ed88d5f0 msgr2=0x5565ed8a0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.860+0000 7fb803fff640 0 client.26080 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:23:19.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.861+0000 7fb82dbea740 1 -- 172.21.15.27:0/3286297194 shutdown_connections 2024-04-25T00:23:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.865+0000 7fb82dbea740 1 -- 172.21.15.27:0/3286297194 wait complete. 2024-04-25T00:23:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.866+0000 7fb82dbea740 20 client.26080 trim_cache size 0 max 16384 2024-04-25T00:23:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.896+0000 7fb82dbea740 1 Processor -- start 2024-04-25T00:23:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.896+0000 7fb82dbea740 1 -- start start 2024-04-25T00:23:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.897+0000 7fb82dbea740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed892700 0x5565ed8a6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.897+0000 7fb82dbea740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed8930a0 0x5565ed8c5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.897+0000 7fb82dbea740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed89f6d0 0x5565ed8c5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.897+0000 7fb82dbea740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb8241ac750 con 0x5565ed892700 2024-04-25T00:23:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.897+0000 7fb82dbea740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb8241ae2f0 con 0x5565ed89f6d0 2024-04-25T00:23:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.897+0000 7fb82dbea740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb8241ace20 con 0x5565ed8930a0 2024-04-25T00:23:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.897+0000 7fb828b9e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed892700 0x5565ed8a6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.897+0000 7fb828b9e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed892700 0x5565ed8a6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49310/0 (socket says 172.21.15.27:49310) 2024-04-25T00:23:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.897+0000 7fb828b9e640 1 -- 172.21.15.27:0/3938887985 learned_addr learned my addr 172.21.15.27:0/3938887985 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:19.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.897+0000 7fb823fff640 1 --2- 172.21.15.27:0/3938887985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed8930a0 0x5565ed8c5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.897+0000 7fb82939f640 1 --2- 172.21.15.27:0/3938887985 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed89f6d0 0x5565ed8c5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.897+0000 7fb823fff640 1 -- 172.21.15.27:0/3938887985 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed89f6d0 msgr2=0x5565ed8c5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.897+0000 7fb823fff640 1 --2- 172.21.15.27:0/3938887985 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed89f6d0 0x5565ed8c5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.897+0000 7fb823fff640 1 -- 172.21.15.27:0/3938887985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed892700 msgr2=0x5565ed8a6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.897+0000 7fb823fff640 1 --2- 172.21.15.27:0/3938887985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed892700 0x5565ed8a6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.897+0000 7fb823fff640 1 -- 172.21.15.27:0/3938887985 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565ed892220 con 0x5565ed8930a0 2024-04-25T00:23:19.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.898+0000 7fb823fff640 1 --2- 172.21.15.27:0/3938887985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed8930a0 0x5565ed8c5800 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb81402a920 tx=0x7fb814000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bac2691cf7ff0624 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.899+0000 7fb828b9e640 1 --2- 172.21.15.27:0/3938887985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed892700 0x5565ed8a6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:19.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.899+0000 7fb81a7fc640 1 -- 172.21.15.27:0/3938887985 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8140305a0 con 0x5565ed8930a0 2024-04-25T00:23:19.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.899+0000 7fb81a7fc640 1 -- 172.21.15.27:0/3938887985 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb814036040 con 0x5565ed8930a0 2024-04-25T00:23:19.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.899+0000 7fb81a7fc640 1 -- 172.21.15.27:0/3938887985 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb814030940 con 0x5565ed8930a0 2024-04-25T00:23:19.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.900+0000 7fb82dbea740 1 -- 172.21.15.27:0/3938887985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed8930a0 msgr2=0x5565ed8c5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.900+0000 7fb82dbea740 1 --2- 172.21.15.27:0/3938887985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed8930a0 0x5565ed8c5800 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb81402a920 tx=0x7fb814000f40 comp rx=0 tx=0).stop 2024-04-25T00:23:19.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.900+0000 7fb82dbea740 1 -- 172.21.15.27:0/3938887985 shutdown_connections 2024-04-25T00:23:19.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.900+0000 7fb82dbea740 1 --2- 172.21.15.27:0/3938887985 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed89f6d0 0x5565ed8c5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.900+0000 7fb82dbea740 1 --2- 172.21.15.27:0/3938887985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed8930a0 0x5565ed8c5800 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.900+0000 7fb82dbea740 1 --2- 172.21.15.27:0/3938887985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed892700 0x5565ed8a6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.900+0000 7fb82dbea740 1 -- 172.21.15.27:0/3938887985 >> 172.21.15.27:0/3938887985 conn(0x5565ed88d5f0 msgr2=0x5565ed89e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.900+0000 7fb82dbea740 1 -- 172.21.15.27:0/3938887985 shutdown_connections 2024-04-25T00:23:19.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.900+0000 7fb82dbea740 1 -- 172.21.15.27:0/3938887985 wait complete. 2024-04-25T00:23:19.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.901+0000 7fb82dbea740 1 -- start start 2024-04-25T00:23:19.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.901+0000 7fb82dbea740 1 Processor -- start 2024-04-25T00:23:19.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.902+0000 7fb82dbea740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 0x5565ed89a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.902+0000 7fb82dbea740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed8930a0 0x5565ed89a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.902+0000 7fb828b9e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 0x5565ed89a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.902+0000 7fb828b9e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 0x5565ed89a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:56094/0 (socket says 172.21.15.27:56094) 2024-04-25T00:23:19.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.902+0000 7fb828b9e640 1 -- 172.21.15.27:0/4238378789 learned_addr learned my addr 172.21.15.27:0/4238378789 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:19.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.902+0000 7fb82dbea740 1 --2- 172.21.15.27:0/4238378789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed89f6d0 0x5565ed897790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.902+0000 7fb82dbea740 1 -- 172.21.15.27:0/4238378789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb8241adc20 con 0x5565ed89f6d0 2024-04-25T00:23:19.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.902+0000 7fb82dbea740 1 -- 172.21.15.27:0/4238378789 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb8241ad520 con 0x5565ed8930a0 2024-04-25T00:23:19.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.902+0000 7fb823fff640 1 --2- 172.21.15.27:0/4238378789 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed8930a0 0x5565ed89a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.903+0000 7fb82dbea740 1 -- 172.21.15.27:0/4238378789 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb8241aa7e0 con 0x5565ed892700 2024-04-25T00:23:19.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.903+0000 7fb82939f640 1 --2- 172.21.15.27:0/4238378789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed89f6d0 0x5565ed897790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.903+0000 7fb82dbea740 10 client.0 _ll_register_callbacks cb 0x7fb81c005f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:19.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.903+0000 7fb823fff640 1 -- 172.21.15.27:0/4238378789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 msgr2=0x5565ed89a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.903+0000 7fb823fff640 1 --2- 172.21.15.27:0/4238378789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 0x5565ed89a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.903+0000 7fb823fff640 1 -- 172.21.15.27:0/4238378789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed89f6d0 msgr2=0x5565ed897790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:19.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.903+0000 7fb823fff640 1 --2- 172.21.15.27:0/4238378789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed89f6d0 0x5565ed897790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.903+0000 7fb823fff640 1 -- 172.21.15.27:0/4238378789 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565ed5d47d0 con 0x5565ed8930a0 2024-04-25T00:23:19.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.904+0000 7fb823fff640 1 --2- 172.21.15.27:0/4238378789 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed8930a0 0x5565ed89a840 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb81402a920 tx=0x7fb814003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7112c317c943692b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:19.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.904+0000 7fb803fff640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T00:23:19.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.904+0000 7fb803fff640 1 -- 172.21.15.27:0/4238378789 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81405d070 con 0x5565ed8930a0 2024-04-25T00:23:19.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.904+0000 7fb803fff640 1 -- 172.21.15.27:0/4238378789 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fb814050040 con 0x5565ed8930a0 2024-04-25T00:23:19.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.904+0000 7fb82dbea740 1 -- 172.21.15.27:0/4238378789 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb814036040 con 0x5565ed8930a0 2024-04-25T00:23:19.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.905+0000 7fb803fff640 1 -- 172.21.15.27:0/4238378789 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fb814031b20 con 0x5565ed8930a0 2024-04-25T00:23:19.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.905+0000 7fb82dbea740 10 client.26108 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:19.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.905+0000 7fb82dbea740 1 -- 172.21.15.27:0/4238378789 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5565ed923940 con 0x5565ed8930a0 2024-04-25T00:23:19.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.906+0000 7fb803fff640 1 -- 172.21.15.27:0/4238378789 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8140308d0 con 0x5565ed8930a0 2024-04-25T00:23:19.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.906+0000 7fb803fff640 1 -- 172.21.15.27:0/4238378789 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7fb814030070 con 0x5565ed8930a0 2024-04-25T00:23:19.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.906+0000 7fb82dbea740 10 client.26108 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:19.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.906+0000 7fb82dbea740 10 client.26108 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:23:19.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.906+0000 7fb82dbea740 20 client.26108 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:19.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.906+0000 7fb82dbea740 1 --2- 172.21.15.27:0/4238378789 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5565ed923b70 0x5565ed925f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:19.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.906+0000 7fb82dbea740 4 client.26108 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:23:19.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.906+0000 7fb82dbea740 1 -- 172.21.15.27:0/4238378789 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fb814036040 con 0x5565ed923b70 2024-04-25T00:23:19.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.906+0000 7fb828b9e640 1 --2- 172.21.15.27:0/4238378789 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5565ed923b70 0x5565ed925f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:19.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.908+0000 7fb803fff640 10 client.26108 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:23:19.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.908+0000 7fb828b9e640 1 --2- 172.21.15.27:0/4238378789 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5565ed923b70 0x5565ed925f60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3e91341663a1973 server_cookie=8d07018459cf9a01 in_seq=0 out_seq=0 2024-04-25T00:23:19.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.909+0000 7fb803fff640 1 -- 172.21.15.27:0/4238378789 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fb814036040 con 0x5565ed923b70 2024-04-25T00:23:19.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.909+0000 7fb803fff640 10 client.26108 handle_command_reply: tid=0 2024-04-25T00:23:19.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.909+0000 7fb82dbea740 1 client.26108 shutdown 2024-04-25T00:23:19.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.909+0000 7fb82dbea740 1 -- 172.21.15.27:0/4238378789 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed8930a0 msgr2=0x5565ed89a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.909+0000 7fb82dbea740 1 --2- 172.21.15.27:0/4238378789 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed8930a0 0x5565ed89a840 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb81402a920 tx=0x7fb814003f90 comp rx=0 tx=0).stop 2024-04-25T00:23:19.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.909+0000 7fb82dbea740 1 -- 172.21.15.27:0/4238378789 shutdown_connections 2024-04-25T00:23:19.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.909+0000 7fb82dbea740 1 --2- 172.21.15.27:0/4238378789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565ed89f6d0 0x5565ed897790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.910+0000 7fb82dbea740 1 --2- 172.21.15.27:0/4238378789 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5565ed923b70 0x5565ed925f60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.910+0000 7fb82dbea740 1 --2- 172.21.15.27:0/4238378789 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565ed8930a0 0x5565ed89a840 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.910+0000 7fb803fff640 0 client.26108 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:23:19.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.910+0000 7fb82dbea740 1 --2- 172.21.15.27:0/4238378789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565ed892700 0x5565ed89a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.910+0000 7fb82dbea740 1 -- 172.21.15.27:0/4238378789 >> 172.21.15.27:0/4238378789 conn(0x5565ed88d5f0 msgr2=0x5565ed88c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.910+0000 7fb82dbea740 1 -- 172.21.15.27:0/4238378789 shutdown_connections 2024-04-25T00:23:19.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.910+0000 7fb82dbea740 1 -- 172.21.15.27:0/4238378789 wait complete. 2024-04-25T00:23:19.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.911+0000 7fb82dbea740 20 client.26108 trim_cache size 0 max 16384 2024-04-25T00:23:19.913 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:23:19.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.912+0000 7fb81b7fe640 1 -- 172.21.15.27:0/2632193704 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb8100a2be0 msgr2=0x7fb8100a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.912+0000 7fb81b7fe640 1 --2- 172.21.15.27:0/2632193704 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb8100a2be0 0x7fb8100a50a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb814004af0 tx=0x7fb814005fd0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.912+0000 7fb81b7fe640 1 -- 172.21.15.27:0/2632193704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8241066a0 msgr2=0x7fb82419b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:19.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.912+0000 7fb81b7fe640 1 --2- 172.21.15.27:0/2632193704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8241066a0 0x7fb82419b7b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb80c00e970 tx=0x7fb80c00ed70 comp rx=0 tx=0).stop 2024-04-25T00:23:19.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.912+0000 7fb81b7fe640 1 -- 172.21.15.27:0/2632193704 shutdown_connections 2024-04-25T00:23:19.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.912+0000 7fb81b7fe640 1 --2- 172.21.15.27:0/2632193704 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb8100a2be0 0x7fb8100a50a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.912+0000 7fb81b7fe640 1 --2- 172.21.15.27:0/2632193704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8241066a0 0x7fb82419b7b0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.912+0000 7fb81b7fe640 1 --2- 172.21.15.27:0/2632193704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb824100fc0 0x7fb824197430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.912+0000 7fb81b7fe640 1 --2- 172.21.15.27:0/2632193704 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb824100600 0x7fb824196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:19.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.912+0000 7fb81b7fe640 1 -- 172.21.15.27:0/2632193704 >> 172.21.15.27:0/2632193704 conn(0x7fb8240776e0 msgr2=0x7fb8240ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:19.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.913+0000 7fb81b7fe640 1 -- 172.21.15.27:0/2632193704 shutdown_connections 2024-04-25T00:23:19.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:19.913+0000 7fb81b7fe640 1 -- 172.21.15.27:0/2632193704 wait complete. 2024-04-25T00:23:19.931 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T00:23:20.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.121+0000 7f8d5bfff640 1 -- 172.21.15.27:0/3243402628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d54006f30 msgr2=0x7f8d54007310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.121+0000 7f8d5bfff640 1 --2- 172.21.15.27:0/3243402628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d54006f30 0x7f8d54007310 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c009900 tx=0x7f8d4c02a4d0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.127+0000 7f8d5bfff640 1 -- 172.21.15.27:0/3243402628 shutdown_connections 2024-04-25T00:23:20.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.127+0000 7f8d5bfff640 1 --2- 172.21.15.27:0/3243402628 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d540a6d50 0x7f8d540a9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.127+0000 7f8d5bfff640 1 --2- 172.21.15.27:0/3243402628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d54007850 0x7f8d540a6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.127+0000 7f8d5bfff640 1 --2- 172.21.15.27:0/3243402628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d54006f30 0x7f8d54007310 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.127+0000 7f8d5bfff640 1 -- 172.21.15.27:0/3243402628 >> 172.21.15.27:0/3243402628 conn(0x7f8d5400cec0 msgr2=0x7f8d5400d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:20.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.129+0000 7f8d5bfff640 1 -- 172.21.15.27:0/3243402628 shutdown_connections 2024-04-25T00:23:20.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.132+0000 7f8d5bfff640 1 -- 172.21.15.27:0/3243402628 wait complete. 2024-04-25T00:23:20.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.133+0000 7f8d5bfff640 1 Processor -- start 2024-04-25T00:23:20.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.133+0000 7f8d5bfff640 1 -- start start 2024-04-25T00:23:20.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.134+0000 7f8d5bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d54006f30 0x7f8d540a65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.134+0000 7f8d5bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d54007850 0x7f8d540a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.134+0000 7f8d5bfff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d540a6d50 0x7f8d540a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.134+0000 7f8d5bfff640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5409aa30 con 0x7f8d54006f30 2024-04-25T00:23:20.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.134+0000 7f8d5bfff640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5409a8b0 con 0x7f8d540a6d50 2024-04-25T00:23:20.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.134+0000 7f8d5b7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d540a6d50 0x7f8d540a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.135+0000 7f8d5b7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d540a6d50 0x7f8d540a0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:60844/0 (socket says 172.21.15.27:60844) 2024-04-25T00:23:20.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.135+0000 7f8d5b7fe640 1 -- 172.21.15.27:0/342916613 learned_addr learned my addr 172.21.15.27:0/342916613 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:20.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.135+0000 7f8d5bfff640 1 -- 172.21.15.27:0/342916613 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5409abb0 con 0x7f8d54007850 2024-04-25T00:23:20.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.135+0000 7f8d5affd640 1 --2- 172.21.15.27:0/342916613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d54006f30 0x7f8d540a65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.136+0000 7f8d5b7fe640 1 -- 172.21.15.27:0/342916613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d54007850 msgr2=0x7f8d540a0650 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:23:20.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.136+0000 7f8d5b7fe640 1 --2- 172.21.15.27:0/342916613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d54007850 0x7f8d540a0650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.136+0000 7f8d5b7fe640 1 -- 172.21.15.27:0/342916613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d54006f30 msgr2=0x7f8d540a65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.136+0000 7f8d5b7fe640 1 --2- 172.21.15.27:0/342916613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d54006f30 0x7f8d540a65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.136+0000 7f8d5b7fe640 1 -- 172.21.15.27:0/342916613 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d540a1450 con 0x7f8d540a6d50 2024-04-25T00:23:20.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.138+0000 7f8d5b7fe640 1 --2- 172.21.15.27:0/342916613 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d540a6d50 0x7f8d540a0b90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400b3e0 tx=0x7f8d4400b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:20.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.139+0000 7f8d53fff640 1 -- 172.21.15.27:0/342916613 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d44015020 con 0x7f8d540a6d50 2024-04-25T00:23:20.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.139+0000 7f8d53fff640 1 -- 172.21.15.27:0/342916613 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d44011d10 con 0x7f8d540a6d50 2024-04-25T00:23:20.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.139+0000 7f8d53fff640 1 -- 172.21.15.27:0/342916613 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4400f2e0 con 0x7f8d540a6d50 2024-04-25T00:23:20.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.139+0000 7f8d5bfff640 1 -- 172.21.15.27:0/342916613 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d54147c00 con 0x7f8d540a6d50 2024-04-25T00:23:20.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.139+0000 7f8d5bfff640 1 -- 172.21.15.27:0/342916613 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d541480d0 con 0x7f8d540a6d50 2024-04-25T00:23:20.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.143+0000 7f8d641ac740 1 Processor -- start 2024-04-25T00:23:20.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.145+0000 7f8d53fff640 1 -- 172.21.15.27:0/342916613 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8d44018020 con 0x7f8d540a6d50 2024-04-25T00:23:20.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.146+0000 7f8d641ac740 1 -- start start 2024-04-25T00:23:20.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.146+0000 7f8d641ac740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc16d710 0x5584cc16daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.146+0000 7f8d641ac740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc16e030 0x5584cc1781c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.146+0000 7f8d641ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc178700 0x5584cc17aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.146+0000 7f8d641ac740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5584cc0d2670 con 0x5584cc16d710 2024-04-25T00:23:20.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.146+0000 7f8d641ac740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5584cc07a970 con 0x5584cc16e030 2024-04-25T00:23:20.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.146+0000 7f8d641ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5584cc0ce630 con 0x5584cc178700 2024-04-25T00:23:20.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.146+0000 7f8d5a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc16d710 0x5584cc16daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.146+0000 7f8d5affd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc16e030 0x5584cc1781c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.147+0000 7f8d5affd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc16e030 0x5584cc1781c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:60848/0 (socket says 172.21.15.27:60848) 2024-04-25T00:23:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.147+0000 7f8d5affd640 1 -- 172.21.15.27:0/2600546517 learned_addr learned my addr 172.21.15.27:0/2600546517 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.147+0000 7f8d5a7fc640 1 --2- 172.21.15.27:0/2600546517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc178700 0x5584cc17aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.147+0000 7f8d5a7fc640 1 -- 172.21.15.27:0/2600546517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc178700 msgr2=0x5584cc17aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.147+0000 7f8d5a7fc640 1 --2- 172.21.15.27:0/2600546517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc178700 0x5584cc17aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.147+0000 7f8d5a7fc640 1 -- 172.21.15.27:0/2600546517 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc16e030 msgr2=0x5584cc1781c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.147+0000 7f8d5a7fc640 1 --2- 172.21.15.27:0/2600546517 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc16e030 0x5584cc1781c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.148+0000 7f8d5a7fc640 1 -- 172.21.15.27:0/2600546517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584cc09e400 con 0x5584cc16d710 2024-04-25T00:23:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.148+0000 7f8d5a7fc640 1 --2- 172.21.15.27:0/2600546517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc16d710 0x5584cc16daf0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c050e00 tx=0x7f8d5c067ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40234ae29986d55c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.148+0000 7f8d53fff640 1 --2- 172.21.15.27:0/342916613 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8d480a2f00 0x7f8d480a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.148+0000 7f8d53fff640 1 -- 172.21.15.27:0/342916613 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8d440cee10 con 0x7f8d540a6d50 2024-04-25T00:23:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.149+0000 7f8d517fa640 1 -- 172.21.15.27:0/2600546517 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c079030 con 0x5584cc16d710 2024-04-25T00:23:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.149+0000 7f8d517fa640 1 -- 172.21.15.27:0/2600546517 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d5c07d040 con 0x5584cc16d710 2024-04-25T00:23:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.149+0000 7f8d517fa640 1 -- 172.21.15.27:0/2600546517 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c079030 con 0x5584cc16d710 2024-04-25T00:23:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.149+0000 7f8d5affd640 1 --2- 172.21.15.27:0/342916613 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8d480a2f00 0x7f8d480a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.150+0000 7f8d641ac740 1 -- 172.21.15.27:0/2600546517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc16d710 msgr2=0x5584cc16daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.150+0000 7f8d641ac740 1 --2- 172.21.15.27:0/2600546517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc16d710 0x5584cc16daf0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c050e00 tx=0x7f8d5c067ab0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.150+0000 7f8d5affd640 1 --2- 172.21.15.27:0/342916613 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8d480a2f00 0x7f8d480a53c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c0096c0 tx=0x7f8d4c02a9d0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.152+0000 7f8d641ac740 1 -- 172.21.15.27:0/2600546517 shutdown_connections 2024-04-25T00:23:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.152+0000 7f8d641ac740 1 --2- 172.21.15.27:0/2600546517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc178700 0x5584cc17aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.153+0000 7f8d641ac740 1 --2- 172.21.15.27:0/2600546517 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc16e030 0x5584cc1781c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.153+0000 7f8d641ac740 1 --2- 172.21.15.27:0/2600546517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc16d710 0x5584cc16daf0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.153+0000 7f8d641ac740 1 -- 172.21.15.27:0/2600546517 >> 172.21.15.27:0/2600546517 conn(0x5584cc169a30 msgr2=0x5584cc169e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.153+0000 7f8d641ac740 1 -- 172.21.15.27:0/2600546517 shutdown_connections 2024-04-25T00:23:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.154+0000 7f8d641ac740 1 -- 172.21.15.27:0/2600546517 wait complete. 2024-04-25T00:23:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.154+0000 7f8d641ac740 1 -- start start 2024-04-25T00:23:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.155+0000 7f8d641ac740 1 Processor -- start 2024-04-25T00:23:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.155+0000 7f8d641ac740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc16d710 0x5584cc177fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.155+0000 7f8d641ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc16e030 0x5584cc1752c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.155+0000 7f8d641ac740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc178700 0x5584cc175800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.156+0000 7f8d5a7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc16e030 0x5584cc1752c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.156+0000 7f8d5a7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc16e030 0x5584cc1752c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:56134/0 (socket says 172.21.15.27:56134) 2024-04-25T00:23:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.156+0000 7f8d641ac740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5584cc17b4f0 con 0x5584cc178700 2024-04-25T00:23:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.156+0000 7f8d641ac740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5584cc08e050 con 0x5584cc16d710 2024-04-25T00:23:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.156+0000 7f8d641ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5584cc17b670 con 0x5584cc16e030 2024-04-25T00:23:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.156+0000 7f8d641ac740 10 client.0 _ll_register_callbacks cb 0x5584cbde5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.156+0000 7f8d5a7fc640 1 -- 172.21.15.27:0/3423390638 learned_addr learned my addr 172.21.15.27:0/3423390638 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:20.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.156+0000 7f8d5affd640 1 --2- 172.21.15.27:0/3423390638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc16d710 0x5584cc177fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.156+0000 7f8d5b7fe640 1 --2- 172.21.15.27:0/3423390638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc178700 0x5584cc175800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.159+0000 7f8d5affd640 1 -- 172.21.15.27:0/3423390638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc16e030 msgr2=0x5584cc1752c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:20.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.159+0000 7f8d5affd640 1 --2- 172.21.15.27:0/3423390638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc16e030 0x5584cc1752c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.159+0000 7f8d5affd640 1 -- 172.21.15.27:0/3423390638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc178700 msgr2=0x5584cc175800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:20.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.159+0000 7f8d5affd640 1 --2- 172.21.15.27:0/3423390638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc178700 0x5584cc175800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.159+0000 7f8d5affd640 1 -- 172.21.15.27:0/3423390638 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584cc05fdc0 con 0x5584cc16d710 2024-04-25T00:23:20.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.161+0000 7f8d5affd640 1 --2- 172.21.15.27:0/3423390638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc16d710 0x5584cc177fd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c000c00 tx=0x7f8d4c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95eae9b20b8008c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:20.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.164+0000 7f8d427fc640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T00:23:20.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.164+0000 7f8d427fc640 1 -- 172.21.15.27:0/3423390638 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c04e070 con 0x5584cc16d710 2024-04-25T00:23:20.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.164+0000 7f8d641ac740 1 -- 172.21.15.27:0/3423390638 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d5c07d040 con 0x5584cc16d710 2024-04-25T00:23:20.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.164+0000 7f8d427fc640 1 -- 172.21.15.27:0/3423390638 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8d4c003a00 con 0x5584cc16d710 2024-04-25T00:23:20.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.168+0000 7f8d427fc640 1 -- 172.21.15.27:0/3423390638 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f8d4c058050 con 0x5584cc16d710 2024-04-25T00:23:20.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.168+0000 7f8d641ac740 10 client.26117 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:20.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.168+0000 7f8d641ac740 1 -- 172.21.15.27:0/3423390638 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5584cbde63f0 con 0x5584cc16d710 2024-04-25T00:23:20.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.169+0000 7f8d427fc640 1 -- 172.21.15.27:0/3423390638 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d5c07d040 con 0x5584cc16d710 2024-04-25T00:23:20.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.169+0000 7f8d427fc640 1 -- 172.21.15.27:0/3423390638 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f8d4c04ac00 con 0x5584cc16d710 2024-04-25T00:23:20.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.169+0000 7f8d641ac740 10 client.26117 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:20.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.169+0000 7f8d641ac740 10 client.26117 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:23:20.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.169+0000 7f8d641ac740 20 client.26117 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:20.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.169+0000 7f8d641ac740 1 --2- 172.21.15.27:0/3423390638 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5584cc188ad0 0x5584cc18af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.169+0000 7f8d641ac740 4 client.26117 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:23:20.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.169+0000 7f8d641ac740 1 -- 172.21.15.27:0/3423390638 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584cc0dfbb0 con 0x5584cc188ad0 2024-04-25T00:23:20.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.173+0000 7f8d5a7fc640 1 --2- 172.21.15.27:0/3423390638 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5584cc188ad0 0x5584cc18af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.177+0000 7f8d5a7fc640 1 --2- 172.21.15.27:0/3423390638 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5584cc188ad0 0x5584cc18af20 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d9751addc89f5e4 server_cookie=69b02c76b3af3268 in_seq=0 out_seq=0 2024-04-25T00:23:20.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.177+0000 7f8d427fc640 10 client.26117 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:23:20.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.178+0000 7f8d427fc640 1 -- 172.21.15.27:0/3423390638 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5584cc0dfbb0 con 0x5584cc188ad0 2024-04-25T00:23:20.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.178+0000 7f8d427fc640 10 client.26117 handle_command_reply: tid=0 2024-04-25T00:23:20.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.178+0000 7f8d641ac740 1 client.26117 shutdown 2024-04-25T00:23:20.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.180+0000 7f8d641ac740 1 -- 172.21.15.27:0/3423390638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc16d710 msgr2=0x5584cc177fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.180+0000 7f8d641ac740 1 --2- 172.21.15.27:0/3423390638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc16d710 0x5584cc177fd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c000c00 tx=0x7f8d4c009260 comp rx=0 tx=0).stop 2024-04-25T00:23:20.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.180+0000 7f8d641ac740 1 -- 172.21.15.27:0/3423390638 shutdown_connections 2024-04-25T00:23:20.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.180+0000 7f8d641ac740 1 --2- 172.21.15.27:0/3423390638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc178700 0x5584cc175800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.180+0000 7f8d641ac740 1 --2- 172.21.15.27:0/3423390638 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5584cc188ad0 0x5584cc18af20 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.180+0000 7f8d641ac740 1 --2- 172.21.15.27:0/3423390638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc16e030 0x5584cc1752c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.180+0000 7f8d641ac740 1 --2- 172.21.15.27:0/3423390638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc16d710 0x5584cc177fd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.180+0000 7f8d641ac740 1 -- 172.21.15.27:0/3423390638 >> 172.21.15.27:0/3423390638 conn(0x5584cc169a30 msgr2=0x5584cc16f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:20.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.180+0000 7f8d427fc640 0 client.26117 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:23:20.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.180+0000 7f8d641ac740 1 -- 172.21.15.27:0/3423390638 shutdown_connections 2024-04-25T00:23:20.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.180+0000 7f8d641ac740 1 -- 172.21.15.27:0/3423390638 wait complete. 2024-04-25T00:23:20.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.181+0000 7f8d641ac740 20 client.26117 trim_cache size 0 max 16384 2024-04-25T00:23:20.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.208+0000 7f8d641ac740 1 Processor -- start 2024-04-25T00:23:20.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.208+0000 7f8d641ac740 1 -- start start 2024-04-25T00:23:20.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.208+0000 7f8d641ac740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc169a30 0x5584cc1843e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.209+0000 7f8d641ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc16d710 0x5584cc1689c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.209+0000 7f8d641ac740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc16e030 0x5584cc190830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.209+0000 7f8d5affd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc169a30 0x5584cc1843e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.209+0000 7f8d5affd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc169a30 0x5584cc1843e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:60870/0 (socket says 172.21.15.27:60870) 2024-04-25T00:23:20.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.209+0000 7f8d5affd640 1 -- 172.21.15.27:0/3680339907 learned_addr learned my addr 172.21.15.27:0/3680339907 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:20.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.209+0000 7f8d5a7fc640 1 --2- 172.21.15.27:0/3680339907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc16d710 0x5584cc1689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.209+0000 7f8d641ac740 1 -- 172.21.15.27:0/3680339907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8d54152230 con 0x5584cc16e030 2024-04-25T00:23:20.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.209+0000 7f8d641ac740 1 -- 172.21.15.27:0/3680339907 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8d54153dd0 con 0x5584cc169a30 2024-04-25T00:23:20.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.209+0000 7f8d641ac740 1 -- 172.21.15.27:0/3680339907 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8d54152900 con 0x5584cc16d710 2024-04-25T00:23:20.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.210+0000 7f8d5a7fc640 1 -- 172.21.15.27:0/3680339907 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc169a30 msgr2=0x5584cc1843e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:20.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.210+0000 7f8d5a7fc640 1 --2- 172.21.15.27:0/3680339907 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc169a30 0x5584cc1843e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.210+0000 7f8d5a7fc640 1 -- 172.21.15.27:0/3680339907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc16e030 msgr2=0x5584cc190830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:20.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.210+0000 7f8d5a7fc640 1 --2- 172.21.15.27:0/3680339907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc16e030 0x5584cc190830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.210+0000 7f8d5a7fc640 1 -- 172.21.15.27:0/3680339907 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584cc16de60 con 0x5584cc16d710 2024-04-25T00:23:20.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.210+0000 7f8d5a7fc640 1 --2- 172.21.15.27:0/3680339907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc16d710 0x5584cc1689c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0671c0 tx=0x7f8d5c0835e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cd73d813ab2d43c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:20.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.210+0000 7f8d50ff9640 1 -- 172.21.15.27:0/3680339907 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c099070 con 0x5584cc16d710 2024-04-25T00:23:20.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.210+0000 7f8d50ff9640 1 -- 172.21.15.27:0/3680339907 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d5c07a040 con 0x5584cc16d710 2024-04-25T00:23:20.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.211+0000 7f8d641ac740 1 -- 172.21.15.27:0/3680339907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc16d710 msgr2=0x5584cc1689c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.211+0000 7f8d641ac740 1 --2- 172.21.15.27:0/3680339907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc16d710 0x5584cc1689c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0671c0 tx=0x7f8d5c0835e0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.212+0000 7f8d641ac740 1 -- 172.21.15.27:0/3680339907 shutdown_connections 2024-04-25T00:23:20.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.212+0000 7f8d641ac740 1 --2- 172.21.15.27:0/3680339907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc16e030 0x5584cc190830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.212+0000 7f8d641ac740 1 --2- 172.21.15.27:0/3680339907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc16d710 0x5584cc1689c0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0671c0 tx=0x7f8d5c0835e0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.212+0000 7f8d641ac740 1 --2- 172.21.15.27:0/3680339907 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc169a30 0x5584cc1843e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.212+0000 7f8d641ac740 1 -- 172.21.15.27:0/3680339907 >> 172.21.15.27:0/3680339907 conn(0x5584cc175960 msgr2=0x5584cc1682b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:20.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.214+0000 7f8d641ac740 1 -- 172.21.15.27:0/3680339907 shutdown_connections 2024-04-25T00:23:20.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.214+0000 7f8d641ac740 1 -- 172.21.15.27:0/3680339907 wait complete. 2024-04-25T00:23:20.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.214+0000 7f8d641ac740 1 -- start start 2024-04-25T00:23:20.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.214+0000 7f8d641ac740 1 Processor -- start 2024-04-25T00:23:20.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.215+0000 7f8d641ac740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc169a30 0x5584cc180590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.215+0000 7f8d641ac740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc16e030 0x5584cc180ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.215+0000 7f8d641ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc178700 0x5584cc17d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.215+0000 7f8d641ac740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8d54153700 con 0x5584cc16e030 2024-04-25T00:23:20.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.215+0000 7f8d641ac740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8d54153000 con 0x5584cc169a30 2024-04-25T00:23:20.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.215+0000 7f8d641ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8d541501c0 con 0x5584cc178700 2024-04-25T00:23:20.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.215+0000 7f8d641ac740 10 client.0 _ll_register_callbacks cb 0x7f8d4c009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:20.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.215+0000 7f8d5a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc16e030 0x5584cc180ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.215+0000 7f8d5a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc16e030 0x5584cc180ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49358/0 (socket says 172.21.15.27:49358) 2024-04-25T00:23:20.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.215+0000 7f8d5a7fc640 1 -- 172.21.15.27:0/2514984773 learned_addr learned my addr 172.21.15.27:0/2514984773 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:20.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.216+0000 7f8d5a7fc640 1 -- 172.21.15.27:0/2514984773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc178700 msgr2=0x5584cc17d9a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:23:20.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.216+0000 7f8d5a7fc640 1 --2- 172.21.15.27:0/2514984773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc178700 0x5584cc17d9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.216+0000 7f8d5affd640 1 --2- 172.21.15.27:0/2514984773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc169a30 0x5584cc180590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.216+0000 7f8d5a7fc640 1 -- 172.21.15.27:0/2514984773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc169a30 msgr2=0x5584cc180590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:20.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.216+0000 7f8d5a7fc640 1 --2- 172.21.15.27:0/2514984773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc169a30 0x5584cc180590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.216+0000 7f8d5a7fc640 1 -- 172.21.15.27:0/2514984773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584cbeb07d0 con 0x5584cc16e030 2024-04-25T00:23:20.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.216+0000 7f8d5a7fc640 1 --2- 172.21.15.27:0/2514984773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc16e030 0x5584cc180ad0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c075010 tx=0x7f8d5c067370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef83c08da7c53cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:20.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.217+0000 7f8d427fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:20.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.217+0000 7f8d427fc640 1 -- 172.21.15.27:0/2514984773 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c099040 con 0x5584cc16e030 2024-04-25T00:23:20.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.217+0000 7f8d641ac740 1 -- 172.21.15.27:0/2514984773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d5c07a040 con 0x5584cc16e030 2024-04-25T00:23:20.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.217+0000 7f8d427fc640 1 -- 172.21.15.27:0/2514984773 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8d5c067940 con 0x5584cc16e030 2024-04-25T00:23:20.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.218+0000 7f8d427fc640 1 -- 172.21.15.27:0/2514984773 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f8d5c0ac050 con 0x5584cc16e030 2024-04-25T00:23:20.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.218+0000 7f8d641ac740 10 client.16614 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:20.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.218+0000 7f8d641ac740 1 -- 172.21.15.27:0/2514984773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5584cc1ee8d0 con 0x5584cc16e030 2024-04-25T00:23:20.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.218+0000 7f8d427fc640 1 -- 172.21.15.27:0/2514984773 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d5c07a040 con 0x5584cc16e030 2024-04-25T00:23:20.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.218+0000 7f8d427fc640 1 -- 172.21.15.27:0/2514984773 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f8d5c083cb0 con 0x5584cc16e030 2024-04-25T00:23:20.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.218+0000 7f8d641ac740 10 client.16614 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:20.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.218+0000 7f8d641ac740 10 client.16614 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:23:20.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.218+0000 7f8d641ac740 20 client.16614 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:20.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.218+0000 7f8d641ac740 1 --2- 172.21.15.27:0/2514984773 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5584cc1eece0 0x5584cc1f1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.218+0000 7f8d641ac740 4 client.16614 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:23:20.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.218+0000 7f8d641ac740 1 -- 172.21.15.27:0/2514984773 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5584cc0d9f00 con 0x5584cc1eece0 2024-04-25T00:23:20.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.219+0000 7f8d5affd640 1 --2- 172.21.15.27:0/2514984773 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5584cc1eece0 0x5584cc1f1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.220+0000 7f8d5affd640 1 --2- 172.21.15.27:0/2514984773 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5584cc1eece0 0x5584cc1f1100 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f9f8f2a4a638a78 server_cookie=bab8b9038015123b in_seq=0 out_seq=0 2024-04-25T00:23:20.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.220+0000 7f8d427fc640 10 client.16614 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:23:20.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.220+0000 7f8d427fc640 1 -- 172.21.15.27:0/2514984773 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5584cc0d9f00 con 0x5584cc1eece0 2024-04-25T00:23:20.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.220+0000 7f8d427fc640 10 client.16614 handle_command_reply: tid=0 2024-04-25T00:23:20.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.220+0000 7f8d641ac740 1 client.16614 shutdown 2024-04-25T00:23:20.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.221+0000 7f8d641ac740 1 -- 172.21.15.27:0/2514984773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc16e030 msgr2=0x5584cc180ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.221+0000 7f8d641ac740 1 --2- 172.21.15.27:0/2514984773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc16e030 0x5584cc180ad0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c075010 tx=0x7f8d5c067370 comp rx=0 tx=0).stop 2024-04-25T00:23:20.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.221+0000 7f8d641ac740 1 -- 172.21.15.27:0/2514984773 shutdown_connections 2024-04-25T00:23:20.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.221+0000 7f8d641ac740 1 --2- 172.21.15.27:0/2514984773 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5584cc1eece0 0x5584cc1f1100 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.221+0000 7f8d641ac740 1 --2- 172.21.15.27:0/2514984773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584cc178700 0x5584cc17d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.221+0000 7f8d641ac740 1 --2- 172.21.15.27:0/2514984773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584cc16e030 0x5584cc180ad0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.221+0000 7f8d641ac740 1 --2- 172.21.15.27:0/2514984773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584cc169a30 0x5584cc180590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.221+0000 7f8d641ac740 1 -- 172.21.15.27:0/2514984773 >> 172.21.15.27:0/2514984773 conn(0x5584cc175960 msgr2=0x5584cc168b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:20.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.221+0000 7f8d427fc640 0 client.16614 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:23:20.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.221+0000 7f8d641ac740 1 -- 172.21.15.27:0/2514984773 shutdown_connections 2024-04-25T00:23:20.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.221+0000 7f8d641ac740 1 -- 172.21.15.27:0/2514984773 wait complete. 2024-04-25T00:23:20.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.222+0000 7f8d641ac740 20 client.16614 trim_cache size 0 max 16384 2024-04-25T00:23:20.231 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:23:20.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.231+0000 7f8d5bfff640 1 -- 172.21.15.27:0/342916613 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8d480a2f00 msgr2=0x7f8d480a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.231+0000 7f8d5bfff640 1 --2- 172.21.15.27:0/342916613 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8d480a2f00 0x7f8d480a53c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c0096c0 tx=0x7f8d4c02a9d0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.231+0000 7f8d5bfff640 1 -- 172.21.15.27:0/342916613 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d540a6d50 msgr2=0x7f8d540a0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.231+0000 7f8d5bfff640 1 --2- 172.21.15.27:0/342916613 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d540a6d50 0x7f8d540a0b90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400b3e0 tx=0x7f8d4400b8a0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.231+0000 7f8d5bfff640 1 -- 172.21.15.27:0/342916613 shutdown_connections 2024-04-25T00:23:20.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.231+0000 7f8d5bfff640 1 --2- 172.21.15.27:0/342916613 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8d480a2f00 0x7f8d480a53c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.231+0000 7f8d5bfff640 1 --2- 172.21.15.27:0/342916613 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d540a6d50 0x7f8d540a0b90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.231+0000 7f8d5bfff640 1 --2- 172.21.15.27:0/342916613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d54007850 0x7f8d540a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.231+0000 7f8d5bfff640 1 --2- 172.21.15.27:0/342916613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d54006f30 0x7f8d540a65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.231+0000 7f8d5bfff640 1 -- 172.21.15.27:0/342916613 >> 172.21.15.27:0/342916613 conn(0x7f8d5400cec0 msgr2=0x7f8d54098360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:20.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.234+0000 7f8d5bfff640 1 -- 172.21.15.27:0/342916613 shutdown_connections 2024-04-25T00:23:20.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.234+0000 7f8d5bfff640 1 -- 172.21.15.27:0/342916613 wait complete. 2024-04-25T00:23:20.253 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T00:23:20.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.452+0000 7fb602109640 1 -- 172.21.15.27:0/104721972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc069dc0 msgr2=0x7fb5fc06a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.452+0000 7fb602109640 1 --2- 172.21.15.27:0/104721972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc069dc0 0x7fb5fc06a240 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009930 tx=0x7fb5f002a420 comp rx=0 tx=0).stop 2024-04-25T00:23:20.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.452+0000 7fb602109640 1 -- 172.21.15.27:0/104721972 shutdown_connections 2024-04-25T00:23:20.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.452+0000 7fb602109640 1 --2- 172.21.15.27:0/104721972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5fc06a780 0x7fb5fc10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.452+0000 7fb602109640 1 --2- 172.21.15.27:0/104721972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc069dc0 0x7fb5fc06a240 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.452+0000 7fb602109640 1 --2- 172.21.15.27:0/104721972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5fc107730 0x7fb5fc107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.452+0000 7fb602109640 1 -- 172.21.15.27:0/104721972 >> 172.21.15.27:0/104721972 conn(0x7fb5fc0776a0 msgr2=0x7fb5fc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:20.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.452+0000 7fb602109640 1 -- 172.21.15.27:0/104721972 shutdown_connections 2024-04-25T00:23:20.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.452+0000 7fb602109640 1 -- 172.21.15.27:0/104721972 wait complete. 2024-04-25T00:23:20.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.453+0000 7fb602109640 1 Processor -- start 2024-04-25T00:23:20.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.453+0000 7fb602109640 1 -- start start 2024-04-25T00:23:20.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.453+0000 7fb602109640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5fc069dc0 0x7fb5fc198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.453+0000 7fb602109640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5fc06a780 0x7fb5fc198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.453+0000 7fb602109640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc107730 0x7fb5fc19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.453+0000 7fb602109640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb5fc111b90 con 0x7fb5fc107730 2024-04-25T00:23:20.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.453+0000 7fb602109640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb5fc111a10 con 0x7fb5fc06a780 2024-04-25T00:23:20.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.453+0000 7fb602109640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb5fc111d10 con 0x7fb5fc069dc0 2024-04-25T00:23:20.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.454+0000 7fb5fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc107730 0x7fb5fc19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.454+0000 7fb5fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc107730 0x7fb5fc19c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49372/0 (socket says 172.21.15.27:49372) 2024-04-25T00:23:20.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.454+0000 7fb5fbfff640 1 -- 172.21.15.27:0/2987412099 learned_addr learned my addr 172.21.15.27:0/2987412099 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:20.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.454+0000 7fb5fb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5fc069dc0 0x7fb5fc198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.454+0000 7fb5faffd640 1 --2- 172.21.15.27:0/2987412099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5fc06a780 0x7fb5fc198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.454+0000 7fb5fbfff640 1 -- 172.21.15.27:0/2987412099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5fc069dc0 msgr2=0x7fb5fc198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.454+0000 7fb5fbfff640 1 --2- 172.21.15.27:0/2987412099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5fc069dc0 0x7fb5fc198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.454+0000 7fb5fbfff640 1 -- 172.21.15.27:0/2987412099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5fc06a780 msgr2=0x7fb5fc198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.454+0000 7fb5fbfff640 1 --2- 172.21.15.27:0/2987412099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5fc06a780 0x7fb5fc198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.454+0000 7fb5fbfff640 1 -- 172.21.15.27:0/2987412099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5fc19cfd0 con 0x7fb5fc107730 2024-04-25T00:23:20.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.454+0000 7fb5fb7fe640 1 --2- 172.21.15.27:0/2987412099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5fc069dc0 0x7fb5fc198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:20.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.454+0000 7fb5fbfff640 1 --2- 172.21.15.27:0/2987412099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc107730 0x7fb5fc19c8a0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb5f400eb10 tx=0x7fb5f400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:20.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.454+0000 7fb5f8ff9640 1 -- 172.21.15.27:0/2987412099 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4004b20 con 0x7fb5fc107730 2024-04-25T00:23:20.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.454+0000 7fb5f8ff9640 1 -- 172.21.15.27:0/2987412099 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5f4004cc0 con 0x7fb5fc107730 2024-04-25T00:23:20.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.454+0000 7fb602109640 1 -- 172.21.15.27:0/2987412099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5fc19d2c0 con 0x7fb5fc107730 2024-04-25T00:23:20.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.454+0000 7fb5f8ff9640 1 -- 172.21.15.27:0/2987412099 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4010760 con 0x7fb5fc107730 2024-04-25T00:23:20.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.454+0000 7fb602109640 1 -- 172.21.15.27:0/2987412099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5fc1a4b50 con 0x7fb5fc107730 2024-04-25T00:23:20.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.456+0000 7fb5f8ff9640 1 -- 172.21.15.27:0/2987412099 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb5f4005020 con 0x7fb5fc107730 2024-04-25T00:23:20.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.457+0000 7fb604e14740 1 Processor -- start 2024-04-25T00:23:20.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.457+0000 7fb604e14740 1 -- start start 2024-04-25T00:23:20.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.457+0000 7fb604e14740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb438292b0 0x55eb43829690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.457+0000 7fb604e14740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43829c60 0x55eb43838440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.457+0000 7fb604e14740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43838980 0x55eb4383ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.457+0000 7fb604e14740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55eb4378e670 con 0x55eb43838980 2024-04-25T00:23:20.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.457+0000 7fb604e14740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55eb43736970 con 0x55eb43829c60 2024-04-25T00:23:20.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.457+0000 7fb5f8ff9640 1 --2- 172.21.15.27:0/2987412099 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb5e40a2d80 0x7fb5e40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.457+0000 7fb604e14740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55eb4378a630 con 0x55eb438292b0 2024-04-25T00:23:20.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.457+0000 7fb5faffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43838980 0x55eb4383ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.457+0000 7fb5fb7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43829c60 0x55eb43838440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.457+0000 7fb5faffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43838980 0x55eb4383ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49374/0 (socket says 172.21.15.27:49374) 2024-04-25T00:23:20.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.457+0000 7fb5faffd640 1 -- 172.21.15.27:0/1022223631 learned_addr learned my addr 172.21.15.27:0/1022223631 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:20.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.457+0000 7fb5fbfff640 1 --2- 172.21.15.27:0/2987412099 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb5e40a2d80 0x7fb5e40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.457+0000 7fb5faffd640 1 --2- 172.21.15.27:0/1022223631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb438292b0 0x55eb43829690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.457+0000 7fb5f8ff9640 1 -- 172.21.15.27:0/2987412099 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fb5f40cfde0 con 0x7fb5fc107730 2024-04-25T00:23:20.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.458+0000 7fb5fb7fe640 1 -- 172.21.15.27:0/1022223631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb438292b0 msgr2=0x55eb43829690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:20.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.458+0000 7fb5fb7fe640 1 --2- 172.21.15.27:0/1022223631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb438292b0 0x55eb43829690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.458+0000 7fb5fb7fe640 1 -- 172.21.15.27:0/1022223631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43838980 msgr2=0x55eb4383ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:20.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.458+0000 7fb5fb7fe640 1 --2- 172.21.15.27:0/1022223631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43838980 0x55eb4383ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.458+0000 7fb5fb7fe640 1 -- 172.21.15.27:0/1022223631 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb4375a400 con 0x55eb43829c60 2024-04-25T00:23:20.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.458+0000 7fb5fbfff640 1 --2- 172.21.15.27:0/2987412099 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb5e40a2d80 0x7fb5e40a5240 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4098960 tx=0x7fb5f400efd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:20.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.458+0000 7fb5fb7fe640 1 --2- 172.21.15.27:0/1022223631 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43829c60 0x55eb43838440 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec00bf90 tx=0x7fb5ec010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=770c5a683bd75e11 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:20.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.458+0000 7fb5e1ffb640 1 -- 172.21.15.27:0/1022223631 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec011700 con 0x55eb43829c60 2024-04-25T00:23:20.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.458+0000 7fb5e1ffb640 1 -- 172.21.15.27:0/1022223631 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5ec011d80 con 0x55eb43829c60 2024-04-25T00:23:20.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.458+0000 7fb5e1ffb640 1 -- 172.21.15.27:0/1022223631 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec01a380 con 0x55eb43829c60 2024-04-25T00:23:20.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.459+0000 7fb604e14740 1 -- 172.21.15.27:0/1022223631 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43829c60 msgr2=0x55eb43838440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.459+0000 7fb604e14740 1 --2- 172.21.15.27:0/1022223631 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43829c60 0x55eb43838440 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec00bf90 tx=0x7fb5ec010860 comp rx=0 tx=0).stop 2024-04-25T00:23:20.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.459+0000 7fb604e14740 1 -- 172.21.15.27:0/1022223631 shutdown_connections 2024-04-25T00:23:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.459+0000 7fb604e14740 1 --2- 172.21.15.27:0/1022223631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43838980 0x55eb4383ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.459+0000 7fb604e14740 1 --2- 172.21.15.27:0/1022223631 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43829c60 0x55eb43838440 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.459+0000 7fb604e14740 1 --2- 172.21.15.27:0/1022223631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb438292b0 0x55eb43829690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.459+0000 7fb604e14740 1 -- 172.21.15.27:0/1022223631 >> 172.21.15.27:0/1022223631 conn(0x55eb438258a0 msgr2=0x55eb43825c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.459+0000 7fb604e14740 1 -- 172.21.15.27:0/1022223631 shutdown_connections 2024-04-25T00:23:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.459+0000 7fb604e14740 1 -- 172.21.15.27:0/1022223631 wait complete. 2024-04-25T00:23:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.459+0000 7fb604e14740 1 -- start start 2024-04-25T00:23:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.459+0000 7fb604e14740 1 Processor -- start 2024-04-25T00:23:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb604e14740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb438292b0 0x55eb43838330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb604e14740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43829c60 0x55eb43835620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb604e14740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43838980 0x55eb43835bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb604e14740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55eb4383b740 con 0x55eb43838980 2024-04-25T00:23:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb604e14740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55eb4374a050 con 0x55eb43829c60 2024-04-25T00:23:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb604e14740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55eb4383b8c0 con 0x55eb438292b0 2024-04-25T00:23:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb604e14740 10 client.0 _ll_register_callbacks cb 0x55eb434a1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb5faffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43838980 0x55eb43835bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb5faffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43838980 0x55eb43835bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49384/0 (socket says 172.21.15.27:49384) 2024-04-25T00:23:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb5faffd640 1 -- 172.21.15.27:0/3271690971 learned_addr learned my addr 172.21.15.27:0/3271690971 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb5fb7fe640 1 --2- 172.21.15.27:0/3271690971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43829c60 0x55eb43835620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb5faffd640 1 --2- 172.21.15.27:0/3271690971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb438292b0 0x55eb43838330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb5fb7fe640 1 -- 172.21.15.27:0/3271690971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb438292b0 msgr2=0x55eb43838330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb5fb7fe640 1 --2- 172.21.15.27:0/3271690971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb438292b0 0x55eb43838330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb5fb7fe640 1 -- 172.21.15.27:0/3271690971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43838980 msgr2=0x55eb43835bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb5fb7fe640 1 --2- 172.21.15.27:0/3271690971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43838980 0x55eb43835bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb5fb7fe640 1 -- 172.21.15.27:0/3271690971 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb4371bdc0 con 0x55eb43829c60 2024-04-25T00:23:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb5faffd640 1 --2- 172.21.15.27:0/3271690971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43838980 0x55eb43835bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb5fb7fe640 1 --2- 172.21.15.27:0/3271690971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43829c60 0x55eb43835620 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec011010 tx=0x7fb5ec01dea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbf5043455fd7660 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb5daffd640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T00:23:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.460+0000 7fb5daffd640 1 -- 172.21.15.27:0/3271690971 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec02b060 con 0x55eb43829c60 2024-04-25T00:23:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.461+0000 7fb604e14740 1 -- 172.21.15.27:0/3271690971 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5ec011d80 con 0x55eb43829c60 2024-04-25T00:23:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.461+0000 7fb5daffd640 1 -- 172.21.15.27:0/3271690971 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fb5ec005040 con 0x55eb43829c60 2024-04-25T00:23:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.461+0000 7fb5daffd640 1 -- 172.21.15.27:0/3271690971 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fb5ec032050 con 0x55eb43829c60 2024-04-25T00:23:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.461+0000 7fb604e14740 10 client.26132 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.461+0000 7fb604e14740 1 -- 172.21.15.27:0/3271690971 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55eb434a23f0 con 0x55eb43829c60 2024-04-25T00:23:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.461+0000 7fb5daffd640 1 -- 172.21.15.27:0/3271690971 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5ec011d80 con 0x55eb43829c60 2024-04-25T00:23:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.461+0000 7fb5daffd640 1 -- 172.21.15.27:0/3271690971 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7fb5ec025070 con 0x55eb43829c60 2024-04-25T00:23:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.461+0000 7fb604e14740 10 client.26132 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.461+0000 7fb604e14740 10 client.26132 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:23:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.461+0000 7fb604e14740 20 client.26132 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.461+0000 7fb604e14740 1 --2- 172.21.15.27:0/3271690971 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55eb438488e0 0x55eb4384acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.461+0000 7fb604e14740 4 client.26132 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:23:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.461+0000 7fb604e14740 1 -- 172.21.15.27:0/3271690971 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb4379bbb0 con 0x55eb438488e0 2024-04-25T00:23:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.462+0000 7fb5faffd640 1 --2- 172.21.15.27:0/3271690971 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55eb438488e0 0x55eb4384acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.462+0000 7fb5faffd640 1 --2- 172.21.15.27:0/3271690971 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55eb438488e0 0x55eb4384acd0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73e02c57b57f988b server_cookie=170a7d80874161ee in_seq=0 out_seq=0 2024-04-25T00:23:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.462+0000 7fb5daffd640 10 client.26132 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.463+0000 7fb5daffd640 1 -- 172.21.15.27:0/3271690971 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eb4379bbb0 con 0x55eb438488e0 2024-04-25T00:23:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.463+0000 7fb5daffd640 10 client.26132 handle_command_reply: tid=0 2024-04-25T00:23:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.463+0000 7fb604e14740 1 client.26132 shutdown 2024-04-25T00:23:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.463+0000 7fb604e14740 1 -- 172.21.15.27:0/3271690971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43829c60 msgr2=0x55eb43835620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.463+0000 7fb604e14740 1 --2- 172.21.15.27:0/3271690971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43829c60 0x55eb43835620 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec011010 tx=0x7fb5ec01dea0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.463+0000 7fb604e14740 1 -- 172.21.15.27:0/3271690971 shutdown_connections 2024-04-25T00:23:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.463+0000 7fb604e14740 1 --2- 172.21.15.27:0/3271690971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43838980 0x55eb43835bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.463+0000 7fb604e14740 1 --2- 172.21.15.27:0/3271690971 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55eb438488e0 0x55eb4384acd0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.463+0000 7fb604e14740 1 --2- 172.21.15.27:0/3271690971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43829c60 0x55eb43835620 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.463+0000 7fb604e14740 1 --2- 172.21.15.27:0/3271690971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb438292b0 0x55eb43838330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.463+0000 7fb5daffd640 0 client.26132 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:20.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.463+0000 7fb604e14740 1 -- 172.21.15.27:0/3271690971 >> 172.21.15.27:0/3271690971 conn(0x55eb438258a0 msgr2=0x55eb4382b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:20.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.463+0000 7fb604e14740 1 -- 172.21.15.27:0/3271690971 shutdown_connections 2024-04-25T00:23:20.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.464+0000 7fb604e14740 1 -- 172.21.15.27:0/3271690971 wait complete. 2024-04-25T00:23:20.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.465+0000 7fb604e14740 20 client.26132 trim_cache size 0 max 16384 2024-04-25T00:23:20.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.497+0000 7fb604e14740 1 Processor -- start 2024-04-25T00:23:20.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.497+0000 7fb604e14740 1 -- start start 2024-04-25T00:23:20.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.498+0000 7fb604e14740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb438292b0 0x55eb43844670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.498+0000 7fb604e14740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb43829c60 0x55eb43825c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.498+0000 7fb604e14740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43838980 0x55eb43837e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.498+0000 7fb604e14740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb5fc1acab0 con 0x55eb438292b0 2024-04-25T00:23:20.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.498+0000 7fb604e14740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb5fc1ae650 con 0x55eb43838980 2024-04-25T00:23:20.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.498+0000 7fb604e14740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb5fc1ad180 con 0x55eb43829c60 2024-04-25T00:23:20.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.498+0000 7fb5faffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb438292b0 0x55eb43844670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.499+0000 7fb5fb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb43829c60 0x55eb43825c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.499+0000 7fb5faffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43838980 0x55eb43837e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.499+0000 7fb5fb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb43829c60 0x55eb43825c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:56216/0 (socket says 172.21.15.27:56216) 2024-04-25T00:23:20.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.499+0000 7fb5fb7fe640 1 -- 172.21.15.27:0/2524046967 learned_addr learned my addr 172.21.15.27:0/2524046967 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:20.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.499+0000 7fb5faffd640 1 -- 172.21.15.27:0/2524046967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb43829c60 msgr2=0x55eb43825c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:20.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.499+0000 7fb5faffd640 1 --2- 172.21.15.27:0/2524046967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb43829c60 0x55eb43825c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.499+0000 7fb5faffd640 1 -- 172.21.15.27:0/2524046967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb438292b0 msgr2=0x55eb43844670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:20.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.499+0000 7fb5faffd640 1 --2- 172.21.15.27:0/2524046967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb438292b0 0x55eb43844670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.499+0000 7fb5faffd640 1 -- 172.21.15.27:0/2524046967 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb4384b040 con 0x55eb43838980 2024-04-25T00:23:20.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.499+0000 7fb5faffd640 1 --2- 172.21.15.27:0/2524046967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43838980 0x55eb43837e60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb5f002a940 tx=0x7fb5f0009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36275760c9169fe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:20.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.500+0000 7fb5e17fa640 1 -- 172.21.15.27:0/2524046967 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0035040 con 0x55eb43838980 2024-04-25T00:23:20.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.500+0000 7fb5e17fa640 1 -- 172.21.15.27:0/2524046967 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5f0031dd0 con 0x55eb43838980 2024-04-25T00:23:20.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.500+0000 7fb5e17fa640 1 -- 172.21.15.27:0/2524046967 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f003db10 con 0x55eb43838980 2024-04-25T00:23:20.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.501+0000 7fb604e14740 1 -- 172.21.15.27:0/2524046967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43838980 msgr2=0x55eb43837e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.501+0000 7fb604e14740 1 --2- 172.21.15.27:0/2524046967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43838980 0x55eb43837e60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb5f002a940 tx=0x7fb5f0009680 comp rx=0 tx=0).stop 2024-04-25T00:23:20.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.501+0000 7fb604e14740 1 -- 172.21.15.27:0/2524046967 shutdown_connections 2024-04-25T00:23:20.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.501+0000 7fb604e14740 1 --2- 172.21.15.27:0/2524046967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43838980 0x55eb43837e60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.501+0000 7fb604e14740 1 --2- 172.21.15.27:0/2524046967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb43829c60 0x55eb43825c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.501+0000 7fb604e14740 1 --2- 172.21.15.27:0/2524046967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb438292b0 0x55eb43844670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.501+0000 7fb604e14740 1 -- 172.21.15.27:0/2524046967 >> 172.21.15.27:0/2524046967 conn(0x55eb438258a0 msgr2=0x55eb43830c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:20.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.501+0000 7fb604e14740 1 -- 172.21.15.27:0/2524046967 shutdown_connections 2024-04-25T00:23:20.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.501+0000 7fb604e14740 1 -- 172.21.15.27:0/2524046967 wait complete. 2024-04-25T00:23:20.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.502+0000 7fb604e14740 1 -- start start 2024-04-25T00:23:20.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.502+0000 7fb604e14740 1 Processor -- start 2024-04-25T00:23:20.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.503+0000 7fb604e14740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb438292b0 0x55eb43840d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.503+0000 7fb604e14740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43829c60 0x55eb43831e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.503+0000 7fb604e14740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43838980 0x55eb43832370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.503+0000 7fb5faffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb438292b0 0x55eb43840d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.503+0000 7fb5fb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43829c60 0x55eb43831e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.503+0000 7fb5fb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43829c60 0x55eb43831e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49392/0 (socket says 172.21.15.27:49392) 2024-04-25T00:23:20.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.503+0000 7fb5faffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb438292b0 0x55eb43840d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:56226/0 (socket says 172.21.15.27:56226) 2024-04-25T00:23:20.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.503+0000 7fb604e14740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb5fc1adf80 con 0x55eb43829c60 2024-04-25T00:23:20.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.503+0000 7fb604e14740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb5fc1ad880 con 0x55eb43838980 2024-04-25T00:23:20.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.503+0000 7fb604e14740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb5fc1aaa40 con 0x55eb438292b0 2024-04-25T00:23:20.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.503+0000 7fb5fb7fe640 1 -- 172.21.15.27:0/3965790362 learned_addr learned my addr 172.21.15.27:0/3965790362 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:20.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.504+0000 7fb604e14740 10 client.0 _ll_register_callbacks cb 0x7fb5f0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:20.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.504+0000 7fb5faffd640 1 -- 172.21.15.27:0/3965790362 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43838980 msgr2=0x55eb43832370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:20.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.504+0000 7fb5faffd640 1 --2- 172.21.15.27:0/3965790362 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43838980 0x55eb43832370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.504+0000 7fb5faffd640 1 -- 172.21.15.27:0/3965790362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43829c60 msgr2=0x55eb43831e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:20.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.504+0000 7fb5faffd640 1 --2- 172.21.15.27:0/3965790362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43829c60 0x55eb43831e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.504+0000 7fb5faffd640 1 -- 172.21.15.27:0/3965790362 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb43835f30 con 0x55eb438292b0 2024-04-25T00:23:20.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.504+0000 7fb5faffd640 1 --2- 172.21.15.27:0/3965790362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb438292b0 0x55eb43840d70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009850 tx=0x7fb5f00505c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18887a8dee2a71dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:20.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.504+0000 7fb5daffd640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:23:20.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.504+0000 7fb5daffd640 1 -- 172.21.15.27:0/3965790362 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0071060 con 0x55eb438292b0 2024-04-25T00:23:20.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.505+0000 7fb5daffd640 1 -- 172.21.15.27:0/3965790362 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fb5f003d450 con 0x55eb438292b0 2024-04-25T00:23:20.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.505+0000 7fb604e14740 1 -- 172.21.15.27:0/3965790362 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5f0031dd0 con 0x55eb438292b0 2024-04-25T00:23:20.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.505+0000 7fb5daffd640 1 -- 172.21.15.27:0/3965790362 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fb5f0035040 con 0x55eb438292b0 2024-04-25T00:23:20.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.505+0000 7fb604e14740 10 client.26116 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:20.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.505+0000 7fb604e14740 1 -- 172.21.15.27:0/3965790362 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55eb4356c7d0 con 0x55eb438292b0 2024-04-25T00:23:20.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.506+0000 7fb5daffd640 1 -- 172.21.15.27:0/3965790362 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5f003d890 con 0x55eb438292b0 2024-04-25T00:23:20.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.506+0000 7fb5daffd640 1 -- 172.21.15.27:0/3965790362 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7fb5f006c630 con 0x55eb438292b0 2024-04-25T00:23:20.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.506+0000 7fb604e14740 10 client.26116 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:20.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.506+0000 7fb604e14740 10 client.26116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:23:20.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.506+0000 7fb604e14740 20 client.26116 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:20.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.506+0000 7fb604e14740 1 --2- 172.21.15.27:0/3965790362 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55eb438aeaa0 0x55eb438b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.506+0000 7fb604e14740 4 client.26116 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:23:20.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.506+0000 7fb604e14740 1 -- 172.21.15.27:0/3965790362 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fb5f0031dd0 con 0x55eb438aeaa0 2024-04-25T00:23:20.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.507+0000 7fb5fb7fe640 1 --2- 172.21.15.27:0/3965790362 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55eb438aeaa0 0x55eb438b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.508+0000 7fb5daffd640 10 client.26116 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:20.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.508+0000 7fb5fb7fe640 1 --2- 172.21.15.27:0/3965790362 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55eb438aeaa0 0x55eb438b0ec0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b40376d9ced4aedb server_cookie=7cd6fc92996603cb in_seq=0 out_seq=0 2024-04-25T00:23:20.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.508+0000 7fb5daffd640 1 -- 172.21.15.27:0/3965790362 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fb5f0031dd0 con 0x55eb438aeaa0 2024-04-25T00:23:20.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.508+0000 7fb5daffd640 10 client.26116 handle_command_reply: tid=0 2024-04-25T00:23:20.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.508+0000 7fb604e14740 1 client.26116 shutdown 2024-04-25T00:23:20.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.510+0000 7fb604e14740 1 -- 172.21.15.27:0/3965790362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb438292b0 msgr2=0x55eb43840d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.510+0000 7fb604e14740 1 --2- 172.21.15.27:0/3965790362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb438292b0 0x55eb43840d70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009850 tx=0x7fb5f00505c0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.515+0000 7fb604e14740 1 -- 172.21.15.27:0/3965790362 shutdown_connections 2024-04-25T00:23:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.515+0000 7fb604e14740 1 --2- 172.21.15.27:0/3965790362 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55eb438aeaa0 0x55eb438b0ec0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.515+0000 7fb5daffd640 0 client.26116 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.515+0000 7fb604e14740 1 --2- 172.21.15.27:0/3965790362 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb43838980 0x55eb43832370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.515+0000 7fb604e14740 1 --2- 172.21.15.27:0/3965790362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb43829c60 0x55eb43831e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.515+0000 7fb604e14740 1 --2- 172.21.15.27:0/3965790362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb438292b0 0x55eb43840d70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.515+0000 7fb604e14740 1 -- 172.21.15.27:0/3965790362 >> 172.21.15.27:0/3965790362 conn(0x55eb438258a0 msgr2=0x55eb43830c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.516+0000 7fb604e14740 1 -- 172.21.15.27:0/3965790362 shutdown_connections 2024-04-25T00:23:20.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.518+0000 7fb604e14740 1 -- 172.21.15.27:0/3965790362 wait complete. 2024-04-25T00:23:20.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.520+0000 7fb604e14740 20 client.26116 trim_cache size 0 max 16384 2024-04-25T00:23:20.524 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:23:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.524+0000 7fb602109640 1 -- 172.21.15.27:0/2987412099 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb5e40a2d80 msgr2=0x7fb5e40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.524+0000 7fb602109640 1 --2- 172.21.15.27:0/2987412099 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb5e40a2d80 0x7fb5e40a5240 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4098960 tx=0x7fb5f400efd0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.524+0000 7fb602109640 1 -- 172.21.15.27:0/2987412099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc107730 msgr2=0x7fb5fc19c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.524+0000 7fb602109640 1 --2- 172.21.15.27:0/2987412099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc107730 0x7fb5fc19c8a0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb5f400eb10 tx=0x7fb5f400c2d0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.528+0000 7fb602109640 1 -- 172.21.15.27:0/2987412099 shutdown_connections 2024-04-25T00:23:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.528+0000 7fb602109640 1 --2- 172.21.15.27:0/2987412099 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb5e40a2d80 0x7fb5e40a5240 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.528+0000 7fb602109640 1 --2- 172.21.15.27:0/2987412099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc107730 0x7fb5fc19c8a0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.528+0000 7fb602109640 1 --2- 172.21.15.27:0/2987412099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5fc06a780 0x7fb5fc198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.528+0000 7fb602109640 1 --2- 172.21.15.27:0/2987412099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5fc069dc0 0x7fb5fc198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.528+0000 7fb602109640 1 -- 172.21.15.27:0/2987412099 >> 172.21.15.27:0/2987412099 conn(0x7fb5fc0776a0 msgr2=0x7fb5fc0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:20.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.530+0000 7fb602109640 1 -- 172.21.15.27:0/2987412099 shutdown_connections 2024-04-25T00:23:20.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:20.531+0000 7fb602109640 1 -- 172.21.15.27:0/2987412099 wait complete. 2024-04-25T00:23:20.555 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:20 smithi027 ceph-mon[29208]: pgmap v230: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 161 KiB/s wr, 135 op/s 2024-04-25T00:23:20.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:20 smithi139 ceph-mon[32833]: pgmap v230: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 161 KiB/s wr, 135 op/s 2024-04-25T00:23:20.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:20 smithi153 ceph-mon[32896]: pgmap v230: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 161 KiB/s wr, 135 op/s 2024-04-25T00:23:21.555 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T00:23:21.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:23:21.555 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T00:23:21.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.733+0000 7f534cc51640 1 -- 172.21.15.27:0/1790330509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f534009be10 msgr2=0x7f534009c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:21.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.733+0000 7f534cc51640 1 --2- 172.21.15.27:0/1790330509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f534009be10 0x7f534009c1f0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f533c00fb40 tx=0x7f533c02a1d0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.736+0000 7f534cc51640 1 -- 172.21.15.27:0/1790330509 shutdown_connections 2024-04-25T00:23:21.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.736+0000 7f534cc51640 1 --2- 172.21.15.27:0/1790330509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5340005100 0x7f53400a2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.736+0000 7f534cc51640 1 --2- 172.21.15.27:0/1790330509 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5340004740 0x7f5340004bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.736+0000 7f534cc51640 1 --2- 172.21.15.27:0/1790330509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f534009be10 0x7f534009c1f0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.736+0000 7f534cc51640 1 -- 172.21.15.27:0/1790330509 >> 172.21.15.27:0/1790330509 conn(0x7f534000a740 msgr2=0x7f534000ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:21.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.737+0000 7f534cc51640 1 -- 172.21.15.27:0/1790330509 shutdown_connections 2024-04-25T00:23:21.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.737+0000 7f534cc51640 1 -- 172.21.15.27:0/1790330509 wait complete. 2024-04-25T00:23:21.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.738+0000 7f534cc51640 1 Processor -- start 2024-04-25T00:23:21.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.738+0000 7f534cc51640 1 -- start start 2024-04-25T00:23:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.738+0000 7f534cc51640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5340004740 0x7f534013b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.738+0000 7f534cc51640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5340005100 0x7f534013b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.738+0000 7f534cc51640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f534009be10 0x7f534013fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.738+0000 7f534cc51640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53400a3f30 con 0x7f5340004740 2024-04-25T00:23:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.738+0000 7f534cc51640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f53400a3db0 con 0x7f534009be10 2024-04-25T00:23:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.738+0000 7f534cc51640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53400a40b0 con 0x7f5340005100 2024-04-25T00:23:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.739+0000 7f5347fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f534009be10 0x7f534013fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.739+0000 7f5346ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5340005100 0x7f534013b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.739+0000 7f5347fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f534009be10 0x7f534013fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:60956/0 (socket says 172.21.15.27:60956) 2024-04-25T00:23:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.739+0000 7f5347fff640 1 -- 172.21.15.27:0/3728785502 learned_addr learned my addr 172.21.15.27:0/3728785502 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:21.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.743+0000 7f53477fe640 1 --2- 172.21.15.27:0/3728785502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5340004740 0x7f534013b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:21.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.743+0000 7f5347fff640 1 -- 172.21.15.27:0/3728785502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5340005100 msgr2=0x7f534013b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:21.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.743+0000 7f5347fff640 1 --2- 172.21.15.27:0/3728785502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5340005100 0x7f534013b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.743+0000 7f5347fff640 1 -- 172.21.15.27:0/3728785502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5340004740 msgr2=0x7f534013b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:21.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.744+0000 7f5347fff640 1 --2- 172.21.15.27:0/3728785502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5340004740 0x7f534013b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.744+0000 7f5347fff640 1 -- 172.21.15.27:0/3728785502 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53401403b0 con 0x7f534009be10 2024-04-25T00:23:21.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.745+0000 7f5347fff640 1 --2- 172.21.15.27:0/3728785502 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f534009be10 0x7f534013fc80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5348050770 tx=0x7f534806b510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:21.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.745+0000 7f5344ff9640 1 -- 172.21.15.27:0/3728785502 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534806bd10 con 0x7f534009be10 2024-04-25T00:23:21.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.745+0000 7f5344ff9640 1 -- 172.21.15.27:0/3728785502 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5348073510 con 0x7f534009be10 2024-04-25T00:23:21.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.745+0000 7f5344ff9640 1 -- 172.21.15.27:0/3728785502 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53480696f0 con 0x7f534009be10 2024-04-25T00:23:21.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.745+0000 7f534cc51640 1 -- 172.21.15.27:0/3728785502 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53401406a0 con 0x7f534009be10 2024-04-25T00:23:21.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.745+0000 7f534cc51640 1 -- 172.21.15.27:0/3728785502 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f534012f3d0 con 0x7f534009be10 2024-04-25T00:23:21.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.746+0000 7f53477fe640 1 --2- 172.21.15.27:0/3728785502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5340004740 0x7f534013b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:23:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.749+0000 7f5350bc4740 1 Processor -- start 2024-04-25T00:23:21.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.750+0000 7f5350bc4740 1 -- start start 2024-04-25T00:23:21.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.750+0000 7f5350bc4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35cdb70 0x5581b35cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.750+0000 7f5350bc4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cc480 0x5581b35cc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.750+0000 7f5350bc4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35d7200 0x5581b35d95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.750+0000 7f5350bc4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5581b35302c0 con 0x5581b35cdb70 2024-04-25T00:23:21.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.750+0000 7f5350bc4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5581b34d8910 con 0x5581b35d7200 2024-04-25T00:23:21.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.750+0000 7f5350bc4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5581b352c280 con 0x5581b35cc480 2024-04-25T00:23:21.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.750+0000 7f5344ff9640 1 -- 172.21.15.27:0/3728785502 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5348069890 con 0x7f534009be10 2024-04-25T00:23:21.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.751+0000 7f5344ff9640 1 --2- 172.21.15.27:0/3728785502 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f53380a2e00 0x7f53380a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.752+0000 7f5344ff9640 1 -- 172.21.15.27:0/3728785502 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f5348133210 con 0x7f534009be10 2024-04-25T00:23:21.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.752+0000 7f5346ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35cdb70 0x5581b35cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:21.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.752+0000 7f5346ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35cdb70 0x5581b35cbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49428/0 (socket says 172.21.15.27:49428) 2024-04-25T00:23:21.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.752+0000 7f5346ffd640 1 -- 172.21.15.27:0/1973466144 learned_addr learned my addr 172.21.15.27:0/1973466144 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:21.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.752+0000 7f5346ffd640 1 --2- 172.21.15.27:0/1973466144 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35d7200 0x5581b35d95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:21.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.753+0000 7f5347fff640 1 --2- 172.21.15.27:0/3728785502 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f53380a2e00 0x7f53380a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.753+0000 7f5346ffd640 1 -- 172.21.15.27:0/1973466144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cc480 msgr2=0x5581b35cc900 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:23:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.753+0000 7f5346ffd640 1 --2- 172.21.15.27:0/1973466144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cc480 0x5581b35cc900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.753+0000 7f5346ffd640 1 -- 172.21.15.27:0/1973466144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35cdb70 msgr2=0x5581b35cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.753+0000 7f5346ffd640 1 --2- 172.21.15.27:0/1973466144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35cdb70 0x5581b35cbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.753+0000 7f5346ffd640 1 -- 172.21.15.27:0/1973466144 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581b34e9fb0 con 0x5581b35d7200 2024-04-25T00:23:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.753+0000 7f5346ffd640 1 --2- 172.21.15.27:0/1973466144 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35d7200 0x5581b35d95f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5334009dd0 tx=0x7f5334005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7aef1f0fe640273 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.756+0000 7f5347fff640 1 --2- 172.21.15.27:0/3728785502 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f53380a2e00 0x7f53380a52c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f53480ff310 tx=0x7f53480738c0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.757+0000 7f5331ffb640 1 -- 172.21.15.27:0/1973466144 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533402a070 con 0x5581b35d7200 2024-04-25T00:23:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.757+0000 7f5331ffb640 1 -- 172.21.15.27:0/1973466144 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5334021b80 con 0x5581b35d7200 2024-04-25T00:23:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.757+0000 7f5331ffb640 1 -- 172.21.15.27:0/1973466144 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53340243e0 con 0x5581b35d7200 2024-04-25T00:23:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.758+0000 7f5350bc4740 1 -- 172.21.15.27:0/1973466144 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35d7200 msgr2=0x5581b35d95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.758+0000 7f5350bc4740 1 --2- 172.21.15.27:0/1973466144 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35d7200 0x5581b35d95f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5334009dd0 tx=0x7f5334005f90 comp rx=0 tx=0).stop 2024-04-25T00:23:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.758+0000 7f5350bc4740 1 -- 172.21.15.27:0/1973466144 shutdown_connections 2024-04-25T00:23:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.758+0000 7f5350bc4740 1 --2- 172.21.15.27:0/1973466144 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35d7200 0x5581b35d95f0 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5334009dd0 tx=0x7f5334005f90 comp rx=0 tx=0).stop 2024-04-25T00:23:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.758+0000 7f5350bc4740 1 --2- 172.21.15.27:0/1973466144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cc480 0x5581b35cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.758+0000 7f5350bc4740 1 --2- 172.21.15.27:0/1973466144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35cdb70 0x5581b35cbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.758+0000 7f5350bc4740 1 -- 172.21.15.27:0/1973466144 >> 172.21.15.27:0/1973466144 conn(0x5581b3594950 msgr2=0x5581b35c89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.761+0000 7f5350bc4740 1 -- 172.21.15.27:0/1973466144 shutdown_connections 2024-04-25T00:23:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.761+0000 7f5350bc4740 1 -- 172.21.15.27:0/1973466144 wait complete. 2024-04-25T00:23:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.761+0000 7f5350bc4740 1 -- start start 2024-04-25T00:23:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.761+0000 7f5350bc4740 1 Processor -- start 2024-04-25T00:23:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.762+0000 7f5350bc4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35cc480 0x5581b35d43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.762+0000 7f5350bc4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cdb70 0x5581b35d48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.762+0000 7f5350bc4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35d4e70 0x5581b35e35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.762+0000 7f5350bc4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5581b35da090 con 0x5581b35d4e70 2024-04-25T00:23:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.762+0000 7f5350bc4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5581b34ebe90 con 0x5581b35cc480 2024-04-25T00:23:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.762+0000 7f5350bc4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5581b35da210 con 0x5581b35cdb70 2024-04-25T00:23:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.762+0000 7f5350bc4740 10 client.0 _ll_register_callbacks cb 0x5581b3527970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.762+0000 7f53477fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cdb70 0x5581b35d48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.762+0000 7f53477fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cdb70 0x5581b35d48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:56266/0 (socket says 172.21.15.27:56266) 2024-04-25T00:23:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.762+0000 7f53477fe640 1 -- 172.21.15.27:0/2815982162 learned_addr learned my addr 172.21.15.27:0/2815982162 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.762+0000 7f5346ffd640 1 --2- 172.21.15.27:0/2815982162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35cc480 0x5581b35d43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:21.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.763+0000 7f5346ffd640 1 --2- 172.21.15.27:0/2815982162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35d4e70 0x5581b35e35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:21.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.763+0000 7f53477fe640 1 -- 172.21.15.27:0/2815982162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35cc480 msgr2=0x5581b35d43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:21.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.763+0000 7f53477fe640 1 --2- 172.21.15.27:0/2815982162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35cc480 0x5581b35d43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.763+0000 7f53477fe640 1 -- 172.21.15.27:0/2815982162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35d4e70 msgr2=0x5581b35e35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.763+0000 7f53477fe640 1 --2- 172.21.15.27:0/2815982162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35d4e70 0x5581b35e35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.763+0000 7f53477fe640 1 -- 172.21.15.27:0/2815982162 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581b330c7e0 con 0x5581b35cdb70 2024-04-25T00:23:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.763+0000 7f53477fe640 1 --2- 172.21.15.27:0/2815982162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cdb70 0x5581b35d48e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f533c005e00 tx=0x7f533c005e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c017bb9d2be0e7e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.763+0000 7f5326ffd640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:23:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.763+0000 7f5326ffd640 1 -- 172.21.15.27:0/2815982162 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c03f020 con 0x5581b35cdb70 2024-04-25T00:23:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.764+0000 7f5350bc4740 1 -- 172.21.15.27:0/2815982162 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5334021b80 con 0x5581b35cdb70 2024-04-25T00:23:21.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.765+0000 7f5326ffd640 1 -- 172.21.15.27:0/2815982162 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f533c02ada0 con 0x5581b35cdb70 2024-04-25T00:23:21.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.766+0000 7f5326ffd640 1 -- 172.21.15.27:0/2815982162 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f533c04a050 con 0x5581b35cdb70 2024-04-25T00:23:21.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.767+0000 7f5350bc4740 10 client.26125 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:21.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.767+0000 7f5350bc4740 1 -- 172.21.15.27:0/2815982162 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5581b336cf70 con 0x5581b35cdb70 2024-04-25T00:23:21.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.767+0000 7f5326ffd640 1 -- 172.21.15.27:0/2815982162 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5334021b80 con 0x5581b35cdb70 2024-04-25T00:23:21.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.768+0000 7f5326ffd640 1 -- 172.21.15.27:0/2815982162 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f533c00ed20 con 0x5581b35cdb70 2024-04-25T00:23:21.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.768+0000 7f5350bc4740 10 client.26125 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:21.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.768+0000 7f5350bc4740 10 client.26125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:23:21.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.768+0000 7f5350bc4740 20 client.26125 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:21.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.768+0000 7f5350bc4740 1 --2- 172.21.15.27:0/2815982162 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5581b35f8f00 0x5581b35fb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.768+0000 7f5350bc4740 4 client.26125 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:23:21.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.768+0000 7f5350bc4740 1 -- 172.21.15.27:0/2815982162 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581b35045a0 con 0x5581b35f8f00 2024-04-25T00:23:21.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.768+0000 7f5346ffd640 1 --2- 172.21.15.27:0/2815982162 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5581b35f8f00 0x5581b35fb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:21.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.771+0000 7f5326ffd640 10 client.26125 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:21.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.771+0000 7f5346ffd640 1 --2- 172.21.15.27:0/2815982162 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5581b35f8f00 0x5581b35fb340 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63de58b5a5ac33a6 server_cookie=1236e1bf5962944 in_seq=0 out_seq=0 2024-04-25T00:23:21.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.771+0000 7f5326ffd640 1 -- 172.21.15.27:0/2815982162 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5581b35045a0 con 0x5581b35f8f00 2024-04-25T00:23:21.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.771+0000 7f5326ffd640 10 client.26125 handle_command_reply: tid=0 2024-04-25T00:23:21.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.773+0000 7f5350bc4740 1 client.26125 shutdown 2024-04-25T00:23:21.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.773+0000 7f5350bc4740 1 -- 172.21.15.27:0/2815982162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cdb70 msgr2=0x5581b35d48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:21.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.773+0000 7f5350bc4740 1 --2- 172.21.15.27:0/2815982162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cdb70 0x5581b35d48e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f533c005e00 tx=0x7f533c005e30 comp rx=0 tx=0).stop 2024-04-25T00:23:21.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.773+0000 7f5350bc4740 1 -- 172.21.15.27:0/2815982162 shutdown_connections 2024-04-25T00:23:21.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.773+0000 7f5350bc4740 1 --2- 172.21.15.27:0/2815982162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35d4e70 0x5581b35e35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.773+0000 7f5350bc4740 1 --2- 172.21.15.27:0/2815982162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cdb70 0x5581b35d48e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.773+0000 7f5350bc4740 1 --2- 172.21.15.27:0/2815982162 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5581b35f8f00 0x5581b35fb340 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.773+0000 7f5350bc4740 1 --2- 172.21.15.27:0/2815982162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35cc480 0x5581b35d43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.773+0000 7f5350bc4740 1 -- 172.21.15.27:0/2815982162 >> 172.21.15.27:0/2815982162 conn(0x5581b3594950 msgr2=0x5581b35ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:21.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.774+0000 7f5326ffd640 0 client.26125 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:21.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.774+0000 7f5350bc4740 1 -- 172.21.15.27:0/2815982162 shutdown_connections 2024-04-25T00:23:21.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.774+0000 7f5350bc4740 1 -- 172.21.15.27:0/2815982162 wait complete. 2024-04-25T00:23:21.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.774+0000 7f5350bc4740 20 client.26125 trim_cache size 0 max 16384 2024-04-25T00:23:21.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.810+0000 7f5350bc4740 1 Processor -- start 2024-04-25T00:23:21.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.810+0000 7f5350bc4740 1 -- start start 2024-04-25T00:23:21.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.810+0000 7f5350bc4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35cc480 0x5581b35d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.810+0000 7f5350bc4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35cdb70 0x5581b3600c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.810+0000 7f5346ffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35cc480 0x5581b35d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:21.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.810+0000 7f5346ffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35cc480 0x5581b35d32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:60988/0 (socket says 172.21.15.27:60988) 2024-04-25T00:23:21.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.810+0000 7f5346ffd640 1 -- 172.21.15.27:0/1029495981 learned_addr learned my addr 172.21.15.27:0/1029495981 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:21.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.810+0000 7f5350bc4740 1 --2- 172.21.15.27:0/1029495981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35d4e70 0x5581b3601190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.811+0000 7f5350bc4740 1 -- 172.21.15.27:0/1029495981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5340165300 con 0x5581b35cdb70 2024-04-25T00:23:21.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.811+0000 7f53477fe640 1 --2- 172.21.15.27:0/1029495981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35cdb70 0x5581b3600c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:21.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.811+0000 7f5350bc4740 1 -- 172.21.15.27:0/1029495981 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5340139530 con 0x5581b35cc480 2024-04-25T00:23:21.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.811+0000 7f5350bc4740 1 -- 172.21.15.27:0/1029495981 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5340164c30 con 0x5581b35d4e70 2024-04-25T00:23:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.811+0000 7f53477fe640 1 -- 172.21.15.27:0/1029495981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35d4e70 msgr2=0x5581b3601190 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:23:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.811+0000 7f53477fe640 1 --2- 172.21.15.27:0/1029495981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35d4e70 0x5581b3601190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.811+0000 7f53477fe640 1 -- 172.21.15.27:0/1029495981 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35cc480 msgr2=0x5581b35d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.811+0000 7f53477fe640 1 --2- 172.21.15.27:0/1029495981 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35cc480 0x5581b35d32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.811+0000 7f53477fe640 1 -- 172.21.15.27:0/1029495981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581b3204170 con 0x5581b35cdb70 2024-04-25T00:23:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.812+0000 7f53477fe640 1 --2- 172.21.15.27:0/1029495981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35cdb70 0x5581b3600c50 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f533c006010 tx=0x7f533c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902e0508b9e32d64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:21.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.813+0000 7f53317fa640 1 -- 172.21.15.27:0/1029495981 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c03f040 con 0x5581b35cdb70 2024-04-25T00:23:21.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.814+0000 7f53317fa640 1 -- 172.21.15.27:0/1029495981 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f533c02e040 con 0x5581b35cdb70 2024-04-25T00:23:21.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.814+0000 7f53317fa640 1 -- 172.21.15.27:0/1029495981 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c02c940 con 0x5581b35cdb70 2024-04-25T00:23:21.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.815+0000 7f5350bc4740 1 -- 172.21.15.27:0/1029495981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35cdb70 msgr2=0x5581b3600c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:21.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.815+0000 7f5350bc4740 1 --2- 172.21.15.27:0/1029495981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35cdb70 0x5581b3600c50 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f533c006010 tx=0x7f533c009260 comp rx=0 tx=0).stop 2024-04-25T00:23:21.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.816+0000 7f5350bc4740 1 -- 172.21.15.27:0/1029495981 shutdown_connections 2024-04-25T00:23:21.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.816+0000 7f5350bc4740 1 --2- 172.21.15.27:0/1029495981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35d4e70 0x5581b3601190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.816+0000 7f5350bc4740 1 --2- 172.21.15.27:0/1029495981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35cdb70 0x5581b3600c50 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.816+0000 7f5350bc4740 1 --2- 172.21.15.27:0/1029495981 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35cc480 0x5581b35d32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.816+0000 7f5350bc4740 1 -- 172.21.15.27:0/1029495981 >> 172.21.15.27:0/1029495981 conn(0x5581b3594950 msgr2=0x5581b35c7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.817+0000 7f5350bc4740 1 -- 172.21.15.27:0/1029495981 shutdown_connections 2024-04-25T00:23:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.817+0000 7f5350bc4740 1 -- 172.21.15.27:0/1029495981 wait complete. 2024-04-25T00:23:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.818+0000 7f5350bc4740 1 -- start start 2024-04-25T00:23:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.818+0000 7f5350bc4740 1 Processor -- start 2024-04-25T00:23:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.818+0000 7f5350bc4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cc480 0x5581b35d85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.818+0000 7f5350bc4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35d4e70 0x5581b35d8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.818+0000 7f5350bc4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35ddc60 0x5581b35de0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.819+0000 7f5346ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cc480 0x5581b35d85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.819+0000 7f5346ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cc480 0x5581b35d85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:56278/0 (socket says 172.21.15.27:56278) 2024-04-25T00:23:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.819+0000 7f53477fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35d4e70 0x5581b35d8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.819+0000 7f5346ffd640 1 -- 172.21.15.27:0/1334766112 learned_addr learned my addr 172.21.15.27:0/1334766112 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.819+0000 7f5350bc4740 1 -- 172.21.15.27:0/1334766112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5340163f10 con 0x5581b35ddc60 2024-04-25T00:23:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.819+0000 7f5350bc4740 1 -- 172.21.15.27:0/1334766112 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5340139c00 con 0x5581b35d4e70 2024-04-25T00:23:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.819+0000 7f5350bc4740 1 -- 172.21.15.27:0/1334766112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5340164560 con 0x5581b35cc480 2024-04-25T00:23:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.819+0000 7f5350bc4740 10 client.0 _ll_register_callbacks cb 0x7f5334005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.819+0000 7f5346ffd640 1 --2- 172.21.15.27:0/1334766112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35ddc60 0x5581b35de0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.819+0000 7f5346ffd640 1 -- 172.21.15.27:0/1334766112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35d4e70 msgr2=0x5581b35d8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.819+0000 7f5346ffd640 1 --2- 172.21.15.27:0/1334766112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35d4e70 0x5581b35d8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.819+0000 7f5346ffd640 1 -- 172.21.15.27:0/1334766112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35ddc60 msgr2=0x5581b35de0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.819+0000 7f5346ffd640 1 --2- 172.21.15.27:0/1334766112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35ddc60 0x5581b35de0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.819+0000 7f5346ffd640 1 -- 172.21.15.27:0/1334766112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581b35de620 con 0x5581b35cc480 2024-04-25T00:23:21.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.820+0000 7f5346ffd640 1 --2- 172.21.15.27:0/1334766112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cc480 0x5581b35d85f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5334009570 tx=0x7f5334025a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d81e95bc1f79ea84 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:21.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.820+0000 7f5326ffd640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:23:21.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.821+0000 7f5326ffd640 1 -- 172.21.15.27:0/1334766112 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5334000e60 con 0x5581b35cc480 2024-04-25T00:23:21.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.821+0000 7f5350bc4740 1 -- 172.21.15.27:0/1334766112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f533c02e040 con 0x5581b35cc480 2024-04-25T00:23:21.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.821+0000 7f5326ffd640 1 -- 172.21.15.27:0/1334766112 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f5334025640 con 0x5581b35cc480 2024-04-25T00:23:21.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.821+0000 7f5326ffd640 1 -- 172.21.15.27:0/1334766112 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f533402a070 con 0x5581b35cc480 2024-04-25T00:23:21.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.821+0000 7f5350bc4740 10 client.26128 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:21.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.821+0000 7f5350bc4740 1 -- 172.21.15.27:0/1334766112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5581b3660150 con 0x5581b35cc480 2024-04-25T00:23:21.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.822+0000 7f5326ffd640 1 -- 172.21.15.27:0/1334766112 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f533c02e040 con 0x5581b35cc480 2024-04-25T00:23:21.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.822+0000 7f5326ffd640 1 -- 172.21.15.27:0/1334766112 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f533401f070 con 0x5581b35cc480 2024-04-25T00:23:21.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.822+0000 7f5350bc4740 10 client.26128 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:21.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.822+0000 7f5350bc4740 10 client.26128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:23:21.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.822+0000 7f5350bc4740 20 client.26128 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:21.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.822+0000 7f5350bc4740 1 --2- 172.21.15.27:0/1334766112 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5581b3660710 0x5581b3662b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:21.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.822+0000 7f5350bc4740 4 client.26128 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T00:23:21.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.822+0000 7f5350bc4740 1 -- 172.21.15.27:0/1334766112 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5581b353d800 con 0x5581b3660710 2024-04-25T00:23:21.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.823+0000 7f53477fe640 1 --2- 172.21.15.27:0/1334766112 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5581b3660710 0x5581b3662b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:21.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.823+0000 7f53477fe640 1 --2- 172.21.15.27:0/1334766112 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5581b3660710 0x5581b3662b30 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=597356e060cd60bf server_cookie=bbe35a7c2708ab68 in_seq=0 out_seq=0 2024-04-25T00:23:21.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.823+0000 7f5326ffd640 10 client.26128 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:21.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.824+0000 7f5326ffd640 1 -- 172.21.15.27:0/1334766112 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5581b353d800 con 0x5581b3660710 2024-04-25T00:23:21.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.824+0000 7f5326ffd640 10 client.26128 handle_command_reply: tid=0 2024-04-25T00:23:21.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.824+0000 7f5350bc4740 1 client.26128 shutdown 2024-04-25T00:23:21.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.825+0000 7f5350bc4740 1 -- 172.21.15.27:0/1334766112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cc480 msgr2=0x5581b35d85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:21.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.825+0000 7f5350bc4740 1 --2- 172.21.15.27:0/1334766112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cc480 0x5581b35d85f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5334009570 tx=0x7f5334025a30 comp rx=0 tx=0).stop 2024-04-25T00:23:21.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.825+0000 7f5350bc4740 1 -- 172.21.15.27:0/1334766112 shutdown_connections 2024-04-25T00:23:21.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.825+0000 7f5350bc4740 1 --2- 172.21.15.27:0/1334766112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581b35ddc60 0x5581b35de0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.825+0000 7f5350bc4740 1 --2- 172.21.15.27:0/1334766112 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5581b3660710 0x5581b3662b30 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.825+0000 7f5350bc4740 1 --2- 172.21.15.27:0/1334766112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5581b35d4e70 0x5581b35d8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.825+0000 7f5350bc4740 1 --2- 172.21.15.27:0/1334766112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5581b35cc480 0x5581b35d85f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.825+0000 7f5350bc4740 1 -- 172.21.15.27:0/1334766112 >> 172.21.15.27:0/1334766112 conn(0x5581b3594950 msgr2=0x5581b35cdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:21.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.825+0000 7f5326ffd640 0 client.26128 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:21.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.825+0000 7f5350bc4740 1 -- 172.21.15.27:0/1334766112 shutdown_connections 2024-04-25T00:23:21.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.825+0000 7f5350bc4740 1 -- 172.21.15.27:0/1334766112 wait complete. 2024-04-25T00:23:21.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.825+0000 7f5350bc4740 20 client.26128 trim_cache size 0 max 16384 2024-04-25T00:23:21.827 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:23:21.827 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T00:23:21.827 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "1c7b0f05-8274-46ba-adc8-928f834ed859", 2024-04-25T00:23:21.828 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T00:23:21.828 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:23:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.826+0000 7f534cc51640 1 -- 172.21.15.27:0/3728785502 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f53380a2e00 msgr2=0x7f53380a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.826+0000 7f534cc51640 1 --2- 172.21.15.27:0/3728785502 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f53380a2e00 0x7f53380a52c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f53480ff310 tx=0x7f53480738c0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.826+0000 7f534cc51640 1 -- 172.21.15.27:0/3728785502 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f534009be10 msgr2=0x7f534013fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.826+0000 7f534cc51640 1 --2- 172.21.15.27:0/3728785502 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f534009be10 0x7f534013fc80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5348050770 tx=0x7f534806b510 comp rx=0 tx=0).stop 2024-04-25T00:23:21.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.826+0000 7f534cc51640 1 -- 172.21.15.27:0/3728785502 shutdown_connections 2024-04-25T00:23:21.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.826+0000 7f534cc51640 1 --2- 172.21.15.27:0/3728785502 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f53380a2e00 0x7f53380a52c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.826+0000 7f534cc51640 1 --2- 172.21.15.27:0/3728785502 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f534009be10 0x7f534013fc80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.826+0000 7f534cc51640 1 --2- 172.21.15.27:0/3728785502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5340005100 0x7f534013b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.826+0000 7f534cc51640 1 --2- 172.21.15.27:0/3728785502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5340004740 0x7f534013b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:21.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.826+0000 7f534cc51640 1 -- 172.21.15.27:0/3728785502 >> 172.21.15.27:0/3728785502 conn(0x7f534000a740 msgr2=0x7f5340093a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:21.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.826+0000 7f534cc51640 1 -- 172.21.15.27:0/3728785502 shutdown_connections 2024-04-25T00:23:21.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:21.826+0000 7f534cc51640 1 -- 172.21.15.27:0/3728785502 wait complete. 2024-04-25T00:23:21.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:23:21.843 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:23:22.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.014+0000 7f4491e4c640 1 -- 172.21.15.27:0/3166336059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c10bd00 msgr2=0x7f448c10e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:22.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.014+0000 7f4491e4c640 1 --2- 172.21.15.27:0/3166336059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c10bd00 0x7f448c10e1a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4484009ca0 tx=0x7f448402a9f0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.015+0000 7f4491e4c640 1 -- 172.21.15.27:0/3166336059 shutdown_connections 2024-04-25T00:23:22.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.015+0000 7f4491e4c640 1 --2- 172.21.15.27:0/3166336059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c10bd00 0x7f448c10e1a0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.015+0000 7f4491e4c640 1 --2- 172.21.15.27:0/3166336059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f448c1032e0 0x7f448c10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.015+0000 7f4491e4c640 1 --2- 172.21.15.27:0/3166336059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f448c1029c0 0x7f448c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.015+0000 7f4491e4c640 1 -- 172.21.15.27:0/3166336059 >> 172.21.15.27:0/3166336059 conn(0x7f448c077740 msgr2=0x7f448c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.015+0000 7f4491e4c640 1 -- 172.21.15.27:0/3166336059 shutdown_connections 2024-04-25T00:23:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.015+0000 7f4491e4c640 1 -- 172.21.15.27:0/3166336059 wait complete. 2024-04-25T00:23:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.015+0000 7f4491e4c640 1 Processor -- start 2024-04-25T00:23:22.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.016+0000 7f4491e4c640 1 -- start start 2024-04-25T00:23:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.016+0000 7f4491e4c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c1029c0 0x7f448c072fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.016+0000 7f4491e4c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f448c1032e0 0x7f448c06c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.016+0000 7f4491e4c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f448c10bd00 0x7f448c06c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.016+0000 7f4491e4c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f448c113e30 con 0x7f448c1029c0 2024-04-25T00:23:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.016+0000 7f4491e4c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f448c113cb0 con 0x7f448c1032e0 2024-04-25T00:23:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.016+0000 7f4491e4c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f448c113fb0 con 0x7f448c10bd00 2024-04-25T00:23:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.016+0000 7f448b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c1029c0 0x7f448c072fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.016+0000 7f448b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c1029c0 0x7f448c072fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49480/0 (socket says 172.21.15.27:49480) 2024-04-25T00:23:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.016+0000 7f448b7fe640 1 -- 172.21.15.27:0/3756398763 learned_addr learned my addr 172.21.15.27:0/3756398763 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.016+0000 7f448affd640 1 --2- 172.21.15.27:0/3756398763 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f448c1032e0 0x7f448c06c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.016+0000 7f448bfff640 1 --2- 172.21.15.27:0/3756398763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f448c10bd00 0x7f448c06c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.016+0000 7f448b7fe640 1 -- 172.21.15.27:0/3756398763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f448c10bd00 msgr2=0x7f448c06c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.016+0000 7f448b7fe640 1 --2- 172.21.15.27:0/3756398763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f448c10bd00 0x7f448c06c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.016+0000 7f448b7fe640 1 -- 172.21.15.27:0/3756398763 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f448c1032e0 msgr2=0x7f448c06c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.016+0000 7f448b7fe640 1 --2- 172.21.15.27:0/3756398763 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f448c1032e0 0x7f448c06c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.016+0000 7f448b7fe640 1 -- 172.21.15.27:0/3756398763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f448c06cbc0 con 0x7f448c1029c0 2024-04-25T00:23:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.017+0000 7f448affd640 1 --2- 172.21.15.27:0/3756398763 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f448c1032e0 0x7f448c06c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.017+0000 7f448bfff640 1 --2- 172.21.15.27:0/3756398763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f448c10bd00 0x7f448c06c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.017+0000 7f448b7fe640 1 --2- 172.21.15.27:0/3756398763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c1029c0 0x7f448c072fa0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f447c00b4a0 tx=0x7f447c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.017+0000 7f4488ff9640 1 -- 172.21.15.27:0/3756398763 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c014070 con 0x7f448c1029c0 2024-04-25T00:23:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.017+0000 7f4488ff9640 1 -- 172.21.15.27:0/3756398763 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f447c010af0 con 0x7f448c1029c0 2024-04-25T00:23:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.017+0000 7f4488ff9640 1 -- 172.21.15.27:0/3756398763 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c017d10 con 0x7f448c1029c0 2024-04-25T00:23:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.017+0000 7f4491e4c640 1 -- 172.21.15.27:0/3756398763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f448c06ceb0 con 0x7f448c1029c0 2024-04-25T00:23:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.017+0000 7f4491e4c640 1 -- 172.21.15.27:0/3756398763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f448c070df0 con 0x7f448c1029c0 2024-04-25T00:23:22.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.020+0000 7f4494c30740 1 Processor -- start 2024-04-25T00:23:22.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.020+0000 7f4494c30740 1 -- start start 2024-04-25T00:23:22.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.020+0000 7f4494c30740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bc0e40 0x55d343bc1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.020+0000 7f4494c30740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc1760 0x55d343bcb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.020+0000 7f4494c30740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bcbe30 0x55d343bce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.020+0000 7f4488ff9640 1 -- 172.21.15.27:0/3756398763 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f447c017070 con 0x7f448c1029c0 2024-04-25T00:23:22.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.020+0000 7f4494c30740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d343b1d6b0 con 0x55d343bc0e40 2024-04-25T00:23:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.020+0000 7f4494c30740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d343ac5880 con 0x55d343bcbe30 2024-04-25T00:23:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.020+0000 7f4494c30740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d343b19670 con 0x55d343bc1760 2024-04-25T00:23:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.021+0000 7f448b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc1760 0x55d343bcb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.021+0000 7f448b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc1760 0x55d343bcb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:56302/0 (socket says 172.21.15.27:56302) 2024-04-25T00:23:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.021+0000 7f448b7fe640 1 -- 172.21.15.27:0/4047010493 learned_addr learned my addr 172.21.15.27:0/4047010493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.021+0000 7f448affd640 1 --2- 172.21.15.27:0/4047010493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bcbe30 0x55d343bce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.021+0000 7f448bfff640 1 --2- 172.21.15.27:0/4047010493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bc0e40 0x55d343bc1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.021+0000 7f448b7fe640 1 -- 172.21.15.27:0/4047010493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bcbe30 msgr2=0x55d343bce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.021+0000 7f448b7fe640 1 --2- 172.21.15.27:0/4047010493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bcbe30 0x55d343bce220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.021+0000 7f448b7fe640 1 -- 172.21.15.27:0/4047010493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bc0e40 msgr2=0x55d343bc1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.021+0000 7f448b7fe640 1 --2- 172.21.15.27:0/4047010493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bc0e40 0x55d343bc1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.021+0000 7f448b7fe640 1 -- 172.21.15.27:0/4047010493 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3438fb7d0 con 0x55d343bc1760 2024-04-25T00:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.021+0000 7f448affd640 1 --2- 172.21.15.27:0/4047010493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bcbe30 0x55d343bce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.021+0000 7f448b7fe640 1 --2- 172.21.15.27:0/4047010493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc1760 0x55d343bcb8f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f447c0cbd20 tx=0x7f447c0d6670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95dc9a59b4c31148 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.021+0000 7f4488ff9640 1 --2- 172.21.15.27:0/3756398763 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f44740a2d80 0x7f44740a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.022+0000 7f4471ffb640 1 -- 172.21.15.27:0/4047010493 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c0dd070 con 0x55d343bc1760 2024-04-25T00:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.022+0000 7f448bfff640 1 --2- 172.21.15.27:0/3756398763 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f44740a2d80 0x7f44740a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.022+0000 7f4471ffb640 1 -- 172.21.15.27:0/4047010493 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f447c06fa70 con 0x55d343bc1760 2024-04-25T00:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.022+0000 7f4471ffb640 1 -- 172.21.15.27:0/4047010493 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c096f00 con 0x55d343bc1760 2024-04-25T00:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.022+0000 7f4488ff9640 1 -- 172.21.15.27:0/3756398763 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f447c0ce930 con 0x7f448c1029c0 2024-04-25T00:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.022+0000 7f4494c30740 1 -- 172.21.15.27:0/4047010493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc1760 msgr2=0x55d343bcb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.022+0000 7f448bfff640 1 --2- 172.21.15.27:0/3756398763 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f44740a2d80 0x7f44740a5240 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4484009430 tx=0x7f4484009800 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.022+0000 7f4494c30740 1 --2- 172.21.15.27:0/4047010493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc1760 0x55d343bcb8f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f447c0cbd20 tx=0x7f447c0d6670 comp rx=0 tx=0).stop 2024-04-25T00:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.023+0000 7f4494c30740 1 -- 172.21.15.27:0/4047010493 shutdown_connections 2024-04-25T00:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.023+0000 7f4494c30740 1 --2- 172.21.15.27:0/4047010493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bcbe30 0x55d343bce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.023+0000 7f4494c30740 1 --2- 172.21.15.27:0/4047010493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc1760 0x55d343bcb8f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.023+0000 7f4494c30740 1 --2- 172.21.15.27:0/4047010493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bc0e40 0x55d343bc1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.023+0000 7f4494c30740 1 -- 172.21.15.27:0/4047010493 >> 172.21.15.27:0/4047010493 conn(0x55d343bbd160 msgr2=0x55d343bbd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.023+0000 7f4494c30740 1 -- 172.21.15.27:0/4047010493 shutdown_connections 2024-04-25T00:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.023+0000 7f4494c30740 1 -- 172.21.15.27:0/4047010493 wait complete. 2024-04-25T00:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.023+0000 7f4494c30740 1 -- start start 2024-04-25T00:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.023+0000 7f4494c30740 1 Processor -- start 2024-04-25T00:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.023+0000 7f4494c30740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 0x55d343bc8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.023+0000 7f4494c30740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bc1760 0x55d343bc9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.023+0000 7f4494c30740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bcbe30 0x55d343bc9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f4494c30740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d343bcec20 con 0x55d343bc1760 2024-04-25T00:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f4494c30740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d343ad8f50 con 0x55d343bcbe30 2024-04-25T00:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f4494c30740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d343bceda0 con 0x55d343bc0e40 2024-04-25T00:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f4494c30740 10 client.0 _ll_register_callbacks cb 0x55d343b14000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f448b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 0x55d343bc8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f448b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 0x55d343bc8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:56316/0 (socket says 172.21.15.27:56316) 2024-04-25T00:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f448b7fe640 1 -- 172.21.15.27:0/4201150891 learned_addr learned my addr 172.21.15.27:0/4201150891 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f448affd640 1 --2- 172.21.15.27:0/4201150891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bc1760 0x55d343bc9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f448bfff640 1 --2- 172.21.15.27:0/4201150891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bcbe30 0x55d343bc9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f448b7fe640 1 -- 172.21.15.27:0/4201150891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bcbe30 msgr2=0x55d343bc9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f448b7fe640 1 --2- 172.21.15.27:0/4201150891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bcbe30 0x55d343bc9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f448b7fe640 1 -- 172.21.15.27:0/4201150891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bc1760 msgr2=0x55d343bc9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f448b7fe640 1 --2- 172.21.15.27:0/4201150891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bc1760 0x55d343bc9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f448b7fe640 1 -- 172.21.15.27:0/4201150891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3438313f0 con 0x55d343bc0e40 2024-04-25T00:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f448b7fe640 1 --2- 172.21.15.27:0/4201150891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 0x55d343bc8dc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f447c000c00 tx=0x7f447c0cbdd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16ae525ac288b839 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f446affd640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f446affd640 1 -- 172.21.15.27:0/4201150891 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c01e070 con 0x55d343bc0e40 2024-04-25T00:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f446affd640 1 -- 172.21.15.27:0/4201150891 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f447c017360 con 0x55d343bc0e40 2024-04-25T00:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.024+0000 7f4494c30740 1 -- 172.21.15.27:0/4201150891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f447c06fa70 con 0x55d343bc0e40 2024-04-25T00:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.025+0000 7f446affd640 1 -- 172.21.15.27:0/4201150891 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f447c0dd070 con 0x55d343bc0e40 2024-04-25T00:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.025+0000 7f4494c30740 10 client.26137 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.025+0000 7f4494c30740 1 -- 172.21.15.27:0/4201150891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55d3437f10f0 con 0x55d343bc0e40 2024-04-25T00:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.025+0000 7f446affd640 1 -- 172.21.15.27:0/4201150891 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f447c0b2070 con 0x55d343bc0e40 2024-04-25T00:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.025+0000 7f446affd640 1 -- 172.21.15.27:0/4201150891 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f447c0cbf50 con 0x55d343bc0e40 2024-04-25T00:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.025+0000 7f4494c30740 10 client.26137 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.025+0000 7f4494c30740 10 client.26137 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.025+0000 7f4494c30740 20 client.26137 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.025+0000 7f4494c30740 1 --2- 172.21.15.27:0/4201150891 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55d343bdbf90 0x55d343bde380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.025+0000 7f4494c30740 4 client.26137 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.025+0000 7f4494c30740 1 -- 172.21.15.27:0/4201150891 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d343b2abf0 con 0x55d343bdbf90 2024-04-25T00:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.025+0000 7f448affd640 1 --2- 172.21.15.27:0/4201150891 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55d343bdbf90 0x55d343bde380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.026+0000 7f446affd640 10 client.26137 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.026+0000 7f448affd640 1 --2- 172.21.15.27:0/4201150891 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55d343bdbf90 0x55d343bde380 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de2fd47d3393192d server_cookie=f459ad8a05bd94ab in_seq=0 out_seq=0 2024-04-25T00:23:22.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.027+0000 7f446affd640 1 -- 172.21.15.27:0/4201150891 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d343b2abf0 con 0x55d343bdbf90 2024-04-25T00:23:22.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.027+0000 7f446affd640 10 client.26137 handle_command_reply: tid=0 2024-04-25T00:23:22.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.027+0000 7f4494c30740 1 client.26137 shutdown 2024-04-25T00:23:22.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.027+0000 7f4494c30740 1 -- 172.21.15.27:0/4201150891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 msgr2=0x55d343bc8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:22.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.027+0000 7f4494c30740 1 --2- 172.21.15.27:0/4201150891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 0x55d343bc8dc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f447c000c00 tx=0x7f447c0cbdd0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.027+0000 7f4494c30740 1 -- 172.21.15.27:0/4201150891 shutdown_connections 2024-04-25T00:23:22.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.027+0000 7f4494c30740 1 --2- 172.21.15.27:0/4201150891 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55d343bdbf90 0x55d343bde380 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.027+0000 7f4494c30740 1 --2- 172.21.15.27:0/4201150891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bcbe30 0x55d343bc9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.027+0000 7f446affd640 0 client.26137 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:22.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.027+0000 7f4494c30740 1 --2- 172.21.15.27:0/4201150891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bc1760 0x55d343bc9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.027+0000 7f4494c30740 1 --2- 172.21.15.27:0/4201150891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 0x55d343bc8dc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.027+0000 7f4494c30740 1 -- 172.21.15.27:0/4201150891 >> 172.21.15.27:0/4201150891 conn(0x55d343bbd160 msgr2=0x55d343bb3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:22.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.027+0000 7f4494c30740 1 -- 172.21.15.27:0/4201150891 shutdown_connections 2024-04-25T00:23:22.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.027+0000 7f4494c30740 1 -- 172.21.15.27:0/4201150891 wait complete. 2024-04-25T00:23:22.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.027+0000 7f4494c30740 20 client.26137 trim_cache size 0 max 16384 2024-04-25T00:23:22.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.057+0000 7f4494c30740 1 Processor -- start 2024-04-25T00:23:22.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.057+0000 7f4494c30740 1 -- start start 2024-04-25T00:23:22.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.057+0000 7f4494c30740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 0x55d343bc86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.057+0000 7f4494c30740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bc1760 0x55d343bc3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.057+0000 7f4494c30740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bcbe30 0x55d343bc9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.057+0000 7f4494c30740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f448c1b5320 con 0x55d343bcbe30 2024-04-25T00:23:22.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.057+0000 7f4494c30740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f448c1b6ec0 con 0x55d343bc1760 2024-04-25T00:23:22.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.057+0000 7f4494c30740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f448c1b59f0 con 0x55d343bc0e40 2024-04-25T00:23:22.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.058+0000 7f448bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bcbe30 0x55d343bc9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.058+0000 7f448affd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bc1760 0x55d343bc3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.058+0000 7f448bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bcbe30 0x55d343bc9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49516/0 (socket says 172.21.15.27:49516) 2024-04-25T00:23:22.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.058+0000 7f448bfff640 1 -- 172.21.15.27:0/2780494544 learned_addr learned my addr 172.21.15.27:0/2780494544 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:22.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.058+0000 7f448b7fe640 1 --2- 172.21.15.27:0/2780494544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 0x55d343bc86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.058+0000 7f448bfff640 1 -- 172.21.15.27:0/2780494544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 msgr2=0x55d343bc86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:22.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.058+0000 7f448bfff640 1 --2- 172.21.15.27:0/2780494544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 0x55d343bc86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.058+0000 7f448bfff640 1 -- 172.21.15.27:0/2780494544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bc1760 msgr2=0x55d343bc3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:22.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.058+0000 7f448bfff640 1 --2- 172.21.15.27:0/2780494544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bc1760 0x55d343bc3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.058+0000 7f448bfff640 1 -- 172.21.15.27:0/2780494544 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d343bc7310 con 0x55d343bcbe30 2024-04-25T00:23:22.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.058+0000 7f448affd640 1 --2- 172.21.15.27:0/2780494544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bc1760 0x55d343bc3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:22.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.058+0000 7f448bfff640 1 --2- 172.21.15.27:0/2780494544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bcbe30 0x55d343bc9490 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4484009ca0 tx=0x7f4484043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7bb5f7cde0908ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:22.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.058+0000 7f44717fa640 1 -- 172.21.15.27:0/2780494544 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4484057070 con 0x55d343bcbe30 2024-04-25T00:23:22.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.059+0000 7f44717fa640 1 -- 172.21.15.27:0/2780494544 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f448404d050 con 0x55d343bcbe30 2024-04-25T00:23:22.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.059+0000 7f44717fa640 1 -- 172.21.15.27:0/2780494544 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4484033040 con 0x55d343bcbe30 2024-04-25T00:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.060+0000 7f4494c30740 1 -- 172.21.15.27:0/2780494544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bcbe30 msgr2=0x55d343bc9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.060+0000 7f4494c30740 1 --2- 172.21.15.27:0/2780494544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bcbe30 0x55d343bc9490 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4484009ca0 tx=0x7f4484043000 comp rx=0 tx=0).stop 2024-04-25T00:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.060+0000 7f4494c30740 1 -- 172.21.15.27:0/2780494544 shutdown_connections 2024-04-25T00:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.060+0000 7f4494c30740 1 --2- 172.21.15.27:0/2780494544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bcbe30 0x55d343bc9490 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.060+0000 7f4494c30740 1 --2- 172.21.15.27:0/2780494544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bc1760 0x55d343bc3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.060+0000 7f4494c30740 1 --2- 172.21.15.27:0/2780494544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 0x55d343bc86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.060+0000 7f4494c30740 1 -- 172.21.15.27:0/2780494544 >> 172.21.15.27:0/2780494544 conn(0x55d343bbd160 msgr2=0x55d343bc8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.060+0000 7f4494c30740 1 -- 172.21.15.27:0/2780494544 shutdown_connections 2024-04-25T00:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.060+0000 7f4494c30740 1 -- 172.21.15.27:0/2780494544 wait complete. 2024-04-25T00:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.060+0000 7f4494c30740 1 -- start start 2024-04-25T00:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.060+0000 7f4494c30740 1 Processor -- start 2024-04-25T00:23:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.061+0000 7f4494c30740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 0x55d343c3a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.061+0000 7f4494c30740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bc1760 0x55d343c3ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.061+0000 7f4494c30740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bcbe30 0x55d343bcf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.061+0000 7f448b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 0x55d343c3a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.061+0000 7f448b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 0x55d343c3a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:56334/0 (socket says 172.21.15.27:56334) 2024-04-25T00:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.061+0000 7f448affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bc1760 0x55d343c3ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.061+0000 7f448b7fe640 1 -- 172.21.15.27:0/2581656497 learned_addr learned my addr 172.21.15.27:0/2581656497 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.061+0000 7f4494c30740 1 -- 172.21.15.27:0/2581656497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f448c1b67f0 con 0x55d343bc1760 2024-04-25T00:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.061+0000 7f4494c30740 1 -- 172.21.15.27:0/2581656497 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f448c1b60f0 con 0x55d343bcbe30 2024-04-25T00:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.061+0000 7f4494c30740 1 -- 172.21.15.27:0/2581656497 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f448c1b32f0 con 0x55d343bc0e40 2024-04-25T00:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.062+0000 7f4494c30740 10 client.0 _ll_register_callbacks cb 0x7f4484005eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.062+0000 7f448b7fe640 1 -- 172.21.15.27:0/2581656497 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bcbe30 msgr2=0x55d343bcf3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.062+0000 7f448b7fe640 1 --2- 172.21.15.27:0/2581656497 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bcbe30 0x55d343bcf3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.062+0000 7f448b7fe640 1 -- 172.21.15.27:0/2581656497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bc1760 msgr2=0x55d343c3ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.062+0000 7f448b7fe640 1 --2- 172.21.15.27:0/2581656497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bc1760 0x55d343c3ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.062+0000 7f448b7fe640 1 -- 172.21.15.27:0/2581656497 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d343bcfa90 con 0x55d343bc0e40 2024-04-25T00:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.062+0000 7f448b7fe640 1 --2- 172.21.15.27:0/2581656497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 0x55d343c3a5d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f447c09a960 tx=0x7f447c0d4300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d876f6d472b58574 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.062+0000 7f4494c30740 1 -- 172.21.15.27:0/2581656497 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f448404d050 con 0x55d343bc0e40 2024-04-25T00:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.062+0000 7f446affd640 10 client.26143 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T00:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.062+0000 7f446affd640 1 -- 172.21.15.27:0/2581656497 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c01e070 con 0x55d343bc0e40 2024-04-25T00:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.063+0000 7f446affd640 1 -- 172.21.15.27:0/2581656497 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f447c017360 con 0x55d343bc0e40 2024-04-25T00:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.063+0000 7f446affd640 1 -- 172.21.15.27:0/2581656497 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f447c09a5c0 con 0x55d343bc0e40 2024-04-25T00:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.063+0000 7f4494c30740 10 client.26143 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.063+0000 7f4494c30740 1 -- 172.21.15.27:0/2581656497 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({fsmap=97}) -- 0x55d343bcfa90 con 0x55d343bc0e40 2024-04-25T00:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.063+0000 7f446affd640 1 -- 172.21.15.27:0/2581656497 <== mon.1 v2:172.21.15.153:3300/0 4 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f447c0a3390 con 0x55d343bc0e40 2024-04-25T00:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.064+0000 7f4494c30740 10 client.26143 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.064+0000 7f4494c30740 10 client.26143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.064+0000 7f4494c30740 20 client.26143 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.064+0000 7f4494c30740 1 --2- 172.21.15.27:0/2581656497 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55d343bdbf90 0x55d343c43d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.064+0000 7f4494c30740 4 client.26143 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:23:22.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.064+0000 7f4494c30740 1 -- 172.21.15.27:0/2581656497 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d343b24f40 con 0x55d343bdbf90 2024-04-25T00:23:22.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.067+0000 7f448affd640 1 --2- 172.21.15.27:0/2581656497 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55d343bdbf90 0x55d343c43d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.069+0000 7f446affd640 10 client.26143 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.069+0000 7f448affd640 1 --2- 172.21.15.27:0/2581656497 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55d343bdbf90 0x55d343c43d60 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d45657410f9ee615 server_cookie=3ad6ecfbfd66347f in_seq=0 out_seq=0 2024-04-25T00:23:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.070+0000 7f446affd640 1 -- 172.21.15.27:0/2581656497 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d343b24f40 con 0x55d343bdbf90 2024-04-25T00:23:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.070+0000 7f446affd640 10 client.26143 handle_command_reply: tid=0 2024-04-25T00:23:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.070+0000 7f4494c30740 1 client.26143 shutdown 2024-04-25T00:23:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.070+0000 7f4494c30740 1 -- 172.21.15.27:0/2581656497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 msgr2=0x55d343c3a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.070+0000 7f4494c30740 1 --2- 172.21.15.27:0/2581656497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 0x55d343c3a5d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f447c09a960 tx=0x7f447c0d4300 comp rx=0 tx=0).stop 2024-04-25T00:23:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.070+0000 7f4494c30740 1 -- 172.21.15.27:0/2581656497 shutdown_connections 2024-04-25T00:23:22.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.071+0000 7f4494c30740 1 --2- 172.21.15.27:0/2581656497 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55d343bdbf90 0x55d343c43d60 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.071+0000 7f4494c30740 1 --2- 172.21.15.27:0/2581656497 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d343bcbe30 0x55d343bcf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.071+0000 7f4494c30740 1 --2- 172.21.15.27:0/2581656497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d343bc1760 0x55d343c3ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.071+0000 7f4494c30740 1 --2- 172.21.15.27:0/2581656497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d343bc0e40 0x55d343c3a5d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.071+0000 7f4494c30740 1 -- 172.21.15.27:0/2581656497 >> 172.21.15.27:0/2581656497 conn(0x55d343bbd160 msgr2=0x55d343bc9610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:22.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.071+0000 7f446affd640 0 client.26143 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:22.074 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:23:22.074 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T00:23:22.074 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:23:22.074 INFO:teuthology.orchestra.run.smithi027.stdout: "1c7b0f05-8274-46ba-adc8-928f834ed859": { 2024-04-25T00:23:22.074 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:23:22.074 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "1c7b0f05-8274-46ba-adc8-928f834ed859", 2024-04-25T00:23:22.074 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:23:22.075 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:23:22.075 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:23:22.075 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:23:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.072+0000 7f4494c30740 1 -- 172.21.15.27:0/2581656497 shutdown_connections 2024-04-25T00:23:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.072+0000 7f4494c30740 1 -- 172.21.15.27:0/2581656497 wait complete. 2024-04-25T00:23:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.073+0000 7f4494c30740 20 client.26143 trim_cache size 0 max 16384 2024-04-25T00:23:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.074+0000 7f4491e4c640 1 -- 172.21.15.27:0/3756398763 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f44740a2d80 msgr2=0x7f44740a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.074+0000 7f4491e4c640 1 --2- 172.21.15.27:0/3756398763 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f44740a2d80 0x7f44740a5240 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4484009430 tx=0x7f4484009800 comp rx=0 tx=0).stop 2024-04-25T00:23:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.074+0000 7f4491e4c640 1 -- 172.21.15.27:0/3756398763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c1029c0 msgr2=0x7f448c072fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.074+0000 7f4491e4c640 1 --2- 172.21.15.27:0/3756398763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c1029c0 0x7f448c072fa0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f447c00b4a0 tx=0x7f447c00b960 comp rx=0 tx=0).stop 2024-04-25T00:23:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.075+0000 7f4491e4c640 1 -- 172.21.15.27:0/3756398763 shutdown_connections 2024-04-25T00:23:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.075+0000 7f4491e4c640 1 --2- 172.21.15.27:0/3756398763 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f44740a2d80 0x7f44740a5240 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.075+0000 7f4491e4c640 1 --2- 172.21.15.27:0/3756398763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f448c10bd00 0x7f448c06c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.075+0000 7f4491e4c640 1 --2- 172.21.15.27:0/3756398763 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f448c1032e0 0x7f448c06c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.075+0000 7f4491e4c640 1 --2- 172.21.15.27:0/3756398763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c1029c0 0x7f448c072fa0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.075+0000 7f4491e4c640 1 -- 172.21.15.27:0/3756398763 >> 172.21.15.27:0/3756398763 conn(0x7f448c077740 msgr2=0x7f448c0fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.076+0000 7f4491e4c640 1 -- 172.21.15.27:0/3756398763 shutdown_connections 2024-04-25T00:23:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:22.076+0000 7f4491e4c640 1 -- 172.21.15.27:0/3756398763 wait complete. 2024-04-25T00:23:22.093 INFO:tasks.cephfs.filesystem:scrub status for tag:1c7b0f05-8274-46ba-adc8-928f834ed859 - {'path': '/', 'tag': '1c7b0f05-8274-46ba-adc8-928f834ed859', 'options': 'recursive,force'} 2024-04-25T00:23:22.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:22 smithi027 ceph-mon[29208]: pgmap v231: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 159 KiB/s wr, 145 op/s 2024-04-25T00:23:22.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:22 smithi139 ceph-mon[32833]: pgmap v231: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 159 KiB/s wr, 145 op/s 2024-04-25T00:23:22.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:22 smithi153 ceph-mon[32896]: pgmap v231: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 159 KiB/s wr, 145 op/s 2024-04-25T00:23:23.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:23 smithi027 ceph-mon[29208]: scrub queued for path: / 2024-04-25T00:23:23.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:23 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:23:23.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:23 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:23:23.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:23 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:23:23.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:23 smithi139 ceph-mon[32833]: scrub queued for path: / 2024-04-25T00:23:23.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:23 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:23:23.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:23 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:23:23.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:23 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:23:23.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:23 smithi153 ceph-mon[32896]: scrub queued for path: / 2024-04-25T00:23:23.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:23 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:23:23.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:23 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:23:23.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:23 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:23:24.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:24 smithi027 ceph-mon[29208]: pgmap v232: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 124 op/s 2024-04-25T00:23:24.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:24 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:23:24.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:24 smithi139 ceph-mon[32833]: pgmap v232: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 124 op/s 2024-04-25T00:23:24.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:24 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:23:24.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:24 smithi153 ceph-mon[32896]: pgmap v232: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 124 op/s 2024-04-25T00:23:24.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:24 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:23:26.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:26 smithi027 ceph-mon[29208]: pgmap v233: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 141 op/s 2024-04-25T00:23:26.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:26 smithi139 ceph-mon[32833]: pgmap v233: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 141 op/s 2024-04-25T00:23:26.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:26 smithi153 ceph-mon[32896]: pgmap v233: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 141 op/s 2024-04-25T00:23:28.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:28 smithi027 ceph-mon[29208]: pgmap v234: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 3.8 KiB/s wr, 110 op/s 2024-04-25T00:23:28.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:28 smithi139 ceph-mon[32833]: pgmap v234: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 3.8 KiB/s wr, 110 op/s 2024-04-25T00:23:28.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:28 smithi153 ceph-mon[32896]: pgmap v234: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 3.8 KiB/s wr, 110 op/s 2024-04-25T00:23:30.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:30 smithi027 ceph-mon[29208]: pgmap v235: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.8 KiB/s wr, 125 op/s 2024-04-25T00:23:30.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:30 smithi139 ceph-mon[32833]: pgmap v235: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.8 KiB/s wr, 125 op/s 2024-04-25T00:23:30.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:30 smithi153 ceph-mon[32896]: pgmap v235: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.8 KiB/s wr, 125 op/s 2024-04-25T00:23:33.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:32 smithi139 ceph-mon[32833]: pgmap v236: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 3.5 KiB/s wr, 128 op/s 2024-04-25T00:23:33.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:32 smithi153 ceph-mon[32896]: pgmap v236: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 3.5 KiB/s wr, 128 op/s 2024-04-25T00:23:33.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:32 smithi027 ceph-mon[29208]: pgmap v236: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 3.5 KiB/s wr, 128 op/s 2024-04-25T00:23:35.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:34 smithi139 ceph-mon[32833]: pgmap v237: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 426 B/s wr, 118 op/s 2024-04-25T00:23:35.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:34 smithi153 ceph-mon[32896]: pgmap v237: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 426 B/s wr, 118 op/s 2024-04-25T00:23:35.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:34 smithi027 ceph-mon[29208]: pgmap v237: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 426 B/s wr, 118 op/s 2024-04-25T00:23:37.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:36 smithi139 ceph-mon[32833]: pgmap v238: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 134 op/s 2024-04-25T00:23:37.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:36 smithi153 ceph-mon[32896]: pgmap v238: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 134 op/s 2024-04-25T00:23:37.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:36 smithi027 ceph-mon[29208]: pgmap v238: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 134 op/s 2024-04-25T00:23:38.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:38 smithi027 ceph-mon[29208]: pgmap v239: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 109 op/s 2024-04-25T00:23:39.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:38 smithi139 ceph-mon[32833]: pgmap v239: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 109 op/s 2024-04-25T00:23:39.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:38 smithi153 ceph-mon[32896]: pgmap v239: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 109 op/s 2024-04-25T00:23:41.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:40 smithi139 ceph-mon[32833]: pgmap v240: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 108 op/s 2024-04-25T00:23:41.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:40 smithi153 ceph-mon[32896]: pgmap v240: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 108 op/s 2024-04-25T00:23:41.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:40 smithi027 ceph-mon[29208]: pgmap v240: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 108 op/s 2024-04-25T00:23:43.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:42 smithi139 ceph-mon[32833]: pgmap v241: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 95 op/s 2024-04-25T00:23:43.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:42 smithi153 ceph-mon[32896]: pgmap v241: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 95 op/s 2024-04-25T00:23:43.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:42 smithi027 ceph-mon[29208]: pgmap v241: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 95 op/s 2024-04-25T00:23:45.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:44 smithi139 ceph-mon[32833]: pgmap v242: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 71 op/s 2024-04-25T00:23:45.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:44 smithi153 ceph-mon[32896]: pgmap v242: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 71 op/s 2024-04-25T00:23:45.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:44 smithi027 ceph-mon[29208]: pgmap v242: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 71 op/s 2024-04-25T00:23:47.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:46 smithi139 ceph-mon[32833]: pgmap v243: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 59 op/s 2024-04-25T00:23:47.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:46 smithi153 ceph-mon[32896]: pgmap v243: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 59 op/s 2024-04-25T00:23:47.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:46 smithi027 ceph-mon[29208]: pgmap v243: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 59 op/s 2024-04-25T00:23:49.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:48 smithi139 ceph-mon[32833]: pgmap v244: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 33 op/s 2024-04-25T00:23:49.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:48 smithi153 ceph-mon[32896]: pgmap v244: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 33 op/s 2024-04-25T00:23:49.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:48 smithi027 ceph-mon[29208]: pgmap v244: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 33 op/s 2024-04-25T00:23:51.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:50 smithi139 ceph-mon[32833]: pgmap v245: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 25 op/s 2024-04-25T00:23:51.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:50 smithi153 ceph-mon[32896]: pgmap v245: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 25 op/s 2024-04-25T00:23:51.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:50 smithi027 ceph-mon[29208]: pgmap v245: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 25 op/s 2024-04-25T00:23:52.094 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:23:52.094 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:23:52.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.211+0000 7f4c090bd640 1 -- 172.21.15.27:0/47416787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c04106630 msgr2=0x7f4c04106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.211+0000 7f4c090bd640 1 --2- 172.21.15.27:0/47416787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c04106630 0x7f4c04106a10 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4009930 tx=0x7f4bf402a420 comp rx=0 tx=0).stop 2024-04-25T00:23:52.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.211+0000 7f4c090bd640 1 -- 172.21.15.27:0/47416787 shutdown_connections 2024-04-25T00:23:52.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.211+0000 7f4c090bd640 1 --2- 172.21.15.27:0/47416787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c04100fe0 0x7f4c0410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.211+0000 7f4c090bd640 1 --2- 172.21.15.27:0/47416787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c04100620 0x7f4c04100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.211+0000 7f4c090bd640 1 --2- 172.21.15.27:0/47416787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c04106630 0x7f4c04106a10 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.211+0000 7f4c090bd640 1 -- 172.21.15.27:0/47416787 >> 172.21.15.27:0/47416787 conn(0x7f4c040776a0 msgr2=0x7f4c04077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.211+0000 7f4c090bd640 1 -- 172.21.15.27:0/47416787 shutdown_connections 2024-04-25T00:23:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.211+0000 7f4c090bd640 1 -- 172.21.15.27:0/47416787 wait complete. 2024-04-25T00:23:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.212+0000 7f4c090bd640 1 Processor -- start 2024-04-25T00:23:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.212+0000 7f4c090bd640 1 -- start start 2024-04-25T00:23:52.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.212+0000 7f4c090bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c04100620 0x7f4c040fd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.212+0000 7f4c090bd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c04100fe0 0x7f4c040fdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4c090bd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c04106630 0x7f4c040fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4c090bd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4c04110a50 con 0x7f4c04100fe0 2024-04-25T00:23:52.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4c090bd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041108d0 con 0x7f4c04106630 2024-04-25T00:23:52.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4c090bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4c04110bd0 con 0x7f4c04100620 2024-04-25T00:23:52.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4bfbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c04100fe0 0x7f4c040fdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4bfbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c04100fe0 0x7f4c040fdb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49748/0 (socket says 172.21.15.27:49748) 2024-04-25T00:23:52.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4bfbfff640 1 -- 172.21.15.27:0/2281776636 learned_addr learned my addr 172.21.15.27:0/2281776636 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:52.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4c0359e640 1 --2- 172.21.15.27:0/2281776636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c04106630 0x7f4c040fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4c02d9d640 1 --2- 172.21.15.27:0/2281776636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c04100620 0x7f4c040fd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4bfbfff640 1 -- 172.21.15.27:0/2281776636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c04100620 msgr2=0x7f4c040fd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4bfbfff640 1 --2- 172.21.15.27:0/2281776636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c04100620 0x7f4c040fd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4bfbfff640 1 -- 172.21.15.27:0/2281776636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c04106630 msgr2=0x7f4c040fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4bfbfff640 1 --2- 172.21.15.27:0/2281776636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c04106630 0x7f4c040fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4bfbfff640 1 -- 172.21.15.27:0/2281776636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c041a6520 con 0x7f4c04100fe0 2024-04-25T00:23:52.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4c0359e640 1 --2- 172.21.15.27:0/2281776636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c04106630 0x7f4c040fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:52.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4bfbfff640 1 --2- 172.21.15.27:0/2281776636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c04100fe0 0x7f4c040fdb90 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f4bec00e9f0 tx=0x7f4bec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4c00d99640 1 -- 172.21.15.27:0/2281776636 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec00cd80 con 0x7f4c04100fe0 2024-04-25T00:23:52.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4c090bd640 1 -- 172.21.15.27:0/2281776636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c041a6810 con 0x7f4c04100fe0 2024-04-25T00:23:52.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4c00d99640 1 -- 172.21.15.27:0/2281776636 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bec010910 con 0x7f4c04100fe0 2024-04-25T00:23:52.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.213+0000 7f4c00d99640 1 -- 172.21.15.27:0/2281776636 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec0188f0 con 0x7f4c04100fe0 2024-04-25T00:23:52.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.214+0000 7f4c090bd640 1 -- 172.21.15.27:0/2281776636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c041a6d60 con 0x7f4c04100fe0 2024-04-25T00:23:52.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.216+0000 7f4c00d99640 1 -- 172.21.15.27:0/2281776636 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4bec010ac0 con 0x7f4c04100fe0 2024-04-25T00:23:52.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.216+0000 7f4c0bdd4740 1 Processor -- start 2024-04-25T00:23:52.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.216+0000 7f4c0bdd4740 1 -- start start 2024-04-25T00:23:52.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c0bdd4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90630 0x56001cf90a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c0bdd4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf90f50 0x56001cf9b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c0bdd4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf9b620 0x56001cf9da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c0bdd4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56001cef56b0 con 0x56001cf90f50 2024-04-25T00:23:52.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c0bdd4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56001ce9d880 con 0x56001cf90630 2024-04-25T00:23:52.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c0bdd4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56001cef1670 con 0x56001cf9b620 2024-04-25T00:23:52.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c00d99640 1 --2- 172.21.15.27:0/2281776636 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4bf00a2d00 0x7f4bf00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c02d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf90f50 0x56001cf9b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c02d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf90f50 0x56001cf9b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49764/0 (socket says 172.21.15.27:49764) 2024-04-25T00:23:52.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c02d9d640 1 -- 172.21.15.27:0/2814421120 learned_addr learned my addr 172.21.15.27:0/2814421120 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:52.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c0359e640 1 --2- 172.21.15.27:0/2814421120 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90630 0x56001cf90a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c0359e640 1 --2- 172.21.15.27:0/2281776636 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4bf00a2d00 0x7f4bf00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c00d99640 1 -- 172.21.15.27:0/2281776636 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f4bec0cfbc0 con 0x7f4c04100fe0 2024-04-25T00:23:52.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4bfbfff640 1 --2- 172.21.15.27:0/2814421120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf9b620 0x56001cf9da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c02d9d640 1 -- 172.21.15.27:0/2814421120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf9b620 msgr2=0x56001cf9da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c02d9d640 1 --2- 172.21.15.27:0/2814421120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf9b620 0x56001cf9da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c02d9d640 1 -- 172.21.15.27:0/2814421120 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90630 msgr2=0x56001cf90a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c02d9d640 1 --2- 172.21.15.27:0/2814421120 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90630 0x56001cf90a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c02d9d640 1 -- 172.21.15.27:0/2814421120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56001ccd37d0 con 0x56001cf90f50 2024-04-25T00:23:52.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c02d9d640 1 --2- 172.21.15.27:0/2814421120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf90f50 0x56001cf9b0e0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4009930 tx=0x7f4bf4005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40525dc53dcc3285 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.217+0000 7f4c0359e640 1 --2- 172.21.15.27:0/2281776636 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4bf00a2d00 0x7f4bf00a51c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009fd0 tx=0x7f4bfc0096a0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.218+0000 7f4bf97fa640 1 -- 172.21.15.27:0/2814421120 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4035040 con 0x56001cf90f50 2024-04-25T00:23:52.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.218+0000 7f4bf97fa640 1 -- 172.21.15.27:0/2814421120 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bf4004480 con 0x56001cf90f50 2024-04-25T00:23:52.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.218+0000 7f4bf97fa640 1 -- 172.21.15.27:0/2814421120 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf40035f0 con 0x56001cf90f50 2024-04-25T00:23:52.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.218+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/2814421120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf90f50 msgr2=0x56001cf9b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.218+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/2814421120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf90f50 0x56001cf9b0e0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4009930 tx=0x7f4bf4005bf0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.218+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/2814421120 shutdown_connections 2024-04-25T00:23:52.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.218+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/2814421120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf9b620 0x56001cf9da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.218+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/2814421120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf90f50 0x56001cf9b0e0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.218+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/2814421120 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90630 0x56001cf90a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.218+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/2814421120 >> 172.21.15.27:0/2814421120 conn(0x56001cf8c950 msgr2=0x56001cf8cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.219+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/2814421120 shutdown_connections 2024-04-25T00:23:52.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.219+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/2814421120 wait complete. 2024-04-25T00:23:52.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.219+0000 7f4c0bdd4740 1 -- start start 2024-04-25T00:23:52.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.219+0000 7f4c0bdd4740 1 Processor -- start 2024-04-25T00:23:52.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.219+0000 7f4c0bdd4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf90630 0x56001cf98560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.219+0000 7f4c0bdd4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90f50 0x56001cf98aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.219+0000 7f4c0bdd4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf9b620 0x56001cf98fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.219+0000 7f4c0bdd4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56001cf9e3b0 con 0x56001cf9b620 2024-04-25T00:23:52.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.219+0000 7f4c0bdd4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56001ceb0f50 con 0x56001cf90f50 2024-04-25T00:23:52.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.219+0000 7f4c0bdd4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56001cf9e530 con 0x56001cf90630 2024-04-25T00:23:52.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.219+0000 7f4c0bdd4740 10 client.0 _ll_register_callbacks cb 0x56001ceec000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:52.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.219+0000 7f4c0359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf9b620 0x56001cf98fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.219+0000 7f4c0359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf9b620 0x56001cf98fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49778/0 (socket says 172.21.15.27:49778) 2024-04-25T00:23:52.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.219+0000 7f4c0359e640 1 -- 172.21.15.27:0/2201932328 learned_addr learned my addr 172.21.15.27:0/2201932328 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:52.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.220+0000 7f4bfbfff640 1 --2- 172.21.15.27:0/2201932328 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90f50 0x56001cf98aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.220+0000 7f4c02d9d640 1 --2- 172.21.15.27:0/2201932328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf90630 0x56001cf98560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.220+0000 7f4c0359e640 1 -- 172.21.15.27:0/2201932328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf90630 msgr2=0x56001cf98560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.220+0000 7f4c0359e640 1 --2- 172.21.15.27:0/2201932328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf90630 0x56001cf98560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.220+0000 7f4c0359e640 1 -- 172.21.15.27:0/2201932328 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90f50 msgr2=0x56001cf98aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.220+0000 7f4c0359e640 1 --2- 172.21.15.27:0/2201932328 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90f50 0x56001cf98aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.220+0000 7f4c0359e640 1 -- 172.21.15.27:0/2201932328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56001cc093f0 con 0x56001cf9b620 2024-04-25T00:23:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.220+0000 7f4bfbfff640 1 --2- 172.21.15.27:0/2201932328 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90f50 0x56001cf98aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:23:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.220+0000 7f4c0359e640 1 --2- 172.21.15.27:0/2201932328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf9b620 0x56001cf98fe0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00bfa0 tx=0x7f4bfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d643d84aaae14d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.220+0000 7f4bea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.220+0000 7f4bea7fc640 1 -- 172.21.15.27:0/2201932328 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc004b20 con 0x56001cf9b620 2024-04-25T00:23:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.220+0000 7f4bea7fc640 1 -- 172.21.15.27:0/2201932328 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f4bfc004cc0 con 0x56001cf9b620 2024-04-25T00:23:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.220+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/2201932328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bf4004480 con 0x56001cf9b620 2024-04-25T00:23:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.220+0000 7f4bea7fc640 1 -- 172.21.15.27:0/2201932328 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f4bfc01e600 con 0x56001cf9b620 2024-04-25T00:23:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.220+0000 7f4c0bdd4740 10 client.16677 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.220+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/2201932328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x56001cbc90f0 con 0x56001cf9b620 2024-04-25T00:23:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.220+0000 7f4bea7fc640 1 -- 172.21.15.27:0/2201932328 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bf4004480 con 0x56001cf9b620 2024-04-25T00:23:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.221+0000 7f4bea7fc640 1 -- 172.21.15.27:0/2201932328 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f4bfc005490 con 0x56001cf9b620 2024-04-25T00:23:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.221+0000 7f4c0bdd4740 10 client.16677 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.221+0000 7f4c0bdd4740 10 client.16677 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:23:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.221+0000 7f4c0bdd4740 20 client.16677 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.221+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/2201932328 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56001cfab900 0x56001cfadd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.221+0000 7f4c0bdd4740 4 client.16677 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:23:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.221+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/2201932328 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56001cf02bf0 con 0x56001cfab900 2024-04-25T00:23:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.221+0000 7f4c02d9d640 1 --2- 172.21.15.27:0/2201932328 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56001cfab900 0x56001cfadd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.222+0000 7f4c02d9d640 1 --2- 172.21.15.27:0/2201932328 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56001cfab900 0x56001cfadd50 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76ae449efb7ef4f1 server_cookie=3cb97eecf992b918 in_seq=0 out_seq=0 2024-04-25T00:23:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.222+0000 7f4bea7fc640 10 client.16677 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.222+0000 7f4bea7fc640 1 -- 172.21.15.27:0/2201932328 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56001cf02bf0 con 0x56001cfab900 2024-04-25T00:23:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.222+0000 7f4bea7fc640 10 client.16677 handle_command_reply: tid=0 2024-04-25T00:23:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.222+0000 7f4c0bdd4740 1 client.16677 shutdown 2024-04-25T00:23:52.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.223+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/2201932328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf9b620 msgr2=0x56001cf98fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.223+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/2201932328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf9b620 0x56001cf98fe0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00bfa0 tx=0x7f4bfc004a40 comp rx=0 tx=0).stop 2024-04-25T00:23:52.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.223+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/2201932328 shutdown_connections 2024-04-25T00:23:52.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.223+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/2201932328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf9b620 0x56001cf98fe0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.223+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/2201932328 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56001cfab900 0x56001cfadd50 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.223+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/2201932328 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90f50 0x56001cf98aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.223+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/2201932328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf90630 0x56001cf98560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.223+0000 7f4bea7fc640 0 client.16677 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:52.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.223+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/2201932328 >> 172.21.15.27:0/2201932328 conn(0x56001cf8c950 msgr2=0x56001cf92750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.223+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/2201932328 shutdown_connections 2024-04-25T00:23:52.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.223+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/2201932328 wait complete. 2024-04-25T00:23:52.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.223+0000 7f4c0bdd4740 20 client.16677 trim_cache size 0 max 16384 2024-04-25T00:23:52.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.240+0000 7f4c0bdd4740 1 Processor -- start 2024-04-25T00:23:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.240+0000 7f4c0bdd4740 1 -- start start 2024-04-25T00:23:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4c0bdd4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf90630 0x56001cfa7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4c0bdd4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90f50 0x56001cf92dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4c0bdd4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf9b620 0x56001cf8b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4c0bdd4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b0ea0 con 0x56001cf9b620 2024-04-25T00:23:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4c0bdd4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b2a40 con 0x56001cf90f50 2024-04-25T00:23:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4c0bdd4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b1570 con 0x56001cf90630 2024-04-25T00:23:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4c0359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf9b620 0x56001cf8b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4c0359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf9b620 0x56001cf8b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49788/0 (socket says 172.21.15.27:49788) 2024-04-25T00:23:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4c0359e640 1 -- 172.21.15.27:0/3423188499 learned_addr learned my addr 172.21.15.27:0/3423188499 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4bfbfff640 1 --2- 172.21.15.27:0/3423188499 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90f50 0x56001cf92dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4c02d9d640 1 --2- 172.21.15.27:0/3423188499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf90630 0x56001cfa7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4c0359e640 1 -- 172.21.15.27:0/3423188499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf90630 msgr2=0x56001cfa7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4c0359e640 1 --2- 172.21.15.27:0/3423188499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf90630 0x56001cfa7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4c0359e640 1 -- 172.21.15.27:0/3423188499 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90f50 msgr2=0x56001cf92dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4c0359e640 1 --2- 172.21.15.27:0/3423188499 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90f50 0x56001cf92dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4c0359e640 1 -- 172.21.15.27:0/3423188499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56001cf7e090 con 0x56001cf9b620 2024-04-25T00:23:52.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4c02d9d640 1 --2- 172.21.15.27:0/3423188499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf90630 0x56001cfa7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:23:52.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.241+0000 7f4c0359e640 1 --2- 172.21.15.27:0/3423188499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf9b620 0x56001cf8b970 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc006010 tx=0x7f4bfc0125e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07bc40c791c2565 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.242+0000 7f4bf8ff9640 1 -- 172.21.15.27:0/3423188499 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc012d30 con 0x56001cf9b620 2024-04-25T00:23:52.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.242+0000 7f4bf8ff9640 1 -- 172.21.15.27:0/3423188499 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bfc0120b0 con 0x56001cf9b620 2024-04-25T00:23:52.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.242+0000 7f4bf8ff9640 1 -- 172.21.15.27:0/3423188499 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc0379b0 con 0x56001cf9b620 2024-04-25T00:23:52.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.242+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/3423188499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf9b620 msgr2=0x56001cf8b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.242+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/3423188499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf9b620 0x56001cf8b970 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc006010 tx=0x7f4bfc0125e0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.242+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/3423188499 shutdown_connections 2024-04-25T00:23:52.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.242+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/3423188499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf9b620 0x56001cf8b970 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.242+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/3423188499 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90f50 0x56001cf92dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.242+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/3423188499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf90630 0x56001cfa7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.242+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/3423188499 >> 172.21.15.27:0/3423188499 conn(0x56001cf8c950 msgr2=0x56001cf8b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.242+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/3423188499 shutdown_connections 2024-04-25T00:23:52.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.242+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/3423188499 wait complete. 2024-04-25T00:23:52.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c0bdd4740 1 -- start start 2024-04-25T00:23:52.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c0bdd4740 1 Processor -- start 2024-04-25T00:23:52.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c0bdd4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf90630 0x56001cfa3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c0bdd4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90f50 0x56001cf94ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c0bdd4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf9b620 0x56001cf95400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c0bdd4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b2370 con 0x56001cf90630 2024-04-25T00:23:52.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c0bdd4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b1c70 con 0x56001cf90f50 2024-04-25T00:23:52.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c0bdd4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041aee30 con 0x56001cf9b620 2024-04-25T00:23:52.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c0bdd4740 10 client.0 _ll_register_callbacks cb 0x7f4bfc0124b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:52.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c02d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf90630 0x56001cfa3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c02d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf90630 0x56001cfa3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49796/0 (socket says 172.21.15.27:49796) 2024-04-25T00:23:52.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c02d9d640 1 -- 172.21.15.27:0/3648157882 learned_addr learned my addr 172.21.15.27:0/3648157882 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:52.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4bfbfff640 1 --2- 172.21.15.27:0/3648157882 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90f50 0x56001cf94ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c0359e640 1 --2- 172.21.15.27:0/3648157882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf9b620 0x56001cf95400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c02d9d640 1 -- 172.21.15.27:0/3648157882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf9b620 msgr2=0x56001cf95400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c02d9d640 1 --2- 172.21.15.27:0/3648157882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf9b620 0x56001cf95400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c02d9d640 1 -- 172.21.15.27:0/3648157882 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90f50 msgr2=0x56001cf94ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c02d9d640 1 --2- 172.21.15.27:0/3648157882 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90f50 0x56001cf94ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c02d9d640 1 -- 172.21.15.27:0/3648157882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56001cf95c30 con 0x56001cf90630 2024-04-25T00:23:52.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4bfbfff640 1 --2- 172.21.15.27:0/3648157882 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90f50 0x56001cf94ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:23:52.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.243+0000 7f4c0359e640 1 --2- 172.21.15.27:0/3648157882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf9b620 0x56001cf95400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:23:52.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.244+0000 7f4c02d9d640 1 --2- 172.21.15.27:0/3648157882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf90630 0x56001cfa3b20 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4004c40 tx=0x7f4bf4005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ded1afbf358f40e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.244+0000 7f4bea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.244+0000 7f4bea7fc640 1 -- 172.21.15.27:0/3648157882 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4057060 con 0x56001cf90630 2024-04-25T00:23:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.244+0000 7f4bea7fc640 1 -- 172.21.15.27:0/3648157882 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f4bf40048c0 con 0x56001cf90630 2024-04-25T00:23:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.244+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/3648157882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bfc0120b0 con 0x56001cf90630 2024-04-25T00:23:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.244+0000 7f4bea7fc640 1 -- 172.21.15.27:0/3648157882 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f4bf405e050 con 0x56001cf90630 2024-04-25T00:23:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.244+0000 7f4c0bdd4740 10 client.16683 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.244+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/3648157882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x56001d011ed0 con 0x56001cf90630 2024-04-25T00:23:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.244+0000 7f4bea7fc640 1 -- 172.21.15.27:0/3648157882 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bfc0120b0 con 0x56001cf90630 2024-04-25T00:23:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.244+0000 7f4bea7fc640 1 -- 172.21.15.27:0/3648157882 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f4bf4052330 con 0x56001cf90630 2024-04-25T00:23:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.244+0000 7f4c0bdd4740 10 client.16683 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.244+0000 7f4c0bdd4740 10 client.16683 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:23:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.244+0000 7f4c0bdd4740 20 client.16683 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.244+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/3648157882 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56001d012230 0x56001d014620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.244+0000 7f4c0bdd4740 4 client.16683 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:23:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.244+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/3648157882 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56001cefcf40 con 0x56001d012230 2024-04-25T00:23:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.245+0000 7f4bfbfff640 1 --2- 172.21.15.27:0/3648157882 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56001d012230 0x56001d014620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.245+0000 7f4bea7fc640 10 client.16683 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.245+0000 7f4bfbfff640 1 --2- 172.21.15.27:0/3648157882 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56001d012230 0x56001d014620 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19bcbdb585a31304 server_cookie=d3d6b8559c6872ff in_seq=0 out_seq=0 2024-04-25T00:23:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.245+0000 7f4bea7fc640 1 -- 172.21.15.27:0/3648157882 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56001cefcf40 con 0x56001d012230 2024-04-25T00:23:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.245+0000 7f4bea7fc640 10 client.16683 handle_command_reply: tid=0 2024-04-25T00:23:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.245+0000 7f4c0bdd4740 1 client.16683 shutdown 2024-04-25T00:23:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.245+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/3648157882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf90630 msgr2=0x56001cfa3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.245+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/3648157882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf90630 0x56001cfa3b20 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4004c40 tx=0x7f4bf4005b50 comp rx=0 tx=0).stop 2024-04-25T00:23:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.245+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/3648157882 shutdown_connections 2024-04-25T00:23:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.245+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/3648157882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56001cf9b620 0x56001cf95400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.245+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/3648157882 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x56001d012230 0x56001d014620 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.245+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/3648157882 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56001cf90f50 0x56001cf94ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.245+0000 7f4c0bdd4740 1 --2- 172.21.15.27:0/3648157882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56001cf90630 0x56001cfa3b20 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.245+0000 7f4bea7fc640 0 client.16683 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.245+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/3648157882 >> 172.21.15.27:0/3648157882 conn(0x56001cf8c950 msgr2=0x56001cf8baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.245+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/3648157882 shutdown_connections 2024-04-25T00:23:52.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.245+0000 7f4c0bdd4740 1 -- 172.21.15.27:0/3648157882 wait complete. 2024-04-25T00:23:52.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.246+0000 7f4c0bdd4740 20 client.16683 trim_cache size 0 max 16384 2024-04-25T00:23:52.253 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:23:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T00:23:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T00:23:52.253 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:23:52.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.246+0000 7f4c090bd640 1 -- 172.21.15.27:0/2281776636 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4bf00a2d00 msgr2=0x7f4bf00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.246+0000 7f4c090bd640 1 --2- 172.21.15.27:0/2281776636 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4bf00a2d00 0x7f4bf00a51c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009fd0 tx=0x7f4bfc0096a0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.246+0000 7f4c090bd640 1 -- 172.21.15.27:0/2281776636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c04100fe0 msgr2=0x7f4c040fdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.246+0000 7f4c090bd640 1 --2- 172.21.15.27:0/2281776636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c04100fe0 0x7f4c040fdb90 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f4bec00e9f0 tx=0x7f4bec00eeb0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.246+0000 7f4c090bd640 1 -- 172.21.15.27:0/2281776636 shutdown_connections 2024-04-25T00:23:52.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.246+0000 7f4c090bd640 1 --2- 172.21.15.27:0/2281776636 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f4bf00a2d00 0x7f4bf00a51c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.246+0000 7f4c090bd640 1 --2- 172.21.15.27:0/2281776636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c04106630 0x7f4c040fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.246+0000 7f4c090bd640 1 --2- 172.21.15.27:0/2281776636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c04100fe0 0x7f4c040fdb90 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.246+0000 7f4c090bd640 1 --2- 172.21.15.27:0/2281776636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c04100620 0x7f4c040fd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.246+0000 7f4c090bd640 1 -- 172.21.15.27:0/2281776636 >> 172.21.15.27:0/2281776636 conn(0x7f4c040776a0 msgr2=0x7f4c0406a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.246+0000 7f4c090bd640 1 -- 172.21.15.27:0/2281776636 shutdown_connections 2024-04-25T00:23:52.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.246+0000 7f4c090bd640 1 -- 172.21.15.27:0/2281776636 wait complete. 2024-04-25T00:23:52.257 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T00:23:52.257 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:23:52.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.360+0000 7feb75e12640 1 -- 172.21.15.27:0/635107056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb70102a20 msgr2=0x7feb70102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.360+0000 7feb75e12640 1 --2- 172.21.15.27:0/635107056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb70102a20 0x7feb70102e00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7feb60009900 tx=0x7feb6002a480 comp rx=0 tx=0).stop 2024-04-25T00:23:52.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.360+0000 7feb75e12640 1 -- 172.21.15.27:0/635107056 shutdown_connections 2024-04-25T00:23:52.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.360+0000 7feb75e12640 1 --2- 172.21.15.27:0/635107056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb7010bd60 0x7feb7010e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.360+0000 7feb75e12640 1 --2- 172.21.15.27:0/635107056 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb70103340 0x7feb7010b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.360+0000 7feb75e12640 1 --2- 172.21.15.27:0/635107056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb70102a20 0x7feb70102e00 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.360+0000 7feb75e12640 1 -- 172.21.15.27:0/635107056 >> 172.21.15.27:0/635107056 conn(0x7feb700777a0 msgr2=0x7feb70077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.360+0000 7feb75e12640 1 -- 172.21.15.27:0/635107056 shutdown_connections 2024-04-25T00:23:52.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.360+0000 7feb75e12640 1 -- 172.21.15.27:0/635107056 wait complete. 2024-04-25T00:23:52.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.360+0000 7feb75e12640 1 Processor -- start 2024-04-25T00:23:52.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb75e12640 1 -- start start 2024-04-25T00:23:52.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb75e12640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb70102a20 0x7feb70198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb75e12640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb70103340 0x7feb701985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb75e12640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7010bd60 0x7feb7019c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb75e12640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feb70111b30 con 0x7feb7010bd60 2024-04-25T00:23:52.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb75e12640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feb701119b0 con 0x7feb70103340 2024-04-25T00:23:52.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb75e12640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feb70111cb0 con 0x7feb70102a20 2024-04-25T00:23:52.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb6ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7010bd60 0x7feb7019c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb6ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7010bd60 0x7feb7019c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49810/0 (socket says 172.21.15.27:49810) 2024-04-25T00:23:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb6ffff640 1 -- 172.21.15.27:0/1931000976 learned_addr learned my addr 172.21.15.27:0/1931000976 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb6effd640 1 --2- 172.21.15.27:0/1931000976 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb70103340 0x7feb701985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb6f7fe640 1 --2- 172.21.15.27:0/1931000976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb70102a20 0x7feb70198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb6ffff640 1 -- 172.21.15.27:0/1931000976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb70102a20 msgr2=0x7feb70198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb6ffff640 1 --2- 172.21.15.27:0/1931000976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb70102a20 0x7feb70198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb6ffff640 1 -- 172.21.15.27:0/1931000976 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb70103340 msgr2=0x7feb701985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb6ffff640 1 --2- 172.21.15.27:0/1931000976 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb70103340 0x7feb701985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb6ffff640 1 -- 172.21.15.27:0/1931000976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7019d030 con 0x7feb7010bd60 2024-04-25T00:23:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.361+0000 7feb6ffff640 1 --2- 172.21.15.27:0/1931000976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7010bd60 0x7feb7019c900 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7feb6800e960 tx=0x7feb6800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.362+0000 7feb6cff9640 1 -- 172.21.15.27:0/1931000976 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb68004d20 con 0x7feb7010bd60 2024-04-25T00:23:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.362+0000 7feb75e12640 1 -- 172.21.15.27:0/1931000976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb7019d320 con 0x7feb7010bd60 2024-04-25T00:23:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.362+0000 7feb6cff9640 1 -- 172.21.15.27:0/1931000976 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb68005020 con 0x7feb7010bd60 2024-04-25T00:23:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.362+0000 7feb6cff9640 1 -- 172.21.15.27:0/1931000976 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb680116d0 con 0x7feb7010bd60 2024-04-25T00:23:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.362+0000 7feb75e12640 1 -- 172.21.15.27:0/1931000976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb701a4bb0 con 0x7feb7010bd60 2024-04-25T00:23:52.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.363+0000 7feb75e12640 1 -- 172.21.15.27:0/1931000976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb700689b0 con 0x7feb7010bd60 2024-04-25T00:23:52.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.364+0000 7feb6cff9640 1 -- 172.21.15.27:0/1931000976 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7feb680054d0 con 0x7feb7010bd60 2024-04-25T00:23:52.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.364+0000 7feb6cff9640 1 --2- 172.21.15.27:0/1931000976 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7feb5c0a2d00 0x7feb5c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.365+0000 7feb6f7fe640 1 --2- 172.21.15.27:0/1931000976 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7feb5c0a2d00 0x7feb5c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.365+0000 7feb6cff9640 1 -- 172.21.15.27:0/1931000976 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7feb680cf830 con 0x7feb7010bd60 2024-04-25T00:23:52.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.365+0000 7feb6f7fe640 1 --2- 172.21.15.27:0/1931000976 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7feb5c0a2d00 0x7feb5c0a51c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7feb60009230 tx=0x7feb600058d0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.369+0000 7feb6cff9640 1 -- 172.21.15.27:0/1931000976 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7feb68016030 con 0x7feb7010bd60 2024-04-25T00:23:52.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.596+0000 7feb75e12640 1 -- 172.21.15.27:0/1931000976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb70106ee0 con 0x7feb7010bd60 2024-04-25T00:23:52.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.596+0000 7feb6cff9640 1 -- 172.21.15.27:0/1931000976 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+11954 (secure 0 0 0) 0x7feb6809ba90 con 0x7feb7010bd60 2024-04-25T00:23:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:23:52.597 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":97,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:23:11.564498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":147,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":148,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":146,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24514,24430,24521,24428,24496]},"id":1}]} 2024-04-25T00:23:52.597 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 97 2024-04-25T00:23:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.598+0000 7feb75e12640 1 -- 172.21.15.27:0/1931000976 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7feb5c0a2d00 msgr2=0x7feb5c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.598+0000 7feb75e12640 1 --2- 172.21.15.27:0/1931000976 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7feb5c0a2d00 0x7feb5c0a51c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7feb60009230 tx=0x7feb600058d0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.598+0000 7feb75e12640 1 -- 172.21.15.27:0/1931000976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7010bd60 msgr2=0x7feb7019c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.598+0000 7feb75e12640 1 --2- 172.21.15.27:0/1931000976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7010bd60 0x7feb7019c900 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7feb6800e960 tx=0x7feb6800ee20 comp rx=0 tx=0).stop 2024-04-25T00:23:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.599+0000 7feb75e12640 1 -- 172.21.15.27:0/1931000976 shutdown_connections 2024-04-25T00:23:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.599+0000 7feb75e12640 1 --2- 172.21.15.27:0/1931000976 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7feb5c0a2d00 0x7feb5c0a51c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.599+0000 7feb75e12640 1 --2- 172.21.15.27:0/1931000976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7010bd60 0x7feb7019c900 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.599+0000 7feb75e12640 1 --2- 172.21.15.27:0/1931000976 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb70103340 0x7feb701985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.599+0000 7feb75e12640 1 --2- 172.21.15.27:0/1931000976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb70102a20 0x7feb70198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.599+0000 7feb75e12640 1 -- 172.21.15.27:0/1931000976 >> 172.21.15.27:0/1931000976 conn(0x7feb700777a0 msgr2=0x7feb700fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.599+0000 7feb75e12640 1 -- 172.21.15.27:0/1931000976 shutdown_connections 2024-04-25T00:23:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.599+0000 7feb75e12640 1 -- 172.21.15.27:0/1931000976 wait complete. 2024-04-25T00:23:52.610 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T00:23:52.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.715+0000 7f318bbb0640 1 -- 172.21.15.27:0/4086028954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f318406a780 msgr2=0x7f318410e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.715+0000 7f318bbb0640 1 --2- 172.21.15.27:0/4086028954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f318406a780 0x7f318410e5f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f317c009cd0 tx=0x7f317c02a8e0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.715+0000 7f318bbb0640 1 -- 172.21.15.27:0/4086028954 shutdown_connections 2024-04-25T00:23:52.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.715+0000 7f318bbb0640 1 --2- 172.21.15.27:0/4086028954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f318406a780 0x7f318410e5f0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.715+0000 7f318bbb0640 1 --2- 172.21.15.27:0/4086028954 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3184069dc0 0x7f318406a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.715+0000 7f318bbb0640 1 --2- 172.21.15.27:0/4086028954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3184107730 0x7f3184107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.715+0000 7f318bbb0640 1 -- 172.21.15.27:0/4086028954 >> 172.21.15.27:0/4086028954 conn(0x7f31840776a0 msgr2=0x7f3184077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.715+0000 7f318bbb0640 1 -- 172.21.15.27:0/4086028954 shutdown_connections 2024-04-25T00:23:52.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.715+0000 7f318bbb0640 1 -- 172.21.15.27:0/4086028954 wait complete. 2024-04-25T00:23:52.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318bbb0640 1 Processor -- start 2024-04-25T00:23:52.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318bbb0640 1 -- start start 2024-04-25T00:23:52.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318bbb0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3184069dc0 0x7f3184197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318bbb0640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f318406a780 0x7f31841984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318bbb0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3184107730 0x7f318419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318bbb0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3184111b40 con 0x7f3184107730 2024-04-25T00:23:52.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318bbb0640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f31841119c0 con 0x7f318406a780 2024-04-25T00:23:52.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318bbb0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3184111cc0 con 0x7f3184069dc0 2024-04-25T00:23:52.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318a14d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3184107730 0x7f318419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318a14d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3184107730 0x7f318419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49826/0 (socket says 172.21.15.27:49826) 2024-04-25T00:23:52.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318a14d640 1 -- 172.21.15.27:0/4286736735 learned_addr learned my addr 172.21.15.27:0/4286736735 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:52.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318994c640 1 --2- 172.21.15.27:0/4286736735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3184069dc0 0x7f3184197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318914b640 1 --2- 172.21.15.27:0/4286736735 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f318406a780 0x7f31841984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318a14d640 1 -- 172.21.15.27:0/4286736735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3184069dc0 msgr2=0x7f3184197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318a14d640 1 --2- 172.21.15.27:0/4286736735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3184069dc0 0x7f3184197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318a14d640 1 -- 172.21.15.27:0/4286736735 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f318406a780 msgr2=0x7f31841984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318a14d640 1 --2- 172.21.15.27:0/4286736735 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f318406a780 0x7f31841984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318a14d640 1 -- 172.21.15.27:0/4286736735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f318419cf50 con 0x7f3184107730 2024-04-25T00:23:52.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.716+0000 7f318914b640 1 --2- 172.21.15.27:0/4286736735 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f318406a780 0x7f31841984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:23:52.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.717+0000 7f318a14d640 1 --2- 172.21.15.27:0/4286736735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3184107730 0x7f318419c820 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f317c005cc0 tx=0x7f317c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.717+0000 7f317affd640 1 -- 172.21.15.27:0/4286736735 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c004040 con 0x7f3184107730 2024-04-25T00:23:52.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.717+0000 7f317affd640 1 -- 172.21.15.27:0/4286736735 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f317c0046c0 con 0x7f3184107730 2024-04-25T00:23:52.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.717+0000 7f317affd640 1 -- 172.21.15.27:0/4286736735 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c030e00 con 0x7f3184107730 2024-04-25T00:23:52.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.717+0000 7f318bbb0640 1 -- 172.21.15.27:0/4286736735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f318419d1e0 con 0x7f3184107730 2024-04-25T00:23:52.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.717+0000 7f318bbb0640 1 -- 172.21.15.27:0/4286736735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31841a4ad0 con 0x7f3184107730 2024-04-25T00:23:52.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.720+0000 7f317affd640 1 -- 172.21.15.27:0/4286736735 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f317c0305a0 con 0x7f3184107730 2024-04-25T00:23:52.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.720+0000 7f318e9a0740 1 Processor -- start 2024-04-25T00:23:52.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.720+0000 7f318e9a0740 1 -- start start 2024-04-25T00:23:52.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.720+0000 7f318e9a0740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabe580 0x55642cabe960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.720+0000 7f318e9a0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabeea0 0x55642cac9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.720+0000 7f318e9a0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cac9570 0x55642cacb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.720+0000 7f318e9a0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55642ca23670 con 0x55642cabeea0 2024-04-25T00:23:52.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.720+0000 7f318e9a0740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55642c9cb970 con 0x55642cabe580 2024-04-25T00:23:52.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.720+0000 7f318e9a0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55642ca1f630 con 0x55642cac9570 2024-04-25T00:23:52.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.720+0000 7f317affd640 1 --2- 172.21.15.27:0/4286736735 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f31700a2d00 0x7f31700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f318994c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabeea0 0x55642cac9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f318994c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabeea0 0x55642cac9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49830/0 (socket says 172.21.15.27:49830) 2024-04-25T00:23:52.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f318994c640 1 -- 172.21.15.27:0/3962880552 learned_addr learned my addr 172.21.15.27:0/3962880552 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:52.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f318914b640 1 --2- 172.21.15.27:0/3962880552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabe580 0x55642cabe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f318a14d640 1 --2- 172.21.15.27:0/4286736735 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f31700a2d00 0x7f31700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f318914b640 1 --2- 172.21.15.27:0/3962880552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cac9570 0x55642cacb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f317affd640 1 -- 172.21.15.27:0/4286736735 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f317c02e030 con 0x7f3184107730 2024-04-25T00:23:52.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f318994c640 1 -- 172.21.15.27:0/3962880552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cac9570 msgr2=0x55642cacb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f318994c640 1 --2- 172.21.15.27:0/3962880552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cac9570 0x55642cacb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f318994c640 1 -- 172.21.15.27:0/3962880552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabe580 msgr2=0x55642cabe960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f318994c640 1 --2- 172.21.15.27:0/3962880552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabe580 0x55642cabe960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f318994c640 1 -- 172.21.15.27:0/3962880552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55642c9ef400 con 0x55642cabeea0 2024-04-25T00:23:52.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f318914b640 1 --2- 172.21.15.27:0/3962880552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabe580 0x55642cabe960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:52.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f318a14d640 1 --2- 172.21.15.27:0/4286736735 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f31700a2d00 0x7f31700a51c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f317c0b9b40 tx=0x7f317c0c1040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f318994c640 1 --2- 172.21.15.27:0/3962880552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabeea0 0x55642cac9030 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3180009ca0 tx=0x7f3180011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=821c1a4dfd97d58d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f316ffff640 1 -- 172.21.15.27:0/3962880552 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318001e070 con 0x55642cabeea0 2024-04-25T00:23:52.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f316ffff640 1 -- 172.21.15.27:0/3962880552 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3180019b10 con 0x55642cabeea0 2024-04-25T00:23:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.721+0000 7f316ffff640 1 -- 172.21.15.27:0/3962880552 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318000ea50 con 0x55642cabeea0 2024-04-25T00:23:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.722+0000 7f318e9a0740 1 -- 172.21.15.27:0/3962880552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabeea0 msgr2=0x55642cac9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.722+0000 7f318e9a0740 1 --2- 172.21.15.27:0/3962880552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabeea0 0x55642cac9030 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3180009ca0 tx=0x7f3180011810 comp rx=0 tx=0).stop 2024-04-25T00:23:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.722+0000 7f318e9a0740 1 -- 172.21.15.27:0/3962880552 shutdown_connections 2024-04-25T00:23:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.722+0000 7f318e9a0740 1 --2- 172.21.15.27:0/3962880552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cac9570 0x55642cacb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.722+0000 7f318e9a0740 1 --2- 172.21.15.27:0/3962880552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabeea0 0x55642cac9030 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.722+0000 7f318e9a0740 1 --2- 172.21.15.27:0/3962880552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabe580 0x55642cabe960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.722+0000 7f318e9a0740 1 -- 172.21.15.27:0/3962880552 >> 172.21.15.27:0/3962880552 conn(0x55642caba8a0 msgr2=0x55642cabac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.722+0000 7f318e9a0740 1 -- 172.21.15.27:0/3962880552 shutdown_connections 2024-04-25T00:23:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.722+0000 7f318e9a0740 1 -- 172.21.15.27:0/3962880552 wait complete. 2024-04-25T00:23:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.723+0000 7f318e9a0740 1 -- start start 2024-04-25T00:23:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.723+0000 7f318e9a0740 1 Processor -- start 2024-04-25T00:23:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.723+0000 7f318e9a0740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabe580 0x55642cac8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.723+0000 7f318e9a0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabeea0 0x55642cac61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.723+0000 7f318e9a0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cac9570 0x55642cac6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.723+0000 7f318e9a0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55642cacc360 con 0x55642cabeea0 2024-04-25T00:23:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.723+0000 7f318e9a0740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55642c9df050 con 0x55642cabe580 2024-04-25T00:23:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.723+0000 7f318e9a0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55642cacc4e0 con 0x55642cac9570 2024-04-25T00:23:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.723+0000 7f318e9a0740 10 client.0 _ll_register_callbacks cb 0x55642c736be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.723+0000 7f318994c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabeea0 0x55642cac61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.723+0000 7f318994c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabeea0 0x55642cac61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49842/0 (socket says 172.21.15.27:49842) 2024-04-25T00:23:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.723+0000 7f318994c640 1 -- 172.21.15.27:0/612356038 learned_addr learned my addr 172.21.15.27:0/612356038 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.723+0000 7f318914b640 1 --2- 172.21.15.27:0/612356038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabe580 0x55642cac8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.723+0000 7f318914b640 1 --2- 172.21.15.27:0/612356038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cac9570 0x55642cac6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.724+0000 7f318994c640 1 -- 172.21.15.27:0/612356038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cac9570 msgr2=0x55642cac6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.724+0000 7f318994c640 1 --2- 172.21.15.27:0/612356038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cac9570 0x55642cac6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.724+0000 7f318994c640 1 -- 172.21.15.27:0/612356038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabe580 msgr2=0x55642cac8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.724+0000 7f318994c640 1 --2- 172.21.15.27:0/612356038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabe580 0x55642cac8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.724+0000 7f318994c640 1 -- 172.21.15.27:0/612356038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55642c9b0dc0 con 0x55642cabeea0 2024-04-25T00:23:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.724+0000 7f318994c640 1 --2- 172.21.15.27:0/612356038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabeea0 0x55642cac61f0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3180011d10 tx=0x7f3180019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c4141cbefb7219 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.724+0000 7f316d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.724+0000 7f316d7fa640 1 -- 172.21.15.27:0/612356038 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3180029020 con 0x55642cabeea0 2024-04-25T00:23:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.724+0000 7f318e9a0740 1 -- 172.21.15.27:0/612356038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3180019b10 con 0x55642cabeea0 2024-04-25T00:23:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.724+0000 7f316d7fa640 1 -- 172.21.15.27:0/612356038 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f3180004220 con 0x55642cabeea0 2024-04-25T00:23:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.724+0000 7f316d7fa640 1 -- 172.21.15.27:0/612356038 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f318001e050 con 0x55642cabeea0 2024-04-25T00:23:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.724+0000 7f318e9a0740 10 client.16701 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.724+0000 7f318e9a0740 1 -- 172.21.15.27:0/612356038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55642c7373f0 con 0x55642cabeea0 2024-04-25T00:23:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.724+0000 7f316d7fa640 1 -- 172.21.15.27:0/612356038 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3180019b10 con 0x55642cabeea0 2024-04-25T00:23:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.724+0000 7f316d7fa640 1 -- 172.21.15.27:0/612356038 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f318000e7e0 con 0x55642cabeea0 2024-04-25T00:23:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.725+0000 7f318e9a0740 10 client.16701 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.725+0000 7f318e9a0740 10 client.16701 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:23:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.725+0000 7f318e9a0740 20 client.16701 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.725+0000 7f318e9a0740 1 --2- 172.21.15.27:0/612356038 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55642cad9550 0x55642cadb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.725+0000 7f318e9a0740 4 client.16701 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:23:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.725+0000 7f318e9a0740 1 -- 172.21.15.27:0/612356038 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55642ca30bb0 con 0x55642cad9550 2024-04-25T00:23:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.725+0000 7f318914b640 1 --2- 172.21.15.27:0/612356038 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55642cad9550 0x55642cadb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.725+0000 7f316d7fa640 10 client.16701 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:23:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.725+0000 7f318914b640 1 --2- 172.21.15.27:0/612356038 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55642cad9550 0x55642cadb940 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d801d3e7dffa671f server_cookie=8aec0410ee5039f5 in_seq=0 out_seq=0 2024-04-25T00:23:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.726+0000 7f316d7fa640 1 -- 172.21.15.27:0/612356038 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55642ca30bb0 con 0x55642cad9550 2024-04-25T00:23:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.726+0000 7f316d7fa640 10 client.16701 handle_command_reply: tid=0 2024-04-25T00:23:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.726+0000 7f318e9a0740 1 client.16701 shutdown 2024-04-25T00:23:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.726+0000 7f318e9a0740 1 -- 172.21.15.27:0/612356038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabeea0 msgr2=0x55642cac61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.726+0000 7f318e9a0740 1 --2- 172.21.15.27:0/612356038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabeea0 0x55642cac61f0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3180011d10 tx=0x7f3180019660 comp rx=0 tx=0).stop 2024-04-25T00:23:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.726+0000 7f318e9a0740 1 -- 172.21.15.27:0/612356038 shutdown_connections 2024-04-25T00:23:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.726+0000 7f318e9a0740 1 --2- 172.21.15.27:0/612356038 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55642cad9550 0x55642cadb940 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.726+0000 7f318e9a0740 1 --2- 172.21.15.27:0/612356038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cac9570 0x55642cac6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.726+0000 7f318e9a0740 1 --2- 172.21.15.27:0/612356038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabeea0 0x55642cac61f0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.726+0000 7f318e9a0740 1 --2- 172.21.15.27:0/612356038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabe580 0x55642cac8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.726+0000 7f316d7fa640 0 client.16701 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:23:52.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.726+0000 7f318e9a0740 1 -- 172.21.15.27:0/612356038 >> 172.21.15.27:0/612356038 conn(0x55642caba8a0 msgr2=0x55642cab9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.726+0000 7f318e9a0740 1 -- 172.21.15.27:0/612356038 shutdown_connections 2024-04-25T00:23:52.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.726+0000 7f318e9a0740 1 -- 172.21.15.27:0/612356038 wait complete. 2024-04-25T00:23:52.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.726+0000 7f318e9a0740 20 client.16701 trim_cache size 0 max 16384 2024-04-25T00:23:52.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.740+0000 7f318e9a0740 1 Processor -- start 2024-04-25T00:23:52.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.740+0000 7f318e9a0740 1 -- start start 2024-04-25T00:23:52.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.740+0000 7f318e9a0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabe580 0x55642cad5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.740+0000 7f318e9a0740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabeea0 0x55642cae1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.740+0000 7f318e9a0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cac9570 0x55642cae1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.740+0000 7f318e9a0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f31841aeac0 con 0x55642cabe580 2024-04-25T00:23:52.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.740+0000 7f318e9a0740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f31841b0660 con 0x55642cabeea0 2024-04-25T00:23:52.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.740+0000 7f318e9a0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f31841af190 con 0x55642cac9570 2024-04-25T00:23:52.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.740+0000 7f318914b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabe580 0x55642cad5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.740+0000 7f318914b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabe580 0x55642cad5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49858/0 (socket says 172.21.15.27:49858) 2024-04-25T00:23:52.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.740+0000 7f318914b640 1 -- 172.21.15.27:0/1889470730 learned_addr learned my addr 172.21.15.27:0/1889470730 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:52.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.740+0000 7f318994c640 1 --2- 172.21.15.27:0/1889470730 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabeea0 0x55642cae1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.740+0000 7f318914b640 1 -- 172.21.15.27:0/1889470730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cac9570 msgr2=0x55642cae1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.740+0000 7f318914b640 1 --2- 172.21.15.27:0/1889470730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cac9570 0x55642cae1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.740+0000 7f318914b640 1 -- 172.21.15.27:0/1889470730 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabeea0 msgr2=0x55642cae1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.740+0000 7f318914b640 1 --2- 172.21.15.27:0/1889470730 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabeea0 0x55642cae1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.740+0000 7f318914b640 1 -- 172.21.15.27:0/1889470730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55642c8017d0 con 0x55642cabe580 2024-04-25T00:23:52.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.741+0000 7f318914b640 1 --2- 172.21.15.27:0/1889470730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabe580 0x55642cad5250 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f31740043e0 tx=0x7f317400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e858ab5a89cc9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.741+0000 7f316f7fe640 1 -- 172.21.15.27:0/1889470730 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317401c040 con 0x55642cabe580 2024-04-25T00:23:52.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.741+0000 7f316f7fe640 1 -- 172.21.15.27:0/1889470730 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3174022610 con 0x55642cabe580 2024-04-25T00:23:52.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.741+0000 7f316f7fe640 1 -- 172.21.15.27:0/1889470730 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31740116a0 con 0x55642cabe580 2024-04-25T00:23:52.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.741+0000 7f318e9a0740 1 -- 172.21.15.27:0/1889470730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabe580 msgr2=0x55642cad5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.741+0000 7f318e9a0740 1 --2- 172.21.15.27:0/1889470730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabe580 0x55642cad5250 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f31740043e0 tx=0x7f317400c8a0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.741+0000 7f318e9a0740 1 -- 172.21.15.27:0/1889470730 shutdown_connections 2024-04-25T00:23:52.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.741+0000 7f318e9a0740 1 --2- 172.21.15.27:0/1889470730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cac9570 0x55642cae1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.741+0000 7f318e9a0740 1 --2- 172.21.15.27:0/1889470730 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabeea0 0x55642cae1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.741+0000 7f318e9a0740 1 --2- 172.21.15.27:0/1889470730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cabe580 0x55642cad5250 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.741+0000 7f318e9a0740 1 -- 172.21.15.27:0/1889470730 >> 172.21.15.27:0/1889470730 conn(0x55642caba8a0 msgr2=0x55642cab9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.741+0000 7f318e9a0740 1 -- 172.21.15.27:0/1889470730 shutdown_connections 2024-04-25T00:23:52.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.741+0000 7f318e9a0740 1 -- 172.21.15.27:0/1889470730 wait complete. 2024-04-25T00:23:52.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.742+0000 7f318e9a0740 1 -- start start 2024-04-25T00:23:52.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.742+0000 7f318e9a0740 1 Processor -- start 2024-04-25T00:23:52.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.742+0000 7f318e9a0740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabe580 0x55642cacc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.742+0000 7f318e9a0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cabeea0 0x55642caccc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.742+0000 7f318e9a0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cac9570 0x55642cb3d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.742+0000 7f318e9a0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f31841aff90 con 0x55642cac9570 2024-04-25T00:23:52.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.742+0000 7f318e9a0740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f31841af890 con 0x55642cabe580 2024-04-25T00:23:52.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.742+0000 7f318e9a0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f31841aca50 con 0x55642cabeea0 2024-04-25T00:23:52.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.742+0000 7f318e9a0740 10 client.0 _ll_register_callbacks cb 0x7f318001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:52.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.742+0000 7f318914b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cac9570 0x55642cb3d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.742+0000 7f318914b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cac9570 0x55642cb3d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49874/0 (socket says 172.21.15.27:49874) 2024-04-25T00:23:52.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.742+0000 7f318914b640 1 -- 172.21.15.27:0/828187382 learned_addr learned my addr 172.21.15.27:0/828187382 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:52.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.742+0000 7f318914b640 1 --2- 172.21.15.27:0/828187382 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabe580 0x55642cacc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.742+0000 7f318994c640 1 --2- 172.21.15.27:0/828187382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cabeea0 0x55642caccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.743+0000 7f318914b640 1 -- 172.21.15.27:0/828187382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cabeea0 msgr2=0x55642caccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.743+0000 7f318914b640 1 --2- 172.21.15.27:0/828187382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cabeea0 0x55642caccc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.743+0000 7f318914b640 1 -- 172.21.15.27:0/828187382 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabe580 msgr2=0x55642cacc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.743+0000 7f318914b640 1 --2- 172.21.15.27:0/828187382 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabe580 0x55642cacc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.743+0000 7f318914b640 1 -- 172.21.15.27:0/828187382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55642cacd4f0 con 0x55642cac9570 2024-04-25T00:23:52.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.743+0000 7f318914b640 1 --2- 172.21.15.27:0/828187382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cac9570 0x55642cb3d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3174013fd0 tx=0x7f317400c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfec692b942b0ed7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.743+0000 7f316d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:52.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.743+0000 7f316d7fa640 1 -- 172.21.15.27:0/828187382 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3174005230 con 0x55642cac9570 2024-04-25T00:23:52.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.743+0000 7f316d7fa640 1 -- 172.21.15.27:0/828187382 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f31740053d0 con 0x55642cac9570 2024-04-25T00:23:52.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.743+0000 7f318e9a0740 1 -- 172.21.15.27:0/828187382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3174022610 con 0x55642cac9570 2024-04-25T00:23:52.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.743+0000 7f316d7fa640 1 -- 172.21.15.27:0/828187382 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f317403e050 con 0x55642cac9570 2024-04-25T00:23:52.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.743+0000 7f318e9a0740 10 client.16707 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:52.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.743+0000 7f318e9a0740 1 -- 172.21.15.27:0/828187382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55642cb3efa0 con 0x55642cac9570 2024-04-25T00:23:52.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.743+0000 7f316d7fa640 1 -- 172.21.15.27:0/828187382 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3174022610 con 0x55642cac9570 2024-04-25T00:23:52.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.744+0000 7f316d7fa640 1 -- 172.21.15.27:0/828187382 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f31740048f0 con 0x55642cac9570 2024-04-25T00:23:52.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.744+0000 7f318e9a0740 10 client.16707 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:52.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.744+0000 7f318e9a0740 10 client.16707 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:23:52.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.744+0000 7f318e9a0740 20 client.16707 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:52.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.744+0000 7f318e9a0740 1 --2- 172.21.15.27:0/828187382 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55642cb3f720 0x55642cb41b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.744+0000 7f318e9a0740 4 client.16707 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:23:52.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.744+0000 7f318e9a0740 1 -- 172.21.15.27:0/828187382 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55642ca2af00 con 0x55642cb3f720 2024-04-25T00:23:52.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.744+0000 7f318994c640 1 --2- 172.21.15.27:0/828187382 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55642cb3f720 0x55642cb41b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.744+0000 7f316d7fa640 10 client.16707 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:23:52.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.744+0000 7f318994c640 1 --2- 172.21.15.27:0/828187382 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55642cb3f720 0x55642cb41b40 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8c347c78e2e04156 server_cookie=186d1776628ac676 in_seq=0 out_seq=0 2024-04-25T00:23:52.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.745+0000 7f316d7fa640 1 -- 172.21.15.27:0/828187382 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55642ca2af00 con 0x55642cb3f720 2024-04-25T00:23:52.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.745+0000 7f316d7fa640 10 client.16707 handle_command_reply: tid=0 2024-04-25T00:23:52.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.745+0000 7f318e9a0740 1 client.16707 shutdown 2024-04-25T00:23:52.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.745+0000 7f318e9a0740 1 -- 172.21.15.27:0/828187382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cac9570 msgr2=0x55642cb3d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.745+0000 7f318e9a0740 1 --2- 172.21.15.27:0/828187382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cac9570 0x55642cb3d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3174013fd0 tx=0x7f317400c720 comp rx=0 tx=0).stop 2024-04-25T00:23:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.745+0000 7f318e9a0740 1 -- 172.21.15.27:0/828187382 shutdown_connections 2024-04-25T00:23:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.745+0000 7f318e9a0740 1 --2- 172.21.15.27:0/828187382 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55642cb3f720 0x55642cb41b40 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.745+0000 7f318e9a0740 1 --2- 172.21.15.27:0/828187382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55642cac9570 0x55642cb3d990 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.745+0000 7f318e9a0740 1 --2- 172.21.15.27:0/828187382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55642cabeea0 0x55642caccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.745+0000 7f318e9a0740 1 --2- 172.21.15.27:0/828187382 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55642cabe580 0x55642cacc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.745+0000 7f318e9a0740 1 -- 172.21.15.27:0/828187382 >> 172.21.15.27:0/828187382 conn(0x55642caba8a0 msgr2=0x55642cac4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.745+0000 7f316d7fa640 0 client.16707 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:23:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.745+0000 7f318e9a0740 1 -- 172.21.15.27:0/828187382 shutdown_connections 2024-04-25T00:23:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.745+0000 7f318e9a0740 1 -- 172.21.15.27:0/828187382 wait complete. 2024-04-25T00:23:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.745+0000 7f318e9a0740 20 client.16707 trim_cache size 0 max 16384 2024-04-25T00:23:52.751 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:23:52.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.745+0000 7f318bbb0640 1 -- 172.21.15.27:0/4286736735 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f31700a2d00 msgr2=0x7f31700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.745+0000 7f318bbb0640 1 --2- 172.21.15.27:0/4286736735 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f31700a2d00 0x7f31700a51c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f317c0b9b40 tx=0x7f317c0c1040 comp rx=0 tx=0).stop 2024-04-25T00:23:52.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.746+0000 7f318bbb0640 1 -- 172.21.15.27:0/4286736735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3184107730 msgr2=0x7f318419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.746+0000 7f318bbb0640 1 --2- 172.21.15.27:0/4286736735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3184107730 0x7f318419c820 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f317c005cc0 tx=0x7f317c032f40 comp rx=0 tx=0).stop 2024-04-25T00:23:52.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.746+0000 7f318bbb0640 1 -- 172.21.15.27:0/4286736735 shutdown_connections 2024-04-25T00:23:52.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.746+0000 7f318bbb0640 1 --2- 172.21.15.27:0/4286736735 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f31700a2d00 0x7f31700a51c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.746+0000 7f318bbb0640 1 --2- 172.21.15.27:0/4286736735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3184107730 0x7f318419c820 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.746+0000 7f318bbb0640 1 --2- 172.21.15.27:0/4286736735 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f318406a780 0x7f31841984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.746+0000 7f318bbb0640 1 --2- 172.21.15.27:0/4286736735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3184069dc0 0x7f3184197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.746+0000 7f318bbb0640 1 -- 172.21.15.27:0/4286736735 >> 172.21.15.27:0/4286736735 conn(0x7f31840776a0 msgr2=0x7f31840fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.746+0000 7f318bbb0640 1 -- 172.21.15.27:0/4286736735 shutdown_connections 2024-04-25T00:23:52.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.746+0000 7f318bbb0640 1 -- 172.21.15.27:0/4286736735 wait complete. 2024-04-25T00:23:52.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T00:23:52.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.861+0000 7f5db6942640 1 -- 172.21.15.27:0/4002508489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5db0101710 msgr2=0x7f5db0101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.861+0000 7f5db6942640 1 --2- 172.21.15.27:0/4002508489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5db0101710 0x7f5db0101b90 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f5da4009930 tx=0x7f5da402a420 comp rx=0 tx=0).stop 2024-04-25T00:23:52.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.861+0000 7f5db6942640 1 -- 172.21.15.27:0/4002508489 shutdown_connections 2024-04-25T00:23:52.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.861+0000 7f5db6942640 1 --2- 172.21.15.27:0/4002508489 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5db01020d0 0x7f5db010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.861+0000 7f5db6942640 1 --2- 172.21.15.27:0/4002508489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5db0101710 0x7f5db0101b90 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.861+0000 7f5db6942640 1 --2- 172.21.15.27:0/4002508489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5db0107720 0x7f5db0107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.861+0000 7f5db6942640 1 -- 172.21.15.27:0/4002508489 >> 172.21.15.27:0/4002508489 conn(0x7f5db00fd180 msgr2=0x7f5db00ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.861+0000 7f5db6942640 1 -- 172.21.15.27:0/4002508489 shutdown_connections 2024-04-25T00:23:52.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.861+0000 7f5db6942640 1 -- 172.21.15.27:0/4002508489 wait complete. 2024-04-25T00:23:52.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.861+0000 7f5db6942640 1 Processor -- start 2024-04-25T00:23:52.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.861+0000 7f5db6942640 1 -- start start 2024-04-25T00:23:52.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5db6942640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5db0101710 0x7f5db019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5db6942640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5db01020d0 0x7f5db019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5db6942640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5db0107720 0x7f5db019ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5db6942640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5db0111b30 con 0x7f5db01020d0 2024-04-25T00:23:52.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5db6942640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01119b0 con 0x7f5db0101710 2024-04-25T00:23:52.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5db6942640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5db0111cb0 con 0x7f5db0107720 2024-04-25T00:23:52.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5daf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5db01020d0 0x7f5db019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5daf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5db01020d0 0x7f5db019a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49900/0 (socket says 172.21.15.27:49900) 2024-04-25T00:23:52.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5daf7fe640 1 -- 172.21.15.27:0/279094170 learned_addr learned my addr 172.21.15.27:0/279094170 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:52.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5daffff640 1 --2- 172.21.15.27:0/279094170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5db0101710 0x7f5db019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5db4edf640 1 --2- 172.21.15.27:0/279094170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5db0107720 0x7f5db019ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5daf7fe640 1 -- 172.21.15.27:0/279094170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5db0107720 msgr2=0x7f5db019ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5daf7fe640 1 --2- 172.21.15.27:0/279094170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5db0107720 0x7f5db019ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5daf7fe640 1 -- 172.21.15.27:0/279094170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5db0101710 msgr2=0x7f5db019a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5daf7fe640 1 --2- 172.21.15.27:0/279094170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5db0101710 0x7f5db019a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5daf7fe640 1 -- 172.21.15.27:0/279094170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db019f1c0 con 0x7f5db01020d0 2024-04-25T00:23:52.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5daf7fe640 1 --2- 172.21.15.27:0/279094170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5db01020d0 0x7f5db019a710 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f5da4009230 tx=0x7f5da4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5dad7fa640 1 -- 172.21.15.27:0/279094170 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da403f030 con 0x7f5db01020d0 2024-04-25T00:23:52.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5dad7fa640 1 -- 172.21.15.27:0/279094170 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5da4003ad0 con 0x7f5db01020d0 2024-04-25T00:23:52.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.862+0000 7f5db6942640 1 -- 172.21.15.27:0/279094170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db019f450 con 0x7f5db01020d0 2024-04-25T00:23:52.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.863+0000 7f5dad7fa640 1 -- 172.21.15.27:0/279094170 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da4035070 con 0x7f5db01020d0 2024-04-25T00:23:52.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.863+0000 7f5db6942640 1 -- 172.21.15.27:0/279094170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db01a6d40 con 0x7f5db01020d0 2024-04-25T00:23:52.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.865+0000 7f5dad7fa640 1 -- 172.21.15.27:0/279094170 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5da4042030 con 0x7f5db01020d0 2024-04-25T00:23:52.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.865+0000 7f5db9651740 1 Processor -- start 2024-04-25T00:23:52.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.865+0000 7f5db9651740 1 -- start start 2024-04-25T00:23:52.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.865+0000 7f5db9651740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e10580 0x5606a3e10960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.865+0000 7f5db9651740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e10ea0 0x5606a3e1b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.865+0000 7f5db9651740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e1b570 0x5606a3e1d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.865+0000 7f5db9651740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5606a3d75670 con 0x5606a3e1b570 2024-04-25T00:23:52.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.865+0000 7f5db9651740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5606a3d1d970 con 0x5606a3e10ea0 2024-04-25T00:23:52.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.865+0000 7f5db9651740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5606a3d71630 con 0x5606a3e10580 2024-04-25T00:23:52.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.865+0000 7f5dad7fa640 1 --2- 172.21.15.27:0/279094170 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5d980a2d00 0x7f5d980a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.865+0000 7f5daf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e1b570 0x5606a3e1d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.865+0000 7f5daf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e1b570 0x5606a3e1d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49912/0 (socket says 172.21.15.27:49912) 2024-04-25T00:23:52.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.865+0000 7f5daf7fe640 1 -- 172.21.15.27:0/163030170 learned_addr learned my addr 172.21.15.27:0/163030170 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:52.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.865+0000 7f5daffff640 1 --2- 172.21.15.27:0/163030170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e10ea0 0x5606a3e1b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.865+0000 7f5db4edf640 1 --2- 172.21.15.27:0/279094170 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5d980a2d00 0x7f5d980a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.866+0000 7f5db4edf640 1 --2- 172.21.15.27:0/163030170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e10580 0x5606a3e10960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.866+0000 7f5dad7fa640 1 -- 172.21.15.27:0/279094170 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f5da40efd50 con 0x7f5db01020d0 2024-04-25T00:23:52.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.866+0000 7f5daf7fe640 1 -- 172.21.15.27:0/163030170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e10580 msgr2=0x5606a3e10960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.866+0000 7f5daf7fe640 1 --2- 172.21.15.27:0/163030170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e10580 0x5606a3e10960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.866+0000 7f5daf7fe640 1 -- 172.21.15.27:0/163030170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e10ea0 msgr2=0x5606a3e1b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.866+0000 7f5daf7fe640 1 --2- 172.21.15.27:0/163030170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e10ea0 0x5606a3e1b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.866+0000 7f5daf7fe640 1 -- 172.21.15.27:0/163030170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606a3d41400 con 0x5606a3e1b570 2024-04-25T00:23:52.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.866+0000 7f5daffff640 1 --2- 172.21.15.27:0/163030170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e10ea0 0x5606a3e1b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:52.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.866+0000 7f5daf7fe640 1 --2- 172.21.15.27:0/163030170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e1b570 0x5606a3e1d960 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f5da40ed0a0 tx=0x7f5da4009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=649a57160f2f1dcc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.866+0000 7f5db4edf640 1 --2- 172.21.15.27:0/279094170 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5d980a2d00 0x7f5d980a51c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5da80077f0 tx=0x7f5da8007780 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.866+0000 7f5d967fc640 1 -- 172.21.15.27:0/163030170 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da4049070 con 0x5606a3e1b570 2024-04-25T00:23:52.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.866+0000 7f5d967fc640 1 -- 172.21.15.27:0/163030170 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5da40f1070 con 0x5606a3e1b570 2024-04-25T00:23:52.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.866+0000 7f5d967fc640 1 -- 172.21.15.27:0/163030170 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da40b8620 con 0x5606a3e1b570 2024-04-25T00:23:52.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.867+0000 7f5db9651740 1 -- 172.21.15.27:0/163030170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e1b570 msgr2=0x5606a3e1d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.867+0000 7f5db9651740 1 --2- 172.21.15.27:0/163030170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e1b570 0x5606a3e1d960 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f5da40ed0a0 tx=0x7f5da4009680 comp rx=0 tx=0).stop 2024-04-25T00:23:52.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.867+0000 7f5db9651740 1 -- 172.21.15.27:0/163030170 shutdown_connections 2024-04-25T00:23:52.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.867+0000 7f5db9651740 1 --2- 172.21.15.27:0/163030170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e1b570 0x5606a3e1d960 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.867+0000 7f5db9651740 1 --2- 172.21.15.27:0/163030170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e10ea0 0x5606a3e1b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.867+0000 7f5db9651740 1 --2- 172.21.15.27:0/163030170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e10580 0x5606a3e10960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.867+0000 7f5db9651740 1 -- 172.21.15.27:0/163030170 >> 172.21.15.27:0/163030170 conn(0x5606a3e0c8a0 msgr2=0x5606a3e0cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.867+0000 7f5db9651740 1 -- 172.21.15.27:0/163030170 shutdown_connections 2024-04-25T00:23:52.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.867+0000 7f5db9651740 1 -- 172.21.15.27:0/163030170 wait complete. 2024-04-25T00:23:52.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.867+0000 7f5db9651740 1 -- start start 2024-04-25T00:23:52.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.867+0000 7f5db9651740 1 Processor -- start 2024-04-25T00:23:52.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.867+0000 7f5db9651740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e10580 0x5606a3e1aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.867+0000 7f5db9651740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e10ea0 0x5606a3e181e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5db9651740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e1b570 0x5606a3e18760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5db9651740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5606a3e1e120 con 0x5606a3e10580 2024-04-25T00:23:52.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5db9651740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5606a3d31050 con 0x5606a3e10ea0 2024-04-25T00:23:52.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5db9651740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5606a3e1e2a0 con 0x5606a3e1b570 2024-04-25T00:23:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5daffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e10580 0x5606a3e1aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5db9651740 10 client.0 _ll_register_callbacks cb 0x5606a3a88be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5daffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e10580 0x5606a3e1aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49926/0 (socket says 172.21.15.27:49926) 2024-04-25T00:23:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5daffff640 1 -- 172.21.15.27:0/1233316650 learned_addr learned my addr 172.21.15.27:0/1233316650 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5daf7fe640 1 --2- 172.21.15.27:0/1233316650 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e10ea0 0x5606a3e181e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5db4edf640 1 --2- 172.21.15.27:0/1233316650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e1b570 0x5606a3e18760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5daffff640 1 -- 172.21.15.27:0/1233316650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e1b570 msgr2=0x5606a3e18760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5daffff640 1 --2- 172.21.15.27:0/1233316650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e1b570 0x5606a3e18760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5daffff640 1 -- 172.21.15.27:0/1233316650 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e10ea0 msgr2=0x5606a3e181e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5daffff640 1 --2- 172.21.15.27:0/1233316650 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e10ea0 0x5606a3e181e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5daffff640 1 -- 172.21.15.27:0/1233316650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606a3d02dc0 con 0x5606a3e10580 2024-04-25T00:23:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5db4edf640 1 --2- 172.21.15.27:0/1233316650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e1b570 0x5606a3e18760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:23:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5daffff640 1 --2- 172.21.15.27:0/1233316650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e10580 0x5606a3e1aef0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f5da00126e0 tx=0x7f5da0012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d44b1e3dd06370 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5d8f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5d8f7fe640 1 -- 172.21.15.27:0/1233316650 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da00120d0 con 0x5606a3e10580 2024-04-25T00:23:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5d8f7fe640 1 -- 172.21.15.27:0/1233316650 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f5da000f920 con 0x5606a3e10580 2024-04-25T00:23:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5db9651740 1 -- 172.21.15.27:0/1233316650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5da40f1070 con 0x5606a3e10580 2024-04-25T00:23:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5d8f7fe640 1 -- 172.21.15.27:0/1233316650 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f5da0012270 con 0x5606a3e10580 2024-04-25T00:23:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5db9651740 10 client.16719 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.868+0000 7f5db9651740 1 -- 172.21.15.27:0/1233316650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5606a3a893f0 con 0x5606a3e10580 2024-04-25T00:23:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.869+0000 7f5d8f7fe640 1 -- 172.21.15.27:0/1233316650 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5da40f1070 con 0x5606a3e10580 2024-04-25T00:23:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.869+0000 7f5d8f7fe640 1 -- 172.21.15.27:0/1233316650 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f5da001c420 con 0x5606a3e10580 2024-04-25T00:23:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.869+0000 7f5db9651740 10 client.16719 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.869+0000 7f5db9651740 10 client.16719 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:23:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.869+0000 7f5db9651740 20 client.16719 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.869+0000 7f5db9651740 1 --2- 172.21.15.27:0/1233316650 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x5606a3e2b3c0 0x5606a3e2d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.869+0000 7f5db9651740 4 client.16719 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:23:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.869+0000 7f5db9651740 1 -- 172.21.15.27:0/1233316650 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606a3d82bb0 con 0x5606a3e2b3c0 2024-04-25T00:23:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.869+0000 7f5daf7fe640 1 --2- 172.21.15.27:0/1233316650 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x5606a3e2b3c0 0x5606a3e2d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.869+0000 7f5d8f7fe640 10 client.16719 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:23:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.869+0000 7f5daf7fe640 1 --2- 172.21.15.27:0/1233316650 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x5606a3e2b3c0 0x5606a3e2d840 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=55590e5ebe1a229b server_cookie=ed72943f44fedce8 in_seq=0 out_seq=0 2024-04-25T00:23:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.870+0000 7f5d8f7fe640 1 -- 172.21.15.27:0/1233316650 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5606a3d82bb0 con 0x5606a3e2b3c0 2024-04-25T00:23:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.870+0000 7f5d8f7fe640 10 client.16719 handle_command_reply: tid=0 2024-04-25T00:23:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.870+0000 7f5db9651740 1 client.16719 shutdown 2024-04-25T00:23:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.870+0000 7f5db9651740 1 -- 172.21.15.27:0/1233316650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e10580 msgr2=0x5606a3e1aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.870+0000 7f5db9651740 1 --2- 172.21.15.27:0/1233316650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e10580 0x5606a3e1aef0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f5da00126e0 tx=0x7f5da0012ba0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.870+0000 7f5db9651740 1 -- 172.21.15.27:0/1233316650 shutdown_connections 2024-04-25T00:23:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.870+0000 7f5db9651740 1 --2- 172.21.15.27:0/1233316650 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x5606a3e2b3c0 0x5606a3e2d840 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.870+0000 7f5db9651740 1 --2- 172.21.15.27:0/1233316650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e1b570 0x5606a3e18760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.870+0000 7f5d8f7fe640 0 client.16719 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:23:52.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.870+0000 7f5db9651740 1 --2- 172.21.15.27:0/1233316650 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e10ea0 0x5606a3e181e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.870+0000 7f5db9651740 1 --2- 172.21.15.27:0/1233316650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e10580 0x5606a3e1aef0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.870+0000 7f5db9651740 1 -- 172.21.15.27:0/1233316650 >> 172.21.15.27:0/1233316650 conn(0x5606a3e0c8a0 msgr2=0x5606a3e126a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.870+0000 7f5db9651740 1 -- 172.21.15.27:0/1233316650 shutdown_connections 2024-04-25T00:23:52.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.871+0000 7f5db9651740 1 -- 172.21.15.27:0/1233316650 wait complete. 2024-04-25T00:23:52.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.871+0000 7f5db9651740 20 client.16719 trim_cache size 0 max 16384 2024-04-25T00:23:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.885+0000 7f5db9651740 1 Processor -- start 2024-04-25T00:23:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.885+0000 7f5db9651740 1 -- start start 2024-04-25T00:23:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.885+0000 7f5db9651740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e10580 0x5606a3e26e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.885+0000 7f5db9651740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e10ea0 0x5606a3e33110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.885+0000 7f5db9651740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e1b570 0x5606a3e33650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.885+0000 7f5db9651740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01b0d50 con 0x5606a3e10ea0 2024-04-25T00:23:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.885+0000 7f5db9651740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01b28f0 con 0x5606a3e1b570 2024-04-25T00:23:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.885+0000 7f5db9651740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01b1420 con 0x5606a3e10580 2024-04-25T00:23:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.885+0000 7f5daf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e10ea0 0x5606a3e33110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.885+0000 7f5daf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e10ea0 0x5606a3e33110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49930/0 (socket says 172.21.15.27:49930) 2024-04-25T00:23:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.885+0000 7f5daf7fe640 1 -- 172.21.15.27:0/2535186872 learned_addr learned my addr 172.21.15.27:0/2535186872 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.885+0000 7f5daffff640 1 --2- 172.21.15.27:0/2535186872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e10580 0x5606a3e26e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.886+0000 7f5db4edf640 1 --2- 172.21.15.27:0/2535186872 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e1b570 0x5606a3e33650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.886+0000 7f5daf7fe640 1 -- 172.21.15.27:0/2535186872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e10580 msgr2=0x5606a3e26e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.886+0000 7f5daf7fe640 1 --2- 172.21.15.27:0/2535186872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e10580 0x5606a3e26e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.886+0000 7f5daf7fe640 1 -- 172.21.15.27:0/2535186872 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e1b570 msgr2=0x5606a3e33650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.886+0000 7f5daf7fe640 1 --2- 172.21.15.27:0/2535186872 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e1b570 0x5606a3e33650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.886+0000 7f5daf7fe640 1 -- 172.21.15.27:0/2535186872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606a3e17190 con 0x5606a3e10ea0 2024-04-25T00:23:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.886+0000 7f5db4edf640 1 --2- 172.21.15.27:0/2535186872 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e1b570 0x5606a3e33650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:23:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.886+0000 7f5daf7fe640 1 --2- 172.21.15.27:0/2535186872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e10ea0 0x5606a3e33110 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f5da40bbf40 tx=0x7f5da4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c7e2f4d2ee169d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.886+0000 7f5d95ffb640 1 -- 172.21.15.27:0/2535186872 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da40d4030 con 0x5606a3e10ea0 2024-04-25T00:23:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.886+0000 7f5d95ffb640 1 -- 172.21.15.27:0/2535186872 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5da403bd60 con 0x5606a3e10ea0 2024-04-25T00:23:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.886+0000 7f5d95ffb640 1 -- 172.21.15.27:0/2535186872 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da4049040 con 0x5606a3e10ea0 2024-04-25T00:23:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 1 -- 172.21.15.27:0/2535186872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e10ea0 msgr2=0x5606a3e33110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 1 --2- 172.21.15.27:0/2535186872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e10ea0 0x5606a3e33110 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f5da40bbf40 tx=0x7f5da4009260 comp rx=0 tx=0).stop 2024-04-25T00:23:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 1 -- 172.21.15.27:0/2535186872 shutdown_connections 2024-04-25T00:23:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 1 --2- 172.21.15.27:0/2535186872 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e1b570 0x5606a3e33650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 1 --2- 172.21.15.27:0/2535186872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e10ea0 0x5606a3e33110 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 1 --2- 172.21.15.27:0/2535186872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e10580 0x5606a3e26e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 1 -- 172.21.15.27:0/2535186872 >> 172.21.15.27:0/2535186872 conn(0x5606a3e0c8a0 msgr2=0x5606a3e0bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 1 -- 172.21.15.27:0/2535186872 shutdown_connections 2024-04-25T00:23:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 1 -- 172.21.15.27:0/2535186872 wait complete. 2024-04-25T00:23:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 1 -- start start 2024-04-25T00:23:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 1 Processor -- start 2024-04-25T00:23:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e10580 0x5606a3e20e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e10ea0 0x5606a3e21350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e1b570 0x5606a3e21890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01b2220 con 0x5606a3e1b570 2024-04-25T00:23:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01b1b20 con 0x5606a3e10ea0 2024-04-25T00:23:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01aece0 con 0x5606a3e10580 2024-04-25T00:23:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.887+0000 7f5db9651740 10 client.0 _ll_register_callbacks cb 0x7f5da0009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5db4edf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e1b570 0x5606a3e21890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5db4edf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e1b570 0x5606a3e21890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49946/0 (socket says 172.21.15.27:49946) 2024-04-25T00:23:52.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5db4edf640 1 -- 172.21.15.27:0/114899374 learned_addr learned my addr 172.21.15.27:0/114899374 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:52.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5daffff640 1 --2- 172.21.15.27:0/114899374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e10580 0x5606a3e20e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5daf7fe640 1 --2- 172.21.15.27:0/114899374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e10ea0 0x5606a3e21350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5db4edf640 1 -- 172.21.15.27:0/114899374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e10580 msgr2=0x5606a3e20e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5db4edf640 1 --2- 172.21.15.27:0/114899374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e10580 0x5606a3e20e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5db4edf640 1 -- 172.21.15.27:0/114899374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e10ea0 msgr2=0x5606a3e21350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:52.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5db4edf640 1 --2- 172.21.15.27:0/114899374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e10ea0 0x5606a3e21350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5db4edf640 1 -- 172.21.15.27:0/114899374 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606a3b537d0 con 0x5606a3e1b570 2024-04-25T00:23:52.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5db4edf640 1 --2- 172.21.15.27:0/114899374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e1b570 0x5606a3e21890 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f5da8014420 tx=0x7f5da8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8032eb421d1d161f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:52.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5d8f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:52.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5d8f7fe640 1 -- 172.21.15.27:0/114899374 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da800fe90 con 0x5606a3e1b570 2024-04-25T00:23:52.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5db9651740 1 -- 172.21.15.27:0/114899374 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5da403bd60 con 0x5606a3e1b570 2024-04-25T00:23:52.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5d8f7fe640 1 -- 172.21.15.27:0/114899374 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f5da80129d0 con 0x5606a3e1b570 2024-04-25T00:23:52.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5d8f7fe640 1 -- 172.21.15.27:0/114899374 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f5da8012bd0 con 0x5606a3e1b570 2024-04-25T00:23:52.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5db9651740 10 client.16725 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:52.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.888+0000 7f5db9651740 1 -- 172.21.15.27:0/114899374 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5606a3e91af0 con 0x5606a3e1b570 2024-04-25T00:23:52.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.889+0000 7f5d8f7fe640 1 -- 172.21.15.27:0/114899374 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5da403bd60 con 0x5606a3e1b570 2024-04-25T00:23:52.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.889+0000 7f5d8f7fe640 1 -- 172.21.15.27:0/114899374 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f5da8029430 con 0x5606a3e1b570 2024-04-25T00:23:52.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.889+0000 7f5db9651740 10 client.16725 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:52.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.889+0000 7f5db9651740 10 client.16725 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:23:52.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.889+0000 7f5db9651740 20 client.16725 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:52.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.889+0000 7f5db9651740 1 --2- 172.21.15.27:0/114899374 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x5606a3e922d0 0x5606a3e946f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:52.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.889+0000 7f5db9651740 4 client.16725 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:23:52.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.889+0000 7f5db9651740 1 -- 172.21.15.27:0/114899374 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5606a3d7cf00 con 0x5606a3e922d0 2024-04-25T00:23:52.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.889+0000 7f5daffff640 1 --2- 172.21.15.27:0/114899374 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x5606a3e922d0 0x5606a3e946f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:52.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.889+0000 7f5daffff640 1 --2- 172.21.15.27:0/114899374 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x5606a3e922d0 0x5606a3e946f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f7bb8e1c95f68dd server_cookie=621fddb6abac2726 in_seq=0 out_seq=0 2024-04-25T00:23:52.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.889+0000 7f5d8f7fe640 10 client.16725 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:23:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.890+0000 7f5d8f7fe640 1 -- 172.21.15.27:0/114899374 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5606a3d7cf00 con 0x5606a3e922d0 2024-04-25T00:23:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.890+0000 7f5d8f7fe640 10 client.16725 handle_command_reply: tid=0 2024-04-25T00:23:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.890+0000 7f5db9651740 1 client.16725 shutdown 2024-04-25T00:23:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.890+0000 7f5db9651740 1 -- 172.21.15.27:0/114899374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e1b570 msgr2=0x5606a3e21890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.890+0000 7f5db9651740 1 --2- 172.21.15.27:0/114899374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e1b570 0x5606a3e21890 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f5da8014420 tx=0x7f5da8004ab0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.890+0000 7f5db9651740 1 -- 172.21.15.27:0/114899374 shutdown_connections 2024-04-25T00:23:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.890+0000 7f5db9651740 1 --2- 172.21.15.27:0/114899374 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x5606a3e922d0 0x5606a3e946f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.890+0000 7f5db9651740 1 --2- 172.21.15.27:0/114899374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5606a3e1b570 0x5606a3e21890 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.890+0000 7f5d8f7fe640 0 client.16725 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:23:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.890+0000 7f5db9651740 1 --2- 172.21.15.27:0/114899374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5606a3e10ea0 0x5606a3e21350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.890+0000 7f5db9651740 1 --2- 172.21.15.27:0/114899374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5606a3e10580 0x5606a3e20e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.890+0000 7f5db9651740 1 -- 172.21.15.27:0/114899374 >> 172.21.15.27:0/114899374 conn(0x5606a3e0c8a0 msgr2=0x5606a3e25e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.890+0000 7f5db9651740 1 -- 172.21.15.27:0/114899374 shutdown_connections 2024-04-25T00:23:52.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.890+0000 7f5db9651740 1 -- 172.21.15.27:0/114899374 wait complete. 2024-04-25T00:23:52.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.890+0000 7f5db9651740 20 client.16725 trim_cache size 0 max 16384 2024-04-25T00:23:52.900 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:23:52.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.891+0000 7f5db6942640 1 -- 172.21.15.27:0/279094170 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5d980a2d00 msgr2=0x7f5d980a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.891+0000 7f5db6942640 1 --2- 172.21.15.27:0/279094170 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5d980a2d00 0x7f5d980a51c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5da80077f0 tx=0x7f5da8007780 comp rx=0 tx=0).stop 2024-04-25T00:23:52.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.891+0000 7f5db6942640 1 -- 172.21.15.27:0/279094170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5db01020d0 msgr2=0x7f5db019a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:52.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.891+0000 7f5db6942640 1 --2- 172.21.15.27:0/279094170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5db01020d0 0x7f5db019a710 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f5da4009230 tx=0x7f5da4004270 comp rx=0 tx=0).stop 2024-04-25T00:23:52.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.891+0000 7f5db6942640 1 -- 172.21.15.27:0/279094170 shutdown_connections 2024-04-25T00:23:52.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.891+0000 7f5db6942640 1 --2- 172.21.15.27:0/279094170 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f5d980a2d00 0x7f5d980a51c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.891+0000 7f5db6942640 1 --2- 172.21.15.27:0/279094170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5db0107720 0x7f5db019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.891+0000 7f5db6942640 1 --2- 172.21.15.27:0/279094170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5db01020d0 0x7f5db019a710 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.891+0000 7f5db6942640 1 --2- 172.21.15.27:0/279094170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5db0101710 0x7f5db019a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:52.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.891+0000 7f5db6942640 1 -- 172.21.15.27:0/279094170 >> 172.21.15.27:0/279094170 conn(0x7f5db00fd180 msgr2=0x7f5db010e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:52.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.891+0000 7f5db6942640 1 -- 172.21.15.27:0/279094170 shutdown_connections 2024-04-25T00:23:52.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:52.891+0000 7f5db6942640 1 -- 172.21.15.27:0/279094170 wait complete. 2024-04-25T00:23:52.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:52 smithi027 ceph-mon[29208]: pgmap v246: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 19 op/s 2024-04-25T00:23:52.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:52 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1931000976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:23:52.903 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T00:23:53.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:52 smithi139 ceph-mon[32833]: pgmap v246: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 19 op/s 2024-04-25T00:23:53.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:52 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1931000976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:23:53.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.007+0000 7f10387c1640 1 -- 172.21.15.27:0/497118865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1030100fd0 msgr2=0x7f103010d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.007+0000 7f10387c1640 1 --2- 172.21.15.27:0/497118865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1030100fd0 0x7f103010d4e0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f1028009cd0 tx=0x7f102802a8e0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.007+0000 7f10387c1640 1 -- 172.21.15.27:0/497118865 shutdown_connections 2024-04-25T00:23:53.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.007+0000 7f10387c1640 1 --2- 172.21.15.27:0/497118865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1030100fd0 0x7f103010d4e0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.007+0000 7f10387c1640 1 --2- 172.21.15.27:0/497118865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1030100610 0x7f1030100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.007+0000 7f10387c1640 1 --2- 172.21.15.27:0/497118865 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10301066b0 0x7f1030106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.007+0000 7f10387c1640 1 -- 172.21.15.27:0/497118865 >> 172.21.15.27:0/497118865 conn(0x7f1030077740 msgr2=0x7f1030077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.007+0000 7f10387c1640 1 -- 172.21.15.27:0/497118865 shutdown_connections 2024-04-25T00:23:53.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.007+0000 7f10387c1640 1 -- 172.21.15.27:0/497118865 wait complete. 2024-04-25T00:23:53.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.008+0000 7f10387c1640 1 Processor -- start 2024-04-25T00:23:53.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.008+0000 7f10387c1640 1 -- start start 2024-04-25T00:23:53.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.008+0000 7f10387c1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1030100610 0x7f1030197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.008+0000 7f10387c1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1030100fd0 0x7f10301984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.008+0000 7f10387c1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10301066b0 0x7f103019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.008+0000 7f10387c1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1030110a30 con 0x7f10301066b0 2024-04-25T00:23:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.008+0000 7f10387c1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f10301108b0 con 0x7f1030100610 2024-04-25T00:23:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.008+0000 7f10387c1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1030110bb0 con 0x7f1030100fd0 2024-04-25T00:23:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.008+0000 7f1036d5e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10301066b0 0x7f103019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.008+0000 7f1036d5e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10301066b0 0x7f103019c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49954/0 (socket says 172.21.15.27:49954) 2024-04-25T00:23:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.008+0000 7f1036d5e640 1 -- 172.21.15.27:0/3439197224 learned_addr learned my addr 172.21.15.27:0/3439197224 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.008+0000 7f103655d640 1 --2- 172.21.15.27:0/3439197224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1030100610 0x7f1030197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.008+0000 7f1035d5c640 1 --2- 172.21.15.27:0/3439197224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1030100fd0 0x7f10301984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.009+0000 7f1036d5e640 1 -- 172.21.15.27:0/3439197224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1030100fd0 msgr2=0x7f10301984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.009+0000 7f1036d5e640 1 --2- 172.21.15.27:0/3439197224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1030100fd0 0x7f10301984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.009+0000 7f1036d5e640 1 -- 172.21.15.27:0/3439197224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1030100610 msgr2=0x7f1030197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.009+0000 7f1036d5e640 1 --2- 172.21.15.27:0/3439197224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1030100610 0x7f1030197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.009+0000 7f1036d5e640 1 -- 172.21.15.27:0/3439197224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f103019cf70 con 0x7f10301066b0 2024-04-25T00:23:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.009+0000 7f1035d5c640 1 --2- 172.21.15.27:0/3439197224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1030100fd0 0x7f10301984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:23:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.009+0000 7f1036d5e640 1 --2- 172.21.15.27:0/3439197224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10301066b0 0x7f103019c840 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f1028005cc0 tx=0x7f1028032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.009+0000 7f10277fe640 1 -- 172.21.15.27:0/3439197224 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1028004040 con 0x7f10301066b0 2024-04-25T00:23:53.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.009+0000 7f10387c1640 1 -- 172.21.15.27:0/3439197224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f103019d200 con 0x7f10301066b0 2024-04-25T00:23:53.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.009+0000 7f10277fe640 1 -- 172.21.15.27:0/3439197224 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10280046c0 con 0x7f10301066b0 2024-04-25T00:23:53.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.009+0000 7f10277fe640 1 -- 172.21.15.27:0/3439197224 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1028030e00 con 0x7f10301066b0 2024-04-25T00:23:53.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.009+0000 7f10387c1640 1 -- 172.21.15.27:0/3439197224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10301a4af0 con 0x7f10301066b0 2024-04-25T00:23:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.011+0000 7f10277fe640 1 -- 172.21.15.27:0/3439197224 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f10280305a0 con 0x7f10301066b0 2024-04-25T00:23:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.011+0000 7f103b5a9740 1 Processor -- start 2024-04-25T00:23:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f103b5a9740 1 -- start start 2024-04-25T00:23:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f103b5a9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4a1580 0x5652bc4a1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f103b5a9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4a1ea0 0x5652bc4ac030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f103b5a9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4ac570 0x5652bc4ae960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f103b5a9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5652bc406670 con 0x5652bc4ac570 2024-04-25T00:23:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f103b5a9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5652bc3ae970 con 0x5652bc4a1580 2024-04-25T00:23:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f103b5a9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5652bc402630 con 0x5652bc4a1ea0 2024-04-25T00:23:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f1035d5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4ac570 0x5652bc4ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f1035d5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4ac570 0x5652bc4ae960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49966/0 (socket says 172.21.15.27:49966) 2024-04-25T00:23:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f1035d5c640 1 -- 172.21.15.27:0/1520268722 learned_addr learned my addr 172.21.15.27:0/1520268722 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:53.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f1035d5c640 1 --2- 172.21.15.27:0/1520268722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4a1580 0x5652bc4a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f103655d640 1 --2- 172.21.15.27:0/1520268722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4a1ea0 0x5652bc4ac030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f10277fe640 1 --2- 172.21.15.27:0/3439197224 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f101c0a2d00 0x7f101c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f1035d5c640 1 -- 172.21.15.27:0/1520268722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4a1ea0 msgr2=0x5652bc4ac030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f1035d5c640 1 --2- 172.21.15.27:0/1520268722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4a1ea0 0x5652bc4ac030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f1035d5c640 1 -- 172.21.15.27:0/1520268722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4a1580 msgr2=0x5652bc4a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f1035d5c640 1 --2- 172.21.15.27:0/1520268722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4a1580 0x5652bc4a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f1035d5c640 1 -- 172.21.15.27:0/1520268722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652bc3d2400 con 0x5652bc4ac570 2024-04-25T00:23:53.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f103655d640 1 --2- 172.21.15.27:0/1520268722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4a1ea0 0x5652bc4ac030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:23:53.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f1036d5e640 1 --2- 172.21.15.27:0/3439197224 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f101c0a2d00 0x7f101c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f10277fe640 1 -- 172.21.15.27:0/3439197224 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f102802e030 con 0x7f10301066b0 2024-04-25T00:23:53.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.012+0000 7f1035d5c640 1 --2- 172.21.15.27:0/1520268722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4ac570 0x5652bc4ae960 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f10200124f0 tx=0x7f102000b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e6bb72489e7e1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.013+0000 7f1024ff9640 1 -- 172.21.15.27:0/1520268722 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1020010490 con 0x5652bc4ac570 2024-04-25T00:23:53.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.013+0000 7f1024ff9640 1 -- 172.21.15.27:0/1520268722 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1020010b10 con 0x5652bc4ac570 2024-04-25T00:23:53.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.013+0000 7f1024ff9640 1 -- 172.21.15.27:0/1520268722 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102000eb10 con 0x5652bc4ac570 2024-04-25T00:23:53.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.013+0000 7f103b5a9740 1 -- 172.21.15.27:0/1520268722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4ac570 msgr2=0x5652bc4ae960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.013+0000 7f1036d5e640 1 --2- 172.21.15.27:0/3439197224 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f101c0a2d00 0x7f101c0a51c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f10280bcc50 tx=0x7f102803d040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.013+0000 7f103b5a9740 1 --2- 172.21.15.27:0/1520268722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4ac570 0x5652bc4ae960 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f10200124f0 tx=0x7f102000b420 comp rx=0 tx=0).stop 2024-04-25T00:23:53.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.013+0000 7f103b5a9740 1 -- 172.21.15.27:0/1520268722 shutdown_connections 2024-04-25T00:23:53.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.013+0000 7f103b5a9740 1 --2- 172.21.15.27:0/1520268722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4ac570 0x5652bc4ae960 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.013+0000 7f103b5a9740 1 --2- 172.21.15.27:0/1520268722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4a1ea0 0x5652bc4ac030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.013+0000 7f103b5a9740 1 --2- 172.21.15.27:0/1520268722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4a1580 0x5652bc4a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.013+0000 7f103b5a9740 1 -- 172.21.15.27:0/1520268722 >> 172.21.15.27:0/1520268722 conn(0x5652bc49d8a0 msgr2=0x5652bc49dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.013+0000 7f103b5a9740 1 -- 172.21.15.27:0/1520268722 shutdown_connections 2024-04-25T00:23:53.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.014+0000 7f103b5a9740 1 -- 172.21.15.27:0/1520268722 wait complete. 2024-04-25T00:23:53.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.014+0000 7f103b5a9740 1 -- start start 2024-04-25T00:23:53.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.014+0000 7f103b5a9740 1 Processor -- start 2024-04-25T00:23:53.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.014+0000 7f103b5a9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4a1580 0x5652bc4abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.014+0000 7f103b5a9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4a9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.014+0000 7f103b5a9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4ac570 0x5652bc4a96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.014+0000 7f103b5a9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5652bc4af360 con 0x5652bc4a1ea0 2024-04-25T00:23:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.014+0000 7f103b5a9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5652bc3c2050 con 0x5652bc4ac570 2024-04-25T00:23:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.014+0000 7f103b5a9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5652bc4af4e0 con 0x5652bc4a1580 2024-04-25T00:23:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.014+0000 7f103b5a9740 10 client.0 _ll_register_callbacks cb 0x5652bc119be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.014+0000 7f103655d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4a9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.014+0000 7f103655d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4a9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49972/0 (socket says 172.21.15.27:49972) 2024-04-25T00:23:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.014+0000 7f103655d640 1 -- 172.21.15.27:0/521893145 learned_addr learned my addr 172.21.15.27:0/521893145 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.014+0000 7f1035d5c640 1 --2- 172.21.15.27:0/521893145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4ac570 0x5652bc4a96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.014+0000 7f1035d5c640 1 --2- 172.21.15.27:0/521893145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4a1580 0x5652bc4abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.014+0000 7f103655d640 1 -- 172.21.15.27:0/521893145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4a1580 msgr2=0x5652bc4abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.014+0000 7f103655d640 1 --2- 172.21.15.27:0/521893145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4a1580 0x5652bc4abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f103655d640 1 -- 172.21.15.27:0/521893145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4ac570 msgr2=0x5652bc4a96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f103655d640 1 --2- 172.21.15.27:0/521893145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4ac570 0x5652bc4a96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f103655d640 1 -- 172.21.15.27:0/521893145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652bc393dc0 con 0x5652bc4a1ea0 2024-04-25T00:23:53.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f1035d5c640 1 --2- 172.21.15.27:0/521893145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4ac570 0x5652bc4a96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:23:53.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f103655d640 1 --2- 172.21.15.27:0/521893145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4a9170 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f102c012700 tx=0x7f102c012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3816c47c5d213e52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f1019ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:53.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f1019ffb640 1 -- 172.21.15.27:0/521893145 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c004030 con 0x5652bc4a1ea0 2024-04-25T00:23:53.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f1019ffb640 1 -- 172.21.15.27:0/521893145 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f102c0041d0 con 0x5652bc4a1ea0 2024-04-25T00:23:53.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f103b5a9740 1 -- 172.21.15.27:0/521893145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1020010b10 con 0x5652bc4a1ea0 2024-04-25T00:23:53.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f1019ffb640 1 -- 172.21.15.27:0/521893145 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f102c0043d0 con 0x5652bc4a1ea0 2024-04-25T00:23:53.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f103b5a9740 10 client.16737 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:53.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f103b5a9740 1 -- 172.21.15.27:0/521893145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5652bc11a3f0 con 0x5652bc4a1ea0 2024-04-25T00:23:53.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f1019ffb640 1 -- 172.21.15.27:0/521893145 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1020010b10 con 0x5652bc4a1ea0 2024-04-25T00:23:53.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f1019ffb640 1 -- 172.21.15.27:0/521893145 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f102c003b90 con 0x5652bc4a1ea0 2024-04-25T00:23:53.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f103b5a9740 10 client.16737 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:53.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f103b5a9740 10 client.16737 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:23:53.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f103b5a9740 20 client.16737 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:53.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f103b5a9740 1 --2- 172.21.15.27:0/521893145 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5652bc4bc650 0x5652bc4bea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f103b5a9740 4 client.16737 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:23:53.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.015+0000 7f103b5a9740 1 -- 172.21.15.27:0/521893145 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652bc413bb0 con 0x5652bc4bc650 2024-04-25T00:23:53.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.016+0000 7f1035d5c640 1 --2- 172.21.15.27:0/521893145 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5652bc4bc650 0x5652bc4bea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.016+0000 7f1035d5c640 1 --2- 172.21.15.27:0/521893145 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5652bc4bc650 0x5652bc4bea70 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a18d1615e518159d server_cookie=95d17eba026a34fc in_seq=0 out_seq=0 2024-04-25T00:23:53.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.016+0000 7f1019ffb640 10 client.16737 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:23:53.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.017+0000 7f1019ffb640 1 -- 172.21.15.27:0/521893145 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5652bc413bb0 con 0x5652bc4bc650 2024-04-25T00:23:53.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.017+0000 7f1019ffb640 10 client.16737 handle_command_reply: tid=0 2024-04-25T00:23:53.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.017+0000 7f103b5a9740 1 client.16737 shutdown 2024-04-25T00:23:53.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.017+0000 7f103b5a9740 1 -- 172.21.15.27:0/521893145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 msgr2=0x5652bc4a9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.017+0000 7f103b5a9740 1 --2- 172.21.15.27:0/521893145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4a9170 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f102c012700 tx=0x7f102c012bc0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.017+0000 7f103b5a9740 1 -- 172.21.15.27:0/521893145 shutdown_connections 2024-04-25T00:23:53.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.017+0000 7f103b5a9740 1 --2- 172.21.15.27:0/521893145 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5652bc4bc650 0x5652bc4bea70 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.017+0000 7f103b5a9740 1 --2- 172.21.15.27:0/521893145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4ac570 0x5652bc4a96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.017+0000 7f103b5a9740 1 --2- 172.21.15.27:0/521893145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4a9170 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.017+0000 7f1019ffb640 0 client.16737 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:23:53.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.017+0000 7f103b5a9740 1 --2- 172.21.15.27:0/521893145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4a1580 0x5652bc4abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.017+0000 7f103b5a9740 1 -- 172.21.15.27:0/521893145 >> 172.21.15.27:0/521893145 conn(0x5652bc49d8a0 msgr2=0x5652bc4a36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.017+0000 7f103b5a9740 1 -- 172.21.15.27:0/521893145 shutdown_connections 2024-04-25T00:23:53.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.017+0000 7f103b5a9740 1 -- 172.21.15.27:0/521893145 wait complete. 2024-04-25T00:23:53.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.017+0000 7f103b5a9740 20 client.16737 trim_cache size 0 max 16384 2024-04-25T00:23:53.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:52 smithi153 ceph-mon[32896]: pgmap v246: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 19 op/s 2024-04-25T00:23:53.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:52 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1931000976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:23:53.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.030+0000 7f103b5a9740 1 Processor -- start 2024-04-25T00:23:53.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f103b5a9740 1 -- start start 2024-04-25T00:23:53.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f103b5a9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4a1580 0x5652bc4b8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f103b5a9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4ab9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f103b5a9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4ac570 0x5652bc4c4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f103b5a9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f10301aeb50 con 0x5652bc4a1ea0 2024-04-25T00:23:53.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f103b5a9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f10301b06f0 con 0x5652bc4ac570 2024-04-25T00:23:53.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f103b5a9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f10301af220 con 0x5652bc4a1580 2024-04-25T00:23:53.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f103655d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4ab9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f103655d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4ab9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49982/0 (socket says 172.21.15.27:49982) 2024-04-25T00:23:53.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f103655d640 1 -- 172.21.15.27:0/3677110980 learned_addr learned my addr 172.21.15.27:0/3677110980 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:53.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f1035d5c640 1 --2- 172.21.15.27:0/3677110980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4a1580 0x5652bc4b8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f103655d640 1 -- 172.21.15.27:0/3677110980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4a1580 msgr2=0x5652bc4b8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f103655d640 1 --2- 172.21.15.27:0/3677110980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4a1580 0x5652bc4b8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f103655d640 1 -- 172.21.15.27:0/3677110980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4ac570 msgr2=0x5652bc4c4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f103655d640 1 --2- 172.21.15.27:0/3677110980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4ac570 0x5652bc4c4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f103655d640 1 -- 172.21.15.27:0/3677110980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652bc1e47d0 con 0x5652bc4a1ea0 2024-04-25T00:23:53.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f103655d640 1 --2- 172.21.15.27:0/3677110980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4ab9b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f102c012410 tx=0x7f102c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56555cf4a33f5b54 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.031+0000 7f101bfff640 1 -- 172.21.15.27:0/3677110980 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c02f070 con 0x5652bc4a1ea0 2024-04-25T00:23:53.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.032+0000 7f101bfff640 1 -- 172.21.15.27:0/3677110980 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f102c00fe80 con 0x5652bc4a1ea0 2024-04-25T00:23:53.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.032+0000 7f101bfff640 1 -- 172.21.15.27:0/3677110980 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c0035f0 con 0x5652bc4a1ea0 2024-04-25T00:23:53.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.032+0000 7f103b5a9740 1 -- 172.21.15.27:0/3677110980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 msgr2=0x5652bc4ab9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.032+0000 7f103b5a9740 1 --2- 172.21.15.27:0/3677110980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4ab9b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f102c012410 tx=0x7f102c00f790 comp rx=0 tx=0).stop 2024-04-25T00:23:53.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.032+0000 7f103b5a9740 1 -- 172.21.15.27:0/3677110980 shutdown_connections 2024-04-25T00:23:53.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.032+0000 7f103b5a9740 1 --2- 172.21.15.27:0/3677110980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4ac570 0x5652bc4c4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.032+0000 7f103b5a9740 1 --2- 172.21.15.27:0/3677110980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4ab9b0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.032+0000 7f103b5a9740 1 --2- 172.21.15.27:0/3677110980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4a1580 0x5652bc4b8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.032+0000 7f103b5a9740 1 -- 172.21.15.27:0/3677110980 >> 172.21.15.27:0/3677110980 conn(0x5652bc49d8a0 msgr2=0x5652bc49cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.032+0000 7f103b5a9740 1 -- 172.21.15.27:0/3677110980 shutdown_connections 2024-04-25T00:23:53.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.032+0000 7f103b5a9740 1 -- 172.21.15.27:0/3677110980 wait complete. 2024-04-25T00:23:53.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.032+0000 7f103b5a9740 1 -- start start 2024-04-25T00:23:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.032+0000 7f103b5a9740 1 Processor -- start 2024-04-25T00:23:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f103b5a9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4a1580 0x5652bc4b0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f103b5a9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4b0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f103b5a9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4ac570 0x5652bc4b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f103b5a9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f10301b0020 con 0x5652bc4a1ea0 2024-04-25T00:23:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f103b5a9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f10301af920 con 0x5652bc4a1580 2024-04-25T00:23:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f103b5a9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f10301acae0 con 0x5652bc4ac570 2024-04-25T00:23:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f103b5a9740 10 client.0 _ll_register_callbacks cb 0x7f102c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f103655d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f103655d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4b0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49996/0 (socket says 172.21.15.27:49996) 2024-04-25T00:23:53.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f103655d640 1 -- 172.21.15.27:0/1551622475 learned_addr learned my addr 172.21.15.27:0/1551622475 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:53.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f1035d5c640 1 --2- 172.21.15.27:0/1551622475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4a1580 0x5652bc4b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f1035d5c640 1 --2- 172.21.15.27:0/1551622475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4ac570 0x5652bc4b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f103655d640 1 -- 172.21.15.27:0/1551622475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4ac570 msgr2=0x5652bc4b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f103655d640 1 --2- 172.21.15.27:0/1551622475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4ac570 0x5652bc4b0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f103655d640 1 -- 172.21.15.27:0/1551622475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4a1580 msgr2=0x5652bc4b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f103655d640 1 --2- 172.21.15.27:0/1551622475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4a1580 0x5652bc4b0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f103655d640 1 -- 172.21.15.27:0/1551622475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652bc521db0 con 0x5652bc4a1ea0 2024-04-25T00:23:53.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f1035d5c640 1 --2- 172.21.15.27:0/1551622475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4ac570 0x5652bc4b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:23:53.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f103655d640 1 --2- 172.21.15.27:0/1551622475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4b0880 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f102c02d040 tx=0x7f102c012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a88d577b0cb8027 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f1019ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:53.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f1019ffb640 1 -- 172.21.15.27:0/1551622475 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c02f070 con 0x5652bc4a1ea0 2024-04-25T00:23:53.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f1019ffb640 1 -- 172.21.15.27:0/1551622475 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f102c00f9a0 con 0x5652bc4a1ea0 2024-04-25T00:23:53.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.033+0000 7f103b5a9740 1 -- 172.21.15.27:0/1551622475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5652bc40df00 con 0x5652bc4a1ea0 2024-04-25T00:23:53.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.034+0000 7f1019ffb640 1 -- 172.21.15.27:0/1551622475 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f102c03d050 con 0x5652bc4a1ea0 2024-04-25T00:23:53.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.034+0000 7f103b5a9740 10 client.16743 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:53.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.034+0000 7f103b5a9740 1 -- 172.21.15.27:0/1551622475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5652bc523070 con 0x5652bc4a1ea0 2024-04-25T00:23:53.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.034+0000 7f1019ffb640 1 -- 172.21.15.27:0/1551622475 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5652bc40df00 con 0x5652bc4a1ea0 2024-04-25T00:23:53.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.034+0000 7f1019ffb640 1 -- 172.21.15.27:0/1551622475 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f102c01c6a0 con 0x5652bc4a1ea0 2024-04-25T00:23:53.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.034+0000 7f103b5a9740 10 client.16743 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:53.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.034+0000 7f103b5a9740 10 client.16743 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:23:53.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.034+0000 7f103b5a9740 20 client.16743 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:53.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.034+0000 7f103b5a9740 1 --2- 172.21.15.27:0/1551622475 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5652bc523730 0x5652bc525b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.034+0000 7f103b5a9740 4 client.16743 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:23:53.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.034+0000 7f103b5a9740 1 -- 172.21.15.27:0/1551622475 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5652bc38b820 con 0x5652bc523730 2024-04-25T00:23:53.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.034+0000 7f1035d5c640 1 --2- 172.21.15.27:0/1551622475 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5652bc523730 0x5652bc525b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.035+0000 7f1019ffb640 10 client.16743 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:23:53.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.035+0000 7f1035d5c640 1 --2- 172.21.15.27:0/1551622475 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5652bc523730 0x5652bc525b50 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7b64a6cbac7461c9 server_cookie=9a665b7e826a97fc in_seq=0 out_seq=0 2024-04-25T00:23:53.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.035+0000 7f1019ffb640 1 -- 172.21.15.27:0/1551622475 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5652bc38b820 con 0x5652bc523730 2024-04-25T00:23:53.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.035+0000 7f1019ffb640 10 client.16743 handle_command_reply: tid=0 2024-04-25T00:23:53.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.035+0000 7f103b5a9740 1 client.16743 shutdown 2024-04-25T00:23:53.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.035+0000 7f103b5a9740 1 -- 172.21.15.27:0/1551622475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 msgr2=0x5652bc4b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.035+0000 7f103b5a9740 1 --2- 172.21.15.27:0/1551622475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4b0880 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f102c02d040 tx=0x7f102c012bc0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.035+0000 7f103b5a9740 1 -- 172.21.15.27:0/1551622475 shutdown_connections 2024-04-25T00:23:53.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.035+0000 7f103b5a9740 1 --2- 172.21.15.27:0/1551622475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5652bc4ac570 0x5652bc4b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.035+0000 7f103b5a9740 1 --2- 172.21.15.27:0/1551622475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652bc4a1ea0 0x5652bc4b0880 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.035+0000 7f103b5a9740 1 --2- 172.21.15.27:0/1551622475 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x5652bc523730 0x5652bc525b50 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.035+0000 7f103b5a9740 1 --2- 172.21.15.27:0/1551622475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5652bc4a1580 0x5652bc4b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.035+0000 7f103b5a9740 1 -- 172.21.15.27:0/1551622475 >> 172.21.15.27:0/1551622475 conn(0x5652bc49d8a0 msgr2=0x5652bc4a89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.035+0000 7f1019ffb640 0 client.16743 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:23:53.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.035+0000 7f103b5a9740 1 -- 172.21.15.27:0/1551622475 shutdown_connections 2024-04-25T00:23:53.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.035+0000 7f103b5a9740 1 -- 172.21.15.27:0/1551622475 wait complete. 2024-04-25T00:23:53.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.035+0000 7f103b5a9740 20 client.16743 trim_cache size 0 max 16384 2024-04-25T00:23:53.047 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:23:53.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.036+0000 7f10387c1640 1 -- 172.21.15.27:0/3439197224 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f101c0a2d00 msgr2=0x7f101c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.036+0000 7f10387c1640 1 --2- 172.21.15.27:0/3439197224 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f101c0a2d00 0x7f101c0a51c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f10280bcc50 tx=0x7f102803d040 comp rx=0 tx=0).stop 2024-04-25T00:23:53.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.036+0000 7f10387c1640 1 -- 172.21.15.27:0/3439197224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10301066b0 msgr2=0x7f103019c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.036+0000 7f10387c1640 1 --2- 172.21.15.27:0/3439197224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10301066b0 0x7f103019c840 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f1028005cc0 tx=0x7f1028032f40 comp rx=0 tx=0).stop 2024-04-25T00:23:53.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.036+0000 7f10387c1640 1 -- 172.21.15.27:0/3439197224 shutdown_connections 2024-04-25T00:23:53.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.036+0000 7f10387c1640 1 --2- 172.21.15.27:0/3439197224 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f101c0a2d00 0x7f101c0a51c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.036+0000 7f10387c1640 1 --2- 172.21.15.27:0/3439197224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10301066b0 0x7f103019c840 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.036+0000 7f10387c1640 1 --2- 172.21.15.27:0/3439197224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1030100fd0 0x7f10301984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.036+0000 7f10387c1640 1 --2- 172.21.15.27:0/3439197224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1030100610 0x7f1030197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.036+0000 7f10387c1640 1 -- 172.21.15.27:0/3439197224 >> 172.21.15.27:0/3439197224 conn(0x7f1030077740 msgr2=0x7f10300ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.036+0000 7f10387c1640 1 -- 172.21.15.27:0/3439197224 shutdown_connections 2024-04-25T00:23:53.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.036+0000 7f10387c1640 1 -- 172.21.15.27:0/3439197224 wait complete. 2024-04-25T00:23:53.051 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T00:23:53.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.154+0000 7f95c5fb0640 1 -- 172.21.15.27:0/614136976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c0106630 msgr2=0x7f95c0106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.154+0000 7f95c5fb0640 1 --2- 172.21.15.27:0/614136976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c0106630 0x7f95c0106a10 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f95b0009cd0 tx=0x7f95b002a8e0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.155+0000 7f95c5fb0640 1 -- 172.21.15.27:0/614136976 shutdown_connections 2024-04-25T00:23:53.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.155+0000 7f95c5fb0640 1 --2- 172.21.15.27:0/614136976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95c0100fe0 0x7f95c010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.155+0000 7f95c5fb0640 1 --2- 172.21.15.27:0/614136976 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f95c0100620 0x7f95c0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.155+0000 7f95c5fb0640 1 --2- 172.21.15.27:0/614136976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c0106630 0x7f95c0106a10 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.155+0000 7f95c5fb0640 1 -- 172.21.15.27:0/614136976 >> 172.21.15.27:0/614136976 conn(0x7f95c00776a0 msgr2=0x7f95c0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.155+0000 7f95c5fb0640 1 -- 172.21.15.27:0/614136976 shutdown_connections 2024-04-25T00:23:53.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.155+0000 7f95c5fb0640 1 -- 172.21.15.27:0/614136976 wait complete. 2024-04-25T00:23:53.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.155+0000 7f95c5fb0640 1 Processor -- start 2024-04-25T00:23:53.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.155+0000 7f95c5fb0640 1 -- start start 2024-04-25T00:23:53.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95c5fb0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c0100620 0x7f95c01980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95c5fb0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95c0100fe0 0x7f95c0198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95c5fb0640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f95c0106630 0x7f95c019c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95c5fb0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f95c0110a00 con 0x7f95c0100620 2024-04-25T00:23:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95c5fb0640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f95c0110880 con 0x7f95c0106630 2024-04-25T00:23:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95c5fb0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f95c0110b80 con 0x7f95c0100fe0 2024-04-25T00:23:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c0100620 0x7f95c01980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c0100620 0x7f95c01980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50004/0 (socket says 172.21.15.27:50004) 2024-04-25T00:23:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95bf7fe640 1 -- 172.21.15.27:0/787724951 learned_addr learned my addr 172.21.15.27:0/787724951 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95beffd640 1 --2- 172.21.15.27:0/787724951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95c0100fe0 0x7f95c0198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95bffff640 1 --2- 172.21.15.27:0/787724951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f95c0106630 0x7f95c019c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95bf7fe640 1 -- 172.21.15.27:0/787724951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95c0100fe0 msgr2=0x7f95c0198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95bf7fe640 1 --2- 172.21.15.27:0/787724951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95c0100fe0 0x7f95c0198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95bf7fe640 1 -- 172.21.15.27:0/787724951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f95c0106630 msgr2=0x7f95c019c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95bf7fe640 1 --2- 172.21.15.27:0/787724951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f95c0106630 0x7f95c019c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95bf7fe640 1 -- 172.21.15.27:0/787724951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c019d0c0 con 0x7f95c0100620 2024-04-25T00:23:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95bf7fe640 1 --2- 172.21.15.27:0/787724951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c0100620 0x7f95c01980d0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f95b002ae00 tx=0x7f95b0003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95bcff9640 1 -- 172.21.15.27:0/787724951 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b0004040 con 0x7f95c0100620 2024-04-25T00:23:53.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95bcff9640 1 -- 172.21.15.27:0/787724951 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95b00046c0 con 0x7f95c0100620 2024-04-25T00:23:53.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95c5fb0640 1 -- 172.21.15.27:0/787724951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95c019d350 con 0x7f95c0100620 2024-04-25T00:23:53.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95bcff9640 1 -- 172.21.15.27:0/787724951 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b0046480 con 0x7f95c0100620 2024-04-25T00:23:53.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.156+0000 7f95c5fb0640 1 -- 172.21.15.27:0/787724951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95c01a4c40 con 0x7f95c0100620 2024-04-25T00:23:53.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.159+0000 7f95bcff9640 1 -- 172.21.15.27:0/787724951 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f95b00305a0 con 0x7f95c0100620 2024-04-25T00:23:53.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.159+0000 7f95c8d9f740 1 Processor -- start 2024-04-25T00:23:53.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.159+0000 7f95c8d9f740 1 -- start start 2024-04-25T00:23:53.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.159+0000 7f95bcff9640 1 --2- 172.21.15.27:0/787724951 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f95a80a2d00 0x7f95a80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.159+0000 7f95c8d9f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1ceb4480 0x55db1ceb4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.159+0000 7f95c8d9f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4da0 0x55db1cec3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.159+0000 7f95c8d9f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1cec3980 0x55db1cec5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.159+0000 7f95c8d9f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55db1ce19670 con 0x55db1cec3980 2024-04-25T00:23:53.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.159+0000 7f95c8d9f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55db1cdc1970 con 0x55db1ceb4da0 2024-04-25T00:23:53.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.159+0000 7f95c8d9f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55db1ce15630 con 0x55db1ceb4480 2024-04-25T00:23:53.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95bf7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1ceb4480 0x55db1ceb4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95bffff640 1 --2- 172.21.15.27:0/787724951 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f95a80a2d00 0x7f95a80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95bf7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1ceb4480 0x55db1ceb4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:51420/0 (socket says 172.21.15.27:51420) 2024-04-25T00:23:53.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95bf7fe640 1 -- 172.21.15.27:0/3925865039 learned_addr learned my addr 172.21.15.27:0/3925865039 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:53.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95beffd640 1 --2- 172.21.15.27:0/3925865039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4da0 0x55db1cec3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95bffff640 1 --2- 172.21.15.27:0/3925865039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1cec3980 0x55db1cec5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95bcff9640 1 -- 172.21.15.27:0/787724951 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f95b002e030 con 0x7f95c0100620 2024-04-25T00:23:53.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95beffd640 1 -- 172.21.15.27:0/3925865039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1ceb4480 msgr2=0x55db1ceb4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95beffd640 1 --2- 172.21.15.27:0/3925865039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1ceb4480 0x55db1ceb4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95beffd640 1 -- 172.21.15.27:0/3925865039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1cec3980 msgr2=0x55db1cec5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95beffd640 1 --2- 172.21.15.27:0/3925865039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1cec3980 0x55db1cec5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95beffd640 1 -- 172.21.15.27:0/3925865039 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db1cde5400 con 0x55db1ceb4da0 2024-04-25T00:23:53.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95bffff640 1 --2- 172.21.15.27:0/3925865039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1cec3980 0x55db1cec5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:53.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95bffff640 1 --2- 172.21.15.27:0/787724951 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f95a80a2d00 0x7f95a80a51c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f95b80058c0 tx=0x7f95b800ca70 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95beffd640 1 --2- 172.21.15.27:0/3925865039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4da0 0x55db1cec3440 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f95b400dbe0 tx=0x7f95b400f360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e51be3597f6a4184 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95a5ffb640 1 -- 172.21.15.27:0/3925865039 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b400fd30 con 0x55db1ceb4da0 2024-04-25T00:23:53.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95a5ffb640 1 -- 172.21.15.27:0/3925865039 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95b4005500 con 0x55db1ceb4da0 2024-04-25T00:23:53.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.160+0000 7f95a5ffb640 1 -- 172.21.15.27:0/3925865039 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b40116a0 con 0x55db1ceb4da0 2024-04-25T00:23:53.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.161+0000 7f95c8d9f740 1 -- 172.21.15.27:0/3925865039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4da0 msgr2=0x55db1cec3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.161+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/3925865039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4da0 0x55db1cec3440 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f95b400dbe0 tx=0x7f95b400f360 comp rx=0 tx=0).stop 2024-04-25T00:23:53.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.161+0000 7f95c8d9f740 1 -- 172.21.15.27:0/3925865039 shutdown_connections 2024-04-25T00:23:53.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.161+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/3925865039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1cec3980 0x55db1cec5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.161+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/3925865039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4da0 0x55db1cec3440 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.161+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/3925865039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1ceb4480 0x55db1ceb4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.161+0000 7f95c8d9f740 1 -- 172.21.15.27:0/3925865039 >> 172.21.15.27:0/3925865039 conn(0x55db1ceb08a0 msgr2=0x55db1ceb0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.161+0000 7f95c8d9f740 1 -- 172.21.15.27:0/3925865039 shutdown_connections 2024-04-25T00:23:53.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.161+0000 7f95c8d9f740 1 -- 172.21.15.27:0/3925865039 wait complete. 2024-04-25T00:23:53.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.161+0000 7f95c8d9f740 1 -- start start 2024-04-25T00:23:53.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95c8d9f740 1 Processor -- start 2024-04-25T00:23:53.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95c8d9f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4480 0x55db1cec32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95c8d9f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4da0 0x55db1cec05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95c8d9f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1cec3980 0x55db1cec0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95c8d9f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55db1cec6740 con 0x55db1ceb4da0 2024-04-25T00:23:53.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95c8d9f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55db1cdd5050 con 0x55db1ceb4480 2024-04-25T00:23:53.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95c8d9f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55db1cec68c0 con 0x55db1cec3980 2024-04-25T00:23:53.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95c8d9f740 10 client.0 _ll_register_callbacks cb 0x55db1cb2cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:53.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95beffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4da0 0x55db1cec05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95beffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4da0 0x55db1cec05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50024/0 (socket says 172.21.15.27:50024) 2024-04-25T00:23:53.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95beffd640 1 -- 172.21.15.27:0/904081433 learned_addr learned my addr 172.21.15.27:0/904081433 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:53.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95bffff640 1 --2- 172.21.15.27:0/904081433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1cec3980 0x55db1cec0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95bf7fe640 1 --2- 172.21.15.27:0/904081433 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4480 0x55db1cec32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95beffd640 1 -- 172.21.15.27:0/904081433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1cec3980 msgr2=0x55db1cec0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95beffd640 1 --2- 172.21.15.27:0/904081433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1cec3980 0x55db1cec0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95beffd640 1 -- 172.21.15.27:0/904081433 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4480 msgr2=0x55db1cec32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95beffd640 1 --2- 172.21.15.27:0/904081433 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4480 0x55db1cec32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95beffd640 1 -- 172.21.15.27:0/904081433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db1cda6dc0 con 0x55db1ceb4da0 2024-04-25T00:23:53.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95bffff640 1 --2- 172.21.15.27:0/904081433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1cec3980 0x55db1cec0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:23:53.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f95beffd640 1 --2- 172.21.15.27:0/904081433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4da0 0x55db1cec05c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f95b4004d10 tx=0x7f95b4004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27cce6dc5b5dfbb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.162+0000 7f959effd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:53.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.163+0000 7f959effd640 1 -- 172.21.15.27:0/904081433 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b400d7e0 con 0x55db1ceb4da0 2024-04-25T00:23:53.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.163+0000 7f959effd640 1 -- 172.21.15.27:0/904081433 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f95b400fd40 con 0x55db1ceb4da0 2024-04-25T00:23:53.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.163+0000 7f95c8d9f740 1 -- 172.21.15.27:0/904081433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95b4005500 con 0x55db1ceb4da0 2024-04-25T00:23:53.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.163+0000 7f959effd640 1 -- 172.21.15.27:0/904081433 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f95b4031050 con 0x55db1ceb4da0 2024-04-25T00:23:53.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.163+0000 7f95c8d9f740 10 client.16755 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:53.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.163+0000 7f95c8d9f740 1 -- 172.21.15.27:0/904081433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55db1cb2d3f0 con 0x55db1ceb4da0 2024-04-25T00:23:53.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.163+0000 7f959effd640 1 -- 172.21.15.27:0/904081433 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95b4005500 con 0x55db1ceb4da0 2024-04-25T00:23:53.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.163+0000 7f959effd640 1 -- 172.21.15.27:0/904081433 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f95b4003710 con 0x55db1ceb4da0 2024-04-25T00:23:53.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.163+0000 7f95c8d9f740 10 client.16755 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:53.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.163+0000 7f95c8d9f740 10 client.16755 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:23:53.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.163+0000 7f95c8d9f740 20 client.16755 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:53.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.163+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/904081433 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55db1ced3a00 0x55db1ced5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.163+0000 7f95c8d9f740 4 client.16755 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:23:53.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.163+0000 7f95c8d9f740 1 -- 172.21.15.27:0/904081433 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db1ce26bb0 con 0x55db1ced3a00 2024-04-25T00:23:53.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.164+0000 7f95bf7fe640 1 --2- 172.21.15.27:0/904081433 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55db1ced3a00 0x55db1ced5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.164+0000 7f95bf7fe640 1 --2- 172.21.15.27:0/904081433 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55db1ced3a00 0x55db1ced5df0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=be8388dce111777e server_cookie=8458dc5b3a74969a in_seq=0 out_seq=0 2024-04-25T00:23:53.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.164+0000 7f959effd640 10 client.16755 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:23:53.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.165+0000 7f959effd640 1 -- 172.21.15.27:0/904081433 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55db1ce26bb0 con 0x55db1ced3a00 2024-04-25T00:23:53.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.165+0000 7f959effd640 10 client.16755 handle_command_reply: tid=0 2024-04-25T00:23:53.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.165+0000 7f95c8d9f740 1 client.16755 shutdown 2024-04-25T00:23:53.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.165+0000 7f95c8d9f740 1 -- 172.21.15.27:0/904081433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4da0 msgr2=0x55db1cec05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.165+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/904081433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4da0 0x55db1cec05c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f95b4004d10 tx=0x7f95b4004a90 comp rx=0 tx=0).stop 2024-04-25T00:23:53.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.165+0000 7f95c8d9f740 1 -- 172.21.15.27:0/904081433 shutdown_connections 2024-04-25T00:23:53.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.165+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/904081433 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55db1ced3a00 0x55db1ced5df0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.165+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/904081433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1cec3980 0x55db1cec0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.165+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/904081433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4da0 0x55db1cec05c0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.165+0000 7f959effd640 0 client.16755 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:23:53.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.165+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/904081433 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4480 0x55db1cec32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.165+0000 7f95c8d9f740 1 -- 172.21.15.27:0/904081433 >> 172.21.15.27:0/904081433 conn(0x55db1ceb08a0 msgr2=0x55db1ceb6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.165+0000 7f95c8d9f740 1 -- 172.21.15.27:0/904081433 shutdown_connections 2024-04-25T00:23:53.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.165+0000 7f95c8d9f740 1 -- 172.21.15.27:0/904081433 wait complete. 2024-04-25T00:23:53.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.165+0000 7f95c8d9f740 20 client.16755 trim_cache size 0 max 16384 2024-04-25T00:23:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.178+0000 7f95c8d9f740 1 Processor -- start 2024-04-25T00:23:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.178+0000 7f95c8d9f740 1 -- start start 2024-04-25T00:23:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.178+0000 7f95c8d9f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4480 0x55db1ceb6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.178+0000 7f95c8d9f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4da0 0x55db1cec2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95c8d9f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1cec3980 0x55db1cedb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95c8d9f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f95c01ac8a0 con 0x55db1ceb4480 2024-04-25T00:23:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95c8d9f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f95c01ae440 con 0x55db1ceb4da0 2024-04-25T00:23:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95c8d9f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f95c01acf70 con 0x55db1cec3980 2024-04-25T00:23:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4480 0x55db1ceb6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4480 0x55db1ceb6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50030/0 (socket says 172.21.15.27:50030) 2024-04-25T00:23:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95bf7fe640 1 -- 172.21.15.27:0/4115564643 learned_addr learned my addr 172.21.15.27:0/4115564643 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95beffd640 1 --2- 172.21.15.27:0/4115564643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4da0 0x55db1cec2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95bffff640 1 --2- 172.21.15.27:0/4115564643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1cec3980 0x55db1cedb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95bf7fe640 1 -- 172.21.15.27:0/4115564643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1cec3980 msgr2=0x55db1cedb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95bf7fe640 1 --2- 172.21.15.27:0/4115564643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1cec3980 0x55db1cedb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95bf7fe640 1 -- 172.21.15.27:0/4115564643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4da0 msgr2=0x55db1cec2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95bf7fe640 1 --2- 172.21.15.27:0/4115564643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4da0 0x55db1cec2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95bf7fe640 1 -- 172.21.15.27:0/4115564643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db1cbf77d0 con 0x55db1ceb4480 2024-04-25T00:23:53.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95bf7fe640 1 --2- 172.21.15.27:0/4115564643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4480 0x55db1ceb6590 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f95b0000f80 tx=0x7f95b00dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeebfe60d366f240 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95a57fa640 1 -- 172.21.15.27:0/4115564643 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b00bd8f0 con 0x55db1ceb4480 2024-04-25T00:23:53.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95a57fa640 1 -- 172.21.15.27:0/4115564643 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95b0036040 con 0x55db1ceb4480 2024-04-25T00:23:53.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.179+0000 7f95a57fa640 1 -- 172.21.15.27:0/4115564643 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b00c5e60 con 0x55db1ceb4480 2024-04-25T00:23:53.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.180+0000 7f95c8d9f740 1 -- 172.21.15.27:0/4115564643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4480 msgr2=0x55db1ceb6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.180+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/4115564643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4480 0x55db1ceb6590 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f95b0000f80 tx=0x7f95b00dd040 comp rx=0 tx=0).stop 2024-04-25T00:23:53.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.180+0000 7f95c8d9f740 1 -- 172.21.15.27:0/4115564643 shutdown_connections 2024-04-25T00:23:53.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.180+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/4115564643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1cec3980 0x55db1cedb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.180+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/4115564643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4da0 0x55db1cec2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.180+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/4115564643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4480 0x55db1ceb6590 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.180+0000 7f95c8d9f740 1 -- 172.21.15.27:0/4115564643 >> 172.21.15.27:0/4115564643 conn(0x55db1ceb08a0 msgr2=0x55db1ceb5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.180+0000 7f95c8d9f740 1 -- 172.21.15.27:0/4115564643 shutdown_connections 2024-04-25T00:23:53.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.180+0000 7f95c8d9f740 1 -- 172.21.15.27:0/4115564643 wait complete. 2024-04-25T00:23:53.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.180+0000 7f95c8d9f740 1 -- start start 2024-04-25T00:23:53.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.180+0000 7f95c8d9f740 1 Processor -- start 2024-04-25T00:23:53.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95c8d9f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4480 0x55db1cebd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95c8d9f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4da0 0x55db1cebd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95c8d9f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1cec3980 0x55db1cebdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95c8d9f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f95c01add70 con 0x55db1ceb4480 2024-04-25T00:23:53.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95c8d9f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f95c01ad670 con 0x55db1ceb4da0 2024-04-25T00:23:53.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95c8d9f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f95c01aa830 con 0x55db1cec3980 2024-04-25T00:23:53.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95c8d9f740 10 client.0 _ll_register_callbacks cb 0x7f95b4015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:53.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4480 0x55db1cebd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4480 0x55db1cebd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50040/0 (socket says 172.21.15.27:50040) 2024-04-25T00:23:53.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95bf7fe640 1 -- 172.21.15.27:0/1620109054 learned_addr learned my addr 172.21.15.27:0/1620109054 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:53.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95beffd640 1 --2- 172.21.15.27:0/1620109054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4da0 0x55db1cebd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95bffff640 1 --2- 172.21.15.27:0/1620109054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1cec3980 0x55db1cebdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95bf7fe640 1 -- 172.21.15.27:0/1620109054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1cec3980 msgr2=0x55db1cebdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95bf7fe640 1 --2- 172.21.15.27:0/1620109054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1cec3980 0x55db1cebdc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95bf7fe640 1 -- 172.21.15.27:0/1620109054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4da0 msgr2=0x55db1cebd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95bf7fe640 1 --2- 172.21.15.27:0/1620109054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4da0 0x55db1cebd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95bf7fe640 1 -- 172.21.15.27:0/1620109054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db1cf38d50 con 0x55db1ceb4480 2024-04-25T00:23:53.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95beffd640 1 --2- 172.21.15.27:0/1620109054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4da0 0x55db1cebd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:23:53.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95bf7fe640 1 --2- 172.21.15.27:0/1620109054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4480 0x55db1cebd190 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f95b00bd180 tx=0x7f95b0003f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb97422520d11eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f959effd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:53.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f959effd640 1 -- 172.21.15.27:0/1620109054 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b00bd8f0 con 0x55db1ceb4480 2024-04-25T00:23:53.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f959effd640 1 -- 172.21.15.27:0/1620109054 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f95b00ee590 con 0x55db1ceb4480 2024-04-25T00:23:53.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.181+0000 7f95c8d9f740 1 -- 172.21.15.27:0/1620109054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95b0036040 con 0x55db1ceb4480 2024-04-25T00:23:53.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.182+0000 7f959effd640 1 -- 172.21.15.27:0/1620109054 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f95b00d9080 con 0x55db1ceb4480 2024-04-25T00:23:53.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.182+0000 7f95c8d9f740 10 client.16761 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:53.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.182+0000 7f95c8d9f740 1 -- 172.21.15.27:0/1620109054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55db1cf39f50 con 0x55db1ceb4480 2024-04-25T00:23:53.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.182+0000 7f959effd640 1 -- 172.21.15.27:0/1620109054 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95b0036040 con 0x55db1ceb4480 2024-04-25T00:23:53.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.182+0000 7f959effd640 1 -- 172.21.15.27:0/1620109054 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f95b004e070 con 0x55db1ceb4480 2024-04-25T00:23:53.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.182+0000 7f95c8d9f740 10 client.16761 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:53.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.182+0000 7f95c8d9f740 10 client.16761 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:23:53.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.182+0000 7f95c8d9f740 20 client.16761 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:53.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.182+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/1620109054 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55db1cf3a4e0 0x55db1cf3c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.182+0000 7f95c8d9f740 4 client.16761 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:23:53.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.182+0000 7f95c8d9f740 1 -- 172.21.15.27:0/1620109054 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55db1ce20f00 con 0x55db1cf3a4e0 2024-04-25T00:23:53.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.182+0000 7f95beffd640 1 --2- 172.21.15.27:0/1620109054 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55db1cf3a4e0 0x55db1cf3c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.183+0000 7f959effd640 10 client.16761 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:23:53.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.183+0000 7f95beffd640 1 --2- 172.21.15.27:0/1620109054 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55db1cf3a4e0 0x55db1cf3c900 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2896875e42c8e81a server_cookie=fef0591d2f17b2ab in_seq=0 out_seq=0 2024-04-25T00:23:53.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.183+0000 7f959effd640 1 -- 172.21.15.27:0/1620109054 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55db1ce20f00 con 0x55db1cf3a4e0 2024-04-25T00:23:53.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.183+0000 7f959effd640 10 client.16761 handle_command_reply: tid=0 2024-04-25T00:23:53.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.183+0000 7f95c8d9f740 1 client.16761 shutdown 2024-04-25T00:23:53.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.183+0000 7f95c8d9f740 1 -- 172.21.15.27:0/1620109054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4480 msgr2=0x55db1cebd190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.183+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/1620109054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4480 0x55db1cebd190 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f95b00bd180 tx=0x7f95b0003f70 comp rx=0 tx=0).stop 2024-04-25T00:23:53.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.183+0000 7f95c8d9f740 1 -- 172.21.15.27:0/1620109054 shutdown_connections 2024-04-25T00:23:53.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.183+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/1620109054 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55db1cf3a4e0 0x55db1cf3c900 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.183+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/1620109054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db1cec3980 0x55db1cebdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.183+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/1620109054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55db1ceb4da0 0x55db1cebd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.183+0000 7f959effd640 0 client.16761 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:23:53.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.183+0000 7f95c8d9f740 1 --2- 172.21.15.27:0/1620109054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1ceb4480 0x55db1cebd190 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.183+0000 7f95c8d9f740 1 -- 172.21.15.27:0/1620109054 >> 172.21.15.27:0/1620109054 conn(0x55db1ceb08a0 msgr2=0x55db1ceb6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.183+0000 7f95c8d9f740 1 -- 172.21.15.27:0/1620109054 shutdown_connections 2024-04-25T00:23:53.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.183+0000 7f95c8d9f740 1 -- 172.21.15.27:0/1620109054 wait complete. 2024-04-25T00:23:53.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.183+0000 7f95c8d9f740 20 client.16761 trim_cache size 0 max 16384 2024-04-25T00:23:53.201 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:23:53.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.184+0000 7f95c5fb0640 1 -- 172.21.15.27:0/787724951 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f95a80a2d00 msgr2=0x7f95a80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.184+0000 7f95c5fb0640 1 --2- 172.21.15.27:0/787724951 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f95a80a2d00 0x7f95a80a51c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f95b80058c0 tx=0x7f95b800ca70 comp rx=0 tx=0).stop 2024-04-25T00:23:53.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.184+0000 7f95c5fb0640 1 -- 172.21.15.27:0/787724951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c0100620 msgr2=0x7f95c01980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.184+0000 7f95c5fb0640 1 --2- 172.21.15.27:0/787724951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c0100620 0x7f95c01980d0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f95b002ae00 tx=0x7f95b0003970 comp rx=0 tx=0).stop 2024-04-25T00:23:53.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.184+0000 7f95c5fb0640 1 -- 172.21.15.27:0/787724951 shutdown_connections 2024-04-25T00:23:53.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.184+0000 7f95c5fb0640 1 --2- 172.21.15.27:0/787724951 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f95a80a2d00 0x7f95a80a51c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.184+0000 7f95c5fb0640 1 --2- 172.21.15.27:0/787724951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f95c0106630 0x7f95c019c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.184+0000 7f95c5fb0640 1 --2- 172.21.15.27:0/787724951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95c0100fe0 0x7f95c0198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.184+0000 7f95c5fb0640 1 --2- 172.21.15.27:0/787724951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c0100620 0x7f95c01980d0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.184+0000 7f95c5fb0640 1 -- 172.21.15.27:0/787724951 >> 172.21.15.27:0/787724951 conn(0x7f95c00776a0 msgr2=0x7f95c006a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.184+0000 7f95c5fb0640 1 -- 172.21.15.27:0/787724951 shutdown_connections 2024-04-25T00:23:53.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.184+0000 7f95c5fb0640 1 -- 172.21.15.27:0/787724951 wait complete. 2024-04-25T00:23:53.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T00:23:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.308+0000 7f7059163640 1 -- 172.21.15.27:0/497297339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705406a780 msgr2=0x7f705410e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.308+0000 7f7059163640 1 --2- 172.21.15.27:0/497297339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705406a780 0x7f705410e5f0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f704c009cd0 tx=0x7f704c02a8e0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.308+0000 7f7059163640 1 -- 172.21.15.27:0/497297339 shutdown_connections 2024-04-25T00:23:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.308+0000 7f7059163640 1 --2- 172.21.15.27:0/497297339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705406a780 0x7f705410e5f0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.308+0000 7f7059163640 1 --2- 172.21.15.27:0/497297339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7054069dc0 0x7f705406a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.308+0000 7f7059163640 1 --2- 172.21.15.27:0/497297339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7054107730 0x7f7054107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.308+0000 7f7059163640 1 -- 172.21.15.27:0/497297339 >> 172.21.15.27:0/497297339 conn(0x7f70540776a0 msgr2=0x7f7054077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.308+0000 7f7059163640 1 -- 172.21.15.27:0/497297339 shutdown_connections 2024-04-25T00:23:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.308+0000 7f7059163640 1 -- 172.21.15.27:0/497297339 wait complete. 2024-04-25T00:23:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.309+0000 7f7059163640 1 Processor -- start 2024-04-25T00:23:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.309+0000 7f7059163640 1 -- start start 2024-04-25T00:23:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.309+0000 7f7059163640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7054069dc0 0x7f7054197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.309+0000 7f7059163640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f705406a780 0x7f7054198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.309+0000 7f7059163640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7054107730 0x7f705419c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.309+0000 7f7059163640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7054111b00 con 0x7f7054069dc0 2024-04-25T00:23:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.309+0000 7f7059163640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7054111980 con 0x7f705406a780 2024-04-25T00:23:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.309+0000 7f7059163640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7054111c80 con 0x7f7054107730 2024-04-25T00:23:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.309+0000 7f7052d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7054069dc0 0x7f7054197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.309+0000 7f7052d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7054069dc0 0x7f7054197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50056/0 (socket says 172.21.15.27:50056) 2024-04-25T00:23:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.309+0000 7f7052d9d640 1 -- 172.21.15.27:0/1182441054 learned_addr learned my addr 172.21.15.27:0/1182441054 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.310+0000 7f704bfff640 1 --2- 172.21.15.27:0/1182441054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f705406a780 0x7f7054198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.310+0000 7f705359e640 1 --2- 172.21.15.27:0/1182441054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7054107730 0x7f705419c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.310+0000 7f7052d9d640 1 -- 172.21.15.27:0/1182441054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7054107730 msgr2=0x7f705419c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.310+0000 7f7052d9d640 1 --2- 172.21.15.27:0/1182441054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7054107730 0x7f705419c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.310+0000 7f7052d9d640 1 -- 172.21.15.27:0/1182441054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f705406a780 msgr2=0x7f7054198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.310+0000 7f7052d9d640 1 --2- 172.21.15.27:0/1182441054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f705406a780 0x7f7054198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.310+0000 7f7052d9d640 1 -- 172.21.15.27:0/1182441054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f705419cee0 con 0x7f7054069dc0 2024-04-25T00:23:53.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.310+0000 7f705359e640 1 --2- 172.21.15.27:0/1182441054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7054107730 0x7f705419c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:23:53.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.310+0000 7f704bfff640 1 --2- 172.21.15.27:0/1182441054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f705406a780 0x7f7054198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:23:53.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.310+0000 7f7052d9d640 1 --2- 172.21.15.27:0/1182441054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7054069dc0 0x7f7054197f40 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f704400c920 tx=0x7f704400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.310+0000 7f7050d99640 1 -- 172.21.15.27:0/1182441054 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044007c70 con 0x7f7054069dc0 2024-04-25T00:23:53.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.310+0000 7f7050d99640 1 -- 172.21.15.27:0/1182441054 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7044007e10 con 0x7f7054069dc0 2024-04-25T00:23:53.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.310+0000 7f7059163640 1 -- 172.21.15.27:0/1182441054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f705419d1d0 con 0x7f7054069dc0 2024-04-25T00:23:53.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.310+0000 7f7050d99640 1 -- 172.21.15.27:0/1182441054 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044018890 con 0x7f7054069dc0 2024-04-25T00:23:53.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.310+0000 7f7059163640 1 -- 172.21.15.27:0/1182441054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70541a4a60 con 0x7f7054069dc0 2024-04-25T00:23:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.312+0000 7f7050d99640 1 -- 172.21.15.27:0/1182441054 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7044018a70 con 0x7f7054069dc0 2024-04-25T00:23:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.312+0000 7f705be74740 1 Processor -- start 2024-04-25T00:23:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f705be74740 1 -- start start 2024-04-25T00:23:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f705be74740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844b42b0 0x5558844b4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f705be74740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844b4c60 0x5558844c3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f705be74740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844c3980 0x5558844c5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f705be74740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555884419670 con 0x5558844c3980 2024-04-25T00:23:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f705be74740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5558843c1970 con 0x5558844b4c60 2024-04-25T00:23:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f705be74740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555884415630 con 0x5558844b42b0 2024-04-25T00:23:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f704bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844c3980 0x5558844c5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f704bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844c3980 0x5558844c5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50058/0 (socket says 172.21.15.27:50058) 2024-04-25T00:23:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f704bfff640 1 -- 172.21.15.27:0/2618167634 learned_addr learned my addr 172.21.15.27:0/2618167634 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f7050d99640 1 --2- 172.21.15.27:0/1182441054 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f70400a2d80 0x7f70400a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f7052d9d640 1 --2- 172.21.15.27:0/2618167634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844b4c60 0x5558844c3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f705359e640 1 --2- 172.21.15.27:0/2618167634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844b42b0 0x5558844b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f704bfff640 1 -- 172.21.15.27:0/2618167634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844b42b0 msgr2=0x5558844b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f704bfff640 1 --2- 172.21.15.27:0/2618167634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844b42b0 0x5558844b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f705359e640 1 --2- 172.21.15.27:0/1182441054 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f70400a2d80 0x7f70400a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f704bfff640 1 -- 172.21.15.27:0/2618167634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844b4c60 msgr2=0x5558844c3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f704bfff640 1 --2- 172.21.15.27:0/2618167634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844b4c60 0x5558844c3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f704bfff640 1 -- 172.21.15.27:0/2618167634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558843e5400 con 0x5558844c3980 2024-04-25T00:23:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f7052d9d640 1 --2- 172.21.15.27:0/2618167634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844b4c60 0x5558844c3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:23:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f7050d99640 1 -- 172.21.15.27:0/1182441054 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f70440cf8c0 con 0x7f7054069dc0 2024-04-25T00:23:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.313+0000 7f704bfff640 1 --2- 172.21.15.27:0/2618167634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844c3980 0x5558844c5d70 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f703c009cf0 tx=0x7f703c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db3b639dc328170a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.314+0000 7f70497fa640 1 -- 172.21.15.27:0/2618167634 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c012450 con 0x5558844c3980 2024-04-25T00:23:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.314+0000 7f705359e640 1 --2- 172.21.15.27:0/1182441054 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f70400a2d80 0x7f70400a5240 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f704c004af0 tx=0x7f704c0095c0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.314+0000 7f70497fa640 1 -- 172.21.15.27:0/2618167634 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f703c012ad0 con 0x5558844c3980 2024-04-25T00:23:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.314+0000 7f70497fa640 1 -- 172.21.15.27:0/2618167634 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c00fc20 con 0x5558844c3980 2024-04-25T00:23:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.314+0000 7f705be74740 1 -- 172.21.15.27:0/2618167634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844c3980 msgr2=0x5558844c5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.314+0000 7f705be74740 1 --2- 172.21.15.27:0/2618167634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844c3980 0x5558844c5d70 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f703c009cf0 tx=0x7f703c0117b0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.314+0000 7f705be74740 1 -- 172.21.15.27:0/2618167634 shutdown_connections 2024-04-25T00:23:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.314+0000 7f705be74740 1 --2- 172.21.15.27:0/2618167634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844c3980 0x5558844c5d70 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.314+0000 7f705be74740 1 --2- 172.21.15.27:0/2618167634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844b4c60 0x5558844c3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.314+0000 7f705be74740 1 --2- 172.21.15.27:0/2618167634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844b42b0 0x5558844b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.314+0000 7f705be74740 1 -- 172.21.15.27:0/2618167634 >> 172.21.15.27:0/2618167634 conn(0x5558844b08a0 msgr2=0x5558844b0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.314+0000 7f705be74740 1 -- 172.21.15.27:0/2618167634 shutdown_connections 2024-04-25T00:23:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.315+0000 7f705be74740 1 -- 172.21.15.27:0/2618167634 wait complete. 2024-04-25T00:23:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.315+0000 7f705be74740 1 -- start start 2024-04-25T00:23:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.315+0000 7f705be74740 1 Processor -- start 2024-04-25T00:23:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.315+0000 7f705be74740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844b42b0 0x5558844c32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.315+0000 7f705be74740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844b4c60 0x5558844c05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.315+0000 7f705be74740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844c3980 0x5558844c0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.315+0000 7f705be74740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5558844c6740 con 0x5558844c3980 2024-04-25T00:23:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.315+0000 7f705be74740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5558843d5050 con 0x5558844b4c60 2024-04-25T00:23:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.315+0000 7f705be74740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5558844c68c0 con 0x5558844b42b0 2024-04-25T00:23:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.315+0000 7f705be74740 10 client.0 _ll_register_callbacks cb 0x55588412cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.315+0000 7f705359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844c3980 0x5558844c0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.315+0000 7f705359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844c3980 0x5558844c0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50068/0 (socket says 172.21.15.27:50068) 2024-04-25T00:23:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.315+0000 7f705359e640 1 -- 172.21.15.27:0/772966510 learned_addr learned my addr 172.21.15.27:0/772966510 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.315+0000 7f7052d9d640 1 --2- 172.21.15.27:0/772966510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844b42b0 0x5558844c32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.315+0000 7f704bfff640 1 --2- 172.21.15.27:0/772966510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844b4c60 0x5558844c05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.316+0000 7f705359e640 1 -- 172.21.15.27:0/772966510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844b42b0 msgr2=0x5558844c32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.316+0000 7f705359e640 1 --2- 172.21.15.27:0/772966510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844b42b0 0x5558844c32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.316+0000 7f705359e640 1 -- 172.21.15.27:0/772966510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844b4c60 msgr2=0x5558844c05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.316+0000 7f705359e640 1 --2- 172.21.15.27:0/772966510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844b4c60 0x5558844c05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.316+0000 7f705359e640 1 -- 172.21.15.27:0/772966510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558843a6dc0 con 0x5558844c3980 2024-04-25T00:23:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.316+0000 7f705359e640 1 --2- 172.21.15.27:0/772966510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844c3980 0x5558844c0b20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f704c005da0 tx=0x7f704c02a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23152896f7e3a7ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.316+0000 7f703a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.316+0000 7f703a7fc640 1 -- 172.21.15.27:0/772966510 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704c04d070 con 0x5558844c3980 2024-04-25T00:23:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.316+0000 7f703a7fc640 1 -- 172.21.15.27:0/772966510 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f704c004510 con 0x5558844c3980 2024-04-25T00:23:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.316+0000 7f705be74740 1 -- 172.21.15.27:0/772966510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f703c012ad0 con 0x5558844c3980 2024-04-25T00:23:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.316+0000 7f703a7fc640 1 -- 172.21.15.27:0/772966510 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f704c004710 con 0x5558844c3980 2024-04-25T00:23:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.316+0000 7f705be74740 10 client.16773 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.316+0000 7f705be74740 1 -- 172.21.15.27:0/772966510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55588412d3f0 con 0x5558844c3980 2024-04-25T00:23:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.316+0000 7f703a7fc640 1 -- 172.21.15.27:0/772966510 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f703c012ad0 con 0x5558844c3980 2024-04-25T00:23:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.317+0000 7f703a7fc640 1 -- 172.21.15.27:0/772966510 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f704c030070 con 0x5558844c3980 2024-04-25T00:23:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.317+0000 7f705be74740 10 client.16773 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.317+0000 7f705be74740 10 client.16773 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:23:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.317+0000 7f705be74740 20 client.16773 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.317+0000 7f705be74740 1 --2- 172.21.15.27:0/772966510 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5558844d3a10 0x5558844d5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.317+0000 7f705be74740 4 client.16773 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:23:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.317+0000 7f705be74740 1 -- 172.21.15.27:0/772966510 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555884426bb0 con 0x5558844d3a10 2024-04-25T00:23:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.317+0000 7f7052d9d640 1 --2- 172.21.15.27:0/772966510 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5558844d3a10 0x5558844d5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.317+0000 7f7052d9d640 1 --2- 172.21.15.27:0/772966510 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5558844d3a10 0x5558844d5e60 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da504844b033a35f server_cookie=eda197ff6181743e in_seq=0 out_seq=0 2024-04-25T00:23:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.317+0000 7f703a7fc640 10 client.16773 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.318+0000 7f703a7fc640 1 -- 172.21.15.27:0/772966510 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555884426bb0 con 0x5558844d3a10 2024-04-25T00:23:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.318+0000 7f703a7fc640 10 client.16773 handle_command_reply: tid=0 2024-04-25T00:23:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.318+0000 7f705be74740 1 client.16773 shutdown 2024-04-25T00:23:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.318+0000 7f705be74740 1 -- 172.21.15.27:0/772966510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844c3980 msgr2=0x5558844c0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.318+0000 7f705be74740 1 --2- 172.21.15.27:0/772966510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844c3980 0x5558844c0b20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f704c005da0 tx=0x7f704c02a8e0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.318+0000 7f705be74740 1 -- 172.21.15.27:0/772966510 shutdown_connections 2024-04-25T00:23:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.318+0000 7f705be74740 1 --2- 172.21.15.27:0/772966510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844c3980 0x5558844c0b20 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.318+0000 7f705be74740 1 --2- 172.21.15.27:0/772966510 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5558844d3a10 0x5558844d5e60 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.318+0000 7f705be74740 1 --2- 172.21.15.27:0/772966510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844b4c60 0x5558844c05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.318+0000 7f705be74740 1 --2- 172.21.15.27:0/772966510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844b42b0 0x5558844c32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.318+0000 7f703a7fc640 0 client.16773 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.318+0000 7f705be74740 1 -- 172.21.15.27:0/772966510 >> 172.21.15.27:0/772966510 conn(0x5558844b08a0 msgr2=0x5558844b66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.318+0000 7f705be74740 1 -- 172.21.15.27:0/772966510 shutdown_connections 2024-04-25T00:23:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.318+0000 7f705be74740 1 -- 172.21.15.27:0/772966510 wait complete. 2024-04-25T00:23:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.319+0000 7f705be74740 20 client.16773 trim_cache size 0 max 16384 2024-04-25T00:23:53.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.333+0000 7f705be74740 1 Processor -- start 2024-04-25T00:23:53.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.333+0000 7f705be74740 1 -- start start 2024-04-25T00:23:53.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.333+0000 7f705be74740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844b42b0 0x5558844cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.333+0000 7f705be74740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844b4c60 0x5558844c2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.333+0000 7f705be74740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844c3980 0x5558844db730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.333+0000 7f705be74740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f70541ac8a0 con 0x5558844b42b0 2024-04-25T00:23:53.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.333+0000 7f705be74740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f70541ae440 con 0x5558844b4c60 2024-04-25T00:23:53.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.333+0000 7f705be74740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f70541acf70 con 0x5558844c3980 2024-04-25T00:23:53.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.333+0000 7f7052d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844b42b0 0x5558844cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.333+0000 7f7052d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844b42b0 0x5558844cf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50080/0 (socket says 172.21.15.27:50080) 2024-04-25T00:23:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.333+0000 7f7052d9d640 1 -- 172.21.15.27:0/598003659 learned_addr learned my addr 172.21.15.27:0/598003659 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.334+0000 7f704bfff640 1 --2- 172.21.15.27:0/598003659 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844b4c60 0x5558844c2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.334+0000 7f705359e640 1 --2- 172.21.15.27:0/598003659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844c3980 0x5558844db730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.334+0000 7f7052d9d640 1 -- 172.21.15.27:0/598003659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844c3980 msgr2=0x5558844db730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.334+0000 7f7052d9d640 1 --2- 172.21.15.27:0/598003659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844c3980 0x5558844db730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.334+0000 7f7052d9d640 1 -- 172.21.15.27:0/598003659 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844b4c60 msgr2=0x5558844c2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.334+0000 7f7052d9d640 1 --2- 172.21.15.27:0/598003659 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844b4c60 0x5558844c2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.334+0000 7f7052d9d640 1 -- 172.21.15.27:0/598003659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558841f77d0 con 0x5558844b42b0 2024-04-25T00:23:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.334+0000 7f704bfff640 1 --2- 172.21.15.27:0/598003659 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844b4c60 0x5558844c2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.334+0000 7f7052d9d640 1 --2- 172.21.15.27:0/598003659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844b42b0 0x5558844cf670 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f70440ccd00 tx=0x7f70440d1d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5d0be82bfd79717 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.334+0000 7f7048ff9640 1 -- 172.21.15.27:0/598003659 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70440a46e0 con 0x5558844b42b0 2024-04-25T00:23:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.334+0000 7f7048ff9640 1 -- 172.21.15.27:0/598003659 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7044099d70 con 0x5558844b42b0 2024-04-25T00:23:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.334+0000 7f7048ff9640 1 -- 172.21.15.27:0/598003659 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70440ccf20 con 0x5558844b42b0 2024-04-25T00:23:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.335+0000 7f705be74740 1 -- 172.21.15.27:0/598003659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844b42b0 msgr2=0x5558844cf670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.335+0000 7f705be74740 1 --2- 172.21.15.27:0/598003659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844b42b0 0x5558844cf670 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f70440ccd00 tx=0x7f70440d1d50 comp rx=0 tx=0).stop 2024-04-25T00:23:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.335+0000 7f705be74740 1 -- 172.21.15.27:0/598003659 shutdown_connections 2024-04-25T00:23:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.335+0000 7f705be74740 1 --2- 172.21.15.27:0/598003659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844c3980 0x5558844db730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.335+0000 7f705be74740 1 --2- 172.21.15.27:0/598003659 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844b4c60 0x5558844c2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.335+0000 7f705be74740 1 --2- 172.21.15.27:0/598003659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844b42b0 0x5558844cf670 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.335+0000 7f705be74740 1 -- 172.21.15.27:0/598003659 >> 172.21.15.27:0/598003659 conn(0x5558844b08a0 msgr2=0x5558844bbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.335+0000 7f705be74740 1 -- 172.21.15.27:0/598003659 shutdown_connections 2024-04-25T00:23:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.335+0000 7f705be74740 1 -- 172.21.15.27:0/598003659 wait complete. 2024-04-25T00:23:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.335+0000 7f705be74740 1 -- start start 2024-04-25T00:23:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.335+0000 7f705be74740 1 Processor -- start 2024-04-25T00:23:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f705be74740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844b42b0 0x5558844bd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f705be74740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844b4c60 0x5558844bde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f705be74740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844c3980 0x555884538fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f705be74740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f70541add70 con 0x5558844b4c60 2024-04-25T00:23:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f705be74740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f70541ad670 con 0x5558844c3980 2024-04-25T00:23:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f705be74740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f70541aa830 con 0x5558844b42b0 2024-04-25T00:23:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f705be74740 10 client.0 _ll_register_callbacks cb 0x7f703c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f704bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844b4c60 0x5558844bde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f704bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844b4c60 0x5558844bde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50090/0 (socket says 172.21.15.27:50090) 2024-04-25T00:23:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f704bfff640 1 -- 172.21.15.27:0/1930312609 learned_addr learned my addr 172.21.15.27:0/1930312609 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f705359e640 1 --2- 172.21.15.27:0/1930312609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844c3980 0x555884538fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f7052d9d640 1 --2- 172.21.15.27:0/1930312609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844b42b0 0x5558844bd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f704bfff640 1 -- 172.21.15.27:0/1930312609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844b42b0 msgr2=0x5558844bd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f704bfff640 1 --2- 172.21.15.27:0/1930312609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844b42b0 0x5558844bd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f704bfff640 1 -- 172.21.15.27:0/1930312609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844c3980 msgr2=0x555884538fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f704bfff640 1 --2- 172.21.15.27:0/1930312609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844c3980 0x555884538fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f704bfff640 1 -- 172.21.15.27:0/1930312609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558845394e0 con 0x5558844b4c60 2024-04-25T00:23:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f704bfff640 1 --2- 172.21.15.27:0/1930312609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844b4c60 0x5558844bde30 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x5558844b1a30 tx=0x7f703c0099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e5b15cac284fb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.336+0000 7f703a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.337+0000 7f703a7fc640 1 -- 172.21.15.27:0/1930312609 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c029070 con 0x5558844b4c60 2024-04-25T00:23:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.337+0000 7f703a7fc640 1 -- 172.21.15.27:0/1930312609 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f703c0053c0 con 0x5558844b4c60 2024-04-25T00:23:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.337+0000 7f705be74740 1 -- 172.21.15.27:0/1930312609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7044099d70 con 0x5558844b4c60 2024-04-25T00:23:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.337+0000 7f703a7fc640 1 -- 172.21.15.27:0/1930312609 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f703c0055c0 con 0x5558844b4c60 2024-04-25T00:23:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.337+0000 7f705be74740 10 client.16779 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.337+0000 7f705be74740 1 -- 172.21.15.27:0/1930312609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55588453a740 con 0x5558844b4c60 2024-04-25T00:23:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.337+0000 7f703a7fc640 1 -- 172.21.15.27:0/1930312609 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7044099d70 con 0x5558844b4c60 2024-04-25T00:23:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.337+0000 7f703a7fc640 1 -- 172.21.15.27:0/1930312609 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f703c003c50 con 0x5558844b4c60 2024-04-25T00:23:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.337+0000 7f705be74740 10 client.16779 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.337+0000 7f705be74740 10 client.16779 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:23:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.337+0000 7f705be74740 20 client.16779 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.337+0000 7f705be74740 1 --2- 172.21.15.27:0/1930312609 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55588453aae0 0x55588453cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.337+0000 7f705be74740 4 client.16779 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:23:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.337+0000 7f705be74740 1 -- 172.21.15.27:0/1930312609 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555884420f00 con 0x55588453aae0 2024-04-25T00:23:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.338+0000 7f7052d9d640 1 --2- 172.21.15.27:0/1930312609 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55588453aae0 0x55588453cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.338+0000 7f7052d9d640 1 --2- 172.21.15.27:0/1930312609 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55588453aae0 0x55588453cf00 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=977b4c5561b63824 server_cookie=7f3e6829b75964e8 in_seq=0 out_seq=0 2024-04-25T00:23:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.338+0000 7f703a7fc640 10 client.16779 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.338+0000 7f703a7fc640 1 -- 172.21.15.27:0/1930312609 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555884420f00 con 0x55588453aae0 2024-04-25T00:23:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.338+0000 7f703a7fc640 10 client.16779 handle_command_reply: tid=0 2024-04-25T00:23:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.338+0000 7f705be74740 1 client.16779 shutdown 2024-04-25T00:23:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.338+0000 7f705be74740 1 -- 172.21.15.27:0/1930312609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844b4c60 msgr2=0x5558844bde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.338+0000 7f705be74740 1 --2- 172.21.15.27:0/1930312609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844b4c60 0x5558844bde30 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x5558844b1a30 tx=0x7f703c0099f0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.338+0000 7f705be74740 1 -- 172.21.15.27:0/1930312609 shutdown_connections 2024-04-25T00:23:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.338+0000 7f705be74740 1 --2- 172.21.15.27:0/1930312609 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55588453aae0 0x55588453cf00 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.338+0000 7f705be74740 1 --2- 172.21.15.27:0/1930312609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5558844c3980 0x555884538fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.338+0000 7f705be74740 1 --2- 172.21.15.27:0/1930312609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558844b4c60 0x5558844bde30 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.338+0000 7f705be74740 1 --2- 172.21.15.27:0/1930312609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558844b42b0 0x5558844bd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.338+0000 7f703a7fc640 0 client.16779 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.338+0000 7f705be74740 1 -- 172.21.15.27:0/1930312609 >> 172.21.15.27:0/1930312609 conn(0x5558844b08a0 msgr2=0x5558844b5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.338+0000 7f705be74740 1 -- 172.21.15.27:0/1930312609 shutdown_connections 2024-04-25T00:23:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.339+0000 7f705be74740 1 -- 172.21.15.27:0/1930312609 wait complete. 2024-04-25T00:23:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.339+0000 7f705be74740 20 client.16779 trim_cache size 0 max 16384 2024-04-25T00:23:53.345 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:23:53.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.339+0000 7f7059163640 1 -- 172.21.15.27:0/1182441054 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f70400a2d80 msgr2=0x7f70400a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.339+0000 7f7059163640 1 --2- 172.21.15.27:0/1182441054 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f70400a2d80 0x7f70400a5240 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f704c004af0 tx=0x7f704c0095c0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.339+0000 7f7059163640 1 -- 172.21.15.27:0/1182441054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7054069dc0 msgr2=0x7f7054197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:53.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.339+0000 7f7059163640 1 --2- 172.21.15.27:0/1182441054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7054069dc0 0x7f7054197f40 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f704400c920 tx=0x7f704400cde0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.339+0000 7f7059163640 1 -- 172.21.15.27:0/1182441054 shutdown_connections 2024-04-25T00:23:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.339+0000 7f7059163640 1 --2- 172.21.15.27:0/1182441054 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f70400a2d80 0x7f70400a5240 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.339+0000 7f7059163640 1 --2- 172.21.15.27:0/1182441054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7054107730 0x7f705419c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.339+0000 7f7059163640 1 --2- 172.21.15.27:0/1182441054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f705406a780 0x7f7054198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.339+0000 7f7059163640 1 --2- 172.21.15.27:0/1182441054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7054069dc0 0x7f7054197f40 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.339+0000 7f7059163640 1 -- 172.21.15.27:0/1182441054 >> 172.21.15.27:0/1182441054 conn(0x7f70540776a0 msgr2=0x7f70540fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.339+0000 7f7059163640 1 -- 172.21.15.27:0/1182441054 shutdown_connections 2024-04-25T00:23:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:53.339+0000 7f7059163640 1 -- 172.21.15.27:0/1182441054 wait complete. 2024-04-25T00:23:54.353 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T00:23:54.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:23:54.354 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T00:23:54.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.472+0000 7fc24d038640 1 -- 172.21.15.27:0/3402551004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc248100620 msgr2=0x7fc248100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.472+0000 7fc24d038640 1 --2- 172.21.15.27:0/3402551004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc248100620 0x7fc248100aa0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc230009930 tx=0x7fc23002a420 comp rx=0 tx=0).stop 2024-04-25T00:23:54.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.472+0000 7fc24d038640 1 -- 172.21.15.27:0/3402551004 shutdown_connections 2024-04-25T00:23:54.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.472+0000 7fc24d038640 1 --2- 172.21.15.27:0/3402551004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc248100fe0 0x7fc24810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.472+0000 7fc24d038640 1 --2- 172.21.15.27:0/3402551004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc248100620 0x7fc248100aa0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.472+0000 7fc24d038640 1 --2- 172.21.15.27:0/3402551004 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc248106630 0x7fc248106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.472+0000 7fc24d038640 1 -- 172.21.15.27:0/3402551004 >> 172.21.15.27:0/3402551004 conn(0x7fc2480776a0 msgr2=0x7fc248077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:54.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.472+0000 7fc24d038640 1 -- 172.21.15.27:0/3402551004 shutdown_connections 2024-04-25T00:23:54.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.472+0000 7fc24d038640 1 -- 172.21.15.27:0/3402551004 wait complete. 2024-04-25T00:23:54.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.473+0000 7fc24d038640 1 Processor -- start 2024-04-25T00:23:54.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.473+0000 7fc24d038640 1 -- start start 2024-04-25T00:23:54.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.473+0000 7fc24d038640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc248100620 0x7fc2481980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.473+0000 7fc24d038640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc248100fe0 0x7fc248198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.473+0000 7fc24d038640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc248106630 0x7fc24819c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.473+0000 7fc24d038640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc248110a40 con 0x7fc248100fe0 2024-04-25T00:23:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.473+0000 7fc24d038640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc2481108c0 con 0x7fc248100620 2024-04-25T00:23:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.473+0000 7fc24d038640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc248110bc0 con 0x7fc248106630 2024-04-25T00:23:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.473+0000 7fc24659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc248100fe0 0x7fc248198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.473+0000 7fc24659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc248100fe0 0x7fc248198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50100/0 (socket says 172.21.15.27:50100) 2024-04-25T00:23:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.473+0000 7fc24659c640 1 -- 172.21.15.27:0/2948960064 learned_addr learned my addr 172.21.15.27:0/2948960064 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.474+0000 7fc246d9d640 1 --2- 172.21.15.27:0/2948960064 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc248100620 0x7fc2481980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.474+0000 7fc24759e640 1 --2- 172.21.15.27:0/2948960064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc248106630 0x7fc24819c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.474+0000 7fc24659c640 1 -- 172.21.15.27:0/2948960064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc248106630 msgr2=0x7fc24819c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.474+0000 7fc24659c640 1 --2- 172.21.15.27:0/2948960064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc248106630 0x7fc24819c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.474+0000 7fc24659c640 1 -- 172.21.15.27:0/2948960064 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc248100620 msgr2=0x7fc2481980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.474+0000 7fc24659c640 1 --2- 172.21.15.27:0/2948960064 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc248100620 0x7fc2481980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.474+0000 7fc24659c640 1 -- 172.21.15.27:0/2948960064 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24819d0c0 con 0x7fc248100fe0 2024-04-25T00:23:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.474+0000 7fc246d9d640 1 --2- 172.21.15.27:0/2948960064 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc248100620 0x7fc2481980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:23:54.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.474+0000 7fc24659c640 1 --2- 172.21.15.27:0/2948960064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc248100fe0 0x7fc248198610 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc230009230 tx=0x7fc230004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:54.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.474+0000 7fc23ffff640 1 -- 172.21.15.27:0/2948960064 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23003f030 con 0x7fc248100fe0 2024-04-25T00:23:54.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.474+0000 7fc24d038640 1 -- 172.21.15.27:0/2948960064 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc24819d350 con 0x7fc248100fe0 2024-04-25T00:23:54.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.474+0000 7fc23ffff640 1 -- 172.21.15.27:0/2948960064 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc230003ad0 con 0x7fc248100fe0 2024-04-25T00:23:54.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.474+0000 7fc23ffff640 1 -- 172.21.15.27:0/2948960064 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc230035070 con 0x7fc248100fe0 2024-04-25T00:23:54.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.474+0000 7fc24d038640 1 -- 172.21.15.27:0/2948960064 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2481a4c40 con 0x7fc248100fe0 2024-04-25T00:23:54.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.477+0000 7fc23ffff640 1 -- 172.21.15.27:0/2948960064 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc230042030 con 0x7fc248100fe0 2024-04-25T00:23:54.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.477+0000 7fc24fe1e740 1 Processor -- start 2024-04-25T00:23:54.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.477+0000 7fc24fe1e740 1 -- start start 2024-04-25T00:23:54.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc24fe1e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559958705980 0x559958703cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc24fe1e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559958704290 0x559958704710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc24fe1e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55995870f010 0x559958711400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc24fe1e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5599586682c0 con 0x55995870f010 2024-04-25T00:23:54.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc24fe1e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x559958610910 con 0x559958704290 2024-04-25T00:23:54.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc24fe1e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559958664280 con 0x559958705980 2024-04-25T00:23:54.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc24659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55995870f010 0x559958711400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc24659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55995870f010 0x559958711400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50106/0 (socket says 172.21.15.27:50106) 2024-04-25T00:23:54.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc24659c640 1 -- 172.21.15.27:0/2604585223 learned_addr learned my addr 172.21.15.27:0/2604585223 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:54.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc23ffff640 1 --2- 172.21.15.27:0/2948960064 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc2340a2d00 0x7fc2340a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc24759e640 1 --2- 172.21.15.27:0/2604585223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559958705980 0x559958703cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc246d9d640 1 --2- 172.21.15.27:0/2604585223 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559958704290 0x559958704710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc24759e640 1 --2- 172.21.15.27:0/2948960064 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc2340a2d00 0x7fc2340a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc23ffff640 1 -- 172.21.15.27:0/2948960064 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fc2300efeb0 con 0x7fc248100fe0 2024-04-25T00:23:54.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc24659c640 1 -- 172.21.15.27:0/2604585223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559958705980 msgr2=0x559958703cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:54.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc24659c640 1 --2- 172.21.15.27:0/2604585223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559958705980 0x559958703cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc24659c640 1 -- 172.21.15.27:0/2604585223 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559958704290 msgr2=0x559958704710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:54.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc24659c640 1 --2- 172.21.15.27:0/2604585223 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559958704290 0x559958704710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.478+0000 7fc24659c640 1 -- 172.21.15.27:0/2604585223 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559958621fb0 con 0x55995870f010 2024-04-25T00:23:54.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.479+0000 7fc24659c640 1 --2- 172.21.15.27:0/2604585223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55995870f010 0x559958711400 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc2300ed1d0 tx=0x7fc230009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30401f8b42ae359e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:54.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.479+0000 7fc24759e640 1 --2- 172.21.15.27:0/2948960064 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc2340a2d00 0x7fc2340a51c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc238005c40 tx=0x7fc238009500 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:54.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.479+0000 7fc23d7fa640 1 -- 172.21.15.27:0/2604585223 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc230049070 con 0x55995870f010 2024-04-25T00:23:54.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.479+0000 7fc23d7fa640 1 -- 172.21.15.27:0/2604585223 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2300bc760 con 0x55995870f010 2024-04-25T00:23:54.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.479+0000 7fc23d7fa640 1 -- 172.21.15.27:0/2604585223 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2300b86d0 con 0x55995870f010 2024-04-25T00:23:54.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.480+0000 7fc24fe1e740 1 -- 172.21.15.27:0/2604585223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55995870f010 msgr2=0x559958711400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.480+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/2604585223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55995870f010 0x559958711400 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc2300ed1d0 tx=0x7fc230009260 comp rx=0 tx=0).stop 2024-04-25T00:23:54.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.480+0000 7fc24fe1e740 1 -- 172.21.15.27:0/2604585223 shutdown_connections 2024-04-25T00:23:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.480+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/2604585223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55995870f010 0x559958711400 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.480+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/2604585223 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559958704290 0x559958704710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.480+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/2604585223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559958705980 0x559958703cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.480+0000 7fc24fe1e740 1 -- 172.21.15.27:0/2604585223 >> 172.21.15.27:0/2604585223 conn(0x5599586cc950 msgr2=0x5599587007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.480+0000 7fc24fe1e740 1 -- 172.21.15.27:0/2604585223 shutdown_connections 2024-04-25T00:23:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.480+0000 7fc24fe1e740 1 -- 172.21.15.27:0/2604585223 wait complete. 2024-04-25T00:23:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.480+0000 7fc24fe1e740 1 -- start start 2024-04-25T00:23:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.480+0000 7fc24fe1e740 1 Processor -- start 2024-04-25T00:23:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.480+0000 7fc24fe1e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559958704290 0x55995870bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.480+0000 7fc24fe1e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958705980 0x55995870c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.480+0000 7fc24fe1e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55995870f010 0x55995870c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.480+0000 7fc24fe1e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5599586682c0 con 0x559958705980 2024-04-25T00:23:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.480+0000 7fc24fe1e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x559958623e90 con 0x559958704290 2024-04-25T00:23:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.480+0000 7fc24fe1e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559958711c60 con 0x55995870f010 2024-04-25T00:23:54.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc24fe1e740 10 client.0 _ll_register_callbacks cb 0x55995865f970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:54.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc24659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958705980 0x55995870c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc24659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958705980 0x55995870c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50114/0 (socket says 172.21.15.27:50114) 2024-04-25T00:23:54.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc24659c640 1 -- 172.21.15.27:0/4017426848 learned_addr learned my addr 172.21.15.27:0/4017426848 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:54.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc246d9d640 1 --2- 172.21.15.27:0/4017426848 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559958704290 0x55995870bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc24759e640 1 --2- 172.21.15.27:0/4017426848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55995870f010 0x55995870c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc24659c640 1 -- 172.21.15.27:0/4017426848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55995870f010 msgr2=0x55995870c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:54.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc24659c640 1 --2- 172.21.15.27:0/4017426848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55995870f010 0x55995870c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc24659c640 1 -- 172.21.15.27:0/4017426848 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559958704290 msgr2=0x55995870bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:54.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc24659c640 1 --2- 172.21.15.27:0/4017426848 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559958704290 0x55995870bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc24659c640 1 -- 172.21.15.27:0/4017426848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599584447e0 con 0x559958705980 2024-04-25T00:23:54.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc24659c640 1 --2- 172.21.15.27:0/4017426848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958705980 0x55995870c460 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fc2300bbff0 tx=0x7fc23003bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d1e72399f4b230 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:54.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc22e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:54.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc22e7fc640 1 -- 172.21.15.27:0/4017426848 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc230049050 con 0x559958705980 2024-04-25T00:23:54.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc22e7fc640 1 -- 172.21.15.27:0/4017426848 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fc2300c00c0 con 0x559958705980 2024-04-25T00:23:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc24fe1e740 1 -- 172.21.15.27:0/4017426848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2300bc760 con 0x559958705980 2024-04-25T00:23:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc22e7fc640 1 -- 172.21.15.27:0/4017426848 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fc2300bc060 con 0x559958705980 2024-04-25T00:23:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc24fe1e740 10 client.16791 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.481+0000 7fc24fe1e740 1 -- 172.21.15.27:0/4017426848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5599584a4f70 con 0x559958705980 2024-04-25T00:23:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.482+0000 7fc22e7fc640 1 -- 172.21.15.27:0/4017426848 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2300bc760 con 0x559958705980 2024-04-25T00:23:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.482+0000 7fc22e7fc640 1 -- 172.21.15.27:0/4017426848 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7fc230003c80 con 0x559958705980 2024-04-25T00:23:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.482+0000 7fc24fe1e740 10 client.16791 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.482+0000 7fc24fe1e740 10 client.16791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:23:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.482+0000 7fc24fe1e740 20 client.16791 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.482+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/4017426848 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55995871ec00 0x559958720ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.482+0000 7fc24fe1e740 4 client.16791 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:23:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.482+0000 7fc24fe1e740 1 -- 172.21.15.27:0/4017426848 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55995863c5a0 con 0x55995871ec00 2024-04-25T00:23:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.482+0000 7fc246d9d640 1 --2- 172.21.15.27:0/4017426848 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55995871ec00 0x559958720ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.483+0000 7fc246d9d640 1 --2- 172.21.15.27:0/4017426848 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55995871ec00 0x559958720ff0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bdfd6795975f2ee server_cookie=91d918e2407f7a1f in_seq=0 out_seq=0 2024-04-25T00:23:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.483+0000 7fc22e7fc640 10 client.16791 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.483+0000 7fc22e7fc640 1 -- 172.21.15.27:0/4017426848 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55995863c5a0 con 0x55995871ec00 2024-04-25T00:23:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.483+0000 7fc22e7fc640 10 client.16791 handle_command_reply: tid=0 2024-04-25T00:23:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.483+0000 7fc24fe1e740 1 client.16791 shutdown 2024-04-25T00:23:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.484+0000 7fc24fe1e740 1 -- 172.21.15.27:0/4017426848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958705980 msgr2=0x55995870c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.484+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/4017426848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958705980 0x55995870c460 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fc2300bbff0 tx=0x7fc23003bf70 comp rx=0 tx=0).stop 2024-04-25T00:23:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.484+0000 7fc24fe1e740 1 -- 172.21.15.27:0/4017426848 shutdown_connections 2024-04-25T00:23:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.484+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/4017426848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55995870f010 0x55995870c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.484+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/4017426848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958705980 0x55995870c460 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.484+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/4017426848 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55995871ec00 0x559958720ff0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.484+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/4017426848 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559958704290 0x55995870bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.484+0000 7fc22e7fc640 0 client.16791 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.484+0000 7fc24fe1e740 1 -- 172.21.15.27:0/4017426848 >> 172.21.15.27:0/4017426848 conn(0x5599586cc950 msgr2=0x559958706660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.484+0000 7fc24fe1e740 1 -- 172.21.15.27:0/4017426848 shutdown_connections 2024-04-25T00:23:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.484+0000 7fc24fe1e740 1 -- 172.21.15.27:0/4017426848 wait complete. 2024-04-25T00:23:54.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.484+0000 7fc24fe1e740 20 client.16791 trim_cache size 0 max 16384 2024-04-25T00:23:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.502+0000 7fc24fe1e740 1 Processor -- start 2024-04-25T00:23:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.502+0000 7fc24fe1e740 1 -- start start 2024-04-25T00:23:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.502+0000 7fc24fe1e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958704290 0x55995871aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.502+0000 7fc24fe1e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559958705980 0x55995870c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.502+0000 7fc24fe1e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55995870f010 0x559958726850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.502+0000 7fc24fe1e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc2481aeba0 con 0x559958704290 2024-04-25T00:23:54.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.502+0000 7fc24fe1e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc2481b0740 con 0x559958705980 2024-04-25T00:23:54.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.502+0000 7fc24fe1e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc2481af270 con 0x55995870f010 2024-04-25T00:23:54.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.502+0000 7fc246d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958704290 0x55995871aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.502+0000 7fc246d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958704290 0x55995871aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50124/0 (socket says 172.21.15.27:50124) 2024-04-25T00:23:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.502+0000 7fc246d9d640 1 -- 172.21.15.27:0/3840412921 learned_addr learned my addr 172.21.15.27:0/3840412921 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.503+0000 7fc24759e640 1 --2- 172.21.15.27:0/3840412921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55995870f010 0x559958726850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.503+0000 7fc24659c640 1 --2- 172.21.15.27:0/3840412921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559958705980 0x55995870c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.503+0000 7fc246d9d640 1 -- 172.21.15.27:0/3840412921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55995870f010 msgr2=0x559958726850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.503+0000 7fc246d9d640 1 --2- 172.21.15.27:0/3840412921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55995870f010 0x559958726850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.503+0000 7fc246d9d640 1 -- 172.21.15.27:0/3840412921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559958705980 msgr2=0x55995870c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.503+0000 7fc246d9d640 1 --2- 172.21.15.27:0/3840412921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559958705980 0x55995870c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.503+0000 7fc246d9d640 1 -- 172.21.15.27:0/3840412921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55995833c170 con 0x559958704290 2024-04-25T00:23:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.503+0000 7fc24659c640 1 --2- 172.21.15.27:0/3840412921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559958705980 0x55995870c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:23:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.503+0000 7fc246d9d640 1 --2- 172.21.15.27:0/3840412921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958704290 0x55995871aa10 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x5599587217f0 tx=0x7fc2400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58cc1fd41f83751 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.503+0000 7fc23cff9640 1 -- 172.21.15.27:0/3840412921 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc240023070 con 0x559958704290 2024-04-25T00:23:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.503+0000 7fc23cff9640 1 -- 172.21.15.27:0/3840412921 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc24001d430 con 0x559958704290 2024-04-25T00:23:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.503+0000 7fc23cff9640 1 -- 172.21.15.27:0/3840412921 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc240003a80 con 0x559958704290 2024-04-25T00:23:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.504+0000 7fc24fe1e740 1 -- 172.21.15.27:0/3840412921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958704290 msgr2=0x55995871aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.504+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/3840412921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958704290 0x55995871aa10 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x5599587217f0 tx=0x7fc2400046c0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.504+0000 7fc24fe1e740 1 -- 172.21.15.27:0/3840412921 shutdown_connections 2024-04-25T00:23:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.504+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/3840412921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55995870f010 0x559958726850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.504+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/3840412921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559958705980 0x55995870c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.504+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/3840412921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958704290 0x55995871aa10 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.504+0000 7fc24fe1e740 1 -- 172.21.15.27:0/3840412921 >> 172.21.15.27:0/3840412921 conn(0x5599586cc950 msgr2=0x5599586ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.504+0000 7fc24fe1e740 1 -- 172.21.15.27:0/3840412921 shutdown_connections 2024-04-25T00:23:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.504+0000 7fc24fe1e740 1 -- 172.21.15.27:0/3840412921 wait complete. 2024-04-25T00:23:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.504+0000 7fc24fe1e740 1 -- start start 2024-04-25T00:23:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.504+0000 7fc24fe1e740 1 Processor -- start 2024-04-25T00:23:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc24fe1e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958704290 0x559958712a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc24fe1e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559958705980 0x559958712f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc24fe1e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55995870f010 0x5599587134b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc24fe1e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc2481b0070 con 0x559958704290 2024-04-25T00:23:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc24fe1e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc2481af970 con 0x55995870f010 2024-04-25T00:23:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc24fe1e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc2481acb30 con 0x559958705980 2024-04-25T00:23:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc24fe1e740 10 client.0 _ll_register_callbacks cb 0x7fc24000b420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc246d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958704290 0x559958712a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc246d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958704290 0x559958712a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50140/0 (socket says 172.21.15.27:50140) 2024-04-25T00:23:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc246d9d640 1 -- 172.21.15.27:0/2721629988 learned_addr learned my addr 172.21.15.27:0/2721629988 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:54.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc24659c640 1 --2- 172.21.15.27:0/2721629988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559958705980 0x559958712f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc246d9d640 1 -- 172.21.15.27:0/2721629988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559958705980 msgr2=0x559958712f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc246d9d640 1 --2- 172.21.15.27:0/2721629988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559958705980 0x559958712f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc24759e640 1 --2- 172.21.15.27:0/2721629988 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55995870f010 0x5599587134b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc246d9d640 1 -- 172.21.15.27:0/2721629988 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55995870f010 msgr2=0x5599587134b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc246d9d640 1 --2- 172.21.15.27:0/2721629988 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55995870f010 0x5599587134b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc246d9d640 1 -- 172.21.15.27:0/2721629988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599587841c0 con 0x559958704290 2024-04-25T00:23:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc246d9d640 1 --2- 172.21.15.27:0/2721629988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958704290 0x559958712a30 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fc24000be00 tx=0x7fc240013260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73feecfbfaf73412 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc22e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.505+0000 7fc22e7fc640 1 -- 172.21.15.27:0/2721629988 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc240023040 con 0x559958704290 2024-04-25T00:23:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.506+0000 7fc22e7fc640 1 -- 172.21.15.27:0/2721629988 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fc24001d660 con 0x559958704290 2024-04-25T00:23:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.506+0000 7fc24fe1e740 1 -- 172.21.15.27:0/2721629988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc24001d430 con 0x559958704290 2024-04-25T00:23:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.506+0000 7fc22e7fc640 1 -- 172.21.15.27:0/2721629988 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fc240035050 con 0x559958704290 2024-04-25T00:23:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.506+0000 7fc24fe1e740 10 client.16797 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.506+0000 7fc24fe1e740 1 -- 172.21.15.27:0/2721629988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x559958785450 con 0x559958704290 2024-04-25T00:23:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.506+0000 7fc22e7fc640 1 -- 172.21.15.27:0/2721629988 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc24001d430 con 0x559958704290 2024-04-25T00:23:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.506+0000 7fc22e7fc640 1 -- 172.21.15.27:0/2721629988 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7fc240004c90 con 0x559958704290 2024-04-25T00:23:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.506+0000 7fc24fe1e740 10 client.16797 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.506+0000 7fc24fe1e740 10 client.16797 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:23:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.506+0000 7fc24fe1e740 20 client.16797 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.506+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/2721629988 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x559958785b60 0x559958787f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.506+0000 7fc24fe1e740 4 client.16797 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T00:23:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.506+0000 7fc24fe1e740 1 -- 172.21.15.27:0/2721629988 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559958675800 con 0x559958785b60 2024-04-25T00:23:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.507+0000 7fc24659c640 1 --2- 172.21.15.27:0/2721629988 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x559958785b60 0x559958787f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.507+0000 7fc24659c640 1 --2- 172.21.15.27:0/2721629988 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x559958785b60 0x559958787f80 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecb74dfdb9e908b9 server_cookie=385ebef6471c13e0 in_seq=0 out_seq=0 2024-04-25T00:23:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.507+0000 7fc22e7fc640 10 client.16797 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.508+0000 7fc22e7fc640 1 -- 172.21.15.27:0/2721629988 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559958675800 con 0x559958785b60 2024-04-25T00:23:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.508+0000 7fc22e7fc640 10 client.16797 handle_command_reply: tid=0 2024-04-25T00:23:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.508+0000 7fc24fe1e740 1 client.16797 shutdown 2024-04-25T00:23:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.508+0000 7fc24fe1e740 1 -- 172.21.15.27:0/2721629988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958704290 msgr2=0x559958712a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.508+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/2721629988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958704290 0x559958712a30 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fc24000be00 tx=0x7fc240013260 comp rx=0 tx=0).stop 2024-04-25T00:23:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.508+0000 7fc24fe1e740 1 -- 172.21.15.27:0/2721629988 shutdown_connections 2024-04-25T00:23:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.508+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/2721629988 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x559958785b60 0x559958787f80 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.508+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/2721629988 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55995870f010 0x5599587134b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.508+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/2721629988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559958705980 0x559958712f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.508+0000 7fc24fe1e740 1 --2- 172.21.15.27:0/2721629988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559958704290 0x559958712a30 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.508+0000 7fc24fe1e740 1 -- 172.21.15.27:0/2721629988 >> 172.21.15.27:0/2721629988 conn(0x5599586cc950 msgr2=0x559958705d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.508+0000 7fc22e7fc640 0 client.16797 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.508+0000 7fc24fe1e740 1 -- 172.21.15.27:0/2721629988 shutdown_connections 2024-04-25T00:23:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.508+0000 7fc24fe1e740 1 -- 172.21.15.27:0/2721629988 wait complete. 2024-04-25T00:23:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.509+0000 7fc24fe1e740 20 client.16797 trim_cache size 0 max 16384 2024-04-25T00:23:54.518 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:23:54.518 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T00:23:54.518 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "0f05fa30-72c1-4da0-85aa-7da7cf034956", 2024-04-25T00:23:54.518 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T00:23:54.518 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:23:54.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.509+0000 7fc24d038640 1 -- 172.21.15.27:0/2948960064 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc2340a2d00 msgr2=0x7fc2340a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.509+0000 7fc24d038640 1 --2- 172.21.15.27:0/2948960064 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc2340a2d00 0x7fc2340a51c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc238005c40 tx=0x7fc238009500 comp rx=0 tx=0).stop 2024-04-25T00:23:54.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.509+0000 7fc24d038640 1 -- 172.21.15.27:0/2948960064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc248100fe0 msgr2=0x7fc248198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.509+0000 7fc24d038640 1 --2- 172.21.15.27:0/2948960064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc248100fe0 0x7fc248198610 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc230009230 tx=0x7fc230004270 comp rx=0 tx=0).stop 2024-04-25T00:23:54.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.509+0000 7fc24d038640 1 -- 172.21.15.27:0/2948960064 shutdown_connections 2024-04-25T00:23:54.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.509+0000 7fc24d038640 1 --2- 172.21.15.27:0/2948960064 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc2340a2d00 0x7fc2340a51c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.509+0000 7fc24d038640 1 --2- 172.21.15.27:0/2948960064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc248106630 0x7fc24819c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.509+0000 7fc24d038640 1 --2- 172.21.15.27:0/2948960064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc248100fe0 0x7fc248198610 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.509+0000 7fc24d038640 1 --2- 172.21.15.27:0/2948960064 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc248100620 0x7fc2481980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.509+0000 7fc24d038640 1 -- 172.21.15.27:0/2948960064 >> 172.21.15.27:0/2948960064 conn(0x7fc2480776a0 msgr2=0x7fc24806a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:54.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.509+0000 7fc24d038640 1 -- 172.21.15.27:0/2948960064 shutdown_connections 2024-04-25T00:23:54.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.509+0000 7fc24d038640 1 -- 172.21.15.27:0/2948960064 wait complete. 2024-04-25T00:23:54.521 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:23:54.522 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:23:54.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.625+0000 7f85412b3640 1 -- 172.21.15.27:0/1726967339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f853c100620 msgr2=0x7f853c100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.625+0000 7f85412b3640 1 --2- 172.21.15.27:0/1726967339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f853c100620 0x7f853c100aa0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f8524009930 tx=0x7f852402a420 comp rx=0 tx=0).stop 2024-04-25T00:23:54.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.625+0000 7f85412b3640 1 -- 172.21.15.27:0/1726967339 shutdown_connections 2024-04-25T00:23:54.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.625+0000 7f85412b3640 1 --2- 172.21.15.27:0/1726967339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f853c100fe0 0x7f853c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.625+0000 7f85412b3640 1 --2- 172.21.15.27:0/1726967339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f853c100620 0x7f853c100aa0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.625+0000 7f85412b3640 1 --2- 172.21.15.27:0/1726967339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f853c106630 0x7f853c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.625+0000 7f85412b3640 1 -- 172.21.15.27:0/1726967339 >> 172.21.15.27:0/1726967339 conn(0x7f853c0776a0 msgr2=0x7f853c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:54.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.625+0000 7f85412b3640 1 -- 172.21.15.27:0/1726967339 shutdown_connections 2024-04-25T00:23:54.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.625+0000 7f85412b3640 1 -- 172.21.15.27:0/1726967339 wait complete. 2024-04-25T00:23:54.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.625+0000 7f85412b3640 1 Processor -- start 2024-04-25T00:23:54.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f85412b3640 1 -- start start 2024-04-25T00:23:54.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f85412b3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f853c100620 0x7f853c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f85412b3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f853c100fe0 0x7f853c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f85412b3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f853c106630 0x7f853c19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f85412b3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f853c111b40 con 0x7f853c106630 2024-04-25T00:23:54.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f85412b3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1119c0 con 0x7f853c100fe0 2024-04-25T00:23:54.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f85412b3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f853c111cc0 con 0x7f853c100620 2024-04-25T00:23:54.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f853b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f853c106630 0x7f853c19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f853b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f853c106630 0x7f853c19c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50152/0 (socket says 172.21.15.27:50152) 2024-04-25T00:23:54.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f853b7fe640 1 -- 172.21.15.27:0/2730126754 learned_addr learned my addr 172.21.15.27:0/2730126754 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:54.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f853affd640 1 --2- 172.21.15.27:0/2730126754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f853c100620 0x7f853c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f853b7fe640 1 -- 172.21.15.27:0/2730126754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f853c100620 msgr2=0x7f853c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f8533fff640 1 --2- 172.21.15.27:0/2730126754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f853c100fe0 0x7f853c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f853b7fe640 1 --2- 172.21.15.27:0/2730126754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f853c100620 0x7f853c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f853b7fe640 1 -- 172.21.15.27:0/2730126754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f853c100fe0 msgr2=0x7f853c198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f853b7fe640 1 --2- 172.21.15.27:0/2730126754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f853c100fe0 0x7f853c198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f853b7fe640 1 -- 172.21.15.27:0/2730126754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853c19d010 con 0x7f853c106630 2024-04-25T00:23:54.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f8533fff640 1 --2- 172.21.15.27:0/2730126754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f853c100fe0 0x7f853c198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:23:54.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.626+0000 7f853b7fe640 1 --2- 172.21.15.27:0/2730126754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f853c106630 0x7f853c19c8e0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f853400eaa0 tx=0x7f853400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:54.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.627+0000 7f8538ff9640 1 -- 172.21.15.27:0/2730126754 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853400cdb0 con 0x7f853c106630 2024-04-25T00:23:54.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.627+0000 7f8538ff9640 1 -- 172.21.15.27:0/2730126754 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8534005020 con 0x7f853c106630 2024-04-25T00:23:54.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.627+0000 7f85412b3640 1 -- 172.21.15.27:0/2730126754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f853c19d300 con 0x7f853c106630 2024-04-25T00:23:54.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.627+0000 7f8538ff9640 1 -- 172.21.15.27:0/2730126754 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85340116b0 con 0x7f853c106630 2024-04-25T00:23:54.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.627+0000 7f85412b3640 1 -- 172.21.15.27:0/2730126754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f853c1a4b90 con 0x7f853c106630 2024-04-25T00:23:54.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.629+0000 7f8538ff9640 1 -- 172.21.15.27:0/2730126754 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8534005560 con 0x7f853c106630 2024-04-25T00:23:54.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.629+0000 7f8543fc4740 1 Processor -- start 2024-04-25T00:23:54.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.629+0000 7f8543fc4740 1 -- start start 2024-04-25T00:23:54.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f8543fc4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5630 0x561d5abb5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f8543fc4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abb5f50 0x561d5abc00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f8543fc4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abc0620 0x561d5abc2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f8543fc4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561d5ab1a6b0 con 0x561d5abb5f50 2024-04-25T00:23:54.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f8543fc4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561d5aac2880 con 0x561d5abb5630 2024-04-25T00:23:54.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f8543fc4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561d5ab16670 con 0x561d5abc0620 2024-04-25T00:23:54.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f853affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abb5f50 0x561d5abc00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f853affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abb5f50 0x561d5abc00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50162/0 (socket says 172.21.15.27:50162) 2024-04-25T00:23:54.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f853affd640 1 -- 172.21.15.27:0/2991725011 learned_addr learned my addr 172.21.15.27:0/2991725011 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:54.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f8538ff9640 1 --2- 172.21.15.27:0/2730126754 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f85280a2d00 0x7f85280a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f8533fff640 1 --2- 172.21.15.27:0/2991725011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abc0620 0x561d5abc2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f853b7fe640 1 --2- 172.21.15.27:0/2730126754 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f85280a2d00 0x7f85280a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f8533fff640 1 --2- 172.21.15.27:0/2991725011 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5630 0x561d5abb5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f853affd640 1 -- 172.21.15.27:0/2991725011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abc0620 msgr2=0x561d5abc2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f853affd640 1 --2- 172.21.15.27:0/2991725011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abc0620 0x561d5abc2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f853affd640 1 -- 172.21.15.27:0/2991725011 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5630 msgr2=0x561d5abb5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f853affd640 1 --2- 172.21.15.27:0/2991725011 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5630 0x561d5abb5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f853affd640 1 -- 172.21.15.27:0/2991725011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d5a8f87d0 con 0x561d5abb5f50 2024-04-25T00:23:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f8533fff640 1 --2- 172.21.15.27:0/2991725011 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5630 0x561d5abb5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:23:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.630+0000 7f8538ff9640 1 -- 172.21.15.27:0/2730126754 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f85340cfa10 con 0x7f853c106630 2024-04-25T00:23:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.631+0000 7f853affd640 1 --2- 172.21.15.27:0/2991725011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abb5f50 0x561d5abc00e0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f852c00bc90 tx=0x7f852c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=175a57a87af8e9d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.631+0000 7f853b7fe640 1 --2- 172.21.15.27:0/2730126754 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f85280a2d00 0x7f85280a51c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f853400efd0 tx=0x7f853406d040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.631+0000 7f85317fa640 1 -- 172.21.15.27:0/2991725011 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852c01c480 con 0x561d5abb5f50 2024-04-25T00:23:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.631+0000 7f85317fa640 1 -- 172.21.15.27:0/2991725011 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f852c01cb00 con 0x561d5abb5f50 2024-04-25T00:23:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.631+0000 7f85317fa640 1 -- 172.21.15.27:0/2991725011 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852c011c00 con 0x561d5abb5f50 2024-04-25T00:23:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.631+0000 7f8543fc4740 1 -- 172.21.15.27:0/2991725011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abb5f50 msgr2=0x561d5abc00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.631+0000 7f8543fc4740 1 --2- 172.21.15.27:0/2991725011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abb5f50 0x561d5abc00e0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f852c00bc90 tx=0x7f852c0107a0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8543fc4740 1 -- 172.21.15.27:0/2991725011 shutdown_connections 2024-04-25T00:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8543fc4740 1 --2- 172.21.15.27:0/2991725011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abc0620 0x561d5abc2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8543fc4740 1 --2- 172.21.15.27:0/2991725011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abb5f50 0x561d5abc00e0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8543fc4740 1 --2- 172.21.15.27:0/2991725011 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5630 0x561d5abb5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8543fc4740 1 -- 172.21.15.27:0/2991725011 >> 172.21.15.27:0/2991725011 conn(0x561d5abb1950 msgr2=0x561d5abb1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8543fc4740 1 -- 172.21.15.27:0/2991725011 shutdown_connections 2024-04-25T00:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8543fc4740 1 -- 172.21.15.27:0/2991725011 wait complete. 2024-04-25T00:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8543fc4740 1 -- start start 2024-04-25T00:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8543fc4740 1 Processor -- start 2024-04-25T00:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8543fc4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5630 0x561d5abbd540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8543fc4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abb5f50 0x561d5abbda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8543fc4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abbdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8543fc4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561d5abc3410 con 0x561d5abc0620 2024-04-25T00:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8543fc4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561d5aad5f50 con 0x561d5abb5630 2024-04-25T00:23:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8543fc4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561d5abc3590 con 0x561d5abb5f50 2024-04-25T00:23:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8543fc4740 10 client.0 _ll_register_callbacks cb 0x561d5ab11000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8533fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abbdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8533fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abbdfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50178/0 (socket says 172.21.15.27:50178) 2024-04-25T00:23:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.632+0000 7f8533fff640 1 -- 172.21.15.27:0/1216242557 learned_addr learned my addr 172.21.15.27:0/1216242557 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.633+0000 7f8533fff640 1 --2- 172.21.15.27:0/1216242557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5630 0x561d5abbd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.633+0000 7f853affd640 1 --2- 172.21.15.27:0/1216242557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abb5f50 0x561d5abbda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.633+0000 7f8533fff640 1 -- 172.21.15.27:0/1216242557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abb5f50 msgr2=0x561d5abbda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.633+0000 7f8533fff640 1 --2- 172.21.15.27:0/1216242557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abb5f50 0x561d5abbda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.633+0000 7f8533fff640 1 -- 172.21.15.27:0/1216242557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5630 msgr2=0x561d5abbd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.633+0000 7f8533fff640 1 --2- 172.21.15.27:0/1216242557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5630 0x561d5abbd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.633+0000 7f8533fff640 1 -- 172.21.15.27:0/1216242557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d5a82e3f0 con 0x561d5abc0620 2024-04-25T00:23:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.633+0000 7f8533fff640 1 --2- 172.21.15.27:0/1216242557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abbdfc0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f8524005ab0 tx=0x7f852402a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b165eea67098f2c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:54.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.633+0000 7f85227fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.633+0000 7f85227fc640 1 -- 172.21.15.27:0/1216242557 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8524035040 con 0x561d5abc0620 2024-04-25T00:23:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.633+0000 7f85227fc640 1 -- 172.21.15.27:0/1216242557 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f852403fb60 con 0x561d5abc0620 2024-04-25T00:23:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.633+0000 7f8543fc4740 1 -- 172.21.15.27:0/1216242557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f852c01cb00 con 0x561d5abc0620 2024-04-25T00:23:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.633+0000 7f85227fc640 1 -- 172.21.15.27:0/1216242557 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f8524052050 con 0x561d5abc0620 2024-04-25T00:23:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.633+0000 7f8543fc4740 10 client.16809 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.633+0000 7f8543fc4740 1 -- 172.21.15.27:0/1216242557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x561d5a7ee0f0 con 0x561d5abc0620 2024-04-25T00:23:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.634+0000 7f85227fc640 1 -- 172.21.15.27:0/1216242557 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f852c01cb00 con 0x561d5abc0620 2024-04-25T00:23:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.634+0000 7f85227fc640 1 -- 172.21.15.27:0/1216242557 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f8524049070 con 0x561d5abc0620 2024-04-25T00:23:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.634+0000 7f8543fc4740 10 client.16809 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.634+0000 7f8543fc4740 10 client.16809 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:23:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.634+0000 7f8543fc4740 20 client.16809 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.634+0000 7f8543fc4740 1 --2- 172.21.15.27:0/1216242557 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x561d5abd0750 0x561d5abd2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.634+0000 7f8543fc4740 4 client.16809 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:23:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.634+0000 7f8543fc4740 1 -- 172.21.15.27:0/1216242557 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d5ab27bf0 con 0x561d5abd0750 2024-04-25T00:23:54.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.634+0000 7f853affd640 1 --2- 172.21.15.27:0/1216242557 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x561d5abd0750 0x561d5abd2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.635+0000 7f853affd640 1 --2- 172.21.15.27:0/1216242557 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x561d5abd0750 0x561d5abd2b60 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc4d69540b4a4337 server_cookie=ee229c920c23be9b in_seq=0 out_seq=0 2024-04-25T00:23:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.635+0000 7f85227fc640 10 client.16809 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.635+0000 7f85227fc640 1 -- 172.21.15.27:0/1216242557 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561d5ab27bf0 con 0x561d5abd0750 2024-04-25T00:23:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.635+0000 7f85227fc640 10 client.16809 handle_command_reply: tid=0 2024-04-25T00:23:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.635+0000 7f8543fc4740 1 client.16809 shutdown 2024-04-25T00:23:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.636+0000 7f8543fc4740 1 -- 172.21.15.27:0/1216242557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 msgr2=0x561d5abbdfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.636+0000 7f8543fc4740 1 --2- 172.21.15.27:0/1216242557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abbdfc0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f8524005ab0 tx=0x7f852402a420 comp rx=0 tx=0).stop 2024-04-25T00:23:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.636+0000 7f8543fc4740 1 -- 172.21.15.27:0/1216242557 shutdown_connections 2024-04-25T00:23:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.636+0000 7f8543fc4740 1 --2- 172.21.15.27:0/1216242557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abbdfc0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.636+0000 7f8543fc4740 1 --2- 172.21.15.27:0/1216242557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abb5f50 0x561d5abbda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.636+0000 7f8543fc4740 1 --2- 172.21.15.27:0/1216242557 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x561d5abd0750 0x561d5abd2b60 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.636+0000 7f8543fc4740 1 --2- 172.21.15.27:0/1216242557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5630 0x561d5abbd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.636+0000 7f85227fc640 0 client.16809 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.636+0000 7f8543fc4740 1 -- 172.21.15.27:0/1216242557 >> 172.21.15.27:0/1216242557 conn(0x561d5abb1950 msgr2=0x561d5abb7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:54.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.636+0000 7f8543fc4740 1 -- 172.21.15.27:0/1216242557 shutdown_connections 2024-04-25T00:23:54.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.636+0000 7f8543fc4740 1 -- 172.21.15.27:0/1216242557 wait complete. 2024-04-25T00:23:54.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.636+0000 7f8543fc4740 20 client.16809 trim_cache size 0 max 16384 2024-04-25T00:23:54.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.653+0000 7f8543fc4740 1 Processor -- start 2024-04-25T00:23:54.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.653+0000 7f8543fc4740 1 -- start start 2024-04-25T00:23:54.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8543fc4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abb5630 0x561d5abcc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8543fc4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5f50 0x561d5abb1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8543fc4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abbcd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8543fc4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1aec10 con 0x561d5abc0620 2024-04-25T00:23:54.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8543fc4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1b07b0 con 0x561d5abb5f50 2024-04-25T00:23:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8543fc4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1af2e0 con 0x561d5abb5630 2024-04-25T00:23:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8533fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abbcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8533fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abbcd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50190/0 (socket says 172.21.15.27:50190) 2024-04-25T00:23:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8533fff640 1 -- 172.21.15.27:0/3421462967 learned_addr learned my addr 172.21.15.27:0/3421462967 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f853affd640 1 --2- 172.21.15.27:0/3421462967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5f50 0x561d5abb1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8533fff640 1 --2- 172.21.15.27:0/3421462967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abb5630 0x561d5abcc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8533fff640 1 -- 172.21.15.27:0/3421462967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abb5630 msgr2=0x561d5abcc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8533fff640 1 --2- 172.21.15.27:0/3421462967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abb5630 0x561d5abcc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8533fff640 1 -- 172.21.15.27:0/3421462967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5f50 msgr2=0x561d5abb1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8533fff640 1 --2- 172.21.15.27:0/3421462967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5f50 0x561d5abb1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8533fff640 1 -- 172.21.15.27:0/3421462967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d5aba3090 con 0x561d5abc0620 2024-04-25T00:23:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8533fff640 1 --2- 172.21.15.27:0/3421462967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abbcd50 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f8524005940 tx=0x7f8524000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b672672b481263 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8530ff9640 1 -- 172.21.15.27:0/3421462967 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8524035040 con 0x561d5abc0620 2024-04-25T00:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8530ff9640 1 -- 172.21.15.27:0/3421462967 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f852403c430 con 0x561d5abc0620 2024-04-25T00:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.654+0000 7f8530ff9640 1 -- 172.21.15.27:0/3421462967 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852403c040 con 0x561d5abc0620 2024-04-25T00:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.655+0000 7f8543fc4740 1 -- 172.21.15.27:0/3421462967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 msgr2=0x561d5abbcd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.655+0000 7f8543fc4740 1 --2- 172.21.15.27:0/3421462967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abbcd50 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f8524005940 tx=0x7f8524000bc0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.655+0000 7f8543fc4740 1 -- 172.21.15.27:0/3421462967 shutdown_connections 2024-04-25T00:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.655+0000 7f8543fc4740 1 --2- 172.21.15.27:0/3421462967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abbcd50 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.655+0000 7f8543fc4740 1 --2- 172.21.15.27:0/3421462967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5f50 0x561d5abb1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.655+0000 7f8543fc4740 1 --2- 172.21.15.27:0/3421462967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abb5630 0x561d5abcc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.655+0000 7f8543fc4740 1 -- 172.21.15.27:0/3421462967 >> 172.21.15.27:0/3421462967 conn(0x561d5abb1950 msgr2=0x561d5abbd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.655+0000 7f8543fc4740 1 -- 172.21.15.27:0/3421462967 shutdown_connections 2024-04-25T00:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.655+0000 7f8543fc4740 1 -- 172.21.15.27:0/3421462967 wait complete. 2024-04-25T00:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8543fc4740 1 -- start start 2024-04-25T00:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8543fc4740 1 Processor -- start 2024-04-25T00:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8543fc4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abb5630 0x561d5abc8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8543fc4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5f50 0x561d5abba030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8543fc4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abba5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8543fc4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1b00e0 con 0x561d5abc0620 2024-04-25T00:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8543fc4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1af9e0 con 0x561d5abb5f50 2024-04-25T00:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8543fc4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1acba0 con 0x561d5abb5630 2024-04-25T00:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8543fc4740 10 client.0 _ll_register_callbacks cb 0x7f8524009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8533fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abba5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8533fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abba5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50206/0 (socket says 172.21.15.27:50206) 2024-04-25T00:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8533fff640 1 -- 172.21.15.27:0/670783935 learned_addr learned my addr 172.21.15.27:0/670783935 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8533fff640 1 --2- 172.21.15.27:0/670783935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abb5630 0x561d5abc8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f853affd640 1 --2- 172.21.15.27:0/670783935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5f50 0x561d5abba030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8533fff640 1 -- 172.21.15.27:0/670783935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abb5630 msgr2=0x561d5abc8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8533fff640 1 --2- 172.21.15.27:0/670783935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abb5630 0x561d5abc8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8533fff640 1 -- 172.21.15.27:0/670783935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5f50 msgr2=0x561d5abba030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8533fff640 1 --2- 172.21.15.27:0/670783935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5f50 0x561d5abba030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f8533fff640 1 -- 172.21.15.27:0/670783935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d5abbade0 con 0x561d5abc0620 2024-04-25T00:23:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.656+0000 7f853affd640 1 --2- 172.21.15.27:0/670783935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5f50 0x561d5abba030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:23:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.657+0000 7f8533fff640 1 --2- 172.21.15.27:0/670783935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abba5b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f852403f320 tx=0x7f852403f220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3682f20843b50b2c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.657+0000 7f85227fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:23:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.657+0000 7f85227fc640 1 -- 172.21.15.27:0/670783935 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8524035070 con 0x561d5abc0620 2024-04-25T00:23:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.657+0000 7f85227fc640 1 -- 172.21.15.27:0/670783935 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8524004460 con 0x561d5abc0620 2024-04-25T00:23:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.657+0000 7f8543fc4740 1 -- 172.21.15.27:0/670783935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f852403c430 con 0x561d5abc0620 2024-04-25T00:23:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.657+0000 7f85227fc640 1 -- 172.21.15.27:0/670783935 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f8524068050 con 0x561d5abc0620 2024-04-25T00:23:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.657+0000 7f8543fc4740 10 client.16815 fetch_fsmap learned FSMap version 97 2024-04-25T00:23:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.657+0000 7f8543fc4740 1 -- 172.21.15.27:0/670783935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x561d5ac36cc0 con 0x561d5abc0620 2024-04-25T00:23:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.657+0000 7f85227fc640 1 -- 172.21.15.27:0/670783935 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f852403c430 con 0x561d5abc0620 2024-04-25T00:23:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.657+0000 7f85227fc640 1 -- 172.21.15.27:0/670783935 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f8524049070 con 0x561d5abc0620 2024-04-25T00:23:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.657+0000 7f8543fc4740 10 client.16815 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:23:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.657+0000 7f8543fc4740 10 client.16815 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:23:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.657+0000 7f8543fc4740 20 client.16815 populate_metadata read hostname 'smithi027' 2024-04-25T00:23:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.657+0000 7f8543fc4740 1 --2- 172.21.15.27:0/670783935 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x561d5ac372d0 0x561d5ac396f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.657+0000 7f8543fc4740 4 client.16815 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:23:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.657+0000 7f8543fc4740 1 -- 172.21.15.27:0/670783935 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561d5ab21f40 con 0x561d5ac372d0 2024-04-25T00:23:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f853affd640 1 --2- 172.21.15.27:0/670783935 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x561d5ac372d0 0x561d5ac396f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f853affd640 1 --2- 172.21.15.27:0/670783935 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x561d5ac372d0 0x561d5ac396f0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e05093f0131b2b8 server_cookie=10c8b2ec0d92cf16 in_seq=0 out_seq=0 2024-04-25T00:23:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f85227fc640 10 client.16815 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f85227fc640 1 -- 172.21.15.27:0/670783935 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561d5ab21f40 con 0x561d5ac372d0 2024-04-25T00:23:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f85227fc640 10 client.16815 handle_command_reply: tid=0 2024-04-25T00:23:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f8543fc4740 1 client.16815 shutdown 2024-04-25T00:23:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f8543fc4740 1 -- 172.21.15.27:0/670783935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 msgr2=0x561d5abba5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f8543fc4740 1 --2- 172.21.15.27:0/670783935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abba5b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f852403f320 tx=0x7f852403f220 comp rx=0 tx=0).stop 2024-04-25T00:23:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f8543fc4740 1 -- 172.21.15.27:0/670783935 shutdown_connections 2024-04-25T00:23:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f8543fc4740 1 --2- 172.21.15.27:0/670783935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d5abc0620 0x561d5abba5b0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f8543fc4740 1 --2- 172.21.15.27:0/670783935 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x561d5ac372d0 0x561d5ac396f0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f8543fc4740 1 --2- 172.21.15.27:0/670783935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561d5abb5f50 0x561d5abba030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f8543fc4740 1 --2- 172.21.15.27:0/670783935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561d5abb5630 0x561d5abc8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f8543fc4740 1 -- 172.21.15.27:0/670783935 >> 172.21.15.27:0/670783935 conn(0x561d5abb1950 msgr2=0x561d5abb7610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:54.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f85227fc640 0 client.16815 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:23:54.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f8543fc4740 1 -- 172.21.15.27:0/670783935 shutdown_connections 2024-04-25T00:23:54.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f8543fc4740 1 -- 172.21.15.27:0/670783935 wait complete. 2024-04-25T00:23:54.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.658+0000 7f8543fc4740 20 client.16815 trim_cache size 0 max 16384 2024-04-25T00:23:54.667 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:23:54.667 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T00:23:54.668 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:23:54.668 INFO:teuthology.orchestra.run.smithi027.stdout: "0f05fa30-72c1-4da0-85aa-7da7cf034956": { 2024-04-25T00:23:54.668 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:23:54.668 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "0f05fa30-72c1-4da0-85aa-7da7cf034956", 2024-04-25T00:23:54.668 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:23:54.668 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:23:54.668 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:23:54.668 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:23:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.659+0000 7f85412b3640 1 -- 172.21.15.27:0/2730126754 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f85280a2d00 msgr2=0x7f85280a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.659+0000 7f85412b3640 1 --2- 172.21.15.27:0/2730126754 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f85280a2d00 0x7f85280a51c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f853400efd0 tx=0x7f853406d040 comp rx=0 tx=0).stop 2024-04-25T00:23:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.659+0000 7f85412b3640 1 -- 172.21.15.27:0/2730126754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f853c106630 msgr2=0x7f853c19c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.659+0000 7f85412b3640 1 --2- 172.21.15.27:0/2730126754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f853c106630 0x7f853c19c8e0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f853400eaa0 tx=0x7f853400ef60 comp rx=0 tx=0).stop 2024-04-25T00:23:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.659+0000 7f85412b3640 1 -- 172.21.15.27:0/2730126754 shutdown_connections 2024-04-25T00:23:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.659+0000 7f85412b3640 1 --2- 172.21.15.27:0/2730126754 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f85280a2d00 0x7f85280a51c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.659+0000 7f85412b3640 1 --2- 172.21.15.27:0/2730126754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f853c106630 0x7f853c19c8e0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.659+0000 7f85412b3640 1 --2- 172.21.15.27:0/2730126754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f853c100fe0 0x7f853c198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.659+0000 7f85412b3640 1 --2- 172.21.15.27:0/2730126754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f853c100620 0x7f853c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:54.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.659+0000 7f85412b3640 1 -- 172.21.15.27:0/2730126754 >> 172.21.15.27:0/2730126754 conn(0x7f853c0776a0 msgr2=0x7f853c103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:54.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.659+0000 7f85412b3640 1 -- 172.21.15.27:0/2730126754 shutdown_connections 2024-04-25T00:23:54.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:23:54.659+0000 7f85412b3640 1 -- 172.21.15.27:0/2730126754 wait complete. 2024-04-25T00:23:54.671 INFO:tasks.cephfs.filesystem:scrub status for tag:0f05fa30-72c1-4da0-85aa-7da7cf034956 - {'path': '/', 'tag': '0f05fa30-72c1-4da0-85aa-7da7cf034956', 'options': 'recursive,force'} 2024-04-25T00:23:54.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:54 smithi027 ceph-mon[29208]: pgmap v247: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:23:55.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:54 smithi153 ceph-mon[32896]: pgmap v247: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:23:55.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:54 smithi139 ceph-mon[32833]: pgmap v247: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:23:56.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:55 smithi153 ceph-mon[32896]: scrub queued for path: / 2024-04-25T00:23:56.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:55 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:23:56.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:55 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:23:56.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:55 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:23:56.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:55 smithi027 ceph-mon[29208]: scrub queued for path: / 2024-04-25T00:23:56.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:55 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:23:56.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:55 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:23:56.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:55 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:23:56.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:55 smithi139 ceph-mon[32833]: scrub queued for path: / 2024-04-25T00:23:56.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:55 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:23:56.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:55 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:23:56.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:55 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:23:57.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:56 smithi153 ceph-mon[32896]: pgmap v248: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T00:23:57.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:56 smithi027 ceph-mon[29208]: pgmap v248: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T00:23:57.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:56 smithi139 ceph-mon[32833]: pgmap v248: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T00:23:59.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:58 smithi153 ceph-mon[32896]: pgmap v249: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-04-25T00:23:59.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:23:58 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:23:59.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:58 smithi027 ceph-mon[29208]: pgmap v249: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-04-25T00:23:59.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:23:58 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:23:59.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:58 smithi139 ceph-mon[32833]: pgmap v249: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-04-25T00:23:59.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:23:58 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:24:01.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:00 smithi153 ceph-mon[32896]: pgmap v250: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-04-25T00:24:01.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:00 smithi027 ceph-mon[29208]: pgmap v250: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-04-25T00:24:01.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:00 smithi139 ceph-mon[32833]: pgmap v250: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-04-25T00:24:03.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:02 smithi153 ceph-mon[32896]: pgmap v251: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T00:24:03.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:02 smithi027 ceph-mon[29208]: pgmap v251: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T00:24:03.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:02 smithi139 ceph-mon[32833]: pgmap v251: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T00:24:05.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:04 smithi153 ceph-mon[32896]: pgmap v252: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:24:05.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:04 smithi027 ceph-mon[29208]: pgmap v252: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:24:05.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:04 smithi139 ceph-mon[32833]: pgmap v252: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:24:07.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:06 smithi027 ceph-mon[29208]: pgmap v253: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:24:07.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:06 smithi139 ceph-mon[32833]: pgmap v253: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:24:07.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:06 smithi153 ceph-mon[32896]: pgmap v253: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:24:09.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:08 smithi027 ceph-mon[29208]: pgmap v254: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:24:09.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:08 smithi139 ceph-mon[32833]: pgmap v254: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:24:09.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:08 smithi153 ceph-mon[32896]: pgmap v254: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:24:11.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:10 smithi027 ceph-mon[29208]: pgmap v255: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T00:24:11.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:10 smithi139 ceph-mon[32833]: pgmap v255: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T00:24:11.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:10 smithi153 ceph-mon[32896]: pgmap v255: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T00:24:13.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:12 smithi027 ceph-mon[29208]: pgmap v256: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:24:13.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:12 smithi139 ceph-mon[32833]: pgmap v256: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:24:13.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:12 smithi153 ceph-mon[32896]: pgmap v256: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:24:14.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:13 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:24:14.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:13 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:24:14.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:13 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:24:15.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:14 smithi027 ceph-mon[29208]: pgmap v257: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:24:15.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:14 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:24:15.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:14 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:24:15.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:14 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:24:15.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:14 smithi139 ceph-mon[32833]: pgmap v257: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:24:15.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:14 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:24:15.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:14 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:24:15.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:14 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:24:15.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:14 smithi153 ceph-mon[32896]: pgmap v257: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:24:15.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:14 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:24:15.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:14 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:24:15.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:14 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:24:17.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:16 smithi027 ceph-mon[29208]: pgmap v258: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:24:17.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:16 smithi139 ceph-mon[32833]: pgmap v258: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:24:17.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:16 smithi153 ceph-mon[32896]: pgmap v258: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:24:19.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:18 smithi027 ceph-mon[29208]: pgmap v259: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:24:19.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:18 smithi139 ceph-mon[32833]: pgmap v259: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:24:19.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:18 smithi153 ceph-mon[32896]: pgmap v259: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:24:21.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:20 smithi027 ceph-mon[29208]: pgmap v260: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T00:24:21.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:20 smithi139 ceph-mon[32833]: pgmap v260: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T00:24:21.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:20 smithi153 ceph-mon[32896]: pgmap v260: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T00:24:23.079 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:22 smithi153 ceph-mon[32896]: pgmap v261: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:24:23.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:22 smithi027 ceph-mon[29208]: pgmap v261: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:24:23.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:22 smithi139 ceph-mon[32833]: pgmap v261: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:24:24.672 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:24:24.673 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:24:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.792+0000 7f9b4b309640 1 -- 172.21.15.27:0/1096657000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b44069dc0 msgr2=0x7f9b4406a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.792+0000 7f9b4b309640 1 --2- 172.21.15.27:0/1096657000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b44069dc0 0x7f9b4406a1a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f9b40009900 tx=0x7f9b4002a410 comp rx=0 tx=0).stop 2024-04-25T00:24:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.792+0000 7f9b4b309640 1 -- 172.21.15.27:0/1096657000 shutdown_connections 2024-04-25T00:24:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.792+0000 7f9b4b309640 1 --2- 172.21.15.27:0/1096657000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b4410e150 0x7f9b44110540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.792+0000 7f9b4b309640 1 --2- 172.21.15.27:0/1096657000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b4406a6e0 0x7f9b4410dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.792+0000 7f9b4b309640 1 --2- 172.21.15.27:0/1096657000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b44069dc0 0x7f9b4406a1a0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.792+0000 7f9b4b309640 1 -- 172.21.15.27:0/1096657000 >> 172.21.15.27:0/1096657000 conn(0x7f9b440776a0 msgr2=0x7f9b44077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.792+0000 7f9b4b309640 1 -- 172.21.15.27:0/1096657000 shutdown_connections 2024-04-25T00:24:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.792+0000 7f9b4b309640 1 -- 172.21.15.27:0/1096657000 wait complete. 2024-04-25T00:24:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.792+0000 7f9b4b309640 1 Processor -- start 2024-04-25T00:24:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.792+0000 7f9b4b309640 1 -- start start 2024-04-25T00:24:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b4b309640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b44069dc0 0x7f9b4419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b4b309640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b4406a6e0 0x7f9b4419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b4b309640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4410e150 0x7f9b4419ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b4b309640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9b44113d10 con 0x7f9b4410e150 2024-04-25T00:24:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b4b309640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9b44113b90 con 0x7f9b4406a6e0 2024-04-25T00:24:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b4b309640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9b44113e90 con 0x7f9b44069dc0 2024-04-25T00:24:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b498a6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4410e150 0x7f9b4419ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b498a6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4410e150 0x7f9b4419ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33962/0 (socket says 172.21.15.27:33962) 2024-04-25T00:24:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b498a6640 1 -- 172.21.15.27:0/2719541801 learned_addr learned my addr 172.21.15.27:0/2719541801 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b488a4640 1 --2- 172.21.15.27:0/2719541801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b4406a6e0 0x7f9b4419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b490a5640 1 --2- 172.21.15.27:0/2719541801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b44069dc0 0x7f9b4419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b498a6640 1 -- 172.21.15.27:0/2719541801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b44069dc0 msgr2=0x7f9b4419a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b498a6640 1 --2- 172.21.15.27:0/2719541801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b44069dc0 0x7f9b4419a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b498a6640 1 -- 172.21.15.27:0/2719541801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b4406a6e0 msgr2=0x7f9b4419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b498a6640 1 --2- 172.21.15.27:0/2719541801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b4406a6e0 0x7f9b4419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b498a6640 1 -- 172.21.15.27:0/2719541801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4419f1b0 con 0x7f9b4410e150 2024-04-25T00:24:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b498a6640 1 --2- 172.21.15.27:0/2719541801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4410e150 0x7f9b4419ea80 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c00e9d0 tx=0x7f9b3c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b367fc640 1 -- 172.21.15.27:0/2719541801 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c00cd80 con 0x7f9b4410e150 2024-04-25T00:24:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b4b309640 1 -- 172.21.15.27:0/2719541801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b4419f4a0 con 0x7f9b4410e150 2024-04-25T00:24:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.793+0000 7f9b367fc640 1 -- 172.21.15.27:0/2719541801 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b3c005020 con 0x7f9b4410e150 2024-04-25T00:24:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.794+0000 7f9b367fc640 1 -- 172.21.15.27:0/2719541801 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c0116d0 con 0x7f9b4410e150 2024-04-25T00:24:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.794+0000 7f9b4b309640 1 -- 172.21.15.27:0/2719541801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b441a6d30 con 0x7f9b4410e150 2024-04-25T00:24:24.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.797+0000 7f9b367fc640 1 -- 172.21.15.27:0/2719541801 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9b3c0051d0 con 0x7f9b4410e150 2024-04-25T00:24:24.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.797+0000 7f9b4e011740 1 Processor -- start 2024-04-25T00:24:24.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.797+0000 7f9b4e011740 1 -- start start 2024-04-25T00:24:24.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.797+0000 7f9b4e011740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04630 0x561da8f04a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.797+0000 7f9b4e011740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f04f50 0x561da8f0f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.797+0000 7f9b4e011740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f0f620 0x561da8f11a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.797+0000 7f9b4e011740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561da8e696b0 con 0x561da8f04630 2024-04-25T00:24:24.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.797+0000 7f9b4e011740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561da8e11880 con 0x561da8f0f620 2024-04-25T00:24:24.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.797+0000 7f9b4e011740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561da8e65670 con 0x561da8f04f50 2024-04-25T00:24:24.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.797+0000 7f9b488a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04630 0x561da8f04a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.797+0000 7f9b488a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04630 0x561da8f04a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33978/0 (socket says 172.21.15.27:33978) 2024-04-25T00:24:24.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.797+0000 7f9b488a4640 1 -- 172.21.15.27:0/212277127 learned_addr learned my addr 172.21.15.27:0/212277127 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.797+0000 7f9b490a5640 1 --2- 172.21.15.27:0/212277127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f04f50 0x561da8f0f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.797+0000 7f9b367fc640 1 --2- 172.21.15.27:0/2719541801 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b300a2d00 0x7f9b300a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.798+0000 7f9b488a4640 1 -- 172.21.15.27:0/212277127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f04f50 msgr2=0x561da8f0f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.798+0000 7f9b488a4640 1 --2- 172.21.15.27:0/212277127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f04f50 0x561da8f0f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.798+0000 7f9b488a4640 1 -- 172.21.15.27:0/212277127 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f0f620 msgr2=0x561da8f11a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.798+0000 7f9b488a4640 1 --2- 172.21.15.27:0/212277127 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f0f620 0x561da8f11a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.798+0000 7f9b488a4640 1 -- 172.21.15.27:0/212277127 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561da8c477d0 con 0x561da8f04630 2024-04-25T00:24:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.798+0000 7f9b490a5640 1 --2- 172.21.15.27:0/212277127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f04f50 0x561da8f0f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:24.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.798+0000 7f9b367fc640 1 -- 172.21.15.27:0/2719541801 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f9b3c0cfcc0 con 0x7f9b4410e150 2024-04-25T00:24:24.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.798+0000 7f9b498a6640 1 --2- 172.21.15.27:0/2719541801 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b300a2d00 0x7f9b300a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.798+0000 7f9b488a4640 1 --2- 172.21.15.27:0/212277127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04630 0x561da8f04a10 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800bd50 tx=0x7f9b380127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba42b823fba224b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:24.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.798+0000 7f9b2f7fe640 1 -- 172.21.15.27:0/212277127 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b380120d0 con 0x561da8f04630 2024-04-25T00:24:24.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.798+0000 7f9b2f7fe640 1 -- 172.21.15.27:0/212277127 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b38016070 con 0x561da8f04630 2024-04-25T00:24:24.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.798+0000 7f9b2f7fe640 1 -- 172.21.15.27:0/212277127 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b380107f0 con 0x561da8f04630 2024-04-25T00:24:24.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.799+0000 7f9b4e011740 1 -- 172.21.15.27:0/212277127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04630 msgr2=0x561da8f04a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:24.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.799+0000 7f9b4e011740 1 --2- 172.21.15.27:0/212277127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04630 0x561da8f04a10 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800bd50 tx=0x7f9b380127c0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.799+0000 7f9b498a6640 1 --2- 172.21.15.27:0/2719541801 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b300a2d00 0x7f9b300a51c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c09bb70 tx=0x7f9b3c00c7f0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:24.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.799+0000 7f9b4e011740 1 -- 172.21.15.27:0/212277127 shutdown_connections 2024-04-25T00:24:24.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.799+0000 7f9b4e011740 1 --2- 172.21.15.27:0/212277127 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f0f620 0x561da8f11a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.799+0000 7f9b4e011740 1 --2- 172.21.15.27:0/212277127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f04f50 0x561da8f0f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.799+0000 7f9b4e011740 1 --2- 172.21.15.27:0/212277127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04630 0x561da8f04a10 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.799+0000 7f9b4e011740 1 -- 172.21.15.27:0/212277127 >> 172.21.15.27:0/212277127 conn(0x561da8f00950 msgr2=0x561da8f00d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.799+0000 7f9b4e011740 1 -- 172.21.15.27:0/212277127 shutdown_connections 2024-04-25T00:24:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.799+0000 7f9b4e011740 1 -- 172.21.15.27:0/212277127 wait complete. 2024-04-25T00:24:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.799+0000 7f9b4e011740 1 -- start start 2024-04-25T00:24:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.799+0000 7f9b4e011740 1 Processor -- start 2024-04-25T00:24:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b4e011740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04630 0x561da8f0c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b4e011740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f04f50 0x561da8f0cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b4e011740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f0f620 0x561da8f0d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b4e011740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561da8f12410 con 0x561da8f04630 2024-04-25T00:24:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b4e011740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561da8e24f50 con 0x561da8f0f620 2024-04-25T00:24:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b4e011740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561da8f12590 con 0x561da8f04f50 2024-04-25T00:24:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b4e011740 10 client.0 _ll_register_callbacks cb 0x561da8e60000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b488a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04630 0x561da8f0c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b488a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04630 0x561da8f0c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33980/0 (socket says 172.21.15.27:33980) 2024-04-25T00:24:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b488a4640 1 -- 172.21.15.27:0/350994136 learned_addr learned my addr 172.21.15.27:0/350994136 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b490a5640 1 --2- 172.21.15.27:0/350994136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f04f50 0x561da8f0cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b488a4640 1 -- 172.21.15.27:0/350994136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f04f50 msgr2=0x561da8f0cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b488a4640 1 --2- 172.21.15.27:0/350994136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f04f50 0x561da8f0cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b488a4640 1 -- 172.21.15.27:0/350994136 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f0f620 msgr2=0x561da8f0d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b488a4640 1 --2- 172.21.15.27:0/350994136 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f0f620 0x561da8f0d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b488a4640 1 -- 172.21.15.27:0/350994136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561da8b7d3f0 con 0x561da8f04630 2024-04-25T00:24:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b490a5640 1 --2- 172.21.15.27:0/350994136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f04f50 0x561da8f0cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b488a4640 1 --2- 172.21.15.27:0/350994136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04630 0x561da8f0c5c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800b3d0 tx=0x7f9b38005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5181ca717a1df383 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b2cff9640 1 -- 172.21.15.27:0/350994136 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38020070 con 0x561da8f04630 2024-04-25T00:24:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b2cff9640 1 -- 172.21.15.27:0/350994136 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f9b380056a0 con 0x561da8f04630 2024-04-25T00:24:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.800+0000 7f9b4e011740 1 -- 172.21.15.27:0/350994136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b38016070 con 0x561da8f04630 2024-04-25T00:24:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.801+0000 7f9b2cff9640 1 -- 172.21.15.27:0/350994136 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f9b3802f050 con 0x561da8f04630 2024-04-25T00:24:24.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.801+0000 7f9b4e011740 10 client.16827 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:24.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.801+0000 7f9b4e011740 1 -- 172.21.15.27:0/350994136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x561da8b3d0f0 con 0x561da8f04630 2024-04-25T00:24:24.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.801+0000 7f9b2cff9640 1 -- 172.21.15.27:0/350994136 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b38016070 con 0x561da8f04630 2024-04-25T00:24:24.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.801+0000 7f9b2cff9640 1 -- 172.21.15.27:0/350994136 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f9b3801b8b0 con 0x561da8f04630 2024-04-25T00:24:24.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.801+0000 7f9b4e011740 10 client.16827 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:24.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.801+0000 7f9b4e011740 10 client.16827 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:24:24.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.801+0000 7f9b4e011740 20 client.16827 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:24.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.801+0000 7f9b4e011740 1 --2- 172.21.15.27:0/350994136 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x561da8f1f820 0x561da8f21c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.801+0000 7f9b4e011740 4 client.16827 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:24:24.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.801+0000 7f9b4e011740 1 -- 172.21.15.27:0/350994136 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561da8e76bf0 con 0x561da8f1f820 2024-04-25T00:24:24.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.801+0000 7f9b490a5640 1 --2- 172.21.15.27:0/350994136 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x561da8f1f820 0x561da8f21c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.802+0000 7f9b490a5640 1 --2- 172.21.15.27:0/350994136 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x561da8f1f820 0x561da8f21c10 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60b23668a55cdf23 server_cookie=b614ac3e36208d87 in_seq=0 out_seq=0 2024-04-25T00:24:24.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.802+0000 7f9b2cff9640 10 client.16827 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:24.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.802+0000 7f9b2cff9640 1 -- 172.21.15.27:0/350994136 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561da8e76bf0 con 0x561da8f1f820 2024-04-25T00:24:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.802+0000 7f9b2cff9640 10 client.16827 handle_command_reply: tid=0 2024-04-25T00:24:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.803+0000 7f9b4e011740 1 client.16827 shutdown 2024-04-25T00:24:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.803+0000 7f9b4e011740 1 -- 172.21.15.27:0/350994136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04630 msgr2=0x561da8f0c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.803+0000 7f9b4e011740 1 --2- 172.21.15.27:0/350994136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04630 0x561da8f0c5c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800b3d0 tx=0x7f9b38005270 comp rx=0 tx=0).stop 2024-04-25T00:24:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.803+0000 7f9b4e011740 1 -- 172.21.15.27:0/350994136 shutdown_connections 2024-04-25T00:24:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.803+0000 7f9b4e011740 1 --2- 172.21.15.27:0/350994136 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x561da8f1f820 0x561da8f21c10 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.803+0000 7f9b4e011740 1 --2- 172.21.15.27:0/350994136 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f0f620 0x561da8f0d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.803+0000 7f9b4e011740 1 --2- 172.21.15.27:0/350994136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f04f50 0x561da8f0cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.803+0000 7f9b4e011740 1 --2- 172.21.15.27:0/350994136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04630 0x561da8f0c5c0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.803+0000 7f9b2cff9640 0 client.16827 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.803+0000 7f9b4e011740 1 -- 172.21.15.27:0/350994136 >> 172.21.15.27:0/350994136 conn(0x561da8f00950 msgr2=0x561da8f06630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.803+0000 7f9b4e011740 1 -- 172.21.15.27:0/350994136 shutdown_connections 2024-04-25T00:24:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.803+0000 7f9b4e011740 1 -- 172.21.15.27:0/350994136 wait complete. 2024-04-25T00:24:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.803+0000 7f9b4e011740 20 client.16827 trim_cache size 0 max 16384 2024-04-25T00:24:24.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b4e011740 1 Processor -- start 2024-04-25T00:24:24.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b4e011740 1 -- start start 2024-04-25T00:24:24.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b4e011740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f04630 0x561da8f0ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b4e011740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04f50 0x561da8f1b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b4e011740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f0f620 0x561da8f0b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b4e011740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441b0eb0 con 0x561da8f04f50 2024-04-25T00:24:24.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b4e011740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441b2a50 con 0x561da8f04630 2024-04-25T00:24:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b4e011740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441b1580 con 0x561da8f0f620 2024-04-25T00:24:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b490a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04f50 0x561da8f1b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b490a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04f50 0x561da8f1b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33982/0 (socket says 172.21.15.27:33982) 2024-04-25T00:24:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b490a5640 1 -- 172.21.15.27:0/1952468012 learned_addr learned my addr 172.21.15.27:0/1952468012 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b488a4640 1 --2- 172.21.15.27:0/1952468012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f04630 0x561da8f0ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b490a5640 1 -- 172.21.15.27:0/1952468012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f0f620 msgr2=0x561da8f0b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b490a5640 1 --2- 172.21.15.27:0/1952468012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f0f620 0x561da8f0b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b490a5640 1 -- 172.21.15.27:0/1952468012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f04630 msgr2=0x561da8f0ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b490a5640 1 --2- 172.21.15.27:0/1952468012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f04630 0x561da8f0ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b490a5640 1 -- 172.21.15.27:0/1952468012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561da8ef2090 con 0x561da8f04f50 2024-04-25T00:24:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.822+0000 7f9b488a4640 1 --2- 172.21.15.27:0/1952468012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f04630 0x561da8f0ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.823+0000 7f9b490a5640 1 --2- 172.21.15.27:0/1952468012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04f50 0x561da8f1b300 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f9b4002a930 tx=0x7f9b400095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d3fcb1f996276e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.823+0000 7f9b2effd640 1 -- 172.21.15.27:0/1952468012 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b40048070 con 0x561da8f04f50 2024-04-25T00:24:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.823+0000 7f9b2effd640 1 -- 172.21.15.27:0/1952468012 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b40005520 con 0x561da8f04f50 2024-04-25T00:24:24.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.823+0000 7f9b2effd640 1 -- 172.21.15.27:0/1952468012 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b40035030 con 0x561da8f04f50 2024-04-25T00:24:24.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.823+0000 7f9b4e011740 1 -- 172.21.15.27:0/1952468012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04f50 msgr2=0x561da8f1b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:24.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.823+0000 7f9b4e011740 1 --2- 172.21.15.27:0/1952468012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04f50 0x561da8f1b300 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f9b4002a930 tx=0x7f9b400095f0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.823+0000 7f9b4e011740 1 -- 172.21.15.27:0/1952468012 shutdown_connections 2024-04-25T00:24:24.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.823+0000 7f9b4e011740 1 --2- 172.21.15.27:0/1952468012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f0f620 0x561da8f0b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.823+0000 7f9b4e011740 1 --2- 172.21.15.27:0/1952468012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f04f50 0x561da8f1b300 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.823+0000 7f9b4e011740 1 --2- 172.21.15.27:0/1952468012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f04630 0x561da8f0ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.823+0000 7f9b4e011740 1 -- 172.21.15.27:0/1952468012 >> 172.21.15.27:0/1952468012 conn(0x561da8f00950 msgr2=0x561da8eff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:24.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.824+0000 7f9b4e011740 1 -- 172.21.15.27:0/1952468012 shutdown_connections 2024-04-25T00:24:24.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.824+0000 7f9b4e011740 1 -- 172.21.15.27:0/1952468012 wait complete. 2024-04-25T00:24:24.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.824+0000 7f9b4e011740 1 -- start start 2024-04-25T00:24:24.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.824+0000 7f9b4e011740 1 Processor -- start 2024-04-25T00:24:24.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.824+0000 7f9b4e011740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f04630 0x561da8f09160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.824+0000 7f9b4e011740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f04f50 0x561da8f096a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.824+0000 7f9b4e011740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f0f620 0x561da8f09be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.824+0000 7f9b4e011740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441b2380 con 0x561da8f0f620 2024-04-25T00:24:24.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.824+0000 7f9b4e011740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441b1c80 con 0x561da8f04f50 2024-04-25T00:24:24.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.824+0000 7f9b4e011740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441aee40 con 0x561da8f04630 2024-04-25T00:24:24.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.824+0000 7f9b4e011740 10 client.0 _ll_register_callbacks cb 0x7f9b3800bc50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:24.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.824+0000 7f9b488a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f0f620 0x561da8f09be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b488a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f0f620 0x561da8f09be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33988/0 (socket says 172.21.15.27:33988) 2024-04-25T00:24:24.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b488a4640 1 -- 172.21.15.27:0/2181714613 learned_addr learned my addr 172.21.15.27:0/2181714613 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:24.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b488a4640 1 --2- 172.21.15.27:0/2181714613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f04630 0x561da8f09160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b490a5640 1 --2- 172.21.15.27:0/2181714613 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f04f50 0x561da8f096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b488a4640 1 -- 172.21.15.27:0/2181714613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f04630 msgr2=0x561da8f09160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b488a4640 1 --2- 172.21.15.27:0/2181714613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f04630 0x561da8f09160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b488a4640 1 -- 172.21.15.27:0/2181714613 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f04f50 msgr2=0x561da8f096a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b488a4640 1 --2- 172.21.15.27:0/2181714613 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f04f50 0x561da8f096a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b488a4640 1 -- 172.21.15.27:0/2181714613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561da8f84a20 con 0x561da8f0f620 2024-04-25T00:24:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b488a4640 1 --2- 172.21.15.27:0/2181714613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f0f620 0x561da8f09be0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800bad0 tx=0x7f9b38005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1923bf3e8f329f01 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b2cff9640 1 -- 172.21.15.27:0/2181714613 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38020070 con 0x561da8f0f620 2024-04-25T00:24:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b2cff9640 1 -- 172.21.15.27:0/2181714613 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f9b38010960 con 0x561da8f0f620 2024-04-25T00:24:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b4e011740 1 -- 172.21.15.27:0/2181714613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b40005520 con 0x561da8f0f620 2024-04-25T00:24:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b2cff9640 1 -- 172.21.15.27:0/2181714613 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f9b3803e050 con 0x561da8f0f620 2024-04-25T00:24:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b4e011740 10 client.16833 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b4e011740 1 -- 172.21.15.27:0/2181714613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x561da8f85b40 con 0x561da8f0f620 2024-04-25T00:24:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.825+0000 7f9b2cff9640 1 -- 172.21.15.27:0/2181714613 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b40005520 con 0x561da8f0f620 2024-04-25T00:24:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.826+0000 7f9b2cff9640 1 -- 172.21.15.27:0/2181714613 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f9b380035f0 con 0x561da8f0f620 2024-04-25T00:24:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.826+0000 7f9b4e011740 10 client.16833 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.826+0000 7f9b4e011740 10 client.16833 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:24:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.826+0000 7f9b4e011740 20 client.16833 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.826+0000 7f9b4e011740 1 --2- 172.21.15.27:0/2181714613 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x561da8f86210 0x561da8f88630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.826+0000 7f9b4e011740 4 client.16833 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:24:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.826+0000 7f9b4e011740 1 -- 172.21.15.27:0/2181714613 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561da8e70f40 con 0x561da8f86210 2024-04-25T00:24:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.826+0000 7f9b490a5640 1 --2- 172.21.15.27:0/2181714613 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x561da8f86210 0x561da8f88630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.826+0000 7f9b490a5640 1 --2- 172.21.15.27:0/2181714613 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x561da8f86210 0x561da8f88630 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fae3ce8918caed server_cookie=5860fafd2c79a708 in_seq=0 out_seq=0 2024-04-25T00:24:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.826+0000 7f9b2cff9640 10 client.16833 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b2cff9640 1 -- 172.21.15.27:0/2181714613 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561da8e70f40 con 0x561da8f86210 2024-04-25T00:24:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b2cff9640 10 client.16833 handle_command_reply: tid=0 2024-04-25T00:24:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b4e011740 1 client.16833 shutdown 2024-04-25T00:24:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b4e011740 1 -- 172.21.15.27:0/2181714613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f0f620 msgr2=0x561da8f09be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b4e011740 1 --2- 172.21.15.27:0/2181714613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f0f620 0x561da8f09be0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800bad0 tx=0x7f9b38005270 comp rx=0 tx=0).stop 2024-04-25T00:24:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b4e011740 1 -- 172.21.15.27:0/2181714613 shutdown_connections 2024-04-25T00:24:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b4e011740 1 --2- 172.21.15.27:0/2181714613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561da8f0f620 0x561da8f09be0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b4e011740 1 --2- 172.21.15.27:0/2181714613 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x561da8f86210 0x561da8f88630 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b4e011740 1 --2- 172.21.15.27:0/2181714613 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561da8f04f50 0x561da8f096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b4e011740 1 --2- 172.21.15.27:0/2181714613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561da8f04630 0x561da8f09160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b2cff9640 0 client.16833 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b4e011740 1 -- 172.21.15.27:0/2181714613 >> 172.21.15.27:0/2181714613 conn(0x561da8f00950 msgr2=0x561da8f0ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b4e011740 1 -- 172.21.15.27:0/2181714613 shutdown_connections 2024-04-25T00:24:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b4e011740 1 -- 172.21.15.27:0/2181714613 wait complete. 2024-04-25T00:24:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b4e011740 20 client.16833 trim_cache size 0 max 16384 2024-04-25T00:24:24.835 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:24:24.836 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T00:24:24.836 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T00:24:24.836 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:24:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b4b309640 1 -- 172.21.15.27:0/2719541801 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b300a2d00 msgr2=0x7f9b300a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b4b309640 1 --2- 172.21.15.27:0/2719541801 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b300a2d00 0x7f9b300a51c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c09bb70 tx=0x7f9b3c00c7f0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b4b309640 1 -- 172.21.15.27:0/2719541801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4410e150 msgr2=0x7f9b4419ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.827+0000 7f9b4b309640 1 --2- 172.21.15.27:0/2719541801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4410e150 0x7f9b4419ea80 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c00e9d0 tx=0x7f9b3c00ee90 comp rx=0 tx=0).stop 2024-04-25T00:24:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.828+0000 7f9b4b309640 1 -- 172.21.15.27:0/2719541801 shutdown_connections 2024-04-25T00:24:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.828+0000 7f9b4b309640 1 --2- 172.21.15.27:0/2719541801 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b300a2d00 0x7f9b300a51c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.828+0000 7f9b4b309640 1 --2- 172.21.15.27:0/2719541801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4410e150 0x7f9b4419ea80 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.828+0000 7f9b4b309640 1 --2- 172.21.15.27:0/2719541801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b4406a6e0 0x7f9b4419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.828+0000 7f9b4b309640 1 --2- 172.21.15.27:0/2719541801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b44069dc0 0x7f9b4419a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.828+0000 7f9b4b309640 1 -- 172.21.15.27:0/2719541801 >> 172.21.15.27:0/2719541801 conn(0x7f9b440776a0 msgr2=0x7f9b4410f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.828+0000 7f9b4b309640 1 -- 172.21.15.27:0/2719541801 shutdown_connections 2024-04-25T00:24:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.828+0000 7f9b4b309640 1 -- 172.21.15.27:0/2719541801 wait complete. 2024-04-25T00:24:24.839 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T00:24:24.839 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:24:24.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.941+0000 7f9b954b3640 1 -- 172.21.15.27:0/1230995643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b9010bd10 msgr2=0x7f9b9010e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:24.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.941+0000 7f9b954b3640 1 --2- 172.21.15.27:0/1230995643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b9010bd10 0x7f9b9010e1b0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009ca0 tx=0x7f9b8002a9e0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.941+0000 7f9b954b3640 1 -- 172.21.15.27:0/1230995643 shutdown_connections 2024-04-25T00:24:24.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.941+0000 7f9b954b3640 1 --2- 172.21.15.27:0/1230995643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b9010bd10 0x7f9b9010e1b0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.941+0000 7f9b954b3640 1 --2- 172.21.15.27:0/1230995643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b901032f0 0x7f9b9010b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.941+0000 7f9b954b3640 1 --2- 172.21.15.27:0/1230995643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b901029d0 0x7f9b90102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.941+0000 7f9b954b3640 1 -- 172.21.15.27:0/1230995643 >> 172.21.15.27:0/1230995643 conn(0x7f9b90077740 msgr2=0x7f9b90077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:24.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.941+0000 7f9b954b3640 1 -- 172.21.15.27:0/1230995643 shutdown_connections 2024-04-25T00:24:24.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.941+0000 7f9b954b3640 1 -- 172.21.15.27:0/1230995643 wait complete. 2024-04-25T00:24:24.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.942+0000 7f9b954b3640 1 Processor -- start 2024-04-25T00:24:24.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.942+0000 7f9b954b3640 1 -- start start 2024-04-25T00:24:24.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.942+0000 7f9b954b3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b901029d0 0x7f9b90197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.942+0000 7f9b954b3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b901032f0 0x7f9b90198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.942+0000 7f9b954b3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b9010bd10 0x7f9b9019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.942+0000 7f9b954b3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9b90111b10 con 0x7f9b9010bd10 2024-04-25T00:24:24.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.942+0000 7f9b954b3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9b90111990 con 0x7f9b901029d0 2024-04-25T00:24:24.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.942+0000 7f9b954b3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9b90111c90 con 0x7f9b901032f0 2024-04-25T00:24:24.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.942+0000 7f9b8f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b9010bd10 0x7f9b9019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.943+0000 7f9b8f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b9010bd10 0x7f9b9019c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33998/0 (socket says 172.21.15.27:33998) 2024-04-25T00:24:24.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.943+0000 7f9b8f7fe640 1 -- 172.21.15.27:0/4241408131 learned_addr learned my addr 172.21.15.27:0/4241408131 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:24.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.943+0000 7f9b8e7fc640 1 --2- 172.21.15.27:0/4241408131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b901032f0 0x7f9b90198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.943+0000 7f9b8effd640 1 --2- 172.21.15.27:0/4241408131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b901029d0 0x7f9b90197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.943+0000 7f9b8f7fe640 1 -- 172.21.15.27:0/4241408131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b901032f0 msgr2=0x7f9b90198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:24.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.943+0000 7f9b8f7fe640 1 --2- 172.21.15.27:0/4241408131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b901032f0 0x7f9b90198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.943+0000 7f9b8f7fe640 1 -- 172.21.15.27:0/4241408131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b901029d0 msgr2=0x7f9b90197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:24.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.943+0000 7f9b8f7fe640 1 --2- 172.21.15.27:0/4241408131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b901029d0 0x7f9b90197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:24.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.943+0000 7f9b8f7fe640 1 -- 172.21.15.27:0/4241408131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9019cfc0 con 0x7f9b9010bd10 2024-04-25T00:24:24.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.943+0000 7f9b8f7fe640 1 --2- 172.21.15.27:0/4241408131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b9010bd10 0x7f9b9019c890 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f9b80006010 tx=0x7f9b80003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:24.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.943+0000 7f9b87fff640 1 -- 172.21.15.27:0/4241408131 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80047070 con 0x7f9b9010bd10 2024-04-25T00:24:24.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.943+0000 7f9b87fff640 1 -- 172.21.15.27:0/4241408131 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b80003e50 con 0x7f9b9010bd10 2024-04-25T00:24:24.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.943+0000 7f9b954b3640 1 -- 172.21.15.27:0/4241408131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b9019d250 con 0x7f9b9010bd10 2024-04-25T00:24:24.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.943+0000 7f9b87fff640 1 -- 172.21.15.27:0/4241408131 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80033050 con 0x7f9b9010bd10 2024-04-25T00:24:24.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.943+0000 7f9b954b3640 1 -- 172.21.15.27:0/4241408131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b901a4b40 con 0x7f9b9010bd10 2024-04-25T00:24:24.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.944+0000 7f9b954b3640 1 -- 172.21.15.27:0/4241408131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b90068560 con 0x7f9b9010bd10 2024-04-25T00:24:24.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.945+0000 7f9b87fff640 1 -- 172.21.15.27:0/4241408131 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9b8004a070 con 0x7f9b9010bd10 2024-04-25T00:24:24.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.946+0000 7f9b87fff640 1 --2- 172.21.15.27:0/4241408131 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b7c0a2d00 0x7f9b7c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:24.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.946+0000 7f9b8effd640 1 --2- 172.21.15.27:0/4241408131 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b7c0a2d00 0x7f9b7c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:24.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.946+0000 7f9b87fff640 1 -- 172.21.15.27:0/4241408131 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f9b800f0420 con 0x7f9b9010bd10 2024-04-25T00:24:24.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.947+0000 7f9b8effd640 1 --2- 172.21.15.27:0/4241408131 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b7c0a2d00 0x7f9b7c0a51c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9b88005fd0 tx=0x7f9b88005850 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:24.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:24.951+0000 7f9b87fff640 1 -- 172.21.15.27:0/4241408131 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9b8007e740 con 0x7f9b9010bd10 2024-04-25T00:24:25.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.173+0000 7f9b954b3640 1 -- 172.21.15.27:0/4241408131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b90106e90 con 0x7f9b9010bd10 2024-04-25T00:24:25.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.174+0000 7f9b87fff640 1 -- 172.21.15.27:0/4241408131 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+11954 (secure 0 0 0) 0x7f9b800bccd0 con 0x7f9b9010bd10 2024-04-25T00:24:25.174 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:24:25.174 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":97,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:23:11.564498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":147,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":148,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":146,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24514,24430,24521,24428,24496]},"id":1}]} 2024-04-25T00:24:25.175 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 97 2024-04-25T00:24:25.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.176+0000 7f9b954b3640 1 -- 172.21.15.27:0/4241408131 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b7c0a2d00 msgr2=0x7f9b7c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.176+0000 7f9b954b3640 1 --2- 172.21.15.27:0/4241408131 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b7c0a2d00 0x7f9b7c0a51c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9b88005fd0 tx=0x7f9b88005850 comp rx=0 tx=0).stop 2024-04-25T00:24:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.176+0000 7f9b954b3640 1 -- 172.21.15.27:0/4241408131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b9010bd10 msgr2=0x7f9b9019c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.176+0000 7f9b954b3640 1 --2- 172.21.15.27:0/4241408131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b9010bd10 0x7f9b9019c890 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f9b80006010 tx=0x7f9b80003ce0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.176+0000 7f9b954b3640 1 -- 172.21.15.27:0/4241408131 shutdown_connections 2024-04-25T00:24:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.176+0000 7f9b954b3640 1 --2- 172.21.15.27:0/4241408131 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f9b7c0a2d00 0x7f9b7c0a51c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.176+0000 7f9b954b3640 1 --2- 172.21.15.27:0/4241408131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b9010bd10 0x7f9b9019c890 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.176+0000 7f9b954b3640 1 --2- 172.21.15.27:0/4241408131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b901032f0 0x7f9b90198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.176+0000 7f9b954b3640 1 --2- 172.21.15.27:0/4241408131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b901029d0 0x7f9b90197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.176+0000 7f9b954b3640 1 -- 172.21.15.27:0/4241408131 >> 172.21.15.27:0/4241408131 conn(0x7f9b90077740 msgr2=0x7f9b900fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.176+0000 7f9b954b3640 1 -- 172.21.15.27:0/4241408131 shutdown_connections 2024-04-25T00:24:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.176+0000 7f9b954b3640 1 -- 172.21.15.27:0/4241408131 wait complete. 2024-04-25T00:24:25.186 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:24 smithi027 ceph-mon[29208]: pgmap v262: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:24:25.188 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T00:24:25.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:24 smithi139 ceph-mon[32833]: pgmap v262: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:24:25.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:24 smithi153 ceph-mon[32896]: pgmap v262: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:24:25.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.292+0000 7f3dbe914640 1 -- 172.21.15.27:0/3765446360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3db8069dc0 msgr2=0x7f3db806a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.292+0000 7f3dbe914640 1 --2- 172.21.15.27:0/3765446360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3db8069dc0 0x7f3db806a240 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009930 tx=0x7f3da002a420 comp rx=0 tx=0).stop 2024-04-25T00:24:25.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.292+0000 7f3dbe914640 1 -- 172.21.15.27:0/3765446360 shutdown_connections 2024-04-25T00:24:25.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.292+0000 7f3dbe914640 1 --2- 172.21.15.27:0/3765446360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db806a780 0x7f3db810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.292+0000 7f3dbe914640 1 --2- 172.21.15.27:0/3765446360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3db8069dc0 0x7f3db806a240 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.292+0000 7f3dbe914640 1 --2- 172.21.15.27:0/3765446360 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3db8107730 0x7f3db8107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.292+0000 7f3dbe914640 1 -- 172.21.15.27:0/3765446360 >> 172.21.15.27:0/3765446360 conn(0x7f3db80776a0 msgr2=0x7f3db8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.292+0000 7f3dbe914640 1 -- 172.21.15.27:0/3765446360 shutdown_connections 2024-04-25T00:24:25.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.292+0000 7f3dbe914640 1 -- 172.21.15.27:0/3765446360 wait complete. 2024-04-25T00:24:25.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3dbe914640 1 Processor -- start 2024-04-25T00:24:25.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3dbe914640 1 -- start start 2024-04-25T00:24:25.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3dbe914640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3db8069dc0 0x7f3db8197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3dbe914640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3db806a780 0x7f3db81984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3dbe914640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db8107730 0x7f3db819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3dbe914640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3db8111b40 con 0x7f3db8069dc0 2024-04-25T00:24:25.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3dbe914640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3db81119c0 con 0x7f3db806a780 2024-04-25T00:24:25.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3dbe914640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3db8111cc0 con 0x7f3db8107730 2024-04-25T00:24:25.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3db7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3db8069dc0 0x7f3db8197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3db7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3db8069dc0 0x7f3db8197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34010/0 (socket says 172.21.15.27:34010) 2024-04-25T00:24:25.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3db7fff640 1 -- 172.21.15.27:0/3269078165 learned_addr learned my addr 172.21.15.27:0/3269078165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3dbceb1640 1 --2- 172.21.15.27:0/3269078165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db8107730 0x7f3db819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3db77fe640 1 --2- 172.21.15.27:0/3269078165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3db806a780 0x7f3db81984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3db7fff640 1 -- 172.21.15.27:0/3269078165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db8107730 msgr2=0x7f3db819c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3db7fff640 1 --2- 172.21.15.27:0/3269078165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db8107730 0x7f3db819c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3db7fff640 1 -- 172.21.15.27:0/3269078165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3db806a780 msgr2=0x7f3db81984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3db7fff640 1 --2- 172.21.15.27:0/3269078165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3db806a780 0x7f3db81984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3db7fff640 1 -- 172.21.15.27:0/3269078165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db819cf90 con 0x7f3db8069dc0 2024-04-25T00:24:25.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.293+0000 7f3db77fe640 1 --2- 172.21.15.27:0/3269078165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3db806a780 0x7f3db81984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:25.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.294+0000 7f3db7fff640 1 --2- 172.21.15.27:0/3269078165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3db8069dc0 0x7f3db8197fa0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3da800e9f0 tx=0x7f3da800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.294+0000 7f3db57fa640 1 -- 172.21.15.27:0/3269078165 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da800cd80 con 0x7f3db8069dc0 2024-04-25T00:24:25.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.294+0000 7f3db57fa640 1 -- 172.21.15.27:0/3269078165 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da8010910 con 0x7f3db8069dc0 2024-04-25T00:24:25.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.294+0000 7f3dbe914640 1 -- 172.21.15.27:0/3269078165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db819d280 con 0x7f3db8069dc0 2024-04-25T00:24:25.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.294+0000 7f3db57fa640 1 -- 172.21.15.27:0/3269078165 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da8018990 con 0x7f3db8069dc0 2024-04-25T00:24:25.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.294+0000 7f3dbe914640 1 -- 172.21.15.27:0/3269078165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db81a4b10 con 0x7f3db8069dc0 2024-04-25T00:24:25.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.296+0000 7f3db57fa640 1 -- 172.21.15.27:0/3269078165 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3da8010ac0 con 0x7f3db8069dc0 2024-04-25T00:24:25.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.296+0000 7f3dc1623740 1 Processor -- start 2024-04-25T00:24:25.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.296+0000 7f3dc1623740 1 -- start start 2024-04-25T00:24:25.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.296+0000 7f3dc1623740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5cd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.296+0000 7f3dc1623740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5cdda0 0x557fed5dc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dc1623740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5dc980 0x557fed5ded70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dc1623740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557fed532670 con 0x557fed5cd480 2024-04-25T00:24:25.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dc1623740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557fed4da970 con 0x557fed5cdda0 2024-04-25T00:24:25.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dc1623740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557fed52e630 con 0x557fed5dc980 2024-04-25T00:24:25.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dbceb1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dbceb1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5cd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34024/0 (socket says 172.21.15.27:34024) 2024-04-25T00:24:25.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dbceb1640 1 -- 172.21.15.27:0/3758705824 learned_addr learned my addr 172.21.15.27:0/3758705824 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3db57fa640 1 --2- 172.21.15.27:0/3269078165 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f3da40a2d80 0x7f3da40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3db77fe640 1 --2- 172.21.15.27:0/3758705824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5dc980 0x557fed5ded70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3db7fff640 1 --2- 172.21.15.27:0/3758705824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5cdda0 0x557fed5dc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dbceb1640 1 --2- 172.21.15.27:0/3269078165 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f3da40a2d80 0x7f3da40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3db57fa640 1 -- 172.21.15.27:0/3269078165 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f3da80cfaf0 con 0x7f3db8069dc0 2024-04-25T00:24:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dbceb1640 1 -- 172.21.15.27:0/3758705824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5dc980 msgr2=0x557fed5ded70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dbceb1640 1 --2- 172.21.15.27:0/3758705824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5dc980 0x557fed5ded70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dbceb1640 1 -- 172.21.15.27:0/3758705824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5cdda0 msgr2=0x557fed5dc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dbceb1640 1 --2- 172.21.15.27:0/3758705824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5cdda0 0x557fed5dc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dbceb1640 1 -- 172.21.15.27:0/3758705824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fed4fe400 con 0x557fed5cd480 2024-04-25T00:24:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3db7fff640 1 --2- 172.21.15.27:0/3758705824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5cdda0 0x557fed5dc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dbceb1640 1 --2- 172.21.15.27:0/3758705824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5cd860 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f3db0012910 tx=0x7f3db0012dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf566819a6428909 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dae7fc640 1 -- 172.21.15.27:0/3758705824 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db0005020 con 0x557fed5cd480 2024-04-25T00:24:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dae7fc640 1 -- 172.21.15.27:0/3758705824 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3db00056a0 con 0x557fed5cd480 2024-04-25T00:24:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dae7fc640 1 -- 172.21.15.27:0/3758705824 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db001c7f0 con 0x557fed5cd480 2024-04-25T00:24:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.297+0000 7f3dbceb1640 1 --2- 172.21.15.27:0/3269078165 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f3da40a2d80 0x7f3da40a5240 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3db001f000 tx=0x7f3db001e000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.298+0000 7f3dc1623740 1 -- 172.21.15.27:0/3758705824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 msgr2=0x557fed5cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.298+0000 7f3dc1623740 1 --2- 172.21.15.27:0/3758705824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5cd860 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f3db0012910 tx=0x7f3db0012dd0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.298+0000 7f3dc1623740 1 -- 172.21.15.27:0/3758705824 shutdown_connections 2024-04-25T00:24:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.298+0000 7f3dc1623740 1 --2- 172.21.15.27:0/3758705824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5dc980 0x557fed5ded70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.298+0000 7f3dc1623740 1 --2- 172.21.15.27:0/3758705824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5cdda0 0x557fed5dc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.298+0000 7f3dc1623740 1 --2- 172.21.15.27:0/3758705824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5cd860 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.298+0000 7f3dc1623740 1 -- 172.21.15.27:0/3758705824 >> 172.21.15.27:0/3758705824 conn(0x557fed5c98a0 msgr2=0x557fed5c9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.298+0000 7f3dc1623740 1 -- 172.21.15.27:0/3758705824 shutdown_connections 2024-04-25T00:24:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.298+0000 7f3dc1623740 1 -- 172.21.15.27:0/3758705824 wait complete. 2024-04-25T00:24:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.298+0000 7f3dc1623740 1 -- start start 2024-04-25T00:24:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3dc1623740 1 Processor -- start 2024-04-25T00:24:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3dc1623740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5dc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3dc1623740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5cdda0 0x557fed5d9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3dc1623740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5dc980 0x557fed5d9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3dc1623740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557fed5df740 con 0x557fed5cd480 2024-04-25T00:24:25.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3dc1623740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557fed4ee050 con 0x557fed5cdda0 2024-04-25T00:24:25.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3dc1623740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557fed5df8c0 con 0x557fed5dc980 2024-04-25T00:24:25.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3dc1623740 10 client.0 _ll_register_callbacks cb 0x557fed245be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:25.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3db7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5dc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3db7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5dc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34032/0 (socket says 172.21.15.27:34032) 2024-04-25T00:24:25.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3db7fff640 1 -- 172.21.15.27:0/3535264733 learned_addr learned my addr 172.21.15.27:0/3535264733 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3dbceb1640 1 --2- 172.21.15.27:0/3535264733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5dc980 0x557fed5d9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3db7fff640 1 -- 172.21.15.27:0/3535264733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5dc980 msgr2=0x557fed5d9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3db7fff640 1 --2- 172.21.15.27:0/3535264733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5dc980 0x557fed5d9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3db7fff640 1 -- 172.21.15.27:0/3535264733 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5cdda0 msgr2=0x557fed5d9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3db7fff640 1 --2- 172.21.15.27:0/3535264733 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5cdda0 0x557fed5d9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3db7fff640 1 -- 172.21.15.27:0/3535264733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fed4bfdc0 con 0x557fed5cd480 2024-04-25T00:24:25.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3db7fff640 1 --2- 172.21.15.27:0/3535264733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5dc270 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f3da800c6a0 tx=0x7f3da8066d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db3a8388508046 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.299+0000 7f3d977fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:25.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.300+0000 7f3d977fe640 1 -- 172.21.15.27:0/3535264733 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da809c3a0 con 0x557fed5cd480 2024-04-25T00:24:25.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.300+0000 7f3d977fe640 1 -- 172.21.15.27:0/3535264733 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f3da80cd150 con 0x557fed5cd480 2024-04-25T00:24:25.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.300+0000 7f3dc1623740 1 -- 172.21.15.27:0/3535264733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3db00056a0 con 0x557fed5cd480 2024-04-25T00:24:25.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.300+0000 7f3d977fe640 1 -- 172.21.15.27:0/3535264733 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f3da80cd350 con 0x557fed5cd480 2024-04-25T00:24:25.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.300+0000 7f3dc1623740 10 client.16851 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:25.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.300+0000 7f3dc1623740 1 -- 172.21.15.27:0/3535264733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x557fed2463f0 con 0x557fed5cd480 2024-04-25T00:24:25.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.300+0000 7f3d977fe640 1 -- 172.21.15.27:0/3535264733 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da806a020 con 0x557fed5cd480 2024-04-25T00:24:25.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.300+0000 7f3d977fe640 1 -- 172.21.15.27:0/3535264733 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f3da809bf00 con 0x557fed5cd480 2024-04-25T00:24:25.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.300+0000 7f3dc1623740 10 client.16851 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:25.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.300+0000 7f3dc1623740 10 client.16851 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:24:25.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.300+0000 7f3dc1623740 20 client.16851 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:25.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.300+0000 7f3dc1623740 1 --2- 172.21.15.27:0/3535264733 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x557fed5eca30 0x557fed5eee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.300+0000 7f3dc1623740 4 client.16851 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:24:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.300+0000 7f3dc1623740 1 -- 172.21.15.27:0/3535264733 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557fed53fbb0 con 0x557fed5eca30 2024-04-25T00:24:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.300+0000 7f3db77fe640 1 --2- 172.21.15.27:0/3535264733 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x557fed5eca30 0x557fed5eee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.301+0000 7f3d977fe640 10 client.16851 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:24:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.301+0000 7f3db77fe640 1 --2- 172.21.15.27:0/3535264733 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x557fed5eca30 0x557fed5eee80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=faa8f781eb82e4d4 server_cookie=38c2b66a8ba1f7ce in_seq=0 out_seq=0 2024-04-25T00:24:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.302+0000 7f3d977fe640 1 -- 172.21.15.27:0/3535264733 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557fed53fbb0 con 0x557fed5eca30 2024-04-25T00:24:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.302+0000 7f3d977fe640 10 client.16851 handle_command_reply: tid=0 2024-04-25T00:24:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.302+0000 7f3dc1623740 1 client.16851 shutdown 2024-04-25T00:24:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.302+0000 7f3dc1623740 1 -- 172.21.15.27:0/3535264733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 msgr2=0x557fed5dc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.302+0000 7f3dc1623740 1 --2- 172.21.15.27:0/3535264733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5dc270 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f3da800c6a0 tx=0x7f3da8066d80 comp rx=0 tx=0).stop 2024-04-25T00:24:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.302+0000 7f3dc1623740 1 -- 172.21.15.27:0/3535264733 shutdown_connections 2024-04-25T00:24:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.302+0000 7f3dc1623740 1 --2- 172.21.15.27:0/3535264733 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x557fed5eca30 0x557fed5eee80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.302+0000 7f3dc1623740 1 --2- 172.21.15.27:0/3535264733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5dc980 0x557fed5d9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.302+0000 7f3dc1623740 1 --2- 172.21.15.27:0/3535264733 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5cdda0 0x557fed5d9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.302+0000 7f3d977fe640 0 client.16851 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:24:25.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.302+0000 7f3dc1623740 1 --2- 172.21.15.27:0/3535264733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5dc270 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.302+0000 7f3dc1623740 1 -- 172.21.15.27:0/3535264733 >> 172.21.15.27:0/3535264733 conn(0x557fed5c98a0 msgr2=0x557fed5c8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.302+0000 7f3dc1623740 1 -- 172.21.15.27:0/3535264733 shutdown_connections 2024-04-25T00:24:25.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.302+0000 7f3dc1623740 1 -- 172.21.15.27:0/3535264733 wait complete. 2024-04-25T00:24:25.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.302+0000 7f3dc1623740 20 client.16851 trim_cache size 0 max 16384 2024-04-25T00:24:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.315+0000 7f3dc1623740 1 Processor -- start 2024-04-25T00:24:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.315+0000 7f3dc1623740 1 -- start start 2024-04-25T00:24:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.315+0000 7f3dc1623740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5cd480 0x557fed5d4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.315+0000 7f3dc1623740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cdda0 0x557fed5cf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.315+0000 7f3dc1623740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5dc980 0x557fed5dbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.315+0000 7f3dc1623740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3db81ac9c0 con 0x557fed5cdda0 2024-04-25T00:24:25.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.315+0000 7f3dc1623740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3db81ae560 con 0x557fed5cd480 2024-04-25T00:24:25.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.315+0000 7f3dc1623740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3db81ad090 con 0x557fed5dc980 2024-04-25T00:24:25.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.315+0000 7f3db77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cdda0 0x557fed5cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.316+0000 7f3db77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cdda0 0x557fed5cf480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34034/0 (socket says 172.21.15.27:34034) 2024-04-25T00:24:25.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.316+0000 7f3db77fe640 1 -- 172.21.15.27:0/3341354104 learned_addr learned my addr 172.21.15.27:0/3341354104 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.316+0000 7f3dbceb1640 1 --2- 172.21.15.27:0/3341354104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5dc980 0x557fed5dbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.316+0000 7f3db7fff640 1 --2- 172.21.15.27:0/3341354104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5cd480 0x557fed5d4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.316+0000 7f3db77fe640 1 -- 172.21.15.27:0/3341354104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5dc980 msgr2=0x557fed5dbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.316+0000 7f3db77fe640 1 --2- 172.21.15.27:0/3341354104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5dc980 0x557fed5dbda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.316+0000 7f3db77fe640 1 -- 172.21.15.27:0/3341354104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5cd480 msgr2=0x557fed5d4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.316+0000 7f3db77fe640 1 --2- 172.21.15.27:0/3341354104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5cd480 0x557fed5d4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.316+0000 7f3db77fe640 1 -- 172.21.15.27:0/3341354104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fed3107d0 con 0x557fed5cdda0 2024-04-25T00:24:25.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.316+0000 7f3db7fff640 1 --2- 172.21.15.27:0/3341354104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5cd480 0x557fed5d4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:25.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.316+0000 7f3db77fe640 1 --2- 172.21.15.27:0/3341354104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cdda0 0x557fed5cf480 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f3da0033000 tx=0x7f3da0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb8e81b8a3a30c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.316+0000 7f3dadffb640 1 -- 172.21.15.27:0/3341354104 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da0035040 con 0x557fed5cdda0 2024-04-25T00:24:25.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.316+0000 7f3dadffb640 1 -- 172.21.15.27:0/3341354104 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da00043c0 con 0x557fed5cdda0 2024-04-25T00:24:25.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.316+0000 7f3dadffb640 1 -- 172.21.15.27:0/3341354104 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da00035f0 con 0x557fed5cdda0 2024-04-25T00:24:25.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 1 -- 172.21.15.27:0/3341354104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cdda0 msgr2=0x557fed5cf480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 1 --2- 172.21.15.27:0/3341354104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cdda0 0x557fed5cf480 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f3da0033000 tx=0x7f3da0005d50 comp rx=0 tx=0).stop 2024-04-25T00:24:25.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 1 -- 172.21.15.27:0/3341354104 shutdown_connections 2024-04-25T00:24:25.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 1 --2- 172.21.15.27:0/3341354104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5dc980 0x557fed5dbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 1 --2- 172.21.15.27:0/3341354104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cdda0 0x557fed5cf480 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 1 --2- 172.21.15.27:0/3341354104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5cd480 0x557fed5d4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 1 -- 172.21.15.27:0/3341354104 >> 172.21.15.27:0/3341354104 conn(0x557fed5c98a0 msgr2=0x557fed5ced70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 1 -- 172.21.15.27:0/3341354104 shutdown_connections 2024-04-25T00:24:25.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 1 -- 172.21.15.27:0/3341354104 wait complete. 2024-04-25T00:24:25.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 1 -- start start 2024-04-25T00:24:25.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 1 Processor -- start 2024-04-25T00:24:25.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5e0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5cdda0 0x557fed5e0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5dc980 0x557fed5e1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3db81ade90 con 0x557fed5cd480 2024-04-25T00:24:25.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3db81ad790 con 0x557fed5dc980 2024-04-25T00:24:25.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3db81aa950 con 0x557fed5cdda0 2024-04-25T00:24:25.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.317+0000 7f3dc1623740 10 client.0 _ll_register_callbacks cb 0x7f3da8009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:25.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3db7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5e0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3db7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5e0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34036/0 (socket says 172.21.15.27:34036) 2024-04-25T00:24:25.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3db7fff640 1 -- 172.21.15.27:0/4271198955 learned_addr learned my addr 172.21.15.27:0/4271198955 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3db77fe640 1 --2- 172.21.15.27:0/4271198955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5cdda0 0x557fed5e0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3dbceb1640 1 --2- 172.21.15.27:0/4271198955 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5dc980 0x557fed5e1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3db7fff640 1 -- 172.21.15.27:0/4271198955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5cdda0 msgr2=0x557fed5e0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3db7fff640 1 --2- 172.21.15.27:0/4271198955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5cdda0 0x557fed5e0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3db7fff640 1 -- 172.21.15.27:0/4271198955 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5dc980 msgr2=0x557fed5e1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3db7fff640 1 --2- 172.21.15.27:0/4271198955 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5dc980 0x557fed5e1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3db7fff640 1 -- 172.21.15.27:0/4271198955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fed652160 con 0x557fed5cd480 2024-04-25T00:24:25.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3dbceb1640 1 --2- 172.21.15.27:0/4271198955 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5dc980 0x557fed5e1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:25.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3db7fff640 1 --2- 172.21.15.27:0/4271198955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5e0710 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f3da80cfef0 tx=0x7f3da809b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=345a9b0aff882b2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3d977fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:25.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3d977fe640 1 -- 172.21.15.27:0/4271198955 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da80cd350 con 0x557fed5cd480 2024-04-25T00:24:25.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3d977fe640 1 -- 172.21.15.27:0/4271198955 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f3da8018b30 con 0x557fed5cd480 2024-04-25T00:24:25.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3dc1623740 1 -- 172.21.15.27:0/4271198955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3da00043c0 con 0x557fed5cd480 2024-04-25T00:24:25.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3d977fe640 1 -- 172.21.15.27:0/4271198955 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f3da809bca0 con 0x557fed5cd480 2024-04-25T00:24:25.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3dc1623740 10 client.16857 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:25.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3dc1623740 1 -- 172.21.15.27:0/4271198955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x557fed653390 con 0x557fed5cd480 2024-04-25T00:24:25.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.318+0000 7f3d977fe640 1 -- 172.21.15.27:0/4271198955 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da00043c0 con 0x557fed5cd480 2024-04-25T00:24:25.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.319+0000 7f3d977fe640 1 -- 172.21.15.27:0/4271198955 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f3da80a4bf0 con 0x557fed5cd480 2024-04-25T00:24:25.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.319+0000 7f3dc1623740 10 client.16857 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:25.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.319+0000 7f3dc1623740 10 client.16857 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:24:25.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.319+0000 7f3dc1623740 20 client.16857 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:25.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.319+0000 7f3dc1623740 1 --2- 172.21.15.27:0/4271198955 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x557fed653ab0 0x557fed655ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.319+0000 7f3dc1623740 4 client.16857 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:24:25.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.319+0000 7f3dc1623740 1 -- 172.21.15.27:0/4271198955 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557fed539f00 con 0x557fed653ab0 2024-04-25T00:24:25.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.319+0000 7f3db77fe640 1 --2- 172.21.15.27:0/4271198955 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x557fed653ab0 0x557fed655ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.319+0000 7f3d977fe640 10 client.16857 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:24:25.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.319+0000 7f3db77fe640 1 --2- 172.21.15.27:0/4271198955 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x557fed653ab0 0x557fed655ea0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=68bfe50079a92756 server_cookie=1158d6f679ae9a33 in_seq=0 out_seq=0 2024-04-25T00:24:25.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.320+0000 7f3d977fe640 1 -- 172.21.15.27:0/4271198955 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557fed539f00 con 0x557fed653ab0 2024-04-25T00:24:25.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.320+0000 7f3d977fe640 10 client.16857 handle_command_reply: tid=0 2024-04-25T00:24:25.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.320+0000 7f3dc1623740 1 client.16857 shutdown 2024-04-25T00:24:25.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.320+0000 7f3dc1623740 1 -- 172.21.15.27:0/4271198955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 msgr2=0x557fed5e0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.320+0000 7f3dc1623740 1 --2- 172.21.15.27:0/4271198955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5e0710 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f3da80cfef0 tx=0x7f3da809b730 comp rx=0 tx=0).stop 2024-04-25T00:24:25.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.320+0000 7f3dc1623740 1 -- 172.21.15.27:0/4271198955 shutdown_connections 2024-04-25T00:24:25.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.320+0000 7f3dc1623740 1 --2- 172.21.15.27:0/4271198955 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x557fed653ab0 0x557fed655ea0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.320+0000 7f3dc1623740 1 --2- 172.21.15.27:0/4271198955 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557fed5dc980 0x557fed5e1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.320+0000 7f3dc1623740 1 --2- 172.21.15.27:0/4271198955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557fed5cdda0 0x557fed5e0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.320+0000 7f3d977fe640 0 client.16857 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:24:25.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.320+0000 7f3dc1623740 1 --2- 172.21.15.27:0/4271198955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fed5cd480 0x557fed5e0710 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.320+0000 7f3dc1623740 1 -- 172.21.15.27:0/4271198955 >> 172.21.15.27:0/4271198955 conn(0x557fed5c98a0 msgr2=0x557fed5cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.320+0000 7f3dc1623740 1 -- 172.21.15.27:0/4271198955 shutdown_connections 2024-04-25T00:24:25.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.320+0000 7f3dc1623740 1 -- 172.21.15.27:0/4271198955 wait complete. 2024-04-25T00:24:25.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.320+0000 7f3dc1623740 20 client.16857 trim_cache size 0 max 16384 2024-04-25T00:24:25.328 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:24:25.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.320+0000 7f3dbe914640 1 -- 172.21.15.27:0/3269078165 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f3da40a2d80 msgr2=0x7f3da40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.320+0000 7f3dbe914640 1 --2- 172.21.15.27:0/3269078165 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f3da40a2d80 0x7f3da40a5240 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3db001f000 tx=0x7f3db001e000 comp rx=0 tx=0).stop 2024-04-25T00:24:25.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.321+0000 7f3dbe914640 1 -- 172.21.15.27:0/3269078165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3db8069dc0 msgr2=0x7f3db8197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.321+0000 7f3dbe914640 1 --2- 172.21.15.27:0/3269078165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3db8069dc0 0x7f3db8197fa0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3da800e9f0 tx=0x7f3da800eeb0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.321+0000 7f3dbe914640 1 -- 172.21.15.27:0/3269078165 shutdown_connections 2024-04-25T00:24:25.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.321+0000 7f3dbe914640 1 --2- 172.21.15.27:0/3269078165 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f3da40a2d80 0x7f3da40a5240 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.321+0000 7f3dbe914640 1 --2- 172.21.15.27:0/3269078165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db8107730 0x7f3db819c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.321+0000 7f3dbe914640 1 --2- 172.21.15.27:0/3269078165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3db806a780 0x7f3db81984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.321+0000 7f3dbe914640 1 --2- 172.21.15.27:0/3269078165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3db8069dc0 0x7f3db8197fa0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.321+0000 7f3dbe914640 1 -- 172.21.15.27:0/3269078165 >> 172.21.15.27:0/3269078165 conn(0x7f3db80776a0 msgr2=0x7f3db80fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.321+0000 7f3dbe914640 1 -- 172.21.15.27:0/3269078165 shutdown_connections 2024-04-25T00:24:25.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.321+0000 7f3dbe914640 1 -- 172.21.15.27:0/3269078165 wait complete. 2024-04-25T00:24:25.331 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T00:24:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.436+0000 7f1cec0bc640 1 -- 172.21.15.27:0/2817484925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ce4069dc0 msgr2=0x7f1ce406a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.436+0000 7f1cec0bc640 1 --2- 172.21.15.27:0/2817484925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ce4069dc0 0x7f1ce406a240 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4009930 tx=0x7f1cd402a420 comp rx=0 tx=0).stop 2024-04-25T00:24:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.436+0000 7f1cec0bc640 1 -- 172.21.15.27:0/2817484925 shutdown_connections 2024-04-25T00:24:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.436+0000 7f1cec0bc640 1 --2- 172.21.15.27:0/2817484925 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ce406a780 0x7f1ce410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.436+0000 7f1cec0bc640 1 --2- 172.21.15.27:0/2817484925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ce4069dc0 0x7f1ce406a240 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.436+0000 7f1cec0bc640 1 --2- 172.21.15.27:0/2817484925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ce4107730 0x7f1ce4107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.436+0000 7f1cec0bc640 1 -- 172.21.15.27:0/2817484925 >> 172.21.15.27:0/2817484925 conn(0x7f1ce40776a0 msgr2=0x7f1ce4077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.436+0000 7f1cec0bc640 1 -- 172.21.15.27:0/2817484925 shutdown_connections 2024-04-25T00:24:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.436+0000 7f1cec0bc640 1 -- 172.21.15.27:0/2817484925 wait complete. 2024-04-25T00:24:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.436+0000 7f1cec0bc640 1 Processor -- start 2024-04-25T00:24:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.436+0000 7f1cec0bc640 1 -- start start 2024-04-25T00:24:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cec0bc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ce4069dc0 0x7f1ce4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cec0bc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ce406a780 0x7f1ce4198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cec0bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ce4107730 0x7f1ce419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cec0bc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce4111b40 con 0x7f1ce4107730 2024-04-25T00:24:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cec0bc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce41119c0 con 0x7f1ce406a780 2024-04-25T00:24:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cec0bc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce4111cc0 con 0x7f1ce4069dc0 2024-04-25T00:24:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cea659640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ce4107730 0x7f1ce419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cea659640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ce4107730 0x7f1ce419c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34052/0 (socket says 172.21.15.27:34052) 2024-04-25T00:24:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cea659640 1 -- 172.21.15.27:0/429913760 learned_addr learned my addr 172.21.15.27:0/429913760 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1ce9e58640 1 --2- 172.21.15.27:0/429913760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ce4069dc0 0x7f1ce4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1ce9657640 1 --2- 172.21.15.27:0/429913760 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ce406a780 0x7f1ce4198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cea659640 1 -- 172.21.15.27:0/429913760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ce4069dc0 msgr2=0x7f1ce4197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cea659640 1 --2- 172.21.15.27:0/429913760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ce4069dc0 0x7f1ce4197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cea659640 1 -- 172.21.15.27:0/429913760 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ce406a780 msgr2=0x7f1ce4198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cea659640 1 --2- 172.21.15.27:0/429913760 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ce406a780 0x7f1ce4198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cea659640 1 -- 172.21.15.27:0/429913760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce419cfd0 con 0x7f1ce4107730 2024-04-25T00:24:25.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1ce9e58640 1 --2- 172.21.15.27:0/429913760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ce4069dc0 0x7f1ce4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:25.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1ce9657640 1 --2- 172.21.15.27:0/429913760 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ce406a780 0x7f1ce4198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:25.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cea659640 1 --2- 172.21.15.27:0/429913760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ce4107730 0x7f1ce419c8a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc00cad0 tx=0x7f1cdc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cdaffd640 1 -- 172.21.15.27:0/429913760 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc00ae20 con 0x7f1ce4107730 2024-04-25T00:24:25.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cdaffd640 1 -- 172.21.15.27:0/429913760 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cdc005020 con 0x7f1ce4107730 2024-04-25T00:24:25.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.437+0000 7f1cec0bc640 1 -- 172.21.15.27:0/429913760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ce419d2c0 con 0x7f1ce4107730 2024-04-25T00:24:25.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.438+0000 7f1cdaffd640 1 -- 172.21.15.27:0/429913760 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc00f6f0 con 0x7f1ce4107730 2024-04-25T00:24:25.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.438+0000 7f1cec0bc640 1 -- 172.21.15.27:0/429913760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ce41a4b50 con 0x7f1ce4107730 2024-04-25T00:24:25.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.440+0000 7f1cdaffd640 1 -- 172.21.15.27:0/429913760 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1cdc0051d0 con 0x7f1ce4107730 2024-04-25T00:24:25.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.440+0000 7f1ceedc5740 1 Processor -- start 2024-04-25T00:24:25.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.440+0000 7f1ceedc5740 1 -- start start 2024-04-25T00:24:25.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.440+0000 7f1ceedc5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d670480 0x56317d670860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.440+0000 7f1ceedc5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d670da0 0x56317d67f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.440+0000 7f1ceedc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67f980 0x56317d681d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.440+0000 7f1ceedc5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56317d5d5670 con 0x56317d670480 2024-04-25T00:24:25.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.440+0000 7f1ceedc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56317d57d970 con 0x56317d67f980 2024-04-25T00:24:25.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.440+0000 7f1ceedc5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56317d5d1630 con 0x56317d670da0 2024-04-25T00:24:25.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1ce9657640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d670480 0x56317d670860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1cdaffd640 1 --2- 172.21.15.27:0/429913760 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1cd00a2d00 0x7f1cd00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1ce9657640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d670480 0x56317d670860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34064/0 (socket says 172.21.15.27:34064) 2024-04-25T00:24:25.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1ce9657640 1 -- 172.21.15.27:0/95101917 learned_addr learned my addr 172.21.15.27:0/95101917 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1ce9e58640 1 --2- 172.21.15.27:0/95101917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d670da0 0x56317d67f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1cea659640 1 --2- 172.21.15.27:0/429913760 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1cd00a2d00 0x7f1cd00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1ce9657640 1 --2- 172.21.15.27:0/95101917 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67f980 0x56317d681d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1cdaffd640 1 -- 172.21.15.27:0/429913760 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f1cdc0cf930 con 0x7f1ce4107730 2024-04-25T00:24:25.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1ce9657640 1 -- 172.21.15.27:0/95101917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d670da0 msgr2=0x56317d67f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1ce9657640 1 --2- 172.21.15.27:0/95101917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d670da0 0x56317d67f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1ce9657640 1 -- 172.21.15.27:0/95101917 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67f980 msgr2=0x56317d681d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1ce9657640 1 --2- 172.21.15.27:0/95101917 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67f980 0x56317d681d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1ce9657640 1 -- 172.21.15.27:0/95101917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56317d5a1400 con 0x56317d670480 2024-04-25T00:24:25.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1ce9e58640 1 --2- 172.21.15.27:0/95101917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d670da0 0x56317d67f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:25.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1ce9657640 1 --2- 172.21.15.27:0/95101917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d670480 0x56317d670860 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4031550 tx=0x7f1cd4005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e4ed4a219d2708 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1cea659640 1 --2- 172.21.15.27:0/429913760 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1cd00a2d00 0x7f1cd00a51c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc09b7e0 tx=0x7f1cdc066b00 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1ccffff640 1 -- 172.21.15.27:0/95101917 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4035050 con 0x56317d670480 2024-04-25T00:24:25.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1ccffff640 1 -- 172.21.15.27:0/95101917 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cd4005040 con 0x56317d670480 2024-04-25T00:24:25.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.441+0000 7f1ccffff640 1 -- 172.21.15.27:0/95101917 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd40048c0 con 0x56317d670480 2024-04-25T00:24:25.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.442+0000 7f1ceedc5740 1 -- 172.21.15.27:0/95101917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d670480 msgr2=0x56317d670860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.442+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/95101917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d670480 0x56317d670860 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4031550 tx=0x7f1cd4005bb0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.442+0000 7f1ceedc5740 1 -- 172.21.15.27:0/95101917 shutdown_connections 2024-04-25T00:24:25.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.442+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/95101917 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67f980 0x56317d681d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.442+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/95101917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d670da0 0x56317d67f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.442+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/95101917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d670480 0x56317d670860 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.442+0000 7f1ceedc5740 1 -- 172.21.15.27:0/95101917 >> 172.21.15.27:0/95101917 conn(0x56317d66c8a0 msgr2=0x56317d66cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.442+0000 7f1ceedc5740 1 -- 172.21.15.27:0/95101917 shutdown_connections 2024-04-25T00:24:25.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.442+0000 7f1ceedc5740 1 -- 172.21.15.27:0/95101917 wait complete. 2024-04-25T00:24:25.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ceedc5740 1 -- start start 2024-04-25T00:24:25.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ceedc5740 1 Processor -- start 2024-04-25T00:24:25.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ceedc5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d670da0 0x56317d67c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ceedc5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d67f980 0x56317d67cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ceedc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67d220 0x56317d68bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ceedc5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56317d6826e0 con 0x56317d67f980 2024-04-25T00:24:25.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ceedc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56317d591050 con 0x56317d67d220 2024-04-25T00:24:25.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ceedc5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56317d682860 con 0x56317d670da0 2024-04-25T00:24:25.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ceedc5740 10 client.0 _ll_register_callbacks cb 0x56317d2e8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:25.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ce9e58640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d67f980 0x56317d67cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ce9e58640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d67f980 0x56317d67cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34076/0 (socket says 172.21.15.27:34076) 2024-04-25T00:24:25.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ce9e58640 1 -- 172.21.15.27:0/2415789542 learned_addr learned my addr 172.21.15.27:0/2415789542 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ce9657640 1 --2- 172.21.15.27:0/2415789542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d670da0 0x56317d67c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ce9657640 1 --2- 172.21.15.27:0/2415789542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67d220 0x56317d68bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ce9e58640 1 -- 172.21.15.27:0/2415789542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d670da0 msgr2=0x56317d67c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ce9e58640 1 --2- 172.21.15.27:0/2415789542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d670da0 0x56317d67c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ce9e58640 1 -- 172.21.15.27:0/2415789542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67d220 msgr2=0x56317d68bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ce9e58640 1 --2- 172.21.15.27:0/2415789542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67d220 0x56317d68bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ce9e58640 1 -- 172.21.15.27:0/2415789542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56317d562dc0 con 0x56317d67f980 2024-04-25T00:24:25.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.443+0000 7f1ce9657640 1 --2- 172.21.15.27:0/2415789542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d670da0 0x56317d67c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:25.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.444+0000 7f1ce9e58640 1 --2- 172.21.15.27:0/2415789542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d67f980 0x56317d67cce0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0010670 tx=0x7f1ce0010b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25c76447f429528 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.444+0000 7f1ccd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:25.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.444+0000 7f1ccd7fa640 1 -- 172.21.15.27:0/2415789542 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0004220 con 0x56317d67f980 2024-04-25T00:24:25.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.444+0000 7f1ccd7fa640 1 -- 172.21.15.27:0/2415789542 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f1ce00043c0 con 0x56317d67f980 2024-04-25T00:24:25.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.444+0000 7f1ceedc5740 1 -- 172.21.15.27:0/2415789542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cd4005040 con 0x56317d67f980 2024-04-25T00:24:25.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.444+0000 7f1ccd7fa640 1 -- 172.21.15.27:0/2415789542 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f1ce0024050 con 0x56317d67f980 2024-04-25T00:24:25.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.444+0000 7f1ceedc5740 10 client.16869 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:25.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.444+0000 7f1ceedc5740 1 -- 172.21.15.27:0/2415789542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x56317d2e93f0 con 0x56317d67f980 2024-04-25T00:24:25.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.444+0000 7f1ccd7fa640 1 -- 172.21.15.27:0/2415789542 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cd4005040 con 0x56317d67f980 2024-04-25T00:24:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.444+0000 7f1ccd7fa640 1 -- 172.21.15.27:0/2415789542 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f1ce000f040 con 0x56317d67f980 2024-04-25T00:24:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.444+0000 7f1ceedc5740 10 client.16869 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.444+0000 7f1ceedc5740 10 client.16869 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:24:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.444+0000 7f1ceedc5740 20 client.16869 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.444+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/2415789542 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x56317d690060 0x56317d692480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.444+0000 7f1ceedc5740 4 client.16869 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:24:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.444+0000 7f1ceedc5740 1 -- 172.21.15.27:0/2415789542 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56317d5e2bb0 con 0x56317d690060 2024-04-25T00:24:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.445+0000 7f1ce9657640 1 --2- 172.21.15.27:0/2415789542 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x56317d690060 0x56317d692480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.445+0000 7f1ccd7fa640 10 client.16869 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:24:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.445+0000 7f1ce9657640 1 --2- 172.21.15.27:0/2415789542 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x56317d690060 0x56317d692480 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ca3216f1143b548e server_cookie=7f5ef228e75ec5bf in_seq=0 out_seq=0 2024-04-25T00:24:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.446+0000 7f1ccd7fa640 1 -- 172.21.15.27:0/2415789542 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56317d5e2bb0 con 0x56317d690060 2024-04-25T00:24:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.446+0000 7f1ccd7fa640 10 client.16869 handle_command_reply: tid=0 2024-04-25T00:24:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.446+0000 7f1ceedc5740 1 client.16869 shutdown 2024-04-25T00:24:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.446+0000 7f1ceedc5740 1 -- 172.21.15.27:0/2415789542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d67f980 msgr2=0x56317d67cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.446+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/2415789542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d67f980 0x56317d67cce0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0010670 tx=0x7f1ce0010b30 comp rx=0 tx=0).stop 2024-04-25T00:24:25.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.446+0000 7f1ceedc5740 1 -- 172.21.15.27:0/2415789542 shutdown_connections 2024-04-25T00:24:25.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.446+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/2415789542 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x56317d690060 0x56317d692480 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.446+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/2415789542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67d220 0x56317d68bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.446+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/2415789542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d67f980 0x56317d67cce0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.446+0000 7f1ccd7fa640 0 client.16869 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:24:25.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.446+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/2415789542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d670da0 0x56317d67c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.446+0000 7f1ceedc5740 1 -- 172.21.15.27:0/2415789542 >> 172.21.15.27:0/2415789542 conn(0x56317d66c8a0 msgr2=0x56317d6726d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.446+0000 7f1ceedc5740 1 -- 172.21.15.27:0/2415789542 shutdown_connections 2024-04-25T00:24:25.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.446+0000 7f1ceedc5740 1 -- 172.21.15.27:0/2415789542 wait complete. 2024-04-25T00:24:25.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.447+0000 7f1ceedc5740 20 client.16869 trim_cache size 0 max 16384 2024-04-25T00:24:25.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ceedc5740 1 Processor -- start 2024-04-25T00:24:25.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ceedc5740 1 -- start start 2024-04-25T00:24:25.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ceedc5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d670da0 0x56317d66c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ceedc5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d67d220 0x56317d6773e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ceedc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67f980 0x56317d697d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ceedc5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce41aca40 con 0x56317d67d220 2024-04-25T00:24:25.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ceedc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce41ae5e0 con 0x56317d67f980 2024-04-25T00:24:25.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ceedc5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce41ad110 con 0x56317d670da0 2024-04-25T00:24:25.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ce9e58640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d67d220 0x56317d6773e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ce9e58640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d67d220 0x56317d6773e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34090/0 (socket says 172.21.15.27:34090) 2024-04-25T00:24:25.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ce9e58640 1 -- 172.21.15.27:0/4137434799 learned_addr learned my addr 172.21.15.27:0/4137434799 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ce9657640 1 --2- 172.21.15.27:0/4137434799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d670da0 0x56317d66c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ce9657640 1 --2- 172.21.15.27:0/4137434799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67f980 0x56317d697d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ce9e58640 1 -- 172.21.15.27:0/4137434799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d670da0 msgr2=0x56317d66c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ce9e58640 1 --2- 172.21.15.27:0/4137434799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d670da0 0x56317d66c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ce9e58640 1 -- 172.21.15.27:0/4137434799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67f980 msgr2=0x56317d697d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ce9e58640 1 --2- 172.21.15.27:0/4137434799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67f980 0x56317d697d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ce9e58640 1 -- 172.21.15.27:0/4137434799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56317d3b37d0 con 0x56317d67d220 2024-04-25T00:24:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.460+0000 7f1ce9657640 1 --2- 172.21.15.27:0/4137434799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67f980 0x56317d697d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.461+0000 7f1ce9e58640 1 --2- 172.21.15.27:0/4137434799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d67d220 0x56317d6773e0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0006010 tx=0x7f1ce0009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c975a7c370f4310 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.461+0000 7f1ccf7fe640 1 -- 172.21.15.27:0/4137434799 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce000e6c0 con 0x56317d67d220 2024-04-25T00:24:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.461+0000 7f1ccf7fe640 1 -- 172.21.15.27:0/4137434799 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ce0010110 con 0x56317d67d220 2024-04-25T00:24:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.461+0000 7f1ccf7fe640 1 -- 172.21.15.27:0/4137434799 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce000ecf0 con 0x56317d67d220 2024-04-25T00:24:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.461+0000 7f1ceedc5740 1 -- 172.21.15.27:0/4137434799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d67d220 msgr2=0x56317d6773e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.461+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/4137434799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d67d220 0x56317d6773e0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0006010 tx=0x7f1ce0009f60 comp rx=0 tx=0).stop 2024-04-25T00:24:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.461+0000 7f1ceedc5740 1 -- 172.21.15.27:0/4137434799 shutdown_connections 2024-04-25T00:24:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.461+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/4137434799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67f980 0x56317d697d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.461+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/4137434799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d67d220 0x56317d6773e0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.461+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/4137434799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d670da0 0x56317d66c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.461+0000 7f1ceedc5740 1 -- 172.21.15.27:0/4137434799 >> 172.21.15.27:0/4137434799 conn(0x56317d66c8a0 msgr2=0x56317d6726a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.461+0000 7f1ceedc5740 1 -- 172.21.15.27:0/4137434799 shutdown_connections 2024-04-25T00:24:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.461+0000 7f1ceedc5740 1 -- 172.21.15.27:0/4137434799 wait complete. 2024-04-25T00:24:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ceedc5740 1 -- start start 2024-04-25T00:24:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ceedc5740 1 Processor -- start 2024-04-25T00:24:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ceedc5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d670da0 0x56317d687fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ceedc5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d67d220 0x56317d683310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ceedc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67f980 0x56317d683920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ceedc5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce41adf10 con 0x56317d670da0 2024-04-25T00:24:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ceedc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce41ad810 con 0x56317d67f980 2024-04-25T00:24:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ceedc5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce41aa9d0 con 0x56317d67d220 2024-04-25T00:24:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ceedc5740 10 client.0 _ll_register_callbacks cb 0x7f1ce0009380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ce9657640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d670da0 0x56317d687fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ce9657640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d670da0 0x56317d687fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34094/0 (socket says 172.21.15.27:34094) 2024-04-25T00:24:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ce9657640 1 -- 172.21.15.27:0/3680623269 learned_addr learned my addr 172.21.15.27:0/3680623269 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ce9e58640 1 --2- 172.21.15.27:0/3680623269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d67d220 0x56317d683310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ce9657640 1 -- 172.21.15.27:0/3680623269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d67d220 msgr2=0x56317d683310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ce9657640 1 --2- 172.21.15.27:0/3680623269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d67d220 0x56317d683310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ce9657640 1 -- 172.21.15.27:0/3680623269 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67f980 msgr2=0x56317d683920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ce9657640 1 --2- 172.21.15.27:0/3680623269 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67f980 0x56317d683920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ce9657640 1 -- 172.21.15.27:0/3680623269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56317d684150 con 0x56317d670da0 2024-04-25T00:24:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.462+0000 7f1ce9e58640 1 --2- 172.21.15.27:0/3680623269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d67d220 0x56317d683310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.463+0000 7f1ce9657640 1 --2- 172.21.15.27:0/3680623269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d670da0 0x56317d687fa0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f1cd402a940 tx=0x7f1cd4005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb93522d3bd13e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.463+0000 7f1ccd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.463+0000 7f1ccd7fa640 1 -- 172.21.15.27:0/3680623269 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4057060 con 0x56317d670da0 2024-04-25T00:24:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.463+0000 7f1ccd7fa640 1 -- 172.21.15.27:0/3680623269 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f1cd4052b50 con 0x56317d670da0 2024-04-25T00:24:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.463+0000 7f1ceedc5740 1 -- 172.21.15.27:0/3680623269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ce0010110 con 0x56317d670da0 2024-04-25T00:24:25.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.463+0000 7f1ccd7fa640 1 -- 172.21.15.27:0/3680623269 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f1cd405e050 con 0x56317d670da0 2024-04-25T00:24:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.463+0000 7f1ceedc5740 10 client.16875 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.463+0000 7f1ceedc5740 1 -- 172.21.15.27:0/3680623269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x56317d6f66d0 con 0x56317d670da0 2024-04-25T00:24:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.463+0000 7f1ccd7fa640 1 -- 172.21.15.27:0/3680623269 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ce0010110 con 0x56317d670da0 2024-04-25T00:24:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.463+0000 7f1ccd7fa640 1 -- 172.21.15.27:0/3680623269 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f1cd4005340 con 0x56317d670da0 2024-04-25T00:24:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.463+0000 7f1ceedc5740 10 client.16875 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.463+0000 7f1ceedc5740 10 client.16875 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:24:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.463+0000 7f1ceedc5740 20 client.16875 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.463+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/3680623269 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x56317d6f6a10 0x56317d6f8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.463+0000 7f1ceedc5740 4 client.16875 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:24:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.463+0000 7f1ceedc5740 1 -- 172.21.15.27:0/3680623269 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56317d5dcf00 con 0x56317d6f6a10 2024-04-25T00:24:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.464+0000 7f1ce9e58640 1 --2- 172.21.15.27:0/3680623269 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x56317d6f6a10 0x56317d6f8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.464+0000 7f1ce9e58640 1 --2- 172.21.15.27:0/3680623269 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x56317d6f6a10 0x56317d6f8e30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=537c9482e203531a server_cookie=8f86fc3a1018f588 in_seq=0 out_seq=0 2024-04-25T00:24:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.464+0000 7f1ccd7fa640 10 client.16875 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:24:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.464+0000 7f1ccd7fa640 1 -- 172.21.15.27:0/3680623269 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56317d5dcf00 con 0x56317d6f6a10 2024-04-25T00:24:25.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.464+0000 7f1ccd7fa640 10 client.16875 handle_command_reply: tid=0 2024-04-25T00:24:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.464+0000 7f1ceedc5740 1 client.16875 shutdown 2024-04-25T00:24:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.464+0000 7f1ceedc5740 1 -- 172.21.15.27:0/3680623269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d670da0 msgr2=0x56317d687fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.464+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/3680623269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d670da0 0x56317d687fa0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f1cd402a940 tx=0x7f1cd4005bb0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.464+0000 7f1ceedc5740 1 -- 172.21.15.27:0/3680623269 shutdown_connections 2024-04-25T00:24:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.464+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/3680623269 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x56317d6f6a10 0x56317d6f8e30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.464+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/3680623269 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56317d67f980 0x56317d683920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.464+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/3680623269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56317d67d220 0x56317d683310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.464+0000 7f1ccd7fa640 0 client.16875 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:24:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.464+0000 7f1ceedc5740 1 --2- 172.21.15.27:0/3680623269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56317d670da0 0x56317d687fa0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.464+0000 7f1ceedc5740 1 -- 172.21.15.27:0/3680623269 >> 172.21.15.27:0/3680623269 conn(0x56317d66c8a0 msgr2=0x56317d671d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.465+0000 7f1ceedc5740 1 -- 172.21.15.27:0/3680623269 shutdown_connections 2024-04-25T00:24:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.465+0000 7f1ceedc5740 1 -- 172.21.15.27:0/3680623269 wait complete. 2024-04-25T00:24:25.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.465+0000 7f1ceedc5740 20 client.16875 trim_cache size 0 max 16384 2024-04-25T00:24:25.472 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:24:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.465+0000 7f1cec0bc640 1 -- 172.21.15.27:0/429913760 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1cd00a2d00 msgr2=0x7f1cd00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.465+0000 7f1cec0bc640 1 --2- 172.21.15.27:0/429913760 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1cd00a2d00 0x7f1cd00a51c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc09b7e0 tx=0x7f1cdc066b00 comp rx=0 tx=0).stop 2024-04-25T00:24:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.465+0000 7f1cec0bc640 1 -- 172.21.15.27:0/429913760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ce4107730 msgr2=0x7f1ce419c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.465+0000 7f1cec0bc640 1 --2- 172.21.15.27:0/429913760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ce4107730 0x7f1ce419c8a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc00cad0 tx=0x7f1cdc00cf90 comp rx=0 tx=0).stop 2024-04-25T00:24:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.465+0000 7f1cec0bc640 1 -- 172.21.15.27:0/429913760 shutdown_connections 2024-04-25T00:24:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.465+0000 7f1cec0bc640 1 --2- 172.21.15.27:0/429913760 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1cd00a2d00 0x7f1cd00a51c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.465+0000 7f1cec0bc640 1 --2- 172.21.15.27:0/429913760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ce4107730 0x7f1ce419c8a0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.465+0000 7f1cec0bc640 1 --2- 172.21.15.27:0/429913760 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ce406a780 0x7f1ce4198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.465+0000 7f1cec0bc640 1 --2- 172.21.15.27:0/429913760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ce4069dc0 0x7f1ce4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.465+0000 7f1cec0bc640 1 -- 172.21.15.27:0/429913760 >> 172.21.15.27:0/429913760 conn(0x7f1ce40776a0 msgr2=0x7f1ce40fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.465+0000 7f1cec0bc640 1 -- 172.21.15.27:0/429913760 shutdown_connections 2024-04-25T00:24:25.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.465+0000 7f1cec0bc640 1 -- 172.21.15.27:0/429913760 wait complete. 2024-04-25T00:24:25.476 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T00:24:25.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.580+0000 7f4248712640 1 -- 172.21.15.27:0/2631698104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f424006a780 msgr2=0x7f424010e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.580+0000 7f4248712640 1 --2- 172.21.15.27:0/2631698104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f424006a780 0x7f424010e5f0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f4238009cd0 tx=0x7f423802a8e0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.580+0000 7f4248712640 1 -- 172.21.15.27:0/2631698104 shutdown_connections 2024-04-25T00:24:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.580+0000 7f4248712640 1 --2- 172.21.15.27:0/2631698104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f424006a780 0x7f424010e5f0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.580+0000 7f4248712640 1 --2- 172.21.15.27:0/2631698104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4240069dc0 0x7f424006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.580+0000 7f4248712640 1 --2- 172.21.15.27:0/2631698104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4240107730 0x7f4240107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.580+0000 7f4248712640 1 -- 172.21.15.27:0/2631698104 >> 172.21.15.27:0/2631698104 conn(0x7f42400776a0 msgr2=0x7f4240077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.580+0000 7f4248712640 1 -- 172.21.15.27:0/2631698104 shutdown_connections 2024-04-25T00:24:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.580+0000 7f4248712640 1 -- 172.21.15.27:0/2631698104 wait complete. 2024-04-25T00:24:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f4248712640 1 Processor -- start 2024-04-25T00:24:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f4248712640 1 -- start start 2024-04-25T00:24:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f4248712640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4240069dc0 0x7f4240100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f4248712640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f424006a780 0x7f4240100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f4248712640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4240107730 0x7f42401010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f4248712640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4240111b90 con 0x7f4240069dc0 2024-04-25T00:24:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f4248712640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4240111a10 con 0x7f4240107730 2024-04-25T00:24:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f4248712640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4240111d10 con 0x7f424006a780 2024-04-25T00:24:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f42464ae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4240069dc0 0x7f4240100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f42464ae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4240069dc0 0x7f4240100670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34100/0 (socket says 172.21.15.27:34100) 2024-04-25T00:24:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f42464ae640 1 -- 172.21.15.27:0/1806238066 learned_addr learned my addr 172.21.15.27:0/1806238066 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f4245cad640 1 --2- 172.21.15.27:0/1806238066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f424006a780 0x7f4240100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f4246caf640 1 --2- 172.21.15.27:0/1806238066 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4240107730 0x7f42401010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f42464ae640 1 -- 172.21.15.27:0/1806238066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f424006a780 msgr2=0x7f4240100bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f42464ae640 1 --2- 172.21.15.27:0/1806238066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f424006a780 0x7f4240100bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f42464ae640 1 -- 172.21.15.27:0/1806238066 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4240107730 msgr2=0x7f42401010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f42464ae640 1 --2- 172.21.15.27:0/1806238066 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4240107730 0x7f42401010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f42464ae640 1 -- 172.21.15.27:0/1806238066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42401a6410 con 0x7f4240069dc0 2024-04-25T00:24:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.581+0000 7f4246caf640 1 --2- 172.21.15.27:0/1806238066 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4240107730 0x7f42401010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.582+0000 7f42464ae640 1 --2- 172.21.15.27:0/1806238066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4240069dc0 0x7f4240100670 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f423c00c920 tx=0x7f423c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.582+0000 7f42377fe640 1 -- 172.21.15.27:0/1806238066 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c007c70 con 0x7f4240069dc0 2024-04-25T00:24:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.582+0000 7f4248712640 1 -- 172.21.15.27:0/1806238066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42401a6700 con 0x7f4240069dc0 2024-04-25T00:24:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.582+0000 7f42377fe640 1 -- 172.21.15.27:0/1806238066 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f423c007e10 con 0x7f4240069dc0 2024-04-25T00:24:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.582+0000 7f42377fe640 1 -- 172.21.15.27:0/1806238066 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c018890 con 0x7f4240069dc0 2024-04-25T00:24:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.582+0000 7f4248712640 1 -- 172.21.15.27:0/1806238066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42401a6ca0 con 0x7f4240069dc0 2024-04-25T00:24:25.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.584+0000 7f42377fe640 1 -- 172.21.15.27:0/1806238066 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f423c018a70 con 0x7f4240069dc0 2024-04-25T00:24:25.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.584+0000 7f424b416740 1 Processor -- start 2024-04-25T00:24:25.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f424b416740 1 -- start start 2024-04-25T00:24:25.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f424b416740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93580 0x562e11c93960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f424b416740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93ea0 0x562e11c9e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f424b416740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11ca0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f424b416740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562e11bf8670 con 0x562e11c9e570 2024-04-25T00:24:25.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f424b416740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x562e11ba0970 con 0x562e11c93580 2024-04-25T00:24:25.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f424b416740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562e11bf4630 con 0x562e11c93ea0 2024-04-25T00:24:25.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f4245cad640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11ca0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f42377fe640 1 --2- 172.21.15.27:0/1806238066 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f422c0a2d00 0x7f422c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f4245cad640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11ca0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34106/0 (socket says 172.21.15.27:34106) 2024-04-25T00:24:25.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f4245cad640 1 -- 172.21.15.27:0/2975667265 learned_addr learned my addr 172.21.15.27:0/2975667265 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f4246caf640 1 --2- 172.21.15.27:0/2975667265 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93580 0x562e11c93960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f4246caf640 1 --2- 172.21.15.27:0/1806238066 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f422c0a2d00 0x7f422c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f42464ae640 1 --2- 172.21.15.27:0/2975667265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93ea0 0x562e11c9e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f42377fe640 1 -- 172.21.15.27:0/1806238066 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f423c0cf900 con 0x7f4240069dc0 2024-04-25T00:24:25.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f4245cad640 1 -- 172.21.15.27:0/2975667265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93ea0 msgr2=0x562e11c9e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f4245cad640 1 --2- 172.21.15.27:0/2975667265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93ea0 0x562e11c9e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f4245cad640 1 -- 172.21.15.27:0/2975667265 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93580 msgr2=0x562e11c93960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f4245cad640 1 --2- 172.21.15.27:0/2975667265 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93580 0x562e11c93960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.585+0000 7f4245cad640 1 -- 172.21.15.27:0/2975667265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e11bc4400 con 0x562e11c9e570 2024-04-25T00:24:25.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.586+0000 7f4245cad640 1 --2- 172.21.15.27:0/2975667265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11ca0960 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f423000b740 tx=0x7f423000bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5758af61a162f6e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.586+0000 7f4246caf640 1 --2- 172.21.15.27:0/1806238066 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f422c0a2d00 0x7f422c0a51c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f423802ae00 tx=0x7f42380095c0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.586+0000 7f4234ff9640 1 -- 172.21.15.27:0/2975667265 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4230003dd0 con 0x562e11c9e570 2024-04-25T00:24:25.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.586+0000 7f4234ff9640 1 -- 172.21.15.27:0/2975667265 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4230015070 con 0x562e11c9e570 2024-04-25T00:24:25.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.586+0000 7f4234ff9640 1 -- 172.21.15.27:0/2975667265 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4230004e00 con 0x562e11c9e570 2024-04-25T00:24:25.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.586+0000 7f424b416740 1 -- 172.21.15.27:0/2975667265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 msgr2=0x562e11ca0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.586+0000 7f424b416740 1 --2- 172.21.15.27:0/2975667265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11ca0960 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f423000b740 tx=0x7f423000bc00 comp rx=0 tx=0).stop 2024-04-25T00:24:25.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f424b416740 1 -- 172.21.15.27:0/2975667265 shutdown_connections 2024-04-25T00:24:25.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f424b416740 1 --2- 172.21.15.27:0/2975667265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11ca0960 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f424b416740 1 --2- 172.21.15.27:0/2975667265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93ea0 0x562e11c9e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f424b416740 1 --2- 172.21.15.27:0/2975667265 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93580 0x562e11c93960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f424b416740 1 -- 172.21.15.27:0/2975667265 >> 172.21.15.27:0/2975667265 conn(0x562e11c8f8a0 msgr2=0x562e11c8fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f424b416740 1 -- 172.21.15.27:0/2975667265 shutdown_connections 2024-04-25T00:24:25.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f424b416740 1 -- 172.21.15.27:0/2975667265 wait complete. 2024-04-25T00:24:25.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f424b416740 1 -- start start 2024-04-25T00:24:25.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f424b416740 1 Processor -- start 2024-04-25T00:24:25.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f424b416740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93580 0x562e11c9dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f424b416740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93ea0 0x562e11c9b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f424b416740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11c9b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f424b416740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562e11ca1360 con 0x562e11c9e570 2024-04-25T00:24:25.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f424b416740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x562e11bb4050 con 0x562e11c93580 2024-04-25T00:24:25.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f424b416740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562e11ca14e0 con 0x562e11c93ea0 2024-04-25T00:24:25.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f424b416740 10 client.0 _ll_register_callbacks cb 0x562e1190bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f4246caf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11c9b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f4246caf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11c9b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34116/0 (socket says 172.21.15.27:34116) 2024-04-25T00:24:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.587+0000 7f4246caf640 1 -- 172.21.15.27:0/389679135 learned_addr learned my addr 172.21.15.27:0/389679135 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.588+0000 7f4245cad640 1 --2- 172.21.15.27:0/389679135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93ea0 0x562e11c9b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.588+0000 7f42464ae640 1 --2- 172.21.15.27:0/389679135 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93580 0x562e11c9dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.588+0000 7f4246caf640 1 -- 172.21.15.27:0/389679135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93ea0 msgr2=0x562e11c9b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.588+0000 7f4246caf640 1 --2- 172.21.15.27:0/389679135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93ea0 0x562e11c9b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.588+0000 7f4246caf640 1 -- 172.21.15.27:0/389679135 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93580 msgr2=0x562e11c9dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.588+0000 7f4246caf640 1 --2- 172.21.15.27:0/389679135 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93580 0x562e11c9dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.588+0000 7f4246caf640 1 -- 172.21.15.27:0/389679135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e11b85dc0 con 0x562e11c9e570 2024-04-25T00:24:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.588+0000 7f4246caf640 1 --2- 172.21.15.27:0/389679135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11c9b710 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f4238035000 tx=0x7f423802a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ad326324d9f3f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.588+0000 7f4229ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.588+0000 7f4229ffb640 1 -- 172.21.15.27:0/389679135 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42380316a0 con 0x562e11c9e570 2024-04-25T00:24:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.588+0000 7f4229ffb640 1 -- 172.21.15.27:0/389679135 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f42380045c0 con 0x562e11c9e570 2024-04-25T00:24:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.588+0000 7f424b416740 1 -- 172.21.15.27:0/389679135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4230015070 con 0x562e11c9e570 2024-04-25T00:24:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.588+0000 7f4229ffb640 1 -- 172.21.15.27:0/389679135 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f42380047c0 con 0x562e11c9e570 2024-04-25T00:24:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.588+0000 7f424b416740 10 client.16887 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.588+0000 7f424b416740 1 -- 172.21.15.27:0/389679135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x562e1190c3f0 con 0x562e11c9e570 2024-04-25T00:24:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.589+0000 7f4229ffb640 1 -- 172.21.15.27:0/389679135 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4230015070 con 0x562e11c9e570 2024-04-25T00:24:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.589+0000 7f4229ffb640 1 -- 172.21.15.27:0/389679135 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f4238030070 con 0x562e11c9e570 2024-04-25T00:24:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.589+0000 7f424b416740 10 client.16887 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.589+0000 7f424b416740 10 client.16887 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:24:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.589+0000 7f424b416740 20 client.16887 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.589+0000 7f424b416740 1 --2- 172.21.15.27:0/389679135 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x562e11cae560 0x562e11cb09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.589+0000 7f424b416740 4 client.16887 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:24:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.589+0000 7f424b416740 1 -- 172.21.15.27:0/389679135 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e11c05bb0 con 0x562e11cae560 2024-04-25T00:24:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.589+0000 7f42464ae640 1 --2- 172.21.15.27:0/389679135 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x562e11cae560 0x562e11cb09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.589+0000 7f42464ae640 1 --2- 172.21.15.27:0/389679135 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x562e11cae560 0x562e11cb09e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6ff06cace31dac13 server_cookie=7188bf284b5ab7e9 in_seq=0 out_seq=0 2024-04-25T00:24:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.589+0000 7f4229ffb640 10 client.16887 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:24:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.590+0000 7f4229ffb640 1 -- 172.21.15.27:0/389679135 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e11c05bb0 con 0x562e11cae560 2024-04-25T00:24:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.590+0000 7f4229ffb640 10 client.16887 handle_command_reply: tid=0 2024-04-25T00:24:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.590+0000 7f424b416740 1 client.16887 shutdown 2024-04-25T00:24:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.590+0000 7f424b416740 1 -- 172.21.15.27:0/389679135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 msgr2=0x562e11c9b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.590+0000 7f424b416740 1 --2- 172.21.15.27:0/389679135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11c9b710 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f4238035000 tx=0x7f423802a8e0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.590+0000 7f424b416740 1 -- 172.21.15.27:0/389679135 shutdown_connections 2024-04-25T00:24:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.590+0000 7f424b416740 1 --2- 172.21.15.27:0/389679135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11c9b710 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.590+0000 7f424b416740 1 --2- 172.21.15.27:0/389679135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93ea0 0x562e11c9b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.590+0000 7f424b416740 1 --2- 172.21.15.27:0/389679135 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x562e11cae560 0x562e11cb09e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.590+0000 7f424b416740 1 --2- 172.21.15.27:0/389679135 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93580 0x562e11c9dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.590+0000 7f4229ffb640 0 client.16887 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:24:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.590+0000 7f424b416740 1 -- 172.21.15.27:0/389679135 >> 172.21.15.27:0/389679135 conn(0x562e11c8f8a0 msgr2=0x562e11c8eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.590+0000 7f424b416740 1 -- 172.21.15.27:0/389679135 shutdown_connections 2024-04-25T00:24:25.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.590+0000 7f424b416740 1 -- 172.21.15.27:0/389679135 wait complete. 2024-04-25T00:24:25.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.590+0000 7f424b416740 20 client.16887 trim_cache size 0 max 16384 2024-04-25T00:24:25.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f424b416740 1 Processor -- start 2024-04-25T00:24:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f424b416740 1 -- start start 2024-04-25T00:24:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f424b416740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93580 0x562e11caa220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f424b416740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93ea0 0x562e11cb62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f424b416740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11cb67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f424b416740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f42401b0d10 con 0x562e11c9e570 2024-04-25T00:24:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f424b416740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f42401b28b0 con 0x562e11c93580 2024-04-25T00:24:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f424b416740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f42401b13e0 con 0x562e11c93ea0 2024-04-25T00:24:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f4246caf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11cb67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f4246caf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11cb67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34126/0 (socket says 172.21.15.27:34126) 2024-04-25T00:24:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f4246caf640 1 -- 172.21.15.27:0/2503062415 learned_addr learned my addr 172.21.15.27:0/2503062415 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f4245cad640 1 --2- 172.21.15.27:0/2503062415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93ea0 0x562e11cb62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f42464ae640 1 --2- 172.21.15.27:0/2503062415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93580 0x562e11caa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f4246caf640 1 -- 172.21.15.27:0/2503062415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93ea0 msgr2=0x562e11cb62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f4246caf640 1 --2- 172.21.15.27:0/2503062415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93ea0 0x562e11cb62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f4246caf640 1 -- 172.21.15.27:0/2503062415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93580 msgr2=0x562e11caa220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f4246caf640 1 --2- 172.21.15.27:0/2503062415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93580 0x562e11caa220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f4246caf640 1 -- 172.21.15.27:0/2503062415 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e11c9ba80 con 0x562e11c9e570 2024-04-25T00:24:25.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.604+0000 7f4246caf640 1 --2- 172.21.15.27:0/2503062415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11cb67f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f42380505e0 tx=0x7f4238005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965148fe8bdbbacc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.605+0000 7f422bfff640 1 -- 172.21.15.27:0/2503062415 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238004410 con 0x562e11c9e570 2024-04-25T00:24:25.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.605+0000 7f422bfff640 1 -- 172.21.15.27:0/2503062415 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4238036040 con 0x562e11c9e570 2024-04-25T00:24:25.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.605+0000 7f422bfff640 1 -- 172.21.15.27:0/2503062415 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238004da0 con 0x562e11c9e570 2024-04-25T00:24:25.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.605+0000 7f424b416740 1 -- 172.21.15.27:0/2503062415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 msgr2=0x562e11cb67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.605+0000 7f424b416740 1 --2- 172.21.15.27:0/2503062415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11cb67f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f42380505e0 tx=0x7f4238005c50 comp rx=0 tx=0).stop 2024-04-25T00:24:25.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.605+0000 7f424b416740 1 -- 172.21.15.27:0/2503062415 shutdown_connections 2024-04-25T00:24:25.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.605+0000 7f424b416740 1 --2- 172.21.15.27:0/2503062415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11cb67f0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.605+0000 7f424b416740 1 --2- 172.21.15.27:0/2503062415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93ea0 0x562e11cb62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.605+0000 7f424b416740 1 --2- 172.21.15.27:0/2503062415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93580 0x562e11caa220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.605+0000 7f424b416740 1 -- 172.21.15.27:0/2503062415 >> 172.21.15.27:0/2503062415 conn(0x562e11c8f8a0 msgr2=0x562e11c8e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.605+0000 7f424b416740 1 -- 172.21.15.27:0/2503062415 shutdown_connections 2024-04-25T00:24:25.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.605+0000 7f424b416740 1 -- 172.21.15.27:0/2503062415 wait complete. 2024-04-25T00:24:25.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f424b416740 1 -- start start 2024-04-25T00:24:25.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f424b416740 1 Processor -- start 2024-04-25T00:24:25.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f424b416740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93580 0x562e11ca6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f424b416740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93ea0 0x562e11ca6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f424b416740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11ca38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f424b416740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f42401b21e0 con 0x562e11c9e570 2024-04-25T00:24:25.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f424b416740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f42401b1ae0 con 0x562e11c93ea0 2024-04-25T00:24:25.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f424b416740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f42401aeca0 con 0x562e11c93580 2024-04-25T00:24:25.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f424b416740 10 client.0 _ll_register_callbacks cb 0x7f4238009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:25.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f4246caf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11ca38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f4246caf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11ca38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34132/0 (socket says 172.21.15.27:34132) 2024-04-25T00:24:25.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f4246caf640 1 -- 172.21.15.27:0/668900201 learned_addr learned my addr 172.21.15.27:0/668900201 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f42464ae640 1 --2- 172.21.15.27:0/668900201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93580 0x562e11ca6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f4246caf640 1 -- 172.21.15.27:0/668900201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93580 msgr2=0x562e11ca6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f4246caf640 1 --2- 172.21.15.27:0/668900201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93580 0x562e11ca6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f4246caf640 1 -- 172.21.15.27:0/668900201 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93ea0 msgr2=0x562e11ca6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f4246caf640 1 --2- 172.21.15.27:0/668900201 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93ea0 0x562e11ca6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f4246caf640 1 -- 172.21.15.27:0/668900201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e119d67d0 con 0x562e11c9e570 2024-04-25T00:24:25.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f42464ae640 1 --2- 172.21.15.27:0/668900201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93580 0x562e11ca6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:25.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f4246caf640 1 --2- 172.21.15.27:0/668900201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11ca38e0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f4238009bf0 tx=0x7f4238003ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a29243fbd7ceff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.606+0000 7f4229ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:25.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.607+0000 7f4229ffb640 1 -- 172.21.15.27:0/668900201 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238065020 con 0x562e11c9e570 2024-04-25T00:24:25.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.607+0000 7f4229ffb640 1 -- 172.21.15.27:0/668900201 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f4238004d40 con 0x562e11c9e570 2024-04-25T00:24:25.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.607+0000 7f424b416740 1 -- 172.21.15.27:0/668900201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4238036040 con 0x562e11c9e570 2024-04-25T00:24:25.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.607+0000 7f4229ffb640 1 -- 172.21.15.27:0/668900201 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f4238070050 con 0x562e11c9e570 2024-04-25T00:24:25.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.607+0000 7f424b416740 10 client.16893 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:25.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.607+0000 7f424b416740 1 -- 172.21.15.27:0/668900201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x562e11d143f0 con 0x562e11c9e570 2024-04-25T00:24:25.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.607+0000 7f4229ffb640 1 -- 172.21.15.27:0/668900201 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4238036040 con 0x562e11c9e570 2024-04-25T00:24:25.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.607+0000 7f4229ffb640 1 -- 172.21.15.27:0/668900201 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f42380306c0 con 0x562e11c9e570 2024-04-25T00:24:25.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.607+0000 7f424b416740 10 client.16893 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:25.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.607+0000 7f424b416740 10 client.16893 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:24:25.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.607+0000 7f424b416740 20 client.16893 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:25.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.607+0000 7f424b416740 1 --2- 172.21.15.27:0/668900201 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x562e11d14aa0 0x562e11d16ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.607+0000 7f424b416740 4 client.16893 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:24:25.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.607+0000 7f424b416740 1 -- 172.21.15.27:0/668900201 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562e11bfff00 con 0x562e11d14aa0 2024-04-25T00:24:25.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.607+0000 7f42464ae640 1 --2- 172.21.15.27:0/668900201 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x562e11d14aa0 0x562e11d16ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.608+0000 7f4229ffb640 10 client.16893 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:24:25.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.608+0000 7f42464ae640 1 --2- 172.21.15.27:0/668900201 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x562e11d14aa0 0x562e11d16ee0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dd0359da99f67d80 server_cookie=fae1699dde383526 in_seq=0 out_seq=0 2024-04-25T00:24:25.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.608+0000 7f4229ffb640 1 -- 172.21.15.27:0/668900201 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562e11bfff00 con 0x562e11d14aa0 2024-04-25T00:24:25.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.608+0000 7f4229ffb640 10 client.16893 handle_command_reply: tid=0 2024-04-25T00:24:25.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.608+0000 7f424b416740 1 client.16893 shutdown 2024-04-25T00:24:25.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.608+0000 7f424b416740 1 -- 172.21.15.27:0/668900201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 msgr2=0x562e11ca38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.608+0000 7f424b416740 1 --2- 172.21.15.27:0/668900201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11ca38e0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f4238009bf0 tx=0x7f4238003ee0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.608+0000 7f424b416740 1 -- 172.21.15.27:0/668900201 shutdown_connections 2024-04-25T00:24:25.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.608+0000 7f424b416740 1 --2- 172.21.15.27:0/668900201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562e11c9e570 0x562e11ca38e0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.608+0000 7f424b416740 1 --2- 172.21.15.27:0/668900201 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x562e11d14aa0 0x562e11d16ee0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.608+0000 7f424b416740 1 --2- 172.21.15.27:0/668900201 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562e11c93ea0 0x562e11ca6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.608+0000 7f424b416740 1 --2- 172.21.15.27:0/668900201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e11c93580 0x562e11ca6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.608+0000 7f424b416740 1 -- 172.21.15.27:0/668900201 >> 172.21.15.27:0/668900201 conn(0x562e11c8f8a0 msgr2=0x562e11c8eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.608+0000 7f4229ffb640 0 client.16893 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:24:25.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.608+0000 7f424b416740 1 -- 172.21.15.27:0/668900201 shutdown_connections 2024-04-25T00:24:25.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.608+0000 7f424b416740 1 -- 172.21.15.27:0/668900201 wait complete. 2024-04-25T00:24:25.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.608+0000 7f424b416740 20 client.16893 trim_cache size 0 max 16384 2024-04-25T00:24:25.616 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:24:25.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.609+0000 7f4248712640 1 -- 172.21.15.27:0/1806238066 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f422c0a2d00 msgr2=0x7f422c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.609+0000 7f4248712640 1 --2- 172.21.15.27:0/1806238066 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f422c0a2d00 0x7f422c0a51c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f423802ae00 tx=0x7f42380095c0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.609+0000 7f4248712640 1 -- 172.21.15.27:0/1806238066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4240069dc0 msgr2=0x7f4240100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.609+0000 7f4248712640 1 --2- 172.21.15.27:0/1806238066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4240069dc0 0x7f4240100670 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f423c00c920 tx=0x7f423c00cde0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.609+0000 7f4248712640 1 -- 172.21.15.27:0/1806238066 shutdown_connections 2024-04-25T00:24:25.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.609+0000 7f4248712640 1 --2- 172.21.15.27:0/1806238066 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f422c0a2d00 0x7f422c0a51c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.609+0000 7f4248712640 1 --2- 172.21.15.27:0/1806238066 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4240107730 0x7f42401010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.609+0000 7f4248712640 1 --2- 172.21.15.27:0/1806238066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f424006a780 0x7f4240100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.609+0000 7f4248712640 1 --2- 172.21.15.27:0/1806238066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4240069dc0 0x7f4240100670 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.609+0000 7f4248712640 1 -- 172.21.15.27:0/1806238066 >> 172.21.15.27:0/1806238066 conn(0x7f42400776a0 msgr2=0x7f42400fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.609+0000 7f4248712640 1 -- 172.21.15.27:0/1806238066 shutdown_connections 2024-04-25T00:24:25.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.609+0000 7f4248712640 1 -- 172.21.15.27:0/1806238066 wait complete. 2024-04-25T00:24:25.620 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T00:24:25.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.723+0000 7f37f20cd640 1 -- 172.21.15.27:0/1825448895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ec100610 msgr2=0x7f37ec100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.723+0000 7f37f20cd640 1 --2- 172.21.15.27:0/1825448895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ec100610 0x7f37ec100a90 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f37e0009930 tx=0x7f37e002a420 comp rx=0 tx=0).stop 2024-04-25T00:24:25.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.724+0000 7f37f20cd640 1 -- 172.21.15.27:0/1825448895 shutdown_connections 2024-04-25T00:24:25.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.724+0000 7f37f20cd640 1 --2- 172.21.15.27:0/1825448895 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37ec100fd0 0x7f37ec10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.724+0000 7f37f20cd640 1 --2- 172.21.15.27:0/1825448895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ec100610 0x7f37ec100a90 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.724+0000 7f37f20cd640 1 --2- 172.21.15.27:0/1825448895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37ec1066b0 0x7f37ec106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.724+0000 7f37f20cd640 1 -- 172.21.15.27:0/1825448895 >> 172.21.15.27:0/1825448895 conn(0x7f37ec077740 msgr2=0x7f37ec077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.724+0000 7f37f20cd640 1 -- 172.21.15.27:0/1825448895 shutdown_connections 2024-04-25T00:24:25.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.724+0000 7f37f20cd640 1 -- 172.21.15.27:0/1825448895 wait complete. 2024-04-25T00:24:25.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.724+0000 7f37f20cd640 1 Processor -- start 2024-04-25T00:24:25.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.724+0000 7f37f20cd640 1 -- start start 2024-04-25T00:24:25.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37f20cd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37ec100610 0x7f37ec19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37f20cd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ec100fd0 0x7f37ec19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37f20cd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37ec1066b0 0x7f37ec19eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37f20cd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec111b30 con 0x7f37ec100fd0 2024-04-25T00:24:25.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37f20cd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec1119b0 con 0x7f37ec1066b0 2024-04-25T00:24:25.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37f20cd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec111cb0 con 0x7f37ec100610 2024-04-25T00:24:25.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37eaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ec100fd0 0x7f37ec19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37eaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ec100fd0 0x7f37ec19a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34148/0 (socket says 172.21.15.27:34148) 2024-04-25T00:24:25.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37eaffd640 1 -- 172.21.15.27:0/3218164426 learned_addr learned my addr 172.21.15.27:0/3218164426 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37ebfff640 1 --2- 172.21.15.27:0/3218164426 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37ec1066b0 0x7f37ec19eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37eb7fe640 1 --2- 172.21.15.27:0/3218164426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37ec100610 0x7f37ec19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37eaffd640 1 -- 172.21.15.27:0/3218164426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37ec100610 msgr2=0x7f37ec19a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37eaffd640 1 --2- 172.21.15.27:0/3218164426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37ec100610 0x7f37ec19a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37eaffd640 1 -- 172.21.15.27:0/3218164426 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37ec1066b0 msgr2=0x7f37ec19eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37eaffd640 1 --2- 172.21.15.27:0/3218164426 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37ec1066b0 0x7f37ec19eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37eaffd640 1 -- 172.21.15.27:0/3218164426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37ec19f1e0 con 0x7f37ec100fd0 2024-04-25T00:24:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37eaffd640 1 --2- 172.21.15.27:0/3218164426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ec100fd0 0x7f37ec19a730 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f37e0009230 tx=0x7f37e0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37e8ff9640 1 -- 172.21.15.27:0/3218164426 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e003f030 con 0x7f37ec100fd0 2024-04-25T00:24:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37f20cd640 1 -- 172.21.15.27:0/3218164426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37ec19f470 con 0x7f37ec100fd0 2024-04-25T00:24:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37e8ff9640 1 -- 172.21.15.27:0/3218164426 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37e0003ad0 con 0x7f37ec100fd0 2024-04-25T00:24:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37e8ff9640 1 -- 172.21.15.27:0/3218164426 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e0035070 con 0x7f37ec100fd0 2024-04-25T00:24:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.725+0000 7f37f20cd640 1 -- 172.21.15.27:0/3218164426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37ec1a6d60 con 0x7f37ec100fd0 2024-04-25T00:24:25.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37e8ff9640 1 -- 172.21.15.27:0/3218164426 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f37e0042030 con 0x7f37ec100fd0 2024-04-25T00:24:25.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37f4dda740 1 Processor -- start 2024-04-25T00:24:25.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37f4dda740 1 -- start start 2024-04-25T00:24:25.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37f4dda740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x564596849960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37f4dda740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596849ea0 0x564596854030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37f4dda740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596854570 0x564596856960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37f4dda740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5645967ae670 con 0x564596849580 2024-04-25T00:24:25.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37f4dda740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564596756970 con 0x564596849ea0 2024-04-25T00:24:25.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37f4dda740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5645967aa630 con 0x564596854570 2024-04-25T00:24:25.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37ebfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x564596849960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37ebfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x564596849960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34156/0 (socket says 172.21.15.27:34156) 2024-04-25T00:24:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37ebfff640 1 -- 172.21.15.27:0/922581509 learned_addr learned my addr 172.21.15.27:0/922581509 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37eaffd640 1 --2- 172.21.15.27:0/922581509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596854570 0x564596856960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37ebfff640 1 -- 172.21.15.27:0/922581509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596854570 msgr2=0x564596856960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37eb7fe640 1 --2- 172.21.15.27:0/922581509 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596849ea0 0x564596854030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37ebfff640 1 --2- 172.21.15.27:0/922581509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596854570 0x564596856960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37e8ff9640 1 --2- 172.21.15.27:0/3218164426 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f37d40a2d00 0x7f37d40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37ebfff640 1 -- 172.21.15.27:0/922581509 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596849ea0 msgr2=0x564596854030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37ebfff640 1 --2- 172.21.15.27:0/922581509 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596849ea0 0x564596854030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37ebfff640 1 -- 172.21.15.27:0/922581509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56459677a400 con 0x564596849580 2024-04-25T00:24:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.728+0000 7f37eb7fe640 1 --2- 172.21.15.27:0/922581509 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596849ea0 0x564596854030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.729+0000 7f37e8ff9640 1 -- 172.21.15.27:0/3218164426 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f37e00efd70 con 0x7f37ec100fd0 2024-04-25T00:24:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.729+0000 7f37ebfff640 1 --2- 172.21.15.27:0/3218164426 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f37d40a2d00 0x7f37d40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.729+0000 7f37ebfff640 1 --2- 172.21.15.27:0/922581509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x564596849960 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f37e4009ee0 tx=0x7f37e4010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2284263b99c42667 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.729+0000 7f37d1ffb640 1 -- 172.21.15.27:0/922581509 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4021070 con 0x564596849580 2024-04-25T00:24:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.729+0000 7f37d1ffb640 1 -- 172.21.15.27:0/922581509 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37e40113e0 con 0x564596849580 2024-04-25T00:24:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.729+0000 7f37d1ffb640 1 -- 172.21.15.27:0/922581509 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4016070 con 0x564596849580 2024-04-25T00:24:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37f4dda740 1 -- 172.21.15.27:0/922581509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 msgr2=0x564596849960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37ebfff640 1 --2- 172.21.15.27:0/3218164426 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f37d40a2d00 0x7f37d40a51c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f37e401e000 tx=0x7f37e4021000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37f4dda740 1 --2- 172.21.15.27:0/922581509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x564596849960 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f37e4009ee0 tx=0x7f37e4010740 comp rx=0 tx=0).stop 2024-04-25T00:24:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37f4dda740 1 -- 172.21.15.27:0/922581509 shutdown_connections 2024-04-25T00:24:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37f4dda740 1 --2- 172.21.15.27:0/922581509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596854570 0x564596856960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37f4dda740 1 --2- 172.21.15.27:0/922581509 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596849ea0 0x564596854030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37f4dda740 1 --2- 172.21.15.27:0/922581509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x564596849960 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37f4dda740 1 -- 172.21.15.27:0/922581509 >> 172.21.15.27:0/922581509 conn(0x5645968458a0 msgr2=0x564596845c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37f4dda740 1 -- 172.21.15.27:0/922581509 shutdown_connections 2024-04-25T00:24:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37f4dda740 1 -- 172.21.15.27:0/922581509 wait complete. 2024-04-25T00:24:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37f4dda740 1 -- start start 2024-04-25T00:24:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37f4dda740 1 Processor -- start 2024-04-25T00:24:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37f4dda740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596849580 0x564596853ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37f4dda740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849ea0 0x5645968511c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37f4dda740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596854570 0x564596851720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37f4dda740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564596857360 con 0x564596849ea0 2024-04-25T00:24:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37f4dda740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56459676a050 con 0x564596854570 2024-04-25T00:24:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.730+0000 7f37f4dda740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5645968574e0 con 0x564596849580 2024-04-25T00:24:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37f4dda740 10 client.0 _ll_register_callbacks cb 0x5645964c1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37eaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849ea0 0x5645968511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37eaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849ea0 0x5645968511c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34172/0 (socket says 172.21.15.27:34172) 2024-04-25T00:24:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37eaffd640 1 -- 172.21.15.27:0/1151600479 learned_addr learned my addr 172.21.15.27:0/1151600479 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37ebfff640 1 --2- 172.21.15.27:0/1151600479 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596854570 0x564596851720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37eb7fe640 1 --2- 172.21.15.27:0/1151600479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596849580 0x564596853ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37eaffd640 1 -- 172.21.15.27:0/1151600479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596849580 msgr2=0x564596853ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37eaffd640 1 --2- 172.21.15.27:0/1151600479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596849580 0x564596853ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37eaffd640 1 -- 172.21.15.27:0/1151600479 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596854570 msgr2=0x564596851720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37eaffd640 1 --2- 172.21.15.27:0/1151600479 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596854570 0x564596851720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37eaffd640 1 -- 172.21.15.27:0/1151600479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56459673bdc0 con 0x564596849ea0 2024-04-25T00:24:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37eb7fe640 1 --2- 172.21.15.27:0/1151600479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596849580 0x564596853ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37eaffd640 1 --2- 172.21.15.27:0/1151600479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849ea0 0x5645968511c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f37e0009900 tx=0x7f37e00bb940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1432c0624f1e5a58 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37caffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37caffd640 1 -- 172.21.15.27:0/1151600479 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e0049050 con 0x564596849ea0 2024-04-25T00:24:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37caffd640 1 -- 172.21.15.27:0/1151600479 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f37e0003c80 con 0x564596849ea0 2024-04-25T00:24:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37f4dda740 1 -- 172.21.15.27:0/1151600479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37e40113e0 con 0x564596849ea0 2024-04-25T00:24:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37caffd640 1 -- 172.21.15.27:0/1151600479 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f37e00c02d0 con 0x564596849ea0 2024-04-25T00:24:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37f4dda740 10 client.16905 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.731+0000 7f37f4dda740 1 -- 172.21.15.27:0/1151600479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5645964c23f0 con 0x564596849ea0 2024-04-25T00:24:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.732+0000 7f37caffd640 1 -- 172.21.15.27:0/1151600479 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37e40113e0 con 0x564596849ea0 2024-04-25T00:24:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.732+0000 7f37caffd640 1 -- 172.21.15.27:0/1151600479 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f37e00c4700 con 0x564596849ea0 2024-04-25T00:24:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.732+0000 7f37f4dda740 10 client.16905 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.732+0000 7f37f4dda740 10 client.16905 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:24:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.732+0000 7f37f4dda740 20 client.16905 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.732+0000 7f37f4dda740 1 --2- 172.21.15.27:0/1151600479 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5645968643f0 0x5645968667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.732+0000 7f37f4dda740 4 client.16905 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:24:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.732+0000 7f37f4dda740 1 -- 172.21.15.27:0/1151600479 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645967bbbb0 con 0x5645968643f0 2024-04-25T00:24:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.732+0000 7f37eb7fe640 1 --2- 172.21.15.27:0/1151600479 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5645968643f0 0x5645968667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.733+0000 7f37caffd640 10 client.16905 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:24:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.733+0000 7f37eb7fe640 1 --2- 172.21.15.27:0/1151600479 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5645968643f0 0x5645968667e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7ecff52a9293a9c1 server_cookie=2e462967c778e2d4 in_seq=0 out_seq=0 2024-04-25T00:24:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.733+0000 7f37caffd640 1 -- 172.21.15.27:0/1151600479 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5645967bbbb0 con 0x5645968643f0 2024-04-25T00:24:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.733+0000 7f37caffd640 10 client.16905 handle_command_reply: tid=0 2024-04-25T00:24:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.733+0000 7f37f4dda740 1 client.16905 shutdown 2024-04-25T00:24:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.733+0000 7f37f4dda740 1 -- 172.21.15.27:0/1151600479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849ea0 msgr2=0x5645968511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.733+0000 7f37f4dda740 1 --2- 172.21.15.27:0/1151600479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849ea0 0x5645968511c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f37e0009900 tx=0x7f37e00bb940 comp rx=0 tx=0).stop 2024-04-25T00:24:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.733+0000 7f37f4dda740 1 -- 172.21.15.27:0/1151600479 shutdown_connections 2024-04-25T00:24:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.733+0000 7f37f4dda740 1 --2- 172.21.15.27:0/1151600479 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596854570 0x564596851720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.733+0000 7f37f4dda740 1 --2- 172.21.15.27:0/1151600479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849ea0 0x5645968511c0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.734+0000 7f37f4dda740 1 --2- 172.21.15.27:0/1151600479 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5645968643f0 0x5645968667e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.734+0000 7f37f4dda740 1 --2- 172.21.15.27:0/1151600479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596849580 0x564596853ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.734+0000 7f37caffd640 0 client.16905 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:24:25.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.734+0000 7f37f4dda740 1 -- 172.21.15.27:0/1151600479 >> 172.21.15.27:0/1151600479 conn(0x5645968458a0 msgr2=0x564596844b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.734+0000 7f37f4dda740 1 -- 172.21.15.27:0/1151600479 shutdown_connections 2024-04-25T00:24:25.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.734+0000 7f37f4dda740 1 -- 172.21.15.27:0/1151600479 wait complete. 2024-04-25T00:24:25.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.734+0000 7f37f4dda740 20 client.16905 trim_cache size 0 max 16384 2024-04-25T00:24:25.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.747+0000 7f37f4dda740 1 Processor -- start 2024-04-25T00:24:25.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.747+0000 7f37f4dda740 1 -- start start 2024-04-25T00:24:25.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.747+0000 7f37f4dda740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x564596853a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.747+0000 7f37f4dda740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596849ea0 0x56459685c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.747+0000 7f37f4dda740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596854570 0x56459685cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.747+0000 7f37f4dda740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec1b0d70 con 0x564596849580 2024-04-25T00:24:25.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.747+0000 7f37f4dda740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec1b2910 con 0x564596849ea0 2024-04-25T00:24:25.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.747+0000 7f37f4dda740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec1b1440 con 0x564596854570 2024-04-25T00:24:25.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.747+0000 7f37eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x564596853a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.747+0000 7f37eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x564596853a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34176/0 (socket says 172.21.15.27:34176) 2024-04-25T00:24:25.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.747+0000 7f37eb7fe640 1 -- 172.21.15.27:0/3619610191 learned_addr learned my addr 172.21.15.27:0/3619610191 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.747+0000 7f37eb7fe640 1 -- 172.21.15.27:0/3619610191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596854570 msgr2=0x56459685cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.747+0000 7f37eb7fe640 1 --2- 172.21.15.27:0/3619610191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596854570 0x56459685cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.747+0000 7f37eb7fe640 1 -- 172.21.15.27:0/3619610191 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596849ea0 msgr2=0x56459685c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.747+0000 7f37eb7fe640 1 --2- 172.21.15.27:0/3619610191 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596849ea0 0x56459685c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.747+0000 7f37eb7fe640 1 -- 172.21.15.27:0/3619610191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56459658c7d0 con 0x564596849580 2024-04-25T00:24:25.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.748+0000 7f37eb7fe640 1 --2- 172.21.15.27:0/3619610191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x564596853a00 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f37ec101750 tx=0x7f37dc00b3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1592aa7c4b6e5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.748+0000 7f37d17fa640 1 -- 172.21.15.27:0/3619610191 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc01e020 con 0x564596849580 2024-04-25T00:24:25.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.748+0000 7f37d17fa640 1 -- 172.21.15.27:0/3619610191 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37dc01a2c0 con 0x564596849580 2024-04-25T00:24:25.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.748+0000 7f37d17fa640 1 -- 172.21.15.27:0/3619610191 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc003850 con 0x564596849580 2024-04-25T00:24:25.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.748+0000 7f37f4dda740 1 -- 172.21.15.27:0/3619610191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 msgr2=0x564596853a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.748+0000 7f37f4dda740 1 --2- 172.21.15.27:0/3619610191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x564596853a00 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f37ec101750 tx=0x7f37dc00b3a0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.748+0000 7f37f4dda740 1 -- 172.21.15.27:0/3619610191 shutdown_connections 2024-04-25T00:24:25.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.748+0000 7f37f4dda740 1 --2- 172.21.15.27:0/3619610191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596854570 0x56459685cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.748+0000 7f37f4dda740 1 --2- 172.21.15.27:0/3619610191 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596849ea0 0x56459685c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.748+0000 7f37f4dda740 1 --2- 172.21.15.27:0/3619610191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x564596853a00 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.748+0000 7f37f4dda740 1 -- 172.21.15.27:0/3619610191 >> 172.21.15.27:0/3619610191 conn(0x5645968458a0 msgr2=0x5645968442b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.748+0000 7f37f4dda740 1 -- 172.21.15.27:0/3619610191 shutdown_connections 2024-04-25T00:24:25.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.748+0000 7f37f4dda740 1 -- 172.21.15.27:0/3619610191 wait complete. 2024-04-25T00:24:25.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.749+0000 7f37f4dda740 1 -- start start 2024-04-25T00:24:25.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.749+0000 7f37f4dda740 1 Processor -- start 2024-04-25T00:24:25.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.749+0000 7f37f4dda740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x56459685a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.749+0000 7f37f4dda740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596849ea0 0x56459685af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.749+0000 7f37f4dda740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596854570 0x5645968d2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.749+0000 7f37f4dda740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec1b2240 con 0x564596849580 2024-04-25T00:24:25.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.749+0000 7f37f4dda740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec1b1b40 con 0x564596849ea0 2024-04-25T00:24:25.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.749+0000 7f37f4dda740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec1aed00 con 0x564596854570 2024-04-25T00:24:25.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.749+0000 7f37f4dda740 10 client.0 _ll_register_callbacks cb 0x7f37e0005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:25.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.749+0000 7f37eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x56459685a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.749+0000 7f37eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x56459685a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34184/0 (socket says 172.21.15.27:34184) 2024-04-25T00:24:25.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.749+0000 7f37eb7fe640 1 -- 172.21.15.27:0/696138599 learned_addr learned my addr 172.21.15.27:0/696138599 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.749+0000 7f37eaffd640 1 --2- 172.21.15.27:0/696138599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596849ea0 0x56459685af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37ebfff640 1 --2- 172.21.15.27:0/696138599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596854570 0x5645968d2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37eb7fe640 1 -- 172.21.15.27:0/696138599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596854570 msgr2=0x5645968d2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37eb7fe640 1 --2- 172.21.15.27:0/696138599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596854570 0x5645968d2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37eb7fe640 1 -- 172.21.15.27:0/696138599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596849ea0 msgr2=0x56459685af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37eb7fe640 1 --2- 172.21.15.27:0/696138599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596849ea0 0x56459685af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37eb7fe640 1 -- 172.21.15.27:0/696138599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645968d2690 con 0x564596849580 2024-04-25T00:24:25.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37eb7fe640 1 --2- 172.21.15.27:0/696138599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x56459685a9f0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00bb00 tx=0x7f37dc01ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8b7a2b73d8222 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37caffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:25.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37caffd640 1 -- 172.21.15.27:0/696138599 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc01a880 con 0x564596849580 2024-04-25T00:24:25.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37caffd640 1 -- 172.21.15.27:0/696138599 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f37dc02d440 con 0x564596849580 2024-04-25T00:24:25.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37f4dda740 1 -- 172.21.15.27:0/696138599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37dc01a2c0 con 0x564596849580 2024-04-25T00:24:25.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37caffd640 1 -- 172.21.15.27:0/696138599 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f37dc036050 con 0x564596849580 2024-04-25T00:24:25.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37f4dda740 10 client.16911 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:25.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37f4dda740 1 -- 172.21.15.27:0/696138599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5645968d3990 con 0x564596849580 2024-04-25T00:24:25.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37caffd640 1 -- 172.21.15.27:0/696138599 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37dc01a2c0 con 0x564596849580 2024-04-25T00:24:25.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37caffd640 1 -- 172.21.15.27:0/696138599 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f37dc02b020 con 0x564596849580 2024-04-25T00:24:25.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37f4dda740 10 client.16911 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:25.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.750+0000 7f37f4dda740 10 client.16911 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:24:25.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.751+0000 7f37f4dda740 20 client.16911 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:25.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.751+0000 7f37f4dda740 1 --2- 172.21.15.27:0/696138599 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5645968d4140 0x5645968d6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.751+0000 7f37f4dda740 4 client.16911 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:24:25.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.751+0000 7f37f4dda740 1 -- 172.21.15.27:0/696138599 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5645967b5f00 con 0x5645968d4140 2024-04-25T00:24:25.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.751+0000 7f37eaffd640 1 --2- 172.21.15.27:0/696138599 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5645968d4140 0x5645968d6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.751+0000 7f37eaffd640 1 --2- 172.21.15.27:0/696138599 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5645968d4140 0x5645968d6560 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=da38ca4da3f3d4e1 server_cookie=fa51abd4268d3bd4 in_seq=0 out_seq=0 2024-04-25T00:24:25.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.751+0000 7f37caffd640 10 client.16911 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:24:25.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.751+0000 7f37caffd640 1 -- 172.21.15.27:0/696138599 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5645967b5f00 con 0x5645968d4140 2024-04-25T00:24:25.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.751+0000 7f37caffd640 10 client.16911 handle_command_reply: tid=0 2024-04-25T00:24:25.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.751+0000 7f37f4dda740 1 client.16911 shutdown 2024-04-25T00:24:25.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f4dda740 1 -- 172.21.15.27:0/696138599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 msgr2=0x56459685a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f4dda740 1 --2- 172.21.15.27:0/696138599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x56459685a9f0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00bb00 tx=0x7f37dc01ad10 comp rx=0 tx=0).stop 2024-04-25T00:24:25.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f4dda740 1 -- 172.21.15.27:0/696138599 shutdown_connections 2024-04-25T00:24:25.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f4dda740 1 --2- 172.21.15.27:0/696138599 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5645968d4140 0x5645968d6560 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f4dda740 1 --2- 172.21.15.27:0/696138599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564596854570 0x5645968d2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f4dda740 1 --2- 172.21.15.27:0/696138599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564596849ea0 0x56459685af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37caffd640 0 client.16911 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:24:25.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f4dda740 1 --2- 172.21.15.27:0/696138599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564596849580 0x56459685a9f0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f4dda740 1 -- 172.21.15.27:0/696138599 >> 172.21.15.27:0/696138599 conn(0x5645968458a0 msgr2=0x564596844b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f4dda740 1 -- 172.21.15.27:0/696138599 shutdown_connections 2024-04-25T00:24:25.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f4dda740 1 -- 172.21.15.27:0/696138599 wait complete. 2024-04-25T00:24:25.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f4dda740 20 client.16911 trim_cache size 0 max 16384 2024-04-25T00:24:25.759 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:24:25.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f20cd640 1 -- 172.21.15.27:0/3218164426 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f37d40a2d00 msgr2=0x7f37d40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f20cd640 1 --2- 172.21.15.27:0/3218164426 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f37d40a2d00 0x7f37d40a51c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f37e401e000 tx=0x7f37e4021000 comp rx=0 tx=0).stop 2024-04-25T00:24:25.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f20cd640 1 -- 172.21.15.27:0/3218164426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ec100fd0 msgr2=0x7f37ec19a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f20cd640 1 --2- 172.21.15.27:0/3218164426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ec100fd0 0x7f37ec19a730 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f37e0009230 tx=0x7f37e0004270 comp rx=0 tx=0).stop 2024-04-25T00:24:25.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f20cd640 1 -- 172.21.15.27:0/3218164426 shutdown_connections 2024-04-25T00:24:25.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f20cd640 1 --2- 172.21.15.27:0/3218164426 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f37d40a2d00 0x7f37d40a51c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f20cd640 1 --2- 172.21.15.27:0/3218164426 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37ec1066b0 0x7f37ec19eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f20cd640 1 --2- 172.21.15.27:0/3218164426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ec100fd0 0x7f37ec19a730 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f20cd640 1 --2- 172.21.15.27:0/3218164426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37ec100610 0x7f37ec19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.752+0000 7f37f20cd640 1 -- 172.21.15.27:0/3218164426 >> 172.21.15.27:0/3218164426 conn(0x7f37ec077740 msgr2=0x7f37ec0feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.753+0000 7f37f20cd640 1 -- 172.21.15.27:0/3218164426 shutdown_connections 2024-04-25T00:24:25.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.753+0000 7f37f20cd640 1 -- 172.21.15.27:0/3218164426 wait complete. 2024-04-25T00:24:25.763 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T00:24:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.868+0000 7f0edf8bd640 1 -- 172.21.15.27:0/514273731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed8069dc0 msgr2=0x7f0ed806a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.868+0000 7f0edf8bd640 1 --2- 172.21.15.27:0/514273731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed8069dc0 0x7f0ed806a240 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8009930 tx=0x7f0ec802a420 comp rx=0 tx=0).stop 2024-04-25T00:24:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.868+0000 7f0edf8bd640 1 -- 172.21.15.27:0/514273731 shutdown_connections 2024-04-25T00:24:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.868+0000 7f0edf8bd640 1 --2- 172.21.15.27:0/514273731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ed806a780 0x7f0ed810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.868+0000 7f0edf8bd640 1 --2- 172.21.15.27:0/514273731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed8069dc0 0x7f0ed806a240 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.868+0000 7f0edf8bd640 1 --2- 172.21.15.27:0/514273731 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ed8107730 0x7f0ed8107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.868+0000 7f0edf8bd640 1 -- 172.21.15.27:0/514273731 >> 172.21.15.27:0/514273731 conn(0x7f0ed80776a0 msgr2=0x7f0ed8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.868+0000 7f0edf8bd640 1 -- 172.21.15.27:0/514273731 shutdown_connections 2024-04-25T00:24:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.868+0000 7f0edf8bd640 1 -- 172.21.15.27:0/514273731 wait complete. 2024-04-25T00:24:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edf8bd640 1 Processor -- start 2024-04-25T00:24:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edf8bd640 1 -- start start 2024-04-25T00:24:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edf8bd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed8069dc0 0x7f0ed8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edf8bd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ed806a780 0x7f0ed8198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edf8bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ed8107730 0x7f0ed819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edf8bd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed8111b90 con 0x7f0ed8069dc0 2024-04-25T00:24:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edf8bd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed8111a10 con 0x7f0ed806a780 2024-04-25T00:24:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edf8bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed8111d10 con 0x7f0ed8107730 2024-04-25T00:24:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edd659640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed8069dc0 0x7f0ed8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edd659640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed8069dc0 0x7f0ed8197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34196/0 (socket says 172.21.15.27:34196) 2024-04-25T00:24:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edd659640 1 -- 172.21.15.27:0/1353392171 learned_addr learned my addr 172.21.15.27:0/1353392171 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edd659640 1 -- 172.21.15.27:0/1353392171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ed8107730 msgr2=0x7f0ed819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edd659640 1 --2- 172.21.15.27:0/1353392171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ed8107730 0x7f0ed819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edce58640 1 --2- 172.21.15.27:0/1353392171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ed806a780 0x7f0ed8198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edd659640 1 -- 172.21.15.27:0/1353392171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ed806a780 msgr2=0x7f0ed8198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edd659640 1 --2- 172.21.15.27:0/1353392171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ed806a780 0x7f0ed8198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edd659640 1 -- 172.21.15.27:0/1353392171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed819cfc0 con 0x7f0ed8069dc0 2024-04-25T00:24:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.869+0000 7f0edce58640 1 --2- 172.21.15.27:0/1353392171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ed806a780 0x7f0ed8198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.870+0000 7f0edd659640 1 --2- 172.21.15.27:0/1353392171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed8069dc0 0x7f0ed8197fd0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0ed400c890 tx=0x7f0ed400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.870+0000 7f0ece7fc640 1 -- 172.21.15.27:0/1353392171 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4004220 con 0x7f0ed8069dc0 2024-04-25T00:24:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.870+0000 7f0edf8bd640 1 -- 172.21.15.27:0/1353392171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ed819d2b0 con 0x7f0ed8069dc0 2024-04-25T00:24:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.870+0000 7f0ece7fc640 1 -- 172.21.15.27:0/1353392171 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ed40043c0 con 0x7f0ed8069dc0 2024-04-25T00:24:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.870+0000 7f0ece7fc640 1 -- 172.21.15.27:0/1353392171 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4004a00 con 0x7f0ed8069dc0 2024-04-25T00:24:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.870+0000 7f0edf8bd640 1 -- 172.21.15.27:0/1353392171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ed81a4b40 con 0x7f0ed8069dc0 2024-04-25T00:24:25.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.872+0000 7f0ece7fc640 1 -- 172.21.15.27:0/1353392171 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0ed40079b0 con 0x7f0ed8069dc0 2024-04-25T00:24:25.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.872+0000 7f0ee25cf740 1 Processor -- start 2024-04-25T00:24:25.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.872+0000 7f0ee25cf740 1 -- start start 2024-04-25T00:24:25.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0ee25cf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a10580 0x556915a10960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0ee25cf740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10ea0 0x556915a1b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0ee25cf740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a1b570 0x556915a1d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0ee25cf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556915975670 con 0x556915a10580 2024-04-25T00:24:25.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0ee25cf740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55691591d970 con 0x556915a10ea0 2024-04-25T00:24:25.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0ee25cf740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556915971630 con 0x556915a1b570 2024-04-25T00:24:25.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0edde5a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a10580 0x556915a10960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0edde5a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a10580 0x556915a10960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34212/0 (socket says 172.21.15.27:34212) 2024-04-25T00:24:25.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0edde5a640 1 -- 172.21.15.27:0/57715708 learned_addr learned my addr 172.21.15.27:0/57715708 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0edce58640 1 --2- 172.21.15.27:0/57715708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a1b570 0x556915a1d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0ece7fc640 1 --2- 172.21.15.27:0/1353392171 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0ec40a2d00 0x7f0ec40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0edd659640 1 --2- 172.21.15.27:0/57715708 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10ea0 0x556915a1b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0edde5a640 1 -- 172.21.15.27:0/57715708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a1b570 msgr2=0x556915a1d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0edde5a640 1 --2- 172.21.15.27:0/57715708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a1b570 0x556915a1d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0edde5a640 1 -- 172.21.15.27:0/57715708 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10ea0 msgr2=0x556915a1b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0edde5a640 1 --2- 172.21.15.27:0/57715708 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10ea0 0x556915a1b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0edde5a640 1 -- 172.21.15.27:0/57715708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556915941400 con 0x556915a10580 2024-04-25T00:24:25.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0ece7fc640 1 -- 172.21.15.27:0/1353392171 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f0ed40cf460 con 0x7f0ed8069dc0 2024-04-25T00:24:25.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0edde5a640 1 --2- 172.21.15.27:0/1353392171 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0ec40a2d00 0x7f0ec40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.873+0000 7f0edde5a640 1 --2- 172.21.15.27:0/57715708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a10580 0x556915a10960 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0007dd0 tx=0x7f0ed000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d0eded2e2815d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.874+0000 7f0ec37fe640 1 -- 172.21.15.27:0/57715708 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0016070 con 0x556915a10580 2024-04-25T00:24:25.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.874+0000 7f0edde5a640 1 --2- 172.21.15.27:0/1353392171 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0ec40a2d00 0x7f0ec40a51c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0ed001d000 tx=0x7f0ed0020000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.874+0000 7f0ec37fe640 1 -- 172.21.15.27:0/57715708 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ed001bae0 con 0x556915a10580 2024-04-25T00:24:25.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.874+0000 7f0ec37fe640 1 -- 172.21.15.27:0/57715708 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed000f9a0 con 0x556915a10580 2024-04-25T00:24:25.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.874+0000 7f0ee25cf740 1 -- 172.21.15.27:0/57715708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a10580 msgr2=0x556915a10960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.874+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/57715708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a10580 0x556915a10960 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0007dd0 tx=0x7f0ed000c6a0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.874+0000 7f0ee25cf740 1 -- 172.21.15.27:0/57715708 shutdown_connections 2024-04-25T00:24:25.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.874+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/57715708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a1b570 0x556915a1d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.874+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/57715708 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10ea0 0x556915a1b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.874+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/57715708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a10580 0x556915a10960 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.874+0000 7f0ee25cf740 1 -- 172.21.15.27:0/57715708 >> 172.21.15.27:0/57715708 conn(0x556915a0c8a0 msgr2=0x556915a0cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.874+0000 7f0ee25cf740 1 -- 172.21.15.27:0/57715708 shutdown_connections 2024-04-25T00:24:25.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.875+0000 7f0ee25cf740 1 -- 172.21.15.27:0/57715708 wait complete. 2024-04-25T00:24:25.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.875+0000 7f0ee25cf740 1 -- start start 2024-04-25T00:24:25.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.875+0000 7f0ee25cf740 1 Processor -- start 2024-04-25T00:24:25.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.875+0000 7f0ee25cf740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10580 0x556915a1af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.875+0000 7f0ee25cf740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a10ea0 0x556915a18220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.875+0000 7f0ee25cf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a1b570 0x556915a18830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.875+0000 7f0ee25cf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556915a1e360 con 0x556915a1b570 2024-04-25T00:24:25.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.875+0000 7f0ee25cf740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556915931050 con 0x556915a10580 2024-04-25T00:24:25.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.875+0000 7f0ee25cf740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556915a1e4e0 con 0x556915a10ea0 2024-04-25T00:24:25.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.875+0000 7f0ee25cf740 10 client.0 _ll_register_callbacks cb 0x556915688be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:25.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.875+0000 7f0edde5a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a1b570 0x556915a18830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.875+0000 7f0edde5a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a1b570 0x556915a18830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34214/0 (socket says 172.21.15.27:34214) 2024-04-25T00:24:25.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.875+0000 7f0edde5a640 1 -- 172.21.15.27:0/1203836281 learned_addr learned my addr 172.21.15.27:0/1203836281 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.876+0000 7f0edde5a640 1 -- 172.21.15.27:0/1203836281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a10ea0 msgr2=0x556915a18220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.876+0000 7f0edde5a640 1 --2- 172.21.15.27:0/1203836281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a10ea0 0x556915a18220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.876+0000 7f0edde5a640 1 -- 172.21.15.27:0/1203836281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10580 msgr2=0x556915a1af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.876+0000 7f0edde5a640 1 --2- 172.21.15.27:0/1203836281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10580 0x556915a1af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.876+0000 7f0edde5a640 1 -- 172.21.15.27:0/1203836281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556915902dc0 con 0x556915a1b570 2024-04-25T00:24:25.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.876+0000 7f0edde5a640 1 --2- 172.21.15.27:0/1203836281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a1b570 0x556915a18830 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000fa50 tx=0x7f0ed000f440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3470fee1fb180c6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.876+0000 7f0ec0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:25.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.876+0000 7f0ec0ff9640 1 -- 172.21.15.27:0/1203836281 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0016040 con 0x556915a1b570 2024-04-25T00:24:25.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.876+0000 7f0ec0ff9640 1 -- 172.21.15.27:0/1203836281 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f0ed001b400 con 0x556915a1b570 2024-04-25T00:24:25.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.876+0000 7f0ee25cf740 1 -- 172.21.15.27:0/1203836281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ed001bae0 con 0x556915a1b570 2024-04-25T00:24:25.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.876+0000 7f0ec0ff9640 1 -- 172.21.15.27:0/1203836281 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f0ed001b600 con 0x556915a1b570 2024-04-25T00:24:25.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.876+0000 7f0ee25cf740 10 client.16923 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:25.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.876+0000 7f0ee25cf740 1 -- 172.21.15.27:0/1203836281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5569156893f0 con 0x556915a1b570 2024-04-25T00:24:25.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.876+0000 7f0ec0ff9640 1 -- 172.21.15.27:0/1203836281 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ed001bae0 con 0x556915a1b570 2024-04-25T00:24:25.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.876+0000 7f0ec0ff9640 1 -- 172.21.15.27:0/1203836281 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f0ed001a070 con 0x556915a1b570 2024-04-25T00:24:25.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.877+0000 7f0ee25cf740 10 client.16923 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:25.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.877+0000 7f0ee25cf740 10 client.16923 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:24:25.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.877+0000 7f0ee25cf740 20 client.16923 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:25.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.877+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/1203836281 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556915a2b620 0x556915a2da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.877+0000 7f0ee25cf740 4 client.16923 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:24:25.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.877+0000 7f0ee25cf740 1 -- 172.21.15.27:0/1203836281 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556915982bb0 con 0x556915a2b620 2024-04-25T00:24:25.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.877+0000 7f0edd659640 1 --2- 172.21.15.27:0/1203836281 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556915a2b620 0x556915a2da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.877+0000 7f0edd659640 1 --2- 172.21.15.27:0/1203836281 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556915a2b620 0x556915a2da10 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c83b62fac4d429c9 server_cookie=12447ad1d2a54b18 in_seq=0 out_seq=0 2024-04-25T00:24:25.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.877+0000 7f0ec0ff9640 10 client.16923 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:25.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.878+0000 7f0ec0ff9640 1 -- 172.21.15.27:0/1203836281 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556915982bb0 con 0x556915a2b620 2024-04-25T00:24:25.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.878+0000 7f0ec0ff9640 10 client.16923 handle_command_reply: tid=0 2024-04-25T00:24:25.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.878+0000 7f0ee25cf740 1 client.16923 shutdown 2024-04-25T00:24:25.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.878+0000 7f0ee25cf740 1 -- 172.21.15.27:0/1203836281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a1b570 msgr2=0x556915a18830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.878+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/1203836281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a1b570 0x556915a18830 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000fa50 tx=0x7f0ed000f440 comp rx=0 tx=0).stop 2024-04-25T00:24:25.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.879+0000 7f0ee25cf740 1 -- 172.21.15.27:0/1203836281 shutdown_connections 2024-04-25T00:24:25.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.879+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/1203836281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a1b570 0x556915a18830 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.879+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/1203836281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a10ea0 0x556915a18220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.879+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/1203836281 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556915a2b620 0x556915a2da10 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.879+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/1203836281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10580 0x556915a1af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.879+0000 7f0ec0ff9640 0 client.16923 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:25.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.879+0000 7f0ee25cf740 1 -- 172.21.15.27:0/1203836281 >> 172.21.15.27:0/1203836281 conn(0x556915a0c8a0 msgr2=0x556915a126a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.879+0000 7f0ee25cf740 1 -- 172.21.15.27:0/1203836281 shutdown_connections 2024-04-25T00:24:25.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.879+0000 7f0ee25cf740 1 -- 172.21.15.27:0/1203836281 wait complete. 2024-04-25T00:24:25.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.879+0000 7f0ee25cf740 20 client.16923 trim_cache size 0 max 16384 2024-04-25T00:24:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.892+0000 7f0ee25cf740 1 Processor -- start 2024-04-25T00:24:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.892+0000 7f0ee25cf740 1 -- start start 2024-04-25T00:24:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.892+0000 7f0ee25cf740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a10580 0x556915a27220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.892+0000 7f0ee25cf740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10ea0 0x556915a169c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.892+0000 7f0ee25cf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a1b570 0x556915a1aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.892+0000 7f0ee25cf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed81aecc0 con 0x556915a1b570 2024-04-25T00:24:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.892+0000 7f0ee25cf740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed81b0860 con 0x556915a10ea0 2024-04-25T00:24:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.892+0000 7f0ee25cf740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed81af390 con 0x556915a10580 2024-04-25T00:24:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.893+0000 7f0edde5a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a1b570 0x556915a1aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.893+0000 7f0edde5a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a1b570 0x556915a1aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34218/0 (socket says 172.21.15.27:34218) 2024-04-25T00:24:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.893+0000 7f0edde5a640 1 -- 172.21.15.27:0/851694275 learned_addr learned my addr 172.21.15.27:0/851694275 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.893+0000 7f0edce58640 1 --2- 172.21.15.27:0/851694275 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10ea0 0x556915a169c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.893+0000 7f0edd659640 1 --2- 172.21.15.27:0/851694275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a10580 0x556915a27220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.893+0000 7f0edde5a640 1 -- 172.21.15.27:0/851694275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a10580 msgr2=0x556915a27220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.893+0000 7f0edde5a640 1 --2- 172.21.15.27:0/851694275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a10580 0x556915a27220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.893+0000 7f0edde5a640 1 -- 172.21.15.27:0/851694275 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10ea0 msgr2=0x556915a169c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.893+0000 7f0edde5a640 1 --2- 172.21.15.27:0/851694275 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10ea0 0x556915a169c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.893+0000 7f0edde5a640 1 -- 172.21.15.27:0/851694275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569157537d0 con 0x556915a1b570 2024-04-25T00:24:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.893+0000 7f0edde5a640 1 --2- 172.21.15.27:0/851694275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a1b570 0x556915a1aa60 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000f4b0 tx=0x7f0ed00077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a21f605ceb740ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.893+0000 7f0ec2ffd640 1 -- 172.21.15.27:0/851694275 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0016070 con 0x556915a1b570 2024-04-25T00:24:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.893+0000 7f0ec2ffd640 1 -- 172.21.15.27:0/851694275 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ed002a070 con 0x556915a1b570 2024-04-25T00:24:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.893+0000 7f0ec2ffd640 1 -- 172.21.15.27:0/851694275 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed000fda0 con 0x556915a1b570 2024-04-25T00:24:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.894+0000 7f0ee25cf740 1 -- 172.21.15.27:0/851694275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a1b570 msgr2=0x556915a1aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.894+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/851694275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a1b570 0x556915a1aa60 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000f4b0 tx=0x7f0ed00077a0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.894+0000 7f0ee25cf740 1 -- 172.21.15.27:0/851694275 shutdown_connections 2024-04-25T00:24:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.894+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/851694275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a1b570 0x556915a1aa60 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.894+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/851694275 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10ea0 0x556915a169c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.894+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/851694275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a10580 0x556915a27220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.894+0000 7f0ee25cf740 1 -- 172.21.15.27:0/851694275 >> 172.21.15.27:0/851694275 conn(0x556915a0c8a0 msgr2=0x556915a125d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.894+0000 7f0ee25cf740 1 -- 172.21.15.27:0/851694275 shutdown_connections 2024-04-25T00:24:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.894+0000 7f0ee25cf740 1 -- 172.21.15.27:0/851694275 wait complete. 2024-04-25T00:24:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.894+0000 7f0ee25cf740 1 -- start start 2024-04-25T00:24:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.894+0000 7f0ee25cf740 1 Processor -- start 2024-04-25T00:24:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0ee25cf740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10580 0x556915a1e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0ee25cf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a10ea0 0x556915a1eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0ee25cf740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a1b570 0x556915a898d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0ee25cf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed81b0190 con 0x556915a10ea0 2024-04-25T00:24:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0ee25cf740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed81afa90 con 0x556915a10580 2024-04-25T00:24:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0ee25cf740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed81acc50 con 0x556915a1b570 2024-04-25T00:24:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0ee25cf740 10 client.0 _ll_register_callbacks cb 0x7f0ec8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0edce58640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a10ea0 0x556915a1eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0edce58640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a10ea0 0x556915a1eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34224/0 (socket says 172.21.15.27:34224) 2024-04-25T00:24:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0edce58640 1 -- 172.21.15.27:0/1062071803 learned_addr learned my addr 172.21.15.27:0/1062071803 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0edd659640 1 --2- 172.21.15.27:0/1062071803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10580 0x556915a1e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0edde5a640 1 --2- 172.21.15.27:0/1062071803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a1b570 0x556915a898d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0edce58640 1 -- 172.21.15.27:0/1062071803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a1b570 msgr2=0x556915a898d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0edce58640 1 --2- 172.21.15.27:0/1062071803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a1b570 0x556915a898d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0edce58640 1 -- 172.21.15.27:0/1062071803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10580 msgr2=0x556915a1e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0edce58640 1 --2- 172.21.15.27:0/1062071803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10580 0x556915a1e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0edce58640 1 -- 172.21.15.27:0/1062071803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556915a89e90 con 0x556915a10ea0 2024-04-25T00:24:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0edde5a640 1 --2- 172.21.15.27:0/1062071803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a1b570 0x556915a898d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0edd659640 1 --2- 172.21.15.27:0/1062071803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10580 0x556915a1e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0edce58640 1 --2- 172.21.15.27:0/1062071803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a10ea0 0x556915a1eea0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8106a80 tx=0x7f0ec80045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476ecc7d58ac6300 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0ec0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0ec0ff9640 1 -- 172.21.15.27:0/1062071803 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8035040 con 0x556915a10ea0 2024-04-25T00:24:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0ec0ff9640 1 -- 172.21.15.27:0/1062071803 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f0ec8005550 con 0x556915a10ea0 2024-04-25T00:24:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0ee25cf740 1 -- 172.21.15.27:0/1062071803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ed002a070 con 0x556915a10ea0 2024-04-25T00:24:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.895+0000 7f0ec0ff9640 1 -- 172.21.15.27:0/1062071803 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f0ec804f050 con 0x556915a10ea0 2024-04-25T00:24:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.896+0000 7f0ee25cf740 10 client.16929 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.896+0000 7f0ee25cf740 1 -- 172.21.15.27:0/1062071803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x556915a91270 con 0x556915a10ea0 2024-04-25T00:24:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.896+0000 7f0ec0ff9640 1 -- 172.21.15.27:0/1062071803 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ed002a070 con 0x556915a10ea0 2024-04-25T00:24:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.896+0000 7f0ec0ff9640 1 -- 172.21.15.27:0/1062071803 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f0ec8003b30 con 0x556915a10ea0 2024-04-25T00:24:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.896+0000 7f0ee25cf740 10 client.16929 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.896+0000 7f0ee25cf740 10 client.16929 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:24:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.896+0000 7f0ee25cf740 20 client.16929 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.896+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/1062071803 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556915a91a50 0x556915a93e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.896+0000 7f0ee25cf740 4 client.16929 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:24:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.896+0000 7f0ee25cf740 1 -- 172.21.15.27:0/1062071803 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55691597cf00 con 0x556915a91a50 2024-04-25T00:24:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.896+0000 7f0edd659640 1 --2- 172.21.15.27:0/1062071803 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556915a91a50 0x556915a93e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.897+0000 7f0ec0ff9640 10 client.16929 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.897+0000 7f0edd659640 1 --2- 172.21.15.27:0/1062071803 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556915a91a50 0x556915a93e70 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccaf21e8c8fe717 server_cookie=8d89d1e6704510b in_seq=0 out_seq=0 2024-04-25T00:24:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.897+0000 7f0ec0ff9640 1 -- 172.21.15.27:0/1062071803 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55691597cf00 con 0x556915a91a50 2024-04-25T00:24:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.897+0000 7f0ec0ff9640 10 client.16929 handle_command_reply: tid=0 2024-04-25T00:24:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.897+0000 7f0ee25cf740 1 client.16929 shutdown 2024-04-25T00:24:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.897+0000 7f0ee25cf740 1 -- 172.21.15.27:0/1062071803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a10ea0 msgr2=0x556915a1eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.897+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/1062071803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a10ea0 0x556915a1eea0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8106a80 tx=0x7f0ec80045c0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.897+0000 7f0ee25cf740 1 -- 172.21.15.27:0/1062071803 shutdown_connections 2024-04-25T00:24:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.897+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/1062071803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556915a1b570 0x556915a898d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.897+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/1062071803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556915a10ea0 0x556915a1eea0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.897+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/1062071803 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556915a91a50 0x556915a93e70 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.897+0000 7f0ee25cf740 1 --2- 172.21.15.27:0/1062071803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556915a10580 0x556915a1e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.897+0000 7f0ec0ff9640 0 client.16929 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:25.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.897+0000 7f0ee25cf740 1 -- 172.21.15.27:0/1062071803 >> 172.21.15.27:0/1062071803 conn(0x556915a0c8a0 msgr2=0x556915a26210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.897+0000 7f0ee25cf740 1 -- 172.21.15.27:0/1062071803 shutdown_connections 2024-04-25T00:24:25.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.897+0000 7f0ee25cf740 1 -- 172.21.15.27:0/1062071803 wait complete. 2024-04-25T00:24:25.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.897+0000 7f0ee25cf740 20 client.16929 trim_cache size 0 max 16384 2024-04-25T00:24:25.904 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:24:25.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.898+0000 7f0edf8bd640 1 -- 172.21.15.27:0/1353392171 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0ec40a2d00 msgr2=0x7f0ec40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.898+0000 7f0edf8bd640 1 --2- 172.21.15.27:0/1353392171 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0ec40a2d00 0x7f0ec40a51c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0ed001d000 tx=0x7f0ed0020000 comp rx=0 tx=0).stop 2024-04-25T00:24:25.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.898+0000 7f0edf8bd640 1 -- 172.21.15.27:0/1353392171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed8069dc0 msgr2=0x7f0ed8197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:25.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.898+0000 7f0edf8bd640 1 --2- 172.21.15.27:0/1353392171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed8069dc0 0x7f0ed8197fd0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0ed400c890 tx=0x7f0ed400cd50 comp rx=0 tx=0).stop 2024-04-25T00:24:25.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.898+0000 7f0edf8bd640 1 -- 172.21.15.27:0/1353392171 shutdown_connections 2024-04-25T00:24:25.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.898+0000 7f0edf8bd640 1 --2- 172.21.15.27:0/1353392171 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0ec40a2d00 0x7f0ec40a51c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.898+0000 7f0edf8bd640 1 --2- 172.21.15.27:0/1353392171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ed8107730 0x7f0ed819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.898+0000 7f0edf8bd640 1 --2- 172.21.15.27:0/1353392171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ed806a780 0x7f0ed8198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.898+0000 7f0edf8bd640 1 --2- 172.21.15.27:0/1353392171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed8069dc0 0x7f0ed8197fd0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:25.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.898+0000 7f0edf8bd640 1 -- 172.21.15.27:0/1353392171 >> 172.21.15.27:0/1353392171 conn(0x7f0ed80776a0 msgr2=0x7f0ed80fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:25.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.898+0000 7f0edf8bd640 1 -- 172.21.15.27:0/1353392171 shutdown_connections 2024-04-25T00:24:25.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:25.898+0000 7f0edf8bd640 1 -- 172.21.15.27:0/1353392171 wait complete. 2024-04-25T00:24:26.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:25 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4241408131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:24:26.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:25 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4241408131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:24:26.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:25 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4241408131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:24:26.909 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T00:24:26.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:24:26.910 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T00:24:27.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.030+0000 7f6f8347b640 1 -- 172.21.15.27:0/938032040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f7c106630 msgr2=0x7f6f7c106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.030+0000 7f6f8347b640 1 --2- 172.21.15.27:0/938032040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f7c106630 0x7f6f7c106a10 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6f78009930 tx=0x7f6f7802a420 comp rx=0 tx=0).stop 2024-04-25T00:24:27.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.030+0000 7f6f8347b640 1 -- 172.21.15.27:0/938032040 shutdown_connections 2024-04-25T00:24:27.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.030+0000 7f6f8347b640 1 --2- 172.21.15.27:0/938032040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f7c100fe0 0x7f6f7c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.030+0000 7f6f8347b640 1 --2- 172.21.15.27:0/938032040 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f7c100620 0x7f6f7c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.030+0000 7f6f8347b640 1 --2- 172.21.15.27:0/938032040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f7c106630 0x7f6f7c106a10 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.030+0000 7f6f8347b640 1 -- 172.21.15.27:0/938032040 >> 172.21.15.27:0/938032040 conn(0x7f6f7c0776a0 msgr2=0x7f6f7c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.030+0000 7f6f8347b640 1 -- 172.21.15.27:0/938032040 shutdown_connections 2024-04-25T00:24:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.030+0000 7f6f8347b640 1 -- 172.21.15.27:0/938032040 wait complete. 2024-04-25T00:24:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.030+0000 7f6f8347b640 1 Processor -- start 2024-04-25T00:24:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.030+0000 7f6f8347b640 1 -- start start 2024-04-25T00:24:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f8347b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f7c100620 0x7f6f7c198e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f8347b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f7c100fe0 0x7f6f7c199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f8347b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f7c106630 0x7f6f7c1a3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f8347b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7c110a40 con 0x7f6f7c100fe0 2024-04-25T00:24:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f8347b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7c1108c0 con 0x7f6f7c106630 2024-04-25T00:24:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f8347b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7c110bc0 con 0x7f6f7c100620 2024-04-25T00:24:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f80a16640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f7c100fe0 0x7f6f7c199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f80a16640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f7c100fe0 0x7f6f7c199380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35700/0 (socket says 172.21.15.27:35700) 2024-04-25T00:24:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f80a16640 1 -- 172.21.15.27:0/2700269730 learned_addr learned my addr 172.21.15.27:0/2700269730 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f81217640 1 --2- 172.21.15.27:0/2700269730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f7c100620 0x7f6f7c198e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f81a18640 1 --2- 172.21.15.27:0/2700269730 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f7c106630 0x7f6f7c1a3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f80a16640 1 -- 172.21.15.27:0/2700269730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f7c100620 msgr2=0x7f6f7c198e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f80a16640 1 --2- 172.21.15.27:0/2700269730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f7c100620 0x7f6f7c198e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f80a16640 1 -- 172.21.15.27:0/2700269730 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f7c106630 msgr2=0x7f6f7c1a3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f80a16640 1 --2- 172.21.15.27:0/2700269730 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f7c106630 0x7f6f7c1a3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f80a16640 1 -- 172.21.15.27:0/2700269730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f7c1a4400 con 0x7f6f7c100fe0 2024-04-25T00:24:27.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f80a16640 1 --2- 172.21.15.27:0/2700269730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f7c100fe0 0x7f6f7c199380 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00c890 tx=0x7f6f6c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:27.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.031+0000 7f6f727fc640 1 -- 172.21.15.27:0/2700269730 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c004220 con 0x7f6f7c100fe0 2024-04-25T00:24:27.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.032+0000 7f6f8347b640 1 -- 172.21.15.27:0/2700269730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f7c1a45d0 con 0x7f6f7c100fe0 2024-04-25T00:24:27.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.032+0000 7f6f727fc640 1 -- 172.21.15.27:0/2700269730 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f6c0043c0 con 0x7f6f7c100fe0 2024-04-25T00:24:27.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.032+0000 7f6f727fc640 1 -- 172.21.15.27:0/2700269730 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c004a00 con 0x7f6f7c100fe0 2024-04-25T00:24:27.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.032+0000 7f6f8347b640 1 -- 172.21.15.27:0/2700269730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f7c1a4b40 con 0x7f6f7c100fe0 2024-04-25T00:24:27.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.034+0000 7f6f727fc640 1 -- 172.21.15.27:0/2700269730 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6f6c0079b0 con 0x7f6f7c100fe0 2024-04-25T00:24:27.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.034+0000 7f6f86181740 1 Processor -- start 2024-04-25T00:24:27.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.034+0000 7f6f86181740 1 -- start start 2024-04-25T00:24:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.034+0000 7f6f86181740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc6e980 0x55f74fc6ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.034+0000 7f6f86181740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6d290 0x55f74fc6d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f86181740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc78010 0x55f74fc7a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f86181740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f74fbd12c0 con 0x55f74fc6d290 2024-04-25T00:24:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f86181740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f74fb79910 con 0x55f74fc6e980 2024-04-25T00:24:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f86181740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f74fbcd280 con 0x55f74fc78010 2024-04-25T00:24:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f727fc640 1 --2- 172.21.15.27:0/2700269730 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6f680a2d00 0x7f6f680a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f81217640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6d290 0x55f74fc6d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f81217640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6d290 0x55f74fc6d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35708/0 (socket says 172.21.15.27:35708) 2024-04-25T00:24:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f81217640 1 -- 172.21.15.27:0/2154834214 learned_addr learned my addr 172.21.15.27:0/2154834214 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f81a18640 1 --2- 172.21.15.27:0/2700269730 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6f680a2d00 0x7f6f680a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f80a16640 1 --2- 172.21.15.27:0/2154834214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc78010 0x55f74fc7a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f81a18640 1 --2- 172.21.15.27:0/2154834214 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc6e980 0x55f74fc6ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f727fc640 1 -- 172.21.15.27:0/2700269730 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f6f6c0cf4c0 con 0x7f6f7c100fe0 2024-04-25T00:24:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f81217640 1 -- 172.21.15.27:0/2154834214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc78010 msgr2=0x55f74fc7a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f81217640 1 --2- 172.21.15.27:0/2154834214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc78010 0x55f74fc7a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f81217640 1 -- 172.21.15.27:0/2154834214 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc6e980 msgr2=0x55f74fc6ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f81217640 1 --2- 172.21.15.27:0/2154834214 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc6e980 0x55f74fc6ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f81217640 1 -- 172.21.15.27:0/2154834214 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f74fb8afb0 con 0x55f74fc6d290 2024-04-25T00:24:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f81a18640 1 --2- 172.21.15.27:0/2700269730 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6f680a2d00 0x7f6f680a51c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6f74005c40 tx=0x7f6f74009c90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.035+0000 7f6f81217640 1 --2- 172.21.15.27:0/2154834214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6d290 0x55f74fc6d710 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6f78009930 tx=0x7f6f78031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44f932f0f51b5a03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.036+0000 7f6f677fe640 1 -- 172.21.15.27:0/2154834214 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f78040070 con 0x55f74fc6d290 2024-04-25T00:24:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.036+0000 7f6f677fe640 1 -- 172.21.15.27:0/2154834214 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f78005020 con 0x55f74fc6d290 2024-04-25T00:24:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.036+0000 7f6f677fe640 1 -- 172.21.15.27:0/2154834214 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f78035040 con 0x55f74fc6d290 2024-04-25T00:24:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.036+0000 7f6f86181740 1 -- 172.21.15.27:0/2154834214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6d290 msgr2=0x55f74fc6d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.036+0000 7f6f86181740 1 --2- 172.21.15.27:0/2154834214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6d290 0x55f74fc6d710 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6f78009930 tx=0x7f6f78031630 comp rx=0 tx=0).stop 2024-04-25T00:24:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.036+0000 7f6f86181740 1 -- 172.21.15.27:0/2154834214 shutdown_connections 2024-04-25T00:24:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.036+0000 7f6f86181740 1 --2- 172.21.15.27:0/2154834214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc78010 0x55f74fc7a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.036+0000 7f6f86181740 1 --2- 172.21.15.27:0/2154834214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6d290 0x55f74fc6d710 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.036+0000 7f6f86181740 1 --2- 172.21.15.27:0/2154834214 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc6e980 0x55f74fc6ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.036+0000 7f6f86181740 1 -- 172.21.15.27:0/2154834214 >> 172.21.15.27:0/2154834214 conn(0x55f74fc35950 msgr2=0x55f74fc697f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:27.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.036+0000 7f6f86181740 1 -- 172.21.15.27:0/2154834214 shutdown_connections 2024-04-25T00:24:27.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.036+0000 7f6f86181740 1 -- 172.21.15.27:0/2154834214 wait complete. 2024-04-25T00:24:27.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.037+0000 7f6f86181740 1 -- start start 2024-04-25T00:24:27.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.037+0000 7f6f86181740 1 Processor -- start 2024-04-25T00:24:27.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.037+0000 7f6f86181740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6d290 0x55f74fc74de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.037+0000 7f6f86181740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc6e980 0x55f74fc75320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.037+0000 7f6f86181740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc78010 0x55f74fc75860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.037+0000 7f6f86181740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f74fc7ae40 con 0x55f74fc6d290 2024-04-25T00:24:27.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.037+0000 7f6f86181740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f74fb8ce90 con 0x55f74fc6e980 2024-04-25T00:24:27.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.037+0000 7f6f86181740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f74fc7afc0 con 0x55f74fc78010 2024-04-25T00:24:27.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.037+0000 7f6f86181740 10 client.0 _ll_register_callbacks cb 0x55f74fbc8970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:27.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.037+0000 7f6f81217640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6d290 0x55f74fc74de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.037+0000 7f6f81217640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6d290 0x55f74fc74de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35716/0 (socket says 172.21.15.27:35716) 2024-04-25T00:24:27.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.037+0000 7f6f81217640 1 -- 172.21.15.27:0/2747780958 learned_addr learned my addr 172.21.15.27:0/2747780958 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:27.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.038+0000 7f6f80a16640 1 --2- 172.21.15.27:0/2747780958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc6e980 0x55f74fc75320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.038+0000 7f6f81a18640 1 --2- 172.21.15.27:0/2747780958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc78010 0x55f74fc75860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.038+0000 7f6f81217640 1 -- 172.21.15.27:0/2747780958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc78010 msgr2=0x55f74fc75860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:27.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.038+0000 7f6f81217640 1 --2- 172.21.15.27:0/2747780958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc78010 0x55f74fc75860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.038+0000 7f6f81217640 1 -- 172.21.15.27:0/2747780958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc6e980 msgr2=0x55f74fc75320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:27.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.038+0000 7f6f81217640 1 --2- 172.21.15.27:0/2747780958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc6e980 0x55f74fc75320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.038+0000 7f6f81217640 1 -- 172.21.15.27:0/2747780958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f74f9ad7e0 con 0x55f74fc6d290 2024-04-25T00:24:27.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.038+0000 7f6f81a18640 1 --2- 172.21.15.27:0/2747780958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc78010 0x55f74fc75860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:27.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.038+0000 7f6f81217640 1 --2- 172.21.15.27:0/2747780958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6d290 0x55f74fc74de0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6f78006010 tx=0x7f6f78009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=963cf1889d39109b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:27.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.038+0000 7f6f64ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:27.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.038+0000 7f6f64ff9640 1 -- 172.21.15.27:0/2747780958 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7804c020 con 0x55f74fc6d290 2024-04-25T00:24:27.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.038+0000 7f6f64ff9640 1 -- 172.21.15.27:0/2747780958 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f6f78004970 con 0x55f74fc6d290 2024-04-25T00:24:27.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.038+0000 7f6f86181740 1 -- 172.21.15.27:0/2747780958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f78005020 con 0x55f74fc6d290 2024-04-25T00:24:27.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.038+0000 7f6f64ff9640 1 -- 172.21.15.27:0/2747780958 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f6f78057050 con 0x55f74fc6d290 2024-04-25T00:24:27.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.038+0000 7f6f86181740 10 client.16941 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:27.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.038+0000 7f6f86181740 1 -- 172.21.15.27:0/2747780958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55f74fa0df70 con 0x55f74fc6d290 2024-04-25T00:24:27.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.039+0000 7f6f64ff9640 1 -- 172.21.15.27:0/2747780958 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f78005020 con 0x55f74fc6d290 2024-04-25T00:24:27.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.039+0000 7f6f64ff9640 1 -- 172.21.15.27:0/2747780958 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f6f7803cc50 con 0x55f74fc6d290 2024-04-25T00:24:27.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.039+0000 7f6f86181740 10 client.16941 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:27.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.039+0000 7f6f86181740 10 client.16941 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:24:27.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.039+0000 7f6f86181740 20 client.16941 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:27.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.039+0000 7f6f86181740 1 --2- 172.21.15.27:0/2747780958 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f74fc87de0 0x55f74fc8a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.039+0000 7f6f86181740 4 client.16941 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:24:27.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.039+0000 7f6f86181740 1 -- 172.21.15.27:0/2747780958 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f74fba55a0 con 0x55f74fc87de0 2024-04-25T00:24:27.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.039+0000 7f6f80a16640 1 --2- 172.21.15.27:0/2747780958 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f74fc87de0 0x55f74fc8a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.040+0000 7f6f80a16640 1 --2- 172.21.15.27:0/2747780958 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f74fc87de0 0x55f74fc8a1d0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22ea5de805d951 server_cookie=c1062abaf3a91fe4 in_seq=0 out_seq=0 2024-04-25T00:24:27.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.040+0000 7f6f64ff9640 10 client.16941 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:27.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.040+0000 7f6f64ff9640 1 -- 172.21.15.27:0/2747780958 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f74fba55a0 con 0x55f74fc87de0 2024-04-25T00:24:27.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.040+0000 7f6f64ff9640 10 client.16941 handle_command_reply: tid=0 2024-04-25T00:24:27.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.040+0000 7f6f86181740 1 client.16941 shutdown 2024-04-25T00:24:27.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.041+0000 7f6f86181740 1 -- 172.21.15.27:0/2747780958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6d290 msgr2=0x55f74fc74de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.041+0000 7f6f86181740 1 --2- 172.21.15.27:0/2747780958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6d290 0x55f74fc74de0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6f78006010 tx=0x7f6f78009640 comp rx=0 tx=0).stop 2024-04-25T00:24:27.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.041+0000 7f6f86181740 1 -- 172.21.15.27:0/2747780958 shutdown_connections 2024-04-25T00:24:27.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.041+0000 7f6f86181740 1 --2- 172.21.15.27:0/2747780958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc78010 0x55f74fc75860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.041+0000 7f6f86181740 1 --2- 172.21.15.27:0/2747780958 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f74fc87de0 0x55f74fc8a1d0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.041+0000 7f6f86181740 1 --2- 172.21.15.27:0/2747780958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc6e980 0x55f74fc75320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.041+0000 7f6f86181740 1 --2- 172.21.15.27:0/2747780958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6d290 0x55f74fc74de0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.041+0000 7f6f64ff9640 0 client.16941 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:27.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.041+0000 7f6f86181740 1 -- 172.21.15.27:0/2747780958 >> 172.21.15.27:0/2747780958 conn(0x55f74fc35950 msgr2=0x55f74fc6f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:27.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.041+0000 7f6f86181740 1 -- 172.21.15.27:0/2747780958 shutdown_connections 2024-04-25T00:24:27.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.041+0000 7f6f86181740 1 -- 172.21.15.27:0/2747780958 wait complete. 2024-04-25T00:24:27.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.041+0000 7f6f86181740 20 client.16941 trim_cache size 0 max 16384 2024-04-25T00:24:27.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f86181740 1 Processor -- start 2024-04-25T00:24:27.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f86181740 1 -- start start 2024-04-25T00:24:27.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f86181740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc6d290 0x55f74fc83d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f86181740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc6e980 0x55f74fc74360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f86181740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc78010 0x55f74fc8f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f86181740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7c1aec20 con 0x55f74fc78010 2024-04-25T00:24:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f86181740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7c1b07c0 con 0x55f74fc6d290 2024-04-25T00:24:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f86181740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7c1af2f0 con 0x55f74fc6e980 2024-04-25T00:24:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f81a18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc78010 0x55f74fc8f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f81a18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc78010 0x55f74fc8f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35728/0 (socket says 172.21.15.27:35728) 2024-04-25T00:24:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f81a18640 1 -- 172.21.15.27:0/1100569387 learned_addr learned my addr 172.21.15.27:0/1100569387 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f81217640 1 --2- 172.21.15.27:0/1100569387 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc6d290 0x55f74fc83d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f80a16640 1 --2- 172.21.15.27:0/1100569387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc6e980 0x55f74fc74360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f81a18640 1 -- 172.21.15.27:0/1100569387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc6e980 msgr2=0x55f74fc74360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f81a18640 1 --2- 172.21.15.27:0/1100569387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc6e980 0x55f74fc74360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f81a18640 1 -- 172.21.15.27:0/1100569387 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc6d290 msgr2=0x55f74fc83d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f81a18640 1 --2- 172.21.15.27:0/1100569387 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc6d290 0x55f74fc83d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f81a18640 1 -- 172.21.15.27:0/1100569387 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f74f8a5170 con 0x55f74fc78010 2024-04-25T00:24:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f81217640 1 --2- 172.21.15.27:0/1100569387 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc6d290 0x55f74fc83d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.059+0000 7f6f81a18640 1 --2- 172.21.15.27:0/1100569387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc78010 0x55f74fc8f800 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6f74009790 tx=0x7f6f74004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b34b7a8b739a3f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.060+0000 7f6f66ffd640 1 -- 172.21.15.27:0/1100569387 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7401e970 con 0x55f74fc78010 2024-04-25T00:24:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.060+0000 7f6f66ffd640 1 -- 172.21.15.27:0/1100569387 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f74020400 con 0x55f74fc78010 2024-04-25T00:24:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.060+0000 7f6f66ffd640 1 -- 172.21.15.27:0/1100569387 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f74011640 con 0x55f74fc78010 2024-04-25T00:24:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.060+0000 7f6f86181740 1 -- 172.21.15.27:0/1100569387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc78010 msgr2=0x55f74fc8f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.060+0000 7f6f86181740 1 --2- 172.21.15.27:0/1100569387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc78010 0x55f74fc8f800 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6f74009790 tx=0x7f6f74004ab0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.060+0000 7f6f86181740 1 -- 172.21.15.27:0/1100569387 shutdown_connections 2024-04-25T00:24:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.060+0000 7f6f86181740 1 --2- 172.21.15.27:0/1100569387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc78010 0x55f74fc8f800 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.060+0000 7f6f86181740 1 --2- 172.21.15.27:0/1100569387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc6e980 0x55f74fc74360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.060+0000 7f6f86181740 1 --2- 172.21.15.27:0/1100569387 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc6d290 0x55f74fc83d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.060+0000 7f6f86181740 1 -- 172.21.15.27:0/1100569387 >> 172.21.15.27:0/1100569387 conn(0x55f74fc35950 msgr2=0x55f74fc76ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:27.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.060+0000 7f6f86181740 1 -- 172.21.15.27:0/1100569387 shutdown_connections 2024-04-25T00:24:27.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.060+0000 7f6f86181740 1 -- 172.21.15.27:0/1100569387 wait complete. 2024-04-25T00:24:27.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.061+0000 7f6f86181740 1 -- start start 2024-04-25T00:24:27.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.061+0000 7f6f86181740 1 Processor -- start 2024-04-25T00:24:27.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.061+0000 7f6f86181740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc6d290 0x55f74fc71a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.061+0000 7f6f86181740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6e980 0x55f74fc71fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.061+0000 7f6f86181740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc78010 0x55f74fc72510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.061+0000 7f6f86181740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7c1b00f0 con 0x55f74fc6e980 2024-04-25T00:24:27.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.061+0000 7f6f86181740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7c1af9f0 con 0x55f74fc78010 2024-04-25T00:24:27.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.061+0000 7f6f86181740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7c1acbb0 con 0x55f74fc6d290 2024-04-25T00:24:27.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.061+0000 7f6f86181740 10 client.0 _ll_register_callbacks cb 0x7f6f78005de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:27.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.061+0000 7f6f80a16640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6e980 0x55f74fc71fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.061+0000 7f6f80a16640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6e980 0x55f74fc71fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35740/0 (socket says 172.21.15.27:35740) 2024-04-25T00:24:27.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.061+0000 7f6f80a16640 1 -- 172.21.15.27:0/1516278060 learned_addr learned my addr 172.21.15.27:0/1516278060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:27.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.061+0000 7f6f81217640 1 --2- 172.21.15.27:0/1516278060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc6d290 0x55f74fc71a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.061+0000 7f6f81a18640 1 --2- 172.21.15.27:0/1516278060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc78010 0x55f74fc72510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f80a16640 1 -- 172.21.15.27:0/1516278060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc6d290 msgr2=0x55f74fc71a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:27.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f80a16640 1 --2- 172.21.15.27:0/1516278060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc6d290 0x55f74fc71a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f80a16640 1 -- 172.21.15.27:0/1516278060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc78010 msgr2=0x55f74fc72510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:27.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f80a16640 1 --2- 172.21.15.27:0/1516278060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc78010 0x55f74fc72510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f80a16640 1 -- 172.21.15.27:0/1516278060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f74fc7e400 con 0x55f74fc6e980 2024-04-25T00:24:27.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f80a16640 1 --2- 172.21.15.27:0/1516278060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6e980 0x55f74fc71fd0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c005450 tx=0x7f6f6c008ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d16ca8e123fe55f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:27.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f64ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:27.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f64ff9640 1 -- 172.21.15.27:0/1516278060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c0b3020 con 0x55f74fc6e980 2024-04-25T00:24:27.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f64ff9640 1 -- 172.21.15.27:0/1516278060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f6f6c097ce0 con 0x55f74fc6e980 2024-04-25T00:24:27.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f86181740 1 -- 172.21.15.27:0/1516278060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f74020400 con 0x55f74fc6e980 2024-04-25T00:24:27.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f64ff9640 1 -- 172.21.15.27:0/1516278060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f6f6c09bd70 con 0x55f74fc6e980 2024-04-25T00:24:27.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f86181740 10 client.16947 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f86181740 1 -- 172.21.15.27:0/1516278060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55f74fcee3b0 con 0x55f74fc6e980 2024-04-25T00:24:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f64ff9640 1 -- 172.21.15.27:0/1516278060 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f74020400 con 0x55f74fc6e980 2024-04-25T00:24:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f64ff9640 1 -- 172.21.15.27:0/1516278060 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f6f6c007ca0 con 0x55f74fc6e980 2024-04-25T00:24:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f86181740 10 client.16947 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f86181740 10 client.16947 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:24:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f86181740 20 client.16947 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f86181740 1 --2- 172.21.15.27:0/1516278060 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f74fceeae0 0x55f74fcf0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f86181740 4 client.16947 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T00:24:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.062+0000 7f6f86181740 1 -- 172.21.15.27:0/1516278060 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f74fbde800 con 0x55f74fceeae0 2024-04-25T00:24:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.063+0000 7f6f81217640 1 --2- 172.21.15.27:0/1516278060 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f74fceeae0 0x55f74fcf0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.063+0000 7f6f64ff9640 10 client.16947 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.063+0000 7f6f81217640 1 --2- 172.21.15.27:0/1516278060 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f74fceeae0 0x55f74fcf0f00 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=752f5bf21bbf2a66 server_cookie=f3b101d5851e9451 in_seq=0 out_seq=0 2024-04-25T00:24:27.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.064+0000 7f6f64ff9640 1 -- 172.21.15.27:0/1516278060 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f74fbde800 con 0x55f74fceeae0 2024-04-25T00:24:27.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.064+0000 7f6f64ff9640 10 client.16947 handle_command_reply: tid=0 2024-04-25T00:24:27.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.064+0000 7f6f86181740 1 client.16947 shutdown 2024-04-25T00:24:27.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.064+0000 7f6f86181740 1 -- 172.21.15.27:0/1516278060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6e980 msgr2=0x55f74fc71fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.064+0000 7f6f86181740 1 --2- 172.21.15.27:0/1516278060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6e980 0x55f74fc71fd0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c005450 tx=0x7f6f6c008ed0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f86181740 1 -- 172.21.15.27:0/1516278060 shutdown_connections 2024-04-25T00:24:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f86181740 1 --2- 172.21.15.27:0/1516278060 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f74fceeae0 0x55f74fcf0f00 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f86181740 1 --2- 172.21.15.27:0/1516278060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f74fc78010 0x55f74fc72510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f86181740 1 --2- 172.21.15.27:0/1516278060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f74fc6e980 0x55f74fc71fd0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f86181740 1 --2- 172.21.15.27:0/1516278060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f74fc6d290 0x55f74fc71a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f86181740 1 -- 172.21.15.27:0/1516278060 >> 172.21.15.27:0/1516278060 conn(0x55f74fc35950 msgr2=0x55f74fc75360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f64ff9640 0 client.16947 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:27.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f86181740 1 -- 172.21.15.27:0/1516278060 shutdown_connections 2024-04-25T00:24:27.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f86181740 1 -- 172.21.15.27:0/1516278060 wait complete. 2024-04-25T00:24:27.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f86181740 20 client.16947 trim_cache size 0 max 16384 2024-04-25T00:24:27.072 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:24:27.072 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T00:24:27.072 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "65fb331c-8c72-48c1-af8b-f63d28fbdfb2", 2024-04-25T00:24:27.072 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T00:24:27.072 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:24:27.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f8347b640 1 -- 172.21.15.27:0/2700269730 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6f680a2d00 msgr2=0x7f6f680a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f8347b640 1 --2- 172.21.15.27:0/2700269730 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6f680a2d00 0x7f6f680a51c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6f74005c40 tx=0x7f6f74009c90 comp rx=0 tx=0).stop 2024-04-25T00:24:27.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f8347b640 1 -- 172.21.15.27:0/2700269730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f7c100fe0 msgr2=0x7f6f7c199380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f8347b640 1 --2- 172.21.15.27:0/2700269730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f7c100fe0 0x7f6f7c199380 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00c890 tx=0x7f6f6c00cd50 comp rx=0 tx=0).stop 2024-04-25T00:24:27.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f8347b640 1 -- 172.21.15.27:0/2700269730 shutdown_connections 2024-04-25T00:24:27.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f8347b640 1 --2- 172.21.15.27:0/2700269730 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6f680a2d00 0x7f6f680a51c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f8347b640 1 --2- 172.21.15.27:0/2700269730 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f7c106630 0x7f6f7c1a3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f8347b640 1 --2- 172.21.15.27:0/2700269730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f7c100fe0 0x7f6f7c199380 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f8347b640 1 --2- 172.21.15.27:0/2700269730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f7c100620 0x7f6f7c198e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.065+0000 7f6f8347b640 1 -- 172.21.15.27:0/2700269730 >> 172.21.15.27:0/2700269730 conn(0x7f6f7c0776a0 msgr2=0x7f6f7c103150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:27.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.066+0000 7f6f8347b640 1 -- 172.21.15.27:0/2700269730 shutdown_connections 2024-04-25T00:24:27.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.066+0000 7f6f8347b640 1 -- 172.21.15.27:0/2700269730 wait complete. 2024-04-25T00:24:27.075 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:26 smithi027 ceph-mon[29208]: pgmap v263: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:24:27.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:24:27.077 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:24:27.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.181+0000 7f557e25d640 1 -- 172.21.15.27:0/2743766792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5578101710 msgr2=0x7f5578101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.181+0000 7f557e25d640 1 --2- 172.21.15.27:0/2743766792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5578101710 0x7f5578101b90 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f556c009930 tx=0x7f556c02a420 comp rx=0 tx=0).stop 2024-04-25T00:24:27.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.181+0000 7f557e25d640 1 -- 172.21.15.27:0/2743766792 shutdown_connections 2024-04-25T00:24:27.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.181+0000 7f557e25d640 1 --2- 172.21.15.27:0/2743766792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55781020d0 0x7f557810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.181+0000 7f557e25d640 1 --2- 172.21.15.27:0/2743766792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5578101710 0x7f5578101b90 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.181+0000 7f557e25d640 1 --2- 172.21.15.27:0/2743766792 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5578107720 0x7f5578107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.181+0000 7f557e25d640 1 -- 172.21.15.27:0/2743766792 >> 172.21.15.27:0/2743766792 conn(0x7f5578078b90 msgr2=0x7f5578078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:27.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.181+0000 7f557e25d640 1 -- 172.21.15.27:0/2743766792 shutdown_connections 2024-04-25T00:24:27.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.181+0000 7f557e25d640 1 -- 172.21.15.27:0/2743766792 wait complete. 2024-04-25T00:24:27.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.181+0000 7f557e25d640 1 Processor -- start 2024-04-25T00:24:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f557e25d640 1 -- start start 2024-04-25T00:24:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f557e25d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5578101710 0x7f5578198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f557e25d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55781020d0 0x7f5578198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f557e25d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5578107720 0x7f557819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f557e25d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5578111b30 con 0x7f55781020d0 2024-04-25T00:24:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f557e25d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f55781119b0 con 0x7f5578107720 2024-04-25T00:24:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f557e25d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5578111cb0 con 0x7f5578101710 2024-04-25T00:24:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f5576ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55781020d0 0x7f5578198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f5576ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55781020d0 0x7f5578198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35762/0 (socket says 172.21.15.27:35762) 2024-04-25T00:24:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f5576ffd640 1 -- 172.21.15.27:0/2928224494 learned_addr learned my addr 172.21.15.27:0/2928224494 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f55777fe640 1 --2- 172.21.15.27:0/2928224494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5578101710 0x7f5578198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f5577fff640 1 --2- 172.21.15.27:0/2928224494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5578107720 0x7f557819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f5576ffd640 1 -- 172.21.15.27:0/2928224494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5578101710 msgr2=0x7f5578198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f5576ffd640 1 --2- 172.21.15.27:0/2928224494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5578101710 0x7f5578198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f5576ffd640 1 -- 172.21.15.27:0/2928224494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5578107720 msgr2=0x7f557819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f5576ffd640 1 --2- 172.21.15.27:0/2928224494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5578107720 0x7f557819c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f5576ffd640 1 -- 172.21.15.27:0/2928224494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557819d020 con 0x7f55781020d0 2024-04-25T00:24:27.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f55777fe640 1 --2- 172.21.15.27:0/2928224494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5578101710 0x7f5578198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:27.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f5576ffd640 1 --2- 172.21.15.27:0/2928224494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55781020d0 0x7f5578198570 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f556c009230 tx=0x7f556c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:27.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f5574ff9640 1 -- 172.21.15.27:0/2928224494 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c03f030 con 0x7f55781020d0 2024-04-25T00:24:27.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.182+0000 7f5574ff9640 1 -- 172.21.15.27:0/2928224494 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f556c003ad0 con 0x7f55781020d0 2024-04-25T00:24:27.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.183+0000 7f5574ff9640 1 -- 172.21.15.27:0/2928224494 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c035070 con 0x7f55781020d0 2024-04-25T00:24:27.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.183+0000 7f557e25d640 1 -- 172.21.15.27:0/2928224494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f557819d2b0 con 0x7f55781020d0 2024-04-25T00:24:27.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.183+0000 7f557e25d640 1 -- 172.21.15.27:0/2928224494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55781a4ba0 con 0x7f55781020d0 2024-04-25T00:24:27.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.185+0000 7f5574ff9640 1 -- 172.21.15.27:0/2928224494 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f556c042030 con 0x7f55781020d0 2024-04-25T00:24:27.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.185+0000 7f5580f6b740 1 Processor -- start 2024-04-25T00:24:27.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f5574ff9640 1 --2- 172.21.15.27:0/2928224494 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f55600a2d80 0x7f55600a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f5580f6b740 1 -- start start 2024-04-25T00:24:27.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f5577fff640 1 --2- 172.21.15.27:0/2928224494 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f55600a2d80 0x7f55600a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f5580f6b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650630 0x55f8e8650a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f5574ff9640 1 -- 172.21.15.27:0/2928224494 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f556c0ef980 con 0x7f55781020d0 2024-04-25T00:24:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f5580f6b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e8650f50 0x55f8e865b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f5580f6b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8e865b620 0x55f8e865da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f5580f6b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f8e85b56b0 con 0x55f8e8650630 2024-04-25T00:24:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f5580f6b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f8e855d880 con 0x55f8e8650f50 2024-04-25T00:24:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f5580f6b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f8e85b1670 con 0x55f8e865b620 2024-04-25T00:24:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f55777fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650630 0x55f8e8650a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f55777fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650630 0x55f8e8650a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35766/0 (socket says 172.21.15.27:35766) 2024-04-25T00:24:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f55777fe640 1 -- 172.21.15.27:0/3272830233 learned_addr learned my addr 172.21.15.27:0/3272830233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f5576ffd640 1 --2- 172.21.15.27:0/3272830233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e8650f50 0x55f8e865b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f5577fff640 1 --2- 172.21.15.27:0/2928224494 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f55600a2d80 0x7f55600a5240 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5570008cf0 tx=0x7f5570008230 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:27.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f55777fe640 1 -- 172.21.15.27:0/3272830233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8e865b620 msgr2=0x55f8e865da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:27.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f55777fe640 1 --2- 172.21.15.27:0/3272830233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8e865b620 0x55f8e865da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f55777fe640 1 -- 172.21.15.27:0/3272830233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e8650f50 msgr2=0x55f8e865b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:27.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f55777fe640 1 --2- 172.21.15.27:0/3272830233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e8650f50 0x55f8e865b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f55777fe640 1 -- 172.21.15.27:0/3272830233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8e83937d0 con 0x55f8e8650630 2024-04-25T00:24:27.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.186+0000 7f55777fe640 1 --2- 172.21.15.27:0/3272830233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650630 0x55f8e8650a10 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f556800bcc0 tx=0x7f556800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea886c459b159a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:27.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.187+0000 7f555dffb640 1 -- 172.21.15.27:0/3272830233 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5568015070 con 0x55f8e8650630 2024-04-25T00:24:27.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.187+0000 7f555dffb640 1 -- 172.21.15.27:0/3272830233 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f556801a940 con 0x55f8e8650630 2024-04-25T00:24:27.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.187+0000 7f555dffb640 1 -- 172.21.15.27:0/3272830233 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5568011a30 con 0x55f8e8650630 2024-04-25T00:24:27.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.187+0000 7f5580f6b740 1 -- 172.21.15.27:0/3272830233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650630 msgr2=0x55f8e8650a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.187+0000 7f5580f6b740 1 --2- 172.21.15.27:0/3272830233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650630 0x55f8e8650a10 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f556800bcc0 tx=0x7f556800f730 comp rx=0 tx=0).stop 2024-04-25T00:24:27.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.187+0000 7f5580f6b740 1 -- 172.21.15.27:0/3272830233 shutdown_connections 2024-04-25T00:24:27.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.187+0000 7f5580f6b740 1 --2- 172.21.15.27:0/3272830233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8e865b620 0x55f8e865da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.187+0000 7f5580f6b740 1 --2- 172.21.15.27:0/3272830233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e8650f50 0x55f8e865b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.187+0000 7f5580f6b740 1 --2- 172.21.15.27:0/3272830233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650630 0x55f8e8650a10 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.187+0000 7f5580f6b740 1 -- 172.21.15.27:0/3272830233 >> 172.21.15.27:0/3272830233 conn(0x55f8e864c950 msgr2=0x55f8e864cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:27.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.187+0000 7f5580f6b740 1 -- 172.21.15.27:0/3272830233 shutdown_connections 2024-04-25T00:24:27.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.187+0000 7f5580f6b740 1 -- 172.21.15.27:0/3272830233 wait complete. 2024-04-25T00:24:27.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.188+0000 7f5580f6b740 1 -- start start 2024-04-25T00:24:27.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.188+0000 7f5580f6b740 1 Processor -- start 2024-04-25T00:24:27.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.188+0000 7f5580f6b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8e8650630 0x55f8e8658570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.188+0000 7f5580f6b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e8658ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.188+0000 7f5580f6b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e865b620 0x55f8e8658ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.188+0000 7f5580f6b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f8e865e410 con 0x55f8e8650f50 2024-04-25T00:24:27.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.188+0000 7f5580f6b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f8e8570f50 con 0x55f8e865b620 2024-04-25T00:24:27.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.188+0000 7f5580f6b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f8e865e590 con 0x55f8e8650630 2024-04-25T00:24:27.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.188+0000 7f5580f6b740 10 client.0 _ll_register_callbacks cb 0x55f8e85ac000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:27.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.188+0000 7f5576ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e8658ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.188+0000 7f5576ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e8658ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35774/0 (socket says 172.21.15.27:35774) 2024-04-25T00:24:27.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.188+0000 7f5576ffd640 1 -- 172.21.15.27:0/3068313267 learned_addr learned my addr 172.21.15.27:0/3068313267 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:27.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.188+0000 7f55777fe640 1 --2- 172.21.15.27:0/3068313267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8e8650630 0x55f8e8658570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.188+0000 7f5577fff640 1 --2- 172.21.15.27:0/3068313267 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e865b620 0x55f8e8658ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.188+0000 7f5576ffd640 1 -- 172.21.15.27:0/3068313267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8e8650630 msgr2=0x55f8e8658570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:27.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.188+0000 7f5576ffd640 1 --2- 172.21.15.27:0/3068313267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8e8650630 0x55f8e8658570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5576ffd640 1 -- 172.21.15.27:0/3068313267 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e865b620 msgr2=0x55f8e8658ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:27.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5576ffd640 1 --2- 172.21.15.27:0/3068313267 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e865b620 0x55f8e8658ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5576ffd640 1 -- 172.21.15.27:0/3068313267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8e82c93f0 con 0x55f8e8650f50 2024-04-25T00:24:27.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5577fff640 1 --2- 172.21.15.27:0/3068313267 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e865b620 0x55f8e8658ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:27.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5576ffd640 1 --2- 172.21.15.27:0/3068313267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e8658ab0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f556c0bbac0 tx=0x7f556c0bc370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=983a6284e14cf123 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:27.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5556ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:27.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5556ffd640 1 -- 172.21.15.27:0/3068313267 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c0bce60 con 0x55f8e8650f50 2024-04-25T00:24:27.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5556ffd640 1 -- 172.21.15.27:0/3068313267 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f556c003c80 con 0x55f8e8650f50 2024-04-25T00:24:27.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5580f6b740 1 -- 172.21.15.27:0/3068313267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f556801a940 con 0x55f8e8650f50 2024-04-25T00:24:27.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5556ffd640 1 -- 172.21.15.27:0/3068313267 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f556c049050 con 0x55f8e8650f50 2024-04-25T00:24:27.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5580f6b740 10 client.16959 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:27.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5580f6b740 1 -- 172.21.15.27:0/3068313267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55f8e82890f0 con 0x55f8e8650f50 2024-04-25T00:24:27.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5556ffd640 1 -- 172.21.15.27:0/3068313267 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f556801a940 con 0x55f8e8650f50 2024-04-25T00:24:27.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5556ffd640 1 -- 172.21.15.27:0/3068313267 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f556c04c060 con 0x55f8e8650f50 2024-04-25T00:24:27.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5580f6b740 10 client.16959 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:27.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5580f6b740 10 client.16959 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:24:27.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5580f6b740 20 client.16959 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:27.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5580f6b740 1 --2- 172.21.15.27:0/3068313267 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f8e866b920 0x55f8e866dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5580f6b740 4 client.16959 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:24:27.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.189+0000 7f5580f6b740 1 -- 172.21.15.27:0/3068313267 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8e85c2bf0 con 0x55f8e866b920 2024-04-25T00:24:27.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.190+0000 7f55777fe640 1 --2- 172.21.15.27:0/3068313267 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f8e866b920 0x55f8e866dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.190+0000 7f5556ffd640 10 client.16959 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:27.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.190+0000 7f55777fe640 1 --2- 172.21.15.27:0/3068313267 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f8e866b920 0x55f8e866dd10 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=723bcb229e3d1ce4 server_cookie=a9f9282721944bd0 in_seq=0 out_seq=0 2024-04-25T00:24:27.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.191+0000 7f5556ffd640 1 -- 172.21.15.27:0/3068313267 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f8e85c2bf0 con 0x55f8e866b920 2024-04-25T00:24:27.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.191+0000 7f5556ffd640 10 client.16959 handle_command_reply: tid=0 2024-04-25T00:24:27.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.191+0000 7f5580f6b740 1 client.16959 shutdown 2024-04-25T00:24:27.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.191+0000 7f5580f6b740 1 -- 172.21.15.27:0/3068313267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 msgr2=0x55f8e8658ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.191+0000 7f5580f6b740 1 --2- 172.21.15.27:0/3068313267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e8658ab0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f556c0bbac0 tx=0x7f556c0bc370 comp rx=0 tx=0).stop 2024-04-25T00:24:27.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.191+0000 7f5580f6b740 1 -- 172.21.15.27:0/3068313267 shutdown_connections 2024-04-25T00:24:27.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.191+0000 7f5580f6b740 1 --2- 172.21.15.27:0/3068313267 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f8e866b920 0x55f8e866dd10 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.191+0000 7f5580f6b740 1 --2- 172.21.15.27:0/3068313267 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e865b620 0x55f8e8658ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.191+0000 7f5556ffd640 0 client.16959 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:27.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.191+0000 7f5580f6b740 1 --2- 172.21.15.27:0/3068313267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e8658ab0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.191+0000 7f5580f6b740 1 --2- 172.21.15.27:0/3068313267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8e8650630 0x55f8e8658570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.191+0000 7f5580f6b740 1 -- 172.21.15.27:0/3068313267 >> 172.21.15.27:0/3068313267 conn(0x55f8e864c950 msgr2=0x55f8e8652650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:27.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.191+0000 7f5580f6b740 1 -- 172.21.15.27:0/3068313267 shutdown_connections 2024-04-25T00:24:27.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.191+0000 7f5580f6b740 1 -- 172.21.15.27:0/3068313267 wait complete. 2024-04-25T00:24:27.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.191+0000 7f5580f6b740 20 client.16959 trim_cache size 0 max 16384 2024-04-25T00:24:27.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5580f6b740 1 Processor -- start 2024-04-25T00:24:27.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5580f6b740 1 -- start start 2024-04-25T00:24:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5580f6b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8e8650630 0x55f8e8667300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5580f6b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e86576d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5580f6b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e865b620 0x55f8e8652d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5580f6b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f55781aeba0 con 0x55f8e8650f50 2024-04-25T00:24:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5580f6b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f55781b0740 con 0x55f8e865b620 2024-04-25T00:24:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5580f6b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55781af270 con 0x55f8e8650630 2024-04-25T00:24:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5576ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e86576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5576ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e86576d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35788/0 (socket says 172.21.15.27:35788) 2024-04-25T00:24:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5576ffd640 1 -- 172.21.15.27:0/323790705 learned_addr learned my addr 172.21.15.27:0/323790705 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5577fff640 1 --2- 172.21.15.27:0/323790705 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e865b620 0x55f8e8652d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5576ffd640 1 -- 172.21.15.27:0/323790705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8e8650630 msgr2=0x55f8e8667300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5576ffd640 1 --2- 172.21.15.27:0/323790705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8e8650630 0x55f8e8667300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5576ffd640 1 -- 172.21.15.27:0/323790705 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e865b620 msgr2=0x55f8e8652d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5576ffd640 1 --2- 172.21.15.27:0/323790705 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e865b620 0x55f8e8652d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5576ffd640 1 -- 172.21.15.27:0/323790705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8e863e090 con 0x55f8e8650f50 2024-04-25T00:24:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5577fff640 1 --2- 172.21.15.27:0/323790705 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e865b620 0x55f8e8652d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f5576ffd640 1 --2- 172.21.15.27:0/323790705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e86576d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f556c0bbcf0 tx=0x7f556c0bb740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54551c936e3b1a3d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.209+0000 7f555d7fa640 1 -- 172.21.15.27:0/323790705 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c0bce60 con 0x55f8e8650f50 2024-04-25T00:24:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.210+0000 7f555d7fa640 1 -- 172.21.15.27:0/323790705 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f556c003c80 con 0x55f8e8650f50 2024-04-25T00:24:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.210+0000 7f555d7fa640 1 -- 172.21.15.27:0/323790705 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c0ecfe0 con 0x55f8e8650f50 2024-04-25T00:24:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.210+0000 7f5580f6b740 1 -- 172.21.15.27:0/323790705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 msgr2=0x55f8e86576d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.210+0000 7f5580f6b740 1 --2- 172.21.15.27:0/323790705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e86576d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f556c0bbcf0 tx=0x7f556c0bb740 comp rx=0 tx=0).stop 2024-04-25T00:24:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.210+0000 7f5580f6b740 1 -- 172.21.15.27:0/323790705 shutdown_connections 2024-04-25T00:24:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.210+0000 7f5580f6b740 1 --2- 172.21.15.27:0/323790705 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e865b620 0x55f8e8652d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.210+0000 7f5580f6b740 1 --2- 172.21.15.27:0/323790705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e86576d0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.210+0000 7f5580f6b740 1 --2- 172.21.15.27:0/323790705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8e8650630 0x55f8e8667300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.210+0000 7f5580f6b740 1 -- 172.21.15.27:0/323790705 >> 172.21.15.27:0/323790705 conn(0x55f8e864c950 msgr2=0x55f8e864b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.210+0000 7f5580f6b740 1 -- 172.21.15.27:0/323790705 shutdown_connections 2024-04-25T00:24:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.210+0000 7f5580f6b740 1 -- 172.21.15.27:0/323790705 wait complete. 2024-04-25T00:24:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.210+0000 7f5580f6b740 1 -- start start 2024-04-25T00:24:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5580f6b740 1 Processor -- start 2024-04-25T00:24:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5580f6b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e8650630 0x55f8e8663b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5580f6b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e8654ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5580f6b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8e865b620 0x55f8e8655400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5580f6b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f55781b0070 con 0x55f8e8650f50 2024-04-25T00:24:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5580f6b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f55781af970 con 0x55f8e8650630 2024-04-25T00:24:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5580f6b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55781acb30 con 0x55f8e865b620 2024-04-25T00:24:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5580f6b740 10 client.0 _ll_register_callbacks cb 0x7f556c0b7dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5576ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e8654ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5576ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e8654ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35798/0 (socket says 172.21.15.27:35798) 2024-04-25T00:24:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5576ffd640 1 -- 172.21.15.27:0/1023521986 learned_addr learned my addr 172.21.15.27:0/1023521986 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f55777fe640 1 --2- 172.21.15.27:0/1023521986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e8650630 0x55f8e8663b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5576ffd640 1 -- 172.21.15.27:0/1023521986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8e865b620 msgr2=0x55f8e8655400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:27.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5576ffd640 1 --2- 172.21.15.27:0/1023521986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8e865b620 0x55f8e8655400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5576ffd640 1 -- 172.21.15.27:0/1023521986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e8650630 msgr2=0x55f8e8663b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:27.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5576ffd640 1 --2- 172.21.15.27:0/1023521986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e8650630 0x55f8e8663b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5576ffd640 1 -- 172.21.15.27:0/1023521986 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8e8655b40 con 0x55f8e8650f50 2024-04-25T00:24:27.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f55777fe640 1 --2- 172.21.15.27:0/1023521986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e8650630 0x55f8e8663b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:27.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5576ffd640 1 --2- 172.21.15.27:0/1023521986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e8654ec0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f556c0efc20 tx=0x7f556c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6f33dcd3b2db98 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:27.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.211+0000 7f5556ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:27.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.212+0000 7f5556ffd640 1 -- 172.21.15.27:0/1023521986 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c0bce60 con 0x55f8e8650f50 2024-04-25T00:24:27.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.212+0000 7f5556ffd640 1 -- 172.21.15.27:0/1023521986 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f556c0c4310 con 0x55f8e8650f50 2024-04-25T00:24:27.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.212+0000 7f5580f6b740 1 -- 172.21.15.27:0/1023521986 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f556c003c80 con 0x55f8e8650f50 2024-04-25T00:24:27.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.212+0000 7f5556ffd640 1 -- 172.21.15.27:0/1023521986 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f556c0bb490 con 0x55f8e8650f50 2024-04-25T00:24:27.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.212+0000 7f5580f6b740 10 client.16965 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:27.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.212+0000 7f5580f6b740 1 -- 172.21.15.27:0/1023521986 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55f8e86d1d80 con 0x55f8e8650f50 2024-04-25T00:24:27.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.212+0000 7f5556ffd640 1 -- 172.21.15.27:0/1023521986 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f556c003c80 con 0x55f8e8650f50 2024-04-25T00:24:27.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.212+0000 7f5556ffd640 1 -- 172.21.15.27:0/1023521986 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f556c0b7880 con 0x55f8e8650f50 2024-04-25T00:24:27.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.212+0000 7f5580f6b740 10 client.16965 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:27.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.212+0000 7f5580f6b740 10 client.16965 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:24:27.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.212+0000 7f5580f6b740 20 client.16965 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:27.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.212+0000 7f5580f6b740 1 --2- 172.21.15.27:0/1023521986 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f8e86d2200 0x55f8e86d45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.212+0000 7f5580f6b740 4 client.16965 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:24:27.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.212+0000 7f5580f6b740 1 -- 172.21.15.27:0/1023521986 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f8e85bcf40 con 0x55f8e86d2200 2024-04-25T00:24:27.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.212+0000 7f55777fe640 1 --2- 172.21.15.27:0/1023521986 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f8e86d2200 0x55f8e86d45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.213+0000 7f5556ffd640 10 client.16965 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:27.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.213+0000 7f55777fe640 1 --2- 172.21.15.27:0/1023521986 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f8e86d2200 0x55f8e86d45f0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2eefd2764a062d33 server_cookie=525bb64fbccefe34 in_seq=0 out_seq=0 2024-04-25T00:24:27.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.213+0000 7f5556ffd640 1 -- 172.21.15.27:0/1023521986 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f8e85bcf40 con 0x55f8e86d2200 2024-04-25T00:24:27.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.213+0000 7f5556ffd640 10 client.16965 handle_command_reply: tid=0 2024-04-25T00:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.213+0000 7f5580f6b740 1 client.16965 shutdown 2024-04-25T00:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.213+0000 7f5580f6b740 1 -- 172.21.15.27:0/1023521986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 msgr2=0x55f8e8654ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.213+0000 7f5580f6b740 1 --2- 172.21.15.27:0/1023521986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e8654ec0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f556c0efc20 tx=0x7f556c009260 comp rx=0 tx=0).stop 2024-04-25T00:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.213+0000 7f5580f6b740 1 -- 172.21.15.27:0/1023521986 shutdown_connections 2024-04-25T00:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.213+0000 7f5580f6b740 1 --2- 172.21.15.27:0/1023521986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f8e865b620 0x55f8e8655400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.213+0000 7f5580f6b740 1 --2- 172.21.15.27:0/1023521986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8e8650f50 0x55f8e8654ec0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.213+0000 7f5580f6b740 1 --2- 172.21.15.27:0/1023521986 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55f8e86d2200 0x55f8e86d45f0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.213+0000 7f5580f6b740 1 --2- 172.21.15.27:0/1023521986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f8e8650630 0x55f8e8663b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.213+0000 7f5556ffd640 0 client.16965 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.213+0000 7f5580f6b740 1 -- 172.21.15.27:0/1023521986 >> 172.21.15.27:0/1023521986 conn(0x55f8e864c950 msgr2=0x55f8e8658530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.213+0000 7f5580f6b740 1 -- 172.21.15.27:0/1023521986 shutdown_connections 2024-04-25T00:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.213+0000 7f5580f6b740 1 -- 172.21.15.27:0/1023521986 wait complete. 2024-04-25T00:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.213+0000 7f5580f6b740 20 client.16965 trim_cache size 0 max 16384 2024-04-25T00:24:27.221 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:24:27.221 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T00:24:27.221 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:24:27.221 INFO:teuthology.orchestra.run.smithi027.stdout: "65fb331c-8c72-48c1-af8b-f63d28fbdfb2": { 2024-04-25T00:24:27.221 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:24:27.221 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "65fb331c-8c72-48c1-af8b-f63d28fbdfb2", 2024-04-25T00:24:27.221 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:24:27.221 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:24:27.221 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:24:27.221 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:24:27.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.214+0000 7f557e25d640 1 -- 172.21.15.27:0/2928224494 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f55600a2d80 msgr2=0x7f55600a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.214+0000 7f557e25d640 1 --2- 172.21.15.27:0/2928224494 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f55600a2d80 0x7f55600a5240 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5570008cf0 tx=0x7f5570008230 comp rx=0 tx=0).stop 2024-04-25T00:24:27.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.214+0000 7f557e25d640 1 -- 172.21.15.27:0/2928224494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55781020d0 msgr2=0x7f5578198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.214+0000 7f557e25d640 1 --2- 172.21.15.27:0/2928224494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55781020d0 0x7f5578198570 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f556c009230 tx=0x7f556c004270 comp rx=0 tx=0).stop 2024-04-25T00:24:27.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.214+0000 7f557e25d640 1 -- 172.21.15.27:0/2928224494 shutdown_connections 2024-04-25T00:24:27.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.214+0000 7f557e25d640 1 --2- 172.21.15.27:0/2928224494 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f55600a2d80 0x7f55600a5240 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.214+0000 7f557e25d640 1 --2- 172.21.15.27:0/2928224494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5578107720 0x7f557819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.214+0000 7f557e25d640 1 --2- 172.21.15.27:0/2928224494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55781020d0 0x7f5578198570 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.214+0000 7f557e25d640 1 --2- 172.21.15.27:0/2928224494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5578101710 0x7f5578198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.214+0000 7f557e25d640 1 -- 172.21.15.27:0/2928224494 >> 172.21.15.27:0/2928224494 conn(0x7f5578078b90 msgr2=0x7f55780ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:27.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.214+0000 7f557e25d640 1 -- 172.21.15.27:0/2928224494 shutdown_connections 2024-04-25T00:24:27.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:27.214+0000 7f557e25d640 1 -- 172.21.15.27:0/2928224494 wait complete. 2024-04-25T00:24:27.225 INFO:tasks.cephfs.filesystem:scrub status for tag:65fb331c-8c72-48c1-af8b-f63d28fbdfb2 - {'path': '/', 'tag': '65fb331c-8c72-48c1-af8b-f63d28fbdfb2', 'options': 'recursive,force'} 2024-04-25T00:24:27.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:26 smithi139 ceph-mon[32833]: pgmap v263: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:24:27.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:26 smithi153 ceph-mon[32896]: pgmap v263: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:24:28.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:27 smithi027 ceph-mon[29208]: scrub queued for path: / 2024-04-25T00:24:28.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:27 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:24:28.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:27 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:24:28.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:27 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:24:28.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:27 smithi139 ceph-mon[32833]: scrub queued for path: / 2024-04-25T00:24:28.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:27 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:24:28.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:27 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:24:28.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:27 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:24:28.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:27 smithi153 ceph-mon[32896]: scrub queued for path: / 2024-04-25T00:24:28.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:27 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:24:28.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:27 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:24:28.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:27 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:24:29.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:28 smithi027 ceph-mon[29208]: pgmap v264: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-25T00:24:29.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:28 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:24:29.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:28 smithi139 ceph-mon[32833]: pgmap v264: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-25T00:24:29.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:28 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:24:29.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:28 smithi153 ceph-mon[32896]: pgmap v264: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-25T00:24:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:28 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:24:31.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:30 smithi027 ceph-mon[29208]: pgmap v265: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:24:31.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:30 smithi139 ceph-mon[32833]: pgmap v265: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:24:31.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:30 smithi153 ceph-mon[32896]: pgmap v265: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:24:33.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:32 smithi027 ceph-mon[29208]: pgmap v266: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-04-25T00:24:33.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:32 smithi139 ceph-mon[32833]: pgmap v266: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-04-25T00:24:33.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:32 smithi153 ceph-mon[32896]: pgmap v266: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-04-25T00:24:35.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:34 smithi027 ceph-mon[29208]: pgmap v267: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-25T00:24:35.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:34 smithi139 ceph-mon[32833]: pgmap v267: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-25T00:24:35.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:34 smithi153 ceph-mon[32896]: pgmap v267: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-25T00:24:37.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:36 smithi027 ceph-mon[29208]: pgmap v268: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T00:24:37.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:36 smithi139 ceph-mon[32833]: pgmap v268: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T00:24:37.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:36 smithi153 ceph-mon[32896]: pgmap v268: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T00:24:39.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:38 smithi027 ceph-mon[29208]: pgmap v269: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-04-25T00:24:39.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:38 smithi139 ceph-mon[32833]: pgmap v269: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-04-25T00:24:39.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:38 smithi153 ceph-mon[32896]: pgmap v269: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-04-25T00:24:41.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:40 smithi027 ceph-mon[29208]: pgmap v270: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-04-25T00:24:41.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:40 smithi139 ceph-mon[32833]: pgmap v270: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-04-25T00:24:41.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:40 smithi153 ceph-mon[32896]: pgmap v270: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-04-25T00:24:43.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:42 smithi027 ceph-mon[29208]: pgmap v271: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T00:24:43.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:42 smithi139 ceph-mon[32833]: pgmap v271: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T00:24:43.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:42 smithi153 ceph-mon[32896]: pgmap v271: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T00:24:45.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:44 smithi027 ceph-mon[29208]: pgmap v272: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-04-25T00:24:45.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:44 smithi139 ceph-mon[32833]: pgmap v272: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-04-25T00:24:45.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:44 smithi153 ceph-mon[32896]: pgmap v272: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-04-25T00:24:47.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:46 smithi027 ceph-mon[29208]: pgmap v273: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:24:47.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:46 smithi139 ceph-mon[32833]: pgmap v273: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:24:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:46 smithi153 ceph-mon[32896]: pgmap v273: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:24:49.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:48 smithi027 ceph-mon[29208]: pgmap v274: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-25T00:24:49.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:48 smithi139 ceph-mon[32833]: pgmap v274: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-25T00:24:49.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:48 smithi153 ceph-mon[32896]: pgmap v274: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-25T00:24:51.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:50 smithi027 ceph-mon[29208]: pgmap v275: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:24:51.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:50 smithi139 ceph-mon[32833]: pgmap v275: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:24:51.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:50 smithi153 ceph-mon[32896]: pgmap v275: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:24:53.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:52 smithi027 ceph-mon[29208]: pgmap v276: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T00:24:53.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:52 smithi139 ceph-mon[32833]: pgmap v276: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T00:24:53.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:52 smithi153 ceph-mon[32896]: pgmap v276: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T00:24:55.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:54 smithi027 ceph-mon[29208]: pgmap v277: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-25T00:24:55.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:54 smithi139 ceph-mon[32833]: pgmap v277: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-25T00:24:55.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:54 smithi153 ceph-mon[32896]: pgmap v277: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-25T00:24:57.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:56 smithi027 ceph-mon[29208]: pgmap v278: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-25T00:24:57.226 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:24:57.227 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:24:57.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:56 smithi139 ceph-mon[32833]: pgmap v278: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-25T00:24:57.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:56 smithi153 ceph-mon[32896]: pgmap v278: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-25T00:24:57.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.344+0000 7fb9b4d96640 1 -- 172.21.15.27:0/3721220988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9b010e020 msgr2=0x7fb9b0110410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.344+0000 7fb9b4d96640 1 --2- 172.21.15.27:0/3721220988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9b010e020 0x7fb9b0110410 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8009ca0 tx=0x7fb9a802a950 comp rx=0 tx=0).stop 2024-04-25T00:24:57.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.344+0000 7fb9b4d96640 1 -- 172.21.15.27:0/3721220988 shutdown_connections 2024-04-25T00:24:57.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.344+0000 7fb9b4d96640 1 --2- 172.21.15.27:0/3721220988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9b010e020 0x7fb9b0110410 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.344+0000 7fb9b4d96640 1 --2- 172.21.15.27:0/3721220988 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9b0074330 0x7fb9b010dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.344+0000 7fb9b4d96640 1 --2- 172.21.15.27:0/3721220988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9b0073a10 0x7fb9b0073df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.344+0000 7fb9b4d96640 1 -- 172.21.15.27:0/3721220988 >> 172.21.15.27:0/3721220988 conn(0x7fb9b0078b90 msgr2=0x7fb9b0078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:57.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.344+0000 7fb9b4d96640 1 -- 172.21.15.27:0/3721220988 shutdown_connections 2024-04-25T00:24:57.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.344+0000 7fb9b4d96640 1 -- 172.21.15.27:0/3721220988 wait complete. 2024-04-25T00:24:57.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.344+0000 7fb9b4d96640 1 Processor -- start 2024-04-25T00:24:57.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.345+0000 7fb9b4d96640 1 -- start start 2024-04-25T00:24:57.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.345+0000 7fb9b4d96640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9b0073a10 0x7fb9b019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.345+0000 7fb9b4d96640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9b0074330 0x7fb9b019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.345+0000 7fb9b4d96640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9b010e020 0x7fb9b019eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.345+0000 7fb9b4d96640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b0113d20 con 0x7fb9b010e020 2024-04-25T00:24:57.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.345+0000 7fb9b4d96640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b0113ba0 con 0x7fb9b0073a10 2024-04-25T00:24:57.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.345+0000 7fb9b4d96640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b0113ea0 con 0x7fb9b0074330 2024-04-25T00:24:57.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.345+0000 7fb9aed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9b010e020 0x7fb9b019eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.345+0000 7fb9aed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9b010e020 0x7fb9b019eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44780/0 (socket says 172.21.15.27:44780) 2024-04-25T00:24:57.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.345+0000 7fb9aed9d640 1 -- 172.21.15.27:0/1257210693 learned_addr learned my addr 172.21.15.27:0/1257210693 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:57.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.345+0000 7fb9aed9d640 1 -- 172.21.15.27:0/1257210693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9b0074330 msgr2=0x7fb9b019a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.345+0000 7fb9aed9d640 1 --2- 172.21.15.27:0/1257210693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9b0074330 0x7fb9b019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.345+0000 7fb9aed9d640 1 -- 172.21.15.27:0/1257210693 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9b0073a10 msgr2=0x7fb9b019a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.345+0000 7fb9aed9d640 1 --2- 172.21.15.27:0/1257210693 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9b0073a10 0x7fb9b019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.345+0000 7fb9aed9d640 1 -- 172.21.15.27:0/1257210693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b019f250 con 0x7fb9b010e020 2024-04-25T00:24:57.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.345+0000 7fb9aed9d640 1 --2- 172.21.15.27:0/1257210693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9b010e020 0x7fb9b019eb20 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8004af0 tx=0x7fb9a802f3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:57.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.346+0000 7fb9a77fe640 1 -- 172.21.15.27:0/1257210693 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8034040 con 0x7fb9b010e020 2024-04-25T00:24:57.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.346+0000 7fb9a77fe640 1 -- 172.21.15.27:0/1257210693 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9a8030920 con 0x7fb9b010e020 2024-04-25T00:24:57.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.346+0000 7fb9a77fe640 1 -- 172.21.15.27:0/1257210693 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a80046a0 con 0x7fb9b010e020 2024-04-25T00:24:57.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.346+0000 7fb9b4d96640 1 -- 172.21.15.27:0/1257210693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9b019f4e0 con 0x7fb9b010e020 2024-04-25T00:24:57.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.346+0000 7fb9b4d96640 1 -- 172.21.15.27:0/1257210693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9b01a6dd0 con 0x7fb9b010e020 2024-04-25T00:24:57.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.348+0000 7fb9a77fe640 1 -- 172.21.15.27:0/1257210693 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb9a8037020 con 0x7fb9b010e020 2024-04-25T00:24:57.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.348+0000 7fb9b7b77740 1 Processor -- start 2024-04-25T00:24:57.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9b7b77740 1 -- start start 2024-04-25T00:24:57.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9b7b77740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c89a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9b7b77740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c89f50 0x559f57c940e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9b7b77740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c94620 0x559f57c96a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9b7b77740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559f57bee6b0 con 0x559f57c89630 2024-04-25T00:24:57.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9b7b77740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x559f57b96880 con 0x559f57c89f50 2024-04-25T00:24:57.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9b7b77740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559f57bea670 con 0x559f57c94620 2024-04-25T00:24:57.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9a77fe640 1 --2- 172.21.15.27:0/1257210693 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb99c0a2d80 0x7fb99c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9a7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c89a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9a7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c89a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44788/0 (socket says 172.21.15.27:44788) 2024-04-25T00:24:57.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9a7fff640 1 -- 172.21.15.27:0/221641618 learned_addr learned my addr 172.21.15.27:0/221641618 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:57.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9aed9d640 1 --2- 172.21.15.27:0/1257210693 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb99c0a2d80 0x7fb99c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9a77fe640 1 -- 172.21.15.27:0/1257210693 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fb9a80f0440 con 0x7fb9b010e020 2024-04-25T00:24:57.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9a7fff640 1 -- 172.21.15.27:0/221641618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c94620 msgr2=0x559f57c96a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:57.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9a7fff640 1 --2- 172.21.15.27:0/221641618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c94620 0x559f57c96a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9a7fff640 1 -- 172.21.15.27:0/221641618 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c89f50 msgr2=0x559f57c940e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:57.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9a7fff640 1 --2- 172.21.15.27:0/221641618 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c89f50 0x559f57c940e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.349+0000 7fb9a7fff640 1 -- 172.21.15.27:0/221641618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f579cc7d0 con 0x559f57c89630 2024-04-25T00:24:57.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.350+0000 7fb9a7fff640 1 --2- 172.21.15.27:0/221641618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c89a10 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fb998009cb0 tx=0x7fb99800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a15bdb7425dc7e3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:57.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.350+0000 7fb9aed9d640 1 --2- 172.21.15.27:0/1257210693 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb99c0a2d80 0x7fb99c0a5240 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb9a80bc650 tx=0x7fb9a808f730 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:57.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.350+0000 7fb9a4ff9640 1 -- 172.21.15.27:0/221641618 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb998004020 con 0x559f57c89630 2024-04-25T00:24:57.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.350+0000 7fb9a4ff9640 1 -- 172.21.15.27:0/221641618 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb998005500 con 0x559f57c89630 2024-04-25T00:24:57.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.350+0000 7fb9a4ff9640 1 -- 172.21.15.27:0/221641618 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb998004c10 con 0x559f57c89630 2024-04-25T00:24:57.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 1 -- 172.21.15.27:0/221641618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 msgr2=0x559f57c89a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/221641618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c89a10 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fb998009cb0 tx=0x7fb99800e720 comp rx=0 tx=0).stop 2024-04-25T00:24:57.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 1 -- 172.21.15.27:0/221641618 shutdown_connections 2024-04-25T00:24:57.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/221641618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c94620 0x559f57c96a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/221641618 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c89f50 0x559f57c940e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/221641618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c89a10 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 1 -- 172.21.15.27:0/221641618 >> 172.21.15.27:0/221641618 conn(0x559f57c85950 msgr2=0x559f57c85d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:57.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 1 -- 172.21.15.27:0/221641618 shutdown_connections 2024-04-25T00:24:57.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 1 -- 172.21.15.27:0/221641618 wait complete. 2024-04-25T00:24:57.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 1 -- start start 2024-04-25T00:24:57.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 1 Processor -- start 2024-04-25T00:24:57.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c89630 0x559f57c914a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c89f50 0x559f57c919e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c94620 0x559f57c91f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559f57c97410 con 0x559f57c94620 2024-04-25T00:24:57.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x559f57ba9f50 con 0x559f57c89f50 2024-04-25T00:24:57.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559f57c97590 con 0x559f57c89630 2024-04-25T00:24:57.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9b7b77740 10 client.0 _ll_register_callbacks cb 0x559f57be5000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:57.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.351+0000 7fb9a7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c94620 0x559f57c91f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb9a7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c94620 0x559f57c91f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44802/0 (socket says 172.21.15.27:44802) 2024-04-25T00:24:57.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb9a7fff640 1 -- 172.21.15.27:0/1341458432 learned_addr learned my addr 172.21.15.27:0/1341458432 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:57.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb9ae59c640 1 --2- 172.21.15.27:0/1341458432 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c89f50 0x559f57c919e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb9a7fff640 1 --2- 172.21.15.27:0/1341458432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c89630 0x559f57c914a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb9a7fff640 1 -- 172.21.15.27:0/1341458432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c89630 msgr2=0x559f57c914a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:57.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb9a7fff640 1 --2- 172.21.15.27:0/1341458432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c89630 0x559f57c914a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb9a7fff640 1 -- 172.21.15.27:0/1341458432 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c89f50 msgr2=0x559f57c919e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:57.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb9a7fff640 1 --2- 172.21.15.27:0/1341458432 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c89f50 0x559f57c919e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb9a7fff640 1 -- 172.21.15.27:0/1341458432 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f579023f0 con 0x559f57c94620 2024-04-25T00:24:57.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb9a7fff640 1 --2- 172.21.15.27:0/1341458432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c94620 0x559f57c91f20 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fb998004370 tx=0x7fb998004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70cc61da312892f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:57.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb995ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:57.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb995ffb640 1 -- 172.21.15.27:0/1341458432 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb998003a80 con 0x559f57c94620 2024-04-25T00:24:57.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb995ffb640 1 -- 172.21.15.27:0/1341458432 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fb998003c20 con 0x559f57c94620 2024-04-25T00:24:57.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb9b7b77740 1 -- 172.21.15.27:0/1341458432 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb998005500 con 0x559f57c94620 2024-04-25T00:24:57.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb995ffb640 1 -- 172.21.15.27:0/1341458432 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fb998003e20 con 0x559f57c94620 2024-04-25T00:24:57.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb9b7b77740 10 client.16977 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:57.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb9b7b77740 1 -- 172.21.15.27:0/1341458432 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x559f578c20f0 con 0x559f57c94620 2024-04-25T00:24:57.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.352+0000 7fb995ffb640 1 -- 172.21.15.27:0/1341458432 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb998005500 con 0x559f57c94620 2024-04-25T00:24:57.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.353+0000 7fb995ffb640 1 -- 172.21.15.27:0/1341458432 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7fb998010690 con 0x559f57c94620 2024-04-25T00:24:57.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.353+0000 7fb9b7b77740 10 client.16977 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:57.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.353+0000 7fb9b7b77740 10 client.16977 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:24:57.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.353+0000 7fb9b7b77740 20 client.16977 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.353+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/1341458432 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x559f57ca4850 0x559f57ca6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.353+0000 7fb9b7b77740 4 client.16977 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.353+0000 7fb9b7b77740 1 -- 172.21.15.27:0/1341458432 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f57bfbbf0 con 0x559f57ca4850 2024-04-25T00:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.353+0000 7fb9ae59c640 1 --2- 172.21.15.27:0/1341458432 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x559f57ca4850 0x559f57ca6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.354+0000 7fb9ae59c640 1 --2- 172.21.15.27:0/1341458432 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x559f57ca4850 0x559f57ca6c40 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0c48bc28c579fd2 server_cookie=81159a915e5930dd in_seq=0 out_seq=0 2024-04-25T00:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.354+0000 7fb995ffb640 10 client.16977 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.354+0000 7fb995ffb640 1 -- 172.21.15.27:0/1341458432 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f57bfbbf0 con 0x559f57ca4850 2024-04-25T00:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.354+0000 7fb995ffb640 10 client.16977 handle_command_reply: tid=0 2024-04-25T00:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.354+0000 7fb9b7b77740 1 client.16977 shutdown 2024-04-25T00:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.355+0000 7fb9b7b77740 1 -- 172.21.15.27:0/1341458432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c94620 msgr2=0x559f57c91f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.355+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/1341458432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c94620 0x559f57c91f20 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fb998004370 tx=0x7fb998004570 comp rx=0 tx=0).stop 2024-04-25T00:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.355+0000 7fb9b7b77740 1 -- 172.21.15.27:0/1341458432 shutdown_connections 2024-04-25T00:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.355+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/1341458432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c94620 0x559f57c91f20 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.355+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/1341458432 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x559f57ca4850 0x559f57ca6c40 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.355+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/1341458432 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c89f50 0x559f57c919e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.355+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/1341458432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c89630 0x559f57c914a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.355+0000 7fb995ffb640 0 client.16977 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:57.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.355+0000 7fb9b7b77740 1 -- 172.21.15.27:0/1341458432 >> 172.21.15.27:0/1341458432 conn(0x559f57c85950 msgr2=0x559f57c8b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:57.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.355+0000 7fb9b7b77740 1 -- 172.21.15.27:0/1341458432 shutdown_connections 2024-04-25T00:24:57.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.355+0000 7fb9b7b77740 1 -- 172.21.15.27:0/1341458432 wait complete. 2024-04-25T00:24:57.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.355+0000 7fb9b7b77740 20 client.16977 trim_cache size 0 max 16384 2024-04-25T00:24:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.372+0000 7fb9b7b77740 1 Processor -- start 2024-04-25T00:24:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.372+0000 7fb9b7b77740 1 -- start start 2024-04-25T00:24:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.372+0000 7fb9b7b77740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c93bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.372+0000 7fb9b7b77740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c89f50 0x559f57ca0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.373+0000 7fb9b7b77740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c94620 0x559f57c85d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.373+0000 7fb9b7b77740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b01b0e10 con 0x559f57c89630 2024-04-25T00:24:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.373+0000 7fb9b7b77740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b01b29b0 con 0x559f57c94620 2024-04-25T00:24:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.373+0000 7fb9b7b77740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b01b14e0 con 0x559f57c89f50 2024-04-25T00:24:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.373+0000 7fb9a7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c93bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.373+0000 7fb9a7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c93bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44804/0 (socket says 172.21.15.27:44804) 2024-04-25T00:24:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.373+0000 7fb9a7fff640 1 -- 172.21.15.27:0/2456999531 learned_addr learned my addr 172.21.15.27:0/2456999531 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.373+0000 7fb9ae59c640 1 --2- 172.21.15.27:0/2456999531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c89f50 0x559f57ca0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.373+0000 7fb9a7fff640 1 -- 172.21.15.27:0/2456999531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c89f50 msgr2=0x559f57ca0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.373+0000 7fb9a7fff640 1 --2- 172.21.15.27:0/2456999531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c89f50 0x559f57ca0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.373+0000 7fb9a7fff640 1 -- 172.21.15.27:0/2456999531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c94620 msgr2=0x559f57c85d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.373+0000 7fb9a7fff640 1 --2- 172.21.15.27:0/2456999531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c94620 0x559f57c85d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.373+0000 7fb9a7fff640 1 -- 172.21.15.27:0/2456999531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f57ca6fb0 con 0x559f57c89630 2024-04-25T00:24:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.373+0000 7fb9a7fff640 1 --2- 172.21.15.27:0/2456999531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c93bd0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb99800e760 tx=0x7fb9980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecdff906152fd2d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.373+0000 7fb997fff640 1 -- 172.21.15.27:0/2456999531 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb998003880 con 0x559f57c89630 2024-04-25T00:24:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.373+0000 7fb997fff640 1 -- 172.21.15.27:0/2456999531 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb998003a20 con 0x559f57c89630 2024-04-25T00:24:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.373+0000 7fb997fff640 1 -- 172.21.15.27:0/2456999531 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9980188d0 con 0x559f57c89630 2024-04-25T00:24:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.374+0000 7fb9b7b77740 1 -- 172.21.15.27:0/2456999531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 msgr2=0x559f57c93bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.374+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/2456999531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c93bd0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb99800e760 tx=0x7fb9980057f0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.374+0000 7fb9b7b77740 1 -- 172.21.15.27:0/2456999531 shutdown_connections 2024-04-25T00:24:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.374+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/2456999531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c94620 0x559f57c85d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.374+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/2456999531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c89f50 0x559f57ca0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.374+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/2456999531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c93bd0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.374+0000 7fb9b7b77740 1 -- 172.21.15.27:0/2456999531 >> 172.21.15.27:0/2456999531 conn(0x559f57c85950 msgr2=0x559f57c84260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.374+0000 7fb9b7b77740 1 -- 172.21.15.27:0/2456999531 shutdown_connections 2024-04-25T00:24:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.374+0000 7fb9b7b77740 1 -- 172.21.15.27:0/2456999531 wait complete. 2024-04-25T00:24:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.374+0000 7fb9b7b77740 1 -- start start 2024-04-25T00:24:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.374+0000 7fb9b7b77740 1 Processor -- start 2024-04-25T00:24:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9b7b77740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c9ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9b7b77740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c89f50 0x559f57c8dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9b7b77740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c94620 0x559f57c8e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9b7b77740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b01b22e0 con 0x559f57c89630 2024-04-25T00:24:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9b7b77740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b01b1be0 con 0x559f57c94620 2024-04-25T00:24:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9b7b77740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b01aeda0 con 0x559f57c89f50 2024-04-25T00:24:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9b7b77740 10 client.0 _ll_register_callbacks cb 0x7fb998009b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9a7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c9ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9a7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c9ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44810/0 (socket says 172.21.15.27:44810) 2024-04-25T00:24:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9a7fff640 1 -- 172.21.15.27:0/1499474678 learned_addr learned my addr 172.21.15.27:0/1499474678 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9ae59c640 1 --2- 172.21.15.27:0/1499474678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c89f50 0x559f57c8dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9a7fff640 1 --2- 172.21.15.27:0/1499474678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c94620 0x559f57c8e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9a7fff640 1 -- 172.21.15.27:0/1499474678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c89f50 msgr2=0x559f57c8dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9a7fff640 1 --2- 172.21.15.27:0/1499474678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c89f50 0x559f57c8dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9a7fff640 1 -- 172.21.15.27:0/1499474678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c94620 msgr2=0x559f57c8e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9a7fff640 1 --2- 172.21.15.27:0/1499474678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c94620 0x559f57c8e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9a7fff640 1 -- 172.21.15.27:0/1499474678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f57c8e750 con 0x559f57c89630 2024-04-25T00:24:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9ae59c640 1 --2- 172.21.15.27:0/1499474678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c89f50 0x559f57c8dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9a7fff640 1 --2- 172.21.15.27:0/1499474678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c9ca30 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb998009670 tx=0x7fb998004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953109d138356fa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb995ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb995ffb640 1 -- 172.21.15.27:0/1499474678 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb998003880 con 0x559f57c89630 2024-04-25T00:24:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb9b7b77740 1 -- 172.21.15.27:0/1499474678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb998003a20 con 0x559f57c89630 2024-04-25T00:24:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.375+0000 7fb995ffb640 1 -- 172.21.15.27:0/1499474678 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fb998018570 con 0x559f57c89630 2024-04-25T00:24:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.376+0000 7fb995ffb640 1 -- 172.21.15.27:0/1499474678 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fb998018790 con 0x559f57c89630 2024-04-25T00:24:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.376+0000 7fb9b7b77740 10 client.16983 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.376+0000 7fb9b7b77740 1 -- 172.21.15.27:0/1499474678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x559f57d0a980 con 0x559f57c89630 2024-04-25T00:24:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.376+0000 7fb995ffb640 1 -- 172.21.15.27:0/1499474678 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb998003a20 con 0x559f57c89630 2024-04-25T00:24:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.376+0000 7fb995ffb640 1 -- 172.21.15.27:0/1499474678 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7fb9980357f0 con 0x559f57c89630 2024-04-25T00:24:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.376+0000 7fb9b7b77740 10 client.16983 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.376+0000 7fb9b7b77740 10 client.16983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:24:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.376+0000 7fb9b7b77740 20 client.16983 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.376+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/1499474678 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x559f57d0ae00 0x559f57d0d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.376+0000 7fb9b7b77740 4 client.16983 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:24:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.376+0000 7fb9b7b77740 1 -- 172.21.15.27:0/1499474678 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559f57bf5f40 con 0x559f57d0ae00 2024-04-25T00:24:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.376+0000 7fb9ae59c640 1 --2- 172.21.15.27:0/1499474678 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x559f57d0ae00 0x559f57d0d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.377+0000 7fb995ffb640 10 client.16983 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.377+0000 7fb9ae59c640 1 --2- 172.21.15.27:0/1499474678 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x559f57d0ae00 0x559f57d0d1f0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f45f3c415f60bc3c server_cookie=8baac45941d8a788 in_seq=0 out_seq=0 2024-04-25T00:24:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.377+0000 7fb995ffb640 1 -- 172.21.15.27:0/1499474678 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559f57bf5f40 con 0x559f57d0ae00 2024-04-25T00:24:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.377+0000 7fb995ffb640 10 client.16983 handle_command_reply: tid=0 2024-04-25T00:24:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.377+0000 7fb9b7b77740 1 client.16983 shutdown 2024-04-25T00:24:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.377+0000 7fb9b7b77740 1 -- 172.21.15.27:0/1499474678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 msgr2=0x559f57c9ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.377+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/1499474678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c9ca30 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb998009670 tx=0x7fb998004570 comp rx=0 tx=0).stop 2024-04-25T00:24:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.377+0000 7fb9b7b77740 1 -- 172.21.15.27:0/1499474678 shutdown_connections 2024-04-25T00:24:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.377+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/1499474678 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x559f57d0ae00 0x559f57d0d1f0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.377+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/1499474678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559f57c94620 0x559f57c8e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.377+0000 7fb995ffb640 0 client.16983 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.377+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/1499474678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f57c89f50 0x559f57c8dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.377+0000 7fb9b7b77740 1 --2- 172.21.15.27:0/1499474678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f57c89630 0x559f57c9ca30 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.377+0000 7fb9b7b77740 1 -- 172.21.15.27:0/1499474678 >> 172.21.15.27:0/1499474678 conn(0x559f57c85950 msgr2=0x559f57c84af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:57.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.377+0000 7fb9b7b77740 1 -- 172.21.15.27:0/1499474678 shutdown_connections 2024-04-25T00:24:57.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.377+0000 7fb9b7b77740 1 -- 172.21.15.27:0/1499474678 wait complete. 2024-04-25T00:24:57.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.378+0000 7fb9b7b77740 20 client.16983 trim_cache size 0 max 16384 2024-04-25T00:24:57.386 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:24:57.386 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T00:24:57.386 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T00:24:57.386 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:24:57.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.378+0000 7fb9b4d96640 1 -- 172.21.15.27:0/1257210693 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb99c0a2d80 msgr2=0x7fb99c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.378+0000 7fb9b4d96640 1 --2- 172.21.15.27:0/1257210693 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb99c0a2d80 0x7fb99c0a5240 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb9a80bc650 tx=0x7fb9a808f730 comp rx=0 tx=0).stop 2024-04-25T00:24:57.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.378+0000 7fb9b4d96640 1 -- 172.21.15.27:0/1257210693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9b010e020 msgr2=0x7fb9b019eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.378+0000 7fb9b4d96640 1 --2- 172.21.15.27:0/1257210693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9b010e020 0x7fb9b019eb20 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8004af0 tx=0x7fb9a802f3a0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.378+0000 7fb9b4d96640 1 -- 172.21.15.27:0/1257210693 shutdown_connections 2024-04-25T00:24:57.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.378+0000 7fb9b4d96640 1 --2- 172.21.15.27:0/1257210693 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb99c0a2d80 0x7fb99c0a5240 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.378+0000 7fb9b4d96640 1 --2- 172.21.15.27:0/1257210693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9b010e020 0x7fb9b019eb20 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.378+0000 7fb9b4d96640 1 --2- 172.21.15.27:0/1257210693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9b0074330 0x7fb9b019a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.378+0000 7fb9b4d96640 1 --2- 172.21.15.27:0/1257210693 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9b0073a10 0x7fb9b019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.378+0000 7fb9b4d96640 1 -- 172.21.15.27:0/1257210693 >> 172.21.15.27:0/1257210693 conn(0x7fb9b0078b90 msgr2=0x7fb9b00fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:57.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.378+0000 7fb9b4d96640 1 -- 172.21.15.27:0/1257210693 shutdown_connections 2024-04-25T00:24:57.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.378+0000 7fb9b4d96640 1 -- 172.21.15.27:0/1257210693 wait complete. 2024-04-25T00:24:57.390 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T00:24:57.390 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:24:57.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.491+0000 7f38791b7640 1 -- 172.21.15.27:0/3427093710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38741032f0 msgr2=0x7f387410b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.491+0000 7f38791b7640 1 --2- 172.21.15.27:0/3427093710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38741032f0 0x7f387410b7d0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f385c009900 tx=0x7f385c02a510 comp rx=0 tx=0).stop 2024-04-25T00:24:57.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.491+0000 7f38791b7640 1 -- 172.21.15.27:0/3427093710 shutdown_connections 2024-04-25T00:24:57.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.491+0000 7f38791b7640 1 --2- 172.21.15.27:0/3427093710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f387410bd10 0x7f387410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.491+0000 7f38791b7640 1 --2- 172.21.15.27:0/3427093710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38741032f0 0x7f387410b7d0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.491+0000 7f38791b7640 1 --2- 172.21.15.27:0/3427093710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38741029d0 0x7f3874102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.491+0000 7f38791b7640 1 -- 172.21.15.27:0/3427093710 >> 172.21.15.27:0/3427093710 conn(0x7f3874077740 msgr2=0x7f3874077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:57.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.491+0000 7f38791b7640 1 -- 172.21.15.27:0/3427093710 shutdown_connections 2024-04-25T00:24:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.491+0000 7f38791b7640 1 -- 172.21.15.27:0/3427093710 wait complete. 2024-04-25T00:24:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.492+0000 7f38791b7640 1 Processor -- start 2024-04-25T00:24:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.492+0000 7f38791b7640 1 -- start start 2024-04-25T00:24:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.492+0000 7f38791b7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38741029d0 0x7f3874100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.492+0000 7f38791b7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38741032f0 0x7f3874100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.492+0000 7f38791b7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f387410bd10 0x7f38741010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.492+0000 7f38791b7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3874111b10 con 0x7f387410bd10 2024-04-25T00:24:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.492+0000 7f38791b7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3874111990 con 0x7f38741032f0 2024-04-25T00:24:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.492+0000 7f38791b7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3874111c90 con 0x7f38741029d0 2024-04-25T00:24:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.492+0000 7f387359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f387410bd10 0x7f38741010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.492+0000 7f387359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f387410bd10 0x7f38741010a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44828/0 (socket says 172.21.15.27:44828) 2024-04-25T00:24:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.492+0000 7f387359e640 1 -- 172.21.15.27:0/3490655703 learned_addr learned my addr 172.21.15.27:0/3490655703 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.492+0000 7f3872d9d640 1 --2- 172.21.15.27:0/3490655703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38741029d0 0x7f3874100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.493+0000 7f386bfff640 1 --2- 172.21.15.27:0/3490655703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38741032f0 0x7f3874100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.493+0000 7f387359e640 1 -- 172.21.15.27:0/3490655703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38741029d0 msgr2=0x7f3874100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.493+0000 7f387359e640 1 --2- 172.21.15.27:0/3490655703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38741029d0 0x7f3874100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.493+0000 7f387359e640 1 -- 172.21.15.27:0/3490655703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38741032f0 msgr2=0x7f3874100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.493+0000 7f387359e640 1 --2- 172.21.15.27:0/3490655703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38741032f0 0x7f3874100b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.493+0000 7f387359e640 1 -- 172.21.15.27:0/3490655703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38741a6510 con 0x7f387410bd10 2024-04-25T00:24:57.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.493+0000 7f386bfff640 1 --2- 172.21.15.27:0/3490655703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38741032f0 0x7f3874100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:57.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.493+0000 7f387359e640 1 --2- 172.21.15.27:0/3490655703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f387410bd10 0x7f38741010a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f386c00cad0 tx=0x7f386c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:57.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.493+0000 7f3870d99640 1 -- 172.21.15.27:0/3490655703 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c00ae20 con 0x7f387410bd10 2024-04-25T00:24:57.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.493+0000 7f3870d99640 1 -- 172.21.15.27:0/3490655703 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f386c005020 con 0x7f387410bd10 2024-04-25T00:24:57.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.493+0000 7f38791b7640 1 -- 172.21.15.27:0/3490655703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38741a6800 con 0x7f387410bd10 2024-04-25T00:24:57.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.493+0000 7f3870d99640 1 -- 172.21.15.27:0/3490655703 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c00f6f0 con 0x7f387410bd10 2024-04-25T00:24:57.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.493+0000 7f38791b7640 1 -- 172.21.15.27:0/3490655703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38741a6d50 con 0x7f387410bd10 2024-04-25T00:24:57.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.494+0000 7f38791b7640 1 -- 172.21.15.27:0/3490655703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3874068560 con 0x7f387410bd10 2024-04-25T00:24:57.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.495+0000 7f3870d99640 1 -- 172.21.15.27:0/3490655703 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f386c0051d0 con 0x7f387410bd10 2024-04-25T00:24:57.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.496+0000 7f3870d99640 1 --2- 172.21.15.27:0/3490655703 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f38600a2d00 0x7f38600a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.496+0000 7f3872d9d640 1 --2- 172.21.15.27:0/3490655703 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f38600a2d00 0x7f38600a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.496+0000 7f3870d99640 1 -- 172.21.15.27:0/3490655703 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f386c0cfaa0 con 0x7f387410bd10 2024-04-25T00:24:57.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.497+0000 7f3872d9d640 1 --2- 172.21.15.27:0/3490655703 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f38600a2d00 0x7f38600a51c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3864009bd0 tx=0x7f3864005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:57.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.501+0000 7f3870d99640 1 -- 172.21.15.27:0/3490655703 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f386c014030 con 0x7f387410bd10 2024-04-25T00:24:57.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.724+0000 7f38791b7640 1 -- 172.21.15.27:0/3490655703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f387402d170 con 0x7f387410bd10 2024-04-25T00:24:57.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.725+0000 7f3870d99640 1 -- 172.21.15.27:0/3490655703 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+11954 (secure 0 0 0) 0x7f386c09c350 con 0x7f387410bd10 2024-04-25T00:24:57.727 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:24:57.727 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":97,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:23:11.564498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":147,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":148,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":146,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24514,24430,24521,24428,24496]},"id":1}]} 2024-04-25T00:24:57.727 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 97 2024-04-25T00:24:57.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.727+0000 7f38791b7640 1 -- 172.21.15.27:0/3490655703 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f38600a2d00 msgr2=0x7f38600a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.727+0000 7f38791b7640 1 --2- 172.21.15.27:0/3490655703 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f38600a2d00 0x7f38600a51c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3864009bd0 tx=0x7f3864005f90 comp rx=0 tx=0).stop 2024-04-25T00:24:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.727+0000 7f38791b7640 1 -- 172.21.15.27:0/3490655703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f387410bd10 msgr2=0x7f38741010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.727+0000 7f38791b7640 1 --2- 172.21.15.27:0/3490655703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f387410bd10 0x7f38741010a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f386c00cad0 tx=0x7f386c00cf90 comp rx=0 tx=0).stop 2024-04-25T00:24:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.727+0000 7f38791b7640 1 -- 172.21.15.27:0/3490655703 shutdown_connections 2024-04-25T00:24:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.727+0000 7f38791b7640 1 --2- 172.21.15.27:0/3490655703 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f38600a2d00 0x7f38600a51c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.727+0000 7f38791b7640 1 --2- 172.21.15.27:0/3490655703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f387410bd10 0x7f38741010a0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.727+0000 7f38791b7640 1 --2- 172.21.15.27:0/3490655703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38741032f0 0x7f3874100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.727+0000 7f38791b7640 1 --2- 172.21.15.27:0/3490655703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38741029d0 0x7f3874100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.727+0000 7f38791b7640 1 -- 172.21.15.27:0/3490655703 >> 172.21.15.27:0/3490655703 conn(0x7f3874077740 msgr2=0x7f38740fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:57.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.727+0000 7f38791b7640 1 -- 172.21.15.27:0/3490655703 shutdown_connections 2024-04-25T00:24:57.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.727+0000 7f38791b7640 1 -- 172.21.15.27:0/3490655703 wait complete. 2024-04-25T00:24:57.740 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T00:24:57.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.842+0000 7f1ec7912640 1 -- 172.21.15.27:0/142333930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ec0107720 msgr2=0x7f1ec0107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.842+0000 7f1ec7912640 1 --2- 172.21.15.27:0/142333930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ec0107720 0x7f1ec0107b00 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc009930 tx=0x7f1ebc02a420 comp rx=0 tx=0).stop 2024-04-25T00:24:57.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.842+0000 7f1ec7912640 1 -- 172.21.15.27:0/142333930 shutdown_connections 2024-04-25T00:24:57.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.842+0000 7f1ec7912640 1 --2- 172.21.15.27:0/142333930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec01020d0 0x7f1ec010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.842+0000 7f1ec7912640 1 --2- 172.21.15.27:0/142333930 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ec0101710 0x7f1ec0101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.842+0000 7f1ec7912640 1 --2- 172.21.15.27:0/142333930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ec0107720 0x7f1ec0107b00 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.842+0000 7f1ec7912640 1 -- 172.21.15.27:0/142333930 >> 172.21.15.27:0/142333930 conn(0x7f1ec00fd180 msgr2=0x7f1ec00ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.842+0000 7f1ec7912640 1 -- 172.21.15.27:0/142333930 shutdown_connections 2024-04-25T00:24:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.842+0000 7f1ec7912640 1 -- 172.21.15.27:0/142333930 wait complete. 2024-04-25T00:24:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec7912640 1 Processor -- start 2024-04-25T00:24:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec7912640 1 -- start start 2024-04-25T00:24:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec7912640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ec0101710 0x7f1ec0198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec7912640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ec01020d0 0x7f1ec01985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec7912640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec0107720 0x7f1ec019c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec7912640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec0111af0 con 0x7f1ec01020d0 2024-04-25T00:24:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec7912640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec0111970 con 0x7f1ec0101710 2024-04-25T00:24:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec7912640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec0111c70 con 0x7f1ec0107720 2024-04-25T00:24:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec4ead640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ec01020d0 0x7f1ec01985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec4ead640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ec01020d0 0x7f1ec01985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44838/0 (socket says 172.21.15.27:44838) 2024-04-25T00:24:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec4ead640 1 -- 172.21.15.27:0/466810885 learned_addr learned my addr 172.21.15.27:0/466810885 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec5eaf640 1 --2- 172.21.15.27:0/466810885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec0107720 0x7f1ec019c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec4ead640 1 -- 172.21.15.27:0/466810885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec0107720 msgr2=0x7f1ec019c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec4ead640 1 --2- 172.21.15.27:0/466810885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec0107720 0x7f1ec019c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec4ead640 1 -- 172.21.15.27:0/466810885 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ec0101710 msgr2=0x7f1ec0198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec56ae640 1 --2- 172.21.15.27:0/466810885 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ec0101710 0x7f1ec0198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec4ead640 1 --2- 172.21.15.27:0/466810885 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ec0101710 0x7f1ec0198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.843+0000 7f1ec4ead640 1 -- 172.21.15.27:0/466810885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec019d010 con 0x7f1ec01020d0 2024-04-25T00:24:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.844+0000 7f1ec4ead640 1 --2- 172.21.15.27:0/466810885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ec01020d0 0x7f1ec01985b0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000d950 tx=0x7f1eb000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.844+0000 7f1eb67fc640 1 -- 172.21.15.27:0/466810885 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0017070 con 0x7f1ec01020d0 2024-04-25T00:24:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.844+0000 7f1eb67fc640 1 -- 172.21.15.27:0/466810885 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eb0009c30 con 0x7f1ec01020d0 2024-04-25T00:24:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.844+0000 7f1ec7912640 1 -- 172.21.15.27:0/466810885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ec019d300 con 0x7f1ec01020d0 2024-04-25T00:24:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.844+0000 7f1eb67fc640 1 -- 172.21.15.27:0/466810885 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb000f6b0 con 0x7f1ec01020d0 2024-04-25T00:24:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.844+0000 7f1ec7912640 1 -- 172.21.15.27:0/466810885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ec01a4b90 con 0x7f1ec01020d0 2024-04-25T00:24:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.846+0000 7f1eb67fc640 1 -- 172.21.15.27:0/466810885 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1eb00109d0 con 0x7f1ec01020d0 2024-04-25T00:24:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.846+0000 7f1eca617740 1 Processor -- start 2024-04-25T00:24:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1eca617740 1 -- start start 2024-04-25T00:24:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1eca617740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c8c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1eca617740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8cea0 0x555b50c97030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1eca617740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c97570 0x555b50c99960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1eca617740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555b50bf1670 con 0x555b50c8c580 2024-04-25T00:24:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1eca617740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555b50b99970 con 0x555b50c97570 2024-04-25T00:24:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1eca617740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555b50bed630 con 0x555b50c8cea0 2024-04-25T00:24:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1eb67fc640 1 --2- 172.21.15.27:0/466810885 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1eac0a2d00 0x7f1eac0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1ec5eaf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c8c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1ec5eaf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c8c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44840/0 (socket says 172.21.15.27:44840) 2024-04-25T00:24:57.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1ec5eaf640 1 -- 172.21.15.27:0/2206196220 learned_addr learned my addr 172.21.15.27:0/2206196220 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:57.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1ec4ead640 1 --2- 172.21.15.27:0/2206196220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c97570 0x555b50c99960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1ec56ae640 1 --2- 172.21.15.27:0/2206196220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8cea0 0x555b50c97030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1eb67fc640 1 -- 172.21.15.27:0/466810885 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f1eb00cf4c0 con 0x7f1ec01020d0 2024-04-25T00:24:57.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1ec5eaf640 1 --2- 172.21.15.27:0/466810885 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1eac0a2d00 0x7f1eac0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1ec5eaf640 1 -- 172.21.15.27:0/2206196220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8cea0 msgr2=0x555b50c97030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:57.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1ec5eaf640 1 --2- 172.21.15.27:0/2206196220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8cea0 0x555b50c97030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1ec5eaf640 1 -- 172.21.15.27:0/2206196220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c97570 msgr2=0x555b50c99960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:57.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1ec5eaf640 1 --2- 172.21.15.27:0/2206196220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c97570 0x555b50c99960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.847+0000 7f1ec5eaf640 1 -- 172.21.15.27:0/2206196220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b50bbd400 con 0x555b50c8c580 2024-04-25T00:24:57.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.848+0000 7f1ec5eaf640 1 --2- 172.21.15.27:0/2206196220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c8c960 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8014940 tx=0x7f1eb8014e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9469dee493abbf54 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:57.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.848+0000 7f1eab7fe640 1 -- 172.21.15.27:0/2206196220 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8014050 con 0x555b50c8c580 2024-04-25T00:24:57.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.848+0000 7f1eab7fe640 1 -- 172.21.15.27:0/2206196220 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eb80141f0 con 0x555b50c8c580 2024-04-25T00:24:57.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.848+0000 7f1eab7fe640 1 -- 172.21.15.27:0/2206196220 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8013660 con 0x555b50c8c580 2024-04-25T00:24:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.848+0000 7f1ec5eaf640 1 --2- 172.21.15.27:0/466810885 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1eac0a2d00 0x7f1eac0a51c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8021000 tx=0x7f1eb8020000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.848+0000 7f1eca617740 1 -- 172.21.15.27:0/2206196220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 msgr2=0x555b50c8c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.848+0000 7f1eca617740 1 --2- 172.21.15.27:0/2206196220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c8c960 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8014940 tx=0x7f1eb8014e00 comp rx=0 tx=0).stop 2024-04-25T00:24:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.848+0000 7f1eca617740 1 -- 172.21.15.27:0/2206196220 shutdown_connections 2024-04-25T00:24:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.848+0000 7f1eca617740 1 --2- 172.21.15.27:0/2206196220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c97570 0x555b50c99960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.848+0000 7f1eca617740 1 --2- 172.21.15.27:0/2206196220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8cea0 0x555b50c97030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.848+0000 7f1eca617740 1 --2- 172.21.15.27:0/2206196220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c8c960 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.848+0000 7f1eca617740 1 -- 172.21.15.27:0/2206196220 >> 172.21.15.27:0/2206196220 conn(0x555b50c888a0 msgr2=0x555b50c88c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.848+0000 7f1eca617740 1 -- 172.21.15.27:0/2206196220 shutdown_connections 2024-04-25T00:24:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.849+0000 7f1eca617740 1 -- 172.21.15.27:0/2206196220 wait complete. 2024-04-25T00:24:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.849+0000 7f1eca617740 1 -- start start 2024-04-25T00:24:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.849+0000 7f1eca617740 1 Processor -- start 2024-04-25T00:24:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.849+0000 7f1eca617740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c96eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.849+0000 7f1eca617740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8cea0 0x555b50c941a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.849+0000 7f1eca617740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c97570 0x555b50c946e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.849+0000 7f1eca617740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555b50c9a360 con 0x555b50c8c580 2024-04-25T00:24:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.849+0000 7f1eca617740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555b50bad050 con 0x555b50c97570 2024-04-25T00:24:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.849+0000 7f1eca617740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555b50c9a4e0 con 0x555b50c8cea0 2024-04-25T00:24:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.849+0000 7f1eca617740 10 client.0 _ll_register_callbacks cb 0x555b50904be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.849+0000 7f1ec56ae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c96eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.849+0000 7f1ec56ae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c96eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44856/0 (socket says 172.21.15.27:44856) 2024-04-25T00:24:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.849+0000 7f1ec56ae640 1 -- 172.21.15.27:0/2372824424 learned_addr learned my addr 172.21.15.27:0/2372824424 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.850+0000 7f1ec5eaf640 1 --2- 172.21.15.27:0/2372824424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c97570 0x555b50c946e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.850+0000 7f1ec4ead640 1 --2- 172.21.15.27:0/2372824424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8cea0 0x555b50c941a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.850+0000 7f1ec56ae640 1 -- 172.21.15.27:0/2372824424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8cea0 msgr2=0x555b50c941a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.850+0000 7f1ec56ae640 1 --2- 172.21.15.27:0/2372824424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8cea0 0x555b50c941a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.850+0000 7f1ec56ae640 1 -- 172.21.15.27:0/2372824424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c97570 msgr2=0x555b50c946e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:57.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.850+0000 7f1ec56ae640 1 --2- 172.21.15.27:0/2372824424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c97570 0x555b50c946e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.850+0000 7f1ec56ae640 1 -- 172.21.15.27:0/2372824424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b50b7edc0 con 0x555b50c8c580 2024-04-25T00:24:57.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.850+0000 7f1ec4ead640 1 --2- 172.21.15.27:0/2372824424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8cea0 0x555b50c941a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:57.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.850+0000 7f1ec56ae640 1 --2- 172.21.15.27:0/2372824424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c96eb0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc009900 tx=0x7f1ebc032e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de860879e9185602 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:57.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.850+0000 7f1ea8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:57.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.850+0000 7f1ea8ff9640 1 -- 172.21.15.27:0/2372824424 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc040070 con 0x555b50c8c580 2024-04-25T00:24:57.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.850+0000 7f1eca617740 1 -- 172.21.15.27:0/2372824424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1eb80141f0 con 0x555b50c8c580 2024-04-25T00:24:57.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.850+0000 7f1ea8ff9640 1 -- 172.21.15.27:0/2372824424 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f1ebc004040 con 0x555b50c8c580 2024-04-25T00:24:57.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.850+0000 7f1ea8ff9640 1 -- 172.21.15.27:0/2372824424 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f1ebc035040 con 0x555b50c8c580 2024-04-25T00:24:57.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.850+0000 7f1eca617740 10 client.17001 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:57.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.850+0000 7f1eca617740 1 -- 172.21.15.27:0/2372824424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x555b509053f0 con 0x555b50c8c580 2024-04-25T00:24:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.850+0000 7f1ea8ff9640 1 -- 172.21.15.27:0/2372824424 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eb80141f0 con 0x555b50c8c580 2024-04-25T00:24:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.851+0000 7f1ea8ff9640 1 -- 172.21.15.27:0/2372824424 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f1ebc0048c0 con 0x555b50c8c580 2024-04-25T00:24:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.851+0000 7f1eca617740 10 client.17001 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.851+0000 7f1eca617740 10 client.17001 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:24:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.851+0000 7f1eca617740 20 client.17001 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.851+0000 7f1eca617740 1 --2- 172.21.15.27:0/2372824424 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x555b50ca7540 0x555b50ca9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.851+0000 7f1eca617740 4 client.17001 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:24:57.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.851+0000 7f1eca617740 1 -- 172.21.15.27:0/2372824424 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b50bfebb0 con 0x555b50ca7540 2024-04-25T00:24:57.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.851+0000 7f1ec4ead640 1 --2- 172.21.15.27:0/2372824424 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x555b50ca7540 0x555b50ca9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.851+0000 7f1ec4ead640 1 --2- 172.21.15.27:0/2372824424 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x555b50ca7540 0x555b50ca9990 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cd9fcb8f355e67be server_cookie=bc68b10308da8251 in_seq=0 out_seq=0 2024-04-25T00:24:57.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.851+0000 7f1ea8ff9640 10 client.17001 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:24:57.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.852+0000 7f1ea8ff9640 1 -- 172.21.15.27:0/2372824424 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555b50bfebb0 con 0x555b50ca7540 2024-04-25T00:24:57.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.852+0000 7f1ea8ff9640 10 client.17001 handle_command_reply: tid=0 2024-04-25T00:24:57.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.852+0000 7f1eca617740 1 client.17001 shutdown 2024-04-25T00:24:57.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.852+0000 7f1eca617740 1 -- 172.21.15.27:0/2372824424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 msgr2=0x555b50c96eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.852+0000 7f1eca617740 1 --2- 172.21.15.27:0/2372824424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c96eb0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc009900 tx=0x7f1ebc032e60 comp rx=0 tx=0).stop 2024-04-25T00:24:57.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.852+0000 7f1eca617740 1 -- 172.21.15.27:0/2372824424 shutdown_connections 2024-04-25T00:24:57.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.852+0000 7f1eca617740 1 --2- 172.21.15.27:0/2372824424 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x555b50ca7540 0x555b50ca9990 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.852+0000 7f1eca617740 1 --2- 172.21.15.27:0/2372824424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c97570 0x555b50c946e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.852+0000 7f1ea8ff9640 0 client.17001 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:24:57.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.852+0000 7f1eca617740 1 --2- 172.21.15.27:0/2372824424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8cea0 0x555b50c941a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.852+0000 7f1eca617740 1 --2- 172.21.15.27:0/2372824424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c96eb0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.852+0000 7f1eca617740 1 -- 172.21.15.27:0/2372824424 >> 172.21.15.27:0/2372824424 conn(0x555b50c888a0 msgr2=0x555b50c8e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:57.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.852+0000 7f1eca617740 1 -- 172.21.15.27:0/2372824424 shutdown_connections 2024-04-25T00:24:57.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.853+0000 7f1eca617740 1 -- 172.21.15.27:0/2372824424 wait complete. 2024-04-25T00:24:57.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.853+0000 7f1eca617740 20 client.17001 trim_cache size 0 max 16384 2024-04-25T00:24:57.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.866+0000 7f1eca617740 1 Processor -- start 2024-04-25T00:24:57.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.866+0000 7f1eca617740 1 -- start start 2024-04-25T00:24:57.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.866+0000 7f1eca617740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8c580 0x555b50ca3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.866+0000 7f1eca617740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c8cea0 0x555b50c8f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.866+0000 7f1eca617740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c97570 0x555b50c969e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.866+0000 7f1eca617740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec01aecd0 con 0x555b50c97570 2024-04-25T00:24:57.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.866+0000 7f1eca617740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec01b0870 con 0x555b50c8cea0 2024-04-25T00:24:57.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.866+0000 7f1eca617740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec01af3a0 con 0x555b50c8c580 2024-04-25T00:24:57.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.866+0000 7f1ec5eaf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c97570 0x555b50c969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.866+0000 7f1ec5eaf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c97570 0x555b50c969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44872/0 (socket says 172.21.15.27:44872) 2024-04-25T00:24:57.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.866+0000 7f1ec5eaf640 1 -- 172.21.15.27:0/1927481454 learned_addr learned my addr 172.21.15.27:0/1927481454 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:57.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.866+0000 7f1ec5eaf640 1 -- 172.21.15.27:0/1927481454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8c580 msgr2=0x555b50ca3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:57.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.866+0000 7f1ec5eaf640 1 --2- 172.21.15.27:0/1927481454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8c580 0x555b50ca3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.866+0000 7f1ec5eaf640 1 -- 172.21.15.27:0/1927481454 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c8cea0 msgr2=0x555b50c8f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:57.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.866+0000 7f1ec4ead640 1 --2- 172.21.15.27:0/1927481454 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c8cea0 0x555b50c8f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.866+0000 7f1ec5eaf640 1 --2- 172.21.15.27:0/1927481454 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c8cea0 0x555b50c8f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.866+0000 7f1ec5eaf640 1 -- 172.21.15.27:0/1927481454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b509cf7d0 con 0x555b50c97570 2024-04-25T00:24:57.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.867+0000 7f1ec5eaf640 1 --2- 172.21.15.27:0/1927481454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c97570 0x555b50c969e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8012fd0 tx=0x7f1eb8014f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e12dcddf079a5893 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:57.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.867+0000 7f1eaaffd640 1 -- 172.21.15.27:0/1927481454 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8013400 con 0x555b50c97570 2024-04-25T00:24:57.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.867+0000 7f1eaaffd640 1 -- 172.21.15.27:0/1927481454 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eb8014530 con 0x555b50c97570 2024-04-25T00:24:57.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.867+0000 7f1eaaffd640 1 -- 172.21.15.27:0/1927481454 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8013b20 con 0x555b50c97570 2024-04-25T00:24:57.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.867+0000 7f1eca617740 1 -- 172.21.15.27:0/1927481454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c97570 msgr2=0x555b50c969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.867+0000 7f1eca617740 1 --2- 172.21.15.27:0/1927481454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c97570 0x555b50c969e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8012fd0 tx=0x7f1eb8014f90 comp rx=0 tx=0).stop 2024-04-25T00:24:57.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.867+0000 7f1eca617740 1 -- 172.21.15.27:0/1927481454 shutdown_connections 2024-04-25T00:24:57.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.867+0000 7f1eca617740 1 --2- 172.21.15.27:0/1927481454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c97570 0x555b50c969e0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.867+0000 7f1eca617740 1 --2- 172.21.15.27:0/1927481454 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c8cea0 0x555b50c8f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.867+0000 7f1eca617740 1 --2- 172.21.15.27:0/1927481454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8c580 0x555b50ca3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.867+0000 7f1eca617740 1 -- 172.21.15.27:0/1927481454 >> 172.21.15.27:0/1927481454 conn(0x555b50c888a0 msgr2=0x555b50c8eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:57.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.867+0000 7f1eca617740 1 -- 172.21.15.27:0/1927481454 shutdown_connections 2024-04-25T00:24:57.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.867+0000 7f1eca617740 1 -- 172.21.15.27:0/1927481454 wait complete. 2024-04-25T00:24:57.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.868+0000 7f1eca617740 1 -- start start 2024-04-25T00:24:57.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.868+0000 7f1eca617740 1 Processor -- start 2024-04-25T00:24:57.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.868+0000 7f1eca617740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c9f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.868+0000 7f1eca617740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8cea0 0x555b50c9fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.868+0000 7f1eca617740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c97570 0x555b50c90a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.868+0000 7f1eca617740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec01b01a0 con 0x555b50c8c580 2024-04-25T00:24:57.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.868+0000 7f1eca617740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec01afaa0 con 0x555b50c97570 2024-04-25T00:24:57.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.868+0000 7f1eca617740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec01acc60 con 0x555b50c8cea0 2024-04-25T00:24:57.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.868+0000 7f1eca617740 10 client.0 _ll_register_callbacks cb 0x7f1ebc031400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:57.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.868+0000 7f1ec56ae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c9f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.868+0000 7f1ec56ae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c9f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44880/0 (socket says 172.21.15.27:44880) 2024-04-25T00:24:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.868+0000 7f1ec56ae640 1 -- 172.21.15.27:0/3737783954 learned_addr learned my addr 172.21.15.27:0/3737783954 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.868+0000 7f1ec4ead640 1 --2- 172.21.15.27:0/3737783954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8cea0 0x555b50c9fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.868+0000 7f1ec56ae640 1 -- 172.21.15.27:0/3737783954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8cea0 msgr2=0x555b50c9fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.869+0000 7f1ec56ae640 1 --2- 172.21.15.27:0/3737783954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8cea0 0x555b50c9fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.869+0000 7f1ec56ae640 1 -- 172.21.15.27:0/3737783954 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c97570 msgr2=0x555b50c90a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.869+0000 7f1ec56ae640 1 --2- 172.21.15.27:0/3737783954 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c97570 0x555b50c90a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.869+0000 7f1ec56ae640 1 -- 172.21.15.27:0/3737783954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b50c91230 con 0x555b50c8c580 2024-04-25T00:24:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.869+0000 7f1ec56ae640 1 --2- 172.21.15.27:0/3737783954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c9f730 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc0163f0 tx=0x7f1ebc004c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83a605b002e90775 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:57.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.869+0000 7f1ea8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:57.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.869+0000 7f1ea8ff9640 1 -- 172.21.15.27:0/3737783954 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc040070 con 0x555b50c8c580 2024-04-25T00:24:57.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.869+0000 7f1eca617740 1 -- 172.21.15.27:0/3737783954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1eb8014530 con 0x555b50c8c580 2024-04-25T00:24:57.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.869+0000 7f1ea8ff9640 1 -- 172.21.15.27:0/3737783954 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f1ebc005000 con 0x555b50c8c580 2024-04-25T00:24:57.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.869+0000 7f1ea8ff9640 1 -- 172.21.15.27:0/3737783954 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f1ebc005d50 con 0x555b50c8c580 2024-04-25T00:24:57.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.869+0000 7f1eca617740 10 client.17007 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:57.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.869+0000 7f1eca617740 1 -- 172.21.15.27:0/3737783954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x555b50d0d710 con 0x555b50c8c580 2024-04-25T00:24:57.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.869+0000 7f1ea8ff9640 1 -- 172.21.15.27:0/3737783954 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eb8014530 con 0x555b50c8c580 2024-04-25T00:24:57.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.869+0000 7f1ea8ff9640 1 -- 172.21.15.27:0/3737783954 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f1ebc03c850 con 0x555b50c8c580 2024-04-25T00:24:57.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.869+0000 7f1eca617740 10 client.17007 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:57.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.869+0000 7f1eca617740 10 client.17007 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:24:57.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.869+0000 7f1eca617740 20 client.17007 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:57.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1eca617740 1 --2- 172.21.15.27:0/3737783954 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x555b50d0df40 0x555b50d10360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1eca617740 4 client.17007 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:24:57.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1eca617740 1 -- 172.21.15.27:0/3737783954 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555b50c94a70 con 0x555b50d0df40 2024-04-25T00:24:57.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1ec4ead640 1 --2- 172.21.15.27:0/3737783954 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x555b50d0df40 0x555b50d10360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:57.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1ec4ead640 1 --2- 172.21.15.27:0/3737783954 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x555b50d0df40 0x555b50d10360 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4b65e941f1d38704 server_cookie=860ca3ca377a0b6f in_seq=0 out_seq=0 2024-04-25T00:24:57.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1ea8ff9640 10 client.17007 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:24:57.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1ea8ff9640 1 -- 172.21.15.27:0/3737783954 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555b50c94a70 con 0x555b50d0df40 2024-04-25T00:24:57.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1ea8ff9640 10 client.17007 handle_command_reply: tid=0 2024-04-25T00:24:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1eca617740 1 client.17007 shutdown 2024-04-25T00:24:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1eca617740 1 -- 172.21.15.27:0/3737783954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 msgr2=0x555b50c9f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1eca617740 1 --2- 172.21.15.27:0/3737783954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c9f730 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc0163f0 tx=0x7f1ebc004c00 comp rx=0 tx=0).stop 2024-04-25T00:24:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1eca617740 1 -- 172.21.15.27:0/3737783954 shutdown_connections 2024-04-25T00:24:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1eca617740 1 --2- 172.21.15.27:0/3737783954 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x555b50d0df40 0x555b50d10360 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1eca617740 1 --2- 172.21.15.27:0/3737783954 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555b50c97570 0x555b50c90a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1eca617740 1 --2- 172.21.15.27:0/3737783954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555b50c8cea0 0x555b50c9fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1ea8ff9640 0 client.17007 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:24:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1eca617740 1 --2- 172.21.15.27:0/3737783954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555b50c8c580 0x555b50c9f730 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.870+0000 7f1eca617740 1 -- 172.21.15.27:0/3737783954 >> 172.21.15.27:0/3737783954 conn(0x555b50c888a0 msgr2=0x555b50c8eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.871+0000 7f1eca617740 1 -- 172.21.15.27:0/3737783954 shutdown_connections 2024-04-25T00:24:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.871+0000 7f1eca617740 1 -- 172.21.15.27:0/3737783954 wait complete. 2024-04-25T00:24:57.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.871+0000 7f1eca617740 20 client.17007 trim_cache size 0 max 16384 2024-04-25T00:24:57.888 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:24:57.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.871+0000 7f1ec7912640 1 -- 172.21.15.27:0/466810885 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1eac0a2d00 msgr2=0x7f1eac0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.871+0000 7f1ec7912640 1 --2- 172.21.15.27:0/466810885 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1eac0a2d00 0x7f1eac0a51c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8021000 tx=0x7f1eb8020000 comp rx=0 tx=0).stop 2024-04-25T00:24:57.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.871+0000 7f1ec7912640 1 -- 172.21.15.27:0/466810885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ec01020d0 msgr2=0x7f1ec01985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.871+0000 7f1ec7912640 1 --2- 172.21.15.27:0/466810885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ec01020d0 0x7f1ec01985b0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000d950 tx=0x7f1eb000de10 comp rx=0 tx=0).stop 2024-04-25T00:24:57.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.871+0000 7f1ec7912640 1 -- 172.21.15.27:0/466810885 shutdown_connections 2024-04-25T00:24:57.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.871+0000 7f1ec7912640 1 --2- 172.21.15.27:0/466810885 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f1eac0a2d00 0x7f1eac0a51c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.871+0000 7f1ec7912640 1 --2- 172.21.15.27:0/466810885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec0107720 0x7f1ec019c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.871+0000 7f1ec7912640 1 --2- 172.21.15.27:0/466810885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ec01020d0 0x7f1ec01985b0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.871+0000 7f1ec7912640 1 --2- 172.21.15.27:0/466810885 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ec0101710 0x7f1ec0198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.871+0000 7f1ec7912640 1 -- 172.21.15.27:0/466810885 >> 172.21.15.27:0/466810885 conn(0x7f1ec00fd180 msgr2=0x7f1ec00fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:57.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.871+0000 7f1ec7912640 1 -- 172.21.15.27:0/466810885 shutdown_connections 2024-04-25T00:24:57.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.871+0000 7f1ec7912640 1 -- 172.21.15.27:0/466810885 wait complete. 2024-04-25T00:24:57.892 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T00:24:57.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.995+0000 7f05dd9fb640 1 -- 172.21.15.27:0/302355892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d8107730 msgr2=0x7f05d8107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:57.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.995+0000 7f05dd9fb640 1 --2- 172.21.15.27:0/302355892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d8107730 0x7f05d8107b10 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f05c8009930 tx=0x7f05c802a420 comp rx=0 tx=0).stop 2024-04-25T00:24:57.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.995+0000 7f05dd9fb640 1 -- 172.21.15.27:0/302355892 shutdown_connections 2024-04-25T00:24:57.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.995+0000 7f05dd9fb640 1 --2- 172.21.15.27:0/302355892 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05d806a780 0x7f05d810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.995+0000 7f05dd9fb640 1 --2- 172.21.15.27:0/302355892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05d8069dc0 0x7f05d806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.995+0000 7f05dd9fb640 1 --2- 172.21.15.27:0/302355892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d8107730 0x7f05d8107b10 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:57.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.995+0000 7f05dd9fb640 1 -- 172.21.15.27:0/302355892 >> 172.21.15.27:0/302355892 conn(0x7f05d80776a0 msgr2=0x7f05d8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:57.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.995+0000 7f05dd9fb640 1 -- 172.21.15.27:0/302355892 shutdown_connections 2024-04-25T00:24:57.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.995+0000 7f05dd9fb640 1 -- 172.21.15.27:0/302355892 wait complete. 2024-04-25T00:24:57.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05dd9fb640 1 Processor -- start 2024-04-25T00:24:57.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05dd9fb640 1 -- start start 2024-04-25T00:24:57.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05dd9fb640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05d8069dc0 0x7f05d81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05dd9fb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05d806a780 0x7f05d81985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05dd9fb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d8107730 0x7f05d819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:57.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05dd9fb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f05d8111b40 con 0x7f05d8107730 2024-04-25T00:24:57.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05dd9fb640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f05d81119c0 con 0x7f05d8069dc0 2024-04-25T00:24:58.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05dd9fb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f05d8111cc0 con 0x7f05d806a780 2024-04-25T00:24:58.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05d77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d8107730 0x7f05d819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05d77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d8107730 0x7f05d819c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44892/0 (socket says 172.21.15.27:44892) 2024-04-25T00:24:58.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05d77fe640 1 -- 172.21.15.27:0/1366172077 learned_addr learned my addr 172.21.15.27:0/1366172077 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05d6ffd640 1 --2- 172.21.15.27:0/1366172077 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05d8069dc0 0x7f05d81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05d77fe640 1 -- 172.21.15.27:0/1366172077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05d806a780 msgr2=0x7f05d81985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05d77fe640 1 --2- 172.21.15.27:0/1366172077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05d806a780 0x7f05d81985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05d77fe640 1 -- 172.21.15.27:0/1366172077 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05d8069dc0 msgr2=0x7f05d81980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05d77fe640 1 --2- 172.21.15.27:0/1366172077 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05d8069dc0 0x7f05d81980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05d77fe640 1 -- 172.21.15.27:0/1366172077 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d819d090 con 0x7f05d8107730 2024-04-25T00:24:58.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.996+0000 7f05d6ffd640 1 --2- 172.21.15.27:0/1366172077 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05d8069dc0 0x7f05d81980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:58.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.997+0000 7f05d77fe640 1 --2- 172.21.15.27:0/1366172077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d8107730 0x7f05d819c960 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f05d000eaa0 tx=0x7f05d000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.997+0000 7f05dc9f9640 1 -- 172.21.15.27:0/1366172077 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d000cdb0 con 0x7f05d8107730 2024-04-25T00:24:58.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.997+0000 7f05dc9f9640 1 -- 172.21.15.27:0/1366172077 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05d0005020 con 0x7f05d8107730 2024-04-25T00:24:58.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.997+0000 7f05dd9fb640 1 -- 172.21.15.27:0/1366172077 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05d819d380 con 0x7f05d8107730 2024-04-25T00:24:58.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.997+0000 7f05dc9f9640 1 -- 172.21.15.27:0/1366172077 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d00116b0 con 0x7f05d8107730 2024-04-25T00:24:58.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.997+0000 7f05dd9fb640 1 -- 172.21.15.27:0/1366172077 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d81a4c10 con 0x7f05d8107730 2024-04-25T00:24:58.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.999+0000 7f05dc9f9640 1 -- 172.21.15.27:0/1366172077 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f05d0005560 con 0x7f05d8107730 2024-04-25T00:24:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.999+0000 7f05e07df740 1 Processor -- start 2024-04-25T00:24:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.999+0000 7f05e07df740 1 -- start start 2024-04-25T00:24:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.999+0000 7f05e07df740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ac8580 0x55fbd3ac8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.999+0000 7f05e07df740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3ad3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.999+0000 7f05e07df740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ad3570 0x55fbd3ad5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.999+0000 7f05e07df740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fbd3a2d670 con 0x55fbd3ac8ea0 2024-04-25T00:24:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.999+0000 7f05e07df740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fbd39d5970 con 0x55fbd3ac8580 2024-04-25T00:24:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:57.999+0000 7f05e07df740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fbd3a29630 con 0x55fbd3ad3570 2024-04-25T00:24:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05dc9f9640 1 --2- 172.21.15.27:0/1366172077 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f05c40a2d80 0x7f05c40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05d6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3ad3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05d6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3ad3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44900/0 (socket says 172.21.15.27:44900) 2024-04-25T00:24:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05d6ffd640 1 -- 172.21.15.27:0/1569514133 learned_addr learned my addr 172.21.15.27:0/1569514133 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05ce7fc640 1 --2- 172.21.15.27:0/1569514133 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ac8580 0x55fbd3ac8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05d77fe640 1 --2- 172.21.15.27:0/1366172077 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f05c40a2d80 0x7f05c40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05ce7fc640 1 --2- 172.21.15.27:0/1569514133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ad3570 0x55fbd3ad5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05dc9f9640 1 -- 172.21.15.27:0/1366172077 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f05d00cf9d0 con 0x7f05d8107730 2024-04-25T00:24:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05d6ffd640 1 -- 172.21.15.27:0/1569514133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ad3570 msgr2=0x55fbd3ad5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05d6ffd640 1 --2- 172.21.15.27:0/1569514133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ad3570 0x55fbd3ad5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05d6ffd640 1 -- 172.21.15.27:0/1569514133 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ac8580 msgr2=0x55fbd3ac8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05d6ffd640 1 --2- 172.21.15.27:0/1569514133 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ac8580 0x55fbd3ac8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05d6ffd640 1 -- 172.21.15.27:0/1569514133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbd39f9400 con 0x55fbd3ac8ea0 2024-04-25T00:24:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05ce7fc640 1 --2- 172.21.15.27:0/1569514133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ad3570 0x55fbd3ad5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05d6ffd640 1 --2- 172.21.15.27:0/1569514133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3ad3030 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f05c8006010 tx=0x7f05c8009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e88e4bd509f3ab40 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05d77fe640 1 --2- 172.21.15.27:0/1366172077 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f05c40a2d80 0x7f05c40a5240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f05d0097fe0 tx=0x7f05d00054d0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05cdffb640 1 -- 172.21.15.27:0/1569514133 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c803f070 con 0x55fbd3ac8ea0 2024-04-25T00:24:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05cdffb640 1 -- 172.21.15.27:0/1569514133 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05c80317b0 con 0x55fbd3ac8ea0 2024-04-25T00:24:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.000+0000 7f05cdffb640 1 -- 172.21.15.27:0/1569514133 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c8035070 con 0x55fbd3ac8ea0 2024-04-25T00:24:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.001+0000 7f05e07df740 1 -- 172.21.15.27:0/1569514133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8ea0 msgr2=0x55fbd3ad3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.001+0000 7f05e07df740 1 --2- 172.21.15.27:0/1569514133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3ad3030 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f05c8006010 tx=0x7f05c8009680 comp rx=0 tx=0).stop 2024-04-25T00:24:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.001+0000 7f05e07df740 1 -- 172.21.15.27:0/1569514133 shutdown_connections 2024-04-25T00:24:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.001+0000 7f05e07df740 1 --2- 172.21.15.27:0/1569514133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ad3570 0x55fbd3ad5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.001+0000 7f05e07df740 1 --2- 172.21.15.27:0/1569514133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3ad3030 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.001+0000 7f05e07df740 1 --2- 172.21.15.27:0/1569514133 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ac8580 0x55fbd3ac8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.001+0000 7f05e07df740 1 -- 172.21.15.27:0/1569514133 >> 172.21.15.27:0/1569514133 conn(0x55fbd3ac48a0 msgr2=0x55fbd3ac4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.001+0000 7f05e07df740 1 -- 172.21.15.27:0/1569514133 shutdown_connections 2024-04-25T00:24:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.001+0000 7f05e07df740 1 -- 172.21.15.27:0/1569514133 wait complete. 2024-04-25T00:24:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.001+0000 7f05e07df740 1 -- start start 2024-04-25T00:24:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05e07df740 1 Processor -- start 2024-04-25T00:24:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05e07df740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ac8580 0x55fbd3ad2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05e07df740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3ad01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05e07df740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ad3570 0x55fbd3ad0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05e07df740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fbd3ad6300 con 0x55fbd3ad3570 2024-04-25T00:24:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05e07df740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fbd39e9050 con 0x55fbd3ac8580 2024-04-25T00:24:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05e07df740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fbd3ad6480 con 0x55fbd3ac8ea0 2024-04-25T00:24:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05e07df740 10 client.0 _ll_register_callbacks cb 0x55fbd3740be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05ce7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ad3570 0x55fbd3ad0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05ce7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ad3570 0x55fbd3ad0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44904/0 (socket says 172.21.15.27:44904) 2024-04-25T00:24:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05ce7fc640 1 -- 172.21.15.27:0/1763177280 learned_addr learned my addr 172.21.15.27:0/1763177280 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05d6ffd640 1 --2- 172.21.15.27:0/1763177280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3ad01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05ce7fc640 1 --2- 172.21.15.27:0/1763177280 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ac8580 0x55fbd3ad2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05ce7fc640 1 -- 172.21.15.27:0/1763177280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ac8ea0 msgr2=0x55fbd3ad01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05ce7fc640 1 --2- 172.21.15.27:0/1763177280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3ad01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05ce7fc640 1 -- 172.21.15.27:0/1763177280 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ac8580 msgr2=0x55fbd3ad2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05ce7fc640 1 --2- 172.21.15.27:0/1763177280 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ac8580 0x55fbd3ad2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.002+0000 7f05ce7fc640 1 -- 172.21.15.27:0/1763177280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbd39badc0 con 0x55fbd3ad3570 2024-04-25T00:24:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.003+0000 7f05ce7fc640 1 --2- 172.21.15.27:0/1763177280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ad3570 0x55fbd3ad0730 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f05c0012970 tx=0x7f05c0005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d792c60debc9d7a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.003+0000 7f05b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.003+0000 7f05b6ffd640 1 -- 172.21.15.27:0/1763177280 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c00118e0 con 0x55fbd3ad3570 2024-04-25T00:24:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.003+0000 7f05b6ffd640 1 -- 172.21.15.27:0/1763177280 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f05c0011a80 con 0x55fbd3ad3570 2024-04-25T00:24:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.003+0000 7f05e07df740 1 -- 172.21.15.27:0/1763177280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05c80317b0 con 0x55fbd3ad3570 2024-04-25T00:24:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.003+0000 7f05b6ffd640 1 -- 172.21.15.27:0/1763177280 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f05c0011c80 con 0x55fbd3ad3570 2024-04-25T00:24:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.003+0000 7f05e07df740 10 client.17019 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.003+0000 7f05e07df740 1 -- 172.21.15.27:0/1763177280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55fbd37413f0 con 0x55fbd3ad3570 2024-04-25T00:24:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.003+0000 7f05b6ffd640 1 -- 172.21.15.27:0/1763177280 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05c80317b0 con 0x55fbd3ad3570 2024-04-25T00:24:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.003+0000 7f05b6ffd640 1 -- 172.21.15.27:0/1763177280 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f05c000f040 con 0x55fbd3ad3570 2024-04-25T00:24:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.003+0000 7f05e07df740 10 client.17019 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.003+0000 7f05e07df740 10 client.17019 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:24:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.003+0000 7f05e07df740 20 client.17019 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.003+0000 7f05e07df740 1 --2- 172.21.15.27:0/1763177280 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55fbd3ae35f0 0x55fbd3ae59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.003+0000 7f05e07df740 4 client.17019 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:24:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.003+0000 7f05e07df740 1 -- 172.21.15.27:0/1763177280 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbd3a3abb0 con 0x55fbd3ae35f0 2024-04-25T00:24:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.004+0000 7f05d6ffd640 1 --2- 172.21.15.27:0/1763177280 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55fbd3ae35f0 0x55fbd3ae59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.004+0000 7f05b6ffd640 10 client.17019 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:24:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.004+0000 7f05d6ffd640 1 --2- 172.21.15.27:0/1763177280 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55fbd3ae35f0 0x55fbd3ae59e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bbcde86fa7794e35 server_cookie=cb19032701db3f17 in_seq=0 out_seq=0 2024-04-25T00:24:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.005+0000 7f05b6ffd640 1 -- 172.21.15.27:0/1763177280 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fbd3a3abb0 con 0x55fbd3ae35f0 2024-04-25T00:24:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.005+0000 7f05b6ffd640 10 client.17019 handle_command_reply: tid=0 2024-04-25T00:24:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.005+0000 7f05e07df740 1 client.17019 shutdown 2024-04-25T00:24:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.005+0000 7f05e07df740 1 -- 172.21.15.27:0/1763177280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ad3570 msgr2=0x55fbd3ad0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.005+0000 7f05e07df740 1 --2- 172.21.15.27:0/1763177280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ad3570 0x55fbd3ad0730 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f05c0012970 tx=0x7f05c0005a50 comp rx=0 tx=0).stop 2024-04-25T00:24:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.005+0000 7f05e07df740 1 -- 172.21.15.27:0/1763177280 shutdown_connections 2024-04-25T00:24:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.005+0000 7f05e07df740 1 --2- 172.21.15.27:0/1763177280 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55fbd3ae35f0 0x55fbd3ae59e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.005+0000 7f05e07df740 1 --2- 172.21.15.27:0/1763177280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ad3570 0x55fbd3ad0730 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.005+0000 7f05e07df740 1 --2- 172.21.15.27:0/1763177280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3ad01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.005+0000 7f05e07df740 1 --2- 172.21.15.27:0/1763177280 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ac8580 0x55fbd3ad2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.005+0000 7f05b6ffd640 0 client.17019 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:24:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.005+0000 7f05e07df740 1 -- 172.21.15.27:0/1763177280 >> 172.21.15.27:0/1763177280 conn(0x55fbd3ac48a0 msgr2=0x55fbd3ac3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.005+0000 7f05e07df740 1 -- 172.21.15.27:0/1763177280 shutdown_connections 2024-04-25T00:24:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.005+0000 7f05e07df740 1 -- 172.21.15.27:0/1763177280 wait complete. 2024-04-25T00:24:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.005+0000 7f05e07df740 20 client.17019 trim_cache size 0 max 16384 2024-04-25T00:24:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.018+0000 7f05e07df740 1 Processor -- start 2024-04-25T00:24:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.018+0000 7f05e07df740 1 -- start start 2024-04-25T00:24:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05e07df740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8580 0x55fbd3adf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05e07df740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3ad2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05e07df740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ad3570 0x55fbd3aeb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05e07df740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f05d81aec80 con 0x55fbd3ac8580 2024-04-25T00:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05e07df740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f05d81b0820 con 0x55fbd3ad3570 2024-04-25T00:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05e07df740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f05d81af350 con 0x55fbd3ac8ea0 2024-04-25T00:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05ce7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8580 0x55fbd3adf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05ce7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8580 0x55fbd3adf1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44914/0 (socket says 172.21.15.27:44914) 2024-04-25T00:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05ce7fc640 1 -- 172.21.15.27:0/2870969475 learned_addr learned my addr 172.21.15.27:0/2870969475 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05d6ffd640 1 --2- 172.21.15.27:0/2870969475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3ad2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05ce7fc640 1 --2- 172.21.15.27:0/2870969475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ad3570 0x55fbd3aeb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05ce7fc640 1 -- 172.21.15.27:0/2870969475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ac8ea0 msgr2=0x55fbd3ad2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05ce7fc640 1 --2- 172.21.15.27:0/2870969475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3ad2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05ce7fc640 1 -- 172.21.15.27:0/2870969475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ad3570 msgr2=0x55fbd3aeb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05ce7fc640 1 --2- 172.21.15.27:0/2870969475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ad3570 0x55fbd3aeb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05ce7fc640 1 -- 172.21.15.27:0/2870969475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbd380b7d0 con 0x55fbd3ac8580 2024-04-25T00:24:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05ce7fc640 1 --2- 172.21.15.27:0/2870969475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8580 0x55fbd3adf1f0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f05c0005cb0 tx=0x7f05c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc9384147f812e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05cd7fa640 1 -- 172.21.15.27:0/2870969475 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c0004ad0 con 0x55fbd3ac8580 2024-04-25T00:24:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05cd7fa640 1 -- 172.21.15.27:0/2870969475 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05c0030070 con 0x55fbd3ac8580 2024-04-25T00:24:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.019+0000 7f05cd7fa640 1 -- 172.21.15.27:0/2870969475 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c002b450 con 0x55fbd3ac8580 2024-04-25T00:24:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.020+0000 7f05e07df740 1 -- 172.21.15.27:0/2870969475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8580 msgr2=0x55fbd3adf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.020+0000 7f05e07df740 1 --2- 172.21.15.27:0/2870969475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8580 0x55fbd3adf1f0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f05c0005cb0 tx=0x7f05c00046c0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.020+0000 7f05e07df740 1 -- 172.21.15.27:0/2870969475 shutdown_connections 2024-04-25T00:24:58.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.020+0000 7f05e07df740 1 --2- 172.21.15.27:0/2870969475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ad3570 0x55fbd3aeb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.020+0000 7f05e07df740 1 --2- 172.21.15.27:0/2870969475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3ad2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.020+0000 7f05e07df740 1 --2- 172.21.15.27:0/2870969475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8580 0x55fbd3adf1f0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.020+0000 7f05e07df740 1 -- 172.21.15.27:0/2870969475 >> 172.21.15.27:0/2870969475 conn(0x55fbd3ac48a0 msgr2=0x55fbd3acb860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.020+0000 7f05e07df740 1 -- 172.21.15.27:0/2870969475 shutdown_connections 2024-04-25T00:24:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.020+0000 7f05e07df740 1 -- 172.21.15.27:0/2870969475 wait complete. 2024-04-25T00:24:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.020+0000 7f05e07df740 1 -- start start 2024-04-25T00:24:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05e07df740 1 Processor -- start 2024-04-25T00:24:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05e07df740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8580 0x55fbd3adb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05e07df740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3adbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05e07df740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ad3570 0x55fbd3ad6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05e07df740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f05d81b0150 con 0x55fbd3ac8580 2024-04-25T00:24:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05e07df740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f05d81afa50 con 0x55fbd3ac8ea0 2024-04-25T00:24:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05e07df740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f05d81acc10 con 0x55fbd3ad3570 2024-04-25T00:24:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05e07df740 10 client.0 _ll_register_callbacks cb 0x7f05c00125b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05ce7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8580 0x55fbd3adb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05ce7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8580 0x55fbd3adb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44926/0 (socket says 172.21.15.27:44926) 2024-04-25T00:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05ce7fc640 1 -- 172.21.15.27:0/2774793943 learned_addr learned my addr 172.21.15.27:0/2774793943 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05ce7fc640 1 --2- 172.21.15.27:0/2774793943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ad3570 0x55fbd3ad6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05d6ffd640 1 --2- 172.21.15.27:0/2774793943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3adbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05ce7fc640 1 -- 172.21.15.27:0/2774793943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ad3570 msgr2=0x55fbd3ad6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05ce7fc640 1 --2- 172.21.15.27:0/2774793943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ad3570 0x55fbd3ad6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05ce7fc640 1 -- 172.21.15.27:0/2774793943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ac8ea0 msgr2=0x55fbd3adbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05ce7fc640 1 --2- 172.21.15.27:0/2774793943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3adbbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05ce7fc640 1 -- 172.21.15.27:0/2774793943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbd3ad73b0 con 0x55fbd3ac8580 2024-04-25T00:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05ce7fc640 1 --2- 172.21.15.27:0/2774793943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8580 0x55fbd3adb670 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f05c0011b80 tx=0x7f05c0012660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8155223973215fae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.021+0000 7f05b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.022+0000 7f05b6ffd640 1 -- 172.21.15.27:0/2774793943 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c003c060 con 0x55fbd3ac8580 2024-04-25T00:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.022+0000 7f05b6ffd640 1 -- 172.21.15.27:0/2774793943 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f05c00053d0 con 0x55fbd3ac8580 2024-04-25T00:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.022+0000 7f05e07df740 1 -- 172.21.15.27:0/2774793943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05c0030070 con 0x55fbd3ac8580 2024-04-25T00:24:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.022+0000 7f05b6ffd640 1 -- 172.21.15.27:0/2774793943 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f05c00117b0 con 0x55fbd3ac8580 2024-04-25T00:24:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.022+0000 7f05e07df740 10 client.17025 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.022+0000 7f05e07df740 1 -- 172.21.15.27:0/2774793943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55fbd3b49700 con 0x55fbd3ac8580 2024-04-25T00:24:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.022+0000 7f05b6ffd640 1 -- 172.21.15.27:0/2774793943 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05c0030070 con 0x55fbd3ac8580 2024-04-25T00:24:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.022+0000 7f05b6ffd640 1 -- 172.21.15.27:0/2774793943 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f05c00035f0 con 0x55fbd3ac8580 2024-04-25T00:24:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.022+0000 7f05e07df740 10 client.17025 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.022+0000 7f05e07df740 10 client.17025 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:24:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.022+0000 7f05e07df740 20 client.17025 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.022+0000 7f05e07df740 1 --2- 172.21.15.27:0/2774793943 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55fbd3b49dc0 0x55fbd3b4c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.022+0000 7f05e07df740 4 client.17025 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:24:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.022+0000 7f05e07df740 1 -- 172.21.15.27:0/2774793943 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fbd3a34f00 con 0x55fbd3b49dc0 2024-04-25T00:24:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05d6ffd640 1 --2- 172.21.15.27:0/2774793943 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55fbd3b49dc0 0x55fbd3b4c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05d6ffd640 1 --2- 172.21.15.27:0/2774793943 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55fbd3b49dc0 0x55fbd3b4c1e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1ca77c1ffceea04b server_cookie=943f72add20893e8 in_seq=0 out_seq=0 2024-04-25T00:24:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05b6ffd640 10 client.17025 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:24:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05b6ffd640 1 -- 172.21.15.27:0/2774793943 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fbd3a34f00 con 0x55fbd3b49dc0 2024-04-25T00:24:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05b6ffd640 10 client.17025 handle_command_reply: tid=0 2024-04-25T00:24:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05e07df740 1 client.17025 shutdown 2024-04-25T00:24:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05e07df740 1 -- 172.21.15.27:0/2774793943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8580 msgr2=0x55fbd3adb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05e07df740 1 --2- 172.21.15.27:0/2774793943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8580 0x55fbd3adb670 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f05c0011b80 tx=0x7f05c0012660 comp rx=0 tx=0).stop 2024-04-25T00:24:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05e07df740 1 -- 172.21.15.27:0/2774793943 shutdown_connections 2024-04-25T00:24:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05e07df740 1 --2- 172.21.15.27:0/2774793943 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55fbd3b49dc0 0x55fbd3b4c1e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05e07df740 1 --2- 172.21.15.27:0/2774793943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fbd3ad3570 0x55fbd3ad6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05e07df740 1 --2- 172.21.15.27:0/2774793943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fbd3ac8ea0 0x55fbd3adbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05b6ffd640 0 client.17025 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:24:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05e07df740 1 --2- 172.21.15.27:0/2774793943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fbd3ac8580 0x55fbd3adb670 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05e07df740 1 -- 172.21.15.27:0/2774793943 >> 172.21.15.27:0/2774793943 conn(0x55fbd3ac48a0 msgr2=0x55fbd3acae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05e07df740 1 -- 172.21.15.27:0/2774793943 shutdown_connections 2024-04-25T00:24:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05e07df740 1 -- 172.21.15.27:0/2774793943 wait complete. 2024-04-25T00:24:58.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.023+0000 7f05e07df740 20 client.17025 trim_cache size 0 max 16384 2024-04-25T00:24:58.034 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:24:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.024+0000 7f05dd9fb640 1 -- 172.21.15.27:0/1366172077 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f05c40a2d80 msgr2=0x7f05c40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.024+0000 7f05dd9fb640 1 --2- 172.21.15.27:0/1366172077 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f05c40a2d80 0x7f05c40a5240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f05d0097fe0 tx=0x7f05d00054d0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.024+0000 7f05dd9fb640 1 -- 172.21.15.27:0/1366172077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d8107730 msgr2=0x7f05d819c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.024+0000 7f05dd9fb640 1 --2- 172.21.15.27:0/1366172077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d8107730 0x7f05d819c960 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f05d000eaa0 tx=0x7f05d000ef60 comp rx=0 tx=0).stop 2024-04-25T00:24:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.024+0000 7f05dd9fb640 1 -- 172.21.15.27:0/1366172077 shutdown_connections 2024-04-25T00:24:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.024+0000 7f05dd9fb640 1 --2- 172.21.15.27:0/1366172077 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f05c40a2d80 0x7f05c40a5240 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.024+0000 7f05dd9fb640 1 --2- 172.21.15.27:0/1366172077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d8107730 0x7f05d819c960 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.024+0000 7f05dd9fb640 1 --2- 172.21.15.27:0/1366172077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05d806a780 0x7f05d81985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.024+0000 7f05dd9fb640 1 --2- 172.21.15.27:0/1366172077 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05d8069dc0 0x7f05d81980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.024+0000 7f05dd9fb640 1 -- 172.21.15.27:0/1366172077 >> 172.21.15.27:0/1366172077 conn(0x7f05d80776a0 msgr2=0x7f05d80fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.024+0000 7f05dd9fb640 1 -- 172.21.15.27:0/1366172077 shutdown_connections 2024-04-25T00:24:58.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.024+0000 7f05dd9fb640 1 -- 172.21.15.27:0/1366172077 wait complete. 2024-04-25T00:24:58.036 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:57 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3490655703' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:24:58.037 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T00:24:58.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.139+0000 7f80edecd640 1 -- 172.21.15.27:0/210142359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e810bd00 msgr2=0x7f80e810e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.139+0000 7f80edecd640 1 --2- 172.21.15.27:0/210142359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e810bd00 0x7f80e810e1a0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f80e0009ca0 tx=0x7f80e002a9e0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.139+0000 7f80edecd640 1 -- 172.21.15.27:0/210142359 shutdown_connections 2024-04-25T00:24:58.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.139+0000 7f80edecd640 1 --2- 172.21.15.27:0/210142359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e810bd00 0x7f80e810e1a0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.139+0000 7f80edecd640 1 --2- 172.21.15.27:0/210142359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80e81032e0 0x7f80e810b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.139+0000 7f80edecd640 1 --2- 172.21.15.27:0/210142359 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f80e81029c0 0x7f80e8102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.139+0000 7f80edecd640 1 -- 172.21.15.27:0/210142359 >> 172.21.15.27:0/210142359 conn(0x7f80e8077740 msgr2=0x7f80e8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.139+0000 7f80edecd640 1 -- 172.21.15.27:0/210142359 shutdown_connections 2024-04-25T00:24:58.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.140+0000 7f80edecd640 1 -- 172.21.15.27:0/210142359 wait complete. 2024-04-25T00:24:58.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.140+0000 7f80edecd640 1 Processor -- start 2024-04-25T00:24:58.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.140+0000 7f80edecd640 1 -- start start 2024-04-25T00:24:58.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.140+0000 7f80edecd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f80e81029c0 0x7f80e8197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.140+0000 7f80edecd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80e81032e0 0x7f80e8198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.140+0000 7f80edecd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e810bd00 0x7f80e819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.140+0000 7f80edecd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f80e8111b00 con 0x7f80e810bd00 2024-04-25T00:24:58.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.140+0000 7f80edecd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f80e8111980 con 0x7f80e81029c0 2024-04-25T00:24:58.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.140+0000 7f80edecd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f80e8111c80 con 0x7f80e81032e0 2024-04-25T00:24:58.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.141+0000 7f80e7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e810bd00 0x7f80e819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.141+0000 7f80e7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e810bd00 0x7f80e819c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44936/0 (socket says 172.21.15.27:44936) 2024-04-25T00:24:58.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.141+0000 7f80e7fff640 1 -- 172.21.15.27:0/2889197466 learned_addr learned my addr 172.21.15.27:0/2889197466 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.141+0000 7f80e77fe640 1 --2- 172.21.15.27:0/2889197466 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f80e81029c0 0x7f80e8197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.141+0000 7f80e6ffd640 1 --2- 172.21.15.27:0/2889197466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80e81032e0 0x7f80e8198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.141+0000 7f80e7fff640 1 -- 172.21.15.27:0/2889197466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80e81032e0 msgr2=0x7f80e8198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.141+0000 7f80e7fff640 1 --2- 172.21.15.27:0/2889197466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80e81032e0 0x7f80e8198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.141+0000 7f80e7fff640 1 -- 172.21.15.27:0/2889197466 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f80e81029c0 msgr2=0x7f80e8197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.141+0000 7f80e7fff640 1 --2- 172.21.15.27:0/2889197466 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f80e81029c0 0x7f80e8197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.141+0000 7f80e7fff640 1 -- 172.21.15.27:0/2889197466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e819cf80 con 0x7f80e810bd00 2024-04-25T00:24:58.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.141+0000 7f80e77fe640 1 --2- 172.21.15.27:0/2889197466 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f80e81029c0 0x7f80e8197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:24:58.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.141+0000 7f80e7fff640 1 --2- 172.21.15.27:0/2889197466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e810bd00 0x7f80e819c850 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f80e0006010 tx=0x7f80e0003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.141+0000 7f80e4ff9640 1 -- 172.21.15.27:0/2889197466 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e0047070 con 0x7f80e810bd00 2024-04-25T00:24:58.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.141+0000 7f80e4ff9640 1 -- 172.21.15.27:0/2889197466 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80e0003e50 con 0x7f80e810bd00 2024-04-25T00:24:58.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.141+0000 7f80edecd640 1 -- 172.21.15.27:0/2889197466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e819d210 con 0x7f80e810bd00 2024-04-25T00:24:58.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.141+0000 7f80e4ff9640 1 -- 172.21.15.27:0/2889197466 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e0033050 con 0x7f80e810bd00 2024-04-25T00:24:58.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.141+0000 7f80edecd640 1 -- 172.21.15.27:0/2889197466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e81a4b00 con 0x7f80e810bd00 2024-04-25T00:24:58.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.144+0000 7f80e4ff9640 1 -- 172.21.15.27:0/2889197466 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f80e004a070 con 0x7f80e810bd00 2024-04-25T00:24:58.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.144+0000 7f80f0bd9740 1 Processor -- start 2024-04-25T00:24:58.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.144+0000 7f80f0bd9740 1 -- start start 2024-04-25T00:24:58.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.144+0000 7f80f0bd9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 0x55b62d479960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.144+0000 7f80f0bd9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d484030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.144+0000 7f80e4ff9640 1 --2- 172.21.15.27:0/2889197466 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f80d00a2d00 0x7f80d00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.144+0000 7f80f0bd9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d484570 0x55b62d486960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.144+0000 7f80f0bd9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b62d3de670 con 0x55b62d479ea0 2024-04-25T00:24:58.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.144+0000 7f80f0bd9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b62d386970 con 0x55b62d479580 2024-04-25T00:24:58.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.144+0000 7f80f0bd9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b62d3da630 con 0x55b62d484570 2024-04-25T00:24:58.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.144+0000 7f80e6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d484030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.144+0000 7f80e6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d484030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44950/0 (socket says 172.21.15.27:44950) 2024-04-25T00:24:58.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.144+0000 7f80e6ffd640 1 -- 172.21.15.27:0/104455006 learned_addr learned my addr 172.21.15.27:0/104455006 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.144+0000 7f80e77fe640 1 --2- 172.21.15.27:0/2889197466 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f80d00a2d00 0x7f80d00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.144+0000 7f80e4ff9640 1 -- 172.21.15.27:0/2889197466 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f80e00f0370 con 0x7f80e810bd00 2024-04-25T00:24:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.145+0000 7f80e6ffd640 1 --2- 172.21.15.27:0/104455006 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 0x55b62d479960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.145+0000 7f80e6ffd640 1 -- 172.21.15.27:0/104455006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d484570 msgr2=0x55b62d486960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:24:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.145+0000 7f80e6ffd640 1 --2- 172.21.15.27:0/104455006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d484570 0x55b62d486960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.145+0000 7f80e6ffd640 1 -- 172.21.15.27:0/104455006 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 msgr2=0x55b62d479960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.145+0000 7f80e6ffd640 1 --2- 172.21.15.27:0/104455006 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 0x55b62d479960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.145+0000 7f80e6ffd640 1 -- 172.21.15.27:0/104455006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b62d3aa400 con 0x55b62d479ea0 2024-04-25T00:24:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.145+0000 7f80e77fe640 1 --2- 172.21.15.27:0/2889197466 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f80d00a2d00 0x7f80d00a51c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f80d800bfd0 tx=0x7f80d800b390 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.145+0000 7f80e6ffd640 1 --2- 172.21.15.27:0/104455006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d484030 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f80dc012540 tx=0x7f80dc00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94a6ade24358b12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.145+0000 7f80cdffb640 1 -- 172.21.15.27:0/104455006 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc004030 con 0x55b62d479ea0 2024-04-25T00:24:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.145+0000 7f80cdffb640 1 -- 172.21.15.27:0/104455006 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80dc0041d0 con 0x55b62d479ea0 2024-04-25T00:24:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.145+0000 7f80cdffb640 1 -- 172.21.15.27:0/104455006 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc00f6e0 con 0x55b62d479ea0 2024-04-25T00:24:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.146+0000 7f80f0bd9740 1 -- 172.21.15.27:0/104455006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 msgr2=0x55b62d484030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.146+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/104455006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d484030 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f80dc012540 tx=0x7f80dc00b790 comp rx=0 tx=0).stop 2024-04-25T00:24:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.146+0000 7f80f0bd9740 1 -- 172.21.15.27:0/104455006 shutdown_connections 2024-04-25T00:24:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.146+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/104455006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d484570 0x55b62d486960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.146+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/104455006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d484030 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.146+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/104455006 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 0x55b62d479960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.146+0000 7f80f0bd9740 1 -- 172.21.15.27:0/104455006 >> 172.21.15.27:0/104455006 conn(0x55b62d4758a0 msgr2=0x55b62d475c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.146+0000 7f80f0bd9740 1 -- 172.21.15.27:0/104455006 shutdown_connections 2024-04-25T00:24:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.146+0000 7f80f0bd9740 1 -- 172.21.15.27:0/104455006 wait complete. 2024-04-25T00:24:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.146+0000 7f80f0bd9740 1 -- start start 2024-04-25T00:24:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.146+0000 7f80f0bd9740 1 Processor -- start 2024-04-25T00:24:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80f0bd9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 0x55b62d483f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80f0bd9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d479ea0 0x55b62d481260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80f0bd9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d484570 0x55b62d481870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80f0bd9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b62d487360 con 0x55b62d484570 2024-04-25T00:24:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80f0bd9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b62d39a050 con 0x55b62d479580 2024-04-25T00:24:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80f0bd9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b62d4874e0 con 0x55b62d479ea0 2024-04-25T00:24:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80f0bd9740 10 client.0 _ll_register_callbacks cb 0x55b62d0f1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80e7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d484570 0x55b62d481870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80e7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d484570 0x55b62d481870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44956/0 (socket says 172.21.15.27:44956) 2024-04-25T00:24:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80e7fff640 1 -- 172.21.15.27:0/4192539818 learned_addr learned my addr 172.21.15.27:0/4192539818 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80e7fff640 1 -- 172.21.15.27:0/4192539818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d479ea0 msgr2=0x55b62d481260 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:24:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80e7fff640 1 --2- 172.21.15.27:0/4192539818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d479ea0 0x55b62d481260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80e7fff640 1 -- 172.21.15.27:0/4192539818 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 msgr2=0x55b62d483f70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:24:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80e7fff640 1 --2- 172.21.15.27:0/4192539818 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 0x55b62d483f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80e7fff640 1 -- 172.21.15.27:0/4192539818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b62d36bdc0 con 0x55b62d484570 2024-04-25T00:24:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80e7fff640 1 --2- 172.21.15.27:0/4192539818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d484570 0x55b62d481870 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f80e00ed7b0 tx=0x7f80e00bc110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec362eb801ce19f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80c6ffd640 1 -- 172.21.15.27:0/4192539818 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e00bc5d0 con 0x55b62d484570 2024-04-25T00:24:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80c6ffd640 1 -- 172.21.15.27:0/4192539818 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f80e00f0150 con 0x55b62d484570 2024-04-25T00:24:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80f0bd9740 1 -- 172.21.15.27:0/4192539818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80dc0041d0 con 0x55b62d484570 2024-04-25T00:24:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80c6ffd640 1 -- 172.21.15.27:0/4192539818 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f80e00c08d0 con 0x55b62d484570 2024-04-25T00:24:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80f0bd9740 10 client.17037 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.147+0000 7f80f0bd9740 1 -- 172.21.15.27:0/4192539818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55b62d0f23f0 con 0x55b62d484570 2024-04-25T00:24:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.148+0000 7f80c6ffd640 1 -- 172.21.15.27:0/4192539818 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80dc0041d0 con 0x55b62d484570 2024-04-25T00:24:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.148+0000 7f80c6ffd640 1 -- 172.21.15.27:0/4192539818 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f80e0004b40 con 0x55b62d484570 2024-04-25T00:24:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.148+0000 7f80f0bd9740 10 client.17037 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.148+0000 7f80f0bd9740 10 client.17037 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:24:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.148+0000 7f80f0bd9740 20 client.17037 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.148+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/4192539818 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55b62d494550 0x55b62d4969a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.148+0000 7f80f0bd9740 4 client.17037 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:24:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.148+0000 7f80f0bd9740 1 -- 172.21.15.27:0/4192539818 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b62d3ebbb0 con 0x55b62d494550 2024-04-25T00:24:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.148+0000 7f80e77fe640 1 --2- 172.21.15.27:0/4192539818 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55b62d494550 0x55b62d4969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.149+0000 7f80c6ffd640 10 client.17037 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:24:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.149+0000 7f80e77fe640 1 --2- 172.21.15.27:0/4192539818 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55b62d494550 0x55b62d4969a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e055f03252c4bcb3 server_cookie=ed7df5d99b3b231d in_seq=0 out_seq=0 2024-04-25T00:24:58.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.149+0000 7f80c6ffd640 1 -- 172.21.15.27:0/4192539818 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b62d3ebbb0 con 0x55b62d494550 2024-04-25T00:24:58.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.149+0000 7f80c6ffd640 10 client.17037 handle_command_reply: tid=0 2024-04-25T00:24:58.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.149+0000 7f80f0bd9740 1 client.17037 shutdown 2024-04-25T00:24:58.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.150+0000 7f80f0bd9740 1 -- 172.21.15.27:0/4192539818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d484570 msgr2=0x55b62d481870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.150+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/4192539818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d484570 0x55b62d481870 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f80e00ed7b0 tx=0x7f80e00bc110 comp rx=0 tx=0).stop 2024-04-25T00:24:58.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.150+0000 7f80f0bd9740 1 -- 172.21.15.27:0/4192539818 shutdown_connections 2024-04-25T00:24:58.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.150+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/4192539818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d484570 0x55b62d481870 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.150+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/4192539818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d479ea0 0x55b62d481260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.150+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/4192539818 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55b62d494550 0x55b62d4969a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.150+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/4192539818 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 0x55b62d483f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.150+0000 7f80c6ffd640 0 client.17037 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:24:58.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.150+0000 7f80f0bd9740 1 -- 172.21.15.27:0/4192539818 >> 172.21.15.27:0/4192539818 conn(0x55b62d4758a0 msgr2=0x55b62d474b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.150+0000 7f80f0bd9740 1 -- 172.21.15.27:0/4192539818 shutdown_connections 2024-04-25T00:24:58.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.150+0000 7f80f0bd9740 1 -- 172.21.15.27:0/4192539818 wait complete. 2024-04-25T00:24:58.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.150+0000 7f80f0bd9740 20 client.17037 trim_cache size 0 max 16384 2024-04-25T00:24:58.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.163+0000 7f80f0bd9740 1 Processor -- start 2024-04-25T00:24:58.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.163+0000 7f80f0bd9740 1 -- start start 2024-04-25T00:24:58.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.163+0000 7f80f0bd9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 0x55b62d490250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.163+0000 7f80f0bd9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d4749c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.163+0000 7f80f0bd9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d484570 0x55b62d483aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.163+0000 7f80f0bd9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f80e81aeb00 con 0x55b62d479ea0 2024-04-25T00:24:58.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.163+0000 7f80f0bd9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f80e81b06a0 con 0x55b62d479580 2024-04-25T00:24:58.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.163+0000 7f80f0bd9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f80e81af1d0 con 0x55b62d484570 2024-04-25T00:24:58.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.163+0000 7f80e6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d4749c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.163+0000 7f80e6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d4749c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44964/0 (socket says 172.21.15.27:44964) 2024-04-25T00:24:58.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.163+0000 7f80e6ffd640 1 -- 172.21.15.27:0/3615798593 learned_addr learned my addr 172.21.15.27:0/3615798593 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.163+0000 7f80e77fe640 1 --2- 172.21.15.27:0/3615798593 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 0x55b62d490250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.163+0000 7f80e7fff640 1 --2- 172.21.15.27:0/3615798593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d484570 0x55b62d483aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.163+0000 7f80e6ffd640 1 -- 172.21.15.27:0/3615798593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d484570 msgr2=0x55b62d483aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80e6ffd640 1 --2- 172.21.15.27:0/3615798593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d484570 0x55b62d483aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80e6ffd640 1 -- 172.21.15.27:0/3615798593 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 msgr2=0x55b62d490250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80e6ffd640 1 --2- 172.21.15.27:0/3615798593 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 0x55b62d490250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80e6ffd640 1 -- 172.21.15.27:0/3615798593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b62d1bc7d0 con 0x55b62d479ea0 2024-04-25T00:24:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80e7fff640 1 --2- 172.21.15.27:0/3615798593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d484570 0x55b62d483aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80e6ffd640 1 --2- 172.21.15.27:0/3615798593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d4749c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f80dc01b000 tx=0x7f80dc00bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1109958654495f82 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80cd7fa640 1 -- 172.21.15.27:0/3615798593 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc005110 con 0x55b62d479ea0 2024-04-25T00:24:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80cd7fa640 1 -- 172.21.15.27:0/3615798593 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80dc0035f0 con 0x55b62d479ea0 2024-04-25T00:24:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80cd7fa640 1 -- 172.21.15.27:0/3615798593 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc0106c0 con 0x55b62d479ea0 2024-04-25T00:24:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80f0bd9740 1 -- 172.21.15.27:0/3615798593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 msgr2=0x55b62d4749c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/3615798593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d4749c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f80dc01b000 tx=0x7f80dc00bf60 comp rx=0 tx=0).stop 2024-04-25T00:24:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80f0bd9740 1 -- 172.21.15.27:0/3615798593 shutdown_connections 2024-04-25T00:24:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/3615798593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d484570 0x55b62d483aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/3615798593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d4749c0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/3615798593 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 0x55b62d490250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80f0bd9740 1 -- 172.21.15.27:0/3615798593 >> 172.21.15.27:0/3615798593 conn(0x55b62d4758a0 msgr2=0x55b62d4742b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80f0bd9740 1 -- 172.21.15.27:0/3615798593 shutdown_connections 2024-04-25T00:24:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.164+0000 7f80f0bd9740 1 -- 172.21.15.27:0/3615798593 wait complete. 2024-04-25T00:24:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.165+0000 7f80f0bd9740 1 -- start start 2024-04-25T00:24:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.165+0000 7f80f0bd9740 1 Processor -- start 2024-04-25T00:24:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.165+0000 7f80f0bd9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 0x55b62d4f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.165+0000 7f80f0bd9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d4f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.165+0000 7f80f0bd9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d484570 0x55b62d4f3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.165+0000 7f80f0bd9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f80e81affd0 con 0x55b62d479ea0 2024-04-25T00:24:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.165+0000 7f80f0bd9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f80e81af8d0 con 0x55b62d479580 2024-04-25T00:24:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.165+0000 7f80f0bd9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f80e81aca90 con 0x55b62d484570 2024-04-25T00:24:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.165+0000 7f80f0bd9740 10 client.0 _ll_register_callbacks cb 0x7f80e0005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.165+0000 7f80e6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d4f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.165+0000 7f80e6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d4f2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44980/0 (socket says 172.21.15.27:44980) 2024-04-25T00:24:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.165+0000 7f80e6ffd640 1 -- 172.21.15.27:0/1694040227 learned_addr learned my addr 172.21.15.27:0/1694040227 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.165+0000 7f80e77fe640 1 --2- 172.21.15.27:0/1694040227 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 0x55b62d4f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.165+0000 7f80e7fff640 1 --2- 172.21.15.27:0/1694040227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d484570 0x55b62d4f3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.165+0000 7f80e6ffd640 1 -- 172.21.15.27:0/1694040227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d484570 msgr2=0x55b62d4f3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.165+0000 7f80e6ffd640 1 --2- 172.21.15.27:0/1694040227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d484570 0x55b62d4f3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80e6ffd640 1 -- 172.21.15.27:0/1694040227 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 msgr2=0x55b62d4f2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80e6ffd640 1 --2- 172.21.15.27:0/1694040227 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 0x55b62d4f2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80e6ffd640 1 -- 172.21.15.27:0/1694040227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b62d4fdd70 con 0x55b62d479ea0 2024-04-25T00:24:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80e6ffd640 1 --2- 172.21.15.27:0/1694040227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d4f2fe0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f80dc012280 tx=0x7f80dc0122b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6c9bf0813519f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80c6ffd640 1 -- 172.21.15.27:0/1694040227 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc005110 con 0x55b62d479ea0 2024-04-25T00:24:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80c6ffd640 1 -- 172.21.15.27:0/1694040227 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f80dc0052b0 con 0x55b62d479ea0 2024-04-25T00:24:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80f0bd9740 1 -- 172.21.15.27:0/1694040227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80dc0035f0 con 0x55b62d479ea0 2024-04-25T00:24:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80c6ffd640 1 -- 172.21.15.27:0/1694040227 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f80dc039050 con 0x55b62d479ea0 2024-04-25T00:24:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80f0bd9740 10 client.17043 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80f0bd9740 1 -- 172.21.15.27:0/1694040227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55b62d4fefe0 con 0x55b62d479ea0 2024-04-25T00:24:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80c6ffd640 1 -- 172.21.15.27:0/1694040227 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80dc0035f0 con 0x55b62d479ea0 2024-04-25T00:24:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80c6ffd640 1 -- 172.21.15.27:0/1694040227 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f80dc01e430 con 0x55b62d479ea0 2024-04-25T00:24:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80f0bd9740 10 client.17043 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80f0bd9740 10 client.17043 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:24:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80f0bd9740 20 client.17043 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/1694040227 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55b62d4ff480 0x55b62d5018a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80f0bd9740 4 client.17043 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:24:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.166+0000 7f80f0bd9740 1 -- 172.21.15.27:0/1694040227 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b62d3e5f00 con 0x55b62d4ff480 2024-04-25T00:24:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.167+0000 7f80e77fe640 1 --2- 172.21.15.27:0/1694040227 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55b62d4ff480 0x55b62d5018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.168+0000 7f80e77fe640 1 --2- 172.21.15.27:0/1694040227 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55b62d4ff480 0x55b62d5018a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=161ebe60204e8017 server_cookie=eeef6d899646cd78 in_seq=0 out_seq=0 2024-04-25T00:24:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.168+0000 7f80c6ffd640 10 client.17043 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:24:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.168+0000 7f80c6ffd640 1 -- 172.21.15.27:0/1694040227 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b62d3e5f00 con 0x55b62d4ff480 2024-04-25T00:24:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.168+0000 7f80c6ffd640 10 client.17043 handle_command_reply: tid=0 2024-04-25T00:24:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.168+0000 7f80f0bd9740 1 client.17043 shutdown 2024-04-25T00:24:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.168+0000 7f80f0bd9740 1 -- 172.21.15.27:0/1694040227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 msgr2=0x55b62d4f2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.168+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/1694040227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d4f2fe0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f80dc012280 tx=0x7f80dc0122b0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80f0bd9740 1 -- 172.21.15.27:0/1694040227 shutdown_connections 2024-04-25T00:24:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/1694040227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b62d484570 0x55b62d4f3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/1694040227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b62d479ea0 0x55b62d4f2fe0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/1694040227 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55b62d4ff480 0x55b62d5018a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80f0bd9740 1 --2- 172.21.15.27:0/1694040227 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b62d479580 0x55b62d4f2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80c6ffd640 0 client.17043 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:24:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80f0bd9740 1 -- 172.21.15.27:0/1694040227 >> 172.21.15.27:0/1694040227 conn(0x55b62d4758a0 msgr2=0x55b62d474b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80f0bd9740 1 -- 172.21.15.27:0/1694040227 shutdown_connections 2024-04-25T00:24:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80f0bd9740 1 -- 172.21.15.27:0/1694040227 wait complete. 2024-04-25T00:24:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80f0bd9740 20 client.17043 trim_cache size 0 max 16384 2024-04-25T00:24:58.177 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:24:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80edecd640 1 -- 172.21.15.27:0/2889197466 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f80d00a2d00 msgr2=0x7f80d00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80edecd640 1 --2- 172.21.15.27:0/2889197466 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f80d00a2d00 0x7f80d00a51c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f80d800bfd0 tx=0x7f80d800b390 comp rx=0 tx=0).stop 2024-04-25T00:24:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80edecd640 1 -- 172.21.15.27:0/2889197466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e810bd00 msgr2=0x7f80e819c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80edecd640 1 --2- 172.21.15.27:0/2889197466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e810bd00 0x7f80e819c850 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f80e0006010 tx=0x7f80e0003ce0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80edecd640 1 -- 172.21.15.27:0/2889197466 shutdown_connections 2024-04-25T00:24:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80edecd640 1 --2- 172.21.15.27:0/2889197466 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f80d00a2d00 0x7f80d00a51c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80edecd640 1 --2- 172.21.15.27:0/2889197466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80e810bd00 0x7f80e819c850 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80edecd640 1 --2- 172.21.15.27:0/2889197466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80e81032e0 0x7f80e8198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80edecd640 1 --2- 172.21.15.27:0/2889197466 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f80e81029c0 0x7f80e8197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80edecd640 1 -- 172.21.15.27:0/2889197466 >> 172.21.15.27:0/2889197466 conn(0x7f80e8077740 msgr2=0x7f80e80fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80edecd640 1 -- 172.21.15.27:0/2889197466 shutdown_connections 2024-04-25T00:24:58.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.169+0000 7f80edecd640 1 -- 172.21.15.27:0/2889197466 wait complete. 2024-04-25T00:24:58.181 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T00:24:58.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:57 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3490655703' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:24:58.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:57 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3490655703' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:24:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.284+0000 7ff28c5b7640 1 -- 172.21.15.27:0/2553183803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff284100fd0 msgr2=0x7ff28410d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.284+0000 7ff28c5b7640 1 --2- 172.21.15.27:0/2553183803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff284100fd0 0x7ff28410d4e0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ff27c009cd0 tx=0x7ff27c02a8e0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.284+0000 7ff28c5b7640 1 -- 172.21.15.27:0/2553183803 shutdown_connections 2024-04-25T00:24:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.284+0000 7ff28c5b7640 1 --2- 172.21.15.27:0/2553183803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff284100fd0 0x7ff28410d4e0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.284+0000 7ff28c5b7640 1 --2- 172.21.15.27:0/2553183803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff284100610 0x7ff284100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.284+0000 7ff28c5b7640 1 --2- 172.21.15.27:0/2553183803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff2841066b0 0x7ff284106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.284+0000 7ff28c5b7640 1 -- 172.21.15.27:0/2553183803 >> 172.21.15.27:0/2553183803 conn(0x7ff284077740 msgr2=0x7ff284077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.284+0000 7ff28c5b7640 1 -- 172.21.15.27:0/2553183803 shutdown_connections 2024-04-25T00:24:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.284+0000 7ff28c5b7640 1 -- 172.21.15.27:0/2553183803 wait complete. 2024-04-25T00:24:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.284+0000 7ff28c5b7640 1 Processor -- start 2024-04-25T00:24:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.284+0000 7ff28c5b7640 1 -- start start 2024-04-25T00:24:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28c5b7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff284100610 0x7ff284197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28c5b7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff284100fd0 0x7ff2841984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28c5b7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2841066b0 0x7ff28419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28c5b7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff284111b30 con 0x7ff2841066b0 2024-04-25T00:24:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28c5b7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff2841119b0 con 0x7ff284100610 2024-04-25T00:24:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28c5b7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff284111cb0 con 0x7ff284100fd0 2024-04-25T00:24:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28ab54640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2841066b0 0x7ff28419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28ab54640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2841066b0 0x7ff28419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44998/0 (socket says 172.21.15.27:44998) 2024-04-25T00:24:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28ab54640 1 -- 172.21.15.27:0/1144398385 learned_addr learned my addr 172.21.15.27:0/1144398385 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28a353640 1 --2- 172.21.15.27:0/1144398385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff284100610 0x7ff284197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff289b52640 1 --2- 172.21.15.27:0/1144398385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff284100fd0 0x7ff2841984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28ab54640 1 -- 172.21.15.27:0/1144398385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff284100fd0 msgr2=0x7ff2841984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28ab54640 1 --2- 172.21.15.27:0/1144398385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff284100fd0 0x7ff2841984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28ab54640 1 -- 172.21.15.27:0/1144398385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff284100610 msgr2=0x7ff284197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28ab54640 1 --2- 172.21.15.27:0/1144398385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff284100610 0x7ff284197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28ab54640 1 -- 172.21.15.27:0/1144398385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff28419cf50 con 0x7ff2841066b0 2024-04-25T00:24:58.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff289b52640 1 --2- 172.21.15.27:0/1144398385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff284100fd0 0x7ff2841984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:58.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28ab54640 1 --2- 172.21.15.27:0/1144398385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2841066b0 0x7ff28419c820 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7ff27c005cc0 tx=0x7ff27c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff27b7fe640 1 -- 172.21.15.27:0/1144398385 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c004040 con 0x7ff2841066b0 2024-04-25T00:24:58.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28c5b7640 1 -- 172.21.15.27:0/1144398385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff28419d1e0 con 0x7ff2841066b0 2024-04-25T00:24:58.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff27b7fe640 1 -- 172.21.15.27:0/1144398385 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff27c0046c0 con 0x7ff2841066b0 2024-04-25T00:24:58.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff27b7fe640 1 -- 172.21.15.27:0/1144398385 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c046480 con 0x7ff2841066b0 2024-04-25T00:24:58.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.285+0000 7ff28c5b7640 1 -- 172.21.15.27:0/1144398385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2841a4ad0 con 0x7ff2841066b0 2024-04-25T00:24:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.287+0000 7ff27b7fe640 1 -- 172.21.15.27:0/1144398385 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff27c0305a0 con 0x7ff2841066b0 2024-04-25T00:24:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.288+0000 7ff28f3a5740 1 Processor -- start 2024-04-25T00:24:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.288+0000 7ff28f3a5740 1 -- start start 2024-04-25T00:24:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.288+0000 7ff28f3a5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af710b580 0x557af710b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.288+0000 7ff28f3a5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af710bea0 0x557af7116030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.288+0000 7ff28f3a5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af7116570 0x557af7118960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.288+0000 7ff28f3a5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557af7070670 con 0x557af7116570 2024-04-25T00:24:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.288+0000 7ff28f3a5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557af7018970 con 0x557af710bea0 2024-04-25T00:24:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.288+0000 7ff28f3a5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557af706c630 con 0x557af710b580 2024-04-25T00:24:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff27b7fe640 1 --2- 172.21.15.27:0/1144398385 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff2700a2d00 0x7ff2700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff289b52640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af7116570 0x557af7118960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff289b52640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af7116570 0x557af7118960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45002/0 (socket says 172.21.15.27:45002) 2024-04-25T00:24:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff289b52640 1 -- 172.21.15.27:0/3598231694 learned_addr learned my addr 172.21.15.27:0/3598231694 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff289b52640 1 --2- 172.21.15.27:0/3598231694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af710b580 0x557af710b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff28ab54640 1 --2- 172.21.15.27:0/1144398385 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff2700a2d00 0x7ff2700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff28a353640 1 --2- 172.21.15.27:0/3598231694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af710bea0 0x557af7116030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff27b7fe640 1 -- 172.21.15.27:0/1144398385 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7ff27c02e030 con 0x7ff2841066b0 2024-04-25T00:24:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff289b52640 1 -- 172.21.15.27:0/3598231694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af710b580 msgr2=0x557af710b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff289b52640 1 --2- 172.21.15.27:0/3598231694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af710b580 0x557af710b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff289b52640 1 -- 172.21.15.27:0/3598231694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af710bea0 msgr2=0x557af7116030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff289b52640 1 --2- 172.21.15.27:0/3598231694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af710bea0 0x557af7116030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff289b52640 1 -- 172.21.15.27:0/3598231694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557af703c400 con 0x557af7116570 2024-04-25T00:24:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff28a353640 1 --2- 172.21.15.27:0/3598231694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af710bea0 0x557af7116030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff28ab54640 1 --2- 172.21.15.27:0/1144398385 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff2700a2d00 0x7ff2700a51c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0bcee0 tx=0x7ff27c0b2670 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff289b52640 1 --2- 172.21.15.27:0/3598231694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af7116570 0x557af7118960 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ff274014a30 tx=0x7ff27400c570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3228d7b7666e3bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff278bfe640 1 -- 172.21.15.27:0/3598231694 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2740142d0 con 0x557af7116570 2024-04-25T00:24:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff278bfe640 1 -- 172.21.15.27:0/3598231694 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff274005500 con 0x557af7116570 2024-04-25T00:24:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.289+0000 7ff278bfe640 1 -- 172.21.15.27:0/3598231694 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff274004a00 con 0x557af7116570 2024-04-25T00:24:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.290+0000 7ff28f3a5740 1 -- 172.21.15.27:0/3598231694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af7116570 msgr2=0x557af7118960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.290+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/3598231694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af7116570 0x557af7118960 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ff274014a30 tx=0x7ff27400c570 comp rx=0 tx=0).stop 2024-04-25T00:24:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.290+0000 7ff28f3a5740 1 -- 172.21.15.27:0/3598231694 shutdown_connections 2024-04-25T00:24:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.290+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/3598231694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af7116570 0x557af7118960 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.290+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/3598231694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af710bea0 0x557af7116030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.290+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/3598231694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af710b580 0x557af710b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.290+0000 7ff28f3a5740 1 -- 172.21.15.27:0/3598231694 >> 172.21.15.27:0/3598231694 conn(0x557af71078a0 msgr2=0x557af7107c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.290+0000 7ff28f3a5740 1 -- 172.21.15.27:0/3598231694 shutdown_connections 2024-04-25T00:24:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.290+0000 7ff28f3a5740 1 -- 172.21.15.27:0/3598231694 wait complete. 2024-04-25T00:24:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.290+0000 7ff28f3a5740 1 -- start start 2024-04-25T00:24:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff28f3a5740 1 Processor -- start 2024-04-25T00:24:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff28f3a5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af710b580 0x557af7115e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff28f3a5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af710bea0 0x557af7113170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff28f3a5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af7116570 0x557af71136b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff28f3a5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557af7119300 con 0x557af710b580 2024-04-25T00:24:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff28f3a5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557af702c050 con 0x557af7116570 2024-04-25T00:24:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff28f3a5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557af7119480 con 0x557af710bea0 2024-04-25T00:24:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff28f3a5740 10 client.0 _ll_register_callbacks cb 0x557af6d83be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff289b52640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af710b580 0x557af7115e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff289b52640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af710b580 0x557af7115e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45004/0 (socket says 172.21.15.27:45004) 2024-04-25T00:24:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff289b52640 1 -- 172.21.15.27:0/3980613180 learned_addr learned my addr 172.21.15.27:0/3980613180 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff28a353640 1 --2- 172.21.15.27:0/3980613180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af710bea0 0x557af7113170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff289b52640 1 -- 172.21.15.27:0/3980613180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af710bea0 msgr2=0x557af7113170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff289b52640 1 --2- 172.21.15.27:0/3980613180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af710bea0 0x557af7113170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff289b52640 1 -- 172.21.15.27:0/3980613180 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af7116570 msgr2=0x557af71136b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff289b52640 1 --2- 172.21.15.27:0/3980613180 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af7116570 0x557af71136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff289b52640 1 -- 172.21.15.27:0/3980613180 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557af6ffddc0 con 0x557af710b580 2024-04-25T00:24:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff28a353640 1 --2- 172.21.15.27:0/3980613180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af710bea0 0x557af7113170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff289b52640 1 --2- 172.21.15.27:0/3980613180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af710b580 0x557af7115e80 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ff27400f730 tx=0x7ff27400fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c0faedc230549a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.291+0000 7ff26dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.292+0000 7ff26dffb640 1 -- 172.21.15.27:0/3980613180 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2740146d0 con 0x557af710b580 2024-04-25T00:24:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.292+0000 7ff26dffb640 1 -- 172.21.15.27:0/3980613180 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7ff274005230 con 0x557af710b580 2024-04-25T00:24:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.292+0000 7ff28f3a5740 1 -- 172.21.15.27:0/3980613180 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff274005500 con 0x557af710b580 2024-04-25T00:24:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.292+0000 7ff26dffb640 1 -- 172.21.15.27:0/3980613180 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7ff274030050 con 0x557af710b580 2024-04-25T00:24:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.292+0000 7ff28f3a5740 10 client.17055 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.292+0000 7ff28f3a5740 1 -- 172.21.15.27:0/3980613180 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x557af6d843f0 con 0x557af710b580 2024-04-25T00:24:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.292+0000 7ff26dffb640 1 -- 172.21.15.27:0/3980613180 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff274005500 con 0x557af710b580 2024-04-25T00:24:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.292+0000 7ff26dffb640 1 -- 172.21.15.27:0/3980613180 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7ff27401f710 con 0x557af710b580 2024-04-25T00:24:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.292+0000 7ff28f3a5740 10 client.17055 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.292+0000 7ff28f3a5740 10 client.17055 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:24:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.292+0000 7ff28f3a5740 20 client.17055 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.292+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/3980613180 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x557af7126490 0x557af7128880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.292+0000 7ff28f3a5740 4 client.17055 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:24:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.292+0000 7ff28f3a5740 1 -- 172.21.15.27:0/3980613180 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557af707dbb0 con 0x557af7126490 2024-04-25T00:24:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.292+0000 7ff28a353640 1 --2- 172.21.15.27:0/3980613180 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x557af7126490 0x557af7128880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.293+0000 7ff28a353640 1 --2- 172.21.15.27:0/3980613180 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x557af7126490 0x557af7128880 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=31701dff29c2deb8 server_cookie=64ab9775e81c97f8 in_seq=0 out_seq=0 2024-04-25T00:24:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.293+0000 7ff26dffb640 10 client.17055 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:24:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.293+0000 7ff26dffb640 1 -- 172.21.15.27:0/3980613180 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557af707dbb0 con 0x557af7126490 2024-04-25T00:24:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.293+0000 7ff26dffb640 10 client.17055 handle_command_reply: tid=0 2024-04-25T00:24:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.294+0000 7ff28f3a5740 1 client.17055 shutdown 2024-04-25T00:24:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.294+0000 7ff28f3a5740 1 -- 172.21.15.27:0/3980613180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af710b580 msgr2=0x557af7115e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.294+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/3980613180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af710b580 0x557af7115e80 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ff27400f730 tx=0x7ff27400fa30 comp rx=0 tx=0).stop 2024-04-25T00:24:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.294+0000 7ff28f3a5740 1 -- 172.21.15.27:0/3980613180 shutdown_connections 2024-04-25T00:24:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.294+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/3980613180 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af7116570 0x557af71136b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.294+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/3980613180 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x557af7126490 0x557af7128880 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.294+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/3980613180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af710bea0 0x557af7113170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.294+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/3980613180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af710b580 0x557af7115e80 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.294+0000 7ff28f3a5740 1 -- 172.21.15.27:0/3980613180 >> 172.21.15.27:0/3980613180 conn(0x557af71078a0 msgr2=0x557af710d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.294+0000 7ff26dffb640 0 client.17055 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:24:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.294+0000 7ff28f3a5740 1 -- 172.21.15.27:0/3980613180 shutdown_connections 2024-04-25T00:24:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.294+0000 7ff28f3a5740 1 -- 172.21.15.27:0/3980613180 wait complete. 2024-04-25T00:24:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.294+0000 7ff28f3a5740 20 client.17055 trim_cache size 0 max 16384 2024-04-25T00:24:58.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff28f3a5740 1 Processor -- start 2024-04-25T00:24:58.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff28f3a5740 1 -- start start 2024-04-25T00:24:58.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff28f3a5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af710b580 0x557af71221f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff28f3a5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af710bea0 0x557af71069c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff28f3a5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af7116570 0x557af712df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff28f3a5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff2841aeb50 con 0x557af7116570 2024-04-25T00:24:58.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff28f3a5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff2841b06f0 con 0x557af710bea0 2024-04-25T00:24:58.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff28f3a5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff2841af220 con 0x557af710b580 2024-04-25T00:24:58.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff289b52640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af7116570 0x557af712df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff289b52640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af7116570 0x557af712df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45016/0 (socket says 172.21.15.27:45016) 2024-04-25T00:24:58.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff289b52640 1 -- 172.21.15.27:0/2974854352 learned_addr learned my addr 172.21.15.27:0/2974854352 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff289b52640 1 --2- 172.21.15.27:0/2974854352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af710b580 0x557af71221f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff289b52640 1 -- 172.21.15.27:0/2974854352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af710b580 msgr2=0x557af71221f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff289b52640 1 --2- 172.21.15.27:0/2974854352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af710b580 0x557af71221f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff289b52640 1 -- 172.21.15.27:0/2974854352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af710bea0 msgr2=0x557af71069c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff289b52640 1 --2- 172.21.15.27:0/2974854352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af710bea0 0x557af71069c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff289b52640 1 -- 172.21.15.27:0/2974854352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557af6e4e7d0 con 0x557af7116570 2024-04-25T00:24:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff289b52640 1 --2- 172.21.15.27:0/2974854352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af7116570 0x557af712df00 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ff274005680 tx=0x7ff27400fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8ae5bddbb6c44dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff26ffff640 1 -- 172.21.15.27:0/2974854352 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2740052e0 con 0x557af7116570 2024-04-25T00:24:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff26ffff640 1 -- 172.21.15.27:0/2974854352 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff274023030 con 0x557af7116570 2024-04-25T00:24:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.308+0000 7ff26ffff640 1 -- 172.21.15.27:0/2974854352 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27402a0d0 con 0x557af7116570 2024-04-25T00:24:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 1 -- 172.21.15.27:0/2974854352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af7116570 msgr2=0x557af712df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/2974854352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af7116570 0x557af712df00 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ff274005680 tx=0x7ff27400fbb0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 1 -- 172.21.15.27:0/2974854352 shutdown_connections 2024-04-25T00:24:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/2974854352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af7116570 0x557af712df00 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/2974854352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af710bea0 0x557af71069c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/2974854352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af710b580 0x557af71221f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 1 -- 172.21.15.27:0/2974854352 >> 172.21.15.27:0/2974854352 conn(0x557af71078a0 msgr2=0x557af71062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 1 -- 172.21.15.27:0/2974854352 shutdown_connections 2024-04-25T00:24:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 1 -- 172.21.15.27:0/2974854352 wait complete. 2024-04-25T00:24:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 1 -- start start 2024-04-25T00:24:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 1 Processor -- start 2024-04-25T00:24:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af710b580 0x557af711e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af710bea0 0x557af711ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af7116570 0x557af711bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff2841b0020 con 0x557af710b580 2024-04-25T00:24:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff2841af920 con 0x557af710bea0 2024-04-25T00:24:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff2841acae0 con 0x557af7116570 2024-04-25T00:24:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.309+0000 7ff28f3a5740 10 client.0 _ll_register_callbacks cb 0x7ff274014470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.310+0000 7ff289b52640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af710b580 0x557af711e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.310+0000 7ff289b52640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af710b580 0x557af711e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45020/0 (socket says 172.21.15.27:45020) 2024-04-25T00:24:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.310+0000 7ff289b52640 1 -- 172.21.15.27:0/1476480643 learned_addr learned my addr 172.21.15.27:0/1476480643 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.310+0000 7ff289b52640 1 -- 172.21.15.27:0/1476480643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af7116570 msgr2=0x557af711bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.310+0000 7ff289b52640 1 --2- 172.21.15.27:0/1476480643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af7116570 0x557af711bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.310+0000 7ff289b52640 1 -- 172.21.15.27:0/1476480643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af710bea0 msgr2=0x557af711ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.310+0000 7ff289b52640 1 --2- 172.21.15.27:0/1476480643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af710bea0 0x557af711ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.310+0000 7ff289b52640 1 -- 172.21.15.27:0/1476480643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557af711c380 con 0x557af710b580 2024-04-25T00:24:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.310+0000 7ff289b52640 1 --2- 172.21.15.27:0/1476480643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af710b580 0x557af711e6a0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7ff27400f730 tx=0x7ff274005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80e6cad15cabafe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.310+0000 7ff26dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.310+0000 7ff26dffb640 1 -- 172.21.15.27:0/1476480643 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2740052e0 con 0x557af710b580 2024-04-25T00:24:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.310+0000 7ff26dffb640 1 -- 172.21.15.27:0/1476480643 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7ff27402a070 con 0x557af710b580 2024-04-25T00:24:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.310+0000 7ff28f3a5740 1 -- 172.21.15.27:0/1476480643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff274023030 con 0x557af710b580 2024-04-25T00:24:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.310+0000 7ff26dffb640 1 -- 172.21.15.27:0/1476480643 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7ff274047050 con 0x557af710b580 2024-04-25T00:24:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.310+0000 7ff28f3a5740 10 client.17061 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.310+0000 7ff28f3a5740 1 -- 172.21.15.27:0/1476480643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x557af718c380 con 0x557af710b580 2024-04-25T00:24:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.311+0000 7ff26dffb640 1 -- 172.21.15.27:0/1476480643 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff274023030 con 0x557af710b580 2024-04-25T00:24:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.311+0000 7ff26dffb640 1 -- 172.21.15.27:0/1476480643 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7ff274004780 con 0x557af710b580 2024-04-25T00:24:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.311+0000 7ff28f3a5740 10 client.17061 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:58.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.311+0000 7ff28f3a5740 10 client.17061 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:24:58.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.311+0000 7ff28f3a5740 20 client.17061 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:58.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.311+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/1476480643 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x557af718c9e0 0x557af718edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.311+0000 7ff28f3a5740 4 client.17061 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:24:58.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.311+0000 7ff28f3a5740 1 -- 172.21.15.27:0/1476480643 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557af7113a40 con 0x557af718c9e0 2024-04-25T00:24:58.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.311+0000 7ff28a353640 1 --2- 172.21.15.27:0/1476480643 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x557af718c9e0 0x557af718edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.312+0000 7ff28a353640 1 --2- 172.21.15.27:0/1476480643 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x557af718c9e0 0x557af718edd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=138c758d06c17776 server_cookie=72112b92c2448ebe in_seq=0 out_seq=0 2024-04-25T00:24:58.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.312+0000 7ff26dffb640 10 client.17061 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:24:58.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.312+0000 7ff26dffb640 1 -- 172.21.15.27:0/1476480643 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557af7113a40 con 0x557af718c9e0 2024-04-25T00:24:58.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.312+0000 7ff26dffb640 10 client.17061 handle_command_reply: tid=0 2024-04-25T00:24:58.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.312+0000 7ff28f3a5740 1 client.17061 shutdown 2024-04-25T00:24:58.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.312+0000 7ff28f3a5740 1 -- 172.21.15.27:0/1476480643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af710b580 msgr2=0x557af711e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.312+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/1476480643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af710b580 0x557af711e6a0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7ff27400f730 tx=0x7ff274005720 comp rx=0 tx=0).stop 2024-04-25T00:24:58.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.312+0000 7ff28f3a5740 1 -- 172.21.15.27:0/1476480643 shutdown_connections 2024-04-25T00:24:58.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.312+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/1476480643 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x557af718c9e0 0x557af718edd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.312+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/1476480643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557af7116570 0x557af711bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.312+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/1476480643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557af710bea0 0x557af711ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.312+0000 7ff26dffb640 0 client.17061 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:24:58.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.312+0000 7ff28f3a5740 1 --2- 172.21.15.27:0/1476480643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557af710b580 0x557af711e6a0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.312+0000 7ff28f3a5740 1 -- 172.21.15.27:0/1476480643 >> 172.21.15.27:0/1476480643 conn(0x557af71078a0 msgr2=0x557af7111600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.312+0000 7ff28f3a5740 1 -- 172.21.15.27:0/1476480643 shutdown_connections 2024-04-25T00:24:58.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.312+0000 7ff28f3a5740 1 -- 172.21.15.27:0/1476480643 wait complete. 2024-04-25T00:24:58.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.312+0000 7ff28f3a5740 20 client.17061 trim_cache size 0 max 16384 2024-04-25T00:24:58.319 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:24:58.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.313+0000 7ff28c5b7640 1 -- 172.21.15.27:0/1144398385 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff2700a2d00 msgr2=0x7ff2700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.313+0000 7ff28c5b7640 1 --2- 172.21.15.27:0/1144398385 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff2700a2d00 0x7ff2700a51c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0bcee0 tx=0x7ff27c0b2670 comp rx=0 tx=0).stop 2024-04-25T00:24:58.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.313+0000 7ff28c5b7640 1 -- 172.21.15.27:0/1144398385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2841066b0 msgr2=0x7ff28419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.313+0000 7ff28c5b7640 1 --2- 172.21.15.27:0/1144398385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2841066b0 0x7ff28419c820 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7ff27c005cc0 tx=0x7ff27c003970 comp rx=0 tx=0).stop 2024-04-25T00:24:58.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.313+0000 7ff28c5b7640 1 -- 172.21.15.27:0/1144398385 shutdown_connections 2024-04-25T00:24:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.313+0000 7ff28c5b7640 1 --2- 172.21.15.27:0/1144398385 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7ff2700a2d00 0x7ff2700a51c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.313+0000 7ff28c5b7640 1 --2- 172.21.15.27:0/1144398385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2841066b0 0x7ff28419c820 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.313+0000 7ff28c5b7640 1 --2- 172.21.15.27:0/1144398385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff284100fd0 0x7ff2841984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.313+0000 7ff28c5b7640 1 --2- 172.21.15.27:0/1144398385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff284100610 0x7ff284197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.313+0000 7ff28c5b7640 1 -- 172.21.15.27:0/1144398385 >> 172.21.15.27:0/1144398385 conn(0x7ff284077740 msgr2=0x7ff2840ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.313+0000 7ff28c5b7640 1 -- 172.21.15.27:0/1144398385 shutdown_connections 2024-04-25T00:24:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.313+0000 7ff28c5b7640 1 -- 172.21.15.27:0/1144398385 wait complete. 2024-04-25T00:24:58.324 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T00:24:58.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.426+0000 7fa0372a5640 1 -- 172.21.15.27:0/196441455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa030100640 msgr2=0x7fa03010e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.426+0000 7fa0372a5640 1 --2- 172.21.15.27:0/196441455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa030100640 0x7fa03010e660 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa028009cd0 tx=0x7fa02802a8e0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.427+0000 7fa0372a5640 1 -- 172.21.15.27:0/196441455 shutdown_connections 2024-04-25T00:24:58.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.427+0000 7fa0372a5640 1 --2- 172.21.15.27:0/196441455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa030100640 0x7fa03010e660 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.427+0000 7fa0372a5640 1 --2- 172.21.15.27:0/196441455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0300ffc80 0x7fa030100100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.427+0000 7fa0372a5640 1 --2- 172.21.15.27:0/196441455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa0301077a0 0x7fa030107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.427+0000 7fa0372a5640 1 -- 172.21.15.27:0/196441455 >> 172.21.15.27:0/196441455 conn(0x7fa030076c00 msgr2=0x7fa0300fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.427+0000 7fa0372a5640 1 -- 172.21.15.27:0/196441455 shutdown_connections 2024-04-25T00:24:58.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.427+0000 7fa0372a5640 1 -- 172.21.15.27:0/196441455 wait complete. 2024-04-25T00:24:58.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.427+0000 7fa0372a5640 1 Processor -- start 2024-04-25T00:24:58.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.427+0000 7fa0372a5640 1 -- start start 2024-04-25T00:24:58.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.427+0000 7fa0372a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0300ffc80 0x7fa030198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.427+0000 7fa0372a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa030100640 0x7fa0301986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.427+0000 7fa0372a5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa0301077a0 0x7fa03019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa0372a5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa030111c80 con 0x7fa0300ffc80 2024-04-25T00:24:58.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa0372a5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa030111b00 con 0x7fa0301077a0 2024-04-25T00:24:58.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa0372a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa030111e00 con 0x7fa030100640 2024-04-25T00:24:58.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa035041640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0300ffc80 0x7fa030198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa035041640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0300ffc80 0x7fa030198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45024/0 (socket says 172.21.15.27:45024) 2024-04-25T00:24:58.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa035041640 1 -- 172.21.15.27:0/1070889589 learned_addr learned my addr 172.21.15.27:0/1070889589 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa034840640 1 --2- 172.21.15.27:0/1070889589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa030100640 0x7fa0301986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa035842640 1 --2- 172.21.15.27:0/1070889589 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa0301077a0 0x7fa03019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa035041640 1 -- 172.21.15.27:0/1070889589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa030100640 msgr2=0x7fa0301986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa035041640 1 --2- 172.21.15.27:0/1070889589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa030100640 0x7fa0301986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa035041640 1 -- 172.21.15.27:0/1070889589 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa0301077a0 msgr2=0x7fa03019ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa035041640 1 --2- 172.21.15.27:0/1070889589 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa0301077a0 0x7fa03019ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa035041640 1 -- 172.21.15.27:0/1070889589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa03019d170 con 0x7fa0300ffc80 2024-04-25T00:24:58.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa035842640 1 --2- 172.21.15.27:0/1070889589 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa0301077a0 0x7fa03019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:58.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa035041640 1 --2- 172.21.15.27:0/1070889589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0300ffc80 0x7fa030198180 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00b4a0 tx=0x7fa02c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa0267fc640 1 -- 172.21.15.27:0/1070889589 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c014070 con 0x7fa0300ffc80 2024-04-25T00:24:58.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa0267fc640 1 -- 172.21.15.27:0/1070889589 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02c010af0 con 0x7fa0300ffc80 2024-04-25T00:24:58.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa0372a5640 1 -- 172.21.15.27:0/1070889589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa03019d460 con 0x7fa0300ffc80 2024-04-25T00:24:58.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa0267fc640 1 -- 172.21.15.27:0/1070889589 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c017d10 con 0x7fa0300ffc80 2024-04-25T00:24:58.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.428+0000 7fa0372a5640 1 -- 172.21.15.27:0/1070889589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0301a4cf0 con 0x7fa0300ffc80 2024-04-25T00:24:58.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.430+0000 7fa0267fc640 1 -- 172.21.15.27:0/1070889589 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa02c017070 con 0x7fa0300ffc80 2024-04-25T00:24:58.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.431+0000 7fa039fb4740 1 Processor -- start 2024-04-25T00:24:58.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.431+0000 7fa039fb4740 1 -- start start 2024-04-25T00:24:58.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.431+0000 7fa039fb4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961580 0x55dd30961960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.431+0000 7fa0267fc640 1 --2- 172.21.15.27:0/1070889589 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa01c0a2d00 0x7fa01c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.431+0000 7fa039fb4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd30961ea0 0x55dd3096c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.431+0000 7fa039fb4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3096c570 0x55dd3096e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.431+0000 7fa039fb4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55dd308c6670 con 0x55dd30961580 2024-04-25T00:24:58.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.431+0000 7fa039fb4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55dd3086e970 con 0x55dd30961ea0 2024-04-25T00:24:58.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.431+0000 7fa039fb4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dd308c2630 con 0x55dd3096c570 2024-04-25T00:24:58.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.431+0000 7fa035041640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961580 0x55dd30961960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.431+0000 7fa035842640 1 --2- 172.21.15.27:0/1070889589 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa01c0a2d00 0x7fa01c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.431+0000 7fa035041640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961580 0x55dd30961960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45030/0 (socket says 172.21.15.27:45030) 2024-04-25T00:24:58.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.432+0000 7fa035041640 1 -- 172.21.15.27:0/4032670497 learned_addr learned my addr 172.21.15.27:0/4032670497 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.431+0000 7fa0267fc640 1 -- 172.21.15.27:0/1070889589 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fa02c0cead0 con 0x7fa0300ffc80 2024-04-25T00:24:58.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.432+0000 7fa034840640 1 --2- 172.21.15.27:0/4032670497 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd30961ea0 0x55dd3096c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.432+0000 7fa035842640 1 --2- 172.21.15.27:0/4032670497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3096c570 0x55dd3096e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.432+0000 7fa035041640 1 -- 172.21.15.27:0/4032670497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3096c570 msgr2=0x55dd3096e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.432+0000 7fa035041640 1 --2- 172.21.15.27:0/4032670497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3096c570 0x55dd3096e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.432+0000 7fa035041640 1 -- 172.21.15.27:0/4032670497 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd30961ea0 msgr2=0x55dd3096c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.432+0000 7fa035041640 1 --2- 172.21.15.27:0/4032670497 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd30961ea0 0x55dd3096c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.432+0000 7fa035041640 1 -- 172.21.15.27:0/4032670497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd30892400 con 0x55dd30961580 2024-04-25T00:24:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.432+0000 7fa035842640 1 --2- 172.21.15.27:0/4032670497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3096c570 0x55dd3096e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.432+0000 7fa035041640 1 --2- 172.21.15.27:0/4032670497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961580 0x55dd30961960 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fa02c097760 tx=0x7fa02c0047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a2e1785875df32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.432+0000 7fa035842640 1 --2- 172.21.15.27:0/1070889589 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa01c0a2d00 0x7fa01c0a51c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55dd3095ea30 tx=0x7fa028031590 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.432+0000 7fa01b7fe640 1 -- 172.21.15.27:0/4032670497 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c01e050 con 0x55dd30961580 2024-04-25T00:24:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.432+0000 7fa01b7fe640 1 -- 172.21.15.27:0/4032670497 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02c09ad90 con 0x55dd30961580 2024-04-25T00:24:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.432+0000 7fa01b7fe640 1 -- 172.21.15.27:0/4032670497 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c0cc130 con 0x55dd30961580 2024-04-25T00:24:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.433+0000 7fa039fb4740 1 -- 172.21.15.27:0/4032670497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961580 msgr2=0x55dd30961960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.433+0000 7fa039fb4740 1 --2- 172.21.15.27:0/4032670497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961580 0x55dd30961960 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fa02c097760 tx=0x7fa02c0047d0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.433+0000 7fa039fb4740 1 -- 172.21.15.27:0/4032670497 shutdown_connections 2024-04-25T00:24:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.433+0000 7fa039fb4740 1 --2- 172.21.15.27:0/4032670497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd3096c570 0x55dd3096e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.433+0000 7fa039fb4740 1 --2- 172.21.15.27:0/4032670497 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd30961ea0 0x55dd3096c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.433+0000 7fa039fb4740 1 --2- 172.21.15.27:0/4032670497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961580 0x55dd30961960 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.433+0000 7fa039fb4740 1 -- 172.21.15.27:0/4032670497 >> 172.21.15.27:0/4032670497 conn(0x55dd3095d8a0 msgr2=0x55dd3095dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.433+0000 7fa039fb4740 1 -- 172.21.15.27:0/4032670497 shutdown_connections 2024-04-25T00:24:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.433+0000 7fa039fb4740 1 -- 172.21.15.27:0/4032670497 wait complete. 2024-04-25T00:24:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.433+0000 7fa039fb4740 1 -- start start 2024-04-25T00:24:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.433+0000 7fa039fb4740 1 Processor -- start 2024-04-25T00:24:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa039fb4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd30961580 0x55dd3096beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa039fb4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd30961ea0 0x55dd309691a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa039fb4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd3096c570 0x55dd309696e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa039fb4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55dd3096f300 con 0x55dd3096c570 2024-04-25T00:24:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa039fb4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55dd30882050 con 0x55dd30961580 2024-04-25T00:24:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa039fb4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dd3096f480 con 0x55dd30961ea0 2024-04-25T00:24:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa039fb4740 10 client.0 _ll_register_callbacks cb 0x55dd305d9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa035842640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd3096c570 0x55dd309696e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa035842640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd3096c570 0x55dd309696e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45038/0 (socket says 172.21.15.27:45038) 2024-04-25T00:24:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa035842640 1 -- 172.21.15.27:0/2367395905 learned_addr learned my addr 172.21.15.27:0/2367395905 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa035041640 1 --2- 172.21.15.27:0/2367395905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd30961580 0x55dd3096beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa034840640 1 --2- 172.21.15.27:0/2367395905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd30961ea0 0x55dd309691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa035842640 1 -- 172.21.15.27:0/2367395905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd30961ea0 msgr2=0x55dd309691a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa035842640 1 --2- 172.21.15.27:0/2367395905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd30961ea0 0x55dd309691a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa035842640 1 -- 172.21.15.27:0/2367395905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd30961580 msgr2=0x55dd3096beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa035842640 1 --2- 172.21.15.27:0/2367395905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd30961580 0x55dd3096beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa035842640 1 -- 172.21.15.27:0/2367395905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd30853dc0 con 0x55dd3096c570 2024-04-25T00:24:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa035842640 1 --2- 172.21.15.27:0/2367395905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd3096c570 0x55dd309696e0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fa028003c20 tx=0x7fa0280095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590f91cd361dafdb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa018ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.434+0000 7fa018ff9640 1 -- 172.21.15.27:0/2367395905 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02804f070 con 0x55dd3096c570 2024-04-25T00:24:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.435+0000 7fa018ff9640 1 -- 172.21.15.27:0/2367395905 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fa0280307a0 con 0x55dd3096c570 2024-04-25T00:24:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.435+0000 7fa039fb4740 1 -- 172.21.15.27:0/2367395905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa02c09ad90 con 0x55dd3096c570 2024-04-25T00:24:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.435+0000 7fa018ff9640 1 -- 172.21.15.27:0/2367395905 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fa028003e60 con 0x55dd3096c570 2024-04-25T00:24:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.435+0000 7fa039fb4740 10 client.17073 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.435+0000 7fa039fb4740 1 -- 172.21.15.27:0/2367395905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55dd305da3f0 con 0x55dd3096c570 2024-04-25T00:24:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.435+0000 7fa018ff9640 1 -- 172.21.15.27:0/2367395905 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02c09ad90 con 0x55dd3096c570 2024-04-25T00:24:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.435+0000 7fa018ff9640 1 -- 172.21.15.27:0/2367395905 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7fa02804a070 con 0x55dd3096c570 2024-04-25T00:24:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.435+0000 7fa039fb4740 10 client.17073 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.435+0000 7fa039fb4740 10 client.17073 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:24:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.435+0000 7fa039fb4740 20 client.17073 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.435+0000 7fa039fb4740 1 --2- 172.21.15.27:0/2367395905 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55dd3097c450 0x55dd3097e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.435+0000 7fa039fb4740 4 client.17073 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:24:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.435+0000 7fa039fb4740 1 -- 172.21.15.27:0/2367395905 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd308d3bb0 con 0x55dd3097c450 2024-04-25T00:24:58.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.438+0000 7fa035041640 1 --2- 172.21.15.27:0/2367395905 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55dd3097c450 0x55dd3097e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.440+0000 7fa018ff9640 10 client.17073 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.440+0000 7fa035041640 1 --2- 172.21.15.27:0/2367395905 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55dd3097c450 0x55dd3097e8b0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bf7dada8aa3c30f server_cookie=d75696670e019c25 in_seq=0 out_seq=0 2024-04-25T00:24:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.441+0000 7fa018ff9640 1 -- 172.21.15.27:0/2367395905 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dd308d3bb0 con 0x55dd3097c450 2024-04-25T00:24:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.441+0000 7fa018ff9640 10 client.17073 handle_command_reply: tid=0 2024-04-25T00:24:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.441+0000 7fa039fb4740 1 client.17073 shutdown 2024-04-25T00:24:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.441+0000 7fa039fb4740 1 -- 172.21.15.27:0/2367395905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd3096c570 msgr2=0x55dd309696e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.441+0000 7fa039fb4740 1 --2- 172.21.15.27:0/2367395905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd3096c570 0x55dd309696e0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fa028003c20 tx=0x7fa0280095c0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.441+0000 7fa039fb4740 1 -- 172.21.15.27:0/2367395905 shutdown_connections 2024-04-25T00:24:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.441+0000 7fa039fb4740 1 --2- 172.21.15.27:0/2367395905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd3096c570 0x55dd309696e0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.441+0000 7fa039fb4740 1 --2- 172.21.15.27:0/2367395905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd30961ea0 0x55dd309691a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.441+0000 7fa039fb4740 1 --2- 172.21.15.27:0/2367395905 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55dd3097c450 0x55dd3097e8b0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.441+0000 7fa039fb4740 1 --2- 172.21.15.27:0/2367395905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd30961580 0x55dd3096beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.441+0000 7fa018ff9640 0 client.17073 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.441+0000 7fa039fb4740 1 -- 172.21.15.27:0/2367395905 >> 172.21.15.27:0/2367395905 conn(0x55dd3095d8a0 msgr2=0x55dd3095cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.441+0000 7fa039fb4740 1 -- 172.21.15.27:0/2367395905 shutdown_connections 2024-04-25T00:24:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.441+0000 7fa039fb4740 1 -- 172.21.15.27:0/2367395905 wait complete. 2024-04-25T00:24:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.441+0000 7fa039fb4740 20 client.17073 trim_cache size 0 max 16384 2024-04-25T00:24:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.455+0000 7fa039fb4740 1 Processor -- start 2024-04-25T00:24:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.455+0000 7fa039fb4740 1 -- start start 2024-04-25T00:24:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.455+0000 7fa039fb4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd30961580 0x55dd309781c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.455+0000 7fa039fb4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961ea0 0x55dd3096b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.455+0000 7fa039fb4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd3096c570 0x55dd30984180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.455+0000 7fa039fb4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa0301aedd0 con 0x55dd30961ea0 2024-04-25T00:24:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.455+0000 7fa039fb4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa0301b0970 con 0x55dd3096c570 2024-04-25T00:24:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.455+0000 7fa039fb4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa0301af4a0 con 0x55dd30961580 2024-04-25T00:24:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.455+0000 7fa034840640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961ea0 0x55dd3096b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.455+0000 7fa034840640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961ea0 0x55dd3096b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45046/0 (socket says 172.21.15.27:45046) 2024-04-25T00:24:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.455+0000 7fa034840640 1 -- 172.21.15.27:0/534871026 learned_addr learned my addr 172.21.15.27:0/534871026 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.455+0000 7fa034840640 1 -- 172.21.15.27:0/534871026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd30961580 msgr2=0x55dd309781c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.455+0000 7fa034840640 1 --2- 172.21.15.27:0/534871026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd30961580 0x55dd309781c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.455+0000 7fa034840640 1 -- 172.21.15.27:0/534871026 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd3096c570 msgr2=0x55dd30984180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.455+0000 7fa034840640 1 --2- 172.21.15.27:0/534871026 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd3096c570 0x55dd30984180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.455+0000 7fa034840640 1 -- 172.21.15.27:0/534871026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd306a47d0 con 0x55dd30961ea0 2024-04-25T00:24:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.456+0000 7fa034840640 1 --2- 172.21.15.27:0/534871026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961ea0 0x55dd3096b9e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fa02000b790 tx=0x7fa02000bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a1ab3c275eff62 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.456+0000 7fa01affd640 1 -- 172.21.15.27:0/534871026 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02001b070 con 0x55dd30961ea0 2024-04-25T00:24:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.456+0000 7fa01affd640 1 -- 172.21.15.27:0/534871026 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa020015450 con 0x55dd30961ea0 2024-04-25T00:24:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.456+0000 7fa01affd640 1 -- 172.21.15.27:0/534871026 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa020014610 con 0x55dd30961ea0 2024-04-25T00:24:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.456+0000 7fa039fb4740 1 -- 172.21.15.27:0/534871026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961ea0 msgr2=0x55dd3096b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.456+0000 7fa039fb4740 1 --2- 172.21.15.27:0/534871026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961ea0 0x55dd3096b9e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fa02000b790 tx=0x7fa02000bc50 comp rx=0 tx=0).stop 2024-04-25T00:24:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.456+0000 7fa039fb4740 1 -- 172.21.15.27:0/534871026 shutdown_connections 2024-04-25T00:24:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.456+0000 7fa039fb4740 1 --2- 172.21.15.27:0/534871026 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd3096c570 0x55dd30984180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.456+0000 7fa039fb4740 1 --2- 172.21.15.27:0/534871026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961ea0 0x55dd3096b9e0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.456+0000 7fa039fb4740 1 --2- 172.21.15.27:0/534871026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd30961580 0x55dd309781c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.456+0000 7fa039fb4740 1 -- 172.21.15.27:0/534871026 >> 172.21.15.27:0/534871026 conn(0x55dd3095d8a0 msgr2=0x55dd309646b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.456+0000 7fa039fb4740 1 -- 172.21.15.27:0/534871026 shutdown_connections 2024-04-25T00:24:58.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.456+0000 7fa039fb4740 1 -- 172.21.15.27:0/534871026 wait complete. 2024-04-25T00:24:58.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.457+0000 7fa039fb4740 1 -- start start 2024-04-25T00:24:58.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.457+0000 7fa039fb4740 1 Processor -- start 2024-04-25T00:24:58.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.457+0000 7fa039fb4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961580 0x55dd309704a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.457+0000 7fa039fb4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd30961ea0 0x55dd309709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.457+0000 7fa039fb4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd3096c570 0x55dd309e19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.457+0000 7fa039fb4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa0301b02a0 con 0x55dd30961580 2024-04-25T00:24:58.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.457+0000 7fa039fb4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa0301afba0 con 0x55dd3096c570 2024-04-25T00:24:58.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.457+0000 7fa039fb4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa0301acd60 con 0x55dd30961ea0 2024-04-25T00:24:58.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.457+0000 7fa039fb4740 10 client.0 _ll_register_callbacks cb 0x7fa028009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:58.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.457+0000 7fa035041640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961580 0x55dd309704a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.457+0000 7fa035041640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961580 0x55dd309704a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45058/0 (socket says 172.21.15.27:45058) 2024-04-25T00:24:58.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.457+0000 7fa035041640 1 -- 172.21.15.27:0/3579215357 learned_addr learned my addr 172.21.15.27:0/3579215357 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:58.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa034840640 1 --2- 172.21.15.27:0/3579215357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd30961ea0 0x55dd309709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa035842640 1 --2- 172.21.15.27:0/3579215357 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd3096c570 0x55dd309e19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa035041640 1 -- 172.21.15.27:0/3579215357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd30961ea0 msgr2=0x55dd309709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa035041640 1 --2- 172.21.15.27:0/3579215357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd30961ea0 0x55dd309709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa035041640 1 -- 172.21.15.27:0/3579215357 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd3096c570 msgr2=0x55dd309e19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:58.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa035041640 1 --2- 172.21.15.27:0/3579215357 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd3096c570 0x55dd309e19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa035041640 1 -- 172.21.15.27:0/3579215357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd309e1f30 con 0x55dd30961580 2024-04-25T00:24:58.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa034840640 1 --2- 172.21.15.27:0/3579215357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd30961ea0 0x55dd309709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:58.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa035842640 1 --2- 172.21.15.27:0/3579215357 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd3096c570 0x55dd309e19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:58.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa035041640 1 --2- 172.21.15.27:0/3579215357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961580 0x55dd309704a0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fa02c09a860 tx=0x7fa02c096ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37cf912f594c02c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:58.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa018ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:58.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa018ff9640 1 -- 172.21.15.27:0/3579215357 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c01e040 con 0x55dd30961580 2024-04-25T00:24:58.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa018ff9640 1 -- 172.21.15.27:0/3579215357 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fa02c0cc130 con 0x55dd30961580 2024-04-25T00:24:58.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa039fb4740 1 -- 172.21.15.27:0/3579215357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa020015450 con 0x55dd30961580 2024-04-25T00:24:58.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa018ff9640 1 -- 172.21.15.27:0/3579215357 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fa02c0cc330 con 0x55dd30961580 2024-04-25T00:24:58.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa039fb4740 10 client.17079 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:58.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa039fb4740 1 -- 172.21.15.27:0/3579215357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55dd309e3090 con 0x55dd30961580 2024-04-25T00:24:58.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa018ff9640 1 -- 172.21.15.27:0/3579215357 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa020015450 con 0x55dd30961580 2024-04-25T00:24:58.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa018ff9640 1 -- 172.21.15.27:0/3579215357 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7fa02c083030 con 0x55dd30961580 2024-04-25T00:24:58.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa039fb4740 10 client.17079 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:58.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa039fb4740 10 client.17079 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:24:58.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa039fb4740 20 client.17079 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:58.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.458+0000 7fa039fb4740 1 --2- 172.21.15.27:0/3579215357 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55dd309e33d0 0x55dd309e57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:58.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa039fb4740 4 client.17079 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:24:58.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa039fb4740 1 -- 172.21.15.27:0/3579215357 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dd308cdf00 con 0x55dd309e33d0 2024-04-25T00:24:58.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa034840640 1 --2- 172.21.15.27:0/3579215357 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55dd309e33d0 0x55dd309e57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:58.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa034840640 1 --2- 172.21.15.27:0/3579215357 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55dd309e33d0 0x55dd309e57c0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=970387147526efdf server_cookie=df6002ebb34311d6 in_seq=0 out_seq=0 2024-04-25T00:24:58.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa018ff9640 10 client.17079 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:58.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa018ff9640 1 -- 172.21.15.27:0/3579215357 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dd308cdf00 con 0x55dd309e33d0 2024-04-25T00:24:58.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa018ff9640 10 client.17079 handle_command_reply: tid=0 2024-04-25T00:24:58.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa039fb4740 1 client.17079 shutdown 2024-04-25T00:24:58.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa039fb4740 1 -- 172.21.15.27:0/3579215357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961580 msgr2=0x55dd309704a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa039fb4740 1 --2- 172.21.15.27:0/3579215357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961580 0x55dd309704a0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fa02c09a860 tx=0x7fa02c096ca0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa039fb4740 1 -- 172.21.15.27:0/3579215357 shutdown_connections 2024-04-25T00:24:58.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa039fb4740 1 --2- 172.21.15.27:0/3579215357 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55dd309e33d0 0x55dd309e57c0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa039fb4740 1 --2- 172.21.15.27:0/3579215357 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dd3096c570 0x55dd309e19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa018ff9640 0 client.17079 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:58.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa039fb4740 1 --2- 172.21.15.27:0/3579215357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dd30961ea0 0x55dd309709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa039fb4740 1 --2- 172.21.15.27:0/3579215357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd30961580 0x55dd309704a0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa039fb4740 1 -- 172.21.15.27:0/3579215357 >> 172.21.15.27:0/3579215357 conn(0x55dd3095d8a0 msgr2=0x55dd30962cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.459+0000 7fa039fb4740 1 -- 172.21.15.27:0/3579215357 shutdown_connections 2024-04-25T00:24:58.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.460+0000 7fa039fb4740 1 -- 172.21.15.27:0/3579215357 wait complete. 2024-04-25T00:24:58.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.460+0000 7fa039fb4740 20 client.17079 trim_cache size 0 max 16384 2024-04-25T00:24:58.467 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:24:58.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.460+0000 7fa0372a5640 1 -- 172.21.15.27:0/1070889589 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa01c0a2d00 msgr2=0x7fa01c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.460+0000 7fa0372a5640 1 --2- 172.21.15.27:0/1070889589 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa01c0a2d00 0x7fa01c0a51c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55dd3095ea30 tx=0x7fa028031590 comp rx=0 tx=0).stop 2024-04-25T00:24:58.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.460+0000 7fa0372a5640 1 -- 172.21.15.27:0/1070889589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0300ffc80 msgr2=0x7fa030198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:58.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.460+0000 7fa0372a5640 1 --2- 172.21.15.27:0/1070889589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0300ffc80 0x7fa030198180 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00b4a0 tx=0x7fa02c00b960 comp rx=0 tx=0).stop 2024-04-25T00:24:58.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.460+0000 7fa0372a5640 1 -- 172.21.15.27:0/1070889589 shutdown_connections 2024-04-25T00:24:58.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.460+0000 7fa0372a5640 1 --2- 172.21.15.27:0/1070889589 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa01c0a2d00 0x7fa01c0a51c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.460+0000 7fa0372a5640 1 --2- 172.21.15.27:0/1070889589 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa0301077a0 0x7fa03019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.460+0000 7fa0372a5640 1 --2- 172.21.15.27:0/1070889589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa030100640 0x7fa0301986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.460+0000 7fa0372a5640 1 --2- 172.21.15.27:0/1070889589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0300ffc80 0x7fa030198180 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:58.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.460+0000 7fa0372a5640 1 -- 172.21.15.27:0/1070889589 >> 172.21.15.27:0/1070889589 conn(0x7fa030076c00 msgr2=0x7fa0301045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:58.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.460+0000 7fa0372a5640 1 -- 172.21.15.27:0/1070889589 shutdown_connections 2024-04-25T00:24:58.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:58.460+0000 7fa0372a5640 1 -- 172.21.15.27:0/1070889589 wait complete. 2024-04-25T00:24:59.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:24:58 smithi027 ceph-mon[29208]: pgmap v279: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 9 op/s 2024-04-25T00:24:59.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:24:58 smithi139 ceph-mon[32833]: pgmap v279: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 9 op/s 2024-04-25T00:24:59.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:24:58 smithi153 ceph-mon[32896]: pgmap v279: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 9 op/s 2024-04-25T00:24:59.473 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T00:24:59.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:24:59.473 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T00:24:59.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.593+0000 7f30be22a640 1 -- 172.21.15.27:0/1421896649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b8069dc0 msgr2=0x7f30b806a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.593+0000 7f30be22a640 1 --2- 172.21.15.27:0/1421896649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b8069dc0 0x7f30b806a240 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f30ac009930 tx=0x7f30ac02a420 comp rx=0 tx=0).stop 2024-04-25T00:24:59.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.593+0000 7f30be22a640 1 -- 172.21.15.27:0/1421896649 shutdown_connections 2024-04-25T00:24:59.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.593+0000 7f30be22a640 1 --2- 172.21.15.27:0/1421896649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30b806a780 0x7f30b810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.593+0000 7f30be22a640 1 --2- 172.21.15.27:0/1421896649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b8069dc0 0x7f30b806a240 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.593+0000 7f30be22a640 1 --2- 172.21.15.27:0/1421896649 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30b8107730 0x7f30b8107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.593+0000 7f30be22a640 1 -- 172.21.15.27:0/1421896649 >> 172.21.15.27:0/1421896649 conn(0x7f30b80776a0 msgr2=0x7f30b8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:59.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.593+0000 7f30be22a640 1 -- 172.21.15.27:0/1421896649 shutdown_connections 2024-04-25T00:24:59.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.593+0000 7f30be22a640 1 -- 172.21.15.27:0/1421896649 wait complete. 2024-04-25T00:24:59.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.593+0000 7f30be22a640 1 Processor -- start 2024-04-25T00:24:59.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.594+0000 7f30be22a640 1 -- start start 2024-04-25T00:24:59.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.594+0000 7f30be22a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30b8069dc0 0x7f30b8198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.594+0000 7f30be22a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30b806a780 0x7f30b81985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.594+0000 7f30be22a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b8107730 0x7f30b819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.594+0000 7f30be22a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f30b8111b40 con 0x7f30b8107730 2024-04-25T00:24:59.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.594+0000 7f30be22a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f30b81119c0 con 0x7f30b8069dc0 2024-04-25T00:24:59.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.594+0000 7f30be22a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f30b8111cc0 con 0x7f30b806a780 2024-04-25T00:24:59.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.594+0000 7f30b7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b8107730 0x7f30b819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.594+0000 7f30b6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30b806a780 0x7f30b81985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.594+0000 7f30b7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b8107730 0x7f30b819c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45080/0 (socket says 172.21.15.27:45080) 2024-04-25T00:24:59.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.594+0000 7f30b77fe640 1 --2- 172.21.15.27:0/2757421679 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30b8069dc0 0x7f30b8198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.594+0000 7f30b7fff640 1 -- 172.21.15.27:0/2757421679 learned_addr learned my addr 172.21.15.27:0/2757421679 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:59.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.595+0000 7f30b6ffd640 1 -- 172.21.15.27:0/2757421679 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30b8069dc0 msgr2=0x7f30b8198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.595+0000 7f30b6ffd640 1 --2- 172.21.15.27:0/2757421679 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30b8069dc0 0x7f30b8198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.595+0000 7f30b6ffd640 1 -- 172.21.15.27:0/2757421679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b8107730 msgr2=0x7f30b819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.595+0000 7f30b6ffd640 1 --2- 172.21.15.27:0/2757421679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b8107730 0x7f30b819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.595+0000 7f30b6ffd640 1 -- 172.21.15.27:0/2757421679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30b819d000 con 0x7f30b806a780 2024-04-25T00:24:59.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.595+0000 7f30b7fff640 1 --2- 172.21.15.27:0/2757421679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b8107730 0x7f30b819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:24:59.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.595+0000 7f30b6ffd640 1 --2- 172.21.15.27:0/2757421679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30b806a780 0x7f30b81985a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f30ac005940 tx=0x7f30ac0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:59.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.595+0000 7f30b4ff9640 1 -- 172.21.15.27:0/2757421679 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac03f070 con 0x7f30b806a780 2024-04-25T00:24:59.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.595+0000 7f30be22a640 1 -- 172.21.15.27:0/2757421679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30b819d290 con 0x7f30b806a780 2024-04-25T00:24:59.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.595+0000 7f30b4ff9640 1 -- 172.21.15.27:0/2757421679 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30ac03b460 con 0x7f30b806a780 2024-04-25T00:24:59.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.595+0000 7f30b4ff9640 1 -- 172.21.15.27:0/2757421679 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac035070 con 0x7f30b806a780 2024-04-25T00:24:59.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.595+0000 7f30be22a640 1 -- 172.21.15.27:0/2757421679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30b81a4b80 con 0x7f30b806a780 2024-04-25T00:24:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30c1011740 1 Processor -- start 2024-04-25T00:24:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30c1011740 1 -- start start 2024-04-25T00:24:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30b4ff9640 1 -- 172.21.15.27:0/2757421679 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f30ac03b610 con 0x7f30b806a780 2024-04-25T00:24:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30c1011740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158237980 0x564158235cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30c1011740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158236290 0x564158236710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30c1011740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158241010 0x564158243400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30c1011740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56415819a2c0 con 0x564158241010 2024-04-25T00:24:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30c1011740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564158142910 con 0x564158236290 2024-04-25T00:24:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30c1011740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564158196280 con 0x564158237980 2024-04-25T00:24:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30b6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158241010 0x564158243400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30b6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158241010 0x564158243400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45094/0 (socket says 172.21.15.27:45094) 2024-04-25T00:24:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30b6ffd640 1 -- 172.21.15.27:0/230540131 learned_addr learned my addr 172.21.15.27:0/230540131 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30b7fff640 1 --2- 172.21.15.27:0/230540131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158237980 0x564158235cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30b77fe640 1 --2- 172.21.15.27:0/230540131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158236290 0x564158236710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30b6ffd640 1 -- 172.21.15.27:0/230540131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158237980 msgr2=0x564158235cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30b6ffd640 1 --2- 172.21.15.27:0/230540131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158237980 0x564158235cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30b6ffd640 1 -- 172.21.15.27:0/230540131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158236290 msgr2=0x564158236710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30b6ffd640 1 --2- 172.21.15.27:0/230540131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158236290 0x564158236710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.598+0000 7f30b6ffd640 1 -- 172.21.15.27:0/230540131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564158153fb0 con 0x564158241010 2024-04-25T00:24:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.599+0000 7f30b7fff640 1 --2- 172.21.15.27:0/230540131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158237980 0x564158235cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.599+0000 7f30b4ff9640 1 --2- 172.21.15.27:0/2757421679 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f30a00a2d00 0x7f30a00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.599+0000 7f30b6ffd640 1 --2- 172.21.15.27:0/230540131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158241010 0x564158243400 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0ecf60 tx=0x7f30ac0f3eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a15743684acde41 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.599+0000 7f30b7fff640 1 --2- 172.21.15.27:0/2757421679 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f30a00a2d00 0x7f30a00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.599+0000 7f309dffb640 1 -- 172.21.15.27:0/230540131 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac0c9070 con 0x564158241010 2024-04-25T00:24:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.599+0000 7f30b4ff9640 1 -- 172.21.15.27:0/2757421679 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f30ac0efc80 con 0x7f30b806a780 2024-04-25T00:24:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.599+0000 7f309dffb640 1 -- 172.21.15.27:0/230540131 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30ac0bf070 con 0x564158241010 2024-04-25T00:24:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.599+0000 7f309dffb640 1 -- 172.21.15.27:0/230540131 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac0bbc70 con 0x564158241010 2024-04-25T00:24:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.600+0000 7f30c1011740 1 -- 172.21.15.27:0/230540131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158241010 msgr2=0x564158243400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.600+0000 7f30b7fff640 1 --2- 172.21.15.27:0/2757421679 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f30a00a2d00 0x7f30a00a51c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f30b0009270 tx=0x7f30b0009680 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.600+0000 7f30c1011740 1 --2- 172.21.15.27:0/230540131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158241010 0x564158243400 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0ecf60 tx=0x7f30ac0f3eb0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.600+0000 7f30c1011740 1 -- 172.21.15.27:0/230540131 shutdown_connections 2024-04-25T00:24:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.600+0000 7f30c1011740 1 --2- 172.21.15.27:0/230540131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158241010 0x564158243400 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.600+0000 7f30c1011740 1 --2- 172.21.15.27:0/230540131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158236290 0x564158236710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.600+0000 7f30c1011740 1 --2- 172.21.15.27:0/230540131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158237980 0x564158235cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.600+0000 7f30c1011740 1 -- 172.21.15.27:0/230540131 >> 172.21.15.27:0/230540131 conn(0x5641581fe950 msgr2=0x5641582327f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.600+0000 7f30c1011740 1 -- 172.21.15.27:0/230540131 shutdown_connections 2024-04-25T00:24:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.600+0000 7f30c1011740 1 -- 172.21.15.27:0/230540131 wait complete. 2024-04-25T00:24:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.600+0000 7f30c1011740 1 -- start start 2024-04-25T00:24:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.600+0000 7f30c1011740 1 Processor -- start 2024-04-25T00:24:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30c1011740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415823dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30c1011740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158237980 0x56415823e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30c1011740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158241010 0x56415823ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30c1011740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564158243c60 con 0x564158236290 2024-04-25T00:24:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30c1011740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564158155e90 con 0x564158241010 2024-04-25T00:24:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30c1011740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564158243de0 con 0x564158237980 2024-04-25T00:24:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30c1011740 10 client.0 _ll_register_callbacks cb 0x564158191970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30b77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415823dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30b77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415823dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45108/0 (socket says 172.21.15.27:45108) 2024-04-25T00:24:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30b77fe640 1 -- 172.21.15.27:0/970511264 learned_addr learned my addr 172.21.15.27:0/970511264 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30b6ffd640 1 --2- 172.21.15.27:0/970511264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158237980 0x56415823e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30b7fff640 1 --2- 172.21.15.27:0/970511264 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158241010 0x56415823ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30b77fe640 1 -- 172.21.15.27:0/970511264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158237980 msgr2=0x56415823e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30b77fe640 1 --2- 172.21.15.27:0/970511264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158237980 0x56415823e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30b77fe640 1 -- 172.21.15.27:0/970511264 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158241010 msgr2=0x56415823ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30b77fe640 1 --2- 172.21.15.27:0/970511264 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158241010 0x56415823ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30b77fe640 1 -- 172.21.15.27:0/970511264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564157f767e0 con 0x564158236290 2024-04-25T00:24:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30b7fff640 1 --2- 172.21.15.27:0/970511264 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158241010 0x56415823ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30b77fe640 1 --2- 172.21.15.27:0/970511264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415823dfa0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f30a800be70 tx=0x7f30a8012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83eb85bba7bdd4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f3096ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f3096ffd640 1 -- 172.21.15.27:0/970511264 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a8004220 con 0x564158236290 2024-04-25T00:24:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f3096ffd640 1 -- 172.21.15.27:0/970511264 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f30a80043c0 con 0x564158236290 2024-04-25T00:24:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.601+0000 7f30c1011740 1 -- 172.21.15.27:0/970511264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30ac0bf070 con 0x564158236290 2024-04-25T00:24:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.602+0000 7f3096ffd640 1 -- 172.21.15.27:0/970511264 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f30a8024050 con 0x564158236290 2024-04-25T00:24:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.602+0000 7f30c1011740 10 client.17091 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.602+0000 7f30c1011740 1 -- 172.21.15.27:0/970511264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x564157fd6f70 con 0x564158236290 2024-04-25T00:24:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.602+0000 7f3096ffd640 1 -- 172.21.15.27:0/970511264 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30ac0bf070 con 0x564158236290 2024-04-25T00:24:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.602+0000 7f3096ffd640 1 -- 172.21.15.27:0/970511264 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f30a8019030 con 0x564158236290 2024-04-25T00:24:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.602+0000 7f30c1011740 10 client.17091 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.602+0000 7f30c1011740 10 client.17091 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:24:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.602+0000 7f30c1011740 20 client.17091 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.602+0000 7f30c1011740 1 --2- 172.21.15.27:0/970511264 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564158250f60 0x5641582533d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.602+0000 7f30c1011740 4 client.17091 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:24:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.602+0000 7f30c1011740 1 -- 172.21.15.27:0/970511264 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56415816e5a0 con 0x564158250f60 2024-04-25T00:24:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.602+0000 7f30b6ffd640 1 --2- 172.21.15.27:0/970511264 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564158250f60 0x5641582533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.603+0000 7f30b6ffd640 1 --2- 172.21.15.27:0/970511264 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564158250f60 0x5641582533d0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20c76d8a06b3ecd0 server_cookie=fd6bf55f7fc8a230 in_seq=0 out_seq=0 2024-04-25T00:24:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.603+0000 7f3096ffd640 10 client.17091 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.603+0000 7f3096ffd640 1 -- 172.21.15.27:0/970511264 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56415816e5a0 con 0x564158250f60 2024-04-25T00:24:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.603+0000 7f3096ffd640 10 client.17091 handle_command_reply: tid=0 2024-04-25T00:24:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.604+0000 7f30c1011740 1 client.17091 shutdown 2024-04-25T00:24:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.604+0000 7f30c1011740 1 -- 172.21.15.27:0/970511264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 msgr2=0x56415823dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.604+0000 7f30c1011740 1 --2- 172.21.15.27:0/970511264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415823dfa0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f30a800be70 tx=0x7f30a8012820 comp rx=0 tx=0).stop 2024-04-25T00:24:59.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.604+0000 7f30c1011740 1 -- 172.21.15.27:0/970511264 shutdown_connections 2024-04-25T00:24:59.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.604+0000 7f30c1011740 1 --2- 172.21.15.27:0/970511264 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x564158250f60 0x5641582533d0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.604+0000 7f30c1011740 1 --2- 172.21.15.27:0/970511264 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158241010 0x56415823ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.604+0000 7f30c1011740 1 --2- 172.21.15.27:0/970511264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158237980 0x56415823e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.604+0000 7f30c1011740 1 --2- 172.21.15.27:0/970511264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415823dfa0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.604+0000 7f3096ffd640 0 client.17091 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:59.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.604+0000 7f30c1011740 1 -- 172.21.15.27:0/970511264 >> 172.21.15.27:0/970511264 conn(0x5641581fe950 msgr2=0x564158238660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:59.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.604+0000 7f30c1011740 1 -- 172.21.15.27:0/970511264 shutdown_connections 2024-04-25T00:24:59.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.604+0000 7f30c1011740 1 -- 172.21.15.27:0/970511264 wait complete. 2024-04-25T00:24:59.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.604+0000 7f30c1011740 20 client.17091 trim_cache size 0 max 16384 2024-04-25T00:24:59.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30c1011740 1 Processor -- start 2024-04-25T00:24:59.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30c1011740 1 -- start start 2024-04-25T00:24:59.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30c1011740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415824cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30c1011740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158237980 0x56415823d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30c1011740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158241010 0x5641582317d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30c1011740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f30b81aebd0 con 0x564158236290 2024-04-25T00:24:59.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30c1011740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f30b81b0770 con 0x564158237980 2024-04-25T00:24:59.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30c1011740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f30b81af2a0 con 0x564158241010 2024-04-25T00:24:59.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30b77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415824cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30b77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415824cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45114/0 (socket says 172.21.15.27:45114) 2024-04-25T00:24:59.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30b77fe640 1 -- 172.21.15.27:0/859789592 learned_addr learned my addr 172.21.15.27:0/859789592 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:59.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30b7fff640 1 --2- 172.21.15.27:0/859789592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158241010 0x5641582317d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30b6ffd640 1 --2- 172.21.15.27:0/859789592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158237980 0x56415823d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30b77fe640 1 -- 172.21.15.27:0/859789592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158241010 msgr2=0x5641582317d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:59.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30b77fe640 1 --2- 172.21.15.27:0/859789592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158241010 0x5641582317d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30b77fe640 1 -- 172.21.15.27:0/859789592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158237980 msgr2=0x56415823d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:59.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30b77fe640 1 --2- 172.21.15.27:0/859789592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158237980 0x56415823d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30b77fe640 1 -- 172.21.15.27:0/859789592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564158232a00 con 0x564158236290 2024-04-25T00:24:59.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.622+0000 7f30b6ffd640 1 --2- 172.21.15.27:0/859789592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158237980 0x56415823d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:24:59.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.623+0000 7f30b77fe640 1 --2- 172.21.15.27:0/859789592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415824cb20 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f30a8004e40 tx=0x7f30a8012d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b22ce00677b9432 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:59.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.623+0000 7f309d7fa640 1 -- 172.21.15.27:0/859789592 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a802c070 con 0x564158236290 2024-04-25T00:24:59.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.623+0000 7f309d7fa640 1 -- 172.21.15.27:0/859789592 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30a8004fe0 con 0x564158236290 2024-04-25T00:24:59.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.623+0000 7f309d7fa640 1 -- 172.21.15.27:0/859789592 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a8010b60 con 0x564158236290 2024-04-25T00:24:59.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.623+0000 7f30c1011740 1 -- 172.21.15.27:0/859789592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 msgr2=0x56415824cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.623+0000 7f30c1011740 1 --2- 172.21.15.27:0/859789592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415824cb20 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f30a8004e40 tx=0x7f30a8012d20 comp rx=0 tx=0).stop 2024-04-25T00:24:59.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.623+0000 7f30c1011740 1 -- 172.21.15.27:0/859789592 shutdown_connections 2024-04-25T00:24:59.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.623+0000 7f30c1011740 1 --2- 172.21.15.27:0/859789592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158241010 0x5641582317d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.624+0000 7f30c1011740 1 --2- 172.21.15.27:0/859789592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158237980 0x56415823d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.624+0000 7f30c1011740 1 --2- 172.21.15.27:0/859789592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415824cb20 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.624+0000 7f30c1011740 1 -- 172.21.15.27:0/859789592 >> 172.21.15.27:0/859789592 conn(0x5641581fe950 msgr2=0x5641582310c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:59.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.624+0000 7f30c1011740 1 -- 172.21.15.27:0/859789592 shutdown_connections 2024-04-25T00:24:59.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.624+0000 7f30c1011740 1 -- 172.21.15.27:0/859789592 wait complete. 2024-04-25T00:24:59.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.624+0000 7f30c1011740 1 -- start start 2024-04-25T00:24:59.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.624+0000 7f30c1011740 1 Processor -- start 2024-04-25T00:24:59.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.624+0000 7f30c1011740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415823adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.624+0000 7f30c1011740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158237980 0x56415823b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.624+0000 7f30c1011740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158241010 0x56415823b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.624+0000 7f30c1011740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f30b81b00a0 con 0x564158236290 2024-04-25T00:24:59.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.624+0000 7f30c1011740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f30b81af9a0 con 0x564158237980 2024-04-25T00:24:59.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.624+0000 7f30c1011740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f30b81acb60 con 0x564158241010 2024-04-25T00:24:59.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.624+0000 7f30c1011740 10 client.0 _ll_register_callbacks cb 0x7f30a800ba80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:59.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f30b77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415823adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f30b77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415823adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45126/0 (socket says 172.21.15.27:45126) 2024-04-25T00:24:59.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f30b77fe640 1 -- 172.21.15.27:0/3041438576 learned_addr learned my addr 172.21.15.27:0/3041438576 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:59.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f30b7fff640 1 --2- 172.21.15.27:0/3041438576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158241010 0x56415823b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f30b6ffd640 1 --2- 172.21.15.27:0/3041438576 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158237980 0x56415823b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f30b77fe640 1 -- 172.21.15.27:0/3041438576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158241010 msgr2=0x56415823b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:59.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f30b77fe640 1 --2- 172.21.15.27:0/3041438576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158241010 0x56415823b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f30b77fe640 1 -- 172.21.15.27:0/3041438576 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158237980 msgr2=0x56415823b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:59.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f30b77fe640 1 --2- 172.21.15.27:0/3041438576 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158237980 0x56415823b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f30b77fe640 1 -- 172.21.15.27:0/3041438576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564157e6e170 con 0x564158236290 2024-04-25T00:24:59.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f30b6ffd640 1 --2- 172.21.15.27:0/3041438576 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158237980 0x56415823b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f30b77fe640 1 --2- 172.21.15.27:0/3041438576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415823adf0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f30a8012860 tx=0x7f30a8005440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a7b57d3c5c17ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f3096ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f3096ffd640 1 -- 172.21.15.27:0/3041438576 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a802c070 con 0x564158236290 2024-04-25T00:24:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f3096ffd640 1 -- 172.21.15.27:0/3041438576 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f30a8004a70 con 0x564158236290 2024-04-25T00:24:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f30c1011740 1 -- 172.21.15.27:0/3041438576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30a8004fe0 con 0x564158236290 2024-04-25T00:24:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f3096ffd640 1 -- 172.21.15.27:0/3041438576 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f30a80108e0 con 0x564158236290 2024-04-25T00:24:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f30c1011740 10 client.17097 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.625+0000 7f30c1011740 1 -- 172.21.15.27:0/3041438576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x5641582bbb90 con 0x564158236290 2024-04-25T00:24:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.626+0000 7f3096ffd640 1 -- 172.21.15.27:0/3041438576 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30a8004fe0 con 0x564158236290 2024-04-25T00:24:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.626+0000 7f3096ffd640 1 -- 172.21.15.27:0/3041438576 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f30a8038020 con 0x564158236290 2024-04-25T00:24:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.626+0000 7f30c1011740 10 client.17097 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.626+0000 7f30c1011740 10 client.17097 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:24:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.626+0000 7f30c1011740 20 client.17097 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.626+0000 7f30c1011740 1 --2- 172.21.15.27:0/3041438576 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5641582bc1f0 0x5641582be610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.626+0000 7f30c1011740 4 client.17097 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T00:24:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.626+0000 7f30c1011740 1 -- 172.21.15.27:0/3041438576 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5641581a7800 con 0x5641582bc1f0 2024-04-25T00:24:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.626+0000 7f30b6ffd640 1 --2- 172.21.15.27:0/3041438576 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5641582bc1f0 0x5641582be610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.627+0000 7f3096ffd640 10 client.17097 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.627+0000 7f30b6ffd640 1 --2- 172.21.15.27:0/3041438576 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5641582bc1f0 0x5641582be610 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dc6da5b3d3905ae server_cookie=a4127d34e36cc429 in_seq=0 out_seq=0 2024-04-25T00:24:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.628+0000 7f3096ffd640 1 -- 172.21.15.27:0/3041438576 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5641581a7800 con 0x5641582bc1f0 2024-04-25T00:24:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.628+0000 7f3096ffd640 10 client.17097 handle_command_reply: tid=0 2024-04-25T00:24:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.628+0000 7f30c1011740 1 client.17097 shutdown 2024-04-25T00:24:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.628+0000 7f30c1011740 1 -- 172.21.15.27:0/3041438576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 msgr2=0x56415823adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.628+0000 7f30c1011740 1 --2- 172.21.15.27:0/3041438576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415823adf0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f30a8012860 tx=0x7f30a8005440 comp rx=0 tx=0).stop 2024-04-25T00:24:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.628+0000 7f30c1011740 1 -- 172.21.15.27:0/3041438576 shutdown_connections 2024-04-25T00:24:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.628+0000 7f30c1011740 1 --2- 172.21.15.27:0/3041438576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564158241010 0x56415823b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.628+0000 7f30c1011740 1 --2- 172.21.15.27:0/3041438576 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5641582bc1f0 0x5641582be610 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.628+0000 7f30c1011740 1 --2- 172.21.15.27:0/3041438576 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564158237980 0x56415823b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.628+0000 7f30c1011740 1 --2- 172.21.15.27:0/3041438576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564158236290 0x56415823adf0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.628+0000 7f30c1011740 1 -- 172.21.15.27:0/3041438576 >> 172.21.15.27:0/3041438576 conn(0x5641581fe950 msgr2=0x56415823df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:59.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.628+0000 7f3096ffd640 0 client.17097 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:59.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.628+0000 7f30c1011740 1 -- 172.21.15.27:0/3041438576 shutdown_connections 2024-04-25T00:24:59.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.628+0000 7f30c1011740 1 -- 172.21.15.27:0/3041438576 wait complete. 2024-04-25T00:24:59.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.628+0000 7f30c1011740 20 client.17097 trim_cache size 0 max 16384 2024-04-25T00:24:59.635 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:24:59.635 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T00:24:59.635 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "63f3f979-d301-4079-99b6-c21a8b242973", 2024-04-25T00:24:59.635 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T00:24:59.635 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:24:59.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.629+0000 7f30be22a640 1 -- 172.21.15.27:0/2757421679 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f30a00a2d00 msgr2=0x7f30a00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.629+0000 7f30be22a640 1 --2- 172.21.15.27:0/2757421679 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f30a00a2d00 0x7f30a00a51c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f30b0009270 tx=0x7f30b0009680 comp rx=0 tx=0).stop 2024-04-25T00:24:59.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.629+0000 7f30be22a640 1 -- 172.21.15.27:0/2757421679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30b806a780 msgr2=0x7f30b81985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.629+0000 7f30be22a640 1 --2- 172.21.15.27:0/2757421679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30b806a780 0x7f30b81985a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f30ac005940 tx=0x7f30ac0042f0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.629+0000 7f30be22a640 1 -- 172.21.15.27:0/2757421679 shutdown_connections 2024-04-25T00:24:59.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.629+0000 7f30be22a640 1 --2- 172.21.15.27:0/2757421679 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f30a00a2d00 0x7f30a00a51c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.629+0000 7f30be22a640 1 --2- 172.21.15.27:0/2757421679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b8107730 0x7f30b819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.629+0000 7f30be22a640 1 --2- 172.21.15.27:0/2757421679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30b806a780 0x7f30b81985a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.629+0000 7f30be22a640 1 --2- 172.21.15.27:0/2757421679 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30b8069dc0 0x7f30b8198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.629+0000 7f30be22a640 1 -- 172.21.15.27:0/2757421679 >> 172.21.15.27:0/2757421679 conn(0x7f30b80776a0 msgr2=0x7f30b80fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:59.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.629+0000 7f30be22a640 1 -- 172.21.15.27:0/2757421679 shutdown_connections 2024-04-25T00:24:59.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.629+0000 7f30be22a640 1 -- 172.21.15.27:0/2757421679 wait complete. 2024-04-25T00:24:59.640 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:24:59.640 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:24:59.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.745+0000 7f2e9cdce640 1 -- 172.21.15.27:0/1136084196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9810bd00 msgr2=0x7f2e9810e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.745+0000 7f2e9cdce640 1 --2- 172.21.15.27:0/1136084196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9810bd00 0x7f2e9810e1a0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009ca0 tx=0x7f2e9002a9e0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.745+0000 7f2e9cdce640 1 -- 172.21.15.27:0/1136084196 shutdown_connections 2024-04-25T00:24:59.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.745+0000 7f2e9cdce640 1 --2- 172.21.15.27:0/1136084196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9810bd00 0x7f2e9810e1a0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.745+0000 7f2e9cdce640 1 --2- 172.21.15.27:0/1136084196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e981032e0 0x7f2e9810b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.745+0000 7f2e9cdce640 1 --2- 172.21.15.27:0/1136084196 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e981029c0 0x7f2e98102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.745+0000 7f2e9cdce640 1 -- 172.21.15.27:0/1136084196 >> 172.21.15.27:0/1136084196 conn(0x7f2e98077740 msgr2=0x7f2e98077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:59.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.746+0000 7f2e9cdce640 1 -- 172.21.15.27:0/1136084196 shutdown_connections 2024-04-25T00:24:59.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.746+0000 7f2e9cdce640 1 -- 172.21.15.27:0/1136084196 wait complete. 2024-04-25T00:24:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.746+0000 7f2e9cdce640 1 Processor -- start 2024-04-25T00:24:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.746+0000 7f2e9cdce640 1 -- start start 2024-04-25T00:24:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e9cdce640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e981029c0 0x7f2e98100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e9cdce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e981032e0 0x7f2e98100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e9cdce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e9810bd00 0x7f2e98101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e9cdce640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2e98111b00 con 0x7f2e981032e0 2024-04-25T00:24:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e9cdce640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2e98111980 con 0x7f2e981029c0 2024-04-25T00:24:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e9cdce640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2e98111c80 con 0x7f2e9810bd00 2024-04-25T00:24:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e8dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e981032e0 0x7f2e98100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e8dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e981032e0 0x7f2e98100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45152/0 (socket says 172.21.15.27:45152) 2024-04-25T00:24:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e8dd9b640 1 -- 172.21.15.27:0/3634507226 learned_addr learned my addr 172.21.15.27:0/3634507226 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e96d9d640 1 --2- 172.21.15.27:0/3634507226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e9810bd00 0x7f2e98101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e9659c640 1 --2- 172.21.15.27:0/3634507226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e981029c0 0x7f2e98100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e8dd9b640 1 -- 172.21.15.27:0/3634507226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e9810bd00 msgr2=0x7f2e98101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e8dd9b640 1 --2- 172.21.15.27:0/3634507226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e9810bd00 0x7f2e98101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e8dd9b640 1 -- 172.21.15.27:0/3634507226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e981029c0 msgr2=0x7f2e98100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e8dd9b640 1 --2- 172.21.15.27:0/3634507226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e981029c0 0x7f2e98100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e8dd9b640 1 -- 172.21.15.27:0/3634507226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e981a6580 con 0x7f2e981032e0 2024-04-25T00:24:59.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e9659c640 1 --2- 172.21.15.27:0/3634507226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e981029c0 0x7f2e98100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:59.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.747+0000 7f2e8dd9b640 1 --2- 172.21.15.27:0/3634507226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e981032e0 0x7f2e98100b50 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000d770 tx=0x7f2e8000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:59.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.748+0000 7f2e8ffff640 1 -- 172.21.15.27:0/3634507226 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e80017070 con 0x7f2e981032e0 2024-04-25T00:24:59.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.748+0000 7f2e9cdce640 1 -- 172.21.15.27:0/3634507226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e981a6870 con 0x7f2e981032e0 2024-04-25T00:24:59.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.748+0000 7f2e8ffff640 1 -- 172.21.15.27:0/3634507226 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e80005500 con 0x7f2e981032e0 2024-04-25T00:24:59.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.748+0000 7f2e8ffff640 1 -- 172.21.15.27:0/3634507226 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e80004e00 con 0x7f2e981032e0 2024-04-25T00:24:59.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.748+0000 7f2e9cdce640 1 -- 172.21.15.27:0/3634507226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e981a6dc0 con 0x7f2e981032e0 2024-04-25T00:24:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.750+0000 7f2e8ffff640 1 -- 172.21.15.27:0/3634507226 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2e80004fa0 con 0x7f2e981032e0 2024-04-25T00:24:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.750+0000 7f2e9fbbb740 1 Processor -- start 2024-04-25T00:24:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.750+0000 7f2e9fbbb740 1 -- start start 2024-04-25T00:24:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.750+0000 7f2e9fbbb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c08a630 0x55e75c08aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.750+0000 7f2e9fbbb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08af50 0x55e75c0950e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.750+0000 7f2e9fbbb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c095620 0x55e75c097a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.750+0000 7f2e9fbbb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e75bfef6b0 con 0x55e75c08af50 2024-04-25T00:24:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.750+0000 7f2e8ffff640 1 --2- 172.21.15.27:0/3634507226 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2e840a2d00 0x7f2e840a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.750+0000 7f2e9fbbb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e75bf97880 con 0x55e75c095620 2024-04-25T00:24:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.750+0000 7f2e9fbbb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e75bfeb670 con 0x55e75c08a630 2024-04-25T00:24:59.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.751+0000 7f2e9659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08af50 0x55e75c0950e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.751+0000 7f2e96d9d640 1 --2- 172.21.15.27:0/3634507226 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2e840a2d00 0x7f2e840a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.751+0000 7f2e9659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08af50 0x55e75c0950e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45156/0 (socket says 172.21.15.27:45156) 2024-04-25T00:24:59.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.751+0000 7f2e9659c640 1 -- 172.21.15.27:0/2089079802 learned_addr learned my addr 172.21.15.27:0/2089079802 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:59.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.751+0000 7f2e8dd9b640 1 --2- 172.21.15.27:0/2089079802 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c095620 0x55e75c097a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.751+0000 7f2e8ffff640 1 -- 172.21.15.27:0/3634507226 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f2e800cf180 con 0x7f2e981032e0 2024-04-25T00:24:59.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.751+0000 7f2e96d9d640 1 --2- 172.21.15.27:0/2089079802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c08a630 0x55e75c08aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.751+0000 7f2e9659c640 1 -- 172.21.15.27:0/2089079802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c08a630 msgr2=0x55e75c08aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:59.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.751+0000 7f2e9659c640 1 --2- 172.21.15.27:0/2089079802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c08a630 0x55e75c08aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.751+0000 7f2e9659c640 1 -- 172.21.15.27:0/2089079802 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c095620 msgr2=0x55e75c097a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:59.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.751+0000 7f2e9659c640 1 --2- 172.21.15.27:0/2089079802 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c095620 0x55e75c097a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.751+0000 7f2e9659c640 1 -- 172.21.15.27:0/2089079802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e75bdcd7d0 con 0x55e75c08af50 2024-04-25T00:24:59.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.751+0000 7f2e9659c640 1 --2- 172.21.15.27:0/2089079802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08af50 0x55e75c0950e0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800bd40 tx=0x7f2e8800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da1f4f9a92cc8263 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:59.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.751+0000 7f2e96d9d640 1 --2- 172.21.15.27:0/3634507226 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2e840a2d00 0x7f2e840a51c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2e90005e30 tx=0x7f2e9003a040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.751+0000 7f2e8cd99640 1 -- 172.21.15.27:0/2089079802 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e88015070 con 0x55e75c08af50 2024-04-25T00:24:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.751+0000 7f2e8cd99640 1 -- 172.21.15.27:0/2089079802 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e8800be30 con 0x55e75c08af50 2024-04-25T00:24:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.751+0000 7f2e8cd99640 1 -- 172.21.15.27:0/2089079802 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e88011a30 con 0x55e75c08af50 2024-04-25T00:24:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.752+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2089079802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08af50 msgr2=0x55e75c0950e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.752+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/2089079802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08af50 0x55e75c0950e0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800bd40 tx=0x7f2e8800f730 comp rx=0 tx=0).stop 2024-04-25T00:24:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.752+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2089079802 shutdown_connections 2024-04-25T00:24:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.752+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/2089079802 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c095620 0x55e75c097a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.752+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/2089079802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08af50 0x55e75c0950e0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.752+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/2089079802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c08a630 0x55e75c08aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.752+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2089079802 >> 172.21.15.27:0/2089079802 conn(0x55e75c086950 msgr2=0x55e75c086d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.752+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2089079802 shutdown_connections 2024-04-25T00:24:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.752+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2089079802 wait complete. 2024-04-25T00:24:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.752+0000 7f2e9fbbb740 1 -- start start 2024-04-25T00:24:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.752+0000 7f2e9fbbb740 1 Processor -- start 2024-04-25T00:24:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e9fbbb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c08a630 0x55e75c092550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e9fbbb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c08af50 0x55e75c092a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e9fbbb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c095620 0x55e75c092fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e9fbbb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e75c098410 con 0x55e75c095620 2024-04-25T00:24:59.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e9fbbb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e75bfaaf50 con 0x55e75c08a630 2024-04-25T00:24:59.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e9fbbb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e75c098590 con 0x55e75c08af50 2024-04-25T00:24:59.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e9fbbb740 10 client.0 _ll_register_callbacks cb 0x55e75bfe6000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:59.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e96d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c095620 0x55e75c092fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e96d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c095620 0x55e75c092fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45168/0 (socket says 172.21.15.27:45168) 2024-04-25T00:24:59.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e96d9d640 1 -- 172.21.15.27:0/2942343757 learned_addr learned my addr 172.21.15.27:0/2942343757 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:59.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e8dd9b640 1 --2- 172.21.15.27:0/2942343757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c08af50 0x55e75c092a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e9659c640 1 --2- 172.21.15.27:0/2942343757 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c08a630 0x55e75c092550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e96d9d640 1 -- 172.21.15.27:0/2942343757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c08af50 msgr2=0x55e75c092a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:59.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e96d9d640 1 --2- 172.21.15.27:0/2942343757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c08af50 0x55e75c092a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e96d9d640 1 -- 172.21.15.27:0/2942343757 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c08a630 msgr2=0x55e75c092550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:59.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e96d9d640 1 --2- 172.21.15.27:0/2942343757 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c08a630 0x55e75c092550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e96d9d640 1 -- 172.21.15.27:0/2942343757 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e75bd033f0 con 0x55e75c095620 2024-04-25T00:24:59.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e96d9d640 1 --2- 172.21.15.27:0/2942343757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c095620 0x55e75c092fd0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f2e90035000 tx=0x7f2e9002a9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16170eb9af33c814 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:59.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.753+0000 7f2e7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:59.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.754+0000 7f2e7dffb640 1 -- 172.21.15.27:0/2942343757 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9004e070 con 0x55e75c095620 2024-04-25T00:24:59.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.754+0000 7f2e7dffb640 1 -- 172.21.15.27:0/2942343757 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f2e900045c0 con 0x55e75c095620 2024-04-25T00:24:59.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.754+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2942343757 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e8800be30 con 0x55e75c095620 2024-04-25T00:24:59.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.754+0000 7f2e7dffb640 1 -- 172.21.15.27:0/2942343757 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f2e90033070 con 0x55e75c095620 2024-04-25T00:24:59.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.754+0000 7f2e9fbbb740 10 client.17109 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:59.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.754+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2942343757 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55e75bcc30f0 con 0x55e75c095620 2024-04-25T00:24:59.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.754+0000 7f2e7dffb640 1 -- 172.21.15.27:0/2942343757 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e8800be30 con 0x55e75c095620 2024-04-25T00:24:59.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.754+0000 7f2e7dffb640 1 -- 172.21.15.27:0/2942343757 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f2e90047040 con 0x55e75c095620 2024-04-25T00:24:59.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.754+0000 7f2e9fbbb740 10 client.17109 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:59.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.754+0000 7f2e9fbbb740 10 client.17109 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:24:59.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.754+0000 7f2e9fbbb740 20 client.17109 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:59.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.754+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/2942343757 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55e75c0a5920 0x55e75c0a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.754+0000 7f2e9fbbb740 4 client.17109 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:24:59.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.754+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2942343757 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e75bffcbf0 con 0x55e75c0a5920 2024-04-25T00:24:59.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.755+0000 7f2e9659c640 1 --2- 172.21.15.27:0/2942343757 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55e75c0a5920 0x55e75c0a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.755+0000 7f2e9659c640 1 --2- 172.21.15.27:0/2942343757 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55e75c0a5920 0x55e75c0a7da0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b380c70ec92bf6b server_cookie=ef2ae8c69788a457 in_seq=0 out_seq=0 2024-04-25T00:24:59.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.755+0000 7f2e7dffb640 10 client.17109 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:59.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.756+0000 7f2e7dffb640 1 -- 172.21.15.27:0/2942343757 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e75bffcbf0 con 0x55e75c0a5920 2024-04-25T00:24:59.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.756+0000 7f2e7dffb640 10 client.17109 handle_command_reply: tid=0 2024-04-25T00:24:59.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.756+0000 7f2e9fbbb740 1 client.17109 shutdown 2024-04-25T00:24:59.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.756+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2942343757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c095620 msgr2=0x55e75c092fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.756+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/2942343757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c095620 0x55e75c092fd0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f2e90035000 tx=0x7f2e9002a9e0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.756+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2942343757 shutdown_connections 2024-04-25T00:24:59.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.756+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/2942343757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c095620 0x55e75c092fd0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.756+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/2942343757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c08af50 0x55e75c092a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.756+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/2942343757 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55e75c0a5920 0x55e75c0a7da0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.756+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/2942343757 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c08a630 0x55e75c092550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.756+0000 7f2e7dffb640 0 client.17109 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:59.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.756+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2942343757 >> 172.21.15.27:0/2942343757 conn(0x55e75c086950 msgr2=0x55e75c08c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:59.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.756+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2942343757 shutdown_connections 2024-04-25T00:24:59.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.756+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2942343757 wait complete. 2024-04-25T00:24:59.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.756+0000 7f2e9fbbb740 20 client.17109 trim_cache size 0 max 16384 2024-04-25T00:24:59.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.773+0000 7f2e9fbbb740 1 Processor -- start 2024-04-25T00:24:59.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.773+0000 7f2e9fbbb740 1 -- start start 2024-04-25T00:24:59.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e9fbbb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08a630 0x55e75c0a1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e9fbbb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c08af50 0x55e75c0916f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e9fbbb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c095620 0x55e75c08cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e9fbbb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2e981b0e20 con 0x55e75c08a630 2024-04-25T00:24:59.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e9fbbb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2e981b29c0 con 0x55e75c095620 2024-04-25T00:24:59.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e9fbbb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2e981b14f0 con 0x55e75c08af50 2024-04-25T00:24:59.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e9659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08a630 0x55e75c0a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e9659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08a630 0x55e75c0a1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45182/0 (socket says 172.21.15.27:45182) 2024-04-25T00:24:59.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e9659c640 1 -- 172.21.15.27:0/2539291438 learned_addr learned my addr 172.21.15.27:0/2539291438 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:59.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e8dd9b640 1 --2- 172.21.15.27:0/2539291438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c08af50 0x55e75c0916f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e96d9d640 1 --2- 172.21.15.27:0/2539291438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c095620 0x55e75c08cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e9659c640 1 -- 172.21.15.27:0/2539291438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c08af50 msgr2=0x55e75c0916f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:59.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e9659c640 1 --2- 172.21.15.27:0/2539291438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c08af50 0x55e75c0916f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e9659c640 1 -- 172.21.15.27:0/2539291438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c095620 msgr2=0x55e75c08cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:59.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e9659c640 1 --2- 172.21.15.27:0/2539291438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c095620 0x55e75c08cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e9659c640 1 -- 172.21.15.27:0/2539291438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e75c078090 con 0x55e75c08a630 2024-04-25T00:24:59.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e8dd9b640 1 --2- 172.21.15.27:0/2539291438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c08af50 0x55e75c0916f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:59.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e9659c640 1 --2- 172.21.15.27:0/2539291438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08a630 0x55e75c0a1300 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f2e88011620 tx=0x7f2e8801af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d8df1bf4dbe987a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:59.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e7ffff640 1 -- 172.21.15.27:0/2539291438 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8802d070 con 0x55e75c08a630 2024-04-25T00:24:59.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e7ffff640 1 -- 172.21.15.27:0/2539291438 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e88011680 con 0x55e75c08a630 2024-04-25T00:24:59.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.774+0000 7f2e7ffff640 1 -- 172.21.15.27:0/2539291438 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e88015040 con 0x55e75c08a630 2024-04-25T00:24:59.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.775+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2539291438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08a630 msgr2=0x55e75c0a1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.775+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/2539291438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08a630 0x55e75c0a1300 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f2e88011620 tx=0x7f2e8801af90 comp rx=0 tx=0).stop 2024-04-25T00:24:59.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.775+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2539291438 shutdown_connections 2024-04-25T00:24:59.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.775+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/2539291438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c095620 0x55e75c08cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.775+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/2539291438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c08af50 0x55e75c0916f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.775+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/2539291438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08a630 0x55e75c0a1300 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.775+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2539291438 >> 172.21.15.27:0/2539291438 conn(0x55e75c086950 msgr2=0x55e75c093c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:59.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.775+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2539291438 shutdown_connections 2024-04-25T00:24:59.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.775+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/2539291438 wait complete. 2024-04-25T00:24:59.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.775+0000 7f2e9fbbb740 1 -- start start 2024-04-25T00:24:59.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.775+0000 7f2e9fbbb740 1 Processor -- start 2024-04-25T00:24:59.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e9fbbb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c08a630 0x55e75c09da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e9fbbb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08af50 0x55e75c08ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e9fbbb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c095620 0x55e75c08f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e9fbbb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2e981b22f0 con 0x55e75c08af50 2024-04-25T00:24:59.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e9fbbb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2e981b1bf0 con 0x55e75c08a630 2024-04-25T00:24:59.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e9fbbb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2e981aedb0 con 0x55e75c095620 2024-04-25T00:24:59.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e9fbbb740 10 client.0 _ll_register_callbacks cb 0x7f2e90003d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:24:59.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e8dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08af50 0x55e75c08ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e8dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08af50 0x55e75c08ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45188/0 (socket says 172.21.15.27:45188) 2024-04-25T00:24:59.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e8dd9b640 1 -- 172.21.15.27:0/3534475897 learned_addr learned my addr 172.21.15.27:0/3534475897 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:24:59.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e96d9d640 1 --2- 172.21.15.27:0/3534475897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c095620 0x55e75c08f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e9659c640 1 --2- 172.21.15.27:0/3534475897 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c08a630 0x55e75c09da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e8dd9b640 1 -- 172.21.15.27:0/3534475897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c095620 msgr2=0x55e75c08f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:59.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e8dd9b640 1 --2- 172.21.15.27:0/3534475897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c095620 0x55e75c08f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e8dd9b640 1 -- 172.21.15.27:0/3534475897 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c08a630 msgr2=0x55e75c09da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e8dd9b640 1 --2- 172.21.15.27:0/3534475897 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c08a630 0x55e75c09da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e8dd9b640 1 -- 172.21.15.27:0/3534475897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e75c08f690 con 0x55e75c08af50 2024-04-25T00:24:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e96d9d640 1 --2- 172.21.15.27:0/3534475897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c095620 0x55e75c08f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e8dd9b640 1 --2- 172.21.15.27:0/3534475897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08af50 0x55e75c08ec10 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f2e8009b030 tx=0x7f2e800bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d374164d87047e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:24:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e7dffb640 1 -- 172.21.15.27:0/3534475897 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e800b4060 con 0x55e75c08af50 2024-04-25T00:24:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e7dffb640 1 -- 172.21.15.27:0/3534475897 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f2e800b4200 con 0x55e75c08af50 2024-04-25T00:24:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/3534475897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e88011680 con 0x55e75c08af50 2024-04-25T00:24:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e7dffb640 1 -- 172.21.15.27:0/3534475897 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f2e80020040 con 0x55e75c08af50 2024-04-25T00:24:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e9fbbb740 10 client.17115 fetch_fsmap learned FSMap version 97 2024-04-25T00:24:59.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.776+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/3534475897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=4+}) -- 0x55e75c10bb90 con 0x55e75c08af50 2024-04-25T00:24:59.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.777+0000 7f2e7dffb640 1 -- 172.21.15.27:0/3534475897 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e88011680 con 0x55e75c08af50 2024-04-25T00:24:59.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.777+0000 7f2e7dffb640 1 -- 172.21.15.27:0/3534475897 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 97) ==== 8378+0+0 (secure 0 0 0) 0x7f2e800979a0 con 0x55e75c08af50 2024-04-25T00:24:59.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.777+0000 7f2e9fbbb740 10 client.17115 fetch_fsmap finished waiting for FSMap version 97 2024-04-25T00:24:59.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.777+0000 7f2e9fbbb740 10 client.17115 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:24:59.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.777+0000 7f2e9fbbb740 20 client.17115 populate_metadata read hostname 'smithi027' 2024-04-25T00:24:59.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.777+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/3534475897 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55e75c10c310 0x55e75c10e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:59.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.777+0000 7f2e9fbbb740 4 client.17115 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:24:59.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.777+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/3534475897 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e75bff6f40 con 0x55e75c10c310 2024-04-25T00:24:59.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.777+0000 7f2e9659c640 1 --2- 172.21.15.27:0/3534475897 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55e75c10c310 0x55e75c10e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:59.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.778+0000 7f2e7dffb640 10 client.17115 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:59.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.778+0000 7f2e9659c640 1 --2- 172.21.15.27:0/3534475897 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55e75c10c310 0x55e75c10e730 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6035efa169385278 server_cookie=ae53c5d14273e16b in_seq=0 out_seq=0 2024-04-25T00:24:59.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.778+0000 7f2e7dffb640 1 -- 172.21.15.27:0/3534475897 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e75bff6f40 con 0x55e75c10c310 2024-04-25T00:24:59.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.778+0000 7f2e7dffb640 10 client.17115 handle_command_reply: tid=0 2024-04-25T00:24:59.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.778+0000 7f2e9fbbb740 1 client.17115 shutdown 2024-04-25T00:24:59.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.778+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/3534475897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08af50 msgr2=0x55e75c08ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.778+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/3534475897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08af50 0x55e75c08ec10 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f2e8009b030 tx=0x7f2e800bb040 comp rx=0 tx=0).stop 2024-04-25T00:24:59.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.778+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/3534475897 shutdown_connections 2024-04-25T00:24:59.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.778+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/3534475897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e75c095620 0x55e75c08f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.778+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/3534475897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e75c08af50 0x55e75c08ec10 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.778+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/3534475897 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55e75c10c310 0x55e75c10e730 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.778+0000 7f2e9fbbb740 1 --2- 172.21.15.27:0/3534475897 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e75c08a630 0x55e75c09da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.778+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/3534475897 >> 172.21.15.27:0/3534475897 conn(0x55e75c086950 msgr2=0x55e75c085af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:59.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.778+0000 7f2e7dffb640 0 client.17115 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:24:59.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.778+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/3534475897 shutdown_connections 2024-04-25T00:24:59.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.778+0000 7f2e9fbbb740 1 -- 172.21.15.27:0/3534475897 wait complete. 2024-04-25T00:24:59.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.778+0000 7f2e9fbbb740 20 client.17115 trim_cache size 0 max 16384 2024-04-25T00:24:59.787 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:24:59.787 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T00:24:59.787 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:24:59.787 INFO:teuthology.orchestra.run.smithi027.stdout: "63f3f979-d301-4079-99b6-c21a8b242973": { 2024-04-25T00:24:59.787 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:24:59.788 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "63f3f979-d301-4079-99b6-c21a8b242973", 2024-04-25T00:24:59.788 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:24:59.788 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:24:59.788 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:24:59.788 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:24:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.779+0000 7f2e9cdce640 1 -- 172.21.15.27:0/3634507226 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2e840a2d00 msgr2=0x7f2e840a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.779+0000 7f2e9cdce640 1 --2- 172.21.15.27:0/3634507226 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2e840a2d00 0x7f2e840a51c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2e90005e30 tx=0x7f2e9003a040 comp rx=0 tx=0).stop 2024-04-25T00:24:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.779+0000 7f2e9cdce640 1 -- 172.21.15.27:0/3634507226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e981032e0 msgr2=0x7f2e98100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.779+0000 7f2e9cdce640 1 --2- 172.21.15.27:0/3634507226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e981032e0 0x7f2e98100b50 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000d770 tx=0x7f2e8000dc30 comp rx=0 tx=0).stop 2024-04-25T00:24:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.779+0000 7f2e9cdce640 1 -- 172.21.15.27:0/3634507226 shutdown_connections 2024-04-25T00:24:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.779+0000 7f2e9cdce640 1 --2- 172.21.15.27:0/3634507226 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2e840a2d00 0x7f2e840a51c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.779+0000 7f2e9cdce640 1 --2- 172.21.15.27:0/3634507226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e9810bd00 0x7f2e98101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.779+0000 7f2e9cdce640 1 --2- 172.21.15.27:0/3634507226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e981032e0 0x7f2e98100b50 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.779+0000 7f2e9cdce640 1 --2- 172.21.15.27:0/3634507226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e981029c0 0x7f2e98100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.779+0000 7f2e9cdce640 1 -- 172.21.15.27:0/3634507226 >> 172.21.15.27:0/3634507226 conn(0x7f2e98077740 msgr2=0x7f2e980fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.779+0000 7f2e9cdce640 1 -- 172.21.15.27:0/3634507226 shutdown_connections 2024-04-25T00:24:59.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:24:59.779+0000 7f2e9cdce640 1 -- 172.21.15.27:0/3634507226 wait complete. 2024-04-25T00:24:59.791 INFO:tasks.cephfs.filesystem:scrub status for tag:63f3f979-d301-4079-99b6-c21a8b242973 - {'path': '/', 'tag': '63f3f979-d301-4079-99b6-c21a8b242973', 'options': 'recursive,force'} 2024-04-25T00:25:01.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:00 smithi027 ceph-mon[29208]: pgmap v280: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-04-25T00:25:01.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:00 smithi027 ceph-mon[29208]: scrub queued for path: / 2024-04-25T00:25:01.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:00 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:25:01.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:00 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:25:01.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:00 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:25:01.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:00 smithi139 ceph-mon[32833]: pgmap v280: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-04-25T00:25:01.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:00 smithi139 ceph-mon[32833]: scrub queued for path: / 2024-04-25T00:25:01.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:00 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:25:01.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:00 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:25:01.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:00 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:25:01.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:00 smithi153 ceph-mon[32896]: pgmap v280: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-04-25T00:25:01.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:00 smithi153 ceph-mon[32896]: scrub queued for path: / 2024-04-25T00:25:01.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:00 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:25:01.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:00 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:25:01.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:00 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:25:03.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:02 smithi027 ceph-mon[29208]: pgmap v281: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-04-25T00:25:03.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:02 smithi139 ceph-mon[32833]: pgmap v281: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-04-25T00:25:03.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:02 smithi153 ceph-mon[32896]: pgmap v281: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-04-25T00:25:04.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:03 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:25:04.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:03 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:25:04.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:03 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:25:05.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:04 smithi027 ceph-mon[29208]: pgmap v282: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:25:05.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:04 smithi139 ceph-mon[32833]: pgmap v282: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:25:05.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:04 smithi153 ceph-mon[32896]: pgmap v282: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:25:07.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:06 smithi027 ceph-mon[29208]: pgmap v283: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:25:07.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:06 smithi139 ceph-mon[32833]: pgmap v283: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:25:07.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:06 smithi153 ceph-mon[32896]: pgmap v283: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:25:09.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:08 smithi027 ceph-mon[29208]: pgmap v284: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:25:09.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:08 smithi139 ceph-mon[32833]: pgmap v284: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:25:09.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:08 smithi153 ceph-mon[32896]: pgmap v284: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:25:10.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:09 smithi027 ceph-mon[29208]: pgmap v285: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T00:25:10.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:09 smithi139 ceph-mon[32833]: pgmap v285: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T00:25:10.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:09 smithi153 ceph-mon[32896]: pgmap v285: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T00:25:13.006 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:12 smithi139 ceph-mon[32833]: pgmap v286: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:25:13.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:12 smithi153 ceph-mon[32896]: pgmap v286: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:25:13.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:12 smithi027 ceph-mon[29208]: pgmap v286: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:25:15.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:14 smithi139 ceph-mon[32833]: pgmap v287: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:25:15.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:14 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:25:15.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:14 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:25:15.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:14 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:25:15.004 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:14 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:25:15.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:14 smithi153 ceph-mon[32896]: pgmap v287: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:25:15.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:14 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:25:15.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:14 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:25:15.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:14 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:25:15.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:14 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:25:15.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:14 smithi027 ceph-mon[29208]: pgmap v287: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:25:15.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:14 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:25:15.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:14 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:25:15.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:14 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:25:15.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:14 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:25:16.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:15 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:25:16.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:15 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:25:16.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:15 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:25:17.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:16 smithi139 ceph-mon[32833]: pgmap v288: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:25:17.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:16 smithi153 ceph-mon[32896]: pgmap v288: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:25:17.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:16 smithi027 ceph-mon[29208]: pgmap v288: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-25T00:25:18.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:17 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:25:18.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:17 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:25:18.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:17 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:25:19.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:18 smithi153 ceph-mon[32896]: pgmap v289: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:25:19.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:18 smithi027 ceph-mon[29208]: pgmap v289: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:25:19.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:18 smithi139 ceph-mon[32833]: pgmap v289: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:25:21.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:20 smithi153 ceph-mon[32896]: pgmap v290: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T00:25:21.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:20 smithi027 ceph-mon[29208]: pgmap v290: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T00:25:21.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:20 smithi139 ceph-mon[32833]: pgmap v290: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T00:25:23.055 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:22 smithi139 ceph-mon[32833]: pgmap v291: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:25:23.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:22 smithi027 ceph-mon[29208]: pgmap v291: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:25:23.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:22 smithi153 ceph-mon[32896]: pgmap v291: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T00:25:25.079 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:24 smithi153 ceph-mon[32896]: pgmap v292: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:25:25.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:24 smithi027 ceph-mon[29208]: pgmap v292: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:25:25.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:24 smithi139 ceph-mon[32833]: pgmap v292: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:25:27.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:26 smithi027 ceph-mon[29208]: pgmap v293: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:25:27.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:26 smithi139 ceph-mon[32833]: pgmap v293: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:25:27.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:26 smithi153 ceph-mon[32896]: pgmap v293: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:25:29.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:28 smithi027 ceph-mon[29208]: pgmap v294: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-25T00:25:29.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:28 smithi139 ceph-mon[32833]: pgmap v294: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-25T00:25:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:28 smithi153 ceph-mon[32896]: pgmap v294: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-25T00:25:29.794 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:25:29.794 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:25:29.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.909+0000 7f62f2329640 1 -- 172.21.15.27:0/1381118997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62ec101710 msgr2=0x7f62ec101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:29.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.909+0000 7f62f2329640 1 --2- 172.21.15.27:0/1381118997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62ec101710 0x7f62ec101b90 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009930 tx=0x7f62e002a420 comp rx=0 tx=0).stop 2024-04-25T00:25:29.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.909+0000 7f62f2329640 1 -- 172.21.15.27:0/1381118997 shutdown_connections 2024-04-25T00:25:29.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.909+0000 7f62f2329640 1 --2- 172.21.15.27:0/1381118997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62ec1020d0 0x7f62ec10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.909+0000 7f62f2329640 1 --2- 172.21.15.27:0/1381118997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62ec101710 0x7f62ec101b90 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.909+0000 7f62f2329640 1 --2- 172.21.15.27:0/1381118997 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f62ec107720 0x7f62ec107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.909+0000 7f62f2329640 1 -- 172.21.15.27:0/1381118997 >> 172.21.15.27:0/1381118997 conn(0x7f62ec0fd180 msgr2=0x7f62ec0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:29.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.909+0000 7f62f2329640 1 -- 172.21.15.27:0/1381118997 shutdown_connections 2024-04-25T00:25:29.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.909+0000 7f62f2329640 1 -- 172.21.15.27:0/1381118997 wait complete. 2024-04-25T00:25:29.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.910+0000 7f62f2329640 1 Processor -- start 2024-04-25T00:25:29.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.910+0000 7f62f2329640 1 -- start start 2024-04-25T00:25:29.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.910+0000 7f62f2329640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62ec101710 0x7f62ec1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.910+0000 7f62f2329640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f62ec1020d0 0x7f62ec198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.910+0000 7f62f2329640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62ec107720 0x7f62ec19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.910+0000 7f62f2329640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec111b80 con 0x7f62ec101710 2024-04-25T00:25:29.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.910+0000 7f62f2329640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec111a00 con 0x7f62ec1020d0 2024-04-25T00:25:29.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.910+0000 7f62f2329640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec111d00 con 0x7f62ec107720 2024-04-25T00:25:29.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.910+0000 7f62ebfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62ec101710 0x7f62ec1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.910+0000 7f62ebfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62ec101710 0x7f62ec1980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52886/0 (socket says 172.21.15.27:52886) 2024-04-25T00:25:29.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.910+0000 7f62ebfff640 1 -- 172.21.15.27:0/2591272932 learned_addr learned my addr 172.21.15.27:0/2591272932 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:29.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.910+0000 7f62eb7fe640 1 --2- 172.21.15.27:0/2591272932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f62ec1020d0 0x7f62ec198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.910+0000 7f62f08c6640 1 --2- 172.21.15.27:0/2591272932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62ec107720 0x7f62ec19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.911+0000 7f62ebfff640 1 -- 172.21.15.27:0/2591272932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62ec107720 msgr2=0x7f62ec19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:29.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.911+0000 7f62ebfff640 1 --2- 172.21.15.27:0/2591272932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62ec107720 0x7f62ec19c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.911+0000 7f62ebfff640 1 -- 172.21.15.27:0/2591272932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f62ec1020d0 msgr2=0x7f62ec198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:29.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.911+0000 7f62ebfff640 1 --2- 172.21.15.27:0/2591272932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f62ec1020d0 0x7f62ec198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.911+0000 7f62ebfff640 1 -- 172.21.15.27:0/2591272932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec19d0b0 con 0x7f62ec101710 2024-04-25T00:25:29.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.911+0000 7f62f08c6640 1 --2- 172.21.15.27:0/2591272932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62ec107720 0x7f62ec19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:25:29.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.911+0000 7f62ebfff640 1 --2- 172.21.15.27:0/2591272932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62ec101710 0x7f62ec1980c0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f62dc00e9f0 tx=0x7f62dc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:29.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.911+0000 7f62e97fa640 1 -- 172.21.15.27:0/2591272932 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc00cd80 con 0x7f62ec101710 2024-04-25T00:25:29.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.911+0000 7f62f2329640 1 -- 172.21.15.27:0/2591272932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ec19d3a0 con 0x7f62ec101710 2024-04-25T00:25:29.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.911+0000 7f62e97fa640 1 -- 172.21.15.27:0/2591272932 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62dc010910 con 0x7f62ec101710 2024-04-25T00:25:29.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.911+0000 7f62e97fa640 1 -- 172.21.15.27:0/2591272932 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0188f0 con 0x7f62ec101710 2024-04-25T00:25:29.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.911+0000 7f62f2329640 1 -- 172.21.15.27:0/2591272932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ec1a4c30 con 0x7f62ec101710 2024-04-25T00:25:29.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.913+0000 7f62e97fa640 1 -- 172.21.15.27:0/2591272932 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f62dc010ac0 con 0x7f62ec101710 2024-04-25T00:25:29.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.914+0000 7f62f5036740 1 Processor -- start 2024-04-25T00:25:29.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.914+0000 7f62f5036740 1 -- start start 2024-04-25T00:25:29.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.914+0000 7f62f5036740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391630 0x556695391a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.914+0000 7f62f5036740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556695391f50 0x55669539c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.914+0000 7f62f5036740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55669539c620 0x55669539ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.914+0000 7f62f5036740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5566952f66b0 con 0x55669539c620 2024-04-25T00:25:29.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.914+0000 7f62f5036740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55669529e880 con 0x556695391630 2024-04-25T00:25:29.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.914+0000 7f62f5036740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5566952f2670 con 0x556695391f50 2024-04-25T00:25:29.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.914+0000 7f62eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55669539c620 0x55669539ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.914+0000 7f62eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55669539c620 0x55669539ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52898/0 (socket says 172.21.15.27:52898) 2024-04-25T00:25:29.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.914+0000 7f62eb7fe640 1 -- 172.21.15.27:0/3199736955 learned_addr learned my addr 172.21.15.27:0/3199736955 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.914+0000 7f62f08c6640 1 --2- 172.21.15.27:0/3199736955 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391630 0x556695391a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.914+0000 7f62e97fa640 1 --2- 172.21.15.27:0/2591272932 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f62d40a2d00 0x7f62d40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.914+0000 7f62ebfff640 1 --2- 172.21.15.27:0/3199736955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556695391f50 0x55669539c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.914+0000 7f62f08c6640 1 --2- 172.21.15.27:0/2591272932 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f62d40a2d00 0x7f62d40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.915+0000 7f62f08c6640 1 -- 172.21.15.27:0/3199736955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556695391f50 msgr2=0x55669539c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.915+0000 7f62f08c6640 1 --2- 172.21.15.27:0/3199736955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556695391f50 0x55669539c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.915+0000 7f62e97fa640 1 -- 172.21.15.27:0/2591272932 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f62dc0cfbf0 con 0x7f62ec101710 2024-04-25T00:25:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.915+0000 7f62f08c6640 1 -- 172.21.15.27:0/3199736955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55669539c620 msgr2=0x55669539ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.915+0000 7f62f08c6640 1 --2- 172.21.15.27:0/3199736955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55669539c620 0x55669539ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.915+0000 7f62f08c6640 1 -- 172.21.15.27:0/3199736955 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566950d47d0 con 0x556695391630 2024-04-25T00:25:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.915+0000 7f62eb7fe640 1 --2- 172.21.15.27:0/3199736955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55669539c620 0x55669539ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:25:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.915+0000 7f62f08c6640 1 --2- 172.21.15.27:0/3199736955 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391630 0x556695391a10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f62e40149c0 tx=0x7f62e4014e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8bac7daba46451b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.915+0000 7f62f08c6640 1 --2- 172.21.15.27:0/2591272932 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f62d40a2d00 0x7f62d40a51c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f62e401c000 tx=0x7f62e401e000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.915+0000 7f62d27fc640 1 -- 172.21.15.27:0/3199736955 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e4014050 con 0x556695391630 2024-04-25T00:25:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.915+0000 7f62d27fc640 1 -- 172.21.15.27:0/3199736955 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62e400cd90 con 0x556695391630 2024-04-25T00:25:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.915+0000 7f62d27fc640 1 -- 172.21.15.27:0/3199736955 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e40108d0 con 0x556695391630 2024-04-25T00:25:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.916+0000 7f62f5036740 1 -- 172.21.15.27:0/3199736955 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391630 msgr2=0x556695391a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.916+0000 7f62f5036740 1 --2- 172.21.15.27:0/3199736955 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391630 0x556695391a10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f62e40149c0 tx=0x7f62e4014e80 comp rx=0 tx=0).stop 2024-04-25T00:25:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.916+0000 7f62f5036740 1 -- 172.21.15.27:0/3199736955 shutdown_connections 2024-04-25T00:25:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.916+0000 7f62f5036740 1 --2- 172.21.15.27:0/3199736955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55669539c620 0x55669539ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.916+0000 7f62f5036740 1 --2- 172.21.15.27:0/3199736955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556695391f50 0x55669539c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.916+0000 7f62f5036740 1 --2- 172.21.15.27:0/3199736955 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391630 0x556695391a10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.916+0000 7f62f5036740 1 -- 172.21.15.27:0/3199736955 >> 172.21.15.27:0/3199736955 conn(0x55669538d950 msgr2=0x55669538dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.916+0000 7f62f5036740 1 -- 172.21.15.27:0/3199736955 shutdown_connections 2024-04-25T00:25:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.916+0000 7f62f5036740 1 -- 172.21.15.27:0/3199736955 wait complete. 2024-04-25T00:25:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.916+0000 7f62f5036740 1 -- start start 2024-04-25T00:25:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62f5036740 1 Processor -- start 2024-04-25T00:25:29.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62f5036740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556695391630 0x5566953995d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62f5036740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556695391f50 0x556695399b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62f5036740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55669539c620 0x55669539a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62f5036740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55669539f3b0 con 0x556695391f50 2024-04-25T00:25:29.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62f5036740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5566952b1f50 con 0x55669539c620 2024-04-25T00:25:29.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62f5036740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55669539f530 con 0x556695391630 2024-04-25T00:25:29.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62f5036740 10 client.0 _ll_register_callbacks cb 0x5566952ed000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:25:29.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556695391f50 0x556695399b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556695391f50 0x556695399b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52904/0 (socket says 172.21.15.27:52904) 2024-04-25T00:25:29.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62f08c6640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55669539c620 0x55669539a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62eb7fe640 1 -- 172.21.15.27:0/1192526813 learned_addr learned my addr 172.21.15.27:0/1192526813 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62ebfff640 1 --2- 172.21.15.27:0/1192526813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556695391630 0x5566953995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62eb7fe640 1 -- 172.21.15.27:0/1192526813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556695391630 msgr2=0x5566953995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62eb7fe640 1 --2- 172.21.15.27:0/1192526813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556695391630 0x5566953995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62eb7fe640 1 -- 172.21.15.27:0/1192526813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55669539c620 msgr2=0x55669539a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62eb7fe640 1 --2- 172.21.15.27:0/1192526813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55669539c620 0x55669539a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62eb7fe640 1 -- 172.21.15.27:0/1192526813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55669500a3f0 con 0x556695391f50 2024-04-25T00:25:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62f08c6640 1 --2- 172.21.15.27:0/1192526813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55669539c620 0x55669539a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:25:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62eb7fe640 1 --2- 172.21.15.27:0/1192526813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556695391f50 0x556695399b10 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009230 tx=0x7f62e00043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b359046464fd77b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.917+0000 7f62cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:25:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.918+0000 7f62cb7fe640 1 -- 172.21.15.27:0/1192526813 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e0035040 con 0x556695391f50 2024-04-25T00:25:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.918+0000 7f62cb7fe640 1 -- 172.21.15.27:0/1192526813 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f62e0005450 con 0x556695391f50 2024-04-25T00:25:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.918+0000 7f62f5036740 1 -- 172.21.15.27:0/1192526813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62e400cd90 con 0x556695391f50 2024-04-25T00:25:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.918+0000 7f62cb7fe640 1 -- 172.21.15.27:0/1192526813 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f62e004f050 con 0x556695391f50 2024-04-25T00:25:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.918+0000 7f62f5036740 10 client.17127 fetch_fsmap learned FSMap version 100 2024-04-25T00:25:29.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.918+0000 7f62f5036740 1 -- 172.21.15.27:0/1192526813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x556694fca0f0 con 0x556695391f50 2024-04-25T00:25:29.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.918+0000 7f62cb7fe640 1 -- 172.21.15.27:0/1192526813 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62e400cd90 con 0x556695391f50 2024-04-25T00:25:29.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.918+0000 7f62cb7fe640 1 -- 172.21.15.27:0/1192526813 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 100) ==== 8366+0+0 (secure 0 0 0) 0x7f62e00318f0 con 0x556695391f50 2024-04-25T00:25:29.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.918+0000 7f62f5036740 10 client.17127 fetch_fsmap finished waiting for FSMap version 100 2024-04-25T00:25:29.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.918+0000 7f62f5036740 10 client.17127 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:25:29.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.918+0000 7f62f5036740 20 client.17127 populate_metadata read hostname 'smithi027' 2024-04-25T00:25:29.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.918+0000 7f62f5036740 1 --2- 172.21.15.27:0/1192526813 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5566953ac8f0 0x5566953aed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.918+0000 7f62f5036740 4 client.17127 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:25:29.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.918+0000 7f62f5036740 1 -- 172.21.15.27:0/1192526813 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556695303bf0 con 0x5566953ac8f0 2024-04-25T00:25:29.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.919+0000 7f62ebfff640 1 --2- 172.21.15.27:0/1192526813 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5566953ac8f0 0x5566953aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.919+0000 7f62cb7fe640 10 client.17127 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:25:29.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.919+0000 7f62ebfff640 1 --2- 172.21.15.27:0/1192526813 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5566953ac8f0 0x5566953aed30 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c3e45d667be484f server_cookie=9ce9c915fd7ccedb in_seq=0 out_seq=0 2024-04-25T00:25:29.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.920+0000 7f62cb7fe640 1 -- 172.21.15.27:0/1192526813 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556695303bf0 con 0x5566953ac8f0 2024-04-25T00:25:29.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.920+0000 7f62cb7fe640 10 client.17127 handle_command_reply: tid=0 2024-04-25T00:25:29.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.920+0000 7f62f5036740 1 client.17127 shutdown 2024-04-25T00:25:29.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.920+0000 7f62f5036740 1 -- 172.21.15.27:0/1192526813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556695391f50 msgr2=0x556695399b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:29.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.920+0000 7f62f5036740 1 --2- 172.21.15.27:0/1192526813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556695391f50 0x556695399b10 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009230 tx=0x7f62e00043b0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.920+0000 7f62f5036740 1 -- 172.21.15.27:0/1192526813 shutdown_connections 2024-04-25T00:25:29.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.920+0000 7f62f5036740 1 --2- 172.21.15.27:0/1192526813 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x5566953ac8f0 0x5566953aed30 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.920+0000 7f62f5036740 1 --2- 172.21.15.27:0/1192526813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55669539c620 0x55669539a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.920+0000 7f62cb7fe640 0 client.17127 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:25:29.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.920+0000 7f62f5036740 1 --2- 172.21.15.27:0/1192526813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556695391f50 0x556695399b10 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.920+0000 7f62f5036740 1 --2- 172.21.15.27:0/1192526813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556695391630 0x5566953995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.920+0000 7f62f5036740 1 -- 172.21.15.27:0/1192526813 >> 172.21.15.27:0/1192526813 conn(0x55669538d950 msgr2=0x556695393750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:29.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.920+0000 7f62f5036740 1 -- 172.21.15.27:0/1192526813 shutdown_connections 2024-04-25T00:25:29.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.920+0000 7f62f5036740 1 -- 172.21.15.27:0/1192526813 wait complete. 2024-04-25T00:25:29.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.920+0000 7f62f5036740 20 client.17127 trim_cache size 0 max 16384 2024-04-25T00:25:29.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62f5036740 1 Processor -- start 2024-04-25T00:25:29.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62f5036740 1 -- start start 2024-04-25T00:25:29.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62f5036740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556695391630 0x556695398510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62f5036740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391f50 0x5566953a82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62f5036740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55669539c620 0x556695399e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62f5036740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec1aeba0 con 0x55669539c620 2024-04-25T00:25:29.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62f5036740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec1b0710 con 0x556695391f50 2024-04-25T00:25:29.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62f5036740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec1af270 con 0x556695391630 2024-04-25T00:25:29.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62f08c6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55669539c620 0x556695399e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62f08c6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55669539c620 0x556695399e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52912/0 (socket says 172.21.15.27:52912) 2024-04-25T00:25:29.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62f08c6640 1 -- 172.21.15.27:0/1017571996 learned_addr learned my addr 172.21.15.27:0/1017571996 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:29.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62eb7fe640 1 --2- 172.21.15.27:0/1017571996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391f50 0x5566953a82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62ebfff640 1 --2- 172.21.15.27:0/1017571996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556695391630 0x556695398510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62f08c6640 1 -- 172.21.15.27:0/1017571996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556695391630 msgr2=0x556695398510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:29.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62f08c6640 1 --2- 172.21.15.27:0/1017571996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556695391630 0x556695398510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62f08c6640 1 -- 172.21.15.27:0/1017571996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391f50 msgr2=0x5566953a82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:29.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62f08c6640 1 --2- 172.21.15.27:0/1017571996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391f50 0x5566953a82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62f08c6640 1 -- 172.21.15.27:0/1017571996 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556695387f00 con 0x55669539c620 2024-04-25T00:25:29.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62eb7fe640 1 --2- 172.21.15.27:0/1017571996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391f50 0x5566953a82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:25:29.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.938+0000 7f62ebfff640 1 --2- 172.21.15.27:0/1017571996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556695391630 0x556695398510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:25:29.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.939+0000 7f62f08c6640 1 --2- 172.21.15.27:0/1017571996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55669539c620 0x556695399e70 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f62e40095f0 tx=0x7f62e4009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f16ffaa4194f22 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:29.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.939+0000 7f62d1ffb640 1 -- 172.21.15.27:0/1017571996 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e4014050 con 0x55669539c620 2024-04-25T00:25:29.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.939+0000 7f62d1ffb640 1 -- 172.21.15.27:0/1017571996 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62e402d030 con 0x55669539c620 2024-04-25T00:25:29.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.939+0000 7f62d1ffb640 1 -- 172.21.15.27:0/1017571996 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e400ccd0 con 0x55669539c620 2024-04-25T00:25:29.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.939+0000 7f62f5036740 1 -- 172.21.15.27:0/1017571996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55669539c620 msgr2=0x556695399e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:29.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.939+0000 7f62f5036740 1 --2- 172.21.15.27:0/1017571996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55669539c620 0x556695399e70 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f62e40095f0 tx=0x7f62e4009f60 comp rx=0 tx=0).stop 2024-04-25T00:25:29.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.939+0000 7f62f5036740 1 -- 172.21.15.27:0/1017571996 shutdown_connections 2024-04-25T00:25:29.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.939+0000 7f62f5036740 1 --2- 172.21.15.27:0/1017571996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55669539c620 0x556695399e70 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.939+0000 7f62f5036740 1 --2- 172.21.15.27:0/1017571996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391f50 0x5566953a82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.939+0000 7f62f5036740 1 --2- 172.21.15.27:0/1017571996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556695391630 0x556695398510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.939+0000 7f62f5036740 1 -- 172.21.15.27:0/1017571996 >> 172.21.15.27:0/1017571996 conn(0x55669538d950 msgr2=0x556695393720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:29.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.939+0000 7f62f5036740 1 -- 172.21.15.27:0/1017571996 shutdown_connections 2024-04-25T00:25:29.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.939+0000 7f62f5036740 1 -- 172.21.15.27:0/1017571996 wait complete. 2024-04-25T00:25:29.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.940+0000 7f62f5036740 1 -- start start 2024-04-25T00:25:29.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.940+0000 7f62f5036740 1 Processor -- start 2024-04-25T00:25:29.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.940+0000 7f62f5036740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391630 0x5566953a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.940+0000 7f62f5036740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556695391f50 0x5566953a00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.940+0000 7f62f5036740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55669539c620 0x5566953a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.940+0000 7f62f5036740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec1b0040 con 0x556695391f50 2024-04-25T00:25:29.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.940+0000 7f62f5036740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec1af940 con 0x556695391630 2024-04-25T00:25:29.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.940+0000 7f62f5036740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec1acb60 con 0x55669539c620 2024-04-25T00:25:29.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.940+0000 7f62f5036740 10 client.0 _ll_register_callbacks cb 0x7f62e0009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:25:29.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.940+0000 7f62eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556695391f50 0x5566953a00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.940+0000 7f62eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556695391f50 0x5566953a00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52928/0 (socket says 172.21.15.27:52928) 2024-04-25T00:25:29.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.940+0000 7f62eb7fe640 1 -- 172.21.15.27:0/249529989 learned_addr learned my addr 172.21.15.27:0/249529989 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:29.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.940+0000 7f62ebfff640 1 --2- 172.21.15.27:0/249529989 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391630 0x5566953a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.941+0000 7f62f08c6640 1 --2- 172.21.15.27:0/249529989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55669539c620 0x5566953a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.941+0000 7f62eb7fe640 1 -- 172.21.15.27:0/249529989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55669539c620 msgr2=0x5566953a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:29.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.941+0000 7f62eb7fe640 1 --2- 172.21.15.27:0/249529989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55669539c620 0x5566953a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.941+0000 7f62eb7fe640 1 -- 172.21.15.27:0/249529989 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391630 msgr2=0x5566953a4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:29.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.941+0000 7f62eb7fe640 1 --2- 172.21.15.27:0/249529989 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391630 0x5566953a4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.941+0000 7f62eb7fe640 1 -- 172.21.15.27:0/249529989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556695411e70 con 0x556695391f50 2024-04-25T00:25:29.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.941+0000 7f62ebfff640 1 --2- 172.21.15.27:0/249529989 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391630 0x5566953a4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:25:29.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.941+0000 7f62eb7fe640 1 --2- 172.21.15.27:0/249529989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556695391f50 0x5566953a00a0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f62e0001eb0 tx=0x7f62e0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c21630317538326 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:29.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.941+0000 7f62cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:25:29.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.941+0000 7f62cb7fe640 1 -- 172.21.15.27:0/249529989 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e0000e60 con 0x556695391f50 2024-04-25T00:25:29.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.941+0000 7f62cb7fe640 1 -- 172.21.15.27:0/249529989 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f62e00315b0 con 0x556695391f50 2024-04-25T00:25:29.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.941+0000 7f62f5036740 1 -- 172.21.15.27:0/249529989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62e402d030 con 0x556695391f50 2024-04-25T00:25:29.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.941+0000 7f62cb7fe640 1 -- 172.21.15.27:0/249529989 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f62e0062050 con 0x556695391f50 2024-04-25T00:25:29.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.941+0000 7f62f5036740 10 client.17133 fetch_fsmap learned FSMap version 100 2024-04-25T00:25:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.941+0000 7f62f5036740 1 -- 172.21.15.27:0/249529989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x556695413070 con 0x556695391f50 2024-04-25T00:25:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.941+0000 7f62cb7fe640 1 -- 172.21.15.27:0/249529989 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62e402d030 con 0x556695391f50 2024-04-25T00:25:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.942+0000 7f62cb7fe640 1 -- 172.21.15.27:0/249529989 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 100) ==== 8366+0+0 (secure 0 0 0) 0x7f62e0003de0 con 0x556695391f50 2024-04-25T00:25:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.942+0000 7f62f5036740 10 client.17133 fetch_fsmap finished waiting for FSMap version 100 2024-04-25T00:25:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.942+0000 7f62f5036740 10 client.17133 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:25:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.942+0000 7f62f5036740 20 client.17133 populate_metadata read hostname 'smithi027' 2024-04-25T00:25:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.942+0000 7f62f5036740 1 --2- 172.21.15.27:0/249529989 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556695413820 0x556695415c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.942+0000 7f62f5036740 4 client.17133 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:25:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.942+0000 7f62f5036740 1 -- 172.21.15.27:0/249529989 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5566952fdf40 con 0x556695413820 2024-04-25T00:25:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.942+0000 7f62ebfff640 1 --2- 172.21.15.27:0/249529989 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556695413820 0x556695415c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.942+0000 7f62cb7fe640 10 client.17133 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:25:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.942+0000 7f62ebfff640 1 --2- 172.21.15.27:0/249529989 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556695413820 0x556695415c40 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39e5a2a63e9cf651 server_cookie=e4f3feb689e2c13d in_seq=0 out_seq=0 2024-04-25T00:25:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.942+0000 7f62cb7fe640 1 -- 172.21.15.27:0/249529989 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5566952fdf40 con 0x556695413820 2024-04-25T00:25:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.942+0000 7f62cb7fe640 10 client.17133 handle_command_reply: tid=0 2024-04-25T00:25:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.942+0000 7f62f5036740 1 client.17133 shutdown 2024-04-25T00:25:29.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f5036740 1 -- 172.21.15.27:0/249529989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556695391f50 msgr2=0x5566953a00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:29.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f5036740 1 --2- 172.21.15.27:0/249529989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556695391f50 0x5566953a00a0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f62e0001eb0 tx=0x7f62e0009260 comp rx=0 tx=0).stop 2024-04-25T00:25:29.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f5036740 1 -- 172.21.15.27:0/249529989 shutdown_connections 2024-04-25T00:25:29.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f5036740 1 --2- 172.21.15.27:0/249529989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55669539c620 0x5566953a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f5036740 1 --2- 172.21.15.27:0/249529989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556695391f50 0x5566953a00a0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f5036740 1 --2- 172.21.15.27:0/249529989 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x556695413820 0x556695415c40 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f5036740 1 --2- 172.21.15.27:0/249529989 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556695391630 0x5566953a4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62cb7fe640 0 client.17133 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:25:29.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f5036740 1 -- 172.21.15.27:0/249529989 >> 172.21.15.27:0/249529989 conn(0x55669538d950 msgr2=0x556695392d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:29.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f5036740 1 -- 172.21.15.27:0/249529989 shutdown_connections 2024-04-25T00:25:29.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f5036740 1 -- 172.21.15.27:0/249529989 wait complete. 2024-04-25T00:25:29.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f5036740 20 client.17133 trim_cache size 0 max 16384 2024-04-25T00:25:29.953 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:25:29.953 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T00:25:29.953 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T00:25:29.953 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:25:29.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f2329640 1 -- 172.21.15.27:0/2591272932 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f62d40a2d00 msgr2=0x7f62d40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:29.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f2329640 1 --2- 172.21.15.27:0/2591272932 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f62d40a2d00 0x7f62d40a51c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f62e401c000 tx=0x7f62e401e000 comp rx=0 tx=0).stop 2024-04-25T00:25:29.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f2329640 1 -- 172.21.15.27:0/2591272932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62ec101710 msgr2=0x7f62ec1980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:29.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f2329640 1 --2- 172.21.15.27:0/2591272932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62ec101710 0x7f62ec1980c0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f62dc00e9f0 tx=0x7f62dc00eeb0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f2329640 1 -- 172.21.15.27:0/2591272932 shutdown_connections 2024-04-25T00:25:29.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f2329640 1 --2- 172.21.15.27:0/2591272932 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f62d40a2d00 0x7f62d40a51c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f2329640 1 --2- 172.21.15.27:0/2591272932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62ec107720 0x7f62ec19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f2329640 1 --2- 172.21.15.27:0/2591272932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f62ec1020d0 0x7f62ec198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f2329640 1 --2- 172.21.15.27:0/2591272932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62ec101710 0x7f62ec1980c0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:29.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f2329640 1 -- 172.21.15.27:0/2591272932 >> 172.21.15.27:0/2591272932 conn(0x7f62ec0fd180 msgr2=0x7f62ec0fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:29.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.943+0000 7f62f2329640 1 -- 172.21.15.27:0/2591272932 shutdown_connections 2024-04-25T00:25:29.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:29.944+0000 7f62f2329640 1 -- 172.21.15.27:0/2591272932 wait complete. 2024-04-25T00:25:29.956 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T00:25:29.956 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:30.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.057+0000 7f6e75314640 1 -- 172.21.15.27:0/59625509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e70101730 msgr2=0x7f6e70101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.057+0000 7f6e75314640 1 --2- 172.21.15.27:0/59625509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e70101730 0x7f6e70101bb0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f6e60009930 tx=0x7f6e6002a420 comp rx=0 tx=0).stop 2024-04-25T00:25:30.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.057+0000 7f6e75314640 1 -- 172.21.15.27:0/59625509 shutdown_connections 2024-04-25T00:25:30.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.057+0000 7f6e75314640 1 --2- 172.21.15.27:0/59625509 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e701020f0 0x7f6e7010e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.057+0000 7f6e75314640 1 --2- 172.21.15.27:0/59625509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e70101730 0x7f6e70101bb0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.057+0000 7f6e75314640 1 --2- 172.21.15.27:0/59625509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e70107740 0x7f6e70107b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.057+0000 7f6e75314640 1 -- 172.21.15.27:0/59625509 >> 172.21.15.27:0/59625509 conn(0x7f6e700fd1e0 msgr2=0x7f6e700ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.057+0000 7f6e75314640 1 -- 172.21.15.27:0/59625509 shutdown_connections 2024-04-25T00:25:30.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.057+0000 7f6e75314640 1 -- 172.21.15.27:0/59625509 wait complete. 2024-04-25T00:25:30.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.057+0000 7f6e75314640 1 Processor -- start 2024-04-25T00:25:30.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.057+0000 7f6e75314640 1 -- start start 2024-04-25T00:25:30.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e75314640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e70101730 0x7f6e701980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e75314640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e701020f0 0x7f6e70198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e75314640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e70107740 0x7f6e7019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e75314640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6e70111b50 con 0x7f6e701020f0 2024-04-25T00:25:30.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e75314640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6e701119d0 con 0x7f6e70101730 2024-04-25T00:25:30.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e75314640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6e70111cd0 con 0x7f6e70107740 2024-04-25T00:25:30.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e6e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e701020f0 0x7f6e70198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e6e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e701020f0 0x7f6e70198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52946/0 (socket says 172.21.15.27:52946) 2024-04-25T00:25:30.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e6e7fc640 1 -- 172.21.15.27:0/2541294640 learned_addr learned my addr 172.21.15.27:0/2541294640 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e6effd640 1 --2- 172.21.15.27:0/2541294640 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e70101730 0x7f6e701980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e6f7fe640 1 --2- 172.21.15.27:0/2541294640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e70107740 0x7f6e7019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e6e7fc640 1 -- 172.21.15.27:0/2541294640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e70107740 msgr2=0x7f6e7019c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e6e7fc640 1 --2- 172.21.15.27:0/2541294640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e70107740 0x7f6e7019c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e6e7fc640 1 -- 172.21.15.27:0/2541294640 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e70101730 msgr2=0x7f6e701980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e6e7fc640 1 --2- 172.21.15.27:0/2541294640 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e70101730 0x7f6e701980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e6e7fc640 1 -- 172.21.15.27:0/2541294640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e7019d070 con 0x7f6e701020f0 2024-04-25T00:25:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e6f7fe640 1 --2- 172.21.15.27:0/2541294640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e70107740 0x7f6e7019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:25:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e6e7fc640 1 --2- 172.21.15.27:0/2541294640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e701020f0 0x7f6e70198610 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f6e60009230 tx=0x7f6e60004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.058+0000 7f6e57fff640 1 -- 172.21.15.27:0/2541294640 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6003f030 con 0x7f6e701020f0 2024-04-25T00:25:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.059+0000 7f6e57fff640 1 -- 172.21.15.27:0/2541294640 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e60003ad0 con 0x7f6e701020f0 2024-04-25T00:25:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.059+0000 7f6e75314640 1 -- 172.21.15.27:0/2541294640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e7019d300 con 0x7f6e701020f0 2024-04-25T00:25:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.059+0000 7f6e57fff640 1 -- 172.21.15.27:0/2541294640 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e60035070 con 0x7f6e701020f0 2024-04-25T00:25:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.059+0000 7f6e75314640 1 -- 172.21.15.27:0/2541294640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e701a4bf0 con 0x7f6e701020f0 2024-04-25T00:25:30.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.059+0000 7f6e75314640 1 -- 172.21.15.27:0/2541294640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e70068560 con 0x7f6e701020f0 2024-04-25T00:25:30.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.061+0000 7f6e57fff640 1 -- 172.21.15.27:0/2541294640 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6e60042030 con 0x7f6e701020f0 2024-04-25T00:25:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.061+0000 7f6e57fff640 1 --2- 172.21.15.27:0/2541294640 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6e580a2d00 0x7f6e580a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.061+0000 7f6e6effd640 1 --2- 172.21.15.27:0/2541294640 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6e580a2d00 0x7f6e580a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.061+0000 7f6e57fff640 1 -- 172.21.15.27:0/2541294640 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f6e600efe00 con 0x7f6e701020f0 2024-04-25T00:25:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.062+0000 7f6e6effd640 1 --2- 172.21.15.27:0/2541294640 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6e580a2d00 0x7f6e580a51c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6e64005fd0 tx=0x7f6e6400b390 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.066+0000 7f6e57fff640 1 -- 172.21.15.27:0/2541294640 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6e600bc6b0 con 0x7f6e701020f0 2024-04-25T00:25:30.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.289+0000 7f6e75314640 1 -- 172.21.15.27:0/2541294640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e70109cf0 con 0x7f6e701020f0 2024-04-25T00:25:30.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.290+0000 7f6e57fff640 1 -- 172.21.15.27:0/2541294640 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+11953 (secure 0 0 0) 0x7f6e600bbfb0 con 0x7f6e701020f0 2024-04-25T00:25:30.291 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:25:30.291 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":100,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:25:29.469285+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":147,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":148,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":146,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24428,24496,24521,24430,24514]},"id":1}]} 2024-04-25T00:25:30.291 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 100 2024-04-25T00:25:30.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.292+0000 7f6e75314640 1 -- 172.21.15.27:0/2541294640 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6e580a2d00 msgr2=0x7f6e580a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.292+0000 7f6e75314640 1 --2- 172.21.15.27:0/2541294640 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6e580a2d00 0x7f6e580a51c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6e64005fd0 tx=0x7f6e6400b390 comp rx=0 tx=0).stop 2024-04-25T00:25:30.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.292+0000 7f6e75314640 1 -- 172.21.15.27:0/2541294640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e701020f0 msgr2=0x7f6e70198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.292+0000 7f6e75314640 1 --2- 172.21.15.27:0/2541294640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e701020f0 0x7f6e70198610 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f6e60009230 tx=0x7f6e60004270 comp rx=0 tx=0).stop 2024-04-25T00:25:30.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.292+0000 7f6e75314640 1 -- 172.21.15.27:0/2541294640 shutdown_connections 2024-04-25T00:25:30.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.292+0000 7f6e75314640 1 --2- 172.21.15.27:0/2541294640 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f6e580a2d00 0x7f6e580a51c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.292+0000 7f6e75314640 1 --2- 172.21.15.27:0/2541294640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e70107740 0x7f6e7019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.292+0000 7f6e75314640 1 --2- 172.21.15.27:0/2541294640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e701020f0 0x7f6e70198610 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.292+0000 7f6e75314640 1 --2- 172.21.15.27:0/2541294640 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e70101730 0x7f6e701980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.292+0000 7f6e75314640 1 -- 172.21.15.27:0/2541294640 >> 172.21.15.27:0/2541294640 conn(0x7f6e700fd1e0 msgr2=0x7f6e700fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.292+0000 7f6e75314640 1 -- 172.21.15.27:0/2541294640 shutdown_connections 2024-04-25T00:25:30.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.292+0000 7f6e75314640 1 -- 172.21.15.27:0/2541294640 wait complete. 2024-04-25T00:25:30.304 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T00:25:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.408+0000 7f096b058640 1 -- 172.21.15.27:0/1453119765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0964100c90 msgr2=0x7f0964104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.408+0000 7f096b058640 1 --2- 172.21.15.27:0/1453119765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0964100c90 0x7f0964104a30 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f0958009900 tx=0x7f095802a410 comp rx=0 tx=0).stop 2024-04-25T00:25:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.408+0000 7f096b058640 1 -- 172.21.15.27:0/1453119765 shutdown_connections 2024-04-25T00:25:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.408+0000 7f096b058640 1 --2- 172.21.15.27:0/1453119765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0964105160 0x7f096410cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.409+0000 7f096b058640 1 --2- 172.21.15.27:0/1453119765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0964100c90 0x7f0964104a30 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.409+0000 7f096b058640 1 --2- 172.21.15.27:0/1453119765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09641002e0 0x7f09641006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.409+0000 7f096b058640 1 -- 172.21.15.27:0/1453119765 >> 172.21.15.27:0/1453119765 conn(0x7f09640758d0 msgr2=0x7f0964075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.409+0000 7f096b058640 1 -- 172.21.15.27:0/1453119765 shutdown_connections 2024-04-25T00:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.409+0000 7f096b058640 1 -- 172.21.15.27:0/1453119765 wait complete. 2024-04-25T00:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.409+0000 7f096b058640 1 Processor -- start 2024-04-25T00:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.409+0000 7f096b058640 1 -- start start 2024-04-25T00:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.409+0000 7f096b058640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09641002e0 0x7f0964197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.409+0000 7f096b058640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0964105160 0x7f0964198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.409+0000 7f096b058640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f096419c8b0 0x7f096419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.409+0000 7f096b058640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f096410f940 con 0x7f096419c8b0 2024-04-25T00:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.409+0000 7f096b058640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f096410f7c0 con 0x7f0964105160 2024-04-25T00:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.409+0000 7f096b058640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f096410fac0 con 0x7f09641002e0 2024-04-25T00:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.409+0000 7f09695f5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f096419c8b0 0x7f096419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.410+0000 7f09695f5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f096419c8b0 0x7f096419cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52966/0 (socket says 172.21.15.27:52966) 2024-04-25T00:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.410+0000 7f0963fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0964105160 0x7f0964198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.410+0000 7f09695f5640 1 -- 172.21.15.27:0/1379374825 learned_addr learned my addr 172.21.15.27:0/1379374825 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.410+0000 7f0968df4640 1 --2- 172.21.15.27:0/1379374825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09641002e0 0x7f0964197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.410+0000 7f09695f5640 1 -- 172.21.15.27:0/1379374825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09641002e0 msgr2=0x7f0964197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.410+0000 7f09695f5640 1 --2- 172.21.15.27:0/1379374825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09641002e0 0x7f0964197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.410+0000 7f09695f5640 1 -- 172.21.15.27:0/1379374825 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0964105160 msgr2=0x7f0964198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.410+0000 7f09695f5640 1 --2- 172.21.15.27:0/1379374825 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0964105160 0x7f0964198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.410+0000 7f09695f5640 1 -- 172.21.15.27:0/1379374825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f096419d3a0 con 0x7f096419c8b0 2024-04-25T00:25:30.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.410+0000 7f09695f5640 1 --2- 172.21.15.27:0/1379374825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f096419c8b0 0x7f096419cd60 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f095c00eb10 tx=0x7f095c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.410+0000 7f0961ffb640 1 -- 172.21.15.27:0/1379374825 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c004b20 con 0x7f096419c8b0 2024-04-25T00:25:30.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.410+0000 7f0961ffb640 1 -- 172.21.15.27:0/1379374825 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f095c004cc0 con 0x7f096419c8b0 2024-04-25T00:25:30.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.410+0000 7f0961ffb640 1 -- 172.21.15.27:0/1379374825 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c010760 con 0x7f096419c8b0 2024-04-25T00:25:30.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.410+0000 7f096b058640 1 -- 172.21.15.27:0/1379374825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f096419d690 con 0x7f096419c8b0 2024-04-25T00:25:30.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.410+0000 7f096b058640 1 -- 172.21.15.27:0/1379374825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09641a4e50 con 0x7f096419c8b0 2024-04-25T00:25:30.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.412+0000 7f0961ffb640 1 -- 172.21.15.27:0/1379374825 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f095c005020 con 0x7f096419c8b0 2024-04-25T00:25:30.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.413+0000 7f096dd64740 1 Processor -- start 2024-04-25T00:25:30.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.413+0000 7f096dd64740 1 -- start start 2024-04-25T00:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.413+0000 7f0961ffb640 1 --2- 172.21.15.27:0/1379374825 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f094c0a2d00 0x7f094c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.413+0000 7f096dd64740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17aeb580 0x55cb17aeb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.413+0000 7f096dd64740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aebea0 0x55cb17af6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.413+0000 7f096dd64740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17af6570 0x55cb17af8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.413+0000 7f096dd64740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cb17a50670 con 0x55cb17aebea0 2024-04-25T00:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.413+0000 7f096dd64740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55cb179f8970 con 0x55cb17af6570 2024-04-25T00:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.413+0000 7f096dd64740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cb17a4c630 con 0x55cb17aeb580 2024-04-25T00:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f0963fff640 1 --2- 172.21.15.27:0/1379374825 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f094c0a2d00 0x7f094c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f0961ffb640 1 -- 172.21.15.27:0/1379374825 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f095c0cfea0 con 0x7f096419c8b0 2024-04-25T00:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f0963fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aebea0 0x55cb17af6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f09695f5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17af6570 0x55cb17af8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f0963fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aebea0 0x55cb17af6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52976/0 (socket says 172.21.15.27:52976) 2024-04-25T00:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f09695f5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17af6570 0x55cb17af8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:51296/0 (socket says 172.21.15.27:51296) 2024-04-25T00:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f0963fff640 1 -- 172.21.15.27:0/3124142685 learned_addr learned my addr 172.21.15.27:0/3124142685 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f0968df4640 1 --2- 172.21.15.27:0/3124142685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17aeb580 0x55cb17aeb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f09695f5640 1 -- 172.21.15.27:0/3124142685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17aeb580 msgr2=0x55cb17aeb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f09695f5640 1 --2- 172.21.15.27:0/3124142685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17aeb580 0x55cb17aeb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f09695f5640 1 -- 172.21.15.27:0/3124142685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aebea0 msgr2=0x55cb17af6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f09695f5640 1 --2- 172.21.15.27:0/3124142685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aebea0 0x55cb17af6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f09695f5640 1 -- 172.21.15.27:0/3124142685 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb17a1c400 con 0x55cb17af6570 2024-04-25T00:25:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f0963fff640 1 --2- 172.21.15.27:0/1379374825 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f094c0a2d00 0x7f094c0a51c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0958005940 tx=0x7f0958005780 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f09695f5640 1 --2- 172.21.15.27:0/3124142685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17af6570 0x55cb17af8960 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f095c0cd240 tx=0x7f095c09bb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79c4fb2cff7a6728 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f094affd640 1 -- 172.21.15.27:0/3124142685 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c04a8f0 con 0x55cb17af6570 2024-04-25T00:25:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f094affd640 1 -- 172.21.15.27:0/3124142685 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f095c09c710 con 0x55cb17af6570 2024-04-25T00:25:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.414+0000 7f094affd640 1 -- 172.21.15.27:0/3124142685 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c04acb0 con 0x55cb17af6570 2024-04-25T00:25:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.415+0000 7f096dd64740 1 -- 172.21.15.27:0/3124142685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17af6570 msgr2=0x55cb17af8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.415+0000 7f096dd64740 1 --2- 172.21.15.27:0/3124142685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17af6570 0x55cb17af8960 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f095c0cd240 tx=0x7f095c09bb50 comp rx=0 tx=0).stop 2024-04-25T00:25:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.415+0000 7f096dd64740 1 -- 172.21.15.27:0/3124142685 shutdown_connections 2024-04-25T00:25:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.415+0000 7f096dd64740 1 --2- 172.21.15.27:0/3124142685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17af6570 0x55cb17af8960 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.415+0000 7f096dd64740 1 --2- 172.21.15.27:0/3124142685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aebea0 0x55cb17af6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.415+0000 7f096dd64740 1 --2- 172.21.15.27:0/3124142685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17aeb580 0x55cb17aeb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.415+0000 7f096dd64740 1 -- 172.21.15.27:0/3124142685 >> 172.21.15.27:0/3124142685 conn(0x55cb17ae78a0 msgr2=0x55cb17ae7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.415+0000 7f096dd64740 1 -- 172.21.15.27:0/3124142685 shutdown_connections 2024-04-25T00:25:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.415+0000 7f096dd64740 1 -- 172.21.15.27:0/3124142685 wait complete. 2024-04-25T00:25:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.415+0000 7f096dd64740 1 -- start start 2024-04-25T00:25:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.415+0000 7f096dd64740 1 Processor -- start 2024-04-25T00:25:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f096dd64740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17af5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f096dd64740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17aebea0 0x55cb17af30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f096dd64740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17af6570 0x55cb17af3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f096dd64740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cb17af9300 con 0x55cb17aeb580 2024-04-25T00:25:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f096dd64740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55cb17a0c050 con 0x55cb17aebea0 2024-04-25T00:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f096dd64740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cb17af9480 con 0x55cb17af6570 2024-04-25T00:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f096dd64740 10 client.0 _ll_register_callbacks cb 0x55cb17763be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f0968df4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17af5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f0968df4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17af5e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52992/0 (socket says 172.21.15.27:52992) 2024-04-25T00:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f0968df4640 1 -- 172.21.15.27:0/3899540905 learned_addr learned my addr 172.21.15.27:0/3899540905 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f0963fff640 1 --2- 172.21.15.27:0/3899540905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17aebea0 0x55cb17af30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f09695f5640 1 --2- 172.21.15.27:0/3899540905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17af6570 0x55cb17af3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f0968df4640 1 -- 172.21.15.27:0/3899540905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17af6570 msgr2=0x55cb17af3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f0968df4640 1 --2- 172.21.15.27:0/3899540905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17af6570 0x55cb17af3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f0968df4640 1 -- 172.21.15.27:0/3899540905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17aebea0 msgr2=0x55cb17af30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f0968df4640 1 --2- 172.21.15.27:0/3899540905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17aebea0 0x55cb17af30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f0968df4640 1 -- 172.21.15.27:0/3899540905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb179dddc0 con 0x55cb17aeb580 2024-04-25T00:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f0968df4640 1 --2- 172.21.15.27:0/3899540905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17af5e00 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f095400e740 tx=0x7f095400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f176d05999007b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.416+0000 7f0943fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.417+0000 7f0943fff640 1 -- 172.21.15.27:0/3899540905 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095401e060 con 0x55cb17aeb580 2024-04-25T00:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.417+0000 7f096dd64740 1 -- 172.21.15.27:0/3899540905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f095c09c710 con 0x55cb17aeb580 2024-04-25T00:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.417+0000 7f0943fff640 1 -- 172.21.15.27:0/3899540905 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f0954004260 con 0x55cb17aeb580 2024-04-25T00:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.417+0000 7f0943fff640 1 -- 172.21.15.27:0/3899540905 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f0954025050 con 0x55cb17aeb580 2024-04-25T00:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.417+0000 7f096dd64740 10 client.17151 fetch_fsmap learned FSMap version 100 2024-04-25T00:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.417+0000 7f096dd64740 1 -- 172.21.15.27:0/3899540905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x55cb177643f0 con 0x55cb17aeb580 2024-04-25T00:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.417+0000 7f0943fff640 1 -- 172.21.15.27:0/3899540905 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f095c09c710 con 0x55cb17aeb580 2024-04-25T00:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.417+0000 7f0943fff640 1 -- 172.21.15.27:0/3899540905 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 100) ==== 8366+0+0 (secure 0 0 0) 0x7f095400b040 con 0x55cb17aeb580 2024-04-25T00:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.417+0000 7f096dd64740 10 client.17151 fetch_fsmap finished waiting for FSMap version 100 2024-04-25T00:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.417+0000 7f096dd64740 10 client.17151 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.417+0000 7f096dd64740 20 client.17151 populate_metadata read hostname 'smithi027' 2024-04-25T00:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.417+0000 7f096dd64740 1 --2- 172.21.15.27:0/3899540905 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55cb17b06520 0x55cb17b089b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.417+0000 7f096dd64740 4 client.17151 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.417+0000 7f096dd64740 1 -- 172.21.15.27:0/3899540905 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb17a5dbb0 con 0x55cb17b06520 2024-04-25T00:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.417+0000 7f0963fff640 1 --2- 172.21.15.27:0/3899540905 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55cb17b06520 0x55cb17b089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.418+0000 7f0963fff640 1 --2- 172.21.15.27:0/3899540905 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55cb17b06520 0x55cb17b089b0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6f431f2c825c82bc server_cookie=ebf5411bb539a89e in_seq=0 out_seq=0 2024-04-25T00:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.418+0000 7f0943fff640 10 client.17151 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:25:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.418+0000 7f0943fff640 1 -- 172.21.15.27:0/3899540905 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cb17a5dbb0 con 0x55cb17b06520 2024-04-25T00:25:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.418+0000 7f0943fff640 10 client.17151 handle_command_reply: tid=0 2024-04-25T00:25:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.418+0000 7f096dd64740 1 client.17151 shutdown 2024-04-25T00:25:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.418+0000 7f096dd64740 1 -- 172.21.15.27:0/3899540905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 msgr2=0x55cb17af5e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.418+0000 7f096dd64740 1 --2- 172.21.15.27:0/3899540905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17af5e00 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f095400e740 tx=0x7f095400ec00 comp rx=0 tx=0).stop 2024-04-25T00:25:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.419+0000 7f096dd64740 1 -- 172.21.15.27:0/3899540905 shutdown_connections 2024-04-25T00:25:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.419+0000 7f096dd64740 1 --2- 172.21.15.27:0/3899540905 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55cb17b06520 0x55cb17b089b0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.419+0000 7f096dd64740 1 --2- 172.21.15.27:0/3899540905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17af6570 0x55cb17af3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.419+0000 7f096dd64740 1 --2- 172.21.15.27:0/3899540905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17aebea0 0x55cb17af30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.419+0000 7f0943fff640 0 client.17151 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:25:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.419+0000 7f096dd64740 1 --2- 172.21.15.27:0/3899540905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17af5e00 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.419+0000 7f096dd64740 1 -- 172.21.15.27:0/3899540905 >> 172.21.15.27:0/3899540905 conn(0x55cb17ae78a0 msgr2=0x55cb17ae6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.419+0000 7f096dd64740 1 -- 172.21.15.27:0/3899540905 shutdown_connections 2024-04-25T00:25:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.419+0000 7f096dd64740 1 -- 172.21.15.27:0/3899540905 wait complete. 2024-04-25T00:25:30.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.419+0000 7f096dd64740 20 client.17151 trim_cache size 0 max 16384 2024-04-25T00:25:30.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.432+0000 7f096dd64740 1 Processor -- start 2024-04-25T00:25:30.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.432+0000 7f096dd64740 1 -- start start 2024-04-25T00:25:30.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.432+0000 7f096dd64740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17b021c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.432+0000 7f096dd64740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17aebea0 0x55cb17aee6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.432+0000 7f096dd64740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17af6570 0x55cb17af5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.432+0000 7f096dd64740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f09641aeb00 con 0x55cb17aeb580 2024-04-25T00:25:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.432+0000 7f096dd64740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f09641b0670 con 0x55cb17af6570 2024-04-25T00:25:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.432+0000 7f096dd64740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f09641af1d0 con 0x55cb17aebea0 2024-04-25T00:25:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.432+0000 7f0968df4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17b021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.432+0000 7f0968df4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17b021c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53002/0 (socket says 172.21.15.27:53002) 2024-04-25T00:25:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.432+0000 7f0968df4640 1 -- 172.21.15.27:0/1339123380 learned_addr learned my addr 172.21.15.27:0/1339123380 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.432+0000 7f09695f5640 1 --2- 172.21.15.27:0/1339123380 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17af6570 0x55cb17af5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.432+0000 7f0963fff640 1 --2- 172.21.15.27:0/1339123380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17aebea0 0x55cb17aee6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.433+0000 7f0968df4640 1 -- 172.21.15.27:0/1339123380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17aebea0 msgr2=0x55cb17aee6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.433+0000 7f0968df4640 1 --2- 172.21.15.27:0/1339123380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17aebea0 0x55cb17aee6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.433+0000 7f0968df4640 1 -- 172.21.15.27:0/1339123380 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17af6570 msgr2=0x55cb17af5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.433+0000 7f0968df4640 1 --2- 172.21.15.27:0/1339123380 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17af6570 0x55cb17af5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.433+0000 7f0968df4640 1 -- 172.21.15.27:0/1339123380 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb17aebcd0 con 0x55cb17aeb580 2024-04-25T00:25:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.433+0000 7f0968df4640 1 --2- 172.21.15.27:0/1339123380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17b021c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f0954003b90 tx=0x7f0954005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d16e03e2d5cdafee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.433+0000 7f094a7fc640 1 -- 172.21.15.27:0/1339123380 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09540035f0 con 0x55cb17aeb580 2024-04-25T00:25:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.433+0000 7f094a7fc640 1 -- 172.21.15.27:0/1339123380 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0954003790 con 0x55cb17aeb580 2024-04-25T00:25:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.433+0000 7f094a7fc640 1 -- 172.21.15.27:0/1339123380 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954019850 con 0x55cb17aeb580 2024-04-25T00:25:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.433+0000 7f096dd64740 1 -- 172.21.15.27:0/1339123380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 msgr2=0x55cb17b021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.433+0000 7f096dd64740 1 --2- 172.21.15.27:0/1339123380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17b021c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f0954003b90 tx=0x7f0954005d30 comp rx=0 tx=0).stop 2024-04-25T00:25:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f096dd64740 1 -- 172.21.15.27:0/1339123380 shutdown_connections 2024-04-25T00:25:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f096dd64740 1 --2- 172.21.15.27:0/1339123380 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17af6570 0x55cb17af5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f096dd64740 1 --2- 172.21.15.27:0/1339123380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17aebea0 0x55cb17aee6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f096dd64740 1 --2- 172.21.15.27:0/1339123380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17b021c0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f096dd64740 1 -- 172.21.15.27:0/1339123380 >> 172.21.15.27:0/1339123380 conn(0x55cb17ae78a0 msgr2=0x55cb17aedfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f096dd64740 1 -- 172.21.15.27:0/1339123380 shutdown_connections 2024-04-25T00:25:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f096dd64740 1 -- 172.21.15.27:0/1339123380 wait complete. 2024-04-25T00:25:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f096dd64740 1 -- start start 2024-04-25T00:25:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f096dd64740 1 Processor -- start 2024-04-25T00:25:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f096dd64740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17afa0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f096dd64740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17aebea0 0x55cb17afa630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f096dd64740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17af6570 0x55cb17afab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f096dd64740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f09641affa0 con 0x55cb17aeb580 2024-04-25T00:25:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f096dd64740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f09641af8a0 con 0x55cb17aebea0 2024-04-25T00:25:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f096dd64740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f09641acac0 con 0x55cb17af6570 2024-04-25T00:25:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f096dd64740 10 client.0 _ll_register_callbacks cb 0x7f095400af20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:25:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f0968df4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17afa0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f0963fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17aebea0 0x55cb17afa630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f0968df4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17afa0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53008/0 (socket says 172.21.15.27:53008) 2024-04-25T00:25:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f0968df4640 1 -- 172.21.15.27:0/2584183071 learned_addr learned my addr 172.21.15.27:0/2584183071 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.434+0000 7f09695f5640 1 --2- 172.21.15.27:0/2584183071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17af6570 0x55cb17afab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.435+0000 7f0968df4640 1 -- 172.21.15.27:0/2584183071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17af6570 msgr2=0x55cb17afab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.435+0000 7f0968df4640 1 --2- 172.21.15.27:0/2584183071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17af6570 0x55cb17afab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.435+0000 7f0968df4640 1 -- 172.21.15.27:0/2584183071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17aebea0 msgr2=0x55cb17afa630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.435+0000 7f0968df4640 1 --2- 172.21.15.27:0/2584183071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17aebea0 0x55cb17afa630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.435+0000 7f0968df4640 1 -- 172.21.15.27:0/2584183071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb1782e7d0 con 0x55cb17aeb580 2024-04-25T00:25:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.435+0000 7f0963fff640 1 --2- 172.21.15.27:0/2584183071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17aebea0 0x55cb17afa630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:25:30.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.435+0000 7f0968df4640 1 --2- 172.21.15.27:0/2584183071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17afa0f0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f095400dd10 tx=0x7f095400d830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd94777fe6146e5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.435+0000 7f0943fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:25:30.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.435+0000 7f0943fff640 1 -- 172.21.15.27:0/2584183071 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095401e040 con 0x55cb17aeb580 2024-04-25T00:25:30.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.435+0000 7f0943fff640 1 -- 172.21.15.27:0/2584183071 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f09540052e0 con 0x55cb17aeb580 2024-04-25T00:25:30.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.435+0000 7f096dd64740 1 -- 172.21.15.27:0/2584183071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0954003790 con 0x55cb17aeb580 2024-04-25T00:25:30.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.435+0000 7f0943fff640 1 -- 172.21.15.27:0/2584183071 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f0954041050 con 0x55cb17aeb580 2024-04-25T00:25:30.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.435+0000 7f096dd64740 10 client.17157 fetch_fsmap learned FSMap version 100 2024-04-25T00:25:30.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.435+0000 7f096dd64740 1 -- 172.21.15.27:0/2584183071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x55cb17b6cc00 con 0x55cb17aeb580 2024-04-25T00:25:30.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.435+0000 7f0943fff640 1 -- 172.21.15.27:0/2584183071 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0954003790 con 0x55cb17aeb580 2024-04-25T00:25:30.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f0943fff640 1 -- 172.21.15.27:0/2584183071 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 100) ==== 8366+0+0 (secure 0 0 0) 0x7f0954038070 con 0x55cb17aeb580 2024-04-25T00:25:30.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f096dd64740 10 client.17157 fetch_fsmap finished waiting for FSMap version 100 2024-04-25T00:25:30.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f096dd64740 10 client.17157 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:25:30.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f096dd64740 20 client.17157 populate_metadata read hostname 'smithi027' 2024-04-25T00:25:30.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f096dd64740 1 --2- 172.21.15.27:0/2584183071 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55cb17b6d1b0 0x55cb17b6f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f096dd64740 4 client.17157 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:25:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f096dd64740 1 -- 172.21.15.27:0/2584183071 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cb17a57f00 con 0x55cb17b6d1b0 2024-04-25T00:25:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f0963fff640 1 --2- 172.21.15.27:0/2584183071 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55cb17b6d1b0 0x55cb17b6f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f0963fff640 1 --2- 172.21.15.27:0/2584183071 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55cb17b6d1b0 0x55cb17b6f5a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b76dd865bb7f88d4 server_cookie=6b87e914a19093f8 in_seq=0 out_seq=0 2024-04-25T00:25:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f0943fff640 10 client.17157 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:25:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f0943fff640 1 -- 172.21.15.27:0/2584183071 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cb17a57f00 con 0x55cb17b6d1b0 2024-04-25T00:25:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f0943fff640 10 client.17157 handle_command_reply: tid=0 2024-04-25T00:25:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f096dd64740 1 client.17157 shutdown 2024-04-25T00:25:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f096dd64740 1 -- 172.21.15.27:0/2584183071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 msgr2=0x55cb17afa0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f096dd64740 1 --2- 172.21.15.27:0/2584183071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17afa0f0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f095400dd10 tx=0x7f095400d830 comp rx=0 tx=0).stop 2024-04-25T00:25:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f096dd64740 1 -- 172.21.15.27:0/2584183071 shutdown_connections 2024-04-25T00:25:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f096dd64740 1 --2- 172.21.15.27:0/2584183071 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x55cb17b6d1b0 0x55cb17b6f5a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f096dd64740 1 --2- 172.21.15.27:0/2584183071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cb17af6570 0x55cb17afab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.436+0000 7f096dd64740 1 --2- 172.21.15.27:0/2584183071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cb17aebea0 0x55cb17afa630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f0943fff640 0 client.17157 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:25:30.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f096dd64740 1 --2- 172.21.15.27:0/2584183071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb17aeb580 0x55cb17afa0f0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f096dd64740 1 -- 172.21.15.27:0/2584183071 >> 172.21.15.27:0/2584183071 conn(0x55cb17ae78a0 msgr2=0x55cb17ae6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f096dd64740 1 -- 172.21.15.27:0/2584183071 shutdown_connections 2024-04-25T00:25:30.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f096dd64740 1 -- 172.21.15.27:0/2584183071 wait complete. 2024-04-25T00:25:30.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f096dd64740 20 client.17157 trim_cache size 0 max 16384 2024-04-25T00:25:30.444 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:25:30.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f096b058640 1 -- 172.21.15.27:0/1379374825 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f094c0a2d00 msgr2=0x7f094c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f096b058640 1 --2- 172.21.15.27:0/1379374825 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f094c0a2d00 0x7f094c0a51c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0958005940 tx=0x7f0958005780 comp rx=0 tx=0).stop 2024-04-25T00:25:30.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f096b058640 1 -- 172.21.15.27:0/1379374825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f096419c8b0 msgr2=0x7f096419cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f096b058640 1 --2- 172.21.15.27:0/1379374825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f096419c8b0 0x7f096419cd60 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f095c00eb10 tx=0x7f095c00c2d0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f096b058640 1 -- 172.21.15.27:0/1379374825 shutdown_connections 2024-04-25T00:25:30.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f096b058640 1 --2- 172.21.15.27:0/1379374825 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f094c0a2d00 0x7f094c0a51c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f096b058640 1 --2- 172.21.15.27:0/1379374825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f096419c8b0 0x7f096419cd60 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f096b058640 1 --2- 172.21.15.27:0/1379374825 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0964105160 0x7f0964198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f096b058640 1 --2- 172.21.15.27:0/1379374825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09641002e0 0x7f0964197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f096b058640 1 -- 172.21.15.27:0/1379374825 >> 172.21.15.27:0/1379374825 conn(0x7f09640758d0 msgr2=0x7f09640fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f096b058640 1 -- 172.21.15.27:0/1379374825 shutdown_connections 2024-04-25T00:25:30.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.437+0000 7f096b058640 1 -- 172.21.15.27:0/1379374825 wait complete. 2024-04-25T00:25:30.449 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T00:25:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.552+0000 7f8e103ee640 1 -- 172.21.15.27:0/4245853642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e0810bda0 msgr2=0x7f8e0810e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.552+0000 7f8e103ee640 1 --2- 172.21.15.27:0/4245853642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e0810bda0 0x7f8e0810e210 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009ca0 tx=0x7f8e0002a950 comp rx=0 tx=0).stop 2024-04-25T00:25:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.552+0000 7f8e103ee640 1 -- 172.21.15.27:0/4245853642 shutdown_connections 2024-04-25T00:25:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.552+0000 7f8e103ee640 1 --2- 172.21.15.27:0/4245853642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e0810bda0 0x7f8e0810e210 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.552+0000 7f8e103ee640 1 --2- 172.21.15.27:0/4245853642 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e081032f0 0x7f8e0810b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.552+0000 7f8e103ee640 1 --2- 172.21.15.27:0/4245853642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e081029d0 0x7f8e08102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.552+0000 7f8e103ee640 1 -- 172.21.15.27:0/4245853642 >> 172.21.15.27:0/4245853642 conn(0x7f8e080777d0 msgr2=0x7f8e08077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.552+0000 7f8e103ee640 1 -- 172.21.15.27:0/4245853642 shutdown_connections 2024-04-25T00:25:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.552+0000 7f8e103ee640 1 -- 172.21.15.27:0/4245853642 wait complete. 2024-04-25T00:25:30.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.553+0000 7f8e103ee640 1 Processor -- start 2024-04-25T00:25:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.553+0000 7f8e103ee640 1 -- start start 2024-04-25T00:25:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.553+0000 7f8e103ee640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e081029d0 0x7f8e08198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.553+0000 7f8e103ee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e081032f0 0x7f8e081985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.553+0000 7f8e103ee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e0810bda0 0x7f8e0819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.553+0000 7f8e103ee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8e08111b70 con 0x7f8e0810bda0 2024-04-25T00:25:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.553+0000 7f8e103ee640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8e081119f0 con 0x7f8e081029d0 2024-04-25T00:25:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.553+0000 7f8e103ee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8e08111cf0 con 0x7f8e081032f0 2024-04-25T00:25:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.553+0000 7f8e0e98b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e0810bda0 0x7f8e0819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.553+0000 7f8e0e98b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e0810bda0 0x7f8e0819c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53012/0 (socket says 172.21.15.27:53012) 2024-04-25T00:25:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.553+0000 7f8e0e98b640 1 -- 172.21.15.27:0/3961477521 learned_addr learned my addr 172.21.15.27:0/3961477521 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.553+0000 7f8e0d989640 1 --2- 172.21.15.27:0/3961477521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e081032f0 0x7f8e081985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.553+0000 7f8e0e18a640 1 --2- 172.21.15.27:0/3961477521 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e081029d0 0x7f8e08198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.554+0000 7f8e0e98b640 1 -- 172.21.15.27:0/3961477521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e081032f0 msgr2=0x7f8e081985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.554+0000 7f8e0e98b640 1 --2- 172.21.15.27:0/3961477521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e081032f0 0x7f8e081985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.554+0000 7f8e0e98b640 1 -- 172.21.15.27:0/3961477521 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e081029d0 msgr2=0x7f8e08198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.554+0000 7f8e0e98b640 1 --2- 172.21.15.27:0/3961477521 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e081029d0 0x7f8e08198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.554+0000 7f8e0e98b640 1 -- 172.21.15.27:0/3961477521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e0819d030 con 0x7f8e0810bda0 2024-04-25T00:25:30.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.554+0000 7f8e0e98b640 1 --2- 172.21.15.27:0/3961477521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e0810bda0 0x7f8e0819c900 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f8e00004af0 tx=0x7f8e0002f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.554+0000 7f8dff7fe640 1 -- 172.21.15.27:0/3961477521 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e00034040 con 0x7f8e0810bda0 2024-04-25T00:25:30.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.554+0000 7f8dff7fe640 1 -- 172.21.15.27:0/3961477521 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e00030920 con 0x7f8e0810bda0 2024-04-25T00:25:30.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.554+0000 7f8dff7fe640 1 -- 172.21.15.27:0/3961477521 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e000047b0 con 0x7f8e0810bda0 2024-04-25T00:25:30.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.554+0000 7f8e103ee640 1 -- 172.21.15.27:0/3961477521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e0819d2c0 con 0x7f8e0810bda0 2024-04-25T00:25:30.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.554+0000 7f8e103ee640 1 -- 172.21.15.27:0/3961477521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e081a4bb0 con 0x7f8e0810bda0 2024-04-25T00:25:30.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.557+0000 7f8dff7fe640 1 -- 172.21.15.27:0/3961477521 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8e00037020 con 0x7f8e0810bda0 2024-04-25T00:25:30.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.557+0000 7f8e131d9740 1 Processor -- start 2024-04-25T00:25:30.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.557+0000 7f8e131d9740 1 -- start start 2024-04-25T00:25:30.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e131d9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf2621580 0x55adf2621960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e131d9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf2621ea0 0x55adf262c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e131d9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf262c570 0x55adf262e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e131d9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55adf2586670 con 0x55adf262c570 2024-04-25T00:25:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e131d9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55adf252e970 con 0x55adf2621580 2024-04-25T00:25:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e131d9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55adf2582630 con 0x55adf2621ea0 2024-04-25T00:25:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e0d989640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf262c570 0x55adf262e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e0d989640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf262c570 0x55adf262e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53020/0 (socket says 172.21.15.27:53020) 2024-04-25T00:25:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e0d989640 1 -- 172.21.15.27:0/3030984814 learned_addr learned my addr 172.21.15.27:0/3030984814 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e0e18a640 1 --2- 172.21.15.27:0/3030984814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf2621ea0 0x55adf262c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e0d989640 1 --2- 172.21.15.27:0/3030984814 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf2621580 0x55adf2621960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8dff7fe640 1 --2- 172.21.15.27:0/3961477521 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8df40a2d00 0x7f8df40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e0e98b640 1 --2- 172.21.15.27:0/3961477521 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8df40a2d00 0x7f8df40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e0d989640 1 -- 172.21.15.27:0/3030984814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf2621ea0 msgr2=0x55adf262c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e0d989640 1 --2- 172.21.15.27:0/3030984814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf2621ea0 0x55adf262c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e0d989640 1 -- 172.21.15.27:0/3030984814 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf2621580 msgr2=0x55adf2621960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e0d989640 1 --2- 172.21.15.27:0/3030984814 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf2621580 0x55adf2621960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8dff7fe640 1 -- 172.21.15.27:0/3961477521 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8e000f0430 con 0x7f8e0810bda0 2024-04-25T00:25:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e0d989640 1 -- 172.21.15.27:0/3030984814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adf2552400 con 0x55adf262c570 2024-04-25T00:25:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.558+0000 7f8e0d989640 1 --2- 172.21.15.27:0/3030984814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf262c570 0x55adf262e960 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f8df8011770 tx=0x7f8df8009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e400551c93a9ebbf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.559+0000 7f8dfcbfe640 1 -- 172.21.15.27:0/3030984814 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df8004030 con 0x55adf262c570 2024-04-25T00:25:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.559+0000 7f8e0e98b640 1 --2- 172.21.15.27:0/3961477521 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8df40a2d00 0x7f8df40a51c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8e000bc3f0 tx=0x7f8e0008e6b0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.559+0000 7f8dfcbfe640 1 -- 172.21.15.27:0/3030984814 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8df800fca0 con 0x55adf262c570 2024-04-25T00:25:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.559+0000 7f8dfcbfe640 1 -- 172.21.15.27:0/3030984814 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df8003a70 con 0x55adf262c570 2024-04-25T00:25:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.559+0000 7f8e131d9740 1 -- 172.21.15.27:0/3030984814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf262c570 msgr2=0x55adf262e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.559+0000 7f8e131d9740 1 --2- 172.21.15.27:0/3030984814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf262c570 0x55adf262e960 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f8df8011770 tx=0x7f8df8009770 comp rx=0 tx=0).stop 2024-04-25T00:25:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.559+0000 7f8e131d9740 1 -- 172.21.15.27:0/3030984814 shutdown_connections 2024-04-25T00:25:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.559+0000 7f8e131d9740 1 --2- 172.21.15.27:0/3030984814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf262c570 0x55adf262e960 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.559+0000 7f8e131d9740 1 --2- 172.21.15.27:0/3030984814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf2621ea0 0x55adf262c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.559+0000 7f8e131d9740 1 --2- 172.21.15.27:0/3030984814 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf2621580 0x55adf2621960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.559+0000 7f8e131d9740 1 -- 172.21.15.27:0/3030984814 >> 172.21.15.27:0/3030984814 conn(0x55adf261d8a0 msgr2=0x55adf261dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.559+0000 7f8e131d9740 1 -- 172.21.15.27:0/3030984814 shutdown_connections 2024-04-25T00:25:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.559+0000 7f8e131d9740 1 -- 172.21.15.27:0/3030984814 wait complete. 2024-04-25T00:25:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.560+0000 7f8e131d9740 1 -- start start 2024-04-25T00:25:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.560+0000 7f8e131d9740 1 Processor -- start 2024-04-25T00:25:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.560+0000 7f8e131d9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621580 0x55adf262be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.560+0000 7f8e131d9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf2621ea0 0x55adf2629170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.560+0000 7f8e131d9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf262c570 0x55adf26296b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.560+0000 7f8e131d9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55adf262f360 con 0x55adf2621580 2024-04-25T00:25:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.560+0000 7f8e131d9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55adf2542050 con 0x55adf2621ea0 2024-04-25T00:25:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.560+0000 7f8e131d9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55adf262f4e0 con 0x55adf262c570 2024-04-25T00:25:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.560+0000 7f8e131d9740 10 client.0 _ll_register_callbacks cb 0x55adf2299be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:25:30.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.560+0000 7f8e0d989640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621580 0x55adf262be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.560+0000 7f8e0d989640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621580 0x55adf262be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53036/0 (socket says 172.21.15.27:53036) 2024-04-25T00:25:30.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.560+0000 7f8e0d989640 1 -- 172.21.15.27:0/3043179593 learned_addr learned my addr 172.21.15.27:0/3043179593 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.560+0000 7f8e0e18a640 1 --2- 172.21.15.27:0/3043179593 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf2621ea0 0x55adf2629170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.560+0000 7f8e0d989640 1 --2- 172.21.15.27:0/3043179593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf262c570 0x55adf26296b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8e0d989640 1 -- 172.21.15.27:0/3043179593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf262c570 msgr2=0x55adf26296b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8e0d989640 1 --2- 172.21.15.27:0/3043179593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf262c570 0x55adf26296b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8e0d989640 1 -- 172.21.15.27:0/3043179593 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf2621ea0 msgr2=0x55adf2629170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8e0d989640 1 --2- 172.21.15.27:0/3043179593 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf2621ea0 0x55adf2629170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8e0d989640 1 -- 172.21.15.27:0/3043179593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adf2513dc0 con 0x55adf2621580 2024-04-25T00:25:30.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8e0e18a640 1 --2- 172.21.15.27:0/3043179593 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf2621ea0 0x55adf2629170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:25:30.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8e0d989640 1 --2- 172.21.15.27:0/3043179593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621580 0x55adf262be80 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8df8003630 tx=0x7f8df8003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5179bee5d9e23aa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8df1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:25:30.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8df1ffb640 1 -- 172.21.15.27:0/3043179593 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df800fe50 con 0x55adf2621580 2024-04-25T00:25:30.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8df1ffb640 1 -- 172.21.15.27:0/3043179593 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8df8003810 con 0x55adf2621580 2024-04-25T00:25:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8e131d9740 1 -- 172.21.15.27:0/3043179593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8df800fca0 con 0x55adf2621580 2024-04-25T00:25:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8df1ffb640 1 -- 172.21.15.27:0/3043179593 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f8df8011020 con 0x55adf2621580 2024-04-25T00:25:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8e131d9740 10 client.17169 fetch_fsmap learned FSMap version 100 2024-04-25T00:25:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8e131d9740 1 -- 172.21.15.27:0/3043179593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x55adf229a3f0 con 0x55adf2621580 2024-04-25T00:25:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8df1ffb640 1 -- 172.21.15.27:0/3043179593 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8df800fca0 con 0x55adf2621580 2024-04-25T00:25:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8df1ffb640 1 -- 172.21.15.27:0/3043179593 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 100) ==== 8366+0+0 (secure 0 0 0) 0x7f8df80304b0 con 0x55adf2621580 2024-04-25T00:25:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8e131d9740 10 client.17169 fetch_fsmap finished waiting for FSMap version 100 2024-04-25T00:25:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.561+0000 7f8e131d9740 10 client.17169 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:25:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.562+0000 7f8e131d9740 20 client.17169 populate_metadata read hostname 'smithi027' 2024-04-25T00:25:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.562+0000 7f8e131d9740 1 --2- 172.21.15.27:0/3043179593 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55adf263c430 0x55adf263e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.562+0000 7f8e131d9740 4 client.17169 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:25:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.562+0000 7f8e131d9740 1 -- 172.21.15.27:0/3043179593 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adf2593bb0 con 0x55adf263c430 2024-04-25T00:25:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.562+0000 7f8e0e18a640 1 --2- 172.21.15.27:0/3043179593 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55adf263c430 0x55adf263e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.562+0000 7f8e0e18a640 1 --2- 172.21.15.27:0/3043179593 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55adf263c430 0x55adf263e820 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fe4ddc385a3668d2 server_cookie=8c29ea60770f7cb6 in_seq=0 out_seq=0 2024-04-25T00:25:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.562+0000 7f8df1ffb640 10 client.17169 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:25:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.563+0000 7f8df1ffb640 1 -- 172.21.15.27:0/3043179593 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55adf2593bb0 con 0x55adf263c430 2024-04-25T00:25:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.563+0000 7f8df1ffb640 10 client.17169 handle_command_reply: tid=0 2024-04-25T00:25:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.563+0000 7f8e131d9740 1 client.17169 shutdown 2024-04-25T00:25:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.563+0000 7f8e131d9740 1 -- 172.21.15.27:0/3043179593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621580 msgr2=0x55adf262be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.563+0000 7f8e131d9740 1 --2- 172.21.15.27:0/3043179593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621580 0x55adf262be80 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8df8003630 tx=0x7f8df8003730 comp rx=0 tx=0).stop 2024-04-25T00:25:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.563+0000 7f8e131d9740 1 -- 172.21.15.27:0/3043179593 shutdown_connections 2024-04-25T00:25:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.563+0000 7f8e131d9740 1 --2- 172.21.15.27:0/3043179593 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55adf263c430 0x55adf263e820 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.563+0000 7f8e131d9740 1 --2- 172.21.15.27:0/3043179593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf262c570 0x55adf26296b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.563+0000 7f8e131d9740 1 --2- 172.21.15.27:0/3043179593 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf2621ea0 0x55adf2629170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.563+0000 7f8df1ffb640 0 client.17169 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:25:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.563+0000 7f8e131d9740 1 --2- 172.21.15.27:0/3043179593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621580 0x55adf262be80 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.563+0000 7f8e131d9740 1 -- 172.21.15.27:0/3043179593 >> 172.21.15.27:0/3043179593 conn(0x55adf261d8a0 msgr2=0x55adf26236a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.563+0000 7f8e131d9740 1 -- 172.21.15.27:0/3043179593 shutdown_connections 2024-04-25T00:25:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.563+0000 7f8e131d9740 1 -- 172.21.15.27:0/3043179593 wait complete. 2024-04-25T00:25:30.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.563+0000 7f8e131d9740 20 client.17169 trim_cache size 0 max 16384 2024-04-25T00:25:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e131d9740 1 Processor -- start 2024-04-25T00:25:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e131d9740 1 -- start start 2024-04-25T00:25:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e131d9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf2621580 0x55adf2638250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e131d9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621ea0 0x55adf261c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e131d9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf262c570 0x55adf2643f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e131d9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8e081aeb60 con 0x55adf2621ea0 2024-04-25T00:25:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e131d9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8e081b06d0 con 0x55adf2621580 2024-04-25T00:25:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e131d9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8e081af230 con 0x55adf262c570 2024-04-25T00:25:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e0e18a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621ea0 0x55adf261c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e0e18a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621ea0 0x55adf261c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53040/0 (socket says 172.21.15.27:53040) 2024-04-25T00:25:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e0e18a640 1 -- 172.21.15.27:0/2861485788 learned_addr learned my addr 172.21.15.27:0/2861485788 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e0d989640 1 --2- 172.21.15.27:0/2861485788 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf2621580 0x55adf2638250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e0d989640 1 --2- 172.21.15.27:0/2861485788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf262c570 0x55adf2643f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e0e18a640 1 -- 172.21.15.27:0/2861485788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf262c570 msgr2=0x55adf2643f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e0e18a640 1 --2- 172.21.15.27:0/2861485788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf262c570 0x55adf2643f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e0e18a640 1 -- 172.21.15.27:0/2861485788 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf2621580 msgr2=0x55adf2638250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e0e18a640 1 --2- 172.21.15.27:0/2861485788 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf2621580 0x55adf2638250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e0e18a640 1 -- 172.21.15.27:0/2861485788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adf23647d0 con 0x55adf2621ea0 2024-04-25T00:25:30.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e0d989640 1 --2- 172.21.15.27:0/2861485788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf262c570 0x55adf2643f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:25:30.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.577+0000 7f8e0e18a640 1 --2- 172.21.15.27:0/2861485788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621ea0 0x55adf261c9c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x55adf26289d0 tx=0x7f8e0400b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52180ea28a72bd47 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.578+0000 7f8df3fff640 1 -- 172.21.15.27:0/2861485788 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e0400bd80 con 0x55adf2621ea0 2024-04-25T00:25:30.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.578+0000 7f8df3fff640 1 -- 172.21.15.27:0/2861485788 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e04005500 con 0x55adf2621ea0 2024-04-25T00:25:30.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.578+0000 7f8df3fff640 1 -- 172.21.15.27:0/2861485788 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e04020410 con 0x55adf2621ea0 2024-04-25T00:25:30.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.578+0000 7f8e131d9740 1 -- 172.21.15.27:0/2861485788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621ea0 msgr2=0x55adf261c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.578+0000 7f8e131d9740 1 --2- 172.21.15.27:0/2861485788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621ea0 0x55adf261c9c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x55adf26289d0 tx=0x7f8e0400b390 comp rx=0 tx=0).stop 2024-04-25T00:25:30.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.578+0000 7f8e131d9740 1 -- 172.21.15.27:0/2861485788 shutdown_connections 2024-04-25T00:25:30.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.578+0000 7f8e131d9740 1 --2- 172.21.15.27:0/2861485788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf262c570 0x55adf2643f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.578+0000 7f8e131d9740 1 --2- 172.21.15.27:0/2861485788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621ea0 0x55adf261c9c0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.578+0000 7f8e131d9740 1 --2- 172.21.15.27:0/2861485788 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf2621580 0x55adf2638250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.578+0000 7f8e131d9740 1 -- 172.21.15.27:0/2861485788 >> 172.21.15.27:0/2861485788 conn(0x55adf261d8a0 msgr2=0x55adf261c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.578+0000 7f8e131d9740 1 -- 172.21.15.27:0/2861485788 shutdown_connections 2024-04-25T00:25:30.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.578+0000 7f8e131d9740 1 -- 172.21.15.27:0/2861485788 wait complete. 2024-04-25T00:25:30.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.578+0000 7f8e131d9740 1 -- start start 2024-04-25T00:25:30.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.578+0000 7f8e131d9740 1 Processor -- start 2024-04-25T00:25:30.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e131d9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf2621580 0x55adf2634440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e131d9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621ea0 0x55adf2634980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e131d9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf262c570 0x55adf2631960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e131d9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8e081b0000 con 0x55adf2621ea0 2024-04-25T00:25:30.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e131d9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8e081af900 con 0x55adf262c570 2024-04-25T00:25:30.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e131d9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8e081acb20 con 0x55adf2621580 2024-04-25T00:25:30.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e131d9740 10 client.0 _ll_register_callbacks cb 0x7f8df8009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:25:30.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e0e18a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621ea0 0x55adf2634980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e0e18a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621ea0 0x55adf2634980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53046/0 (socket says 172.21.15.27:53046) 2024-04-25T00:25:30.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e0e18a640 1 -- 172.21.15.27:0/1520853936 learned_addr learned my addr 172.21.15.27:0/1520853936 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e0d989640 1 --2- 172.21.15.27:0/1520853936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf262c570 0x55adf2631960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e0e18a640 1 -- 172.21.15.27:0/1520853936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf2621580 msgr2=0x55adf2634440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e0d989640 1 --2- 172.21.15.27:0/1520853936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf2621580 0x55adf2634440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e0e18a640 1 --2- 172.21.15.27:0/1520853936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf2621580 0x55adf2634440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e0e18a640 1 -- 172.21.15.27:0/1520853936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf262c570 msgr2=0x55adf2631960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e0e18a640 1 --2- 172.21.15.27:0/1520853936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf262c570 0x55adf2631960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e0e18a640 1 -- 172.21.15.27:0/1520853936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adf2632050 con 0x55adf2621ea0 2024-04-25T00:25:30.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e0e18a640 1 --2- 172.21.15.27:0/1520853936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621ea0 0x55adf2634980 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f8e04024000 tx=0x7f8e04005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47182e276c9cdd71 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8df1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:25:30.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8df1ffb640 1 -- 172.21.15.27:0/1520853936 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e04004360 con 0x55adf2621ea0 2024-04-25T00:25:30.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8df1ffb640 1 -- 172.21.15.27:0/1520853936 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8e0400bd80 con 0x55adf2621ea0 2024-04-25T00:25:30.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.579+0000 7f8e131d9740 1 -- 172.21.15.27:0/1520853936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e04005500 con 0x55adf2621ea0 2024-04-25T00:25:30.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.580+0000 7f8df1ffb640 1 -- 172.21.15.27:0/1520853936 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f8e04037050 con 0x55adf2621ea0 2024-04-25T00:25:30.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.580+0000 7f8e131d9740 10 client.17175 fetch_fsmap learned FSMap version 100 2024-04-25T00:25:30.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.580+0000 7f8e131d9740 1 -- 172.21.15.27:0/1520853936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x55adf26a21d0 con 0x55adf2621ea0 2024-04-25T00:25:30.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.580+0000 7f8df1ffb640 1 -- 172.21.15.27:0/1520853936 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e04005500 con 0x55adf2621ea0 2024-04-25T00:25:30.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.580+0000 7f8df1ffb640 1 -- 172.21.15.27:0/1520853936 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 100) ==== 8366+0+0 (secure 0 0 0) 0x7f8e04032030 con 0x55adf2621ea0 2024-04-25T00:25:30.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.580+0000 7f8e131d9740 10 client.17175 fetch_fsmap finished waiting for FSMap version 100 2024-04-25T00:25:30.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.580+0000 7f8e131d9740 10 client.17175 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:25:30.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.580+0000 7f8e131d9740 20 client.17175 populate_metadata read hostname 'smithi027' 2024-04-25T00:25:30.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.580+0000 7f8e131d9740 1 --2- 172.21.15.27:0/1520853936 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55adf26a2970 0x55adf26a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.580+0000 7f8e131d9740 4 client.17175 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:25:30.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.580+0000 7f8e131d9740 1 -- 172.21.15.27:0/1520853936 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55adf2629a40 con 0x55adf26a2970 2024-04-25T00:25:30.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.580+0000 7f8e0d989640 1 --2- 172.21.15.27:0/1520853936 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55adf26a2970 0x55adf26a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.581+0000 7f8e0d989640 1 --2- 172.21.15.27:0/1520853936 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55adf26a2970 0x55adf26a4d90 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bfcc73ad1051601 server_cookie=5ba73849102504b6 in_seq=0 out_seq=0 2024-04-25T00:25:30.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.581+0000 7f8df1ffb640 10 client.17175 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:25:30.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.581+0000 7f8df1ffb640 1 -- 172.21.15.27:0/1520853936 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55adf2629a40 con 0x55adf26a2970 2024-04-25T00:25:30.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.581+0000 7f8df1ffb640 10 client.17175 handle_command_reply: tid=0 2024-04-25T00:25:30.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.581+0000 7f8e131d9740 1 client.17175 shutdown 2024-04-25T00:25:30.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.581+0000 7f8e131d9740 1 -- 172.21.15.27:0/1520853936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621ea0 msgr2=0x55adf2634980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.581+0000 7f8e131d9740 1 --2- 172.21.15.27:0/1520853936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621ea0 0x55adf2634980 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f8e04024000 tx=0x7f8e04005740 comp rx=0 tx=0).stop 2024-04-25T00:25:30.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.581+0000 7f8e131d9740 1 -- 172.21.15.27:0/1520853936 shutdown_connections 2024-04-25T00:25:30.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.581+0000 7f8e131d9740 1 --2- 172.21.15.27:0/1520853936 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x55adf26a2970 0x55adf26a4d90 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.581+0000 7f8e131d9740 1 --2- 172.21.15.27:0/1520853936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55adf262c570 0x55adf2631960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.581+0000 7f8e131d9740 1 --2- 172.21.15.27:0/1520853936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55adf2621ea0 0x55adf2634980 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.581+0000 7f8df1ffb640 0 client.17175 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:25:30.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.581+0000 7f8e131d9740 1 --2- 172.21.15.27:0/1520853936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55adf2621580 0x55adf2634440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.581+0000 7f8e131d9740 1 -- 172.21.15.27:0/1520853936 >> 172.21.15.27:0/1520853936 conn(0x55adf261d8a0 msgr2=0x55adf261c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.582+0000 7f8e131d9740 1 -- 172.21.15.27:0/1520853936 shutdown_connections 2024-04-25T00:25:30.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.582+0000 7f8e131d9740 1 -- 172.21.15.27:0/1520853936 wait complete. 2024-04-25T00:25:30.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.582+0000 7f8e131d9740 20 client.17175 trim_cache size 0 max 16384 2024-04-25T00:25:30.591 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:25:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.582+0000 7f8e103ee640 1 -- 172.21.15.27:0/3961477521 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8df40a2d00 msgr2=0x7f8df40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.582+0000 7f8e103ee640 1 --2- 172.21.15.27:0/3961477521 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8df40a2d00 0x7f8df40a51c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8e000bc3f0 tx=0x7f8e0008e6b0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.582+0000 7f8e103ee640 1 -- 172.21.15.27:0/3961477521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e0810bda0 msgr2=0x7f8e0819c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.582+0000 7f8e103ee640 1 --2- 172.21.15.27:0/3961477521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e0810bda0 0x7f8e0819c900 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f8e00004af0 tx=0x7f8e0002f5b0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.582+0000 7f8e103ee640 1 -- 172.21.15.27:0/3961477521 shutdown_connections 2024-04-25T00:25:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.582+0000 7f8e103ee640 1 --2- 172.21.15.27:0/3961477521 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8df40a2d00 0x7f8df40a51c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.582+0000 7f8e103ee640 1 --2- 172.21.15.27:0/3961477521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e0810bda0 0x7f8e0819c900 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.582+0000 7f8e103ee640 1 --2- 172.21.15.27:0/3961477521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e081032f0 0x7f8e081985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.582+0000 7f8e103ee640 1 --2- 172.21.15.27:0/3961477521 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e081029d0 0x7f8e08198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.582+0000 7f8e103ee640 1 -- 172.21.15.27:0/3961477521 >> 172.21.15.27:0/3961477521 conn(0x7f8e080777d0 msgr2=0x7f8e080fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.582+0000 7f8e103ee640 1 -- 172.21.15.27:0/3961477521 shutdown_connections 2024-04-25T00:25:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.582+0000 7f8e103ee640 1 -- 172.21.15.27:0/3961477521 wait complete. 2024-04-25T00:25:30.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T00:25:30.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.698+0000 7feeb3838640 1 -- 172.21.15.27:0/1798062870 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feeac100610 msgr2=0x7feeac100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.698+0000 7feeb3838640 1 --2- 172.21.15.27:0/1798062870 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feeac100610 0x7feeac100a90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7feea8009be0 tx=0x7feea802a6c0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.698+0000 7feeb3838640 1 -- 172.21.15.27:0/1798062870 shutdown_connections 2024-04-25T00:25:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.698+0000 7feeb3838640 1 --2- 172.21.15.27:0/1798062870 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feeac100fd0 0x7feeac10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.698+0000 7feeb3838640 1 --2- 172.21.15.27:0/1798062870 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feeac100610 0x7feeac100a90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.698+0000 7feeb3838640 1 --2- 172.21.15.27:0/1798062870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeac1066b0 0x7feeac106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.698+0000 7feeb3838640 1 -- 172.21.15.27:0/1798062870 >> 172.21.15.27:0/1798062870 conn(0x7feeac077740 msgr2=0x7feeac077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.698+0000 7feeb3838640 1 -- 172.21.15.27:0/1798062870 shutdown_connections 2024-04-25T00:25:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.698+0000 7feeb3838640 1 -- 172.21.15.27:0/1798062870 wait complete. 2024-04-25T00:25:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.698+0000 7feeb3838640 1 Processor -- start 2024-04-25T00:25:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb3838640 1 -- start start 2024-04-25T00:25:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb3838640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feeac100610 0x7feeac1022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb3838640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feeac100fd0 0x7feeac1027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb3838640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeac1066b0 0x7feeac102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb3838640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feeac1109e0 con 0x7feeac1066b0 2024-04-25T00:25:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb3838640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feeac110860 con 0x7feeac100fd0 2024-04-25T00:25:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb3838640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feeac110b60 con 0x7feeac100610 2024-04-25T00:25:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb1dd5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeac1066b0 0x7feeac102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb1dd5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeac1066b0 0x7feeac102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53072/0 (socket says 172.21.15.27:53072) 2024-04-25T00:25:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb1dd5640 1 -- 172.21.15.27:0/1459524831 learned_addr learned my addr 172.21.15.27:0/1459524831 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb0dd3640 1 --2- 172.21.15.27:0/1459524831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feeac100fd0 0x7feeac1027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb15d4640 1 --2- 172.21.15.27:0/1459524831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feeac100610 0x7feeac1022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb1dd5640 1 -- 172.21.15.27:0/1459524831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feeac100610 msgr2=0x7feeac1022b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb1dd5640 1 --2- 172.21.15.27:0/1459524831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feeac100610 0x7feeac1022b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb1dd5640 1 -- 172.21.15.27:0/1459524831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feeac100fd0 msgr2=0x7feeac1027f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb1dd5640 1 --2- 172.21.15.27:0/1459524831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feeac100fd0 0x7feeac1027f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb1dd5640 1 -- 172.21.15.27:0/1459524831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeac1a6460 con 0x7feeac1066b0 2024-04-25T00:25:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.699+0000 7feeb1dd5640 1 --2- 172.21.15.27:0/1459524831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeac1066b0 0x7feeac102d30 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7feea400e970 tx=0x7feea400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.700+0000 7fee9e7fc640 1 -- 172.21.15.27:0/1459524831 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea400ccc0 con 0x7feeac1066b0 2024-04-25T00:25:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.700+0000 7feeb3838640 1 -- 172.21.15.27:0/1459524831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeac1a6750 con 0x7feeac1066b0 2024-04-25T00:25:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.700+0000 7fee9e7fc640 1 -- 172.21.15.27:0/1459524831 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feea400ce60 con 0x7feeac1066b0 2024-04-25T00:25:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.700+0000 7fee9e7fc640 1 -- 172.21.15.27:0/1459524831 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea4010430 con 0x7feeac1066b0 2024-04-25T00:25:30.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.700+0000 7feeb3838640 1 -- 172.21.15.27:0/1459524831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeac1a6cc0 con 0x7feeac1066b0 2024-04-25T00:25:30.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.702+0000 7fee9e7fc640 1 -- 172.21.15.27:0/1459524831 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7feea4005020 con 0x7feeac1066b0 2024-04-25T00:25:30.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.702+0000 7feeb6623740 1 Processor -- start 2024-04-25T00:25:30.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.702+0000 7feeb6623740 1 -- start start 2024-04-25T00:25:30.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb6623740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84580 0x562bbbd84960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb6623740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84ea0 0x562bbbd8f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb6623740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 0x562bbbd91960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb6623740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562bbbce9670 con 0x562bbbd84ea0 2024-04-25T00:25:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb6623740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x562bbbc91970 con 0x562bbbd8f570 2024-04-25T00:25:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb6623740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562bbbce5630 con 0x562bbbd84580 2024-04-25T00:25:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7fee9e7fc640 1 --2- 172.21.15.27:0/1459524831 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fee980a2d00 0x7fee980a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb15d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84ea0 0x562bbbd8f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb15d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84ea0 0x562bbbd8f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53082/0 (socket says 172.21.15.27:53082) 2024-04-25T00:25:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb15d4640 1 -- 172.21.15.27:0/1193710796 learned_addr learned my addr 172.21.15.27:0/1193710796 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb1dd5640 1 --2- 172.21.15.27:0/1459524831 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fee980a2d00 0x7fee980a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb0dd3640 1 --2- 172.21.15.27:0/1193710796 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 0x562bbbd91960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb0dd3640 1 --2- 172.21.15.27:0/1193710796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84580 0x562bbbd84960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7fee9e7fc640 1 -- 172.21.15.27:0/1459524831 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7feea40cfad0 con 0x7feeac1066b0 2024-04-25T00:25:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb15d4640 1 -- 172.21.15.27:0/1193710796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84580 msgr2=0x562bbbd84960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb15d4640 1 --2- 172.21.15.27:0/1193710796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84580 0x562bbbd84960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb15d4640 1 -- 172.21.15.27:0/1193710796 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 msgr2=0x562bbbd91960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb15d4640 1 --2- 172.21.15.27:0/1193710796 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 0x562bbbd91960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb15d4640 1 -- 172.21.15.27:0/1193710796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bbbcb5400 con 0x562bbbd84ea0 2024-04-25T00:25:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb15d4640 1 --2- 172.21.15.27:0/1193710796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84ea0 0x562bbbd8f030 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7feea0009c60 tx=0x7feea0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=859cecbb6c82b54c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.703+0000 7feeb1dd5640 1 --2- 172.21.15.27:0/1459524831 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fee980a2d00 0x7fee980a51c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7feea40980e0 tx=0x7feea406dc40 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.704+0000 7fee977fe640 1 -- 172.21.15.27:0/1193710796 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea0004230 con 0x562bbbd84ea0 2024-04-25T00:25:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.704+0000 7fee977fe640 1 -- 172.21.15.27:0/1193710796 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feea0009e10 con 0x562bbbd84ea0 2024-04-25T00:25:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.704+0000 7fee977fe640 1 -- 172.21.15.27:0/1193710796 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea0004de0 con 0x562bbbd84ea0 2024-04-25T00:25:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.704+0000 7feeb6623740 1 -- 172.21.15.27:0/1193710796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84ea0 msgr2=0x562bbbd8f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.704+0000 7feeb6623740 1 --2- 172.21.15.27:0/1193710796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84ea0 0x562bbbd8f030 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7feea0009c60 tx=0x7feea0011810 comp rx=0 tx=0).stop 2024-04-25T00:25:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.704+0000 7feeb6623740 1 -- 172.21.15.27:0/1193710796 shutdown_connections 2024-04-25T00:25:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.704+0000 7feeb6623740 1 --2- 172.21.15.27:0/1193710796 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 0x562bbbd91960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.704+0000 7feeb6623740 1 --2- 172.21.15.27:0/1193710796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84ea0 0x562bbbd8f030 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.704+0000 7feeb6623740 1 --2- 172.21.15.27:0/1193710796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84580 0x562bbbd84960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.704+0000 7feeb6623740 1 -- 172.21.15.27:0/1193710796 >> 172.21.15.27:0/1193710796 conn(0x562bbbd808a0 msgr2=0x562bbbd80c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.704+0000 7feeb6623740 1 -- 172.21.15.27:0/1193710796 shutdown_connections 2024-04-25T00:25:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.704+0000 7feeb6623740 1 -- 172.21.15.27:0/1193710796 wait complete. 2024-04-25T00:25:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb6623740 1 -- start start 2024-04-25T00:25:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb6623740 1 Processor -- start 2024-04-25T00:25:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb6623740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84580 0x562bbbd8eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb6623740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84ea0 0x562bbbd8c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb6623740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 0x562bbbd8c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb6623740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562bbbd92360 con 0x562bbbd84ea0 2024-04-25T00:25:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb6623740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x562bbbca5050 con 0x562bbbd8f570 2024-04-25T00:25:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb6623740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562bbbd924e0 con 0x562bbbd84580 2024-04-25T00:25:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb6623740 10 client.0 _ll_register_callbacks cb 0x562bbb9fcbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:25:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb15d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84ea0 0x562bbbd8c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb15d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84ea0 0x562bbbd8c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53090/0 (socket says 172.21.15.27:53090) 2024-04-25T00:25:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb15d4640 1 -- 172.21.15.27:0/3126091494 learned_addr learned my addr 172.21.15.27:0/3126091494 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb0dd3640 1 --2- 172.21.15.27:0/3126091494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 0x562bbbd8c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb0dd3640 1 --2- 172.21.15.27:0/3126091494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84580 0x562bbbd8eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb15d4640 1 -- 172.21.15.27:0/3126091494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84580 msgr2=0x562bbbd8eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb15d4640 1 --2- 172.21.15.27:0/3126091494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84580 0x562bbbd8eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb15d4640 1 -- 172.21.15.27:0/3126091494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 msgr2=0x562bbbd8c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.705+0000 7feeb15d4640 1 --2- 172.21.15.27:0/3126091494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 0x562bbbd8c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7feeb15d4640 1 -- 172.21.15.27:0/3126091494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bbbc76dc0 con 0x562bbbd84ea0 2024-04-25T00:25:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7feeb0dd3640 1 --2- 172.21.15.27:0/3126091494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 0x562bbbd8c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:25:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7feeb15d4640 1 --2- 172.21.15.27:0/3126091494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84ea0 0x562bbbd8c1d0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7feea0009230 tx=0x7feea00044b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19520b984de1ba6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7fee94ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:25:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7fee94ff9640 1 -- 172.21.15.27:0/3126091494 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea0027020 con 0x562bbbd84ea0 2024-04-25T00:25:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7fee94ff9640 1 -- 172.21.15.27:0/3126091494 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7feea00035f0 con 0x562bbbd84ea0 2024-04-25T00:25:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7feeb6623740 1 -- 172.21.15.27:0/3126091494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feea0009e10 con 0x562bbbd84ea0 2024-04-25T00:25:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7fee94ff9640 1 -- 172.21.15.27:0/3126091494 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7feea0032050 con 0x562bbbd84ea0 2024-04-25T00:25:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7feeb6623740 10 client.17187 fetch_fsmap learned FSMap version 100 2024-04-25T00:25:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7feeb6623740 1 -- 172.21.15.27:0/3126091494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x562bbb9fd3f0 con 0x562bbbd84ea0 2024-04-25T00:25:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7fee94ff9640 1 -- 172.21.15.27:0/3126091494 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feea0009e10 con 0x562bbbd84ea0 2024-04-25T00:25:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7fee94ff9640 1 -- 172.21.15.27:0/3126091494 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 100) ==== 8366+0+0 (secure 0 0 0) 0x7feea00238d0 con 0x562bbbd84ea0 2024-04-25T00:25:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7feeb6623740 10 client.17187 fetch_fsmap finished waiting for FSMap version 100 2024-04-25T00:25:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7feeb6623740 10 client.17187 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:25:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7feeb6623740 20 client.17187 populate_metadata read hostname 'smithi027' 2024-04-25T00:25:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7feeb6623740 1 --2- 172.21.15.27:0/3126091494 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x562bbbd9f560 0x562bbbda1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7feeb6623740 4 client.17187 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:25:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.706+0000 7feeb6623740 1 -- 172.21.15.27:0/3126091494 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562bbbcf6bb0 con 0x562bbbd9f560 2024-04-25T00:25:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.707+0000 7feeb0dd3640 1 --2- 172.21.15.27:0/3126091494 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x562bbbd9f560 0x562bbbda1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.707+0000 7fee94ff9640 10 client.17187 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:25:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.707+0000 7feeb0dd3640 1 --2- 172.21.15.27:0/3126091494 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x562bbbd9f560 0x562bbbda1950 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e8f39422fb3a7c34 server_cookie=4c536156baeabb64 in_seq=0 out_seq=0 2024-04-25T00:25:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.708+0000 7fee94ff9640 1 -- 172.21.15.27:0/3126091494 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562bbbcf6bb0 con 0x562bbbd9f560 2024-04-25T00:25:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.708+0000 7fee94ff9640 10 client.17187 handle_command_reply: tid=0 2024-04-25T00:25:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.708+0000 7feeb6623740 1 client.17187 shutdown 2024-04-25T00:25:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.708+0000 7feeb6623740 1 -- 172.21.15.27:0/3126091494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84ea0 msgr2=0x562bbbd8c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.708+0000 7feeb6623740 1 --2- 172.21.15.27:0/3126091494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84ea0 0x562bbbd8c1d0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7feea0009230 tx=0x7feea00044b0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.708+0000 7feeb6623740 1 -- 172.21.15.27:0/3126091494 shutdown_connections 2024-04-25T00:25:30.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.708+0000 7feeb6623740 1 --2- 172.21.15.27:0/3126091494 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x562bbbd9f560 0x562bbbda1950 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.708+0000 7feeb6623740 1 --2- 172.21.15.27:0/3126091494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 0x562bbbd8c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.708+0000 7feeb6623740 1 --2- 172.21.15.27:0/3126091494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84ea0 0x562bbbd8c1d0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.708+0000 7feeb6623740 1 --2- 172.21.15.27:0/3126091494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84580 0x562bbbd8eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.708+0000 7fee94ff9640 0 client.17187 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:25:30.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.708+0000 7feeb6623740 1 -- 172.21.15.27:0/3126091494 >> 172.21.15.27:0/3126091494 conn(0x562bbbd808a0 msgr2=0x562bbbd7fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.708+0000 7feeb6623740 1 -- 172.21.15.27:0/3126091494 shutdown_connections 2024-04-25T00:25:30.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.708+0000 7feeb6623740 1 -- 172.21.15.27:0/3126091494 wait complete. 2024-04-25T00:25:30.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.708+0000 7feeb6623740 20 client.17187 trim_cache size 0 max 16384 2024-04-25T00:25:30.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb6623740 1 Processor -- start 2024-04-25T00:25:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb6623740 1 -- start start 2024-04-25T00:25:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb6623740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84580 0x562bbbd9b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb6623740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84ea0 0x562bbbd8ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb6623740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 0x562bbbda70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb6623740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feeac1b0e30 con 0x562bbbd84580 2024-04-25T00:25:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb6623740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feeac1b29a0 con 0x562bbbd8f570 2024-04-25T00:25:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb6623740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feeac1b1500 con 0x562bbbd84ea0 2024-04-25T00:25:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb0dd3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84580 0x562bbbd9b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb0dd3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84580 0x562bbbd9b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53102/0 (socket says 172.21.15.27:53102) 2024-04-25T00:25:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb0dd3640 1 -- 172.21.15.27:0/3248286976 learned_addr learned my addr 172.21.15.27:0/3248286976 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb0dd3640 1 --2- 172.21.15.27:0/3248286976 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 0x562bbbda70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb0dd3640 1 -- 172.21.15.27:0/3248286976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84ea0 msgr2=0x562bbbd8ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb0dd3640 1 --2- 172.21.15.27:0/3248286976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84ea0 0x562bbbd8ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb0dd3640 1 -- 172.21.15.27:0/3248286976 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 msgr2=0x562bbbda70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb0dd3640 1 --2- 172.21.15.27:0/3248286976 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 0x562bbbda70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb0dd3640 1 -- 172.21.15.27:0/3248286976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bbbd8caa0 con 0x562bbbd84580 2024-04-25T00:25:30.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.722+0000 7feeb0dd3640 1 --2- 172.21.15.27:0/3248286976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84580 0x562bbbd9b220 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7feea8009be0 tx=0x7feea80303a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36c325d5b0c62d55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.723+0000 7fee96ffd640 1 -- 172.21.15.27:0/3248286976 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea8050070 con 0x562bbbd84580 2024-04-25T00:25:30.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.723+0000 7fee96ffd640 1 -- 172.21.15.27:0/3248286976 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feea8037040 con 0x562bbbd84580 2024-04-25T00:25:30.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.723+0000 7fee96ffd640 1 -- 172.21.15.27:0/3248286976 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea8036040 con 0x562bbbd84580 2024-04-25T00:25:30.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.723+0000 7feeb6623740 1 -- 172.21.15.27:0/3248286976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84580 msgr2=0x562bbbd9b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.723+0000 7feeb6623740 1 --2- 172.21.15.27:0/3248286976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84580 0x562bbbd9b220 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7feea8009be0 tx=0x7feea80303a0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.723+0000 7feeb6623740 1 -- 172.21.15.27:0/3248286976 shutdown_connections 2024-04-25T00:25:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.723+0000 7feeb6623740 1 --2- 172.21.15.27:0/3248286976 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 0x562bbbda70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.723+0000 7feeb6623740 1 --2- 172.21.15.27:0/3248286976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84ea0 0x562bbbd8ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.723+0000 7feeb6623740 1 --2- 172.21.15.27:0/3248286976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84580 0x562bbbd9b220 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.723+0000 7feeb6623740 1 -- 172.21.15.27:0/3248286976 >> 172.21.15.27:0/3248286976 conn(0x562bbbd808a0 msgr2=0x562bbbd87860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.723+0000 7feeb6623740 1 -- 172.21.15.27:0/3248286976 shutdown_connections 2024-04-25T00:25:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.723+0000 7feeb6623740 1 -- 172.21.15.27:0/3248286976 wait complete. 2024-04-25T00:25:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.724+0000 7feeb6623740 1 -- start start 2024-04-25T00:25:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.724+0000 7feeb6623740 1 Processor -- start 2024-04-25T00:25:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.724+0000 7feeb6623740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84580 0x562bbbd97b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.724+0000 7feeb6623740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84ea0 0x562bbbd92e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.724+0000 7feeb6623740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 0x562bbbd933d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.724+0000 7feeb6623740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feeac1b22d0 con 0x562bbbd84580 2024-04-25T00:25:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.724+0000 7feeb6623740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feeac1b1bd0 con 0x562bbbd8f570 2024-04-25T00:25:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.724+0000 7feeb6623740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feeac1aedf0 con 0x562bbbd84ea0 2024-04-25T00:25:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.724+0000 7feeb6623740 10 client.0 _ll_register_callbacks cb 0x7feea0005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:25:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.724+0000 7feeb0dd3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84580 0x562bbbd97b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.724+0000 7feeb0dd3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84580 0x562bbbd97b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53118/0 (socket says 172.21.15.27:53118) 2024-04-25T00:25:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.724+0000 7feeb0dd3640 1 -- 172.21.15.27:0/2228159438 learned_addr learned my addr 172.21.15.27:0/2228159438 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.724+0000 7feeb0dd3640 1 --2- 172.21.15.27:0/2228159438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 0x562bbbd933d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7feeb15d4640 1 --2- 172.21.15.27:0/2228159438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84ea0 0x562bbbd92e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7feeb0dd3640 1 -- 172.21.15.27:0/2228159438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84ea0 msgr2=0x562bbbd92e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7feeb0dd3640 1 --2- 172.21.15.27:0/2228159438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84ea0 0x562bbbd92e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7feeb0dd3640 1 -- 172.21.15.27:0/2228159438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 msgr2=0x562bbbd933d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7feeb0dd3640 1 --2- 172.21.15.27:0/2228159438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 0x562bbbd933d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7feeb0dd3640 1 -- 172.21.15.27:0/2228159438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bbbac77d0 con 0x562bbbd84580 2024-04-25T00:25:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7feeb15d4640 1 --2- 172.21.15.27:0/2228159438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84ea0 0x562bbbd92e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:25:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7feeb0dd3640 1 --2- 172.21.15.27:0/2228159438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84580 0x562bbbd97b20 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7feea8009d00 tx=0x7feea804bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29b644fdd3fe57dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7fee94ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:25:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7fee94ff9640 1 -- 172.21.15.27:0/2228159438 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea8036070 con 0x562bbbd84580 2024-04-25T00:25:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7fee94ff9640 1 -- 172.21.15.27:0/2228159438 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7feea8003970 con 0x562bbbd84580 2024-04-25T00:25:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7feeb6623740 1 -- 172.21.15.27:0/2228159438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feea8037040 con 0x562bbbd84580 2024-04-25T00:25:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7fee94ff9640 1 -- 172.21.15.27:0/2228159438 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7feea8050040 con 0x562bbbd84580 2024-04-25T00:25:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7feeb6623740 10 client.17193 fetch_fsmap learned FSMap version 100 2024-04-25T00:25:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7feeb6623740 1 -- 172.21.15.27:0/2228159438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x562bbbe057b0 con 0x562bbbd84580 2024-04-25T00:25:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7fee94ff9640 1 -- 172.21.15.27:0/2228159438 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feea8037040 con 0x562bbbd84580 2024-04-25T00:25:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7fee94ff9640 1 -- 172.21.15.27:0/2228159438 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 100) ==== 8366+0+0 (secure 0 0 0) 0x7feea8056070 con 0x562bbbd84580 2024-04-25T00:25:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7feeb6623740 10 client.17193 fetch_fsmap finished waiting for FSMap version 100 2024-04-25T00:25:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7feeb6623740 10 client.17193 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:25:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.725+0000 7feeb6623740 20 client.17193 populate_metadata read hostname 'smithi027' 2024-04-25T00:25:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7feeb6623740 1 --2- 172.21.15.27:0/2228159438 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x562bbbe05da0 0x562bbbe081c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7feeb6623740 4 client.17193 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:25:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7feeb6623740 1 -- 172.21.15.27:0/2228159438 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562bbbcf0f00 con 0x562bbbe05da0 2024-04-25T00:25:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7feeb15d4640 1 --2- 172.21.15.27:0/2228159438 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x562bbbe05da0 0x562bbbe081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7fee94ff9640 10 client.17193 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:25:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7feeb15d4640 1 --2- 172.21.15.27:0/2228159438 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x562bbbe05da0 0x562bbbe081c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5c65a248376d67d5 server_cookie=f80489468392926b in_seq=0 out_seq=0 2024-04-25T00:25:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7fee94ff9640 1 -- 172.21.15.27:0/2228159438 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562bbbcf0f00 con 0x562bbbe05da0 2024-04-25T00:25:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7fee94ff9640 10 client.17193 handle_command_reply: tid=0 2024-04-25T00:25:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7feeb6623740 1 client.17193 shutdown 2024-04-25T00:25:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7feeb6623740 1 -- 172.21.15.27:0/2228159438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84580 msgr2=0x562bbbd97b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7feeb6623740 1 --2- 172.21.15.27:0/2228159438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84580 0x562bbbd97b20 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7feea8009d00 tx=0x7feea804bf90 comp rx=0 tx=0).stop 2024-04-25T00:25:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7feeb6623740 1 -- 172.21.15.27:0/2228159438 shutdown_connections 2024-04-25T00:25:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7feeb6623740 1 --2- 172.21.15.27:0/2228159438 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x562bbbe05da0 0x562bbbe081c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7feeb6623740 1 --2- 172.21.15.27:0/2228159438 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562bbbd8f570 0x562bbbd933d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7feeb6623740 1 --2- 172.21.15.27:0/2228159438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562bbbd84ea0 0x562bbbd92e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7feeb6623740 1 --2- 172.21.15.27:0/2228159438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562bbbd84580 0x562bbbd97b20 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7fee94ff9640 0 client.17193 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:25:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.726+0000 7feeb6623740 1 -- 172.21.15.27:0/2228159438 >> 172.21.15.27:0/2228159438 conn(0x562bbbd808a0 msgr2=0x562bbbd9a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.727+0000 7feeb6623740 1 -- 172.21.15.27:0/2228159438 shutdown_connections 2024-04-25T00:25:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.727+0000 7feeb6623740 1 -- 172.21.15.27:0/2228159438 wait complete. 2024-04-25T00:25:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.727+0000 7feeb6623740 20 client.17193 trim_cache size 0 max 16384 2024-04-25T00:25:30.735 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:25:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.727+0000 7feeb3838640 1 -- 172.21.15.27:0/1459524831 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fee980a2d00 msgr2=0x7fee980a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.727+0000 7feeb3838640 1 --2- 172.21.15.27:0/1459524831 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fee980a2d00 0x7fee980a51c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7feea40980e0 tx=0x7feea406dc40 comp rx=0 tx=0).stop 2024-04-25T00:25:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.727+0000 7feeb3838640 1 -- 172.21.15.27:0/1459524831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeac1066b0 msgr2=0x7feeac102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.727+0000 7feeb3838640 1 --2- 172.21.15.27:0/1459524831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeac1066b0 0x7feeac102d30 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7feea400e970 tx=0x7feea400ed70 comp rx=0 tx=0).stop 2024-04-25T00:25:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.727+0000 7feeb3838640 1 -- 172.21.15.27:0/1459524831 shutdown_connections 2024-04-25T00:25:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.727+0000 7feeb3838640 1 --2- 172.21.15.27:0/1459524831 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fee980a2d00 0x7fee980a51c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.727+0000 7feeb3838640 1 --2- 172.21.15.27:0/1459524831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeac1066b0 0x7feeac102d30 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.727+0000 7feeb3838640 1 --2- 172.21.15.27:0/1459524831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feeac100fd0 0x7feeac1027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.727+0000 7feeb3838640 1 --2- 172.21.15.27:0/1459524831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feeac100610 0x7feeac1022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.727+0000 7feeb3838640 1 -- 172.21.15.27:0/1459524831 >> 172.21.15.27:0/1459524831 conn(0x7feeac077740 msgr2=0x7feeac0ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.727+0000 7feeb3838640 1 -- 172.21.15.27:0/1459524831 shutdown_connections 2024-04-25T00:25:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.727+0000 7feeb3838640 1 -- 172.21.15.27:0/1459524831 wait complete. 2024-04-25T00:25:30.739 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T00:25:30.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.842+0000 7fedb80fc640 1 -- 172.21.15.27:0/619882383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb0069dc0 msgr2=0x7fedb006a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.842+0000 7fedb80fc640 1 --2- 172.21.15.27:0/619882383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb0069dc0 0x7fedb006a240 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7feda0009930 tx=0x7feda002a420 comp rx=0 tx=0).stop 2024-04-25T00:25:30.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.842+0000 7fedb80fc640 1 -- 172.21.15.27:0/619882383 shutdown_connections 2024-04-25T00:25:30.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.842+0000 7fedb80fc640 1 --2- 172.21.15.27:0/619882383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fedb006a780 0x7fedb010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.842+0000 7fedb80fc640 1 --2- 172.21.15.27:0/619882383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb0069dc0 0x7fedb006a240 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.842+0000 7fedb80fc640 1 --2- 172.21.15.27:0/619882383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fedb0107730 0x7fedb0107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.842+0000 7fedb80fc640 1 -- 172.21.15.27:0/619882383 >> 172.21.15.27:0/619882383 conn(0x7fedb00776a0 msgr2=0x7fedb0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.842+0000 7fedb80fc640 1 -- 172.21.15.27:0/619882383 shutdown_connections 2024-04-25T00:25:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.842+0000 7fedb80fc640 1 -- 172.21.15.27:0/619882383 wait complete. 2024-04-25T00:25:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.843+0000 7fedb80fc640 1 Processor -- start 2024-04-25T00:25:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.843+0000 7fedb80fc640 1 -- start start 2024-04-25T00:25:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.843+0000 7fedb80fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb0069dc0 0x7fedb0100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.843+0000 7fedb80fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fedb006a780 0x7fedb0100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.843+0000 7fedb80fc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fedb0107730 0x7fedb01010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.843+0000 7fedb80fc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fedb0111b40 con 0x7fedb0069dc0 2024-04-25T00:25:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.843+0000 7fedb80fc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fedb01119c0 con 0x7fedb0107730 2024-04-25T00:25:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.843+0000 7fedb80fc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fedb0111cc0 con 0x7fedb006a780 2024-04-25T00:25:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.843+0000 7fedb5e98640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb0069dc0 0x7fedb0100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.843+0000 7fedb5e98640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb0069dc0 0x7fedb0100670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53140/0 (socket says 172.21.15.27:53140) 2024-04-25T00:25:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.843+0000 7fedb5e98640 1 -- 172.21.15.27:0/1947490217 learned_addr learned my addr 172.21.15.27:0/1947490217 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.843+0000 7fedb6699640 1 --2- 172.21.15.27:0/1947490217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fedb0107730 0x7fedb01010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.843+0000 7fedb5e98640 1 -- 172.21.15.27:0/1947490217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fedb006a780 msgr2=0x7fedb0100bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.843+0000 7fedb5697640 1 --2- 172.21.15.27:0/1947490217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fedb006a780 0x7fedb0100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.843+0000 7fedb5e98640 1 --2- 172.21.15.27:0/1947490217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fedb006a780 0x7fedb0100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.844+0000 7fedb5e98640 1 -- 172.21.15.27:0/1947490217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fedb0107730 msgr2=0x7fedb01010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.844+0000 7fedb5e98640 1 --2- 172.21.15.27:0/1947490217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fedb0107730 0x7fedb01010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.844+0000 7fedb5e98640 1 -- 172.21.15.27:0/1947490217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb01a6560 con 0x7fedb0069dc0 2024-04-25T00:25:30.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.844+0000 7fedb6699640 1 --2- 172.21.15.27:0/1947490217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fedb0107730 0x7fedb01010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:25:30.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.844+0000 7fedb5e98640 1 --2- 172.21.15.27:0/1947490217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb0069dc0 0x7fedb0100670 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fedac00d770 tx=0x7fedac00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.844+0000 7feda6ffd640 1 -- 172.21.15.27:0/1947490217 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac017070 con 0x7fedb0069dc0 2024-04-25T00:25:30.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.844+0000 7feda6ffd640 1 -- 172.21.15.27:0/1947490217 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedac005500 con 0x7fedb0069dc0 2024-04-25T00:25:30.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.844+0000 7fedb80fc640 1 -- 172.21.15.27:0/1947490217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedb01a6850 con 0x7fedb0069dc0 2024-04-25T00:25:30.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.844+0000 7feda6ffd640 1 -- 172.21.15.27:0/1947490217 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac004e00 con 0x7fedb0069dc0 2024-04-25T00:25:30.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.844+0000 7fedb80fc640 1 -- 172.21.15.27:0/1947490217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedb01a6d50 con 0x7fedb0069dc0 2024-04-25T00:25:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.847+0000 7feda6ffd640 1 -- 172.21.15.27:0/1947490217 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fedac004fa0 con 0x7fedb0069dc0 2024-04-25T00:25:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.847+0000 7fedbae06740 1 Processor -- start 2024-04-25T00:25:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.847+0000 7fedbae06740 1 -- start start 2024-04-25T00:25:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.847+0000 7fedbae06740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096580 0x559821096960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.847+0000 7fedbae06740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559821096ea0 0x5598210a1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.847+0000 7fedbae06740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5598210a1570 0x5598210a3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.847+0000 7fedbae06740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559820ffb670 con 0x5598210a1570 2024-04-25T00:25:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.847+0000 7fedbae06740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x559820fa3970 con 0x559821096580 2024-04-25T00:25:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.847+0000 7fedbae06740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559820ff7630 con 0x559821096ea0 2024-04-25T00:25:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.847+0000 7fedb5697640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5598210a1570 0x5598210a3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.847+0000 7fedb5697640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5598210a1570 0x5598210a3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53154/0 (socket says 172.21.15.27:53154) 2024-04-25T00:25:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.847+0000 7fedb5697640 1 -- 172.21.15.27:0/1375700992 learned_addr learned my addr 172.21.15.27:0/1375700992 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.848+0000 7fedb5e98640 1 --2- 172.21.15.27:0/1375700992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559821096ea0 0x5598210a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.848+0000 7fedb6699640 1 --2- 172.21.15.27:0/1375700992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096580 0x559821096960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.848+0000 7feda6ffd640 1 --2- 172.21.15.27:0/1947490217 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fed9c0a2d30 0x7fed9c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.848+0000 7fedb5697640 1 -- 172.21.15.27:0/1375700992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559821096ea0 msgr2=0x5598210a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.848+0000 7fedb5697640 1 --2- 172.21.15.27:0/1375700992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559821096ea0 0x5598210a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.848+0000 7fedb5697640 1 -- 172.21.15.27:0/1375700992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096580 msgr2=0x559821096960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.848+0000 7fedb5697640 1 --2- 172.21.15.27:0/1375700992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096580 0x559821096960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.848+0000 7fedb5697640 1 -- 172.21.15.27:0/1375700992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559820fc7400 con 0x5598210a1570 2024-04-25T00:25:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.848+0000 7fedb6699640 1 --2- 172.21.15.27:0/1375700992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096580 0x559821096960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:25:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.848+0000 7fedb6699640 1 --2- 172.21.15.27:0/1947490217 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fed9c0a2d30 0x7fed9c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.848+0000 7feda6ffd640 1 -- 172.21.15.27:0/1947490217 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fedac0cf260 con 0x7fedb0069dc0 2024-04-25T00:25:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.848+0000 7fedb5697640 1 --2- 172.21.15.27:0/1375700992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5598210a1570 0x5598210a3960 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7feda0006010 tx=0x7feda0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de0d2b91ed1dc276 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.848+0000 7fed9bfff640 1 -- 172.21.15.27:0/1375700992 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda003f070 con 0x5598210a1570 2024-04-25T00:25:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.848+0000 7fed9bfff640 1 -- 172.21.15.27:0/1375700992 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feda0004030 con 0x5598210a1570 2024-04-25T00:25:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.848+0000 7fed9bfff640 1 -- 172.21.15.27:0/1375700992 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda0035070 con 0x5598210a1570 2024-04-25T00:25:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.849+0000 7fedbae06740 1 -- 172.21.15.27:0/1375700992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5598210a1570 msgr2=0x5598210a3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.849+0000 7fedb6699640 1 --2- 172.21.15.27:0/1947490217 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fed9c0a2d30 0x7fed9c0a51f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7feda8005fd0 tx=0x7feda8004a40 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.849+0000 7fedbae06740 1 --2- 172.21.15.27:0/1375700992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5598210a1570 0x5598210a3960 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7feda0006010 tx=0x7feda0005dd0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.849+0000 7fedbae06740 1 -- 172.21.15.27:0/1375700992 shutdown_connections 2024-04-25T00:25:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.849+0000 7fedbae06740 1 --2- 172.21.15.27:0/1375700992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5598210a1570 0x5598210a3960 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.849+0000 7fedbae06740 1 --2- 172.21.15.27:0/1375700992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559821096ea0 0x5598210a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.849+0000 7fedbae06740 1 --2- 172.21.15.27:0/1375700992 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096580 0x559821096960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.849+0000 7fedbae06740 1 -- 172.21.15.27:0/1375700992 >> 172.21.15.27:0/1375700992 conn(0x5598210928a0 msgr2=0x559821092c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.849+0000 7fedbae06740 1 -- 172.21.15.27:0/1375700992 shutdown_connections 2024-04-25T00:25:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.849+0000 7fedbae06740 1 -- 172.21.15.27:0/1375700992 wait complete. 2024-04-25T00:25:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.849+0000 7fedbae06740 1 -- start start 2024-04-25T00:25:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.849+0000 7fedbae06740 1 Processor -- start 2024-04-25T00:25:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedbae06740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559821096580 0x5598210a0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedbae06740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096ea0 0x55982109e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedbae06740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598210a1570 0x55982109e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedbae06740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5598210a4300 con 0x559821096580 2024-04-25T00:25:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedbae06740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x559820fb7050 con 0x559821096ea0 2024-04-25T00:25:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedbae06740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5598210a4480 con 0x5598210a1570 2024-04-25T00:25:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedbae06740 10 client.0 _ll_register_callbacks cb 0x559820d0ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:25:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedb5e98640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559821096580 0x5598210a0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedb5e98640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559821096580 0x5598210a0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53164/0 (socket says 172.21.15.27:53164) 2024-04-25T00:25:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedb5e98640 1 -- 172.21.15.27:0/3850387037 learned_addr learned my addr 172.21.15.27:0/3850387037 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedb6699640 1 --2- 172.21.15.27:0/3850387037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598210a1570 0x55982109e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedb5e98640 1 -- 172.21.15.27:0/3850387037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598210a1570 msgr2=0x55982109e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedb5697640 1 --2- 172.21.15.27:0/3850387037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096ea0 0x55982109e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedb5e98640 1 --2- 172.21.15.27:0/3850387037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598210a1570 0x55982109e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedb5e98640 1 -- 172.21.15.27:0/3850387037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096ea0 msgr2=0x55982109e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedb5e98640 1 --2- 172.21.15.27:0/3850387037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096ea0 0x55982109e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedb5e98640 1 -- 172.21.15.27:0/3850387037 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559820f88dc0 con 0x559821096580 2024-04-25T00:25:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedb6699640 1 --2- 172.21.15.27:0/3850387037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598210a1570 0x55982109e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:25:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fedb5e98640 1 --2- 172.21.15.27:0/3850387037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559821096580 0x5598210a0ea0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fedac00d740 tx=0x7fedac005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54922a813e01eb48 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.850+0000 7fed997fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:25:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.851+0000 7fed997fa640 1 -- 172.21.15.27:0/3850387037 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac020040 con 0x559821096580 2024-04-25T00:25:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.851+0000 7fed997fa640 1 -- 172.21.15.27:0/3850387037 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fedac097a80 con 0x559821096580 2024-04-25T00:25:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.851+0000 7fedbae06740 1 -- 172.21.15.27:0/3850387037 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feda0004030 con 0x559821096580 2024-04-25T00:25:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.851+0000 7fed997fa640 1 -- 172.21.15.27:0/3850387037 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7fedac0046c0 con 0x559821096580 2024-04-25T00:25:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.851+0000 7fedbae06740 10 client.17205 fetch_fsmap learned FSMap version 100 2024-04-25T00:25:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.851+0000 7fedbae06740 1 -- 172.21.15.27:0/3850387037 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x559820d0f3f0 con 0x559821096580 2024-04-25T00:25:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.851+0000 7fed997fa640 1 -- 172.21.15.27:0/3850387037 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feda0004030 con 0x559821096580 2024-04-25T00:25:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.851+0000 7fed997fa640 1 -- 172.21.15.27:0/3850387037 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 100) ==== 8366+0+0 (secure 0 0 0) 0x7fedac0d2c80 con 0x559821096580 2024-04-25T00:25:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.851+0000 7fedbae06740 10 client.17205 fetch_fsmap finished waiting for FSMap version 100 2024-04-25T00:25:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.851+0000 7fedbae06740 10 client.17205 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:25:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.851+0000 7fedbae06740 20 client.17205 populate_metadata read hostname 'smithi027' 2024-04-25T00:25:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.851+0000 7fedbae06740 1 --2- 172.21.15.27:0/3850387037 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5598210b1580 0x5598210b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.851+0000 7fedbae06740 4 client.17205 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:25:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.851+0000 7fedbae06740 1 -- 172.21.15.27:0/3850387037 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559821008bb0 con 0x5598210b1580 2024-04-25T00:25:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.851+0000 7fedb5697640 1 --2- 172.21.15.27:0/3850387037 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5598210b1580 0x5598210b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.852+0000 7fed997fa640 10 client.17205 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:25:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.852+0000 7fedb5697640 1 --2- 172.21.15.27:0/3850387037 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5598210b1580 0x5598210b3a40 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=32cea5fe6eae303 server_cookie=87e572fab22bc2d9 in_seq=0 out_seq=0 2024-04-25T00:25:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.853+0000 7fed997fa640 1 -- 172.21.15.27:0/3850387037 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559821008bb0 con 0x5598210b1580 2024-04-25T00:25:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.853+0000 7fed997fa640 10 client.17205 handle_command_reply: tid=0 2024-04-25T00:25:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.853+0000 7fedbae06740 1 client.17205 shutdown 2024-04-25T00:25:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.853+0000 7fedbae06740 1 -- 172.21.15.27:0/3850387037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559821096580 msgr2=0x5598210a0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.853+0000 7fedbae06740 1 --2- 172.21.15.27:0/3850387037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559821096580 0x5598210a0ea0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fedac00d740 tx=0x7fedac005d30 comp rx=0 tx=0).stop 2024-04-25T00:25:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.853+0000 7fedbae06740 1 -- 172.21.15.27:0/3850387037 shutdown_connections 2024-04-25T00:25:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.853+0000 7fedbae06740 1 --2- 172.21.15.27:0/3850387037 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x5598210b1580 0x5598210b3a40 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.853+0000 7fedbae06740 1 --2- 172.21.15.27:0/3850387037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5598210a1570 0x55982109e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.853+0000 7fedbae06740 1 --2- 172.21.15.27:0/3850387037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096ea0 0x55982109e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.853+0000 7fed997fa640 0 client.17205 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:25:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.853+0000 7fedbae06740 1 --2- 172.21.15.27:0/3850387037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559821096580 0x5598210a0ea0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.853+0000 7fedbae06740 1 -- 172.21.15.27:0/3850387037 >> 172.21.15.27:0/3850387037 conn(0x5598210928a0 msgr2=0x559821091b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.853+0000 7fedbae06740 1 -- 172.21.15.27:0/3850387037 shutdown_connections 2024-04-25T00:25:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.853+0000 7fedbae06740 1 -- 172.21.15.27:0/3850387037 wait complete. 2024-04-25T00:25:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.853+0000 7fedbae06740 20 client.17205 trim_cache size 0 max 16384 2024-04-25T00:25:30.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.866+0000 7fedbae06740 1 Processor -- start 2024-04-25T00:25:30.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.866+0000 7fedbae06740 1 -- start start 2024-04-25T00:25:30.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.866+0000 7fedbae06740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096580 0x5598210ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.866+0000 7fedbae06740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559821096ea0 0x5598210b9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.866+0000 7fedbae06740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5598210a1570 0x5598210b9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.866+0000 7fedbae06740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fedb01b0cd0 con 0x5598210a1570 2024-04-25T00:25:30.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.866+0000 7fedbae06740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fedb01b2840 con 0x559821096580 2024-04-25T00:25:30.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.866+0000 7fedbae06740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fedb01b13a0 con 0x559821096ea0 2024-04-25T00:25:30.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.867+0000 7fedb6699640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5598210a1570 0x5598210b9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.867+0000 7fedb5e98640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096580 0x5598210ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.867+0000 7fedb6699640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5598210a1570 0x5598210b9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53176/0 (socket says 172.21.15.27:53176) 2024-04-25T00:25:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.867+0000 7fedb5e98640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096580 0x5598210ad1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:51500/0 (socket says 172.21.15.27:51500) 2024-04-25T00:25:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.867+0000 7fedb6699640 1 -- 172.21.15.27:0/861459543 learned_addr learned my addr 172.21.15.27:0/861459543 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.867+0000 7fedb5697640 1 --2- 172.21.15.27:0/861459543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559821096ea0 0x5598210b9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.867+0000 7fedb6699640 1 -- 172.21.15.27:0/861459543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559821096ea0 msgr2=0x5598210b9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.867+0000 7fedb6699640 1 --2- 172.21.15.27:0/861459543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559821096ea0 0x5598210b9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.867+0000 7fedb6699640 1 -- 172.21.15.27:0/861459543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096580 msgr2=0x5598210ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.867+0000 7fedb6699640 1 --2- 172.21.15.27:0/861459543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096580 0x5598210ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.867+0000 7fedb6699640 1 -- 172.21.15.27:0/861459543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559820dd97d0 con 0x5598210a1570 2024-04-25T00:25:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.867+0000 7fedb5e98640 1 --2- 172.21.15.27:0/861459543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096580 0x5598210ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:25:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.867+0000 7fedb6699640 1 --2- 172.21.15.27:0/861459543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5598210a1570 0x5598210b9850 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7feda80128c0 tx=0x7feda8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d12c8b5a4249d17 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.867+0000 7fed9b7fe640 1 -- 172.21.15.27:0/861459543 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda8010bd0 con 0x5598210a1570 2024-04-25T00:25:30.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.867+0000 7fed9b7fe640 1 -- 172.21.15.27:0/861459543 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feda8016070 con 0x5598210a1570 2024-04-25T00:25:30.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.867+0000 7fed9b7fe640 1 -- 172.21.15.27:0/861459543 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda80045f0 con 0x5598210a1570 2024-04-25T00:25:30.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.868+0000 7fedbae06740 1 -- 172.21.15.27:0/861459543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5598210a1570 msgr2=0x5598210b9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.868+0000 7fedbae06740 1 --2- 172.21.15.27:0/861459543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5598210a1570 0x5598210b9850 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7feda80128c0 tx=0x7feda8014000 comp rx=0 tx=0).stop 2024-04-25T00:25:30.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.868+0000 7fedbae06740 1 -- 172.21.15.27:0/861459543 shutdown_connections 2024-04-25T00:25:30.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.868+0000 7fedbae06740 1 --2- 172.21.15.27:0/861459543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5598210a1570 0x5598210b9850 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.868+0000 7fedbae06740 1 --2- 172.21.15.27:0/861459543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559821096ea0 0x5598210b9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.868+0000 7fedbae06740 1 --2- 172.21.15.27:0/861459543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559821096580 0x5598210ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.868+0000 7fedbae06740 1 -- 172.21.15.27:0/861459543 >> 172.21.15.27:0/861459543 conn(0x5598210928a0 msgr2=0x559821091b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.868+0000 7fedbae06740 1 -- 172.21.15.27:0/861459543 shutdown_connections 2024-04-25T00:25:30.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.868+0000 7fedbae06740 1 -- 172.21.15.27:0/861459543 wait complete. 2024-04-25T00:25:30.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.868+0000 7fedbae06740 1 -- start start 2024-04-25T00:25:30.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.868+0000 7fedbae06740 1 Processor -- start 2024-04-25T00:25:30.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedbae06740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559821096580 0x5598210a9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedbae06740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559821096ea0 0x5598210a6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedbae06740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598210a1570 0x5598210a7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedbae06740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fedb01b2170 con 0x559821096ea0 2024-04-25T00:25:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedbae06740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fedb01b1a70 con 0x5598210a1570 2024-04-25T00:25:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedbae06740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fedb01aec90 con 0x559821096580 2024-04-25T00:25:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedbae06740 10 client.0 _ll_register_callbacks cb 0x7fedac09b260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:25:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedb5697640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559821096ea0 0x5598210a6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedb5697640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559821096ea0 0x5598210a6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53188/0 (socket says 172.21.15.27:53188) 2024-04-25T00:25:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedb5697640 1 -- 172.21.15.27:0/2175119945 learned_addr learned my addr 172.21.15.27:0/2175119945 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedb5e98640 1 --2- 172.21.15.27:0/2175119945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559821096580 0x5598210a9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedb6699640 1 --2- 172.21.15.27:0/2175119945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598210a1570 0x5598210a7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedb5697640 1 -- 172.21.15.27:0/2175119945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559821096580 msgr2=0x5598210a9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedb5697640 1 --2- 172.21.15.27:0/2175119945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559821096580 0x5598210a9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedb5697640 1 -- 172.21.15.27:0/2175119945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598210a1570 msgr2=0x5598210a7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedb5697640 1 --2- 172.21.15.27:0/2175119945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598210a1570 0x5598210a7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedb5697640 1 -- 172.21.15.27:0/2175119945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598210a7b40 con 0x559821096ea0 2024-04-25T00:25:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedb6699640 1 --2- 172.21.15.27:0/2175119945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598210a1570 0x5598210a7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:25:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedb5697640 1 --2- 172.21.15.27:0/2175119945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559821096ea0 0x5598210a6e00 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7feda002a940 tx=0x7feda0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f112d052502f0bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fed997fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:25:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fed997fa640 1 -- 172.21.15.27:0/2175119945 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda0035040 con 0x559821096ea0 2024-04-25T00:25:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fed997fa640 1 -- 172.21.15.27:0/2175119945 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7feda0004020 con 0x559821096ea0 2024-04-25T00:25:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.869+0000 7fedbae06740 1 -- 172.21.15.27:0/2175119945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feda8016070 con 0x559821096ea0 2024-04-25T00:25:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.870+0000 7fed997fa640 1 -- 172.21.15.27:0/2175119945 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7feda005a050 con 0x559821096ea0 2024-04-25T00:25:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.870+0000 7fedbae06740 10 client.17211 fetch_fsmap learned FSMap version 100 2024-04-25T00:25:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.870+0000 7fedbae06740 1 -- 172.21.15.27:0/2175119945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x559821117bb0 con 0x559821096ea0 2024-04-25T00:25:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.870+0000 7fed997fa640 1 -- 172.21.15.27:0/2175119945 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feda8016070 con 0x559821096ea0 2024-04-25T00:25:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.870+0000 7fed997fa640 1 -- 172.21.15.27:0/2175119945 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 100) ==== 8366+0+0 (secure 0 0 0) 0x7feda0005240 con 0x559821096ea0 2024-04-25T00:25:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.870+0000 7fedbae06740 10 client.17211 fetch_fsmap finished waiting for FSMap version 100 2024-04-25T00:25:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.870+0000 7fedbae06740 10 client.17211 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:25:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.870+0000 7fedbae06740 20 client.17211 populate_metadata read hostname 'smithi027' 2024-04-25T00:25:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.870+0000 7fedbae06740 1 --2- 172.21.15.27:0/2175119945 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x559821118000 0x55982111a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.870+0000 7fedbae06740 4 client.17211 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:25:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.870+0000 7fedbae06740 1 -- 172.21.15.27:0/2175119945 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559821002f00 con 0x559821118000 2024-04-25T00:25:30.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.870+0000 7fedb5e98640 1 --2- 172.21.15.27:0/2175119945 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x559821118000 0x55982111a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.871+0000 7fedb5e98640 1 --2- 172.21.15.27:0/2175119945 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x559821118000 0x55982111a3f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b3433446352ff4f3 server_cookie=8abca4b922d4a254 in_seq=0 out_seq=0 2024-04-25T00:25:30.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.871+0000 7fed997fa640 10 client.17211 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:25:30.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.871+0000 7fed997fa640 1 -- 172.21.15.27:0/2175119945 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559821002f00 con 0x559821118000 2024-04-25T00:25:30.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.871+0000 7fed997fa640 10 client.17211 handle_command_reply: tid=0 2024-04-25T00:25:30.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.871+0000 7fedbae06740 1 client.17211 shutdown 2024-04-25T00:25:30.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.871+0000 7fedbae06740 1 -- 172.21.15.27:0/2175119945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559821096ea0 msgr2=0x5598210a6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.871+0000 7fedbae06740 1 --2- 172.21.15.27:0/2175119945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559821096ea0 0x5598210a6e00 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7feda002a940 tx=0x7feda0005d50 comp rx=0 tx=0).stop 2024-04-25T00:25:30.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.871+0000 7fedbae06740 1 -- 172.21.15.27:0/2175119945 shutdown_connections 2024-04-25T00:25:30.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.871+0000 7fedbae06740 1 --2- 172.21.15.27:0/2175119945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5598210a1570 0x5598210a7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.871+0000 7fedbae06740 1 --2- 172.21.15.27:0/2175119945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559821096ea0 0x5598210a6e00 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.871+0000 7fedbae06740 1 --2- 172.21.15.27:0/2175119945 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x559821118000 0x55982111a3f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.871+0000 7fedbae06740 1 --2- 172.21.15.27:0/2175119945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559821096580 0x5598210a9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.871+0000 7fed997fa640 0 client.17211 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:25:30.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.871+0000 7fedbae06740 1 -- 172.21.15.27:0/2175119945 >> 172.21.15.27:0/2175119945 conn(0x5598210928a0 msgr2=0x5598210ac1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.871+0000 7fedbae06740 1 -- 172.21.15.27:0/2175119945 shutdown_connections 2024-04-25T00:25:30.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.871+0000 7fedbae06740 1 -- 172.21.15.27:0/2175119945 wait complete. 2024-04-25T00:25:30.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.871+0000 7fedbae06740 20 client.17211 trim_cache size 0 max 16384 2024-04-25T00:25:30.879 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:25:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.872+0000 7fedb80fc640 1 -- 172.21.15.27:0/1947490217 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fed9c0a2d30 msgr2=0x7fed9c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.872+0000 7fedb80fc640 1 --2- 172.21.15.27:0/1947490217 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fed9c0a2d30 0x7fed9c0a51f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7feda8005fd0 tx=0x7feda8004a40 comp rx=0 tx=0).stop 2024-04-25T00:25:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.872+0000 7fedb80fc640 1 -- 172.21.15.27:0/1947490217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb0069dc0 msgr2=0x7fedb0100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.872+0000 7fedb80fc640 1 --2- 172.21.15.27:0/1947490217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb0069dc0 0x7fedb0100670 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fedac00d770 tx=0x7fedac00dc30 comp rx=0 tx=0).stop 2024-04-25T00:25:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.872+0000 7fedb80fc640 1 -- 172.21.15.27:0/1947490217 shutdown_connections 2024-04-25T00:25:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.872+0000 7fedb80fc640 1 --2- 172.21.15.27:0/1947490217 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fed9c0a2d30 0x7fed9c0a51f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.872+0000 7fedb80fc640 1 --2- 172.21.15.27:0/1947490217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fedb0107730 0x7fedb01010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.872+0000 7fedb80fc640 1 --2- 172.21.15.27:0/1947490217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fedb006a780 0x7fedb0100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.872+0000 7fedb80fc640 1 --2- 172.21.15.27:0/1947490217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb0069dc0 0x7fedb0100670 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.872+0000 7fedb80fc640 1 -- 172.21.15.27:0/1947490217 >> 172.21.15.27:0/1947490217 conn(0x7fedb00776a0 msgr2=0x7fedb00fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.872+0000 7fedb80fc640 1 -- 172.21.15.27:0/1947490217 shutdown_connections 2024-04-25T00:25:30.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.872+0000 7fedb80fc640 1 -- 172.21.15.27:0/1947490217 wait complete. 2024-04-25T00:25:30.883 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:30 smithi027 ceph-mon[29208]: pgmap v295: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:25:30.883 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:30 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:25:30.883 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:30 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2541294640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:25:30.884 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T00:25:30.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.987+0000 7f8c7b7e2640 1 -- 172.21.15.27:0/3172688197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c741029c0 msgr2=0x7f8c74102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.987+0000 7f8c7b7e2640 1 --2- 172.21.15.27:0/3172688197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c741029c0 0x7f8c74102da0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f8c70009900 tx=0x7f8c7002a510 comp rx=0 tx=0).stop 2024-04-25T00:25:30.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.987+0000 7f8c7b7e2640 1 -- 172.21.15.27:0/3172688197 shutdown_connections 2024-04-25T00:25:30.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.987+0000 7f8c7b7e2640 1 --2- 172.21.15.27:0/3172688197 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c7410bd00 0x7f8c7410e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.987+0000 7f8c7b7e2640 1 --2- 172.21.15.27:0/3172688197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c741032e0 0x7f8c7410b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.987+0000 7f8c7b7e2640 1 --2- 172.21.15.27:0/3172688197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c741029c0 0x7f8c74102da0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.987+0000 7f8c7b7e2640 1 -- 172.21.15.27:0/3172688197 >> 172.21.15.27:0/3172688197 conn(0x7f8c74077740 msgr2=0x7f8c74077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.987+0000 7f8c7b7e2640 1 -- 172.21.15.27:0/3172688197 shutdown_connections 2024-04-25T00:25:30.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.987+0000 7f8c7b7e2640 1 -- 172.21.15.27:0/3172688197 wait complete. 2024-04-25T00:25:30.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.987+0000 7f8c7b7e2640 1 Processor -- start 2024-04-25T00:25:30.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.987+0000 7f8c7b7e2640 1 -- start start 2024-04-25T00:25:30.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c7b7e2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c741029c0 0x7f8c741980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c7b7e2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c741032e0 0x7f8c74198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c7b7e2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c7410bd00 0x7f8c7419c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c7b7e2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8c74111b00 con 0x7f8c741029c0 2024-04-25T00:25:30.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c7b7e2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8c74111980 con 0x7f8c741032e0 2024-04-25T00:25:30.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c7b7e2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8c74111c80 con 0x7f8c7410bd00 2024-04-25T00:25:30.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c7957e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c741029c0 0x7f8c741980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c7957e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c741029c0 0x7f8c741980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53206/0 (socket says 172.21.15.27:53206) 2024-04-25T00:25:30.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c7957e640 1 -- 172.21.15.27:0/1589330188 learned_addr learned my addr 172.21.15.27:0/1589330188 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c7957e640 1 -- 172.21.15.27:0/1589330188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c7410bd00 msgr2=0x7f8c7419c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c79d7f640 1 --2- 172.21.15.27:0/1589330188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c7410bd00 0x7f8c7419c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c78d7d640 1 --2- 172.21.15.27:0/1589330188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c741032e0 0x7f8c74198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c7957e640 1 --2- 172.21.15.27:0/1589330188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c7410bd00 0x7f8c7419c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c7957e640 1 -- 172.21.15.27:0/1589330188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c741032e0 msgr2=0x7f8c74198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c7957e640 1 --2- 172.21.15.27:0/1589330188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c741032e0 0x7f8c74198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c7957e640 1 -- 172.21.15.27:0/1589330188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7419d0b0 con 0x7f8c741029c0 2024-04-25T00:25:30.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c7957e640 1 --2- 172.21.15.27:0/1589330188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c741029c0 0x7f8c741980c0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f8c70006010 tx=0x7f8c700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c6a7fc640 1 -- 172.21.15.27:0/1589330188 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7003f030 con 0x7f8c741029c0 2024-04-25T00:25:30.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c6a7fc640 1 -- 172.21.15.27:0/1589330188 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c70003ce0 con 0x7f8c741029c0 2024-04-25T00:25:30.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c7b7e2640 1 -- 172.21.15.27:0/1589330188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c7419d340 con 0x7f8c741029c0 2024-04-25T00:25:30.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c6a7fc640 1 -- 172.21.15.27:0/1589330188 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c70035070 con 0x7f8c741029c0 2024-04-25T00:25:30.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.988+0000 7f8c7b7e2640 1 -- 172.21.15.27:0/1589330188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c741a4c30 con 0x7f8c741029c0 2024-04-25T00:25:30.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.991+0000 7f8c6a7fc640 1 -- 172.21.15.27:0/1589330188 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8c70042030 con 0x7f8c741029c0 2024-04-25T00:25:30.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.991+0000 7f8c7e5c5740 1 Processor -- start 2024-04-25T00:25:30.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.991+0000 7f8c7e5c5740 1 -- start start 2024-04-25T00:25:30.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c7e5c5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 0x55a4d0c6e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c7e5c5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c79030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c7e5c5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c79570 0x55a4d0c7b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c7e5c5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a4d0bd3670 con 0x55a4d0c79570 2024-04-25T00:25:30.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c7e5c5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a4d0b7b970 con 0x55a4d0c6eea0 2024-04-25T00:25:30.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c7e5c5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a4d0bcf630 con 0x55a4d0c6e580 2024-04-25T00:25:30.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c78d7d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c79570 0x55a4d0c7b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c78d7d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c79570 0x55a4d0c7b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53222/0 (socket says 172.21.15.27:53222) 2024-04-25T00:25:30.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c78d7d640 1 -- 172.21.15.27:0/1174403012 learned_addr learned my addr 172.21.15.27:0/1174403012 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:30.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c79d7f640 1 --2- 172.21.15.27:0/1174403012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 0x55a4d0c6e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c7957e640 1 --2- 172.21.15.27:0/1174403012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c79030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c6a7fc640 1 --2- 172.21.15.27:0/1589330188 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8c600a2d00 0x7f8c600a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c78d7d640 1 -- 172.21.15.27:0/1174403012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 msgr2=0x55a4d0c6e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c78d7d640 1 --2- 172.21.15.27:0/1174403012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 0x55a4d0c6e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c78d7d640 1 -- 172.21.15.27:0/1174403012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c6eea0 msgr2=0x55a4d0c79030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c78d7d640 1 --2- 172.21.15.27:0/1174403012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c79030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c78d7d640 1 -- 172.21.15.27:0/1174403012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4d0b9f400 con 0x55a4d0c79570 2024-04-25T00:25:30.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c79d7f640 1 --2- 172.21.15.27:0/1589330188 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8c600a2d00 0x7f8c600a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c6a7fc640 1 -- 172.21.15.27:0/1589330188 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8c700efe50 con 0x7f8c741029c0 2024-04-25T00:25:30.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.992+0000 7f8c78d7d640 1 --2- 172.21.15.27:0/1174403012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c79570 0x55a4d0c7b960 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009c00 tx=0x7f8c64010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fbe98b0d53c7108 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.993+0000 7f8c5f7fe640 1 -- 172.21.15.27:0/1174403012 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6401b480 con 0x55a4d0c79570 2024-04-25T00:25:30.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.993+0000 7f8c5f7fe640 1 -- 172.21.15.27:0/1174403012 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c6401bb00 con 0x55a4d0c79570 2024-04-25T00:25:30.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.993+0000 7f8c5f7fe640 1 -- 172.21.15.27:0/1174403012 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c64012ac0 con 0x55a4d0c79570 2024-04-25T00:25:30.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.993+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/1174403012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c79570 msgr2=0x55a4d0c7b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.993+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/1174403012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c79570 0x55a4d0c7b960 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009c00 tx=0x7f8c64010740 comp rx=0 tx=0).stop 2024-04-25T00:25:30.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.993+0000 7f8c79d7f640 1 --2- 172.21.15.27:0/1589330188 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8c600a2d00 0x7f8c600a51c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00bd50 tx=0x7f8c6c004a40 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.993+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/1174403012 shutdown_connections 2024-04-25T00:25:30.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.993+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/1174403012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c79570 0x55a4d0c7b960 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.993+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/1174403012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c79030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.993+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/1174403012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 0x55a4d0c6e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.993+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/1174403012 >> 172.21.15.27:0/1174403012 conn(0x55a4d0c6a8a0 msgr2=0x55a4d0c6ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.993+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/1174403012 shutdown_connections 2024-04-25T00:25:30.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.993+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/1174403012 wait complete. 2024-04-25T00:25:31.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.994+0000 7f8c7e5c5740 1 -- start start 2024-04-25T00:25:31.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.994+0000 7f8c7e5c5740 1 Processor -- start 2024-04-25T00:25:31.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.994+0000 7f8c7e5c5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 0x55a4d0c78e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:31.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.994+0000 7f8c7e5c5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c76130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:31.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.994+0000 7f8c7e5c5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c79570 0x55a4d0c76670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:31.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.994+0000 7f8c7e5c5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a4d0c7c360 con 0x55a4d0c6eea0 2024-04-25T00:25:31.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.994+0000 7f8c7e5c5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a4d0b8f050 con 0x55a4d0c79570 2024-04-25T00:25:31.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.994+0000 7f8c7e5c5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a4d0c7c4e0 con 0x55a4d0c6e580 2024-04-25T00:25:31.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.994+0000 7f8c7e5c5740 10 client.0 _ll_register_callbacks cb 0x55a4d08e6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:25:31.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.994+0000 7f8c78d7d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c76130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:31.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.994+0000 7f8c78d7d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c76130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53228/0 (socket says 172.21.15.27:53228) 2024-04-25T00:25:31.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.994+0000 7f8c78d7d640 1 -- 172.21.15.27:0/3774990277 learned_addr learned my addr 172.21.15.27:0/3774990277 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:31.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.994+0000 7f8c7957e640 1 --2- 172.21.15.27:0/3774990277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 0x55a4d0c78e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:31.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c78d7d640 1 -- 172.21.15.27:0/3774990277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 msgr2=0x55a4d0c78e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:31.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c78d7d640 1 --2- 172.21.15.27:0/3774990277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 0x55a4d0c78e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c78d7d640 1 -- 172.21.15.27:0/3774990277 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c79570 msgr2=0x55a4d0c76670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:31.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c78d7d640 1 --2- 172.21.15.27:0/3774990277 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c79570 0x55a4d0c76670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c78d7d640 1 -- 172.21.15.27:0/3774990277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4d0b60dc0 con 0x55a4d0c6eea0 2024-04-25T00:25:31.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c78d7d640 1 --2- 172.21.15.27:0/3774990277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c76130 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009f20 tx=0x7f8c64009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a99cd48cc81c1148 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:31.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c5cff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:25:31.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c5cff9640 1 -- 172.21.15.27:0/3774990277 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6402a060 con 0x55a4d0c6eea0 2024-04-25T00:25:31.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c5cff9640 1 -- 172.21.15.27:0/3774990277 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8c64005250 con 0x55a4d0c6eea0 2024-04-25T00:25:31.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3774990277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c6401bb00 con 0x55a4d0c6eea0 2024-04-25T00:25:31.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c5cff9640 1 -- 172.21.15.27:0/3774990277 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f8c64031050 con 0x55a4d0c6eea0 2024-04-25T00:25:31.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c7e5c5740 10 client.17223 fetch_fsmap learned FSMap version 100 2024-04-25T00:25:31.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3774990277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x55a4d08e73f0 con 0x55a4d0c6eea0 2024-04-25T00:25:31.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c5cff9640 1 -- 172.21.15.27:0/3774990277 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c6401bb00 con 0x55a4d0c6eea0 2024-04-25T00:25:31.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c5cff9640 1 -- 172.21.15.27:0/3774990277 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 100) ==== 8366+0+0 (secure 0 0 0) 0x7f8c6400f040 con 0x55a4d0c6eea0 2024-04-25T00:25:31.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c7e5c5740 10 client.17223 fetch_fsmap finished waiting for FSMap version 100 2024-04-25T00:25:31.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c7e5c5740 10 client.17223 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:25:31.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c7e5c5740 20 client.17223 populate_metadata read hostname 'smithi027' 2024-04-25T00:25:31.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/3774990277 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55a4d0c894d0 0x55a4d0c8b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:31.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c7e5c5740 4 client.17223 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:25:31.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.995+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3774990277 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4d0be0bb0 con 0x55a4d0c894d0 2024-04-25T00:25:31.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.996+0000 7f8c7957e640 1 --2- 172.21.15.27:0/3774990277 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55a4d0c894d0 0x55a4d0c8b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:31.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.996+0000 7f8c7957e640 1 --2- 172.21.15.27:0/3774990277 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55a4d0c894d0 0x55a4d0c8b8c0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f3de378334bd2a0 server_cookie=875cc6681a5ae874 in_seq=0 out_seq=0 2024-04-25T00:25:31.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.996+0000 7f8c5cff9640 10 client.17223 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:25:31.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.997+0000 7f8c5cff9640 1 -- 172.21.15.27:0/3774990277 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a4d0be0bb0 con 0x55a4d0c894d0 2024-04-25T00:25:31.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.997+0000 7f8c5cff9640 10 client.17223 handle_command_reply: tid=0 2024-04-25T00:25:31.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.997+0000 7f8c7e5c5740 1 client.17223 shutdown 2024-04-25T00:25:31.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.997+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3774990277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 msgr2=0x55a4d0c76130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:31.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.997+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/3774990277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c76130 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009f20 tx=0x7f8c64009f50 comp rx=0 tx=0).stop 2024-04-25T00:25:31.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.997+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3774990277 shutdown_connections 2024-04-25T00:25:31.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.997+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/3774990277 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55a4d0c894d0 0x55a4d0c8b8c0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.997+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/3774990277 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c79570 0x55a4d0c76670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.997+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/3774990277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c76130 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.997+0000 7f8c5cff9640 0 client.17223 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:25:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.997+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/3774990277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 0x55a4d0c78e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.997+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3774990277 >> 172.21.15.27:0/3774990277 conn(0x55a4d0c6a8a0 msgr2=0x55a4d0c69b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.997+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3774990277 shutdown_connections 2024-04-25T00:25:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.997+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3774990277 wait complete. 2024-04-25T00:25:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:30.997+0000 7f8c7e5c5740 20 client.17223 trim_cache size 0 max 16384 2024-04-25T00:25:31.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.010+0000 7f8c7e5c5740 1 Processor -- start 2024-04-25T00:25:31.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.010+0000 7f8c7e5c5740 1 -- start start 2024-04-25T00:25:31.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c7e5c5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 0x55a4d0c85220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:31.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c7e5c5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c715a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:31.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c7e5c5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c79570 0x55a4d0c78970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:31.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c7e5c5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8c741aeb40 con 0x55a4d0c6eea0 2024-04-25T00:25:31.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c7e5c5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8c741b06b0 con 0x55a4d0c79570 2024-04-25T00:25:31.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c7e5c5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8c741af210 con 0x55a4d0c6e580 2024-04-25T00:25:31.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c78d7d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c715a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c78d7d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c715a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53242/0 (socket says 172.21.15.27:53242) 2024-04-25T00:25:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c78d7d640 1 -- 172.21.15.27:0/3201597801 learned_addr learned my addr 172.21.15.27:0/3201597801 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c7957e640 1 --2- 172.21.15.27:0/3201597801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 0x55a4d0c85220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c79d7f640 1 --2- 172.21.15.27:0/3201597801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c79570 0x55a4d0c78970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c78d7d640 1 -- 172.21.15.27:0/3201597801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 msgr2=0x55a4d0c85220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c78d7d640 1 --2- 172.21.15.27:0/3201597801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 0x55a4d0c85220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c78d7d640 1 -- 172.21.15.27:0/3201597801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c79570 msgr2=0x55a4d0c78970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c78d7d640 1 --2- 172.21.15.27:0/3201597801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c79570 0x55a4d0c78970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c78d7d640 1 -- 172.21.15.27:0/3201597801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4d09b17d0 con 0x55a4d0c6eea0 2024-04-25T00:25:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c78d7d640 1 --2- 172.21.15.27:0/3201597801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c715a0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f8c6401b810 tx=0x7f8c64005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0f819e5fd36a0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c5effd640 1 -- 172.21.15.27:0/3201597801 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6402a070 con 0x55a4d0c6eea0 2024-04-25T00:25:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c5effd640 1 -- 172.21.15.27:0/3201597801 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c64004da0 con 0x55a4d0c6eea0 2024-04-25T00:25:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.011+0000 7f8c5effd640 1 -- 172.21.15.27:0/3201597801 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c64012450 con 0x55a4d0c6eea0 2024-04-25T00:25:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.012+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3201597801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 msgr2=0x55a4d0c715a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:31.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.012+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/3201597801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c715a0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f8c6401b810 tx=0x7f8c64005850 comp rx=0 tx=0).stop 2024-04-25T00:25:31.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.012+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3201597801 shutdown_connections 2024-04-25T00:25:31.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.012+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/3201597801 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c79570 0x55a4d0c78970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.012+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/3201597801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c715a0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.012+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/3201597801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 0x55a4d0c85220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.012+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3201597801 >> 172.21.15.27:0/3201597801 conn(0x55a4d0c6a8a0 msgr2=0x55a4d0c70e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:31.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.012+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3201597801 shutdown_connections 2024-04-25T00:25:31.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.012+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3201597801 wait complete. 2024-04-25T00:25:31.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.012+0000 7f8c7e5c5740 1 -- start start 2024-04-25T00:25:31.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.012+0000 7f8c7e5c5740 1 Processor -- start 2024-04-25T00:25:31.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c7e5c5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 0x55a4d0c7d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:31.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c7e5c5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c7d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:31.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c7e5c5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c79570 0x55a4d0c7dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:31.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c7e5c5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8c741affe0 con 0x55a4d0c6eea0 2024-04-25T00:25:31.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c7e5c5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8c741af8e0 con 0x55a4d0c79570 2024-04-25T00:25:31.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c7e5c5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8c741acb00 con 0x55a4d0c6e580 2024-04-25T00:25:31.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c7e5c5740 10 client.0 _ll_register_callbacks cb 0x7f8c64009a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:25:31.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c78d7d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c7d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:31.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c78d7d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c7d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53250/0 (socket says 172.21.15.27:53250) 2024-04-25T00:25:31.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c78d7d640 1 -- 172.21.15.27:0/3643633163 learned_addr learned my addr 172.21.15.27:0/3643633163 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:31.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c7957e640 1 --2- 172.21.15.27:0/3643633163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 0x55a4d0c7d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:31.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c79d7f640 1 --2- 172.21.15.27:0/3643633163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c79570 0x55a4d0c7dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:31.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c78d7d640 1 -- 172.21.15.27:0/3643633163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 msgr2=0x55a4d0c7d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:31.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c78d7d640 1 --2- 172.21.15.27:0/3643633163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 0x55a4d0c7d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c78d7d640 1 -- 172.21.15.27:0/3643633163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c79570 msgr2=0x55a4d0c7dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:31.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c78d7d640 1 --2- 172.21.15.27:0/3643633163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c79570 0x55a4d0c7dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c78d7d640 1 -- 172.21.15.27:0/3643633163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4d0cee7e0 con 0x55a4d0c6eea0 2024-04-25T00:25:31.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c78d7d640 1 --2- 172.21.15.27:0/3643633163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c7d690 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f8c64015040 tx=0x7f8c64004f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21fbef74be5c7b55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:31.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c5cff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:25:31.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.013+0000 7f8c5cff9640 1 -- 172.21.15.27:0/3643633163 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6402a040 con 0x55a4d0c6eea0 2024-04-25T00:25:31.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.014+0000 7f8c5cff9640 1 -- 172.21.15.27:0/3643633163 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8c6401bb20 con 0x55a4d0c6eea0 2024-04-25T00:25:31.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.014+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3643633163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c64004da0 con 0x55a4d0c6eea0 2024-04-25T00:25:31.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.014+0000 7f8c5cff9640 1 -- 172.21.15.27:0/3643633163 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f8c6401bd20 con 0x55a4d0c6eea0 2024-04-25T00:25:31.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.014+0000 7f8c7e5c5740 10 client.17229 fetch_fsmap learned FSMap version 100 2024-04-25T00:25:31.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.014+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3643633163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x55a4d0cefa80 con 0x55a4d0c6eea0 2024-04-25T00:25:31.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.014+0000 7f8c5cff9640 1 -- 172.21.15.27:0/3643633163 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c64004da0 con 0x55a4d0c6eea0 2024-04-25T00:25:31.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.014+0000 7f8c5cff9640 1 -- 172.21.15.27:0/3643633163 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 100) ==== 8366+0+0 (secure 0 0 0) 0x7f8c6403d070 con 0x55a4d0c6eea0 2024-04-25T00:25:31.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.014+0000 7f8c7e5c5740 10 client.17229 fetch_fsmap finished waiting for FSMap version 100 2024-04-25T00:25:31.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.014+0000 7f8c7e5c5740 10 client.17229 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:25:31.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.014+0000 7f8c7e5c5740 20 client.17229 populate_metadata read hostname 'smithi027' 2024-04-25T00:25:31.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.014+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/3643633163 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55a4d0cf00f0 0x55a4d0cf2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:31.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.014+0000 7f8c7e5c5740 4 client.17229 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:25:31.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.014+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3643633163 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a4d0bdaf00 con 0x55a4d0cf00f0 2024-04-25T00:25:31.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.014+0000 7f8c7957e640 1 --2- 172.21.15.27:0/3643633163 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55a4d0cf00f0 0x55a4d0cf2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:31.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.015+0000 7f8c5cff9640 10 client.17229 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:25:31.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.015+0000 7f8c7957e640 1 --2- 172.21.15.27:0/3643633163 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55a4d0cf00f0 0x55a4d0cf2510 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ef00e0bc8959b73 server_cookie=a014bd59e2e639c3 in_seq=0 out_seq=0 2024-04-25T00:25:31.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.015+0000 7f8c5cff9640 1 -- 172.21.15.27:0/3643633163 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a4d0bdaf00 con 0x55a4d0cf00f0 2024-04-25T00:25:31.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.015+0000 7f8c5cff9640 10 client.17229 handle_command_reply: tid=0 2024-04-25T00:25:31.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.015+0000 7f8c7e5c5740 1 client.17229 shutdown 2024-04-25T00:25:31.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.015+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3643633163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 msgr2=0x55a4d0c7d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:31.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.015+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/3643633163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c7d690 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f8c64015040 tx=0x7f8c64004f50 comp rx=0 tx=0).stop 2024-04-25T00:25:31.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.015+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3643633163 shutdown_connections 2024-04-25T00:25:31.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.015+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/3643633163 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55a4d0cf00f0 0x55a4d0cf2510 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.015+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/3643633163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4d0c79570 0x55a4d0c7dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.015+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/3643633163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4d0c6eea0 0x55a4d0c7d690 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.015+0000 7f8c5cff9640 0 client.17229 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:25:31.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.015+0000 7f8c7e5c5740 1 --2- 172.21.15.27:0/3643633163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4d0c6e580 0x55a4d0c7d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.015+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3643633163 >> 172.21.15.27:0/3643633163 conn(0x55a4d0c6a8a0 msgr2=0x55a4d0c70670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:31.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.015+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3643633163 shutdown_connections 2024-04-25T00:25:31.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.015+0000 7f8c7e5c5740 1 -- 172.21.15.27:0/3643633163 wait complete. 2024-04-25T00:25:31.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.015+0000 7f8c7e5c5740 20 client.17229 trim_cache size 0 max 16384 2024-04-25T00:25:31.023 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:25:31.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.016+0000 7f8c7b7e2640 1 -- 172.21.15.27:0/1589330188 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8c600a2d00 msgr2=0x7f8c600a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:31.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.016+0000 7f8c7b7e2640 1 --2- 172.21.15.27:0/1589330188 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8c600a2d00 0x7f8c600a51c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00bd50 tx=0x7f8c6c004a40 comp rx=0 tx=0).stop 2024-04-25T00:25:31.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.016+0000 7f8c7b7e2640 1 -- 172.21.15.27:0/1589330188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c741029c0 msgr2=0x7f8c741980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:31.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.016+0000 7f8c7b7e2640 1 --2- 172.21.15.27:0/1589330188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c741029c0 0x7f8c741980c0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f8c70006010 tx=0x7f8c700042f0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.016+0000 7f8c7b7e2640 1 -- 172.21.15.27:0/1589330188 shutdown_connections 2024-04-25T00:25:31.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.016+0000 7f8c7b7e2640 1 --2- 172.21.15.27:0/1589330188 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8c600a2d00 0x7f8c600a51c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.016+0000 7f8c7b7e2640 1 --2- 172.21.15.27:0/1589330188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c7410bd00 0x7f8c7419c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.016+0000 7f8c7b7e2640 1 --2- 172.21.15.27:0/1589330188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c741032e0 0x7f8c74198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.016+0000 7f8c7b7e2640 1 --2- 172.21.15.27:0/1589330188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c741029c0 0x7f8c741980c0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.016+0000 7f8c7b7e2640 1 -- 172.21.15.27:0/1589330188 >> 172.21.15.27:0/1589330188 conn(0x7f8c74077740 msgr2=0x7f8c740fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:31.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.016+0000 7f8c7b7e2640 1 -- 172.21.15.27:0/1589330188 shutdown_connections 2024-04-25T00:25:31.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:31.016+0000 7f8c7b7e2640 1 -- 172.21.15.27:0/1589330188 wait complete. 2024-04-25T00:25:31.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:30 smithi139 ceph-mon[32833]: pgmap v295: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:25:31.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:30 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:25:31.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:30 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2541294640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:25:31.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:30 smithi153 ceph-mon[32896]: pgmap v295: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:25:31.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:30 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:25:31.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:30 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2541294640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:25:32.029 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T00:25:32.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:25:32.029 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T00:25:32.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.148+0000 7fb31eedd640 1 -- 172.21.15.27:0/958699085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318100620 msgr2=0x7fb318100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.148+0000 7fb31eedd640 1 --2- 172.21.15.27:0/958699085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318100620 0x7fb318100aa0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fb304009930 tx=0x7fb30402a420 comp rx=0 tx=0).stop 2024-04-25T00:25:32.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.148+0000 7fb31eedd640 1 -- 172.21.15.27:0/958699085 shutdown_connections 2024-04-25T00:25:32.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.148+0000 7fb31eedd640 1 --2- 172.21.15.27:0/958699085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb318100fe0 0x7fb31810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.148+0000 7fb31eedd640 1 --2- 172.21.15.27:0/958699085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318100620 0x7fb318100aa0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.148+0000 7fb31eedd640 1 --2- 172.21.15.27:0/958699085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb318106630 0x7fb318106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.148+0000 7fb31eedd640 1 -- 172.21.15.27:0/958699085 >> 172.21.15.27:0/958699085 conn(0x7fb3180776a0 msgr2=0x7fb318077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:32.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.148+0000 7fb31eedd640 1 -- 172.21.15.27:0/958699085 shutdown_connections 2024-04-25T00:25:32.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.148+0000 7fb31eedd640 1 -- 172.21.15.27:0/958699085 wait complete. 2024-04-25T00:25:32.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb31eedd640 1 Processor -- start 2024-04-25T00:25:32.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb31eedd640 1 -- start start 2024-04-25T00:25:32.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb31eedd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb318100620 0x7fb318197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb31eedd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb318100fe0 0x7fb3181984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb31eedd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318106630 0x7fb31819c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb31eedd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb318110a40 con 0x7fb318106630 2024-04-25T00:25:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb31eedd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181108c0 con 0x7fb318100620 2024-04-25T00:25:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb31eedd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb318110bc0 con 0x7fb318100fe0 2024-04-25T00:25:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb31d47a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318106630 0x7fb31819c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb31d47a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318106630 0x7fb31819c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53278/0 (socket says 172.21.15.27:53278) 2024-04-25T00:25:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb31d47a640 1 -- 172.21.15.27:0/2219458373 learned_addr learned my addr 172.21.15.27:0/2219458373 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb317fff640 1 --2- 172.21.15.27:0/2219458373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb318100fe0 0x7fb3181984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb31cc79640 1 --2- 172.21.15.27:0/2219458373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb318100620 0x7fb318197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb31d47a640 1 -- 172.21.15.27:0/2219458373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb318100fe0 msgr2=0x7fb3181984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb31d47a640 1 --2- 172.21.15.27:0/2219458373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb318100fe0 0x7fb3181984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb31d47a640 1 -- 172.21.15.27:0/2219458373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb318100620 msgr2=0x7fb318197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb31d47a640 1 --2- 172.21.15.27:0/2219458373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb318100620 0x7fb318197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb31d47a640 1 -- 172.21.15.27:0/2219458373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb31819cf60 con 0x7fb318106630 2024-04-25T00:25:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.149+0000 7fb317fff640 1 --2- 172.21.15.27:0/2219458373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb318100fe0 0x7fb3181984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:25:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.150+0000 7fb31d47a640 1 --2- 172.21.15.27:0/2219458373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318106630 0x7fb31819c830 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb31000eaa0 tx=0x7fb31000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:32.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.150+0000 7fb315ffb640 1 -- 172.21.15.27:0/2219458373 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31000cdb0 con 0x7fb318106630 2024-04-25T00:25:32.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.150+0000 7fb315ffb640 1 -- 172.21.15.27:0/2219458373 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb310005020 con 0x7fb318106630 2024-04-25T00:25:32.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.150+0000 7fb31eedd640 1 -- 172.21.15.27:0/2219458373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb31819d250 con 0x7fb318106630 2024-04-25T00:25:32.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.150+0000 7fb315ffb640 1 -- 172.21.15.27:0/2219458373 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3100116b0 con 0x7fb318106630 2024-04-25T00:25:32.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.150+0000 7fb31eedd640 1 -- 172.21.15.27:0/2219458373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3181a4ae0 con 0x7fb318106630 2024-04-25T00:25:32.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.153+0000 7fb315ffb640 1 -- 172.21.15.27:0/2219458373 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb310005560 con 0x7fb318106630 2024-04-25T00:25:32.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.153+0000 7fb321bea740 1 Processor -- start 2024-04-25T00:25:32.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.153+0000 7fb321bea740 1 -- start start 2024-04-25T00:25:32.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.153+0000 7fb321bea740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99452980 0x55ee99450cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.153+0000 7fb321bea740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99451290 0x55ee99451710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.153+0000 7fb321bea740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee9945c010 0x55ee9945e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.153+0000 7fb321bea740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ee993b52c0 con 0x55ee99452980 2024-04-25T00:25:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.153+0000 7fb321bea740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ee9935d910 con 0x55ee9945c010 2024-04-25T00:25:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.153+0000 7fb321bea740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ee993b1280 con 0x55ee99451290 2024-04-25T00:25:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb315ffb640 1 --2- 172.21.15.27:0/2219458373 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb3080a2d80 0x7fb3080a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb317fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99452980 0x55ee99450cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb317fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99452980 0x55ee99450cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53294/0 (socket says 172.21.15.27:53294) 2024-04-25T00:25:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb317fff640 1 -- 172.21.15.27:0/573096229 learned_addr learned my addr 172.21.15.27:0/573096229 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb31d47a640 1 --2- 172.21.15.27:0/2219458373 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb3080a2d80 0x7fb3080a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb31cc79640 1 --2- 172.21.15.27:0/573096229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99451290 0x55ee99451710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb315ffb640 1 -- 172.21.15.27:0/2219458373 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fb3100cfa40 con 0x7fb318106630 2024-04-25T00:25:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb317fff640 1 --2- 172.21.15.27:0/573096229 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee9945c010 0x55ee9945e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb317fff640 1 -- 172.21.15.27:0/573096229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99451290 msgr2=0x55ee99451710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb317fff640 1 --2- 172.21.15.27:0/573096229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99451290 0x55ee99451710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb317fff640 1 -- 172.21.15.27:0/573096229 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee9945c010 msgr2=0x55ee9945e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb317fff640 1 --2- 172.21.15.27:0/573096229 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee9945c010 0x55ee9945e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb317fff640 1 -- 172.21.15.27:0/573096229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee9936efb0 con 0x55ee99452980 2024-04-25T00:25:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb317fff640 1 --2- 172.21.15.27:0/573096229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99452980 0x55ee99450cc0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fb304005940 tx=0x7fb304031480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af5a89280481c57 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb31d47a640 1 --2- 172.21.15.27:0/2219458373 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb3080a2d80 0x7fb3080a5240 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb31009bb80 tx=0x7fb31006dbb0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb302ffd640 1 -- 172.21.15.27:0/573096229 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb304035050 con 0x55ee99452980 2024-04-25T00:25:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb302ffd640 1 -- 172.21.15.27:0/573096229 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb304031890 con 0x55ee99452980 2024-04-25T00:25:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.154+0000 7fb302ffd640 1 -- 172.21.15.27:0/573096229 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3040046c0 con 0x55ee99452980 2024-04-25T00:25:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.155+0000 7fb321bea740 1 -- 172.21.15.27:0/573096229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99452980 msgr2=0x55ee99450cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.155+0000 7fb321bea740 1 --2- 172.21.15.27:0/573096229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99452980 0x55ee99450cc0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fb304005940 tx=0x7fb304031480 comp rx=0 tx=0).stop 2024-04-25T00:25:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.155+0000 7fb321bea740 1 -- 172.21.15.27:0/573096229 shutdown_connections 2024-04-25T00:25:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.155+0000 7fb321bea740 1 --2- 172.21.15.27:0/573096229 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee9945c010 0x55ee9945e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.155+0000 7fb321bea740 1 --2- 172.21.15.27:0/573096229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99451290 0x55ee99451710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.155+0000 7fb321bea740 1 --2- 172.21.15.27:0/573096229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99452980 0x55ee99450cc0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.155+0000 7fb321bea740 1 -- 172.21.15.27:0/573096229 >> 172.21.15.27:0/573096229 conn(0x55ee99419950 msgr2=0x55ee9944d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.155+0000 7fb321bea740 1 -- 172.21.15.27:0/573096229 shutdown_connections 2024-04-25T00:25:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.155+0000 7fb321bea740 1 -- 172.21.15.27:0/573096229 wait complete. 2024-04-25T00:25:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb321bea740 1 -- start start 2024-04-25T00:25:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb321bea740 1 Processor -- start 2024-04-25T00:25:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb321bea740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99451290 0x55ee99458ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb321bea740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99452980 0x55ee99459400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb321bea740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee9945c010 0x55ee99459940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb321bea740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ee9945ee40 con 0x55ee99451290 2024-04-25T00:25:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb321bea740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ee99370e90 con 0x55ee9945c010 2024-04-25T00:25:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb321bea740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ee9945efc0 con 0x55ee99452980 2024-04-25T00:25:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb321bea740 10 client.0 _ll_register_callbacks cb 0x55ee993ac970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:25:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb317fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99451290 0x55ee99458ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb317fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99451290 0x55ee99458ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53310/0 (socket says 172.21.15.27:53310) 2024-04-25T00:25:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb317fff640 1 -- 172.21.15.27:0/394769859 learned_addr learned my addr 172.21.15.27:0/394769859 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb31cc79640 1 --2- 172.21.15.27:0/394769859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99452980 0x55ee99459400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb317fff640 1 -- 172.21.15.27:0/394769859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99452980 msgr2=0x55ee99459400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb317fff640 1 --2- 172.21.15.27:0/394769859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99452980 0x55ee99459400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb317fff640 1 -- 172.21.15.27:0/394769859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee9945c010 msgr2=0x55ee99459940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb317fff640 1 --2- 172.21.15.27:0/394769859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee9945c010 0x55ee99459940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb317fff640 1 -- 172.21.15.27:0/394769859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee991917e0 con 0x55ee99451290 2024-04-25T00:25:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.156+0000 7fb31cc79640 1 --2- 172.21.15.27:0/394769859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99452980 0x55ee99459400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:25:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.157+0000 7fb317fff640 1 --2- 172.21.15.27:0/394769859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99451290 0x55ee99458ec0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fb304005e60 tx=0x7fb304004500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e0adfae7f9cbd79 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.157+0000 7fb2f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:25:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.157+0000 7fb2f7fff640 1 -- 172.21.15.27:0/394769859 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb304035040 con 0x55ee99451290 2024-04-25T00:25:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.157+0000 7fb2f7fff640 1 -- 172.21.15.27:0/394769859 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fb3040035f0 con 0x55ee99451290 2024-04-25T00:25:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.157+0000 7fb321bea740 1 -- 172.21.15.27:0/394769859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb304031890 con 0x55ee99451290 2024-04-25T00:25:32.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.157+0000 7fb2f7fff640 1 -- 172.21.15.27:0/394769859 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7fb304056050 con 0x55ee99451290 2024-04-25T00:25:32.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.157+0000 7fb321bea740 10 client.17241 fetch_fsmap learned FSMap version 100 2024-04-25T00:25:32.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.157+0000 7fb321bea740 1 -- 172.21.15.27:0/394769859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x55ee991f1f70 con 0x55ee99451290 2024-04-25T00:25:32.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.157+0000 7fb2f7fff640 1 -- 172.21.15.27:0/394769859 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb304031890 con 0x55ee99451290 2024-04-25T00:25:32.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.157+0000 7fb2f7fff640 1 -- 172.21.15.27:0/394769859 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 100) ==== 8366+0+0 (secure 0 0 0) 0x7fb30403aa90 con 0x55ee99451290 2024-04-25T00:25:32.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.157+0000 7fb321bea740 10 client.17241 fetch_fsmap finished waiting for FSMap version 100 2024-04-25T00:25:32.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.157+0000 7fb321bea740 10 client.17241 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:25:32.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.157+0000 7fb321bea740 20 client.17241 populate_metadata read hostname 'smithi027' 2024-04-25T00:25:32.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.158+0000 7fb321bea740 1 --2- 172.21.15.27:0/394769859 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55ee9946bf20 0x55ee9946e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.158+0000 7fb321bea740 4 client.17241 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:25:32.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.158+0000 7fb321bea740 1 -- 172.21.15.27:0/394769859 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee993895a0 con 0x55ee9946bf20 2024-04-25T00:25:32.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.158+0000 7fb31cc79640 1 --2- 172.21.15.27:0/394769859 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55ee9946bf20 0x55ee9946e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.158+0000 7fb31cc79640 1 --2- 172.21.15.27:0/394769859 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55ee9946bf20 0x55ee9946e310 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83ee5bf69bf80ba0 server_cookie=879b1a1611312300 in_seq=0 out_seq=0 2024-04-25T00:25:32.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.158+0000 7fb2f7fff640 10 client.17241 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:25:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.159+0000 7fb2f7fff640 1 -- 172.21.15.27:0/394769859 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ee993895a0 con 0x55ee9946bf20 2024-04-25T00:25:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.159+0000 7fb2f7fff640 10 client.17241 handle_command_reply: tid=0 2024-04-25T00:25:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.159+0000 7fb321bea740 1 client.17241 shutdown 2024-04-25T00:25:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.159+0000 7fb321bea740 1 -- 172.21.15.27:0/394769859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99451290 msgr2=0x55ee99458ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.159+0000 7fb321bea740 1 --2- 172.21.15.27:0/394769859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99451290 0x55ee99458ec0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fb304005e60 tx=0x7fb304004500 comp rx=0 tx=0).stop 2024-04-25T00:25:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.159+0000 7fb321bea740 1 -- 172.21.15.27:0/394769859 shutdown_connections 2024-04-25T00:25:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.159+0000 7fb321bea740 1 --2- 172.21.15.27:0/394769859 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55ee9946bf20 0x55ee9946e310 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.159+0000 7fb321bea740 1 --2- 172.21.15.27:0/394769859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee9945c010 0x55ee99459940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.159+0000 7fb321bea740 1 --2- 172.21.15.27:0/394769859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99452980 0x55ee99459400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.159+0000 7fb2f7fff640 0 client.17241 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:25:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.159+0000 7fb321bea740 1 --2- 172.21.15.27:0/394769859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99451290 0x55ee99458ec0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.159+0000 7fb321bea740 1 -- 172.21.15.27:0/394769859 >> 172.21.15.27:0/394769859 conn(0x55ee99419950 msgr2=0x55ee994535f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.160+0000 7fb321bea740 1 -- 172.21.15.27:0/394769859 shutdown_connections 2024-04-25T00:25:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.160+0000 7fb321bea740 1 -- 172.21.15.27:0/394769859 wait complete. 2024-04-25T00:25:32.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.160+0000 7fb321bea740 20 client.17241 trim_cache size 0 max 16384 2024-04-25T00:25:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.177+0000 7fb321bea740 1 Processor -- start 2024-04-25T00:25:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb321bea740 1 -- start start 2024-04-25T00:25:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb321bea740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99451290 0x55ee9944c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb321bea740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee99452980 0x55ee99471330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb321bea740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9945c010 0x55ee99471870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb321bea740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181aeaf0 con 0x55ee9945c010 2024-04-25T00:25:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb321bea740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181b0660 con 0x55ee99452980 2024-04-25T00:25:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb321bea740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181af1c0 con 0x55ee99451290 2024-04-25T00:25:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb317fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9945c010 0x55ee99471870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb317fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9945c010 0x55ee99471870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53322/0 (socket says 172.21.15.27:53322) 2024-04-25T00:25:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb317fff640 1 -- 172.21.15.27:0/1984507530 learned_addr learned my addr 172.21.15.27:0/1984507530 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb317fff640 1 --2- 172.21.15.27:0/1984507530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99451290 0x55ee9944c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb31cc79640 1 --2- 172.21.15.27:0/1984507530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee99452980 0x55ee99471330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb317fff640 1 -- 172.21.15.27:0/1984507530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99451290 msgr2=0x55ee9944c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb317fff640 1 --2- 172.21.15.27:0/1984507530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99451290 0x55ee9944c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb317fff640 1 -- 172.21.15.27:0/1984507530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee99452980 msgr2=0x55ee99471330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb317fff640 1 --2- 172.21.15.27:0/1984507530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee99452980 0x55ee99471330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb317fff640 1 -- 172.21.15.27:0/1984507530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee99089170 con 0x55ee9945c010 2024-04-25T00:25:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb31cc79640 1 --2- 172.21.15.27:0/1984507530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee99452980 0x55ee99471330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:25:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb317fff640 1 --2- 172.21.15.27:0/1984507530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9945c010 0x55ee99471870 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fb3040045f0 tx=0x7fb304005530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58d022de2e396d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb3027fc640 1 -- 172.21.15.27:0/1984507530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb304035070 con 0x55ee9945c010 2024-04-25T00:25:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb3027fc640 1 -- 172.21.15.27:0/1984507530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb30404d070 con 0x55ee9945c010 2024-04-25T00:25:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.178+0000 7fb3027fc640 1 -- 172.21.15.27:0/1984507530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb304048c80 con 0x55ee9945c010 2024-04-25T00:25:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.179+0000 7fb321bea740 1 -- 172.21.15.27:0/1984507530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9945c010 msgr2=0x55ee99471870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.179+0000 7fb321bea740 1 --2- 172.21.15.27:0/1984507530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9945c010 0x55ee99471870 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fb3040045f0 tx=0x7fb304005530 comp rx=0 tx=0).stop 2024-04-25T00:25:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.179+0000 7fb321bea740 1 -- 172.21.15.27:0/1984507530 shutdown_connections 2024-04-25T00:25:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.179+0000 7fb321bea740 1 --2- 172.21.15.27:0/1984507530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9945c010 0x55ee99471870 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.179+0000 7fb321bea740 1 --2- 172.21.15.27:0/1984507530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee99452980 0x55ee99471330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.179+0000 7fb321bea740 1 --2- 172.21.15.27:0/1984507530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99451290 0x55ee9944c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.179+0000 7fb321bea740 1 -- 172.21.15.27:0/1984507530 >> 172.21.15.27:0/1984507530 conn(0x55ee99419950 msgr2=0x55ee9944c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.179+0000 7fb321bea740 1 -- 172.21.15.27:0/1984507530 shutdown_connections 2024-04-25T00:25:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.179+0000 7fb321bea740 1 -- 172.21.15.27:0/1984507530 wait complete. 2024-04-25T00:25:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.179+0000 7fb321bea740 1 -- start start 2024-04-25T00:25:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.179+0000 7fb321bea740 1 Processor -- start 2024-04-25T00:25:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb321bea740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99451290 0x55ee9946f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb321bea740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99452980 0x55ee9946fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb321bea740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee9945c010 0x55ee994d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb321bea740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181aff90 con 0x55ee99452980 2024-04-25T00:25:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb321bea740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181af890 con 0x55ee9945c010 2024-04-25T00:25:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb321bea740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181acab0 con 0x55ee99451290 2024-04-25T00:25:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb321bea740 10 client.0 _ll_register_callbacks cb 0x7fb30c00c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:25:32.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb31cc79640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99452980 0x55ee9946fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb31cc79640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99452980 0x55ee9946fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53328/0 (socket says 172.21.15.27:53328) 2024-04-25T00:25:32.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb31cc79640 1 -- 172.21.15.27:0/103221705 learned_addr learned my addr 172.21.15.27:0/103221705 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:32.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb317fff640 1 --2- 172.21.15.27:0/103221705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99451290 0x55ee9946f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb317fff640 1 --2- 172.21.15.27:0/103221705 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee9945c010 0x55ee994d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb31cc79640 1 -- 172.21.15.27:0/103221705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99451290 msgr2=0x55ee9946f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:32.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb31cc79640 1 --2- 172.21.15.27:0/103221705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99451290 0x55ee9946f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb31cc79640 1 -- 172.21.15.27:0/103221705 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee9945c010 msgr2=0x55ee994d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:32.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb31cc79640 1 --2- 172.21.15.27:0/103221705 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee9945c010 0x55ee994d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb31cc79640 1 -- 172.21.15.27:0/103221705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee994d16f0 con 0x55ee99452980 2024-04-25T00:25:32.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb31cc79640 1 --2- 172.21.15.27:0/103221705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99452980 0x55ee9946fa60 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fb30c000c00 tx=0x7fb30c00cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e949f414474a8b36 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.180+0000 7fb2f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:25:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.181+0000 7fb2f7fff640 1 -- 172.21.15.27:0/103221705 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c003d20 con 0x55ee99452980 2024-04-25T00:25:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.181+0000 7fb2f7fff640 1 -- 172.21.15.27:0/103221705 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fb30c01a2c0 con 0x55ee99452980 2024-04-25T00:25:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.181+0000 7fb321bea740 1 -- 172.21.15.27:0/103221705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb30404d070 con 0x55ee99452980 2024-04-25T00:25:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.181+0000 7fb2f7fff640 1 -- 172.21.15.27:0/103221705 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7fb30c02b050 con 0x55ee99452980 2024-04-25T00:25:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.181+0000 7fb321bea740 10 client.17247 fetch_fsmap learned FSMap version 100 2024-04-25T00:25:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.181+0000 7fb321bea740 1 -- 172.21.15.27:0/103221705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x55ee994d2a10 con 0x55ee99452980 2024-04-25T00:25:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.181+0000 7fb2f7fff640 1 -- 172.21.15.27:0/103221705 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb30404d070 con 0x55ee99452980 2024-04-25T00:25:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.181+0000 7fb2f7fff640 1 -- 172.21.15.27:0/103221705 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 100) ==== 8366+0+0 (secure 0 0 0) 0x7fb30c011a40 con 0x55ee99452980 2024-04-25T00:25:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.181+0000 7fb321bea740 10 client.17247 fetch_fsmap finished waiting for FSMap version 100 2024-04-25T00:25:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.181+0000 7fb321bea740 10 client.17247 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:25:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.181+0000 7fb321bea740 20 client.17247 populate_metadata read hostname 'smithi027' 2024-04-25T00:25:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.181+0000 7fb321bea740 1 --2- 172.21.15.27:0/103221705 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55ee994d31c0 0x55ee994d55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.181+0000 7fb321bea740 4 client.17247 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T00:25:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.181+0000 7fb321bea740 1 -- 172.21.15.27:0/103221705 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ee993c2800 con 0x55ee994d31c0 2024-04-25T00:25:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.181+0000 7fb317fff640 1 --2- 172.21.15.27:0/103221705 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55ee994d31c0 0x55ee994d55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.182+0000 7fb2f7fff640 10 client.17247 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:25:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.182+0000 7fb317fff640 1 --2- 172.21.15.27:0/103221705 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55ee994d31c0 0x55ee994d55e0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9fd6de9f2845634 server_cookie=6b3ac0ee0fb7037 in_seq=0 out_seq=0 2024-04-25T00:25:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.183+0000 7fb2f7fff640 1 -- 172.21.15.27:0/103221705 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ee993c2800 con 0x55ee994d31c0 2024-04-25T00:25:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.183+0000 7fb2f7fff640 10 client.17247 handle_command_reply: tid=0 2024-04-25T00:25:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.183+0000 7fb321bea740 1 client.17247 shutdown 2024-04-25T00:25:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.183+0000 7fb321bea740 1 -- 172.21.15.27:0/103221705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99452980 msgr2=0x55ee9946fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.183+0000 7fb321bea740 1 --2- 172.21.15.27:0/103221705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99452980 0x55ee9946fa60 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fb30c000c00 tx=0x7fb30c00cb80 comp rx=0 tx=0).stop 2024-04-25T00:25:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.183+0000 7fb321bea740 1 -- 172.21.15.27:0/103221705 shutdown_connections 2024-04-25T00:25:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.183+0000 7fb321bea740 1 --2- 172.21.15.27:0/103221705 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55ee994d31c0 0x55ee994d55e0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.183+0000 7fb321bea740 1 --2- 172.21.15.27:0/103221705 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ee9945c010 0x55ee994d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.183+0000 7fb321bea740 1 --2- 172.21.15.27:0/103221705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee99452980 0x55ee9946fa60 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.183+0000 7fb321bea740 1 --2- 172.21.15.27:0/103221705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ee99451290 0x55ee9946f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.183+0000 7fb2f7fff640 0 client.17247 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:25:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.183+0000 7fb321bea740 1 -- 172.21.15.27:0/103221705 >> 172.21.15.27:0/103221705 conn(0x55ee99419950 msgr2=0x55ee9944c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.183+0000 7fb321bea740 1 -- 172.21.15.27:0/103221705 shutdown_connections 2024-04-25T00:25:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.183+0000 7fb321bea740 1 -- 172.21.15.27:0/103221705 wait complete. 2024-04-25T00:25:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.183+0000 7fb321bea740 20 client.17247 trim_cache size 0 max 16384 2024-04-25T00:25:32.190 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:25:32.190 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T00:25:32.190 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "007349c9-5c25-4100-bbfe-6f2104ab4d96", 2024-04-25T00:25:32.190 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T00:25:32.190 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:25:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.184+0000 7fb31eedd640 1 -- 172.21.15.27:0/2219458373 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb3080a2d80 msgr2=0x7fb3080a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.184+0000 7fb31eedd640 1 --2- 172.21.15.27:0/2219458373 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb3080a2d80 0x7fb3080a5240 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb31009bb80 tx=0x7fb31006dbb0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.184+0000 7fb31eedd640 1 -- 172.21.15.27:0/2219458373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318106630 msgr2=0x7fb31819c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.184+0000 7fb31eedd640 1 --2- 172.21.15.27:0/2219458373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318106630 0x7fb31819c830 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb31000eaa0 tx=0x7fb31000ef60 comp rx=0 tx=0).stop 2024-04-25T00:25:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.184+0000 7fb31eedd640 1 -- 172.21.15.27:0/2219458373 shutdown_connections 2024-04-25T00:25:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.184+0000 7fb31eedd640 1 --2- 172.21.15.27:0/2219458373 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb3080a2d80 0x7fb3080a5240 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.184+0000 7fb31eedd640 1 --2- 172.21.15.27:0/2219458373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318106630 0x7fb31819c830 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.184+0000 7fb31eedd640 1 --2- 172.21.15.27:0/2219458373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb318100fe0 0x7fb3181984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.184+0000 7fb31eedd640 1 --2- 172.21.15.27:0/2219458373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb318100620 0x7fb318197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.184+0000 7fb31eedd640 1 -- 172.21.15.27:0/2219458373 >> 172.21.15.27:0/2219458373 conn(0x7fb3180776a0 msgr2=0x7fb318069f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.184+0000 7fb31eedd640 1 -- 172.21.15.27:0/2219458373 shutdown_connections 2024-04-25T00:25:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.184+0000 7fb31eedd640 1 -- 172.21.15.27:0/2219458373 wait complete. 2024-04-25T00:25:32.196 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:25:32.196 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:25:32.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.300+0000 7fa5b4189640 1 -- 172.21.15.27:0/2392996446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ac06a780 msgr2=0x7fa5ac10e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.300+0000 7fa5b4189640 1 --2- 172.21.15.27:0/2392996446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ac06a780 0x7fa5ac10e5f0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4009cd0 tx=0x7fa5a402a8e0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.300+0000 7fa5b4189640 1 -- 172.21.15.27:0/2392996446 shutdown_connections 2024-04-25T00:25:32.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.300+0000 7fa5b4189640 1 --2- 172.21.15.27:0/2392996446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ac06a780 0x7fa5ac10e5f0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.300+0000 7fa5b4189640 1 --2- 172.21.15.27:0/2392996446 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5ac069dc0 0x7fa5ac06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.300+0000 7fa5b4189640 1 --2- 172.21.15.27:0/2392996446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5ac107730 0x7fa5ac107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.300+0000 7fa5b4189640 1 -- 172.21.15.27:0/2392996446 >> 172.21.15.27:0/2392996446 conn(0x7fa5ac0776a0 msgr2=0x7fa5ac077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:32.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.300+0000 7fa5b4189640 1 -- 172.21.15.27:0/2392996446 shutdown_connections 2024-04-25T00:25:32.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.300+0000 7fa5b4189640 1 -- 172.21.15.27:0/2392996446 wait complete. 2024-04-25T00:25:32.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.301+0000 7fa5b4189640 1 Processor -- start 2024-04-25T00:25:32.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.301+0000 7fa5b4189640 1 -- start start 2024-04-25T00:25:32.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.301+0000 7fa5b4189640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ac069dc0 0x7fa5ac197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.301+0000 7fa5b4189640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5ac06a780 0x7fa5ac1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.301+0000 7fa5b4189640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5ac107730 0x7fa5ac19c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.301+0000 7fa5b4189640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ac111b00 con 0x7fa5ac069dc0 2024-04-25T00:25:32.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.301+0000 7fa5b4189640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ac111980 con 0x7fa5ac06a780 2024-04-25T00:25:32.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.301+0000 7fa5b4189640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ac111c80 con 0x7fa5ac107730 2024-04-25T00:25:32.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.301+0000 7fa5b1f25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ac069dc0 0x7fa5ac197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.301+0000 7fa5b1f25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ac069dc0 0x7fa5ac197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53354/0 (socket says 172.21.15.27:53354) 2024-04-25T00:25:32.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.301+0000 7fa5b1f25640 1 -- 172.21.15.27:0/159466708 learned_addr learned my addr 172.21.15.27:0/159466708 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:32.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.301+0000 7fa5b2726640 1 --2- 172.21.15.27:0/159466708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5ac107730 0x7fa5ac19c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.302+0000 7fa5b1f25640 1 -- 172.21.15.27:0/159466708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5ac107730 msgr2=0x7fa5ac19c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.302+0000 7fa5b1f25640 1 --2- 172.21.15.27:0/159466708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5ac107730 0x7fa5ac19c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.302+0000 7fa5b1f25640 1 -- 172.21.15.27:0/159466708 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5ac06a780 msgr2=0x7fa5ac1984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.302+0000 7fa5b1f25640 1 --2- 172.21.15.27:0/159466708 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5ac06a780 0x7fa5ac1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.302+0000 7fa5b1f25640 1 -- 172.21.15.27:0/159466708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5ac19cf00 con 0x7fa5ac069dc0 2024-04-25T00:25:32.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.302+0000 7fa5b1f25640 1 --2- 172.21.15.27:0/159466708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ac069dc0 0x7fa5ac197f60 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fa5a800c920 tx=0x7fa5a800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:32.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.302+0000 7fa5a2ffd640 1 -- 172.21.15.27:0/159466708 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8007c70 con 0x7fa5ac069dc0 2024-04-25T00:25:32.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.302+0000 7fa5a2ffd640 1 -- 172.21.15.27:0/159466708 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5a8007e10 con 0x7fa5ac069dc0 2024-04-25T00:25:32.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.302+0000 7fa5b4189640 1 -- 172.21.15.27:0/159466708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5ac19d1f0 con 0x7fa5ac069dc0 2024-04-25T00:25:32.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.302+0000 7fa5a2ffd640 1 -- 172.21.15.27:0/159466708 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8018890 con 0x7fa5ac069dc0 2024-04-25T00:25:32.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.302+0000 7fa5b4189640 1 -- 172.21.15.27:0/159466708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5ac1a4a80 con 0x7fa5ac069dc0 2024-04-25T00:25:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.305+0000 7fa5a2ffd640 1 -- 172.21.15.27:0/159466708 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa5a8018a70 con 0x7fa5ac069dc0 2024-04-25T00:25:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.305+0000 7fa5b6f75740 1 Processor -- start 2024-04-25T00:25:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.305+0000 7fa5b6f75740 1 -- start start 2024-04-25T00:25:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.305+0000 7fa5b6f75740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3630 0x55d8be2a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.305+0000 7fa5b6f75740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2a3f50 0x55d8be2ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.305+0000 7fa5b6f75740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.305+0000 7fa5b6f75740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d8be2086b0 con 0x55d8be2ae620 2024-04-25T00:25:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.305+0000 7fa5b6f75740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d8be1b0880 con 0x55d8be2a3630 2024-04-25T00:25:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.305+0000 7fa5b6f75740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d8be204670 con 0x55d8be2a3f50 2024-04-25T00:25:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.305+0000 7fa5b1724640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa5b1724640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2b0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53358/0 (socket says 172.21.15.27:53358) 2024-04-25T00:25:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa5b1724640 1 -- 172.21.15.27:0/2170139147 learned_addr learned my addr 172.21.15.27:0/2170139147 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa5b1f25640 1 --2- 172.21.15.27:0/2170139147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2a3f50 0x55d8be2ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa5a2ffd640 1 --2- 172.21.15.27:0/159466708 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa5980a2d80 0x7fa5980a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa5b2726640 1 --2- 172.21.15.27:0/2170139147 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3630 0x55d8be2a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa5b2726640 1 --2- 172.21.15.27:0/159466708 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa5980a2d80 0x7fa5980a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa5a2ffd640 1 -- 172.21.15.27:0/159466708 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fa5a80cf8c0 con 0x7fa5ac069dc0 2024-04-25T00:25:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa5b1f25640 1 -- 172.21.15.27:0/2170139147 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3630 msgr2=0x55d8be2a3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa5b1f25640 1 --2- 172.21.15.27:0/2170139147 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3630 0x55d8be2a3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa5b1f25640 1 -- 172.21.15.27:0/2170139147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 msgr2=0x55d8be2b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa5b1f25640 1 --2- 172.21.15.27:0/2170139147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa5b1f25640 1 -- 172.21.15.27:0/2170139147 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8bdfe67d0 con 0x55d8be2a3f50 2024-04-25T00:25:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa5b1724640 1 --2- 172.21.15.27:0/2170139147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:25:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa5b1f25640 1 --2- 172.21.15.27:0/2170139147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2a3f50 0x55d8be2ae0e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa5a80ccca0 tx=0x7fa5a8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abefabdf583dcaa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:32.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa5b2726640 1 --2- 172.21.15.27:0/159466708 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa5980a2d80 0x7fa5980a5240 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4009430 tx=0x7fa5a403f040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:32.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa597fff640 1 -- 172.21.15.27:0/2170139147 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a800de20 con 0x55d8be2a3f50 2024-04-25T00:25:32.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa597fff640 1 -- 172.21.15.27:0/2170139147 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5a809ba70 con 0x55d8be2a3f50 2024-04-25T00:25:32.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.306+0000 7fa597fff640 1 -- 172.21.15.27:0/2170139147 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a80980e0 con 0x55d8be2a3f50 2024-04-25T00:25:32.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.307+0000 7fa5b6f75740 1 -- 172.21.15.27:0/2170139147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2a3f50 msgr2=0x55d8be2ae0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.307+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/2170139147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2a3f50 0x55d8be2ae0e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa5a80ccca0 tx=0x7fa5a8000bc0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.307+0000 7fa5b6f75740 1 -- 172.21.15.27:0/2170139147 shutdown_connections 2024-04-25T00:25:32.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.307+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/2170139147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.307+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/2170139147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2a3f50 0x55d8be2ae0e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.307+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/2170139147 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3630 0x55d8be2a3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.307+0000 7fa5b6f75740 1 -- 172.21.15.27:0/2170139147 >> 172.21.15.27:0/2170139147 conn(0x55d8be29f950 msgr2=0x55d8be29fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:32.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.307+0000 7fa5b6f75740 1 -- 172.21.15.27:0/2170139147 shutdown_connections 2024-04-25T00:25:32.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.307+0000 7fa5b6f75740 1 -- 172.21.15.27:0/2170139147 wait complete. 2024-04-25T00:25:32.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.307+0000 7fa5b6f75740 1 -- start start 2024-04-25T00:25:32.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b6f75740 1 Processor -- start 2024-04-25T00:25:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b6f75740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2a3630 0x55d8be2ab630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b6f75740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 0x55d8be2abb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b6f75740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2ae620 0x55d8be2ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b6f75740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d8be2b13b0 con 0x55d8be2a3630 2024-04-25T00:25:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b6f75740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d8be1c3f50 con 0x55d8be2a3f50 2024-04-25T00:25:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b6f75740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d8be2b1530 con 0x55d8be2ae620 2024-04-25T00:25:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b6f75740 10 client.0 _ll_register_callbacks cb 0x55d8be1ff000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:25:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b1f25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2a3630 0x55d8be2ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b1f25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2a3630 0x55d8be2ab630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53372/0 (socket says 172.21.15.27:53372) 2024-04-25T00:25:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b1f25640 1 -- 172.21.15.27:0/3862281701 learned_addr learned my addr 172.21.15.27:0/3862281701 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b2726640 1 --2- 172.21.15.27:0/3862281701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2ae620 0x55d8be2ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b1f25640 1 -- 172.21.15.27:0/3862281701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2ae620 msgr2=0x55d8be2ac0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b1724640 1 --2- 172.21.15.27:0/3862281701 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 0x55d8be2abb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b1f25640 1 --2- 172.21.15.27:0/3862281701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2ae620 0x55d8be2ac0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b1f25640 1 -- 172.21.15.27:0/3862281701 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 msgr2=0x55d8be2abb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b1f25640 1 --2- 172.21.15.27:0/3862281701 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 0x55d8be2abb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b1f25640 1 -- 172.21.15.27:0/3862281701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8bdf1c3f0 con 0x55d8be2a3630 2024-04-25T00:25:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.308+0000 7fa5b1724640 1 --2- 172.21.15.27:0/3862281701 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 0x55d8be2abb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:25:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.309+0000 7fa5b1f25640 1 --2- 172.21.15.27:0/3862281701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2a3630 0x55d8be2ab630 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fa5a809ba00 tx=0x7fa5a809fe80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a98a76f8dd03aba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.309+0000 7fa5957fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:25:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.309+0000 7fa5957fa640 1 -- 172.21.15.27:0/3862281701 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a80b9060 con 0x55d8be2a3630 2024-04-25T00:25:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.309+0000 7fa5957fa640 1 -- 172.21.15.27:0/3862281701 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fa5a80980e0 con 0x55d8be2a3630 2024-04-25T00:25:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.309+0000 7fa5b6f75740 1 -- 172.21.15.27:0/3862281701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5a809ba70 con 0x55d8be2a3630 2024-04-25T00:25:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.309+0000 7fa5957fa640 1 -- 172.21.15.27:0/3862281701 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7fa5a800de20 con 0x55d8be2a3630 2024-04-25T00:25:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.309+0000 7fa5b6f75740 10 client.17259 fetch_fsmap learned FSMap version 100 2024-04-25T00:25:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.309+0000 7fa5b6f75740 1 -- 172.21.15.27:0/3862281701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x55d8bdedc0f0 con 0x55d8be2a3630 2024-04-25T00:25:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.309+0000 7fa5957fa640 1 -- 172.21.15.27:0/3862281701 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5a809ba70 con 0x55d8be2a3630 2024-04-25T00:25:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.309+0000 7fa5957fa640 1 -- 172.21.15.27:0/3862281701 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 100) ==== 8366+0+0 (secure 0 0 0) 0x7fa5a80a42e0 con 0x55d8be2a3630 2024-04-25T00:25:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.309+0000 7fa5b6f75740 10 client.17259 fetch_fsmap finished waiting for FSMap version 100 2024-04-25T00:25:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.309+0000 7fa5b6f75740 10 client.17259 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:25:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.309+0000 7fa5b6f75740 20 client.17259 populate_metadata read hostname 'smithi027' 2024-04-25T00:25:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.309+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/3862281701 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55d8be2be7b0 0x55d8be2c0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.309+0000 7fa5b6f75740 4 client.17259 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:25:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.309+0000 7fa5b6f75740 1 -- 172.21.15.27:0/3862281701 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8be215bf0 con 0x55d8be2be7b0 2024-04-25T00:25:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.310+0000 7fa5b1724640 1 --2- 172.21.15.27:0/3862281701 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55d8be2be7b0 0x55d8be2c0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.310+0000 7fa5957fa640 10 client.17259 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:25:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.310+0000 7fa5b1724640 1 --2- 172.21.15.27:0/3862281701 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55d8be2be7b0 0x55d8be2c0ba0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff72845870d7633e server_cookie=9b04bae83b07788 in_seq=0 out_seq=0 2024-04-25T00:25:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.311+0000 7fa5957fa640 1 -- 172.21.15.27:0/3862281701 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d8be215bf0 con 0x55d8be2be7b0 2024-04-25T00:25:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.311+0000 7fa5957fa640 10 client.17259 handle_command_reply: tid=0 2024-04-25T00:25:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.311+0000 7fa5b6f75740 1 client.17259 shutdown 2024-04-25T00:25:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.311+0000 7fa5b6f75740 1 -- 172.21.15.27:0/3862281701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2a3630 msgr2=0x55d8be2ab630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.311+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/3862281701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2a3630 0x55d8be2ab630 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fa5a809ba00 tx=0x7fa5a809fe80 comp rx=0 tx=0).stop 2024-04-25T00:25:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.311+0000 7fa5b6f75740 1 -- 172.21.15.27:0/3862281701 shutdown_connections 2024-04-25T00:25:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.311+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/3862281701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2ae620 0x55d8be2ac0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.311+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/3862281701 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55d8be2be7b0 0x55d8be2c0ba0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.311+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/3862281701 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 0x55d8be2abb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.311+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/3862281701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2a3630 0x55d8be2ab630 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.311+0000 7fa5957fa640 0 client.17259 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:25:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.311+0000 7fa5b6f75740 1 -- 172.21.15.27:0/3862281701 >> 172.21.15.27:0/3862281701 conn(0x55d8be29f950 msgr2=0x55d8be2a5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.311+0000 7fa5b6f75740 1 -- 172.21.15.27:0/3862281701 shutdown_connections 2024-04-25T00:25:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.311+0000 7fa5b6f75740 1 -- 172.21.15.27:0/3862281701 wait complete. 2024-04-25T00:25:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.311+0000 7fa5b6f75740 20 client.17259 trim_cache size 0 max 16384 2024-04-25T00:25:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.328+0000 7fa5b6f75740 1 Processor -- start 2024-04-25T00:25:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.328+0000 7fa5b6f75740 1 -- start start 2024-04-25T00:25:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b6f75740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2a3630 0x55d8be2add60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b6f75740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 0x55d8be2a5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b6f75740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2c3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b6f75740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ac1aeb60 con 0x55d8be2ae620 2024-04-25T00:25:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b6f75740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ac1b06d0 con 0x55d8be2a3f50 2024-04-25T00:25:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b6f75740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ac1af230 con 0x55d8be2a3630 2024-04-25T00:25:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b2726640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2c3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b2726640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2c3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53376/0 (socket says 172.21.15.27:53376) 2024-04-25T00:25:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b2726640 1 -- 172.21.15.27:0/1675955185 learned_addr learned my addr 172.21.15.27:0/1675955185 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b1724640 1 --2- 172.21.15.27:0/1675955185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 0x55d8be2a5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b1f25640 1 --2- 172.21.15.27:0/1675955185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2a3630 0x55d8be2add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b2726640 1 -- 172.21.15.27:0/1675955185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2a3630 msgr2=0x55d8be2add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b2726640 1 --2- 172.21.15.27:0/1675955185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2a3630 0x55d8be2add60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b2726640 1 -- 172.21.15.27:0/1675955185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 msgr2=0x55d8be2a5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b2726640 1 --2- 172.21.15.27:0/1675955185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 0x55d8be2a5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b2726640 1 -- 172.21.15.27:0/1675955185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8be2a4450 con 0x55d8be2ae620 2024-04-25T00:25:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b1724640 1 --2- 172.21.15.27:0/1675955185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 0x55d8be2a5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:25:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5b2726640 1 --2- 172.21.15.27:0/1675955185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2c3bc0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4035000 tx=0x7fa5a40093c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d256cd4b383eb20 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5977fe640 1 -- 172.21.15.27:0/1675955185 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4003b70 con 0x55d8be2ae620 2024-04-25T00:25:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5977fe640 1 -- 172.21.15.27:0/1675955185 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5a4036040 con 0x55d8be2ae620 2024-04-25T00:25:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.329+0000 7fa5977fe640 1 -- 172.21.15.27:0/1675955185 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4030800 con 0x55d8be2ae620 2024-04-25T00:25:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.330+0000 7fa5b6f75740 1 -- 172.21.15.27:0/1675955185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 msgr2=0x55d8be2c3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.330+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/1675955185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2c3bc0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4035000 tx=0x7fa5a40093c0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.330+0000 7fa5b6f75740 1 -- 172.21.15.27:0/1675955185 shutdown_connections 2024-04-25T00:25:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.330+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/1675955185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2c3bc0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.330+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/1675955185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 0x55d8be2a5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.330+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/1675955185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2a3630 0x55d8be2add60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.330+0000 7fa5b6f75740 1 -- 172.21.15.27:0/1675955185 >> 172.21.15.27:0/1675955185 conn(0x55d8be29f950 msgr2=0x55d8be2ba010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.330+0000 7fa5b6f75740 1 -- 172.21.15.27:0/1675955185 shutdown_connections 2024-04-25T00:25:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.330+0000 7fa5b6f75740 1 -- 172.21.15.27:0/1675955185 wait complete. 2024-04-25T00:25:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.330+0000 7fa5b6f75740 1 -- start start 2024-04-25T00:25:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.330+0000 7fa5b6f75740 1 Processor -- start 2024-04-25T00:25:32.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b6f75740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2a3630 0x55d8be2c1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b6f75740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 0x55d8be2c1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b6f75740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2c2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b6f75740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ac1b0000 con 0x55d8be2ae620 2024-04-25T00:25:32.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b6f75740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ac1af900 con 0x55d8be2a3f50 2024-04-25T00:25:32.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b6f75740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ac1acb20 con 0x55d8be2a3630 2024-04-25T00:25:32.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b6f75740 10 client.0 _ll_register_callbacks cb 0x7fa59c009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:25:32.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b2726640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2c2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b2726640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2c2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53388/0 (socket says 172.21.15.27:53388) 2024-04-25T00:25:32.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b2726640 1 -- 172.21.15.27:0/455871450 learned_addr learned my addr 172.21.15.27:0/455871450 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:32.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b1f25640 1 --2- 172.21.15.27:0/455871450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2a3630 0x55d8be2c1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b1724640 1 --2- 172.21.15.27:0/455871450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 0x55d8be2c1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b2726640 1 -- 172.21.15.27:0/455871450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2a3630 msgr2=0x55d8be2c1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b2726640 1 --2- 172.21.15.27:0/455871450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2a3630 0x55d8be2c1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b2726640 1 -- 172.21.15.27:0/455871450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 msgr2=0x55d8be2c1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b2726640 1 --2- 172.21.15.27:0/455871450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 0x55d8be2c1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b2726640 1 -- 172.21.15.27:0/455871450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8be323990 con 0x55d8be2ae620 2024-04-25T00:25:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.331+0000 7fa5b1724640 1 --2- 172.21.15.27:0/455871450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 0x55d8be2c1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:25:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.332+0000 7fa5b2726640 1 --2- 172.21.15.27:0/455871450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2c2000 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4004810 tx=0x7fa5a4032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c60feeb167cf3cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.332+0000 7fa5957fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:25:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.332+0000 7fa5957fa640 1 -- 172.21.15.27:0/455871450 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4062060 con 0x55d8be2ae620 2024-04-25T00:25:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.332+0000 7fa5b6f75740 1 -- 172.21.15.27:0/455871450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5a4036040 con 0x55d8be2ae620 2024-04-25T00:25:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.332+0000 7fa5957fa640 1 -- 172.21.15.27:0/455871450 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fa5a4004310 con 0x55d8be2ae620 2024-04-25T00:25:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.332+0000 7fa5957fa640 1 -- 172.21.15.27:0/455871450 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7fa5a4069050 con 0x55d8be2ae620 2024-04-25T00:25:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.332+0000 7fa5b6f75740 10 client.17265 fetch_fsmap learned FSMap version 100 2024-04-25T00:25:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.332+0000 7fa5b6f75740 1 -- 172.21.15.27:0/455871450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=4+}) -- 0x55d8be324c00 con 0x55d8be2ae620 2024-04-25T00:25:32.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.332+0000 7fa5957fa640 1 -- 172.21.15.27:0/455871450 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5a4036040 con 0x55d8be2ae620 2024-04-25T00:25:32.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.332+0000 7fa5957fa640 1 -- 172.21.15.27:0/455871450 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 100) ==== 8366+0+0 (secure 0 0 0) 0x7fa5a4030070 con 0x55d8be2ae620 2024-04-25T00:25:32.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.332+0000 7fa5b6f75740 10 client.17265 fetch_fsmap finished waiting for FSMap version 100 2024-04-25T00:25:32.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.332+0000 7fa5b6f75740 10 client.17265 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:25:32.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.332+0000 7fa5b6f75740 20 client.17265 populate_metadata read hostname 'smithi027' 2024-04-25T00:25:32.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.332+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/455871450 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55d8be325330 0x55d8be327750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.332+0000 7fa5b6f75740 4 client.17265 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:25:32.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.332+0000 7fa5b6f75740 1 -- 172.21.15.27:0/455871450 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d8be20ff40 con 0x55d8be325330 2024-04-25T00:25:32.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.333+0000 7fa5b1f25640 1 --2- 172.21.15.27:0/455871450 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55d8be325330 0x55d8be327750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.333+0000 7fa5957fa640 10 client.17265 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:25:32.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.333+0000 7fa5b1f25640 1 --2- 172.21.15.27:0/455871450 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55d8be325330 0x55d8be327750 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=698d1447b42a27ae server_cookie=2ab4db3a6476fd84 in_seq=0 out_seq=0 2024-04-25T00:25:32.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.333+0000 7fa5957fa640 1 -- 172.21.15.27:0/455871450 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d8be20ff40 con 0x55d8be325330 2024-04-25T00:25:32.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.333+0000 7fa5957fa640 10 client.17265 handle_command_reply: tid=0 2024-04-25T00:25:32.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.333+0000 7fa5b6f75740 1 client.17265 shutdown 2024-04-25T00:25:32.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.333+0000 7fa5b6f75740 1 -- 172.21.15.27:0/455871450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 msgr2=0x55d8be2c2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.333+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/455871450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2c2000 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4004810 tx=0x7fa5a4032f40 comp rx=0 tx=0).stop 2024-04-25T00:25:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.333+0000 7fa5b6f75740 1 -- 172.21.15.27:0/455871450 shutdown_connections 2024-04-25T00:25:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.333+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/455871450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8be2ae620 0x55d8be2c2000 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.333+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/455871450 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x55d8be325330 0x55d8be327750 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.333+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/455871450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d8be2a3f50 0x55d8be2c1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.333+0000 7fa5957fa640 0 client.17265 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:25:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.333+0000 7fa5b6f75740 1 --2- 172.21.15.27:0/455871450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8be2a3630 0x55d8be2c1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.333+0000 7fa5b6f75740 1 -- 172.21.15.27:0/455871450 >> 172.21.15.27:0/455871450 conn(0x55d8be29f950 msgr2=0x55d8be29e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.333+0000 7fa5b6f75740 1 -- 172.21.15.27:0/455871450 shutdown_connections 2024-04-25T00:25:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.333+0000 7fa5b6f75740 1 -- 172.21.15.27:0/455871450 wait complete. 2024-04-25T00:25:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.334+0000 7fa5b6f75740 20 client.17265 trim_cache size 0 max 16384 2024-04-25T00:25:32.340 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:25:32.340 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T00:25:32.341 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T00:25:32.341 INFO:teuthology.orchestra.run.smithi027.stdout: "007349c9-5c25-4100-bbfe-6f2104ab4d96": { 2024-04-25T00:25:32.341 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T00:25:32.341 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "007349c9-5c25-4100-bbfe-6f2104ab4d96", 2024-04-25T00:25:32.341 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T00:25:32.341 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:25:32.341 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:25:32.341 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:25:32.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.334+0000 7fa5b4189640 1 -- 172.21.15.27:0/159466708 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa5980a2d80 msgr2=0x7fa5980a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.334+0000 7fa5b4189640 1 --2- 172.21.15.27:0/159466708 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa5980a2d80 0x7fa5980a5240 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4009430 tx=0x7fa5a403f040 comp rx=0 tx=0).stop 2024-04-25T00:25:32.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.334+0000 7fa5b4189640 1 -- 172.21.15.27:0/159466708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ac069dc0 msgr2=0x7fa5ac197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.334+0000 7fa5b4189640 1 --2- 172.21.15.27:0/159466708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ac069dc0 0x7fa5ac197f60 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fa5a800c920 tx=0x7fa5a800cde0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.334+0000 7fa5b4189640 1 -- 172.21.15.27:0/159466708 shutdown_connections 2024-04-25T00:25:32.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.334+0000 7fa5b4189640 1 --2- 172.21.15.27:0/159466708 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa5980a2d80 0x7fa5980a5240 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.334+0000 7fa5b4189640 1 --2- 172.21.15.27:0/159466708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5ac107730 0x7fa5ac19c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.334+0000 7fa5b4189640 1 --2- 172.21.15.27:0/159466708 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5ac06a780 0x7fa5ac1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.334+0000 7fa5b4189640 1 --2- 172.21.15.27:0/159466708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ac069dc0 0x7fa5ac197f60 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.334+0000 7fa5b4189640 1 -- 172.21.15.27:0/159466708 >> 172.21.15.27:0/159466708 conn(0x7fa5ac0776a0 msgr2=0x7fa5ac0fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:32.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.334+0000 7fa5b4189640 1 -- 172.21.15.27:0/159466708 shutdown_connections 2024-04-25T00:25:32.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:32.334+0000 7fa5b4189640 1 -- 172.21.15.27:0/159466708 wait complete. 2024-04-25T00:25:32.346 INFO:tasks.cephfs.filesystem:scrub status for tag:007349c9-5c25-4100-bbfe-6f2104ab4d96 - {'path': '/', 'tag': '007349c9-5c25-4100-bbfe-6f2104ab4d96', 'options': 'recursive,force'} 2024-04-25T00:25:33.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:32 smithi027 ceph-mon[29208]: pgmap v296: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T00:25:33.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:32 smithi027 ceph-mon[29208]: scrub queued for path: / 2024-04-25T00:25:33.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:32 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:25:33.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:32 smithi027 ceph-mon[29208]: scrub summary: active paths [/] 2024-04-25T00:25:33.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:32 smithi027 ceph-mon[29208]: scrub summary: idle+waiting paths [/] 2024-04-25T00:25:33.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:32 smithi139 ceph-mon[32833]: pgmap v296: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T00:25:33.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:32 smithi139 ceph-mon[32833]: scrub queued for path: / 2024-04-25T00:25:33.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:32 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:25:33.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:32 smithi139 ceph-mon[32833]: scrub summary: active paths [/] 2024-04-25T00:25:33.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:32 smithi139 ceph-mon[32833]: scrub summary: idle+waiting paths [/] 2024-04-25T00:25:33.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:32 smithi153 ceph-mon[32896]: pgmap v296: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T00:25:33.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:32 smithi153 ceph-mon[32896]: scrub queued for path: / 2024-04-25T00:25:33.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:32 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:25:33.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:32 smithi153 ceph-mon[32896]: scrub summary: active paths [/] 2024-04-25T00:25:33.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:32 smithi153 ceph-mon[32896]: scrub summary: idle+waiting paths [/] 2024-04-25T00:25:34.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:33 smithi027 ceph-mon[29208]: scrub summary: idle 2024-04-25T00:25:34.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:33 smithi139 ceph-mon[32833]: scrub summary: idle 2024-04-25T00:25:34.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:33 smithi153 ceph-mon[32896]: scrub summary: idle 2024-04-25T00:25:35.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:34 smithi027 ceph-mon[29208]: pgmap v297: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:25:35.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:34 smithi139 ceph-mon[32833]: pgmap v297: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:25:35.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:34 smithi153 ceph-mon[32896]: pgmap v297: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:25:36.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:35 smithi027 ceph-mon[29208]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:25:36.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:35 smithi139 ceph-mon[32833]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:25:36.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:35 smithi153 ceph-mon[32896]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T00:25:37.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:36 smithi027 ceph-mon[29208]: pgmap v298: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T00:25:37.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:36 smithi139 ceph-mon[32833]: pgmap v298: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T00:25:37.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:36 smithi153 ceph-mon[32896]: pgmap v298: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T00:25:39.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:38 smithi027 ceph-mon[29208]: pgmap v299: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-04-25T00:25:39.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:38 smithi139 ceph-mon[32833]: pgmap v299: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-04-25T00:25:39.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:38 smithi153 ceph-mon[32896]: pgmap v299: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-04-25T00:25:41.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:40 smithi027 ceph-mon[29208]: pgmap v300: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-04-25T00:25:41.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:40 smithi139 ceph-mon[32833]: pgmap v300: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-04-25T00:25:41.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:40 smithi153 ceph-mon[32896]: pgmap v300: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-04-25T00:25:43.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:42 smithi027 ceph-mon[29208]: pgmap v301: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T00:25:43.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:42 smithi139 ceph-mon[32833]: pgmap v301: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T00:25:43.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:42 smithi153 ceph-mon[32896]: pgmap v301: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T00:25:45.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:44 smithi027 ceph-mon[29208]: pgmap v302: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-25T00:25:45.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:44 smithi139 ceph-mon[32833]: pgmap v302: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-25T00:25:45.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:44 smithi153 ceph-mon[32896]: pgmap v302: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-25T00:25:47.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:46 smithi027 ceph-mon[29208]: pgmap v303: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T00:25:47.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:46 smithi139 ceph-mon[32833]: pgmap v303: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T00:25:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:46 smithi153 ceph-mon[32896]: pgmap v303: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T00:25:49.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:48 smithi027 ceph-mon[29208]: pgmap v304: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 9 op/s 2024-04-25T00:25:49.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:48 smithi139 ceph-mon[32833]: pgmap v304: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 9 op/s 2024-04-25T00:25:49.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:48 smithi153 ceph-mon[32896]: pgmap v304: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 9 op/s 2024-04-25T00:25:51.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:50 smithi027 ceph-mon[29208]: pgmap v305: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:25:51.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:50 smithi139 ceph-mon[32833]: pgmap v305: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:25:51.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:50 smithi153 ceph-mon[32896]: pgmap v305: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:25:53.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:52 smithi027 ceph-mon[29208]: pgmap v306: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T00:25:53.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:52 smithi139 ceph-mon[32833]: pgmap v306: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T00:25:53.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:52 smithi153 ceph-mon[32896]: pgmap v306: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T00:25:54.902 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T00:25:55.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.017+0000 7f8eae529640 1 -- 172.21.15.27:0/1104935863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea8106630 msgr2=0x7f8ea8106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:55.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.017+0000 7f8eae529640 1 --2- 172.21.15.27:0/1104935863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea8106630 0x7f8ea8106a10 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009930 tx=0x7f8e9802a420 comp rx=0 tx=0).stop 2024-04-25T00:25:55.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.017+0000 7f8eae529640 1 -- 172.21.15.27:0/1104935863 shutdown_connections 2024-04-25T00:25:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.017+0000 7f8eae529640 1 --2- 172.21.15.27:0/1104935863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ea8101030 0x7f8ea810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.017+0000 7f8eae529640 1 --2- 172.21.15.27:0/1104935863 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ea8100670 0x7f8ea8100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.017+0000 7f8eae529640 1 --2- 172.21.15.27:0/1104935863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea8106630 0x7f8ea8106a10 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.017+0000 7f8eae529640 1 -- 172.21.15.27:0/1104935863 >> 172.21.15.27:0/1104935863 conn(0x7f8ea80777c0 msgr2=0x7f8ea8077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.017+0000 7f8eae529640 1 -- 172.21.15.27:0/1104935863 shutdown_connections 2024-04-25T00:25:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.017+0000 7f8eae529640 1 -- 172.21.15.27:0/1104935863 wait complete. 2024-04-25T00:25:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.017+0000 7f8eae529640 1 Processor -- start 2024-04-25T00:25:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.017+0000 7f8eae529640 1 -- start start 2024-04-25T00:25:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.018+0000 7f8eae529640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea8100670 0x7f8ea8198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.018+0000 7f8eae529640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ea8101030 0x7f8ea8199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.018+0000 7f8eae529640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ea8106630 0x7f8ea81a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.018+0000 7f8eae529640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea8110a00 con 0x7f8ea8100670 2024-04-25T00:25:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.018+0000 7f8eae529640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea8110880 con 0x7f8ea8106630 2024-04-25T00:25:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.018+0000 7f8eae529640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea8110b80 con 0x7f8ea8101030 2024-04-25T00:25:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.018+0000 7f8ea7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea8100670 0x7f8ea8198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.018+0000 7f8ea7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea8100670 0x7f8ea8198de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49050/0 (socket says 172.21.15.27:49050) 2024-04-25T00:25:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.018+0000 7f8ea7fff640 1 -- 172.21.15.27:0/977756935 learned_addr learned my addr 172.21.15.27:0/977756935 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.018+0000 7f8eacac6640 1 --2- 172.21.15.27:0/977756935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ea8106630 0x7f8ea81a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.018+0000 7f8ea77fe640 1 --2- 172.21.15.27:0/977756935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ea8101030 0x7f8ea8199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.018+0000 7f8ea7fff640 1 -- 172.21.15.27:0/977756935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ea8101030 msgr2=0x7f8ea8199320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.018+0000 7f8ea7fff640 1 --2- 172.21.15.27:0/977756935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ea8101030 0x7f8ea8199320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.018+0000 7f8ea7fff640 1 -- 172.21.15.27:0/977756935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ea8106630 msgr2=0x7f8ea81a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.018+0000 7f8ea7fff640 1 --2- 172.21.15.27:0/977756935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ea8106630 0x7f8ea81a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.018+0000 7f8ea7fff640 1 -- 172.21.15.27:0/977756935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ea81a43a0 con 0x7f8ea8100670 2024-04-25T00:25:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.019+0000 7f8ea7fff640 1 --2- 172.21.15.27:0/977756935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea8100670 0x7f8ea8198de0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009a50 tx=0x7f8e98004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.019+0000 7f8ea57fa640 1 -- 172.21.15.27:0/977756935 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9803f030 con 0x7f8ea8100670 2024-04-25T00:25:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.019+0000 7f8ea57fa640 1 -- 172.21.15.27:0/977756935 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e98003ce0 con 0x7f8ea8100670 2024-04-25T00:25:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.019+0000 7f8eae529640 1 -- 172.21.15.27:0/977756935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ea81a4570 con 0x7f8ea8100670 2024-04-25T00:25:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.019+0000 7f8ea57fa640 1 -- 172.21.15.27:0/977756935 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e98035070 con 0x7f8ea8100670 2024-04-25T00:25:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.019+0000 7f8eae529640 1 -- 172.21.15.27:0/977756935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ea81a49d0 con 0x7f8ea8100670 2024-04-25T00:25:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.020+0000 7f8eae529640 1 -- 172.21.15.27:0/977756935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ea8101d60 con 0x7f8ea8100670 2024-04-25T00:25:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.021+0000 7f8ea57fa640 1 -- 172.21.15.27:0/977756935 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8e98042030 con 0x7f8ea8100670 2024-04-25T00:25:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.022+0000 7f8ea57fa640 1 --2- 172.21.15.27:0/977756935 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8e940a2da0 0x7f8e940a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.022+0000 7f8ea77fe640 1 --2- 172.21.15.27:0/977756935 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8e940a2da0 0x7f8e940a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.022+0000 7f8ea57fa640 1 -- 172.21.15.27:0/977756935 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f8e980efe30 con 0x7f8ea8100670 2024-04-25T00:25:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.022+0000 7f8ea77fe640 1 --2- 172.21.15.27:0/977756935 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8e940a2da0 0x7f8e940a5260 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8069940 tx=0x7f8e9000a4c0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.026+0000 7f8ea57fa640 1 -- 172.21.15.27:0/977756935 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8e980bc6e0 con 0x7f8ea8100670 2024-04-25T00:25:55.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:54 smithi027 ceph-mon[29208]: pgmap v307: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:25:55.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.232+0000 7f8eae529640 1 -- 172.21.15.27:0/977756935 --> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f8ea8061b80 con 0x7f8e940a2da0 2024-04-25T00:25:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.241+0000 7f8ea57fa640 1 -- 172.21.15.27:0/977756935 <== mgr.15564 v2:172.21.15.27:6800/1076858552 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+332 (secure 0 0 0) 0x7f8ea8061b80 con 0x7f8e940a2da0 2024-04-25T00:25:55.242 INFO:teuthology.orchestra.run.smithi027.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-25T00:00:00", "created": "2024-04-25T00:15:53", "first": "2024-04-25T00:16:00", "last": "2024-04-25T00:25:00", "last_pruned": null, "created_count": 6, "pruned_count": 0, "active": true} 2024-04-25T00:25:55.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.243+0000 7f8eae529640 1 -- 172.21.15.27:0/977756935 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8e940a2da0 msgr2=0x7f8e940a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:55.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.243+0000 7f8eae529640 1 --2- 172.21.15.27:0/977756935 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8e940a2da0 0x7f8e940a5260 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8069940 tx=0x7f8e9000a4c0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.243+0000 7f8eae529640 1 -- 172.21.15.27:0/977756935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea8100670 msgr2=0x7f8ea8198de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:55.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.243+0000 7f8eae529640 1 --2- 172.21.15.27:0/977756935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea8100670 0x7f8ea8198de0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009a50 tx=0x7f8e98004270 comp rx=0 tx=0).stop 2024-04-25T00:25:55.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.243+0000 7f8eae529640 1 -- 172.21.15.27:0/977756935 shutdown_connections 2024-04-25T00:25:55.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.243+0000 7f8eae529640 1 --2- 172.21.15.27:0/977756935 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f8e940a2da0 0x7f8e940a5260 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.243+0000 7f8eae529640 1 --2- 172.21.15.27:0/977756935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ea8106630 0x7f8ea81a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.243+0000 7f8eae529640 1 --2- 172.21.15.27:0/977756935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ea8101030 0x7f8ea8199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.243+0000 7f8eae529640 1 --2- 172.21.15.27:0/977756935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea8100670 0x7f8ea8198de0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.243+0000 7f8eae529640 1 -- 172.21.15.27:0/977756935 >> 172.21.15.27:0/977756935 conn(0x7f8ea80777c0 msgr2=0x7f8ea8103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:55.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.243+0000 7f8eae529640 1 -- 172.21.15.27:0/977756935 shutdown_connections 2024-04-25T00:25:55.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.243+0000 7f8eae529640 1 -- 172.21.15.27:0/977756935 wait complete. 2024-04-25T00:25:55.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:54 smithi139 ceph-mon[32833]: pgmap v307: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:25:55.255 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-25T00:25:55.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:54 smithi153 ceph-mon[32896]: pgmap v307: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:25:55.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.358+0000 7f3e88d10640 1 -- 172.21.15.27:0/3586606645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e84103310 msgr2=0x7f3e8410b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:55.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.358+0000 7f3e88d10640 1 --2- 172.21.15.27:0/3586606645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e84103310 0x7f3e8410b7f0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f3e78009900 tx=0x7f3e7802a480 comp rx=0 tx=0).stop 2024-04-25T00:25:55.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.358+0000 7f3e88d10640 1 -- 172.21.15.27:0/3586606645 shutdown_connections 2024-04-25T00:25:55.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.358+0000 7f3e88d10640 1 --2- 172.21.15.27:0/3586606645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e8410bd30 0x7f3e8410e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.358+0000 7f3e88d10640 1 --2- 172.21.15.27:0/3586606645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e84103310 0x7f3e8410b7f0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.358+0000 7f3e88d10640 1 --2- 172.21.15.27:0/3586606645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e841029f0 0x7f3e84102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.358+0000 7f3e88d10640 1 -- 172.21.15.27:0/3586606645 >> 172.21.15.27:0/3586606645 conn(0x7f3e840777c0 msgr2=0x7f3e84077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.358+0000 7f3e88d10640 1 -- 172.21.15.27:0/3586606645 shutdown_connections 2024-04-25T00:25:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.358+0000 7f3e88d10640 1 -- 172.21.15.27:0/3586606645 wait complete. 2024-04-25T00:25:55.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.358+0000 7f3e88d10640 1 Processor -- start 2024-04-25T00:25:55.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e88d10640 1 -- start start 2024-04-25T00:25:55.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e88d10640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e841029f0 0x7f3e84198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:55.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e88d10640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e84103310 0x7f3e84198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:55.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e88d10640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e8410bd30 0x7f3e8419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:55.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e88d10640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3e84111b00 con 0x7f3e841029f0 2024-04-25T00:25:55.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e88d10640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3e84111980 con 0x7f3e84103310 2024-04-25T00:25:55.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e88d10640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3e84111c80 con 0x7f3e8410bd30 2024-04-25T00:25:55.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e8259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e841029f0 0x7f3e84198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:55.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e8259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e841029f0 0x7f3e84198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49062/0 (socket says 172.21.15.27:49062) 2024-04-25T00:25:55.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e8259c640 1 -- 172.21.15.27:0/580708828 learned_addr learned my addr 172.21.15.27:0/580708828 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:25:55.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e81d9b640 1 --2- 172.21.15.27:0/580708828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e84103310 0x7f3e84198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:55.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e82d9d640 1 --2- 172.21.15.27:0/580708828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e8410bd30 0x7f3e8419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:55.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e8259c640 1 -- 172.21.15.27:0/580708828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e8410bd30 msgr2=0x7f3e8419c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:55.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e8259c640 1 --2- 172.21.15.27:0/580708828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e8410bd30 0x7f3e8419c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e8259c640 1 -- 172.21.15.27:0/580708828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e84103310 msgr2=0x7f3e84198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e8259c640 1 --2- 172.21.15.27:0/580708828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e84103310 0x7f3e84198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e8259c640 1 -- 172.21.15.27:0/580708828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8419d030 con 0x7f3e841029f0 2024-04-25T00:25:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e81d9b640 1 --2- 172.21.15.27:0/580708828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e84103310 0x7f3e84198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:25:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e8259c640 1 --2- 172.21.15.27:0/580708828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e841029f0 0x7f3e84198040 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00d950 tx=0x7f3e7c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.359+0000 7f3e737fe640 1 -- 172.21.15.27:0/580708828 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c017070 con 0x7f3e841029f0 2024-04-25T00:25:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.360+0000 7f3e737fe640 1 -- 172.21.15.27:0/580708828 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e7c009c30 con 0x7f3e841029f0 2024-04-25T00:25:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.360+0000 7f3e88d10640 1 -- 172.21.15.27:0/580708828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e8419d320 con 0x7f3e841029f0 2024-04-25T00:25:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.360+0000 7f3e737fe640 1 -- 172.21.15.27:0/580708828 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c00f6b0 con 0x7f3e841029f0 2024-04-25T00:25:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.360+0000 7f3e88d10640 1 -- 172.21.15.27:0/580708828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e841a4bb0 con 0x7f3e841029f0 2024-04-25T00:25:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.361+0000 7f3e88d10640 1 -- 172.21.15.27:0/580708828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e8404f500 con 0x7f3e841029f0 2024-04-25T00:25:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.362+0000 7f3e737fe640 1 -- 172.21.15.27:0/580708828 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3e7c0109d0 con 0x7f3e841029f0 2024-04-25T00:25:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.362+0000 7f3e737fe640 1 --2- 172.21.15.27:0/580708828 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f3e6c0a2e20 0x7f3e6c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.362+0000 7f3e81d9b640 1 --2- 172.21.15.27:0/580708828 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f3e6c0a2e20 0x7f3e6c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.362+0000 7f3e737fe640 1 -- 172.21.15.27:0/580708828 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f3e7c0cf400 con 0x7f3e841029f0 2024-04-25T00:25:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.363+0000 7f3e81d9b640 1 --2- 172.21.15.27:0/580708828 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f3e6c0a2e20 0x7f3e6c0a52e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3e8410dac0 tx=0x7f3e78005a20 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:55.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.367+0000 7f3e737fe640 1 -- 172.21.15.27:0/580708828 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3e7c009de0 con 0x7f3e841029f0 2024-04-25T00:25:55.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.573+0000 7f3e88d10640 1 -- 172.21.15.27:0/580708828 --> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f3e84061b80 con 0x7f3e6c0a2e20 2024-04-25T00:25:55.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.580+0000 7f3e737fe640 1 -- 172.21.15.27:0/580708828 <== mgr.15564 v2:172.21.15.27:6800/1076858552 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f3e84061b80 con 0x7f3e6c0a2e20 2024-04-25T00:25:55.582 INFO:teuthology.orchestra.run.smithi027.stdout:/ 1m 6m3h 2024-04-25T00:25:55.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.582+0000 7f3e88d10640 1 -- 172.21.15.27:0/580708828 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f3e6c0a2e20 msgr2=0x7f3e6c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:55.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.582+0000 7f3e88d10640 1 --2- 172.21.15.27:0/580708828 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f3e6c0a2e20 0x7f3e6c0a52e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3e8410dac0 tx=0x7f3e78005a20 comp rx=0 tx=0).stop 2024-04-25T00:25:55.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.582+0000 7f3e88d10640 1 -- 172.21.15.27:0/580708828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e841029f0 msgr2=0x7f3e84198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:55.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.582+0000 7f3e88d10640 1 --2- 172.21.15.27:0/580708828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e841029f0 0x7f3e84198040 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00d950 tx=0x7f3e7c00de10 comp rx=0 tx=0).stop 2024-04-25T00:25:55.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.583+0000 7f3e88d10640 1 -- 172.21.15.27:0/580708828 shutdown_connections 2024-04-25T00:25:55.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.583+0000 7f3e88d10640 1 --2- 172.21.15.27:0/580708828 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f3e6c0a2e20 0x7f3e6c0a52e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.583+0000 7f3e88d10640 1 --2- 172.21.15.27:0/580708828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e8410bd30 0x7f3e8419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.583+0000 7f3e88d10640 1 --2- 172.21.15.27:0/580708828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e84103310 0x7f3e84198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.583+0000 7f3e88d10640 1 --2- 172.21.15.27:0/580708828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e841029f0 0x7f3e84198040 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.583+0000 7f3e88d10640 1 -- 172.21.15.27:0/580708828 >> 172.21.15.27:0/580708828 conn(0x7f3e840777c0 msgr2=0x7f3e840fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:55.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.583+0000 7f3e88d10640 1 -- 172.21.15.27:0/580708828 shutdown_connections 2024-04-25T00:25:55.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:25:55.583+0000 7f3e88d10640 1 -- 172.21.15.27:0/580708828 wait complete. 2024-04-25T00:25:55.595 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-25T00:25:55.609 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-25T00:25:57.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:56 smithi027 ceph-mon[29208]: from='client.17271 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:25:57.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:56 smithi027 ceph-mon[29208]: pgmap v308: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:25:57.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:56 smithi027 ceph-mon[29208]: from='client.17277 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:25:57.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:56 smithi139 ceph-mon[32833]: from='client.17271 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:25:57.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:56 smithi139 ceph-mon[32833]: pgmap v308: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:25:57.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:56 smithi139 ceph-mon[32833]: from='client.17277 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:25:57.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:56 smithi153 ceph-mon[32896]: from='client.17271 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:25:57.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:56 smithi153 ceph-mon[32896]: pgmap v308: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:25:57.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:56 smithi153 ceph-mon[32896]: from='client.17277 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T00:25:59.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:25:58 smithi027 ceph-mon[29208]: pgmap v309: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-25T00:25:59.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:25:58 smithi139 ceph-mon[32833]: pgmap v309: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-25T00:25:59.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:25:58 smithi153 ceph-mon[32896]: pgmap v309: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-25T00:26:01.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:00 smithi027 ceph-mon[29208]: pgmap v310: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:26:01.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:00 smithi139 ceph-mon[32833]: pgmap v310: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:26:01.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:00 smithi153 ceph-mon[32896]: pgmap v310: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:26:02.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T00:26:02.347 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T00:26:02.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.464+0000 7fee8c589640 1 -- 172.21.15.27:0/2048417176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee84069dc0 msgr2=0x7fee8406a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.464+0000 7fee8c589640 1 --2- 172.21.15.27:0/2048417176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee84069dc0 0x7fee8406a240 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fee74009930 tx=0x7fee7402a420 comp rx=0 tx=0).stop 2024-04-25T00:26:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.464+0000 7fee8c589640 1 -- 172.21.15.27:0/2048417176 shutdown_connections 2024-04-25T00:26:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.464+0000 7fee8c589640 1 --2- 172.21.15.27:0/2048417176 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee8406a780 0x7fee8410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.464+0000 7fee8c589640 1 --2- 172.21.15.27:0/2048417176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee84069dc0 0x7fee8406a240 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.464+0000 7fee8c589640 1 --2- 172.21.15.27:0/2048417176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee84107730 0x7fee84107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.464+0000 7fee8c589640 1 -- 172.21.15.27:0/2048417176 >> 172.21.15.27:0/2048417176 conn(0x7fee840776a0 msgr2=0x7fee84077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.464+0000 7fee8c589640 1 -- 172.21.15.27:0/2048417176 shutdown_connections 2024-04-25T00:26:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.464+0000 7fee8c589640 1 -- 172.21.15.27:0/2048417176 wait complete. 2024-04-25T00:26:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.465+0000 7fee8c589640 1 Processor -- start 2024-04-25T00:26:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.465+0000 7fee8c589640 1 -- start start 2024-04-25T00:26:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.465+0000 7fee8c589640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee84069dc0 0x7fee84198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.465+0000 7fee8c589640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee8406a780 0x7fee841985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.465+0000 7fee8c589640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee84107730 0x7fee8419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.465+0000 7fee8c589640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fee84111b00 con 0x7fee8406a780 2024-04-25T00:26:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.465+0000 7fee8c589640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fee84111980 con 0x7fee84107730 2024-04-25T00:26:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.465+0000 7fee8c589640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fee84111c80 con 0x7fee84069dc0 2024-04-25T00:26:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.465+0000 7fee89b24640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee8406a780 0x7fee841985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.465+0000 7fee89b24640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee8406a780 0x7fee841985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54254/0 (socket says 172.21.15.27:54254) 2024-04-25T00:26:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.465+0000 7fee89b24640 1 -- 172.21.15.27:0/1991233280 learned_addr learned my addr 172.21.15.27:0/1991233280 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.466+0000 7fee8ab26640 1 --2- 172.21.15.27:0/1991233280 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee84107730 0x7fee8419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.466+0000 7fee8a325640 1 --2- 172.21.15.27:0/1991233280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee84069dc0 0x7fee84198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.466+0000 7fee89b24640 1 -- 172.21.15.27:0/1991233280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee84069dc0 msgr2=0x7fee84198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.466+0000 7fee89b24640 1 --2- 172.21.15.27:0/1991233280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee84069dc0 0x7fee84198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.466+0000 7fee89b24640 1 -- 172.21.15.27:0/1991233280 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee84107730 msgr2=0x7fee8419c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.466+0000 7fee89b24640 1 --2- 172.21.15.27:0/1991233280 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee84107730 0x7fee8419c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.466+0000 7fee89b24640 1 -- 172.21.15.27:0/1991233280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee8419d050 con 0x7fee8406a780 2024-04-25T00:26:02.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.466+0000 7fee89b24640 1 --2- 172.21.15.27:0/1991233280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee8406a780 0x7fee841985a0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fee74009230 tx=0x7fee74004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:02.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.466+0000 7fee7b7fe640 1 -- 172.21.15.27:0/1991233280 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7403f030 con 0x7fee8406a780 2024-04-25T00:26:02.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.466+0000 7fee7b7fe640 1 -- 172.21.15.27:0/1991233280 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee74003ad0 con 0x7fee8406a780 2024-04-25T00:26:02.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.466+0000 7fee8c589640 1 -- 172.21.15.27:0/1991233280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee8419d2e0 con 0x7fee8406a780 2024-04-25T00:26:02.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.466+0000 7fee7b7fe640 1 -- 172.21.15.27:0/1991233280 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee74035070 con 0x7fee8406a780 2024-04-25T00:26:02.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.466+0000 7fee8c589640 1 -- 172.21.15.27:0/1991233280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee841a4bd0 con 0x7fee8406a780 2024-04-25T00:26:02.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.468+0000 7fee7b7fe640 1 -- 172.21.15.27:0/1991233280 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fee74042030 con 0x7fee8406a780 2024-04-25T00:26:02.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.469+0000 7fee8f374740 1 Processor -- start 2024-04-25T00:26:02.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.469+0000 7fee8f374740 1 -- start start 2024-04-25T00:26:02.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.469+0000 7fee8f374740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfb4630 0x562b7bfb4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.469+0000 7fee8f374740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfb4f50 0x562b7bfbf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.469+0000 7fee8f374740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfbf620 0x562b7bfc1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.469+0000 7fee8f374740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562b7bf196b0 con 0x562b7bfbf620 2024-04-25T00:26:02.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.469+0000 7fee8f374740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x562b7bec1880 con 0x562b7bfb4f50 2024-04-25T00:26:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.469+0000 7fee8f374740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562b7bf15670 con 0x562b7bfb4630 2024-04-25T00:26:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.469+0000 7fee7b7fe640 1 --2- 172.21.15.27:0/1991233280 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fee700a2d00 0x7fee700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.469+0000 7fee89b24640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfbf620 0x562b7bfc1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.469+0000 7fee8ab26640 1 --2- 172.21.15.27:0/1991233280 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fee700a2d00 0x7fee700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.469+0000 7fee8a325640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfb4f50 0x562b7bfbf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.470+0000 7fee89b24640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfbf620 0x562b7bfc1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54260/0 (socket says 172.21.15.27:54260) 2024-04-25T00:26:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.470+0000 7fee89b24640 1 -- 172.21.15.27:0/1405214148 learned_addr learned my addr 172.21.15.27:0/1405214148 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.470+0000 7fee8ab26640 1 --2- 172.21.15.27:0/1405214148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfb4630 0x562b7bfb4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.470+0000 7fee7b7fe640 1 -- 172.21.15.27:0/1991233280 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fee740efd60 con 0x7fee8406a780 2024-04-25T00:26:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.470+0000 7fee8a325640 1 -- 172.21.15.27:0/1405214148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfb4630 msgr2=0x562b7bfb4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.470+0000 7fee8a325640 1 --2- 172.21.15.27:0/1405214148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfb4630 0x562b7bfb4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.470+0000 7fee8a325640 1 -- 172.21.15.27:0/1405214148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfbf620 msgr2=0x562b7bfc1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.470+0000 7fee8a325640 1 --2- 172.21.15.27:0/1405214148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfbf620 0x562b7bfc1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.470+0000 7fee8a325640 1 -- 172.21.15.27:0/1405214148 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b7bcf77d0 con 0x562b7bfb4f50 2024-04-25T00:26:02.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.470+0000 7fee8ab26640 1 --2- 172.21.15.27:0/1991233280 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fee700a2d00 0x7fee700a51c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fee7c005fd0 tx=0x7fee7c009d10 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:02.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.470+0000 7fee8a325640 1 --2- 172.21.15.27:0/1405214148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfb4f50 0x562b7bfbf0e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fee8000db70 tx=0x7fee8000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=946195b853d51494 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:02.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.470+0000 7fee78bfe640 1 -- 172.21.15.27:0/1405214148 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8000de60 con 0x562b7bfb4f50 2024-04-25T00:26:02.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.470+0000 7fee78bfe640 1 -- 172.21.15.27:0/1405214148 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee80005500 con 0x562b7bfb4f50 2024-04-25T00:26:02.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.470+0000 7fee78bfe640 1 -- 172.21.15.27:0/1405214148 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee800116a0 con 0x562b7bfb4f50 2024-04-25T00:26:02.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.471+0000 7fee8f374740 1 -- 172.21.15.27:0/1405214148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfb4f50 msgr2=0x562b7bfbf0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.471+0000 7fee8f374740 1 --2- 172.21.15.27:0/1405214148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfb4f50 0x562b7bfbf0e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fee8000db70 tx=0x7fee8000f730 comp rx=0 tx=0).stop 2024-04-25T00:26:02.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.471+0000 7fee8f374740 1 -- 172.21.15.27:0/1405214148 shutdown_connections 2024-04-25T00:26:02.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.471+0000 7fee8f374740 1 --2- 172.21.15.27:0/1405214148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfbf620 0x562b7bfc1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.471+0000 7fee8f374740 1 --2- 172.21.15.27:0/1405214148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfb4f50 0x562b7bfbf0e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.471+0000 7fee8f374740 1 --2- 172.21.15.27:0/1405214148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfb4630 0x562b7bfb4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.471+0000 7fee8f374740 1 -- 172.21.15.27:0/1405214148 >> 172.21.15.27:0/1405214148 conn(0x562b7bfb0950 msgr2=0x562b7bfb0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:02.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.471+0000 7fee8f374740 1 -- 172.21.15.27:0/1405214148 shutdown_connections 2024-04-25T00:26:02.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.471+0000 7fee8f374740 1 -- 172.21.15.27:0/1405214148 wait complete. 2024-04-25T00:26:02.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8f374740 1 -- start start 2024-04-25T00:26:02.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8f374740 1 Processor -- start 2024-04-25T00:26:02.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8f374740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7bfbc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8f374740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfb4f50 0x562b7bfbcae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8f374740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfbf620 0x562b7bfbd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8f374740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562b7bfc2410 con 0x562b7bfb4630 2024-04-25T00:26:02.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8f374740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x562b7bed4f50 con 0x562b7bfb4f50 2024-04-25T00:26:02.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8f374740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562b7bfc2590 con 0x562b7bfbf620 2024-04-25T00:26:02.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8f374740 10 client.0 _ll_register_callbacks cb 0x562b7bf10000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:26:02.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8a325640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7bfbc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8a325640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7bfbc5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54272/0 (socket says 172.21.15.27:54272) 2024-04-25T00:26:02.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8a325640 1 -- 172.21.15.27:0/1874934552 learned_addr learned my addr 172.21.15.27:0/1874934552 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:02.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee89b24640 1 --2- 172.21.15.27:0/1874934552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfb4f50 0x562b7bfbcae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8ab26640 1 --2- 172.21.15.27:0/1874934552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfbf620 0x562b7bfbd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8a325640 1 -- 172.21.15.27:0/1874934552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfbf620 msgr2=0x562b7bfbd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:02.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8a325640 1 --2- 172.21.15.27:0/1874934552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfbf620 0x562b7bfbd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8a325640 1 -- 172.21.15.27:0/1874934552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfb4f50 msgr2=0x562b7bfbcae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:02.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8a325640 1 --2- 172.21.15.27:0/1874934552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfb4f50 0x562b7bfbcae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8a325640 1 -- 172.21.15.27:0/1874934552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b7bc2d3f0 con 0x562b7bfb4630 2024-04-25T00:26:02.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.472+0000 7fee8ab26640 1 --2- 172.21.15.27:0/1874934552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfbf620 0x562b7bfbd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:26:02.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.473+0000 7fee8a325640 1 --2- 172.21.15.27:0/1874934552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7bfbc5a0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fee80000c00 tx=0x7fee8000d8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a45dd0fb85ced0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:02.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.473+0000 7fee6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:26:02.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.473+0000 7fee6dffb640 1 -- 172.21.15.27:0/1874934552 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8002a060 con 0x562b7bfb4630 2024-04-25T00:26:02.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.473+0000 7fee6dffb640 1 -- 172.21.15.27:0/1874934552 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fee8000dc60 con 0x562b7bfb4630 2024-04-25T00:26:02.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.473+0000 7fee8f374740 1 -- 172.21.15.27:0/1874934552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee80005500 con 0x562b7bfb4630 2024-04-25T00:26:02.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.473+0000 7fee6dffb640 1 -- 172.21.15.27:0/1874934552 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fee80031050 con 0x562b7bfb4630 2024-04-25T00:26:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.473+0000 7fee8f374740 10 client.17289 fetch_fsmap learned FSMap version 101 2024-04-25T00:26:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.473+0000 7fee8f374740 1 -- 172.21.15.27:0/1874934552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x562b7bbed0f0 con 0x562b7bfb4630 2024-04-25T00:26:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.473+0000 7fee6dffb640 1 -- 172.21.15.27:0/1874934552 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee80005500 con 0x562b7bfb4630 2024-04-25T00:26:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.473+0000 7fee6dffb640 1 -- 172.21.15.27:0/1874934552 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 101) ==== 8362+0+0 (secure 0 0 0) 0x7fee80011940 con 0x562b7bfb4630 2024-04-25T00:26:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.473+0000 7fee8f374740 10 client.17289 fetch_fsmap finished waiting for FSMap version 101 2024-04-25T00:26:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.473+0000 7fee8f374740 10 client.17289 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:26:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.473+0000 7fee8f374740 20 client.17289 populate_metadata read hostname 'smithi027' 2024-04-25T00:26:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.473+0000 7fee8f374740 1 --2- 172.21.15.27:0/1874934552 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x562b7bfcf880 0x562b7bfd1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.473+0000 7fee8f374740 4 client.17289 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:26:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.473+0000 7fee8f374740 1 -- 172.21.15.27:0/1874934552 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b7bf26bf0 con 0x562b7bfcf880 2024-04-25T00:26:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.474+0000 7fee89b24640 1 --2- 172.21.15.27:0/1874934552 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x562b7bfcf880 0x562b7bfd1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.474+0000 7fee89b24640 1 --2- 172.21.15.27:0/1874934552 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x562b7bfcf880 0x562b7bfd1c70 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=537f597fcd9dbb3f server_cookie=de673c121d984978 in_seq=0 out_seq=0 2024-04-25T00:26:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.474+0000 7fee6dffb640 10 client.17289 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:26:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.475+0000 7fee6dffb640 1 -- 172.21.15.27:0/1874934552 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562b7bf26bf0 con 0x562b7bfcf880 2024-04-25T00:26:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.475+0000 7fee6dffb640 10 client.17289 handle_command_reply: tid=0 2024-04-25T00:26:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.475+0000 7fee8f374740 1 client.17289 shutdown 2024-04-25T00:26:02.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.475+0000 7fee8f374740 1 -- 172.21.15.27:0/1874934552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 msgr2=0x562b7bfbc5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.475+0000 7fee8f374740 1 --2- 172.21.15.27:0/1874934552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7bfbc5a0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fee80000c00 tx=0x7fee8000d8c0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.475+0000 7fee8f374740 1 -- 172.21.15.27:0/1874934552 shutdown_connections 2024-04-25T00:26:02.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.475+0000 7fee8f374740 1 --2- 172.21.15.27:0/1874934552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfbf620 0x562b7bfbd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.475+0000 7fee8f374740 1 --2- 172.21.15.27:0/1874934552 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x562b7bfcf880 0x562b7bfd1c70 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.475+0000 7fee8f374740 1 --2- 172.21.15.27:0/1874934552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfb4f50 0x562b7bfbcae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.475+0000 7fee8f374740 1 --2- 172.21.15.27:0/1874934552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7bfbc5a0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.475+0000 7fee8f374740 1 -- 172.21.15.27:0/1874934552 >> 172.21.15.27:0/1874934552 conn(0x562b7bfb0950 msgr2=0x562b7bfb6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:02.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.475+0000 7fee6dffb640 0 client.17289 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:26:02.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.475+0000 7fee8f374740 1 -- 172.21.15.27:0/1874934552 shutdown_connections 2024-04-25T00:26:02.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.475+0000 7fee8f374740 1 -- 172.21.15.27:0/1874934552 wait complete. 2024-04-25T00:26:02.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.475+0000 7fee8f374740 20 client.17289 trim_cache size 0 max 16384 2024-04-25T00:26:02.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.493+0000 7fee8f374740 1 Processor -- start 2024-04-25T00:26:02.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.493+0000 7fee8f374740 1 -- start start 2024-04-25T00:26:02.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee8f374740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7bfcb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee8f374740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfb4f50 0x562b7bfb6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee8f374740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfbf620 0x562b7bfc7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee8f374740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fee841aeb60 con 0x562b7bfb4630 2024-04-25T00:26:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee8f374740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fee841b06a0 con 0x562b7bfbf620 2024-04-25T00:26:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee8f374740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fee841af230 con 0x562b7bfb4f50 2024-04-25T00:26:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee8a325640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7bfcb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee8a325640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7bfcb2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54286/0 (socket says 172.21.15.27:54286) 2024-04-25T00:26:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee8a325640 1 -- 172.21.15.27:0/2081605347 learned_addr learned my addr 172.21.15.27:0/2081605347 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee8ab26640 1 --2- 172.21.15.27:0/2081605347 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfbf620 0x562b7bfc7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee8a325640 1 -- 172.21.15.27:0/2081605347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfb4f50 msgr2=0x562b7bfb6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee8a325640 1 --2- 172.21.15.27:0/2081605347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfb4f50 0x562b7bfb6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee8a325640 1 -- 172.21.15.27:0/2081605347 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfbf620 msgr2=0x562b7bfc7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee8a325640 1 --2- 172.21.15.27:0/2081605347 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfbf620 0x562b7bfc7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee8a325640 1 -- 172.21.15.27:0/2081605347 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b7bfa2090 con 0x562b7bfb4630 2024-04-25T00:26:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee8a325640 1 --2- 172.21.15.27:0/2081605347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7bfcb2d0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fee8000db60 tx=0x7fee800048d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa11d15b6772a0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:02.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee6ffff640 1 -- 172.21.15.27:0/2081605347 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8002a070 con 0x562b7bfb4630 2024-04-25T00:26:02.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee6ffff640 1 -- 172.21.15.27:0/2081605347 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee8000fc30 con 0x562b7bfb4630 2024-04-25T00:26:02.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.494+0000 7fee6ffff640 1 -- 172.21.15.27:0/2081605347 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee80011440 con 0x562b7bfb4630 2024-04-25T00:26:02.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.495+0000 7fee8f374740 1 -- 172.21.15.27:0/2081605347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 msgr2=0x562b7bfcb2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.495+0000 7fee8f374740 1 --2- 172.21.15.27:0/2081605347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7bfcb2d0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fee8000db60 tx=0x7fee800048d0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.495+0000 7fee8f374740 1 -- 172.21.15.27:0/2081605347 shutdown_connections 2024-04-25T00:26:02.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.495+0000 7fee8f374740 1 --2- 172.21.15.27:0/2081605347 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfbf620 0x562b7bfc7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.495+0000 7fee8f374740 1 --2- 172.21.15.27:0/2081605347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfb4f50 0x562b7bfb6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.495+0000 7fee8f374740 1 --2- 172.21.15.27:0/2081605347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7bfcb2d0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.495+0000 7fee8f374740 1 -- 172.21.15.27:0/2081605347 >> 172.21.15.27:0/2081605347 conn(0x562b7bfb0950 msgr2=0x562b7bfaf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:02.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.495+0000 7fee8f374740 1 -- 172.21.15.27:0/2081605347 shutdown_connections 2024-04-25T00:26:02.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.495+0000 7fee8f374740 1 -- 172.21.15.27:0/2081605347 wait complete. 2024-04-25T00:26:02.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.495+0000 7fee8f374740 1 -- start start 2024-04-25T00:26:02.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.495+0000 7fee8f374740 1 Processor -- start 2024-04-25T00:26:02.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8f374740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7c0321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8f374740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfb4f50 0x562b7c032720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8f374740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfbf620 0x562b7bfd6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8f374740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fee841affd0 con 0x562b7bfb4630 2024-04-25T00:26:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8f374740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fee841af900 con 0x562b7bfb4f50 2024-04-25T00:26:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8f374740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fee841acb20 con 0x562b7bfbf620 2024-04-25T00:26:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8f374740 10 client.0 _ll_register_callbacks cb 0x7fee80015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:26:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8a325640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7c0321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8a325640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7c0321e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54296/0 (socket says 172.21.15.27:54296) 2024-04-25T00:26:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8a325640 1 -- 172.21.15.27:0/841015376 learned_addr learned my addr 172.21.15.27:0/841015376 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8ab26640 1 --2- 172.21.15.27:0/841015376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfbf620 0x562b7bfd6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee89b24640 1 --2- 172.21.15.27:0/841015376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfb4f50 0x562b7c032720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8a325640 1 -- 172.21.15.27:0/841015376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfbf620 msgr2=0x562b7bfd6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8a325640 1 --2- 172.21.15.27:0/841015376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfbf620 0x562b7bfd6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8a325640 1 -- 172.21.15.27:0/841015376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfb4f50 msgr2=0x562b7c032720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8a325640 1 --2- 172.21.15.27:0/841015376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfb4f50 0x562b7c032720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8a325640 1 -- 172.21.15.27:0/841015376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b7bfd6fa0 con 0x562b7bfb4630 2024-04-25T00:26:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee89b24640 1 --2- 172.21.15.27:0/841015376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfb4f50 0x562b7c032720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:26:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8a325640 1 --2- 172.21.15.27:0/841015376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7c0321e0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fee80005880 tx=0x7fee8000d5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b4190562de5c3d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:26:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee6dffb640 1 -- 172.21.15.27:0/841015376 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8002a050 con 0x562b7bfb4630 2024-04-25T00:26:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee6dffb640 1 -- 172.21.15.27:0/841015376 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fee80025960 con 0x562b7bfb4630 2024-04-25T00:26:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.496+0000 7fee8f374740 1 -- 172.21.15.27:0/841015376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee8000fc30 con 0x562b7bfb4630 2024-04-25T00:26:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.497+0000 7fee6dffb640 1 -- 172.21.15.27:0/841015376 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fee80048050 con 0x562b7bfb4630 2024-04-25T00:26:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.497+0000 7fee8f374740 10 client.17295 fetch_fsmap learned FSMap version 101 2024-04-25T00:26:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.497+0000 7fee8f374740 1 -- 172.21.15.27:0/841015376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x562b7c039a10 con 0x562b7bfb4630 2024-04-25T00:26:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.497+0000 7fee6dffb640 1 -- 172.21.15.27:0/841015376 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee8000fc30 con 0x562b7bfb4630 2024-04-25T00:26:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.497+0000 7fee6dffb640 1 -- 172.21.15.27:0/841015376 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 101) ==== 8362+0+0 (secure 0 0 0) 0x7fee80035430 con 0x562b7bfb4630 2024-04-25T00:26:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.497+0000 7fee8f374740 10 client.17295 fetch_fsmap finished waiting for FSMap version 101 2024-04-25T00:26:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.497+0000 7fee8f374740 10 client.17295 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:26:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.497+0000 7fee8f374740 20 client.17295 populate_metadata read hostname 'smithi027' 2024-04-25T00:26:02.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.497+0000 7fee8f374740 1 --2- 172.21.15.27:0/841015376 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x562b7c03a0c0 0x562b7c03c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.497+0000 7fee8f374740 4 client.17295 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T00:26:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.497+0000 7fee8f374740 1 -- 172.21.15.27:0/841015376 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562b7bf20f40 con 0x562b7c03a0c0 2024-04-25T00:26:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.497+0000 7fee89b24640 1 --2- 172.21.15.27:0/841015376 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x562b7c03a0c0 0x562b7c03c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.498+0000 7fee89b24640 1 --2- 172.21.15.27:0/841015376 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x562b7c03a0c0 0x562b7c03c4e0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d21b8b1fd42aff42 server_cookie=4ebacae757d474c6 in_seq=0 out_seq=0 2024-04-25T00:26:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.498+0000 7fee6dffb640 10 client.17295 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:26:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.498+0000 7fee6dffb640 1 -- 172.21.15.27:0/841015376 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562b7bf20f40 con 0x562b7c03a0c0 2024-04-25T00:26:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.498+0000 7fee6dffb640 10 client.17295 handle_command_reply: tid=0 2024-04-25T00:26:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.498+0000 7fee8f374740 1 client.17295 shutdown 2024-04-25T00:26:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.498+0000 7fee8f374740 1 -- 172.21.15.27:0/841015376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 msgr2=0x562b7c0321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.498+0000 7fee8f374740 1 --2- 172.21.15.27:0/841015376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7c0321e0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fee80005880 tx=0x7fee8000d5c0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.498+0000 7fee8f374740 1 -- 172.21.15.27:0/841015376 shutdown_connections 2024-04-25T00:26:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.498+0000 7fee8f374740 1 --2- 172.21.15.27:0/841015376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562b7bfbf620 0x562b7bfd6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.498+0000 7fee8f374740 1 --2- 172.21.15.27:0/841015376 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x562b7c03a0c0 0x562b7c03c4e0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.498+0000 7fee8f374740 1 --2- 172.21.15.27:0/841015376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562b7bfb4f50 0x562b7c032720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.498+0000 7fee6dffb640 0 client.17295 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:26:02.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.498+0000 7fee8f374740 1 --2- 172.21.15.27:0/841015376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b7bfb4630 0x562b7c0321e0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.498+0000 7fee8f374740 1 -- 172.21.15.27:0/841015376 >> 172.21.15.27:0/841015376 conn(0x562b7bfb0950 msgr2=0x562b7bfaf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:02.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.498+0000 7fee8f374740 1 -- 172.21.15.27:0/841015376 shutdown_connections 2024-04-25T00:26:02.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.498+0000 7fee8f374740 1 -- 172.21.15.27:0/841015376 wait complete. 2024-04-25T00:26:02.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.498+0000 7fee8f374740 20 client.17295 trim_cache size 0 max 16384 2024-04-25T00:26:02.505 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:26:02.505 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T00:26:02.505 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T00:26:02.505 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:26:02.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.499+0000 7fee8c589640 1 -- 172.21.15.27:0/1991233280 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fee700a2d00 msgr2=0x7fee700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.499+0000 7fee8c589640 1 --2- 172.21.15.27:0/1991233280 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fee700a2d00 0x7fee700a51c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fee7c005fd0 tx=0x7fee7c009d10 comp rx=0 tx=0).stop 2024-04-25T00:26:02.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.499+0000 7fee8c589640 1 -- 172.21.15.27:0/1991233280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee8406a780 msgr2=0x7fee841985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.499+0000 7fee8c589640 1 --2- 172.21.15.27:0/1991233280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee8406a780 0x7fee841985a0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fee74009230 tx=0x7fee74004270 comp rx=0 tx=0).stop 2024-04-25T00:26:02.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.499+0000 7fee8c589640 1 -- 172.21.15.27:0/1991233280 shutdown_connections 2024-04-25T00:26:02.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.499+0000 7fee8c589640 1 --2- 172.21.15.27:0/1991233280 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fee700a2d00 0x7fee700a51c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.499+0000 7fee8c589640 1 --2- 172.21.15.27:0/1991233280 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee84107730 0x7fee8419c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.499+0000 7fee8c589640 1 --2- 172.21.15.27:0/1991233280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee8406a780 0x7fee841985a0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.499+0000 7fee8c589640 1 --2- 172.21.15.27:0/1991233280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee84069dc0 0x7fee84198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.499+0000 7fee8c589640 1 -- 172.21.15.27:0/1991233280 >> 172.21.15.27:0/1991233280 conn(0x7fee840776a0 msgr2=0x7fee840fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:02.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.499+0000 7fee8c589640 1 -- 172.21.15.27:0/1991233280 shutdown_connections 2024-04-25T00:26:02.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.499+0000 7fee8c589640 1 -- 172.21.15.27:0/1991233280 wait complete. 2024-04-25T00:26:02.511 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T00:26:02.511 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:02.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.612+0000 7f57edebd640 1 -- 172.21.15.27:0/4285579724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e8102a20 msgr2=0x7f57e8102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.612+0000 7f57edebd640 1 --2- 172.21.15.27:0/4285579724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e8102a20 0x7f57e8102e00 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f57d8009900 tx=0x7f57d802a480 comp rx=0 tx=0).stop 2024-04-25T00:26:02.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.612+0000 7f57edebd640 1 -- 172.21.15.27:0/4285579724 shutdown_connections 2024-04-25T00:26:02.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.612+0000 7f57edebd640 1 --2- 172.21.15.27:0/4285579724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57e810bd60 0x7f57e810e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.612+0000 7f57edebd640 1 --2- 172.21.15.27:0/4285579724 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57e8103340 0x7f57e810b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.612+0000 7f57edebd640 1 --2- 172.21.15.27:0/4285579724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e8102a20 0x7f57e8102e00 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.612+0000 7f57edebd640 1 -- 172.21.15.27:0/4285579724 >> 172.21.15.27:0/4285579724 conn(0x7f57e80777a0 msgr2=0x7f57e8077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:02.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.612+0000 7f57edebd640 1 -- 172.21.15.27:0/4285579724 shutdown_connections 2024-04-25T00:26:02.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.612+0000 7f57edebd640 1 -- 172.21.15.27:0/4285579724 wait complete. 2024-04-25T00:26:02.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.613+0000 7f57edebd640 1 Processor -- start 2024-04-25T00:26:02.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.613+0000 7f57edebd640 1 -- start start 2024-04-25T00:26:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.613+0000 7f57edebd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e8102a20 0x7f57e8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.613+0000 7f57edebd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57e8103340 0x7f57e8198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.613+0000 7f57edebd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57e810bd60 0x7f57e819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.613+0000 7f57edebd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f57e8111b30 con 0x7f57e8102a20 2024-04-25T00:26:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.613+0000 7f57edebd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f57e81119b0 con 0x7f57e810bd60 2024-04-25T00:26:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.613+0000 7f57edebd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f57e8111cb0 con 0x7f57e8103340 2024-04-25T00:26:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.613+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e8102a20 0x7f57e8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.613+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e8102a20 0x7f57e8197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54310/0 (socket says 172.21.15.27:54310) 2024-04-25T00:26:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.613+0000 7f57e77fe640 1 -- 172.21.15.27:0/2411839499 learned_addr learned my addr 172.21.15.27:0/2411839499 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.613+0000 7f57e6ffd640 1 --2- 172.21.15.27:0/2411839499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57e8103340 0x7f57e8198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.613+0000 7f57e7fff640 1 --2- 172.21.15.27:0/2411839499 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57e810bd60 0x7f57e819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.613+0000 7f57e77fe640 1 -- 172.21.15.27:0/2411839499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57e8103340 msgr2=0x7f57e8198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.613+0000 7f57e77fe640 1 --2- 172.21.15.27:0/2411839499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57e8103340 0x7f57e8198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.614+0000 7f57e77fe640 1 -- 172.21.15.27:0/2411839499 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57e810bd60 msgr2=0x7f57e819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.614+0000 7f57e77fe640 1 --2- 172.21.15.27:0/2411839499 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57e810bd60 0x7f57e819c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.614+0000 7f57e77fe640 1 -- 172.21.15.27:0/2411839499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e819cfc0 con 0x7f57e8102a20 2024-04-25T00:26:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.614+0000 7f57e77fe640 1 --2- 172.21.15.27:0/2411839499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e8102a20 0x7f57e8197fd0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f57d8006010 tx=0x7f57d80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:02.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.614+0000 7f57e4ff9640 1 -- 172.21.15.27:0/2411839499 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d803f030 con 0x7f57e8102a20 2024-04-25T00:26:02.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.614+0000 7f57edebd640 1 -- 172.21.15.27:0/2411839499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57e819d250 con 0x7f57e8102a20 2024-04-25T00:26:02.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.614+0000 7f57e4ff9640 1 -- 172.21.15.27:0/2411839499 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57d8003ce0 con 0x7f57e8102a20 2024-04-25T00:26:02.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.614+0000 7f57e4ff9640 1 -- 172.21.15.27:0/2411839499 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d8035070 con 0x7f57e8102a20 2024-04-25T00:26:02.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.614+0000 7f57edebd640 1 -- 172.21.15.27:0/2411839499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57e81a4b40 con 0x7f57e8102a20 2024-04-25T00:26:02.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.615+0000 7f57edebd640 1 -- 172.21.15.27:0/2411839499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57e80689b0 con 0x7f57e8102a20 2024-04-25T00:26:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.616+0000 7f57e4ff9640 1 -- 172.21.15.27:0/2411839499 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f57d8042030 con 0x7f57e8102a20 2024-04-25T00:26:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.617+0000 7f57e4ff9640 1 --2- 172.21.15.27:0/2411839499 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f57d00a2d00 0x7f57d00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.617+0000 7f57e6ffd640 1 --2- 172.21.15.27:0/2411839499 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f57d00a2d00 0x7f57d00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.617+0000 7f57e4ff9640 1 -- 172.21.15.27:0/2411839499 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f57d80efd80 con 0x7f57e8102a20 2024-04-25T00:26:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.617+0000 7f57e6ffd640 1 --2- 172.21.15.27:0/2411839499 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f57d00a2d00 0x7f57d00a51c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f57dc005fd0 tx=0x7f57dc009470 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:02.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.621+0000 7f57e4ff9640 1 -- 172.21.15.27:0/2411839499 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f57d80bc6f0 con 0x7f57e8102a20 2024-04-25T00:26:02.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.843+0000 7f57edebd640 1 -- 172.21.15.27:0/2411839499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57e81a4d10 con 0x7f57e8102a20 2024-04-25T00:26:02.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.844+0000 7f57e4ff9640 1 -- 172.21.15.27:0/2411839499 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+11952 (secure 0 0 0) 0x7f57d80bbff0 con 0x7f57e8102a20 2024-04-25T00:26:02.845 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:26:02.846 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":101,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:25:34.747257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":147,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":148,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":146,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24514,24430,24521,24428,24496]},"id":1}]} 2024-04-25T00:26:02.846 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-04-25T00:26:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.846+0000 7f57edebd640 1 -- 172.21.15.27:0/2411839499 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f57d00a2d00 msgr2=0x7f57d00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.846+0000 7f57edebd640 1 --2- 172.21.15.27:0/2411839499 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f57d00a2d00 0x7f57d00a51c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f57dc005fd0 tx=0x7f57dc009470 comp rx=0 tx=0).stop 2024-04-25T00:26:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.846+0000 7f57edebd640 1 -- 172.21.15.27:0/2411839499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e8102a20 msgr2=0x7f57e8197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.846+0000 7f57edebd640 1 --2- 172.21.15.27:0/2411839499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e8102a20 0x7f57e8197fd0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f57d8006010 tx=0x7f57d80042f0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.846+0000 7f57edebd640 1 -- 172.21.15.27:0/2411839499 shutdown_connections 2024-04-25T00:26:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.846+0000 7f57edebd640 1 --2- 172.21.15.27:0/2411839499 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f57d00a2d00 0x7f57d00a51c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.846+0000 7f57edebd640 1 --2- 172.21.15.27:0/2411839499 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57e810bd60 0x7f57e819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.846+0000 7f57edebd640 1 --2- 172.21.15.27:0/2411839499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57e8103340 0x7f57e8198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.846+0000 7f57edebd640 1 --2- 172.21.15.27:0/2411839499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e8102a20 0x7f57e8197fd0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.846+0000 7f57edebd640 1 -- 172.21.15.27:0/2411839499 >> 172.21.15.27:0/2411839499 conn(0x7f57e80777a0 msgr2=0x7f57e80fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.846+0000 7f57edebd640 1 -- 172.21.15.27:0/2411839499 shutdown_connections 2024-04-25T00:26:02.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.846+0000 7f57edebd640 1 -- 172.21.15.27:0/2411839499 wait complete. 2024-04-25T00:26:02.859 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T00:26:02.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.962+0000 7f2a0325d640 1 -- 172.21.15.27:0/2536227887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29fc107720 msgr2=0x7f29fc107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.962+0000 7f2a0325d640 1 --2- 172.21.15.27:0/2536227887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29fc107720 0x7f29fc107b00 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f29f0009930 tx=0x7f29f002a420 comp rx=0 tx=0).stop 2024-04-25T00:26:02.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.962+0000 7f2a0325d640 1 -- 172.21.15.27:0/2536227887 shutdown_connections 2024-04-25T00:26:02.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.962+0000 7f2a0325d640 1 --2- 172.21.15.27:0/2536227887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29fc1020d0 0x7f29fc10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.962+0000 7f2a0325d640 1 --2- 172.21.15.27:0/2536227887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29fc101710 0x7f29fc101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.962+0000 7f2a0325d640 1 --2- 172.21.15.27:0/2536227887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29fc107720 0x7f29fc107b00 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.962+0000 7f2a0325d640 1 -- 172.21.15.27:0/2536227887 >> 172.21.15.27:0/2536227887 conn(0x7f29fc078b90 msgr2=0x7f29fc078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:02.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.962+0000 7f2a0325d640 1 -- 172.21.15.27:0/2536227887 shutdown_connections 2024-04-25T00:26:02.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.962+0000 7f2a0325d640 1 -- 172.21.15.27:0/2536227887 wait complete. 2024-04-25T00:26:02.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.963+0000 7f2a0325d640 1 Processor -- start 2024-04-25T00:26:02.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.963+0000 7f2a0325d640 1 -- start start 2024-04-25T00:26:02.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.963+0000 7f2a0325d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29fc101710 0x7f29fc198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.963+0000 7f2a0325d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29fc1020d0 0x7f29fc198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.963+0000 7f2a0325d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29fc107720 0x7f29fc19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.963+0000 7f2a0325d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f29fc111b40 con 0x7f29fc1020d0 2024-04-25T00:26:02.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.963+0000 7f2a0325d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f29fc1119c0 con 0x7f29fc101710 2024-04-25T00:26:02.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.963+0000 7f2a0325d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29fc111cc0 con 0x7f29fc107720 2024-04-25T00:26:02.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.963+0000 7f29fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29fc1020d0 0x7f29fc198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.963+0000 7f29fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29fc1020d0 0x7f29fc198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54340/0 (socket says 172.21.15.27:54340) 2024-04-25T00:26:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.963+0000 7f29fbfff640 1 -- 172.21.15.27:0/3065374155 learned_addr learned my addr 172.21.15.27:0/3065374155 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.963+0000 7f29fbfff640 1 -- 172.21.15.27:0/3065374155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29fc107720 msgr2=0x7f29fc19c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.963+0000 7f2a017fa640 1 --2- 172.21.15.27:0/3065374155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29fc107720 0x7f29fc19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.963+0000 7f2a00ff9640 1 --2- 172.21.15.27:0/3065374155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29fc101710 0x7f29fc198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.964+0000 7f29fbfff640 1 --2- 172.21.15.27:0/3065374155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29fc107720 0x7f29fc19c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.964+0000 7f29fbfff640 1 -- 172.21.15.27:0/3065374155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29fc101710 msgr2=0x7f29fc198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.964+0000 7f29fbfff640 1 --2- 172.21.15.27:0/3065374155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29fc101710 0x7f29fc198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.964+0000 7f29fbfff640 1 -- 172.21.15.27:0/3065374155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29fc19d010 con 0x7f29fc1020d0 2024-04-25T00:26:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.964+0000 7f2a017fa640 1 --2- 172.21.15.27:0/3065374155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29fc107720 0x7f29fc19c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:26:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.964+0000 7f29fbfff640 1 --2- 172.21.15.27:0/3065374155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29fc1020d0 0x7f29fc198560 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f29e800d950 tx=0x7f29e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.964+0000 7f29f9ffb640 1 -- 172.21.15.27:0/3065374155 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e8017070 con 0x7f29fc1020d0 2024-04-25T00:26:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.964+0000 7f29f9ffb640 1 -- 172.21.15.27:0/3065374155 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29e8009c30 con 0x7f29fc1020d0 2024-04-25T00:26:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.964+0000 7f2a0325d640 1 -- 172.21.15.27:0/3065374155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29fc19d300 con 0x7f29fc1020d0 2024-04-25T00:26:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.964+0000 7f29f9ffb640 1 -- 172.21.15.27:0/3065374155 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e800f6b0 con 0x7f29fc1020d0 2024-04-25T00:26:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.964+0000 7f2a0325d640 1 -- 172.21.15.27:0/3065374155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29fc1a4b90 con 0x7f29fc1020d0 2024-04-25T00:26:02.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.966+0000 7f29f9ffb640 1 -- 172.21.15.27:0/3065374155 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f29e80109d0 con 0x7f29fc1020d0 2024-04-25T00:26:02.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a05f6c740 1 Processor -- start 2024-04-25T00:26:02.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a05f6c740 1 -- start start 2024-04-25T00:26:02.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f29f9ffb640 1 --2- 172.21.15.27:0/3065374155 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f29ec0a2d80 0x7f29ec0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a017fa640 1 --2- 172.21.15.27:0/3065374155 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f29ec0a2d80 0x7f29ec0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a05f6c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d0ed960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a05f6c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0edea0 0x56160d0f8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f29f9ffb640 1 -- 172.21.15.27:0/3065374155 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f29e80cf400 con 0x7f29fc1020d0 2024-04-25T00:26:02.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a05f6c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0f8570 0x56160d0fa960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a05f6c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56160d052670 con 0x56160d0ed580 2024-04-25T00:26:02.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a05f6c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56160cffa970 con 0x56160d0f8570 2024-04-25T00:26:02.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a05f6c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56160d04e630 con 0x56160d0edea0 2024-04-25T00:26:02.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a00ff9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d0ed960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a00ff9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d0ed960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54346/0 (socket says 172.21.15.27:54346) 2024-04-25T00:26:02.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a00ff9640 1 -- 172.21.15.27:0/1582517783 learned_addr learned my addr 172.21.15.27:0/1582517783 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:02.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f29fbfff640 1 --2- 172.21.15.27:0/1582517783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0edea0 0x56160d0f8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a017fa640 1 --2- 172.21.15.27:0/1582517783 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0f8570 0x56160d0fa960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a017fa640 1 --2- 172.21.15.27:0/3065374155 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f29ec0a2d80 0x7f29ec0a5240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f29f4008c60 tx=0x7f29f400d710 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:02.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a00ff9640 1 -- 172.21.15.27:0/1582517783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0edea0 msgr2=0x56160d0f8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:02.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a00ff9640 1 --2- 172.21.15.27:0/1582517783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0edea0 0x56160d0f8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a00ff9640 1 -- 172.21.15.27:0/1582517783 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0f8570 msgr2=0x56160d0fa960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:02.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a00ff9640 1 --2- 172.21.15.27:0/1582517783 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0f8570 0x56160d0fa960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a00ff9640 1 -- 172.21.15.27:0/1582517783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56160d01e400 con 0x56160d0ed580 2024-04-25T00:26:02.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.967+0000 7f2a017fa640 1 --2- 172.21.15.27:0/1582517783 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0f8570 0x56160d0fa960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:26:02.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.968+0000 7f2a00ff9640 1 --2- 172.21.15.27:0/1582517783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d0ed960 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f29f0006010 tx=0x7f29f00040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78af5bd31eaf89ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:02.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.968+0000 7f29e6ffd640 1 -- 172.21.15.27:0/1582517783 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f003f070 con 0x56160d0ed580 2024-04-25T00:26:02.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.968+0000 7f29e6ffd640 1 -- 172.21.15.27:0/1582517783 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29f0005520 con 0x56160d0ed580 2024-04-25T00:26:02.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.968+0000 7f29e6ffd640 1 -- 172.21.15.27:0/1582517783 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f0035030 con 0x56160d0ed580 2024-04-25T00:26:02.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.968+0000 7f2a05f6c740 1 -- 172.21.15.27:0/1582517783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 msgr2=0x56160d0ed960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.968+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/1582517783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d0ed960 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f29f0006010 tx=0x7f29f00040c0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.968+0000 7f2a05f6c740 1 -- 172.21.15.27:0/1582517783 shutdown_connections 2024-04-25T00:26:02.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.968+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/1582517783 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0f8570 0x56160d0fa960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.968+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/1582517783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0edea0 0x56160d0f8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.968+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/1582517783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d0ed960 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.968+0000 7f2a05f6c740 1 -- 172.21.15.27:0/1582517783 >> 172.21.15.27:0/1582517783 conn(0x56160d0e98a0 msgr2=0x56160d0e9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:02.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.968+0000 7f2a05f6c740 1 -- 172.21.15.27:0/1582517783 shutdown_connections 2024-04-25T00:26:02.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.968+0000 7f2a05f6c740 1 -- 172.21.15.27:0/1582517783 wait complete. 2024-04-25T00:26:02.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.969+0000 7f2a05f6c740 1 -- start start 2024-04-25T00:26:02.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.969+0000 7f2a05f6c740 1 Processor -- start 2024-04-25T00:26:02.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.969+0000 7f2a05f6c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0ed580 0x56160d0f7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.969+0000 7f2a05f6c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0edea0 0x56160d0f5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.969+0000 7f2a05f6c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0f8570 0x56160d0f56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.969+0000 7f2a05f6c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56160d0fb300 con 0x56160d0edea0 2024-04-25T00:26:02.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.969+0000 7f2a05f6c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56160d00e050 con 0x56160d0ed580 2024-04-25T00:26:02.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.969+0000 7f2a05f6c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56160d0fb480 con 0x56160d0f8570 2024-04-25T00:26:02.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.969+0000 7f2a05f6c740 10 client.0 _ll_register_callbacks cb 0x56160cd65be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:26:02.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.969+0000 7f29fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0edea0 0x56160d0f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.969+0000 7f29fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0edea0 0x56160d0f5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54350/0 (socket says 172.21.15.27:54350) 2024-04-25T00:26:02.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.969+0000 7f29fbfff640 1 -- 172.21.15.27:0/1700536540 learned_addr learned my addr 172.21.15.27:0/1700536540 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:02.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.969+0000 7f2a017fa640 1 --2- 172.21.15.27:0/1700536540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0f8570 0x56160d0f56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.970+0000 7f2a00ff9640 1 --2- 172.21.15.27:0/1700536540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0ed580 0x56160d0f7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.970+0000 7f29fbfff640 1 -- 172.21.15.27:0/1700536540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0f8570 msgr2=0x56160d0f56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:02.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.970+0000 7f29fbfff640 1 --2- 172.21.15.27:0/1700536540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0f8570 0x56160d0f56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.970+0000 7f29fbfff640 1 -- 172.21.15.27:0/1700536540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0ed580 msgr2=0x56160d0f7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:02.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.970+0000 7f29fbfff640 1 --2- 172.21.15.27:0/1700536540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0ed580 0x56160d0f7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.970+0000 7f29fbfff640 1 -- 172.21.15.27:0/1700536540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56160cfdfdc0 con 0x56160d0edea0 2024-04-25T00:26:02.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.970+0000 7f2a00ff9640 1 --2- 172.21.15.27:0/1700536540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0ed580 0x56160d0f7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:26:02.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.970+0000 7f29fbfff640 1 --2- 172.21.15.27:0/1700536540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0edea0 0x56160d0f5170 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f29e80cc840 tx=0x7f29e8009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=286ba9e9c71246c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:02.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.970+0000 7f29dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:26:02.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.970+0000 7f29dbfff640 1 -- 172.21.15.27:0/1700536540 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e8020040 con 0x56160d0edea0 2024-04-25T00:26:02.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.970+0000 7f29dbfff640 1 -- 172.21.15.27:0/1700536540 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f29e8097c20 con 0x56160d0edea0 2024-04-25T00:26:02.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.970+0000 7f2a05f6c740 1 -- 172.21.15.27:0/1700536540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29f0005520 con 0x56160d0edea0 2024-04-25T00:26:02.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.970+0000 7f29dbfff640 1 -- 172.21.15.27:0/1700536540 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f29e809f960 con 0x56160d0edea0 2024-04-25T00:26:02.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.970+0000 7f2a05f6c740 10 client.17313 fetch_fsmap learned FSMap version 101 2024-04-25T00:26:02.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.970+0000 7f2a05f6c740 1 -- 172.21.15.27:0/1700536540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x56160cd663f0 con 0x56160d0edea0 2024-04-25T00:26:02.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.970+0000 7f29dbfff640 1 -- 172.21.15.27:0/1700536540 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29f0005520 con 0x56160d0edea0 2024-04-25T00:26:02.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.971+0000 7f29dbfff640 1 -- 172.21.15.27:0/1700536540 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 101) ==== 8362+0+0 (secure 0 0 0) 0x7f29e8097300 con 0x56160d0edea0 2024-04-25T00:26:02.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.971+0000 7f2a05f6c740 10 client.17313 fetch_fsmap finished waiting for FSMap version 101 2024-04-25T00:26:02.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.971+0000 7f2a05f6c740 10 client.17313 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:26:02.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.971+0000 7f2a05f6c740 20 client.17313 populate_metadata read hostname 'smithi027' 2024-04-25T00:26:02.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.971+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/1700536540 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x56160d1084f0 0x56160d10a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.971+0000 7f2a05f6c740 4 client.17313 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:26:02.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.971+0000 7f2a05f6c740 1 -- 172.21.15.27:0/1700536540 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56160d05fbb0 con 0x56160d1084f0 2024-04-25T00:26:02.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.971+0000 7f2a00ff9640 1 --2- 172.21.15.27:0/1700536540 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x56160d1084f0 0x56160d10a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.971+0000 7f2a00ff9640 1 --2- 172.21.15.27:0/1700536540 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x56160d1084f0 0x56160d10a940 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1f8352f3cf693882 server_cookie=878144d0d196ec4e in_seq=0 out_seq=0 2024-04-25T00:26:02.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.971+0000 7f29dbfff640 10 client.17313 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:26:02.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.972+0000 7f29dbfff640 1 -- 172.21.15.27:0/1700536540 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56160d05fbb0 con 0x56160d1084f0 2024-04-25T00:26:02.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.972+0000 7f29dbfff640 10 client.17313 handle_command_reply: tid=0 2024-04-25T00:26:02.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.972+0000 7f2a05f6c740 1 client.17313 shutdown 2024-04-25T00:26:02.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.972+0000 7f2a05f6c740 1 -- 172.21.15.27:0/1700536540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0edea0 msgr2=0x56160d0f5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.972+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/1700536540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0edea0 0x56160d0f5170 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f29e80cc840 tx=0x7f29e8009ea0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.972+0000 7f2a05f6c740 1 -- 172.21.15.27:0/1700536540 shutdown_connections 2024-04-25T00:26:02.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.972+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/1700536540 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x56160d1084f0 0x56160d10a940 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.972+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/1700536540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0f8570 0x56160d0f56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.972+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/1700536540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0edea0 0x56160d0f5170 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.972+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/1700536540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0ed580 0x56160d0f7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.972+0000 7f29dbfff640 0 client.17313 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:26:02.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.972+0000 7f2a05f6c740 1 -- 172.21.15.27:0/1700536540 >> 172.21.15.27:0/1700536540 conn(0x56160d0e98a0 msgr2=0x56160d0e8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:02.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.972+0000 7f2a05f6c740 1 -- 172.21.15.27:0/1700536540 shutdown_connections 2024-04-25T00:26:02.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.972+0000 7f2a05f6c740 1 -- 172.21.15.27:0/1700536540 wait complete. 2024-04-25T00:26:02.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.973+0000 7f2a05f6c740 20 client.17313 trim_cache size 0 max 16384 2024-04-25T00:26:02.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f2a05f6c740 1 Processor -- start 2024-04-25T00:26:02.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f2a05f6c740 1 -- start start 2024-04-25T00:26:02.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f2a05f6c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d1041f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f2a05f6c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0edea0 0x56160d0f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f2a05f6c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0f8570 0x56160d110210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f2a05f6c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f29fc1aeae0 con 0x56160d0ed580 2024-04-25T00:26:02.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f2a05f6c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f29fc1b0620 con 0x56160d0f8570 2024-04-25T00:26:02.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f2a05f6c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29fc1af1b0 con 0x56160d0edea0 2024-04-25T00:26:02.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f2a00ff9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d1041f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f2a00ff9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d1041f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54354/0 (socket says 172.21.15.27:54354) 2024-04-25T00:26:02.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f2a00ff9640 1 -- 172.21.15.27:0/3602393255 learned_addr learned my addr 172.21.15.27:0/3602393255 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:02.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f29fbfff640 1 --2- 172.21.15.27:0/3602393255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0edea0 0x56160d0f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f2a00ff9640 1 -- 172.21.15.27:0/3602393255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0edea0 msgr2=0x56160d0f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:02.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f2a00ff9640 1 --2- 172.21.15.27:0/3602393255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0edea0 0x56160d0f79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f2a00ff9640 1 -- 172.21.15.27:0/3602393255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0f8570 msgr2=0x56160d110210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:02.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f2a00ff9640 1 --2- 172.21.15.27:0/3602393255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0f8570 0x56160d110210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f2a00ff9640 1 -- 172.21.15.27:0/3602393255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56160ce307d0 con 0x56160d0ed580 2024-04-25T00:26:02.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.986+0000 7f2a00ff9640 1 --2- 172.21.15.27:0/3602393255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d1041f0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f29f0004fd0 tx=0x7f29f0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29de6e2cecad8e41 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:02.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.987+0000 7f29e67fc640 1 -- 172.21.15.27:0/3602393255 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f003f040 con 0x56160d0ed580 2024-04-25T00:26:02.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.987+0000 7f29e67fc640 1 -- 172.21.15.27:0/3602393255 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29f00054e0 con 0x56160d0ed580 2024-04-25T00:26:02.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.987+0000 7f29e67fc640 1 -- 172.21.15.27:0/3602393255 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f0035040 con 0x56160d0ed580 2024-04-25T00:26:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.987+0000 7f2a05f6c740 1 -- 172.21.15.27:0/3602393255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 msgr2=0x56160d1041f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.987+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/3602393255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d1041f0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f29f0004fd0 tx=0x7f29f0000bc0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.987+0000 7f2a05f6c740 1 -- 172.21.15.27:0/3602393255 shutdown_connections 2024-04-25T00:26:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.987+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/3602393255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0f8570 0x56160d110210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.987+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/3602393255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0edea0 0x56160d0f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.987+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/3602393255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d1041f0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.987+0000 7f2a05f6c740 1 -- 172.21.15.27:0/3602393255 >> 172.21.15.27:0/3602393255 conn(0x56160d0e98a0 msgr2=0x56160d0f0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.987+0000 7f2a05f6c740 1 -- 172.21.15.27:0/3602393255 shutdown_connections 2024-04-25T00:26:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.987+0000 7f2a05f6c740 1 -- 172.21.15.27:0/3602393255 wait complete. 2024-04-25T00:26:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.987+0000 7f2a05f6c740 1 -- start start 2024-04-25T00:26:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f2a05f6c740 1 Processor -- start 2024-04-25T00:26:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f2a05f6c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d1008f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f2a05f6c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0edea0 0x56160d0f1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f2a05f6c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0f8570 0x56160d0f1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f2a05f6c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f29fc1aff50 con 0x56160d0ed580 2024-04-25T00:26:02.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f2a05f6c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f29fc1af880 con 0x56160d0edea0 2024-04-25T00:26:02.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f2a05f6c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29fc1acaa0 con 0x56160d0f8570 2024-04-25T00:26:02.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f2a05f6c740 10 client.0 _ll_register_callbacks cb 0x7f29e800df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:26:02.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f2a00ff9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d1008f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f2a00ff9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d1008f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54370/0 (socket says 172.21.15.27:54370) 2024-04-25T00:26:02.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f2a00ff9640 1 -- 172.21.15.27:0/3877992446 learned_addr learned my addr 172.21.15.27:0/3877992446 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:02.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f29fbfff640 1 --2- 172.21.15.27:0/3877992446 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0edea0 0x56160d0f1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f2a017fa640 1 --2- 172.21.15.27:0/3877992446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0f8570 0x56160d0f1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f2a00ff9640 1 -- 172.21.15.27:0/3877992446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0f8570 msgr2=0x56160d0f1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:02.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f2a00ff9640 1 --2- 172.21.15.27:0/3877992446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0f8570 0x56160d0f1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f2a00ff9640 1 -- 172.21.15.27:0/3877992446 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0edea0 msgr2=0x56160d0f1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:02.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f2a00ff9640 1 --2- 172.21.15.27:0/3877992446 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0edea0 0x56160d0f1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.988+0000 7f2a00ff9640 1 -- 172.21.15.27:0/3877992446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56160d0f2500 con 0x56160d0ed580 2024-04-25T00:26:02.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.989+0000 7f2a00ff9640 1 --2- 172.21.15.27:0/3877992446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d1008f0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f29f0005920 tx=0x7f29f0005950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d826014bf0dab7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:02.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.989+0000 7f29dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:26:02.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.989+0000 7f29dbfff640 1 -- 172.21.15.27:0/3877992446 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f0035070 con 0x56160d0ed580 2024-04-25T00:26:02.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.989+0000 7f2a05f6c740 1 -- 172.21.15.27:0/3877992446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29f00054e0 con 0x56160d0ed580 2024-04-25T00:26:02.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.989+0000 7f29dbfff640 1 -- 172.21.15.27:0/3877992446 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f29f00046c0 con 0x56160d0ed580 2024-04-25T00:26:02.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.989+0000 7f29dbfff640 1 -- 172.21.15.27:0/3877992446 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f29f00048c0 con 0x56160d0ed580 2024-04-25T00:26:02.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.989+0000 7f2a05f6c740 10 client.17319 fetch_fsmap learned FSMap version 101 2024-04-25T00:26:02.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.989+0000 7f2a05f6c740 1 -- 172.21.15.27:0/3877992446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x56160d16e870 con 0x56160d0ed580 2024-04-25T00:26:02.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.989+0000 7f29dbfff640 1 -- 172.21.15.27:0/3877992446 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29f00054e0 con 0x56160d0ed580 2024-04-25T00:26:02.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.989+0000 7f29dbfff640 1 -- 172.21.15.27:0/3877992446 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 101) ==== 8362+0+0 (secure 0 0 0) 0x7f29f00035f0 con 0x56160d0ed580 2024-04-25T00:26:02.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.989+0000 7f2a05f6c740 10 client.17319 fetch_fsmap finished waiting for FSMap version 101 2024-04-25T00:26:02.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.989+0000 7f2a05f6c740 10 client.17319 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T00:26:02.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.989+0000 7f2a05f6c740 20 client.17319 populate_metadata read hostname 'smithi027' 2024-04-25T00:26:02.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.989+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/3877992446 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x56160d16ec00 0x56160d171020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.989+0000 7f2a05f6c740 4 client.17319 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:26:02.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.989+0000 7f2a05f6c740 1 -- 172.21.15.27:0/3877992446 --> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56160d059f00 con 0x56160d16ec00 2024-04-25T00:26:02.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.990+0000 7f29fbfff640 1 --2- 172.21.15.27:0/3877992446 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x56160d16ec00 0x56160d171020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.990+0000 7f29fbfff640 1 --2- 172.21.15.27:0/3877992446 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x56160d16ec00 0x56160d171020 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ad27277c9688de4a server_cookie=9fc39c59c78faf70 in_seq=0 out_seq=0 2024-04-25T00:26:02.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.990+0000 7f29dbfff640 10 client.17319 ms_handle_connect on v2:172.21.15.139:6832/3964352148 2024-04-25T00:26:02.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.990+0000 7f29dbfff640 1 -- 172.21.15.27:0/3877992446 <== mds.4 v2:172.21.15.139:6832/3964352148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56160d059f00 con 0x56160d16ec00 2024-04-25T00:26:02.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.990+0000 7f29dbfff640 10 client.17319 handle_command_reply: tid=0 2024-04-25T00:26:02.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.990+0000 7f2a05f6c740 1 client.17319 shutdown 2024-04-25T00:26:02.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.990+0000 7f2a05f6c740 1 -- 172.21.15.27:0/3877992446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 msgr2=0x56160d1008f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.990+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/3877992446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d1008f0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f29f0005920 tx=0x7f29f0005950 comp rx=0 tx=0).stop 2024-04-25T00:26:02.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.990+0000 7f2a05f6c740 1 -- 172.21.15.27:0/3877992446 shutdown_connections 2024-04-25T00:26:02.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.990+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/3877992446 >> [v2:172.21.15.139:6832/3964352148,v1:172.21.15.139:6833/3964352148] conn(0x56160d16ec00 0x56160d171020 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.990+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/3877992446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56160d0f8570 0x56160d0f1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.990+0000 7f29dbfff640 0 client.17319 ms_handle_reset on v2:172.21.15.139:6832/3964352148 2024-04-25T00:26:02.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.990+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/3877992446 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56160d0edea0 0x56160d0f1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.990+0000 7f2a05f6c740 1 --2- 172.21.15.27:0/3877992446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56160d0ed580 0x56160d1008f0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.990+0000 7f2a05f6c740 1 -- 172.21.15.27:0/3877992446 >> 172.21.15.27:0/3877992446 conn(0x56160d0e98a0 msgr2=0x56160d0efeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:02.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.990+0000 7f2a05f6c740 1 -- 172.21.15.27:0/3877992446 shutdown_connections 2024-04-25T00:26:02.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.991+0000 7f2a05f6c740 1 -- 172.21.15.27:0/3877992446 wait complete. 2024-04-25T00:26:02.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.991+0000 7f2a05f6c740 20 client.17319 trim_cache size 0 max 16384 2024-04-25T00:26:02.998 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:26:02.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.991+0000 7f2a0325d640 1 -- 172.21.15.27:0/3065374155 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f29ec0a2d80 msgr2=0x7f29ec0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.991+0000 7f2a0325d640 1 --2- 172.21.15.27:0/3065374155 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f29ec0a2d80 0x7f29ec0a5240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f29f4008c60 tx=0x7f29f400d710 comp rx=0 tx=0).stop 2024-04-25T00:26:02.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.991+0000 7f2a0325d640 1 -- 172.21.15.27:0/3065374155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29fc1020d0 msgr2=0x7f29fc198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.991+0000 7f2a0325d640 1 --2- 172.21.15.27:0/3065374155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29fc1020d0 0x7f29fc198560 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f29e800d950 tx=0x7f29e800de10 comp rx=0 tx=0).stop 2024-04-25T00:26:02.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.991+0000 7f2a0325d640 1 -- 172.21.15.27:0/3065374155 shutdown_connections 2024-04-25T00:26:02.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.991+0000 7f2a0325d640 1 --2- 172.21.15.27:0/3065374155 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f29ec0a2d80 0x7f29ec0a5240 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.991+0000 7f2a0325d640 1 --2- 172.21.15.27:0/3065374155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29fc107720 0x7f29fc19c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.991+0000 7f2a0325d640 1 --2- 172.21.15.27:0/3065374155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29fc1020d0 0x7f29fc198560 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.991+0000 7f2a0325d640 1 --2- 172.21.15.27:0/3065374155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29fc101710 0x7f29fc198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.991+0000 7f2a0325d640 1 -- 172.21.15.27:0/3065374155 >> 172.21.15.27:0/3065374155 conn(0x7f29fc078b90 msgr2=0x7f29fc0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.991+0000 7f2a0325d640 1 -- 172.21.15.27:0/3065374155 shutdown_connections 2024-04-25T00:26:03.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:02.991+0000 7f2a0325d640 1 -- 172.21.15.27:0/3065374155 wait complete. 2024-04-25T00:26:03.003 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T00:26:03.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.106+0000 7f7eee109640 1 -- 172.21.15.27:0/4151066733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee8069dc0 msgr2=0x7f7ee806a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.106+0000 7f7eee109640 1 --2- 172.21.15.27:0/4151066733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee8069dc0 0x7f7ee806a240 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f7edc009930 tx=0x7f7edc02a420 comp rx=0 tx=0).stop 2024-04-25T00:26:03.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.106+0000 7f7eee109640 1 -- 172.21.15.27:0/4151066733 shutdown_connections 2024-04-25T00:26:03.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.106+0000 7f7eee109640 1 --2- 172.21.15.27:0/4151066733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ee806a780 0x7f7ee810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.106+0000 7f7eee109640 1 --2- 172.21.15.27:0/4151066733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee8069dc0 0x7f7ee806a240 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.106+0000 7f7eee109640 1 --2- 172.21.15.27:0/4151066733 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ee8107730 0x7f7ee8107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.106+0000 7f7eee109640 1 -- 172.21.15.27:0/4151066733 >> 172.21.15.27:0/4151066733 conn(0x7f7ee80776a0 msgr2=0x7f7ee8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.106+0000 7f7eee109640 1 -- 172.21.15.27:0/4151066733 shutdown_connections 2024-04-25T00:26:03.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.106+0000 7f7eee109640 1 -- 172.21.15.27:0/4151066733 wait complete. 2024-04-25T00:26:03.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.107+0000 7f7eee109640 1 Processor -- start 2024-04-25T00:26:03.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.107+0000 7f7eee109640 1 -- start start 2024-04-25T00:26:03.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.107+0000 7f7eee109640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee8069dc0 0x7f7ee8198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.107+0000 7f7eee109640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ee806a780 0x7f7ee81985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.107+0000 7f7eee109640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ee8107730 0x7f7ee819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.107+0000 7f7eee109640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee8111b40 con 0x7f7ee8069dc0 2024-04-25T00:26:03.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.107+0000 7f7eee109640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee81119c0 con 0x7f7ee8107730 2024-04-25T00:26:03.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.107+0000 7f7eee109640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee8111cc0 con 0x7f7ee806a780 2024-04-25T00:26:03.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.107+0000 7f7ee77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee8069dc0 0x7f7ee8198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.107+0000 7f7ee77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee8069dc0 0x7f7ee8198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54382/0 (socket says 172.21.15.27:54382) 2024-04-25T00:26:03.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.107+0000 7f7ee77fe640 1 -- 172.21.15.27:0/1423941565 learned_addr learned my addr 172.21.15.27:0/1423941565 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.108+0000 7f7ee7fff640 1 --2- 172.21.15.27:0/1423941565 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ee8107730 0x7f7ee819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.108+0000 7f7ee6ffd640 1 --2- 172.21.15.27:0/1423941565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ee806a780 0x7f7ee81985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.108+0000 7f7ee77fe640 1 -- 172.21.15.27:0/1423941565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ee806a780 msgr2=0x7f7ee81985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.108+0000 7f7ee77fe640 1 --2- 172.21.15.27:0/1423941565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ee806a780 0x7f7ee81985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.108+0000 7f7ee77fe640 1 -- 172.21.15.27:0/1423941565 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ee8107730 msgr2=0x7f7ee819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.108+0000 7f7ee77fe640 1 --2- 172.21.15.27:0/1423941565 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ee8107730 0x7f7ee819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.108+0000 7f7ee77fe640 1 -- 172.21.15.27:0/1423941565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee819d000 con 0x7f7ee8069dc0 2024-04-25T00:26:03.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.108+0000 7f7ee7fff640 1 --2- 172.21.15.27:0/1423941565 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ee8107730 0x7f7ee819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:26:03.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.108+0000 7f7ee77fe640 1 --2- 172.21.15.27:0/1423941565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee8069dc0 0x7f7ee8198060 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800d950 tx=0x7f7ed800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.108+0000 7f7ee4ff9640 1 -- 172.21.15.27:0/1423941565 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8017070 con 0x7f7ee8069dc0 2024-04-25T00:26:03.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.108+0000 7f7ee4ff9640 1 -- 172.21.15.27:0/1423941565 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ed8009c30 con 0x7f7ee8069dc0 2024-04-25T00:26:03.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.108+0000 7f7eee109640 1 -- 172.21.15.27:0/1423941565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ee819d2f0 con 0x7f7ee8069dc0 2024-04-25T00:26:03.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.108+0000 7f7ee4ff9640 1 -- 172.21.15.27:0/1423941565 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed800f6b0 con 0x7f7ee8069dc0 2024-04-25T00:26:03.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.108+0000 7f7eee109640 1 -- 172.21.15.27:0/1423941565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ee81a4b80 con 0x7f7ee8069dc0 2024-04-25T00:26:03.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.110+0000 7f7ee4ff9640 1 -- 172.21.15.27:0/1423941565 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7ed80109d0 con 0x7f7ee8069dc0 2024-04-25T00:26:03.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ef0e0b740 1 Processor -- start 2024-04-25T00:26:03.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ef0e0b740 1 -- start start 2024-04-25T00:26:03.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ee4ff9640 1 --2- 172.21.15.27:0/1423941565 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7ed00a2d80 0x7f7ed00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ef0e0b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca17690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ef0e0b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca17c60 0x5619fca26440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ef0e0b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca26980 0x5619fca28d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ef0e0b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5619fc97c670 con 0x5619fca172b0 2024-04-25T00:26:03.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ef0e0b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5619fc924970 con 0x5619fca26980 2024-04-25T00:26:03.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ef0e0b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5619fc978630 con 0x5619fca17c60 2024-04-25T00:26:03.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ee7fff640 1 --2- 172.21.15.27:0/1423941565 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7ed00a2d80 0x7f7ed00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ee4ff9640 1 -- 172.21.15.27:0/1423941565 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f7ed80cf390 con 0x7f7ee8069dc0 2024-04-25T00:26:03.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ee77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca17690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ee77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca17690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54388/0 (socket says 172.21.15.27:54388) 2024-04-25T00:26:03.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ee77fe640 1 -- 172.21.15.27:0/1877209692 learned_addr learned my addr 172.21.15.27:0/1877209692 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ee6ffd640 1 --2- 172.21.15.27:0/1877209692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca17c60 0x5619fca26440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ee77fe640 1 -- 172.21.15.27:0/1877209692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca17c60 msgr2=0x5619fca26440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ee77fe640 1 --2- 172.21.15.27:0/1877209692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca17c60 0x5619fca26440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ee77fe640 1 -- 172.21.15.27:0/1877209692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca26980 msgr2=0x5619fca28d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ee77fe640 1 --2- 172.21.15.27:0/1877209692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca26980 0x5619fca28d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.111+0000 7f7ee77fe640 1 -- 172.21.15.27:0/1877209692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619fc948400 con 0x5619fca172b0 2024-04-25T00:26:03.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.112+0000 7f7ee7fff640 1 --2- 172.21.15.27:0/1423941565 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7ed00a2d80 0x7f7ed00a5240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0005fd0 tx=0x7f7ee0009b40 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.112+0000 7f7ee77fe640 1 --2- 172.21.15.27:0/1877209692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca17690 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80a3c80 tx=0x7f7ed8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9b275235fb0fe4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.112+0000 7f7ecdffb640 1 -- 172.21.15.27:0/1877209692 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8020070 con 0x5619fca172b0 2024-04-25T00:26:03.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.112+0000 7f7ecdffb640 1 -- 172.21.15.27:0/1877209692 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ed80bb040 con 0x5619fca172b0 2024-04-25T00:26:03.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.112+0000 7f7ecdffb640 1 -- 172.21.15.27:0/1877209692 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8097bb0 con 0x5619fca172b0 2024-04-25T00:26:03.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.112+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/1877209692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 msgr2=0x5619fca17690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.112+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/1877209692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca17690 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80a3c80 tx=0x7f7ed8005880 comp rx=0 tx=0).stop 2024-04-25T00:26:03.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.112+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/1877209692 shutdown_connections 2024-04-25T00:26:03.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.112+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/1877209692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca26980 0x5619fca28d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.112+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/1877209692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca17c60 0x5619fca26440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.112+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/1877209692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca17690 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.112+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/1877209692 >> 172.21.15.27:0/1877209692 conn(0x5619fca138a0 msgr2=0x5619fca13c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.112+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/1877209692 shutdown_connections 2024-04-25T00:26:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.112+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/1877209692 wait complete. 2024-04-25T00:26:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.113+0000 7f7ef0e0b740 1 -- start start 2024-04-25T00:26:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.113+0000 7f7ef0e0b740 1 Processor -- start 2024-04-25T00:26:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.113+0000 7f7ef0e0b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca262e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.113+0000 7f7ef0e0b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca17c60 0x5619fca235d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.113+0000 7f7ef0e0b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca26980 0x5619fca23b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.113+0000 7f7ef0e0b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5619fca296e0 con 0x5619fca172b0 2024-04-25T00:26:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.113+0000 7f7ef0e0b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5619fc938050 con 0x5619fca17c60 2024-04-25T00:26:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.113+0000 7f7ef0e0b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5619fca29860 con 0x5619fca26980 2024-04-25T00:26:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.113+0000 7f7ef0e0b740 10 client.0 _ll_register_callbacks cb 0x5619fc68fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:26:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.113+0000 7f7ee77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca262e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.113+0000 7f7ee77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca262e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54394/0 (socket says 172.21.15.27:54394) 2024-04-25T00:26:03.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.113+0000 7f7ee77fe640 1 -- 172.21.15.27:0/1374579481 learned_addr learned my addr 172.21.15.27:0/1374579481 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.114+0000 7f7ee7fff640 1 --2- 172.21.15.27:0/1374579481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca26980 0x5619fca23b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.114+0000 7f7ee6ffd640 1 --2- 172.21.15.27:0/1374579481 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca17c60 0x5619fca235d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.114+0000 7f7ee77fe640 1 -- 172.21.15.27:0/1374579481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca26980 msgr2=0x5619fca23b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.114+0000 7f7ee77fe640 1 --2- 172.21.15.27:0/1374579481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca26980 0x5619fca23b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.114+0000 7f7ee77fe640 1 -- 172.21.15.27:0/1374579481 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca17c60 msgr2=0x5619fca235d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.114+0000 7f7ee77fe640 1 --2- 172.21.15.27:0/1374579481 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca17c60 0x5619fca235d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.114+0000 7f7ee77fe640 1 -- 172.21.15.27:0/1374579481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619fc909dc0 con 0x5619fca172b0 2024-04-25T00:26:03.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.114+0000 7f7ee6ffd640 1 --2- 172.21.15.27:0/1374579481 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca17c60 0x5619fca235d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:26:03.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.114+0000 7f7ee77fe640 1 --2- 172.21.15.27:0/1374579481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca262e0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8012040 tx=0x7f7ed8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22150aaa9e8afb0b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.114+0000 7f7ec6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:26:03.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.114+0000 7f7ec6ffd640 1 -- 172.21.15.27:0/1374579481 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8020040 con 0x5619fca172b0 2024-04-25T00:26:03.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.114+0000 7f7ec6ffd640 1 -- 172.21.15.27:0/1374579481 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f7ed80cc9f0 con 0x5619fca172b0 2024-04-25T00:26:03.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.114+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/1374579481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ed80bb040 con 0x5619fca172b0 2024-04-25T00:26:03.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.114+0000 7f7ec6ffd640 1 -- 172.21.15.27:0/1374579481 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f7ed8099920 con 0x5619fca172b0 2024-04-25T00:26:03.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.114+0000 7f7ef0e0b740 10 client.17331 fetch_fsmap learned FSMap version 101 2024-04-25T00:26:03.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.114+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/1374579481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x5619fc6903f0 con 0x5619fca172b0 2024-04-25T00:26:03.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.114+0000 7f7ec6ffd640 1 -- 172.21.15.27:0/1374579481 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ed80bb040 con 0x5619fca172b0 2024-04-25T00:26:03.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.115+0000 7f7ec6ffd640 1 -- 172.21.15.27:0/1374579481 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 101) ==== 8362+0+0 (secure 0 0 0) 0x7f7ed809a990 con 0x5619fca172b0 2024-04-25T00:26:03.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.115+0000 7f7ef0e0b740 10 client.17331 fetch_fsmap finished waiting for FSMap version 101 2024-04-25T00:26:03.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.115+0000 7f7ef0e0b740 10 client.17331 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:26:03.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.115+0000 7f7ef0e0b740 20 client.17331 populate_metadata read hostname 'smithi027' 2024-04-25T00:26:03.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.115+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/1374579481 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x5619fca36810 0x5619fca38c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.115+0000 7f7ef0e0b740 4 client.17331 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:26:03.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.115+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/1374579481 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5619fc989bb0 con 0x5619fca36810 2024-04-25T00:26:03.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.115+0000 7f7ee6ffd640 1 --2- 172.21.15.27:0/1374579481 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x5619fca36810 0x5619fca38c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.115+0000 7f7ee6ffd640 1 --2- 172.21.15.27:0/1374579481 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x5619fca36810 0x5619fca38c00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b986d0e99de2ed23 server_cookie=7757a26457702136 in_seq=0 out_seq=0 2024-04-25T00:26:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.115+0000 7f7ec6ffd640 10 client.17331 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:26:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.116+0000 7f7ec6ffd640 1 -- 172.21.15.27:0/1374579481 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5619fc989bb0 con 0x5619fca36810 2024-04-25T00:26:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.116+0000 7f7ec6ffd640 10 client.17331 handle_command_reply: tid=0 2024-04-25T00:26:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.116+0000 7f7ef0e0b740 1 client.17331 shutdown 2024-04-25T00:26:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.116+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/1374579481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 msgr2=0x5619fca262e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.116+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/1374579481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca262e0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8012040 tx=0x7f7ed8000bc0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.116+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/1374579481 shutdown_connections 2024-04-25T00:26:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.116+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/1374579481 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x5619fca36810 0x5619fca38c00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.116+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/1374579481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca26980 0x5619fca23b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.116+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/1374579481 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca17c60 0x5619fca235d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.116+0000 7f7ec6ffd640 0 client.17331 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:26:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.116+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/1374579481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca262e0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.116+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/1374579481 >> 172.21.15.27:0/1374579481 conn(0x5619fca138a0 msgr2=0x5619fca19710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.116+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/1374579481 shutdown_connections 2024-04-25T00:26:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.117+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/1374579481 wait complete. 2024-04-25T00:26:03.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.117+0000 7f7ef0e0b740 20 client.17331 trim_cache size 0 max 16384 2024-04-25T00:26:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.130+0000 7f7ef0e0b740 1 Processor -- start 2024-04-25T00:26:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.130+0000 7f7ef0e0b740 1 -- start start 2024-04-25T00:26:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.130+0000 7f7ef0e0b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca172b0 0x5619fca25e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.130+0000 7f7ef0e0b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca17c60 0x5619fca3bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.130+0000 7f7ef0e0b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca26980 0x5619fca3c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.130+0000 7f7ef0e0b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee81ac9f0 con 0x5619fca17c60 2024-04-25T00:26:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.130+0000 7f7ef0e0b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee81ae530 con 0x5619fca172b0 2024-04-25T00:26:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.130+0000 7f7ef0e0b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee81ad0c0 con 0x5619fca26980 2024-04-25T00:26:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.130+0000 7f7ee6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca17c60 0x5619fca3bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.130+0000 7f7ee6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca17c60 0x5619fca3bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54400/0 (socket says 172.21.15.27:54400) 2024-04-25T00:26:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.130+0000 7f7ee6ffd640 1 -- 172.21.15.27:0/3659988145 learned_addr learned my addr 172.21.15.27:0/3659988145 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.130+0000 7f7ee6ffd640 1 -- 172.21.15.27:0/3659988145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca26980 msgr2=0x5619fca3c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.130+0000 7f7ee6ffd640 1 --2- 172.21.15.27:0/3659988145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca26980 0x5619fca3c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.130+0000 7f7ee6ffd640 1 -- 172.21.15.27:0/3659988145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca172b0 msgr2=0x5619fca25e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.130+0000 7f7ee6ffd640 1 --2- 172.21.15.27:0/3659988145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca172b0 0x5619fca25e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.130+0000 7f7ee6ffd640 1 -- 172.21.15.27:0/3659988145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619fc75a7d0 con 0x5619fca17c60 2024-04-25T00:26:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.130+0000 7f7ee6ffd640 1 --2- 172.21.15.27:0/3659988145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca17c60 0x5619fca3bc20 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f7edc000c00 tx=0x7f7edc005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d571530ab0440d9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.131+0000 7f7ecd7fa640 1 -- 172.21.15.27:0/3659988145 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc035040 con 0x5619fca17c60 2024-04-25T00:26:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.131+0000 7f7ecd7fa640 1 -- 172.21.15.27:0/3659988145 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7edc047070 con 0x5619fca17c60 2024-04-25T00:26:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.131+0000 7f7ecd7fa640 1 -- 172.21.15.27:0/3659988145 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc005500 con 0x5619fca17c60 2024-04-25T00:26:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.131+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/3659988145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca17c60 msgr2=0x5619fca3bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.131+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/3659988145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca17c60 0x5619fca3bc20 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f7edc000c00 tx=0x7f7edc005a20 comp rx=0 tx=0).stop 2024-04-25T00:26:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.131+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/3659988145 shutdown_connections 2024-04-25T00:26:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.131+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/3659988145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca26980 0x5619fca3c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.131+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/3659988145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca17c60 0x5619fca3bc20 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.131+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/3659988145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca172b0 0x5619fca25e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.131+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/3659988145 >> 172.21.15.27:0/3659988145 conn(0x5619fca138a0 msgr2=0x5619fca1ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.131+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/3659988145 shutdown_connections 2024-04-25T00:26:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.131+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/3659988145 wait complete. 2024-04-25T00:26:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.132+0000 7f7ef0e0b740 1 -- start start 2024-04-25T00:26:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.132+0000 7f7ef0e0b740 1 Processor -- start 2024-04-25T00:26:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.132+0000 7f7ef0e0b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca3c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.132+0000 7f7ef0e0b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca17c60 0x5619fca39510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.132+0000 7f7ef0e0b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca26980 0x5619fca39c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.132+0000 7f7ef0e0b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee81ade60 con 0x5619fca172b0 2024-04-25T00:26:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.132+0000 7f7ef0e0b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee81ad790 con 0x5619fca26980 2024-04-25T00:26:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.132+0000 7f7ef0e0b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee81aa9b0 con 0x5619fca17c60 2024-04-25T00:26:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.132+0000 7f7ef0e0b740 10 client.0 _ll_register_callbacks cb 0x7f7ed80cf070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:26:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.132+0000 7f7ee77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca3c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.132+0000 7f7ee77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca3c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54416/0 (socket says 172.21.15.27:54416) 2024-04-25T00:26:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.132+0000 7f7ee77fe640 1 -- 172.21.15.27:0/4293023945 learned_addr learned my addr 172.21.15.27:0/4293023945 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.132+0000 7f7ee7fff640 1 --2- 172.21.15.27:0/4293023945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca26980 0x5619fca39c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.132+0000 7f7ee6ffd640 1 --2- 172.21.15.27:0/4293023945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca17c60 0x5619fca39510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ee77fe640 1 -- 172.21.15.27:0/4293023945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca17c60 msgr2=0x5619fca39510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ee77fe640 1 --2- 172.21.15.27:0/4293023945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca17c60 0x5619fca39510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ee77fe640 1 -- 172.21.15.27:0/4293023945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca26980 msgr2=0x5619fca39c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ee77fe640 1 --2- 172.21.15.27:0/4293023945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca26980 0x5619fca39c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ee77fe640 1 -- 172.21.15.27:0/4293023945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619fca9b9f0 con 0x5619fca172b0 2024-04-25T00:26:03.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ee6ffd640 1 --2- 172.21.15.27:0/4293023945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca17c60 0x5619fca39510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:26:03.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ee77fe640 1 --2- 172.21.15.27:0/4293023945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca3c130 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009610 tx=0x7f7ed807e0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1501ce8e7f38f7e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ec6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:26:03.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ec6ffd640 1 -- 172.21.15.27:0/4293023945 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8020030 con 0x5619fca172b0 2024-04-25T00:26:03.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ec6ffd640 1 -- 172.21.15.27:0/4293023945 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f7ed8097290 con 0x5619fca172b0 2024-04-25T00:26:03.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/4293023945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7edc047070 con 0x5619fca172b0 2024-04-25T00:26:03.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ec6ffd640 1 -- 172.21.15.27:0/4293023945 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f7ed809ab90 con 0x5619fca172b0 2024-04-25T00:26:03.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ef0e0b740 10 client.17337 fetch_fsmap learned FSMap version 101 2024-04-25T00:26:03.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/4293023945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x5619fca9cb90 con 0x5619fca172b0 2024-04-25T00:26:03.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ec6ffd640 1 -- 172.21.15.27:0/4293023945 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7edc047070 con 0x5619fca172b0 2024-04-25T00:26:03.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ec6ffd640 1 -- 172.21.15.27:0/4293023945 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 101) ==== 8362+0+0 (secure 0 0 0) 0x7f7ed8010460 con 0x5619fca172b0 2024-04-25T00:26:03.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ef0e0b740 10 client.17337 fetch_fsmap finished waiting for FSMap version 101 2024-04-25T00:26:03.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ef0e0b740 10 client.17337 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-25T00:26:03.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ef0e0b740 20 client.17337 populate_metadata read hostname 'smithi027' 2024-04-25T00:26:03.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/4293023945 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x5619fca9d070 0x5619fca9f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ef0e0b740 4 client.17337 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:26:03.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.133+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/4293023945 --> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5619fc983f00 con 0x5619fca9d070 2024-04-25T00:26:03.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.134+0000 7f7ee6ffd640 1 --2- 172.21.15.27:0/4293023945 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x5619fca9d070 0x5619fca9f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.134+0000 7f7ee6ffd640 1 --2- 172.21.15.27:0/4293023945 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x5619fca9d070 0x5619fca9f460 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d2956f15570d8536 server_cookie=70e402364f0c434c in_seq=0 out_seq=0 2024-04-25T00:26:03.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.134+0000 7f7ec6ffd640 10 client.17337 ms_handle_connect on v2:172.21.15.153:6832/2700717364 2024-04-25T00:26:03.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.134+0000 7f7ec6ffd640 1 -- 172.21.15.27:0/4293023945 <== mds.3 v2:172.21.15.153:6832/2700717364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5619fc983f00 con 0x5619fca9d070 2024-04-25T00:26:03.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.134+0000 7f7ec6ffd640 10 client.17337 handle_command_reply: tid=0 2024-04-25T00:26:03.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.134+0000 7f7ef0e0b740 1 client.17337 shutdown 2024-04-25T00:26:03.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.134+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/4293023945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 msgr2=0x5619fca3c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.134+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/4293023945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca3c130 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009610 tx=0x7f7ed807e0d0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/4293023945 shutdown_connections 2024-04-25T00:26:03.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/4293023945 >> [v2:172.21.15.153:6832/2700717364,v1:172.21.15.153:6833/2700717364] conn(0x5619fca9d070 0x5619fca9f460 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/4293023945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5619fca26980 0x5619fca39c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7ec6ffd640 0 client.17337 ms_handle_reset on v2:172.21.15.153:6832/2700717364 2024-04-25T00:26:03.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/4293023945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5619fca17c60 0x5619fca39510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7ef0e0b740 1 --2- 172.21.15.27:0/4293023945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5619fca172b0 0x5619fca3c130 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/4293023945 >> 172.21.15.27:0/4293023945 conn(0x5619fca138a0 msgr2=0x5619fca31af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/4293023945 shutdown_connections 2024-04-25T00:26:03.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7ef0e0b740 1 -- 172.21.15.27:0/4293023945 wait complete. 2024-04-25T00:26:03.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7ef0e0b740 20 client.17337 trim_cache size 0 max 16384 2024-04-25T00:26:03.142 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:26:03.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7eee109640 1 -- 172.21.15.27:0/1423941565 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7ed00a2d80 msgr2=0x7f7ed00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7eee109640 1 --2- 172.21.15.27:0/1423941565 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7ed00a2d80 0x7f7ed00a5240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0005fd0 tx=0x7f7ee0009b40 comp rx=0 tx=0).stop 2024-04-25T00:26:03.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7eee109640 1 -- 172.21.15.27:0/1423941565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee8069dc0 msgr2=0x7f7ee8198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7eee109640 1 --2- 172.21.15.27:0/1423941565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee8069dc0 0x7f7ee8198060 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800d950 tx=0x7f7ed800de10 comp rx=0 tx=0).stop 2024-04-25T00:26:03.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7eee109640 1 -- 172.21.15.27:0/1423941565 shutdown_connections 2024-04-25T00:26:03.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7eee109640 1 --2- 172.21.15.27:0/1423941565 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7ed00a2d80 0x7f7ed00a5240 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7eee109640 1 --2- 172.21.15.27:0/1423941565 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7ee8107730 0x7f7ee819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7eee109640 1 --2- 172.21.15.27:0/1423941565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ee806a780 0x7f7ee81985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7eee109640 1 --2- 172.21.15.27:0/1423941565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee8069dc0 0x7f7ee8198060 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7eee109640 1 -- 172.21.15.27:0/1423941565 >> 172.21.15.27:0/1423941565 conn(0x7f7ee80776a0 msgr2=0x7f7ee80fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7eee109640 1 -- 172.21.15.27:0/1423941565 shutdown_connections 2024-04-25T00:26:03.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.135+0000 7f7eee109640 1 -- 172.21.15.27:0/1423941565 wait complete. 2024-04-25T00:26:03.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:02 smithi027 ceph-mon[29208]: pgmap v311: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-25T00:26:03.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:02 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2411839499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:26:03.148 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T00:26:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.250+0000 7f7f494ae640 1 -- 172.21.15.27:0/1862148066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f4410bd00 msgr2=0x7f7f4410e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.250+0000 7f7f494ae640 1 --2- 172.21.15.27:0/1862148066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f4410bd00 0x7f7f4410e1a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c009ca0 tx=0x7f7f3c02a9e0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.250+0000 7f7f494ae640 1 -- 172.21.15.27:0/1862148066 shutdown_connections 2024-04-25T00:26:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.250+0000 7f7f494ae640 1 --2- 172.21.15.27:0/1862148066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f4410bd00 0x7f7f4410e1a0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.250+0000 7f7f494ae640 1 --2- 172.21.15.27:0/1862148066 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f441032e0 0x7f7f4410b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.250+0000 7f7f494ae640 1 --2- 172.21.15.27:0/1862148066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f441029c0 0x7f7f44102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.250+0000 7f7f494ae640 1 -- 172.21.15.27:0/1862148066 >> 172.21.15.27:0/1862148066 conn(0x7f7f44077740 msgr2=0x7f7f44077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.250+0000 7f7f494ae640 1 -- 172.21.15.27:0/1862148066 shutdown_connections 2024-04-25T00:26:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.250+0000 7f7f494ae640 1 -- 172.21.15.27:0/1862148066 wait complete. 2024-04-25T00:26:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.251+0000 7f7f494ae640 1 Processor -- start 2024-04-25T00:26:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.251+0000 7f7f494ae640 1 -- start start 2024-04-25T00:26:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.251+0000 7f7f494ae640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f441029c0 0x7f7f44198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.251+0000 7f7f494ae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f441032e0 0x7f7f44198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.251+0000 7f7f494ae640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f4410bd00 0x7f7f4419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.251+0000 7f7f494ae640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f44111b00 con 0x7f7f441032e0 2024-04-25T00:26:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.251+0000 7f7f494ae640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7f44111980 con 0x7f7f441029c0 2024-04-25T00:26:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.251+0000 7f7f494ae640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f44111c80 con 0x7f7f4410bd00 2024-04-25T00:26:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.251+0000 7f7f427fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f441032e0 0x7f7f44198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.251+0000 7f7f427fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f441032e0 0x7f7f44198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54434/0 (socket says 172.21.15.27:54434) 2024-04-25T00:26:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.251+0000 7f7f427fc640 1 -- 172.21.15.27:0/1875993086 learned_addr learned my addr 172.21.15.27:0/1875993086 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.252+0000 7f7f42ffd640 1 --2- 172.21.15.27:0/1875993086 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f441029c0 0x7f7f44198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.252+0000 7f7f437fe640 1 --2- 172.21.15.27:0/1875993086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f4410bd00 0x7f7f4419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.252+0000 7f7f427fc640 1 -- 172.21.15.27:0/1875993086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f4410bd00 msgr2=0x7f7f4419c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.252+0000 7f7f427fc640 1 --2- 172.21.15.27:0/1875993086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f4410bd00 0x7f7f4419c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.252+0000 7f7f427fc640 1 -- 172.21.15.27:0/1875993086 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f441029c0 msgr2=0x7f7f44198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.252+0000 7f7f427fc640 1 --2- 172.21.15.27:0/1875993086 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f441029c0 0x7f7f44198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.252+0000 7f7f427fc640 1 -- 172.21.15.27:0/1875993086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f4419cfc0 con 0x7f7f441032e0 2024-04-25T00:26:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.252+0000 7f7f437fe640 1 --2- 172.21.15.27:0/1875993086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f4410bd00 0x7f7f4419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:26:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.252+0000 7f7f427fc640 1 --2- 172.21.15.27:0/1875993086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f441032e0 0x7f7f44198560 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400d950 tx=0x7f7f3400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.252+0000 7f7f2bfff640 1 -- 172.21.15.27:0/1875993086 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f34017030 con 0x7f7f441032e0 2024-04-25T00:26:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.252+0000 7f7f494ae640 1 -- 172.21.15.27:0/1875993086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f4419d2b0 con 0x7f7f441032e0 2024-04-25T00:26:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.252+0000 7f7f2bfff640 1 -- 172.21.15.27:0/1875993086 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f34004020 con 0x7f7f441032e0 2024-04-25T00:26:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.252+0000 7f7f2bfff640 1 -- 172.21.15.27:0/1875993086 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f340047a0 con 0x7f7f441032e0 2024-04-25T00:26:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.252+0000 7f7f494ae640 1 -- 172.21.15.27:0/1875993086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f441a4b40 con 0x7f7f441032e0 2024-04-25T00:26:03.257 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:02 smithi139 ceph-mon[32833]: pgmap v311: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-25T00:26:03.257 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:02 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2411839499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:26:03.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.254+0000 7f7f2bfff640 1 -- 172.21.15.27:0/1875993086 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7f340041d0 con 0x7f7f441032e0 2024-04-25T00:26:03.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.255+0000 7f7f4c1b6740 1 Processor -- start 2024-04-25T00:26:03.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.255+0000 7f7f4c1b6740 1 -- start start 2024-04-25T00:26:03.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.255+0000 7f7f4c1b6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593b5480 0x55f1593b5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.255+0000 7f7f4c1b6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5da0 0x55f1593c4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.255+0000 7f7f4c1b6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593c4980 0x55f1593c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.255+0000 7f7f4c1b6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f15931a670 con 0x55f1593c4980 2024-04-25T00:26:03.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.255+0000 7f7f4c1b6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f1592c2970 con 0x55f1593b5480 2024-04-25T00:26:03.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.255+0000 7f7f4c1b6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f159316630 con 0x55f1593b5da0 2024-04-25T00:26:03.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.255+0000 7f7f2bfff640 1 --2- 172.21.15.27:0/1875993086 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7f2c0a2d00 0x7f7f2c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.255+0000 7f7f427fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593c4980 0x55f1593c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.255+0000 7f7f427fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593c4980 0x55f1593c6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54440/0 (socket says 172.21.15.27:54440) 2024-04-25T00:26:03.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.255+0000 7f7f427fc640 1 -- 172.21.15.27:0/833881085 learned_addr learned my addr 172.21.15.27:0/833881085 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.255+0000 7f7f437fe640 1 --2- 172.21.15.27:0/1875993086 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7f2c0a2d00 0x7f7f2c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.255+0000 7f7f42ffd640 1 --2- 172.21.15.27:0/833881085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5da0 0x55f1593c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.255+0000 7f7f437fe640 1 --2- 172.21.15.27:0/833881085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593b5480 0x55f1593b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.255+0000 7f7f2bfff640 1 -- 172.21.15.27:0/1875993086 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f7f340cf1f0 con 0x7f7f441032e0 2024-04-25T00:26:03.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.256+0000 7f7f427fc640 1 -- 172.21.15.27:0/833881085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5da0 msgr2=0x55f1593c4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.256+0000 7f7f427fc640 1 --2- 172.21.15.27:0/833881085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5da0 0x55f1593c4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.256+0000 7f7f427fc640 1 -- 172.21.15.27:0/833881085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593b5480 msgr2=0x55f1593b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.256+0000 7f7f427fc640 1 --2- 172.21.15.27:0/833881085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593b5480 0x55f1593b5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.256+0000 7f7f427fc640 1 -- 172.21.15.27:0/833881085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1592e6400 con 0x55f1593c4980 2024-04-25T00:26:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.256+0000 7f7f437fe640 1 --2- 172.21.15.27:0/833881085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593b5480 0x55f1593b5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:26:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.256+0000 7f7f427fc640 1 --2- 172.21.15.27:0/833881085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593c4980 0x55f1593c6d70 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f7f34017000 tx=0x7f7f34021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50fc05c85fd09f20 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.256+0000 7f7f437fe640 1 --2- 172.21.15.27:0/1875993086 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7f2c0a2d00 0x7f7f2c0a51c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c009a60 tx=0x7f7f3c004a40 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.256+0000 7f7f297fa640 1 -- 172.21.15.27:0/833881085 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3409baa0 con 0x55f1593c4980 2024-04-25T00:26:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.256+0000 7f7f297fa640 1 -- 172.21.15.27:0/833881085 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f3409b3a0 con 0x55f1593c4980 2024-04-25T00:26:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.256+0000 7f7f297fa640 1 -- 172.21.15.27:0/833881085 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3409b600 con 0x55f1593c4980 2024-04-25T00:26:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/833881085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593c4980 msgr2=0x55f1593c6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/833881085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593c4980 0x55f1593c6d70 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f7f34017000 tx=0x7f7f34021000 comp rx=0 tx=0).stop 2024-04-25T00:26:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/833881085 shutdown_connections 2024-04-25T00:26:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/833881085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593c4980 0x55f1593c6d70 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/833881085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5da0 0x55f1593c4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/833881085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593b5480 0x55f1593b5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/833881085 >> 172.21.15.27:0/833881085 conn(0x55f1593b18a0 msgr2=0x55f1593b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/833881085 shutdown_connections 2024-04-25T00:26:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/833881085 wait complete. 2024-04-25T00:26:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 1 -- start start 2024-04-25T00:26:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 1 Processor -- start 2024-04-25T00:26:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5480 0x55f1593c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593b5da0 0x55f1593c1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593c4980 0x55f1593c1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f1593c76e0 con 0x55f1593c4980 2024-04-25T00:26:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f1592d6050 con 0x55f1593b5da0 2024-04-25T00:26:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f1593c7860 con 0x55f1593b5480 2024-04-25T00:26:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f4c1b6740 10 client.0 _ll_register_callbacks cb 0x55f15902dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:26:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f437fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593c4980 0x55f1593c1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f42ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5480 0x55f1593c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f437fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593c4980 0x55f1593c1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54448/0 (socket says 172.21.15.27:54448) 2024-04-25T00:26:03.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.257+0000 7f7f437fe640 1 -- 172.21.15.27:0/794863070 learned_addr learned my addr 172.21.15.27:0/794863070 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.258+0000 7f7f427fc640 1 --2- 172.21.15.27:0/794863070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593b5da0 0x55f1593c1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.258+0000 7f7f437fe640 1 -- 172.21.15.27:0/794863070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5480 msgr2=0x55f1593c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.258+0000 7f7f437fe640 1 --2- 172.21.15.27:0/794863070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5480 0x55f1593c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.258+0000 7f7f437fe640 1 -- 172.21.15.27:0/794863070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593b5da0 msgr2=0x55f1593c1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.258+0000 7f7f437fe640 1 --2- 172.21.15.27:0/794863070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593b5da0 0x55f1593c1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.258+0000 7f7f437fe640 1 -- 172.21.15.27:0/794863070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1592a7dc0 con 0x55f1593c4980 2024-04-25T00:26:03.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.258+0000 7f7f437fe640 1 --2- 172.21.15.27:0/794863070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593c4980 0x55f1593c1ad0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c036360 tx=0x7f7f3c02a9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e699c82d5a1fd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.258+0000 7f7f26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:26:03.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.258+0000 7f7f26ff5640 1 -- 172.21.15.27:0/794863070 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c033050 con 0x55f1593c4980 2024-04-25T00:26:03.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.258+0000 7f7f26ff5640 1 -- 172.21.15.27:0/794863070 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f7f3c02fbc0 con 0x55f1593c4980 2024-04-25T00:26:03.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.258+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/794863070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f3409b3a0 con 0x55f1593c4980 2024-04-25T00:26:03.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.258+0000 7f7f26ff5640 1 -- 172.21.15.27:0/794863070 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f7f3c041040 con 0x55f1593c4980 2024-04-25T00:26:03.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.258+0000 7f7f4c1b6740 10 client.17349 fetch_fsmap learned FSMap version 101 2024-04-25T00:26:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.258+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/794863070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55f15902e3f0 con 0x55f1593c4980 2024-04-25T00:26:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.258+0000 7f7f26ff5640 1 -- 172.21.15.27:0/794863070 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f3409b3a0 con 0x55f1593c4980 2024-04-25T00:26:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.259+0000 7f7f26ff5640 1 -- 172.21.15.27:0/794863070 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 101) ==== 8362+0+0 (secure 0 0 0) 0x7f7f3c0365d0 con 0x55f1593c4980 2024-04-25T00:26:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.259+0000 7f7f4c1b6740 10 client.17349 fetch_fsmap finished waiting for FSMap version 101 2024-04-25T00:26:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.259+0000 7f7f4c1b6740 10 client.17349 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:26:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.259+0000 7f7f4c1b6740 20 client.17349 populate_metadata read hostname 'smithi027' 2024-04-25T00:26:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.259+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/794863070 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55f1593d4900 0x55f1593d6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.259+0000 7f7f4c1b6740 4 client.17349 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:26:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.259+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/794863070 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f159327bb0 con 0x55f1593d4900 2024-04-25T00:26:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.259+0000 7f7f42ffd640 1 --2- 172.21.15.27:0/794863070 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55f1593d4900 0x55f1593d6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.259+0000 7f7f42ffd640 1 --2- 172.21.15.27:0/794863070 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55f1593d4900 0x55f1593d6d50 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e19ad6547272a9d4 server_cookie=e8d2bdc6240aee30 in_seq=0 out_seq=0 2024-04-25T00:26:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.259+0000 7f7f26ff5640 10 client.17349 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:26:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.260+0000 7f7f26ff5640 1 -- 172.21.15.27:0/794863070 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f159327bb0 con 0x55f1593d4900 2024-04-25T00:26:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.260+0000 7f7f26ff5640 10 client.17349 handle_command_reply: tid=0 2024-04-25T00:26:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.260+0000 7f7f4c1b6740 1 client.17349 shutdown 2024-04-25T00:26:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.260+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/794863070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593c4980 msgr2=0x55f1593c1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.260+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/794863070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593c4980 0x55f1593c1ad0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c036360 tx=0x7f7f3c02a9e0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.260+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/794863070 shutdown_connections 2024-04-25T00:26:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.260+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/794863070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593c4980 0x55f1593c1ad0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.260+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/794863070 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55f1593d4900 0x55f1593d6d50 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.260+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/794863070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593b5da0 0x55f1593c1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.260+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/794863070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5480 0x55f1593c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.260+0000 7f7f26ff5640 0 client.17349 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:26:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.260+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/794863070 >> 172.21.15.27:0/794863070 conn(0x55f1593b18a0 msgr2=0x55f1593b76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.261+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/794863070 shutdown_connections 2024-04-25T00:26:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.261+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/794863070 wait complete. 2024-04-25T00:26:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.261+0000 7f7f4c1b6740 20 client.17349 trim_cache size 0 max 16384 2024-04-25T00:26:03.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.274+0000 7f7f4c1b6740 1 Processor -- start 2024-04-25T00:26:03.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.274+0000 7f7f4c1b6740 1 -- start start 2024-04-25T00:26:03.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.274+0000 7f7f4c1b6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593b5480 0x55f1593c06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.274+0000 7f7f4c1b6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5da0 0x55f1593b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.274+0000 7f7f4c1b6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593c4980 0x55f1593c3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.274+0000 7f7f4c1b6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f441ac9a0 con 0x55f1593b5480 2024-04-25T00:26:03.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.274+0000 7f7f4c1b6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7f441ae4e0 con 0x55f1593c4980 2024-04-25T00:26:03.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.274+0000 7f7f4c1b6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f441ad070 con 0x55f1593b5da0 2024-04-25T00:26:03.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.274+0000 7f7f42ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593b5480 0x55f1593c06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.274+0000 7f7f42ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593b5480 0x55f1593c06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54460/0 (socket says 172.21.15.27:54460) 2024-04-25T00:26:03.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.274+0000 7f7f42ffd640 1 -- 172.21.15.27:0/1185904959 learned_addr learned my addr 172.21.15.27:0/1185904959 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.274+0000 7f7f42ffd640 1 -- 172.21.15.27:0/1185904959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5da0 msgr2=0x55f1593b09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.274+0000 7f7f42ffd640 1 --2- 172.21.15.27:0/1185904959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5da0 0x55f1593b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.274+0000 7f7f42ffd640 1 -- 172.21.15.27:0/1185904959 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593c4980 msgr2=0x55f1593c3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.274+0000 7f7f42ffd640 1 --2- 172.21.15.27:0/1185904959 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593c4980 0x55f1593c3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.274+0000 7f7f42ffd640 1 -- 172.21.15.27:0/1185904959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1590f87d0 con 0x55f1593b5480 2024-04-25T00:26:03.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.275+0000 7f7f42ffd640 1 --2- 172.21.15.27:0/1185904959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593b5480 0x55f1593c06f0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x55f1593d7570 tx=0x7f7f38013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cfea0ad2dd39a0c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.275+0000 7f7f28ff9640 1 -- 172.21.15.27:0/1185904959 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3801d430 con 0x55f1593b5480 2024-04-25T00:26:03.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.275+0000 7f7f28ff9640 1 -- 172.21.15.27:0/1185904959 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f38005020 con 0x55f1593b5480 2024-04-25T00:26:03.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.275+0000 7f7f28ff9640 1 -- 172.21.15.27:0/1185904959 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f38004bc0 con 0x55f1593b5480 2024-04-25T00:26:03.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:02 smithi153 ceph-mon[32896]: pgmap v311: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-25T00:26:03.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:02 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2411839499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:26:03.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.275+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/1185904959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593b5480 msgr2=0x55f1593c06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.275+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/1185904959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593b5480 0x55f1593c06f0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x55f1593d7570 tx=0x7f7f38013830 comp rx=0 tx=0).stop 2024-04-25T00:26:03.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.275+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/1185904959 shutdown_connections 2024-04-25T00:26:03.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.275+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/1185904959 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593c4980 0x55f1593c3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.275+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/1185904959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5da0 0x55f1593b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.275+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/1185904959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593b5480 0x55f1593c06f0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.275+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/1185904959 >> 172.21.15.27:0/1185904959 conn(0x55f1593b18a0 msgr2=0x55f1593b02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.276+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/1185904959 shutdown_connections 2024-04-25T00:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.276+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/1185904959 wait complete. 2024-04-25T00:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.276+0000 7f7f4c1b6740 1 -- start start 2024-04-25T00:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.276+0000 7f7f4c1b6740 1 Processor -- start 2024-04-25T00:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.276+0000 7f7f4c1b6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5480 0x55f1593ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.276+0000 7f7f4c1b6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593b5da0 0x55f1593bdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.276+0000 7f7f4c1b6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593c4980 0x55f1593be320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.276+0000 7f7f4c1b6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f441ade10 con 0x55f1593b5da0 2024-04-25T00:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.276+0000 7f7f4c1b6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7f441ad740 con 0x55f1593c4980 2024-04-25T00:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.276+0000 7f7f4c1b6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f441aa960 con 0x55f1593b5480 2024-04-25T00:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.276+0000 7f7f4c1b6740 10 client.0 _ll_register_callbacks cb 0x7f7f3c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.276+0000 7f7f427fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593b5da0 0x55f1593bdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.276+0000 7f7f427fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593b5da0 0x55f1593bdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54468/0 (socket says 172.21.15.27:54468) 2024-04-25T00:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.276+0000 7f7f427fc640 1 -- 172.21.15.27:0/3994010478 learned_addr learned my addr 172.21.15.27:0/3994010478 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.276+0000 7f7f42ffd640 1 --2- 172.21.15.27:0/3994010478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5480 0x55f1593ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f427fc640 1 -- 172.21.15.27:0/3994010478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5480 msgr2=0x55f1593ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f427fc640 1 --2- 172.21.15.27:0/3994010478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5480 0x55f1593ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f427fc640 1 -- 172.21.15.27:0/3994010478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593c4980 msgr2=0x55f1593be320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f427fc640 1 --2- 172.21.15.27:0/3994010478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593c4980 0x55f1593be320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f427fc640 1 -- 172.21.15.27:0/3994010478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1593be890 con 0x55f1593b5da0 2024-04-25T00:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f42ffd640 1 --2- 172.21.15.27:0/3994010478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5480 0x55f1593ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f427fc640 1 --2- 172.21.15.27:0/3994010478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593b5da0 0x55f1593bdde0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f7f34097800 tx=0x7f7f34097c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c4d6eb10e0859e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f26ff5640 1 -- 172.21.15.27:0/3994010478 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f340cca60 con 0x55f1593b5da0 2024-04-25T00:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f26ff5640 1 -- 172.21.15.27:0/3994010478 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f7f340b4120 con 0x55f1593b5da0 2024-04-25T00:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/3994010478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f38005020 con 0x55f1593b5da0 2024-04-25T00:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f26ff5640 1 -- 172.21.15.27:0/3994010478 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f7f3409b3a0 con 0x55f1593b5da0 2024-04-25T00:26:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f4c1b6740 10 client.17355 fetch_fsmap learned FSMap version 101 2024-04-25T00:26:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/3994010478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55f15943abc0 con 0x55f1593b5da0 2024-04-25T00:26:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f26ff5640 1 -- 172.21.15.27:0/3994010478 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f38005020 con 0x55f1593b5da0 2024-04-25T00:26:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f26ff5640 1 -- 172.21.15.27:0/3994010478 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 101) ==== 8362+0+0 (secure 0 0 0) 0x7f7f340a3b80 con 0x55f1593b5da0 2024-04-25T00:26:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f4c1b6740 10 client.17355 fetch_fsmap finished waiting for FSMap version 101 2024-04-25T00:26:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f4c1b6740 10 client.17355 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T00:26:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f4c1b6740 20 client.17355 populate_metadata read hostname 'smithi027' 2024-04-25T00:26:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/3994010478 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55f15943af50 0x55f15943d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f4c1b6740 4 client.17355 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:26:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.277+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/3994010478 --> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f159321f00 con 0x55f15943af50 2024-04-25T00:26:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.278+0000 7f7f42ffd640 1 --2- 172.21.15.27:0/3994010478 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55f15943af50 0x55f15943d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.278+0000 7f7f42ffd640 1 --2- 172.21.15.27:0/3994010478 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55f15943af50 0x55f15943d340 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=da15af2aa2eb139f server_cookie=de8c41839a14f0ed in_seq=0 out_seq=0 2024-04-25T00:26:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.278+0000 7f7f26ff5640 10 client.17355 ms_handle_connect on v2:172.21.15.153:6836/4066071541 2024-04-25T00:26:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.278+0000 7f7f26ff5640 1 -- 172.21.15.27:0/3994010478 <== mds.2 v2:172.21.15.153:6836/4066071541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f159321f00 con 0x55f15943af50 2024-04-25T00:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.278+0000 7f7f26ff5640 10 client.17355 handle_command_reply: tid=0 2024-04-25T00:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.278+0000 7f7f4c1b6740 1 client.17355 shutdown 2024-04-25T00:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.278+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/3994010478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593b5da0 msgr2=0x55f1593bdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.278+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/3994010478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593b5da0 0x55f1593bdde0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f7f34097800 tx=0x7f7f34097c40 comp rx=0 tx=0).stop 2024-04-25T00:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.278+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/3994010478 shutdown_connections 2024-04-25T00:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.278+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/3994010478 >> [v2:172.21.15.153:6836/4066071541,v1:172.21.15.153:6837/4066071541] conn(0x55f15943af50 0x55f15943d340 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.278+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/3994010478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f1593c4980 0x55f1593be320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.278+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/3994010478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1593b5da0 0x55f1593bdde0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.278+0000 7f7f4c1b6740 1 --2- 172.21.15.27:0/3994010478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1593b5480 0x55f1593ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.278+0000 7f7f26ff5640 0 client.17355 ms_handle_reset on v2:172.21.15.153:6836/4066071541 2024-04-25T00:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.278+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/3994010478 >> 172.21.15.27:0/3994010478 conn(0x55f1593b18a0 msgr2=0x55f1593b02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.278+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/3994010478 shutdown_connections 2024-04-25T00:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.278+0000 7f7f4c1b6740 1 -- 172.21.15.27:0/3994010478 wait complete. 2024-04-25T00:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.279+0000 7f7f4c1b6740 20 client.17355 trim_cache size 0 max 16384 2024-04-25T00:26:03.287 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:26:03.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.279+0000 7f7f494ae640 1 -- 172.21.15.27:0/1875993086 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7f2c0a2d00 msgr2=0x7f7f2c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.279+0000 7f7f494ae640 1 --2- 172.21.15.27:0/1875993086 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7f2c0a2d00 0x7f7f2c0a51c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c009a60 tx=0x7f7f3c004a40 comp rx=0 tx=0).stop 2024-04-25T00:26:03.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.279+0000 7f7f494ae640 1 -- 172.21.15.27:0/1875993086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f441032e0 msgr2=0x7f7f44198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.279+0000 7f7f494ae640 1 --2- 172.21.15.27:0/1875993086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f441032e0 0x7f7f44198560 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400d950 tx=0x7f7f3400de10 comp rx=0 tx=0).stop 2024-04-25T00:26:03.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.279+0000 7f7f494ae640 1 -- 172.21.15.27:0/1875993086 shutdown_connections 2024-04-25T00:26:03.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.279+0000 7f7f494ae640 1 --2- 172.21.15.27:0/1875993086 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f7f2c0a2d00 0x7f7f2c0a51c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.279+0000 7f7f494ae640 1 --2- 172.21.15.27:0/1875993086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f4410bd00 0x7f7f4419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.279+0000 7f7f494ae640 1 --2- 172.21.15.27:0/1875993086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f441032e0 0x7f7f44198560 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.279+0000 7f7f494ae640 1 --2- 172.21.15.27:0/1875993086 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f441029c0 0x7f7f44198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.279+0000 7f7f494ae640 1 -- 172.21.15.27:0/1875993086 >> 172.21.15.27:0/1875993086 conn(0x7f7f44077740 msgr2=0x7f7f440fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.279+0000 7f7f494ae640 1 -- 172.21.15.27:0/1875993086 shutdown_connections 2024-04-25T00:26:03.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.279+0000 7f7f494ae640 1 -- 172.21.15.27:0/1875993086 wait complete. 2024-04-25T00:26:03.292 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T00:26:03.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.394+0000 7fe4e454f640 1 -- 172.21.15.27:0/4220872248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4dc100540 msgr2=0x7fe4dc100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.394+0000 7fe4e454f640 1 --2- 172.21.15.27:0/4220872248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4dc100540 0x7fe4dc100920 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4009900 tx=0x7fe4d402a410 comp rx=0 tx=0).stop 2024-04-25T00:26:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.394+0000 7fe4e454f640 1 -- 172.21.15.27:0/4220872248 shutdown_connections 2024-04-25T00:26:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.394+0000 7fe4e454f640 1 --2- 172.21.15.27:0/4220872248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4dc10e0f0 0x7fe4dc1104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.395+0000 7fe4e454f640 1 --2- 172.21.15.27:0/4220872248 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4dc100e60 0x7fe4dc10dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.395+0000 7fe4e454f640 1 --2- 172.21.15.27:0/4220872248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4dc100540 0x7fe4dc100920 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.395+0000 7fe4e454f640 1 -- 172.21.15.27:0/4220872248 >> 172.21.15.27:0/4220872248 conn(0x7fe4dc077740 msgr2=0x7fe4dc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.395+0000 7fe4e454f640 1 -- 172.21.15.27:0/4220872248 shutdown_connections 2024-04-25T00:26:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.395+0000 7fe4e454f640 1 -- 172.21.15.27:0/4220872248 wait complete. 2024-04-25T00:26:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.395+0000 7fe4e454f640 1 Processor -- start 2024-04-25T00:26:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.395+0000 7fe4e454f640 1 -- start start 2024-04-25T00:26:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.395+0000 7fe4e454f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4dc100540 0x7fe4dc19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.395+0000 7fe4e454f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4dc100e60 0x7fe4dc19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.395+0000 7fe4e454f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4dc10e0f0 0x7fe4dc19ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.395+0000 7fe4e454f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc113d20 con 0x7fe4dc10e0f0 2024-04-25T00:26:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.395+0000 7fe4e454f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc113ba0 con 0x7fe4dc100540 2024-04-25T00:26:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.395+0000 7fe4e454f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc113ea0 con 0x7fe4dc100e60 2024-04-25T00:26:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.396+0000 7fe4e2aec640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4dc10e0f0 0x7fe4dc19ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.396+0000 7fe4e2aec640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4dc10e0f0 0x7fe4dc19ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54490/0 (socket says 172.21.15.27:54490) 2024-04-25T00:26:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.396+0000 7fe4e2aec640 1 -- 172.21.15.27:0/1347112683 learned_addr learned my addr 172.21.15.27:0/1347112683 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.396+0000 7fe4e22eb640 1 --2- 172.21.15.27:0/1347112683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4dc100540 0x7fe4dc19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.396+0000 7fe4e1aea640 1 --2- 172.21.15.27:0/1347112683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4dc100e60 0x7fe4dc19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.396+0000 7fe4e2aec640 1 -- 172.21.15.27:0/1347112683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4dc100e60 msgr2=0x7fe4dc19a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.396+0000 7fe4e2aec640 1 --2- 172.21.15.27:0/1347112683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4dc100e60 0x7fe4dc19a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.396+0000 7fe4e2aec640 1 -- 172.21.15.27:0/1347112683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4dc100540 msgr2=0x7fe4dc19a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.396+0000 7fe4e2aec640 1 --2- 172.21.15.27:0/1347112683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4dc100540 0x7fe4dc19a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.396+0000 7fe4e2aec640 1 -- 172.21.15.27:0/1347112683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4dc19f170 con 0x7fe4dc10e0f0 2024-04-25T00:26:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.396+0000 7fe4e2aec640 1 --2- 172.21.15.27:0/1347112683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4dc10e0f0 0x7fe4dc19ea40 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc00e9d0 tx=0x7fe4cc00edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.396+0000 7fe4db7fe640 1 -- 172.21.15.27:0/1347112683 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc004d20 con 0x7fe4dc10e0f0 2024-04-25T00:26:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.396+0000 7fe4db7fe640 1 -- 172.21.15.27:0/1347112683 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4cc005020 con 0x7fe4dc10e0f0 2024-04-25T00:26:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.396+0000 7fe4db7fe640 1 -- 172.21.15.27:0/1347112683 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc011690 con 0x7fe4dc10e0f0 2024-04-25T00:26:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.396+0000 7fe4e454f640 1 -- 172.21.15.27:0/1347112683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4dc19f460 con 0x7fe4dc10e0f0 2024-04-25T00:26:03.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.396+0000 7fe4e454f640 1 -- 172.21.15.27:0/1347112683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4dc1a6cf0 con 0x7fe4dc10e0f0 2024-04-25T00:26:03.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.398+0000 7fe4db7fe640 1 -- 172.21.15.27:0/1347112683 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe4cc005540 con 0x7fe4dc10e0f0 2024-04-25T00:26:03.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.399+0000 7fe4e7259740 1 Processor -- start 2024-04-25T00:26:03.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.399+0000 7fe4e7259740 1 -- start start 2024-04-25T00:26:03.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.399+0000 7fe4db7fe640 1 --2- 172.21.15.27:0/1347112683 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fe4c40a2d00 0x7fe4c40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.399+0000 7fe4e7259740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820e9580 0x5556820e9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.399+0000 7fe4e7259740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820e9ea0 0x5556820f4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.399+0000 7fe4e7259740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820f4570 0x5556820f6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.399+0000 7fe4e7259740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55568204e670 con 0x5556820f4570 2024-04-25T00:26:03.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.399+0000 7fe4e7259740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555681ff6970 con 0x5556820e9ea0 2024-04-25T00:26:03.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.399+0000 7fe4e7259740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55568204a630 con 0x5556820e9580 2024-04-25T00:26:03.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4e1aea640 1 --2- 172.21.15.27:0/1347112683 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fe4c40a2d00 0x7fe4c40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4e2aec640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820f4570 0x5556820f6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4e2aec640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820f4570 0x5556820f6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54498/0 (socket says 172.21.15.27:54498) 2024-04-25T00:26:03.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4e22eb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820e9580 0x5556820e9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4e2aec640 1 -- 172.21.15.27:0/955089167 learned_addr learned my addr 172.21.15.27:0/955089167 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4db7fe640 1 -- 172.21.15.27:0/1347112683 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fe4cc0cfba0 con 0x7fe4dc10e0f0 2024-04-25T00:26:03.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4e1aea640 1 --2- 172.21.15.27:0/955089167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820e9ea0 0x5556820f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4e22eb640 1 -- 172.21.15.27:0/955089167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820e9ea0 msgr2=0x5556820f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4e22eb640 1 --2- 172.21.15.27:0/955089167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820e9ea0 0x5556820f4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4e22eb640 1 -- 172.21.15.27:0/955089167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820f4570 msgr2=0x5556820f6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4e22eb640 1 --2- 172.21.15.27:0/955089167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820f4570 0x5556820f6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4e22eb640 1 -- 172.21.15.27:0/955089167 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55568201a400 con 0x5556820e9580 2024-04-25T00:26:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4e2aec640 1 --2- 172.21.15.27:0/955089167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820f4570 0x5556820f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:26:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4e1aea640 1 --2- 172.21.15.27:0/1347112683 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fe4c40a2d00 0x7fe4c40a51c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe4d00058c0 tx=0x7fe4d00138f0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4e22eb640 1 --2- 172.21.15.27:0/955089167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820e9580 0x5556820e9960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4005940 tx=0x7fe4d4031c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9491febfa7cd00f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4d8bfe640 1 -- 172.21.15.27:0/955089167 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4040070 con 0x5556820e9580 2024-04-25T00:26:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4d8bfe640 1 -- 172.21.15.27:0/955089167 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d40317b0 con 0x5556820e9580 2024-04-25T00:26:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.400+0000 7fe4d8bfe640 1 -- 172.21.15.27:0/955089167 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4035040 con 0x5556820e9580 2024-04-25T00:26:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.401+0000 7fe4e7259740 1 -- 172.21.15.27:0/955089167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820e9580 msgr2=0x5556820e9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.401+0000 7fe4e7259740 1 --2- 172.21.15.27:0/955089167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820e9580 0x5556820e9960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4005940 tx=0x7fe4d4031c60 comp rx=0 tx=0).stop 2024-04-25T00:26:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.401+0000 7fe4e7259740 1 -- 172.21.15.27:0/955089167 shutdown_connections 2024-04-25T00:26:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.401+0000 7fe4e7259740 1 --2- 172.21.15.27:0/955089167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820f4570 0x5556820f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.401+0000 7fe4e7259740 1 --2- 172.21.15.27:0/955089167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820e9ea0 0x5556820f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.401+0000 7fe4e7259740 1 --2- 172.21.15.27:0/955089167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820e9580 0x5556820e9960 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.401+0000 7fe4e7259740 1 -- 172.21.15.27:0/955089167 >> 172.21.15.27:0/955089167 conn(0x5556820e58a0 msgr2=0x5556820e5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.401+0000 7fe4e7259740 1 -- 172.21.15.27:0/955089167 shutdown_connections 2024-04-25T00:26:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.401+0000 7fe4e7259740 1 -- 172.21.15.27:0/955089167 wait complete. 2024-04-25T00:26:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e7259740 1 -- start start 2024-04-25T00:26:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e7259740 1 Processor -- start 2024-04-25T00:26:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e7259740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9580 0x5556820f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e7259740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820e9ea0 0x5556820f11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e7259740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820f4570 0x5556820f1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e7259740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5556820f7300 con 0x5556820e9580 2024-04-25T00:26:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e7259740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55568200a050 con 0x5556820f4570 2024-04-25T00:26:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e7259740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5556820f7480 con 0x5556820e9ea0 2024-04-25T00:26:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e22eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9580 0x5556820f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e7259740 10 client.0 _ll_register_callbacks cb 0x555681d61be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:26:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e22eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9580 0x5556820f3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54504/0 (socket says 172.21.15.27:54504) 2024-04-25T00:26:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e22eb640 1 -- 172.21.15.27:0/3053595192 learned_addr learned my addr 172.21.15.27:0/3053595192 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e1aea640 1 --2- 172.21.15.27:0/3053595192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820e9ea0 0x5556820f11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e2aec640 1 --2- 172.21.15.27:0/3053595192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820f4570 0x5556820f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e22eb640 1 -- 172.21.15.27:0/3053595192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820e9ea0 msgr2=0x5556820f11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e22eb640 1 --2- 172.21.15.27:0/3053595192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820e9ea0 0x5556820f11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e22eb640 1 -- 172.21.15.27:0/3053595192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820f4570 msgr2=0x5556820f1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e22eb640 1 --2- 172.21.15.27:0/3053595192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820f4570 0x5556820f1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.402+0000 7fe4e22eb640 1 -- 172.21.15.27:0/3053595192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555681fdbdc0 con 0x5556820e9580 2024-04-25T00:26:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.403+0000 7fe4e22eb640 1 --2- 172.21.15.27:0/3053595192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9580 0x5556820f3e80 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fe4d40052c0 tx=0x7fe4d40316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a692f427d96d81f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.403+0000 7fe4c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:26:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.403+0000 7fe4c1ffb640 1 -- 172.21.15.27:0/3053595192 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4040030 con 0x5556820e9580 2024-04-25T00:26:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.403+0000 7fe4c1ffb640 1 -- 172.21.15.27:0/3053595192 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fe4d4004230 con 0x5556820e9580 2024-04-25T00:26:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.403+0000 7fe4e7259740 1 -- 172.21.15.27:0/3053595192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4d40317b0 con 0x5556820e9580 2024-04-25T00:26:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.403+0000 7fe4c1ffb640 1 -- 172.21.15.27:0/3053595192 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fe4d4056050 con 0x5556820e9580 2024-04-25T00:26:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.403+0000 7fe4e7259740 10 client.17367 fetch_fsmap learned FSMap version 101 2024-04-25T00:26:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.403+0000 7fe4e7259740 1 -- 172.21.15.27:0/3053595192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x555681d623f0 con 0x5556820e9580 2024-04-25T00:26:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.403+0000 7fe4c1ffb640 1 -- 172.21.15.27:0/3053595192 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d40317b0 con 0x5556820e9580 2024-04-25T00:26:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.403+0000 7fe4c1ffb640 1 -- 172.21.15.27:0/3053595192 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 101) ==== 8362+0+0 (secure 0 0 0) 0x7fe4d40037b0 con 0x5556820e9580 2024-04-25T00:26:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.403+0000 7fe4e7259740 10 client.17367 fetch_fsmap finished waiting for FSMap version 101 2024-04-25T00:26:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.403+0000 7fe4e7259740 10 client.17367 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:26:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.403+0000 7fe4e7259740 20 client.17367 populate_metadata read hostname 'smithi027' 2024-04-25T00:26:03.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.403+0000 7fe4e7259740 1 --2- 172.21.15.27:0/3053595192 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x555682104410 0x555682106800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.403+0000 7fe4e7259740 4 client.17367 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:26:03.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.403+0000 7fe4e7259740 1 -- 172.21.15.27:0/3053595192 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55568205bbb0 con 0x555682104410 2024-04-25T00:26:03.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.404+0000 7fe4e1aea640 1 --2- 172.21.15.27:0/3053595192 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x555682104410 0x555682106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.404+0000 7fe4e1aea640 1 --2- 172.21.15.27:0/3053595192 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x555682104410 0x555682106800 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1788b699c2f8c28 server_cookie=f42a5a5b407ebd8a in_seq=0 out_seq=0 2024-04-25T00:26:03.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.404+0000 7fe4c1ffb640 10 client.17367 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:26:03.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.405+0000 7fe4c1ffb640 1 -- 172.21.15.27:0/3053595192 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55568205bbb0 con 0x555682104410 2024-04-25T00:26:03.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.405+0000 7fe4c1ffb640 10 client.17367 handle_command_reply: tid=0 2024-04-25T00:26:03.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.405+0000 7fe4e7259740 1 client.17367 shutdown 2024-04-25T00:26:03.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.405+0000 7fe4e7259740 1 -- 172.21.15.27:0/3053595192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9580 msgr2=0x5556820f3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.405+0000 7fe4e7259740 1 --2- 172.21.15.27:0/3053595192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9580 0x5556820f3e80 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fe4d40052c0 tx=0x7fe4d40316d0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.405+0000 7fe4e7259740 1 -- 172.21.15.27:0/3053595192 shutdown_connections 2024-04-25T00:26:03.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.405+0000 7fe4e7259740 1 --2- 172.21.15.27:0/3053595192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820f4570 0x5556820f1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.405+0000 7fe4e7259740 1 --2- 172.21.15.27:0/3053595192 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x555682104410 0x555682106800 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.405+0000 7fe4e7259740 1 --2- 172.21.15.27:0/3053595192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820e9ea0 0x5556820f11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.405+0000 7fe4e7259740 1 --2- 172.21.15.27:0/3053595192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9580 0x5556820f3e80 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.405+0000 7fe4e7259740 1 -- 172.21.15.27:0/3053595192 >> 172.21.15.27:0/3053595192 conn(0x5556820e58a0 msgr2=0x5556820eb6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.405+0000 7fe4c1ffb640 0 client.17367 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:26:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.405+0000 7fe4e7259740 1 -- 172.21.15.27:0/3053595192 shutdown_connections 2024-04-25T00:26:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.405+0000 7fe4e7259740 1 -- 172.21.15.27:0/3053595192 wait complete. 2024-04-25T00:26:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.405+0000 7fe4e7259740 20 client.17367 trim_cache size 0 max 16384 2024-04-25T00:26:03.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.418+0000 7fe4e7259740 1 Processor -- start 2024-04-25T00:26:03.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e7259740 1 -- start start 2024-04-25T00:26:03.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e7259740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820e9580 0x5556820f0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e7259740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9ea0 0x5556820eb640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e7259740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820f4570 0x5556820f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e7259740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc1b0d80 con 0x5556820e9ea0 2024-04-25T00:26:03.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e7259740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc1b28c0 con 0x5556820f4570 2024-04-25T00:26:03.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e7259740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc1b1450 con 0x5556820e9580 2024-04-25T00:26:03.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e1aea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9ea0 0x5556820eb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e1aea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9ea0 0x5556820eb640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54510/0 (socket says 172.21.15.27:54510) 2024-04-25T00:26:03.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e1aea640 1 -- 172.21.15.27:0/3420309677 learned_addr learned my addr 172.21.15.27:0/3420309677 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e2aec640 1 --2- 172.21.15.27:0/3420309677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820f4570 0x5556820f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e1aea640 1 -- 172.21.15.27:0/3420309677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820e9580 msgr2=0x5556820f0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e22eb640 1 --2- 172.21.15.27:0/3420309677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820e9580 0x5556820f0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e1aea640 1 --2- 172.21.15.27:0/3420309677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820e9580 0x5556820f0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e1aea640 1 -- 172.21.15.27:0/3420309677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820f4570 msgr2=0x5556820f3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e1aea640 1 --2- 172.21.15.27:0/3420309677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820f4570 0x5556820f3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e1aea640 1 -- 172.21.15.27:0/3420309677 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555681e2c7d0 con 0x5556820e9ea0 2024-04-25T00:26:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e22eb640 1 --2- 172.21.15.27:0/3420309677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820e9580 0x5556820f0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:26:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4e1aea640 1 --2- 172.21.15.27:0/3420309677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9ea0 0x5556820eb640 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x5556820eab40 tx=0x7fe4d0013f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29b3a177f98dc2c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4c3fff640 1 -- 172.21.15.27:0/3420309677 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d002e030 con 0x5556820e9ea0 2024-04-25T00:26:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4c3fff640 1 -- 172.21.15.27:0/3420309677 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d0020db0 con 0x5556820e9ea0 2024-04-25T00:26:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.419+0000 7fe4c3fff640 1 -- 172.21.15.27:0/3420309677 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d002a6c0 con 0x5556820e9ea0 2024-04-25T00:26:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.420+0000 7fe4e7259740 1 -- 172.21.15.27:0/3420309677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9ea0 msgr2=0x5556820eb640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.420+0000 7fe4e7259740 1 --2- 172.21.15.27:0/3420309677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9ea0 0x5556820eb640 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x5556820eab40 tx=0x7fe4d0013f90 comp rx=0 tx=0).stop 2024-04-25T00:26:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.420+0000 7fe4e7259740 1 -- 172.21.15.27:0/3420309677 shutdown_connections 2024-04-25T00:26:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.420+0000 7fe4e7259740 1 --2- 172.21.15.27:0/3420309677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820f4570 0x5556820f3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.420+0000 7fe4e7259740 1 --2- 172.21.15.27:0/3420309677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9ea0 0x5556820eb640 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.420+0000 7fe4e7259740 1 --2- 172.21.15.27:0/3420309677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820e9580 0x5556820f0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.420+0000 7fe4e7259740 1 -- 172.21.15.27:0/3420309677 >> 172.21.15.27:0/3420309677 conn(0x5556820e58a0 msgr2=0x5556820ec7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.420+0000 7fe4e7259740 1 -- 172.21.15.27:0/3420309677 shutdown_connections 2024-04-25T00:26:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.420+0000 7fe4e7259740 1 -- 172.21.15.27:0/3420309677 wait complete. 2024-04-25T00:26:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.420+0000 7fe4e7259740 1 -- start start 2024-04-25T00:26:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.420+0000 7fe4e7259740 1 Processor -- start 2024-04-25T00:26:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e7259740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820e9580 0x5556820fc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e7259740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9ea0 0x5556820ed860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e7259740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820f4570 0x5556820edda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e7259740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc1b21f0 con 0x5556820e9ea0 2024-04-25T00:26:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e7259740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc1b1b20 con 0x5556820e9580 2024-04-25T00:26:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e7259740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc1aed40 con 0x5556820f4570 2024-04-25T00:26:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e7259740 10 client.0 _ll_register_callbacks cb 0x7fe4cc005830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:26:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e1aea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9ea0 0x5556820ed860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e1aea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9ea0 0x5556820ed860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54514/0 (socket says 172.21.15.27:54514) 2024-04-25T00:26:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e1aea640 1 -- 172.21.15.27:0/2592308634 learned_addr learned my addr 172.21.15.27:0/2592308634 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e22eb640 1 --2- 172.21.15.27:0/2592308634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820e9580 0x5556820fc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e2aec640 1 --2- 172.21.15.27:0/2592308634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820f4570 0x5556820edda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e1aea640 1 -- 172.21.15.27:0/2592308634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820f4570 msgr2=0x5556820edda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e1aea640 1 --2- 172.21.15.27:0/2592308634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820f4570 0x5556820edda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e1aea640 1 -- 172.21.15.27:0/2592308634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820e9580 msgr2=0x5556820fc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e1aea640 1 --2- 172.21.15.27:0/2592308634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820e9580 0x5556820fc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e1aea640 1 -- 172.21.15.27:0/2592308634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556820ee2e0 con 0x5556820e9ea0 2024-04-25T00:26:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e2aec640 1 --2- 172.21.15.27:0/2592308634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820f4570 0x5556820edda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:26:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4e1aea640 1 --2- 172.21.15.27:0/2592308634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9ea0 0x5556820ed860 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000ad70 tx=0x7fe4d002a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6473c628c7dfb07d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:26:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.421+0000 7fe4c1ffb640 1 -- 172.21.15.27:0/2592308634 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0037030 con 0x5556820e9ea0 2024-04-25T00:26:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.422+0000 7fe4c1ffb640 1 -- 172.21.15.27:0/2592308634 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fe4d00208d0 con 0x5556820e9ea0 2024-04-25T00:26:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.422+0000 7fe4e7259740 1 -- 172.21.15.27:0/2592308634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4d0020db0 con 0x5556820e9ea0 2024-04-25T00:26:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.422+0000 7fe4c1ffb640 1 -- 172.21.15.27:0/2592308634 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fe4d0044050 con 0x5556820e9ea0 2024-04-25T00:26:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.422+0000 7fe4e7259740 10 client.17373 fetch_fsmap learned FSMap version 101 2024-04-25T00:26:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.422+0000 7fe4e7259740 1 -- 172.21.15.27:0/2592308634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55568216a360 con 0x5556820e9ea0 2024-04-25T00:26:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.422+0000 7fe4c1ffb640 1 -- 172.21.15.27:0/2592308634 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d0020db0 con 0x5556820e9ea0 2024-04-25T00:26:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.422+0000 7fe4c1ffb640 1 -- 172.21.15.27:0/2592308634 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 101) ==== 8362+0+0 (secure 0 0 0) 0x7fe4d0005250 con 0x5556820e9ea0 2024-04-25T00:26:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.422+0000 7fe4e7259740 10 client.17373 fetch_fsmap finished waiting for FSMap version 101 2024-04-25T00:26:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.422+0000 7fe4e7259740 10 client.17373 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T00:26:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.422+0000 7fe4e7259740 20 client.17373 populate_metadata read hostname 'smithi027' 2024-04-25T00:26:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.422+0000 7fe4e7259740 1 --2- 172.21.15.27:0/2592308634 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55568216a9c0 0x55568216cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.422+0000 7fe4e7259740 4 client.17373 mds_command: new command op to 24514 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:26:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.422+0000 7fe4e7259740 1 -- 172.21.15.27:0/2592308634 --> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555682055f00 con 0x55568216a9c0 2024-04-25T00:26:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.422+0000 7fe4e22eb640 1 --2- 172.21.15.27:0/2592308634 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55568216a9c0 0x55568216cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.423+0000 7fe4e22eb640 1 --2- 172.21.15.27:0/2592308634 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55568216a9c0 0x55568216cdb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2639ddfea9541193 server_cookie=e074ea4035ae2edf in_seq=0 out_seq=0 2024-04-25T00:26:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.423+0000 7fe4c1ffb640 10 client.17373 ms_handle_connect on v2:172.21.15.153:6838/1144169732 2024-04-25T00:26:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.423+0000 7fe4c1ffb640 1 -- 172.21.15.27:0/2592308634 <== mds.1 v2:172.21.15.153:6838/1144169732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555682055f00 con 0x55568216a9c0 2024-04-25T00:26:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.423+0000 7fe4c1ffb640 10 client.17373 handle_command_reply: tid=0 2024-04-25T00:26:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.423+0000 7fe4e7259740 1 client.17373 shutdown 2024-04-25T00:26:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.423+0000 7fe4e7259740 1 -- 172.21.15.27:0/2592308634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9ea0 msgr2=0x5556820ed860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.423+0000 7fe4e7259740 1 --2- 172.21.15.27:0/2592308634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9ea0 0x5556820ed860 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000ad70 tx=0x7fe4d002a340 comp rx=0 tx=0).stop 2024-04-25T00:26:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.423+0000 7fe4e7259740 1 -- 172.21.15.27:0/2592308634 shutdown_connections 2024-04-25T00:26:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.423+0000 7fe4e7259740 1 --2- 172.21.15.27:0/2592308634 >> [v2:172.21.15.153:6838/1144169732,v1:172.21.15.153:6839/1144169732] conn(0x55568216a9c0 0x55568216cdb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.423+0000 7fe4e7259740 1 --2- 172.21.15.27:0/2592308634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5556820f4570 0x5556820edda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.423+0000 7fe4c1ffb640 0 client.17373 ms_handle_reset on v2:172.21.15.153:6838/1144169732 2024-04-25T00:26:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.423+0000 7fe4e7259740 1 --2- 172.21.15.27:0/2592308634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556820e9ea0 0x5556820ed860 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.423+0000 7fe4e7259740 1 --2- 172.21.15.27:0/2592308634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5556820e9580 0x5556820fc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.423+0000 7fe4e7259740 1 -- 172.21.15.27:0/2592308634 >> 172.21.15.27:0/2592308634 conn(0x5556820e58a0 msgr2=0x5556820eb640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.423+0000 7fe4e7259740 1 -- 172.21.15.27:0/2592308634 shutdown_connections 2024-04-25T00:26:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.423+0000 7fe4e7259740 1 -- 172.21.15.27:0/2592308634 wait complete. 2024-04-25T00:26:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.423+0000 7fe4e7259740 20 client.17373 trim_cache size 0 max 16384 2024-04-25T00:26:03.431 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:26:03.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.424+0000 7fe4e454f640 1 -- 172.21.15.27:0/1347112683 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fe4c40a2d00 msgr2=0x7fe4c40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.424+0000 7fe4e454f640 1 --2- 172.21.15.27:0/1347112683 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fe4c40a2d00 0x7fe4c40a51c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe4d00058c0 tx=0x7fe4d00138f0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.424+0000 7fe4e454f640 1 -- 172.21.15.27:0/1347112683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4dc10e0f0 msgr2=0x7fe4dc19ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.424+0000 7fe4e454f640 1 --2- 172.21.15.27:0/1347112683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4dc10e0f0 0x7fe4dc19ea40 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc00e9d0 tx=0x7fe4cc00edd0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.424+0000 7fe4e454f640 1 -- 172.21.15.27:0/1347112683 shutdown_connections 2024-04-25T00:26:03.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.424+0000 7fe4e454f640 1 --2- 172.21.15.27:0/1347112683 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fe4c40a2d00 0x7fe4c40a51c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.424+0000 7fe4e454f640 1 --2- 172.21.15.27:0/1347112683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4dc10e0f0 0x7fe4dc19ea40 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.424+0000 7fe4e454f640 1 --2- 172.21.15.27:0/1347112683 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4dc100e60 0x7fe4dc19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.424+0000 7fe4e454f640 1 --2- 172.21.15.27:0/1347112683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4dc100540 0x7fe4dc19a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.424+0000 7fe4e454f640 1 -- 172.21.15.27:0/1347112683 >> 172.21.15.27:0/1347112683 conn(0x7fe4dc077740 msgr2=0x7fe4dc10f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.424+0000 7fe4e454f640 1 -- 172.21.15.27:0/1347112683 shutdown_connections 2024-04-25T00:26:03.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.424+0000 7fe4e454f640 1 -- 172.21.15.27:0/1347112683 wait complete. 2024-04-25T00:26:03.436 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T00:26:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.538+0000 7f2c44631640 1 -- 172.21.15.27:0/2105174019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c3c074410 msgr2=0x7f2c3c10da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.538+0000 7f2c44631640 1 --2- 172.21.15.27:0/2105174019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c3c074410 0x7f2c3c10da90 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c009900 tx=0x7f2c2c02a480 comp rx=0 tx=0).stop 2024-04-25T00:26:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.538+0000 7f2c44631640 1 -- 172.21.15.27:0/2105174019 shutdown_connections 2024-04-25T00:26:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.538+0000 7f2c44631640 1 --2- 172.21.15.27:0/2105174019 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c3c10dfd0 0x7f2c3c1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.538+0000 7f2c44631640 1 --2- 172.21.15.27:0/2105174019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c3c074410 0x7f2c3c10da90 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.538+0000 7f2c44631640 1 --2- 172.21.15.27:0/2105174019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c3c073af0 0x7f2c3c073ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.538+0000 7f2c44631640 1 -- 172.21.15.27:0/2105174019 >> 172.21.15.27:0/2105174019 conn(0x7f2c3c0fd180 msgr2=0x7f2c3c0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.538+0000 7f2c44631640 1 -- 172.21.15.27:0/2105174019 shutdown_connections 2024-04-25T00:26:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.538+0000 7f2c44631640 1 -- 172.21.15.27:0/2105174019 wait complete. 2024-04-25T00:26:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.539+0000 7f2c44631640 1 Processor -- start 2024-04-25T00:26:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.539+0000 7f2c44631640 1 -- start start 2024-04-25T00:26:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.539+0000 7f2c44631640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c3c073af0 0x7f2c3c19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.539+0000 7f2c44631640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c3c074410 0x7f2c3c19a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.539+0000 7f2c44631640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c3c10dfd0 0x7f2c3c19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.539+0000 7f2c44631640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3c113cf0 con 0x7f2c3c073af0 2024-04-25T00:26:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.539+0000 7f2c44631640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3c113b70 con 0x7f2c3c10dfd0 2024-04-25T00:26:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.539+0000 7f2c44631640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3c113e70 con 0x7f2c3c074410 2024-04-25T00:26:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.539+0000 7f2c423cd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c3c073af0 0x7f2c3c19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.539+0000 7f2c423cd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c3c073af0 0x7f2c3c19a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54532/0 (socket says 172.21.15.27:54532) 2024-04-25T00:26:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.539+0000 7f2c423cd640 1 -- 172.21.15.27:0/2248052604 learned_addr learned my addr 172.21.15.27:0/2248052604 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.539+0000 7f2c41bcc640 1 --2- 172.21.15.27:0/2248052604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c3c074410 0x7f2c3c19a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.539+0000 7f2c42bce640 1 --2- 172.21.15.27:0/2248052604 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c3c10dfd0 0x7f2c3c19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.540+0000 7f2c423cd640 1 -- 172.21.15.27:0/2248052604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c3c074410 msgr2=0x7f2c3c19a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.540+0000 7f2c423cd640 1 --2- 172.21.15.27:0/2248052604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c3c074410 0x7f2c3c19a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.540+0000 7f2c423cd640 1 -- 172.21.15.27:0/2248052604 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c3c10dfd0 msgr2=0x7f2c3c19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.540+0000 7f2c423cd640 1 --2- 172.21.15.27:0/2248052604 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c3c10dfd0 0x7f2c3c19eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.540+0000 7f2c423cd640 1 -- 172.21.15.27:0/2248052604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c3c19f210 con 0x7f2c3c073af0 2024-04-25T00:26:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.540+0000 7f2c423cd640 1 --2- 172.21.15.27:0/2248052604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c3c073af0 0x7f2c3c19a270 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f2c3800e9f0 tx=0x7f2c3800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.540+0000 7f2c337fe640 1 -- 172.21.15.27:0/2248052604 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3800cd80 con 0x7f2c3c073af0 2024-04-25T00:26:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.540+0000 7f2c337fe640 1 -- 172.21.15.27:0/2248052604 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c38005500 con 0x7f2c3c073af0 2024-04-25T00:26:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.540+0000 7f2c44631640 1 -- 172.21.15.27:0/2248052604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c3c19f500 con 0x7f2c3c073af0 2024-04-25T00:26:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.540+0000 7f2c337fe640 1 -- 172.21.15.27:0/2248052604 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c38010690 con 0x7f2c3c073af0 2024-04-25T00:26:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.540+0000 7f2c44631640 1 -- 172.21.15.27:0/2248052604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c3c1a6d90 con 0x7f2c3c073af0 2024-04-25T00:26:03.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.543+0000 7f2c337fe640 1 -- 172.21.15.27:0/2248052604 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2c380046c0 con 0x7f2c3c073af0 2024-04-25T00:26:03.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.543+0000 7f2c4741c740 1 Processor -- start 2024-04-25T00:26:03.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.543+0000 7f2c4741c740 1 -- start start 2024-04-25T00:26:03.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c4741c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557c3580 0x558f557c3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c4741c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557c3ea0 0x558f557ce030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c4741c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557ce570 0x558f557d0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c4741c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558f55728670 con 0x558f557ce570 2024-04-25T00:26:03.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c4741c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558f556d0970 con 0x558f557c3ea0 2024-04-25T00:26:03.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c4741c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558f55724630 con 0x558f557c3580 2024-04-25T00:26:03.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c41bcc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557ce570 0x558f557d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c41bcc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557ce570 0x558f557d0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54538/0 (socket says 172.21.15.27:54538) 2024-04-25T00:26:03.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c41bcc640 1 -- 172.21.15.27:0/3661710905 learned_addr learned my addr 172.21.15.27:0/3661710905 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c337fe640 1 --2- 172.21.15.27:0/2248052604 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2c280a2d00 0x7f2c280a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c423cd640 1 --2- 172.21.15.27:0/3661710905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557c3ea0 0x558f557ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c42bce640 1 --2- 172.21.15.27:0/3661710905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557c3580 0x558f557c3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c42bce640 1 --2- 172.21.15.27:0/2248052604 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2c280a2d00 0x7f2c280a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c41bcc640 1 -- 172.21.15.27:0/3661710905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557c3580 msgr2=0x558f557c3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c41bcc640 1 --2- 172.21.15.27:0/3661710905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557c3580 0x558f557c3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c337fe640 1 -- 172.21.15.27:0/2248052604 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f2c380cfbe0 con 0x7f2c3c073af0 2024-04-25T00:26:03.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c41bcc640 1 -- 172.21.15.27:0/3661710905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557c3ea0 msgr2=0x558f557ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c41bcc640 1 --2- 172.21.15.27:0/3661710905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557c3ea0 0x558f557ce030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c41bcc640 1 -- 172.21.15.27:0/3661710905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f556f4400 con 0x558f557ce570 2024-04-25T00:26:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.544+0000 7f2c41bcc640 1 --2- 172.21.15.27:0/3661710905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557ce570 0x558f557d0960 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c009900 tx=0x7f2c2c005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45176d9275d8f151 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.545+0000 7f2c30bfe640 1 -- 172.21.15.27:0/3661710905 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c040070 con 0x558f557ce570 2024-04-25T00:26:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.545+0000 7f2c42bce640 1 --2- 172.21.15.27:0/2248052604 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2c280a2d00 0x7f2c280a51c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2c34005fd0 tx=0x7f2c34009600 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.545+0000 7f2c30bfe640 1 -- 172.21.15.27:0/3661710905 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c2c0318f0 con 0x558f557ce570 2024-04-25T00:26:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.545+0000 7f2c30bfe640 1 -- 172.21.15.27:0/3661710905 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c035040 con 0x558f557ce570 2024-04-25T00:26:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.545+0000 7f2c4741c740 1 -- 172.21.15.27:0/3661710905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557ce570 msgr2=0x558f557d0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.545+0000 7f2c4741c740 1 --2- 172.21.15.27:0/3661710905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557ce570 0x558f557d0960 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c009900 tx=0x7f2c2c005e00 comp rx=0 tx=0).stop 2024-04-25T00:26:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.545+0000 7f2c4741c740 1 -- 172.21.15.27:0/3661710905 shutdown_connections 2024-04-25T00:26:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.545+0000 7f2c4741c740 1 --2- 172.21.15.27:0/3661710905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557ce570 0x558f557d0960 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.545+0000 7f2c4741c740 1 --2- 172.21.15.27:0/3661710905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557c3ea0 0x558f557ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.545+0000 7f2c4741c740 1 --2- 172.21.15.27:0/3661710905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557c3580 0x558f557c3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.545+0000 7f2c4741c740 1 -- 172.21.15.27:0/3661710905 >> 172.21.15.27:0/3661710905 conn(0x558f557bf8a0 msgr2=0x558f557bfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.545+0000 7f2c4741c740 1 -- 172.21.15.27:0/3661710905 shutdown_connections 2024-04-25T00:26:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.545+0000 7f2c4741c740 1 -- 172.21.15.27:0/3661710905 wait complete. 2024-04-25T00:26:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c4741c740 1 -- start start 2024-04-25T00:26:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c4741c740 1 Processor -- start 2024-04-25T00:26:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c4741c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557c3580 0x558f557cde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c4741c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557c3ea0 0x558f557cb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c4741c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557ce570 0x558f557cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c4741c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558f557d1300 con 0x558f557c3ea0 2024-04-25T00:26:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c4741c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558f556e4050 con 0x558f557c3580 2024-04-25T00:26:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c4741c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558f557d1480 con 0x558f557ce570 2024-04-25T00:26:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c4741c740 10 client.0 _ll_register_callbacks cb 0x558f5543bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:26:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c41bcc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557c3ea0 0x558f557cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c41bcc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557c3ea0 0x558f557cb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54548/0 (socket says 172.21.15.27:54548) 2024-04-25T00:26:03.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c41bcc640 1 -- 172.21.15.27:0/3717705741 learned_addr learned my addr 172.21.15.27:0/3717705741 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c42bce640 1 --2- 172.21.15.27:0/3717705741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557ce570 0x558f557cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c41bcc640 1 -- 172.21.15.27:0/3717705741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557ce570 msgr2=0x558f557cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c41bcc640 1 --2- 172.21.15.27:0/3717705741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557ce570 0x558f557cb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c41bcc640 1 -- 172.21.15.27:0/3717705741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557c3580 msgr2=0x558f557cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c41bcc640 1 --2- 172.21.15.27:0/3717705741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557c3580 0x558f557cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c41bcc640 1 -- 172.21.15.27:0/3717705741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f556b5dc0 con 0x558f557c3ea0 2024-04-25T00:26:03.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.546+0000 7f2c42bce640 1 --2- 172.21.15.27:0/3717705741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557ce570 0x558f557cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:26:03.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.547+0000 7f2c41bcc640 1 --2- 172.21.15.27:0/3717705741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557c3ea0 0x558f557cb180 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c005940 tx=0x7f2c2c005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c39e429287f4a3b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.547+0000 7f2c25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:26:03.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.547+0000 7f2c25ffb640 1 -- 172.21.15.27:0/3717705741 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c04c030 con 0x558f557c3ea0 2024-04-25T00:26:03.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.547+0000 7f2c25ffb640 1 -- 172.21.15.27:0/3717705741 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f2c2c0316f0 con 0x558f557c3ea0 2024-04-25T00:26:03.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.547+0000 7f2c4741c740 1 -- 172.21.15.27:0/3717705741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c2c0318f0 con 0x558f557c3ea0 2024-04-25T00:26:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.547+0000 7f2c25ffb640 1 -- 172.21.15.27:0/3717705741 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f2c2c056050 con 0x558f557c3ea0 2024-04-25T00:26:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.547+0000 7f2c4741c740 10 client.17385 fetch_fsmap learned FSMap version 101 2024-04-25T00:26:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.547+0000 7f2c4741c740 1 -- 172.21.15.27:0/3717705741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x558f5543c3f0 con 0x558f557c3ea0 2024-04-25T00:26:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.547+0000 7f2c25ffb640 1 -- 172.21.15.27:0/3717705741 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c2c0318f0 con 0x558f557c3ea0 2024-04-25T00:26:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.547+0000 7f2c25ffb640 1 -- 172.21.15.27:0/3717705741 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 101) ==== 8362+0+0 (secure 0 0 0) 0x7f2c2c003680 con 0x558f557c3ea0 2024-04-25T00:26:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.547+0000 7f2c4741c740 10 client.17385 fetch_fsmap finished waiting for FSMap version 101 2024-04-25T00:26:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.547+0000 7f2c4741c740 10 client.17385 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:26:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.547+0000 7f2c4741c740 20 client.17385 populate_metadata read hostname 'smithi027' 2024-04-25T00:26:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.547+0000 7f2c4741c740 1 --2- 172.21.15.27:0/3717705741 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x558f557de3d0 0x558f557e07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.547+0000 7f2c4741c740 4 client.17385 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:26:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.547+0000 7f2c4741c740 1 -- 172.21.15.27:0/3717705741 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f55735bb0 con 0x558f557de3d0 2024-04-25T00:26:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.548+0000 7f2c423cd640 1 --2- 172.21.15.27:0/3717705741 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x558f557de3d0 0x558f557e07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.548+0000 7f2c423cd640 1 --2- 172.21.15.27:0/3717705741 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x558f557de3d0 0x558f557e07c0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ada32d5cb163eff server_cookie=d5127375086da9de in_seq=0 out_seq=0 2024-04-25T00:26:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.548+0000 7f2c25ffb640 10 client.17385 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:26:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.549+0000 7f2c25ffb640 1 -- 172.21.15.27:0/3717705741 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558f55735bb0 con 0x558f557de3d0 2024-04-25T00:26:03.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.549+0000 7f2c25ffb640 10 client.17385 handle_command_reply: tid=0 2024-04-25T00:26:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.549+0000 7f2c4741c740 1 client.17385 shutdown 2024-04-25T00:26:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.549+0000 7f2c4741c740 1 -- 172.21.15.27:0/3717705741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557c3ea0 msgr2=0x558f557cb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.549+0000 7f2c4741c740 1 --2- 172.21.15.27:0/3717705741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557c3ea0 0x558f557cb180 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c005940 tx=0x7f2c2c005250 comp rx=0 tx=0).stop 2024-04-25T00:26:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.549+0000 7f2c4741c740 1 -- 172.21.15.27:0/3717705741 shutdown_connections 2024-04-25T00:26:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.549+0000 7f2c4741c740 1 --2- 172.21.15.27:0/3717705741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557ce570 0x558f557cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.549+0000 7f2c4741c740 1 --2- 172.21.15.27:0/3717705741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557c3ea0 0x558f557cb180 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.549+0000 7f2c4741c740 1 --2- 172.21.15.27:0/3717705741 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x558f557de3d0 0x558f557e07c0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.549+0000 7f2c4741c740 1 --2- 172.21.15.27:0/3717705741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557c3580 0x558f557cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.549+0000 7f2c25ffb640 0 client.17385 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:26:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.549+0000 7f2c4741c740 1 -- 172.21.15.27:0/3717705741 >> 172.21.15.27:0/3717705741 conn(0x558f557bf8a0 msgr2=0x558f557beb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.549+0000 7f2c4741c740 1 -- 172.21.15.27:0/3717705741 shutdown_connections 2024-04-25T00:26:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.549+0000 7f2c4741c740 1 -- 172.21.15.27:0/3717705741 wait complete. 2024-04-25T00:26:03.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.549+0000 7f2c4741c740 20 client.17385 trim_cache size 0 max 16384 2024-04-25T00:26:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c4741c740 1 Processor -- start 2024-04-25T00:26:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c4741c740 1 -- start start 2024-04-25T00:26:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c4741c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557c3580 0x558f557da1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c4741c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557c3ea0 0x558f557e5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c4741c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557ce570 0x558f557e6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c4741c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3c1b0de0 con 0x558f557c3ea0 2024-04-25T00:26:03.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c4741c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3c1b2920 con 0x558f557ce570 2024-04-25T00:26:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c4741c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3c1b14b0 con 0x558f557c3580 2024-04-25T00:26:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c41bcc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557c3ea0 0x558f557e5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c41bcc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557c3ea0 0x558f557e5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54554/0 (socket says 172.21.15.27:54554) 2024-04-25T00:26:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c41bcc640 1 -- 172.21.15.27:0/2121313953 learned_addr learned my addr 172.21.15.27:0/2121313953 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c42bce640 1 --2- 172.21.15.27:0/2121313953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557ce570 0x558f557e6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c41bcc640 1 -- 172.21.15.27:0/2121313953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557c3580 msgr2=0x558f557da1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c41bcc640 1 --2- 172.21.15.27:0/2121313953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557c3580 0x558f557da1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c41bcc640 1 -- 172.21.15.27:0/2121313953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557ce570 msgr2=0x558f557e6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c41bcc640 1 --2- 172.21.15.27:0/2121313953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557ce570 0x558f557e6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c41bcc640 1 -- 172.21.15.27:0/2121313953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f555067d0 con 0x558f557c3ea0 2024-04-25T00:26:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c42bce640 1 --2- 172.21.15.27:0/2121313953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557ce570 0x558f557e6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:26:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c41bcc640 1 --2- 172.21.15.27:0/2121313953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557c3ea0 0x558f557e5e20 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c000c00 tx=0x7f2c2c0095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec91a536e9d58d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.563+0000 7f2c27fff640 1 -- 172.21.15.27:0/2121313953 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c04c040 con 0x558f557c3ea0 2024-04-25T00:26:03.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.564+0000 7f2c27fff640 1 -- 172.21.15.27:0/2121313953 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c2c0056a0 con 0x558f557c3ea0 2024-04-25T00:26:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.564+0000 7f2c27fff640 1 -- 172.21.15.27:0/2121313953 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c040070 con 0x558f557c3ea0 2024-04-25T00:26:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.564+0000 7f2c4741c740 1 -- 172.21.15.27:0/2121313953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557c3ea0 msgr2=0x558f557e5e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.564+0000 7f2c4741c740 1 --2- 172.21.15.27:0/2121313953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557c3ea0 0x558f557e5e20 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c000c00 tx=0x7f2c2c0095a0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.564+0000 7f2c4741c740 1 -- 172.21.15.27:0/2121313953 shutdown_connections 2024-04-25T00:26:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.564+0000 7f2c4741c740 1 --2- 172.21.15.27:0/2121313953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557ce570 0x558f557e6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.564+0000 7f2c4741c740 1 --2- 172.21.15.27:0/2121313953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557c3ea0 0x558f557e5e20 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.564+0000 7f2c4741c740 1 --2- 172.21.15.27:0/2121313953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557c3580 0x558f557da1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.564+0000 7f2c4741c740 1 -- 172.21.15.27:0/2121313953 >> 172.21.15.27:0/2121313953 conn(0x558f557bf8a0 msgr2=0x558f557beb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.564+0000 7f2c4741c740 1 -- 172.21.15.27:0/2121313953 shutdown_connections 2024-04-25T00:26:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.564+0000 7f2c4741c740 1 -- 172.21.15.27:0/2121313953 wait complete. 2024-04-25T00:26:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.564+0000 7f2c4741c740 1 -- start start 2024-04-25T00:26:03.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.564+0000 7f2c4741c740 1 Processor -- start 2024-04-25T00:26:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c4741c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557c3580 0x558f557d66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c4741c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557c3ea0 0x558f557d6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c4741c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557ce570 0x558f557d3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c4741c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3c1b2250 con 0x558f557ce570 2024-04-25T00:26:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c4741c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3c1b1b80 con 0x558f557c3580 2024-04-25T00:26:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c4741c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3c1aeda0 con 0x558f557c3ea0 2024-04-25T00:26:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c4741c740 10 client.0 _ll_register_callbacks cb 0x7f2c2c005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:26:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c42bce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557ce570 0x558f557d3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c42bce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557ce570 0x558f557d3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54570/0 (socket says 172.21.15.27:54570) 2024-04-25T00:26:03.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c42bce640 1 -- 172.21.15.27:0/2635165922 learned_addr learned my addr 172.21.15.27:0/2635165922 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c423cd640 1 --2- 172.21.15.27:0/2635165922 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557c3580 0x558f557d66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c41bcc640 1 --2- 172.21.15.27:0/2635165922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557c3ea0 0x558f557d6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c42bce640 1 -- 172.21.15.27:0/2635165922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557c3ea0 msgr2=0x558f557d6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c42bce640 1 --2- 172.21.15.27:0/2635165922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557c3ea0 0x558f557d6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c42bce640 1 -- 172.21.15.27:0/2635165922 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557c3580 msgr2=0x558f557d66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c42bce640 1 --2- 172.21.15.27:0/2635165922 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557c3580 0x558f557d66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c42bce640 1 -- 172.21.15.27:0/2635165922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f557d44b0 con 0x558f557ce570 2024-04-25T00:26:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c42bce640 1 --2- 172.21.15.27:0/2635165922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557ce570 0x558f557d3cb0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f2c34010a70 tx=0x7f2c34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a7f925f70c78af0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T00:26:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c25ffb640 1 -- 172.21.15.27:0/2635165922 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3402e060 con 0x558f557ce570 2024-04-25T00:26:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c25ffb640 1 -- 172.21.15.27:0/2635165922 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f2c340293f0 con 0x558f557ce570 2024-04-25T00:26:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.565+0000 7f2c4741c740 1 -- 172.21.15.27:0/2635165922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c2c0056a0 con 0x558f557ce570 2024-04-25T00:26:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.566+0000 7f2c25ffb640 1 -- 172.21.15.27:0/2635165922 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f2c34021e60 con 0x558f557ce570 2024-04-25T00:26:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.566+0000 7f2c4741c740 10 client.17391 fetch_fsmap learned FSMap version 101 2024-04-25T00:26:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.566+0000 7f2c4741c740 1 -- 172.21.15.27:0/2635165922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x558f55844280 con 0x558f557ce570 2024-04-25T00:26:03.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.566+0000 7f2c25ffb640 1 -- 172.21.15.27:0/2635165922 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c2c0056a0 con 0x558f557ce570 2024-04-25T00:26:03.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.566+0000 7f2c25ffb640 1 -- 172.21.15.27:0/2635165922 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 101) ==== 8362+0+0 (secure 0 0 0) 0x7f2c34028030 con 0x558f557ce570 2024-04-25T00:26:03.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.566+0000 7f2c4741c740 10 client.17391 fetch_fsmap finished waiting for FSMap version 101 2024-04-25T00:26:03.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.566+0000 7f2c4741c740 10 client.17391 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T00:26:03.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.566+0000 7f2c4741c740 20 client.17391 populate_metadata read hostname 'smithi027' 2024-04-25T00:26:03.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.566+0000 7f2c4741c740 1 --2- 172.21.15.27:0/2635165922 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x558f55844a30 0x558f55846e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:03.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.566+0000 7f2c4741c740 4 client.17391 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T00:26:03.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.566+0000 7f2c4741c740 1 -- 172.21.15.27:0/2635165922 --> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558f5572ff00 con 0x558f55844a30 2024-04-25T00:26:03.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.566+0000 7f2c423cd640 1 --2- 172.21.15.27:0/2635165922 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x558f55844a30 0x558f55846e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:03.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.567+0000 7f2c423cd640 1 --2- 172.21.15.27:0/2635165922 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x558f55844a30 0x558f55846e20 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65264312d1638ad8 server_cookie=2f6f7120bd1b08d9 in_seq=0 out_seq=0 2024-04-25T00:26:03.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.567+0000 7f2c25ffb640 10 client.17391 ms_handle_connect on v2:172.21.15.139:6838/2728523692 2024-04-25T00:26:03.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.567+0000 7f2c25ffb640 1 -- 172.21.15.27:0/2635165922 <== mds.0 v2:172.21.15.139:6838/2728523692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558f5572ff00 con 0x558f55844a30 2024-04-25T00:26:03.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.567+0000 7f2c25ffb640 10 client.17391 handle_command_reply: tid=0 2024-04-25T00:26:03.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.567+0000 7f2c4741c740 1 client.17391 shutdown 2024-04-25T00:26:03.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.567+0000 7f2c4741c740 1 -- 172.21.15.27:0/2635165922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557ce570 msgr2=0x558f557d3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.567+0000 7f2c4741c740 1 --2- 172.21.15.27:0/2635165922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557ce570 0x558f557d3cb0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f2c34010a70 tx=0x7f2c34004a40 comp rx=0 tx=0).stop 2024-04-25T00:26:03.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.567+0000 7f2c4741c740 1 -- 172.21.15.27:0/2635165922 shutdown_connections 2024-04-25T00:26:03.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.567+0000 7f2c4741c740 1 --2- 172.21.15.27:0/2635165922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558f557ce570 0x558f557d3cb0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.567+0000 7f2c4741c740 1 --2- 172.21.15.27:0/2635165922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558f557c3ea0 0x558f557d6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.567+0000 7f2c4741c740 1 --2- 172.21.15.27:0/2635165922 >> [v2:172.21.15.139:6838/2728523692,v1:172.21.15.139:6839/2728523692] conn(0x558f55844a30 0x558f55846e20 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.567+0000 7f2c4741c740 1 --2- 172.21.15.27:0/2635165922 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f557c3580 0x558f557d66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.567+0000 7f2c4741c740 1 -- 172.21.15.27:0/2635165922 >> 172.21.15.27:0/2635165922 conn(0x558f557bf8a0 msgr2=0x558f557d91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.567+0000 7f2c25ffb640 0 client.17391 ms_handle_reset on v2:172.21.15.139:6838/2728523692 2024-04-25T00:26:03.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.567+0000 7f2c4741c740 1 -- 172.21.15.27:0/2635165922 shutdown_connections 2024-04-25T00:26:03.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.567+0000 7f2c4741c740 1 -- 172.21.15.27:0/2635165922 wait complete. 2024-04-25T00:26:03.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.567+0000 7f2c4741c740 20 client.17391 trim_cache size 0 max 16384 2024-04-25T00:26:03.575 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T00:26:03.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.568+0000 7f2c44631640 1 -- 172.21.15.27:0/2248052604 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2c280a2d00 msgr2=0x7f2c280a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.568+0000 7f2c44631640 1 --2- 172.21.15.27:0/2248052604 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2c280a2d00 0x7f2c280a51c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2c34005fd0 tx=0x7f2c34009600 comp rx=0 tx=0).stop 2024-04-25T00:26:03.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.568+0000 7f2c44631640 1 -- 172.21.15.27:0/2248052604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c3c073af0 msgr2=0x7f2c3c19a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.568+0000 7f2c44631640 1 --2- 172.21.15.27:0/2248052604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c3c073af0 0x7f2c3c19a270 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f2c3800e9f0 tx=0x7f2c3800edf0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.568+0000 7f2c44631640 1 -- 172.21.15.27:0/2248052604 shutdown_connections 2024-04-25T00:26:03.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.568+0000 7f2c44631640 1 --2- 172.21.15.27:0/2248052604 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2c280a2d00 0x7f2c280a51c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.568+0000 7f2c44631640 1 --2- 172.21.15.27:0/2248052604 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c3c10dfd0 0x7f2c3c19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.568+0000 7f2c44631640 1 --2- 172.21.15.27:0/2248052604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c3c074410 0x7f2c3c19a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.568+0000 7f2c44631640 1 --2- 172.21.15.27:0/2248052604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c3c073af0 0x7f2c3c19a270 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.568+0000 7f2c44631640 1 -- 172.21.15.27:0/2248052604 >> 172.21.15.27:0/2248052604 conn(0x7f2c3c0fd180 msgr2=0x7f2c3c0fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.568+0000 7f2c44631640 1 -- 172.21.15.27:0/2248052604 shutdown_connections 2024-04-25T00:26:03.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:03.568+0000 7f2c44631640 1 -- 172.21.15.27:0/2248052604 wait complete. 2024-04-25T00:26:04.580 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-04-25T00:26:04.580 INFO:tasks.fwd_scrub:done joining 2024-04-25T00:26:04.580 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-25T00:26:04.591 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:26:04.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.705+0000 7f5525c37640 1 -- 172.21.15.27:0/641752773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f552010bd60 msgr2=0x7f552010e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.705+0000 7f5525c37640 1 --2- 172.21.15.27:0/641752773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f552010bd60 0x7f552010e1d0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f5518009ca0 tx=0x7f551802a950 comp rx=0 tx=0).stop 2024-04-25T00:26:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.705+0000 7f5525c37640 1 -- 172.21.15.27:0/641752773 shutdown_connections 2024-04-25T00:26:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.705+0000 7f5525c37640 1 --2- 172.21.15.27:0/641752773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f552010bd60 0x7f552010e1d0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.705+0000 7f5525c37640 1 --2- 172.21.15.27:0/641752773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5520103340 0x7f552010b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.705+0000 7f5525c37640 1 --2- 172.21.15.27:0/641752773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5520102a20 0x7f5520102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.705+0000 7f5525c37640 1 -- 172.21.15.27:0/641752773 >> 172.21.15.27:0/641752773 conn(0x7f55200777a0 msgr2=0x7f5520077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.705+0000 7f5525c37640 1 -- 172.21.15.27:0/641752773 shutdown_connections 2024-04-25T00:26:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.705+0000 7f5525c37640 1 -- 172.21.15.27:0/641752773 wait complete. 2024-04-25T00:26:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.705+0000 7f5525c37640 1 Processor -- start 2024-04-25T00:26:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.706+0000 7f5525c37640 1 -- start start 2024-04-25T00:26:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.706+0000 7f5525c37640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5520102a20 0x7f5520197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.706+0000 7f5525c37640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5520103340 0x7f5520198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.706+0000 7f5525c37640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f552010bd60 0x7f552019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.706+0000 7f5525c37640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5520111b30 con 0x7f5520103340 2024-04-25T00:26:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.706+0000 7f5525c37640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f55201119b0 con 0x7f552010bd60 2024-04-25T00:26:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.706+0000 7f5525c37640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5520111cb0 con 0x7f5520102a20 2024-04-25T00:26:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.706+0000 7f551effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5520103340 0x7f5520198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.706+0000 7f551f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5520102a20 0x7f5520197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.706+0000 7f551effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5520103340 0x7f5520198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54594/0 (socket says 172.21.15.27:54594) 2024-04-25T00:26:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.706+0000 7f551effd640 1 -- 172.21.15.27:0/2269064935 learned_addr learned my addr 172.21.15.27:0/2269064935 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.706+0000 7f551ffff640 1 --2- 172.21.15.27:0/2269064935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f552010bd60 0x7f552019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.706+0000 7f551effd640 1 -- 172.21.15.27:0/2269064935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5520102a20 msgr2=0x7f5520197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.706+0000 7f551effd640 1 --2- 172.21.15.27:0/2269064935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5520102a20 0x7f5520197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.706+0000 7f551effd640 1 -- 172.21.15.27:0/2269064935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f552010bd60 msgr2=0x7f552019c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.706+0000 7f551effd640 1 --2- 172.21.15.27:0/2269064935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f552010bd60 0x7f552019c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.707+0000 7f551effd640 1 -- 172.21.15.27:0/2269064935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f552019cee0 con 0x7f5520103340 2024-04-25T00:26:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.707+0000 7f551effd640 1 --2- 172.21.15.27:0/2269064935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5520103340 0x7f5520198430 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f551400c890 tx=0x7f551400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.707+0000 7f551cff9640 1 -- 172.21.15.27:0/2269064935 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5514004220 con 0x7f5520103340 2024-04-25T00:26:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.707+0000 7f551cff9640 1 -- 172.21.15.27:0/2269064935 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55140043c0 con 0x7f5520103340 2024-04-25T00:26:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.707+0000 7f5525c37640 1 -- 172.21.15.27:0/2269064935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f552019d1d0 con 0x7f5520103340 2024-04-25T00:26:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.707+0000 7f551cff9640 1 -- 172.21.15.27:0/2269064935 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5514004a00 con 0x7f5520103340 2024-04-25T00:26:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.707+0000 7f5525c37640 1 -- 172.21.15.27:0/2269064935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55201a4a60 con 0x7f5520103340 2024-04-25T00:26:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.708+0000 7f5525c37640 1 -- 172.21.15.27:0/2269064935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55200689b0 con 0x7f5520103340 2024-04-25T00:26:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.709+0000 7f551cff9640 1 -- 172.21.15.27:0/2269064935 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f55140079b0 con 0x7f5520103340 2024-04-25T00:26:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.710+0000 7f551cff9640 1 --2- 172.21.15.27:0/2269064935 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f55080a2d00 0x7f55080a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.710+0000 7f551f7fe640 1 --2- 172.21.15.27:0/2269064935 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f55080a2d00 0x7f55080a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.710+0000 7f551cff9640 1 -- 172.21.15.27:0/2269064935 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f55140cf780 con 0x7f5520103340 2024-04-25T00:26:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.710+0000 7f551f7fe640 1 --2- 172.21.15.27:0/2269064935 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f55080a2d00 0x7f55080a51c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f551000abe0 tx=0x7f551000a2d0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:04.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.713+0000 7f551cff9640 1 -- 172.21.15.27:0/2269064935 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5514014030 con 0x7f5520103340 2024-04-25T00:26:04.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.900+0000 7f5525c37640 1 -- 172.21.15.27:0/2269064935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5520199240 con 0x7f5520103340 2024-04-25T00:26:04.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.902+0000 7f551cff9640 1 -- 172.21.15.27:0/2269064935 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25113 (secure 0 0 0) 0x7f5520199240 con 0x7f5520103340 2024-04-25T00:26:04.903 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:26:04.903 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:15:45.034931+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1885111026":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/2528869729":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/4085720546":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/236412562":"2024-04-26T00:15:45.034899+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/2161554778":"2024-04-26T00:15:45.034899+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6800/2161554778":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:26:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.904+0000 7f5525c37640 1 -- 172.21.15.27:0/2269064935 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f55080a2d00 msgr2=0x7f55080a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.904+0000 7f5525c37640 1 --2- 172.21.15.27:0/2269064935 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f55080a2d00 0x7f55080a51c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f551000abe0 tx=0x7f551000a2d0 comp rx=0 tx=0).stop 2024-04-25T00:26:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.904+0000 7f5525c37640 1 -- 172.21.15.27:0/2269064935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5520103340 msgr2=0x7f5520198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.904+0000 7f5525c37640 1 --2- 172.21.15.27:0/2269064935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5520103340 0x7f5520198430 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f551400c890 tx=0x7f551400cc90 comp rx=0 tx=0).stop 2024-04-25T00:26:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.904+0000 7f5525c37640 1 -- 172.21.15.27:0/2269064935 shutdown_connections 2024-04-25T00:26:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.904+0000 7f5525c37640 1 --2- 172.21.15.27:0/2269064935 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f55080a2d00 0x7f55080a51c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.904+0000 7f5525c37640 1 --2- 172.21.15.27:0/2269064935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f552010bd60 0x7f552019c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.904+0000 7f5525c37640 1 --2- 172.21.15.27:0/2269064935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5520103340 0x7f5520198430 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.904+0000 7f5525c37640 1 --2- 172.21.15.27:0/2269064935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5520102a20 0x7f5520197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.904+0000 7f5525c37640 1 -- 172.21.15.27:0/2269064935 >> 172.21.15.27:0/2269064935 conn(0x7f55200777a0 msgr2=0x7f55200fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.904+0000 7f5525c37640 1 -- 172.21.15.27:0/2269064935 shutdown_connections 2024-04-25T00:26:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:04.904+0000 7f5525c37640 1 -- 172.21.15.27:0/2269064935 wait complete. 2024-04-25T00:26:04.915 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:04 smithi027 ceph-mon[29208]: pgmap v312: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T00:26:04.916 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:08:34.006973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T00:13:17.337307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T00:13:18.253713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T00:13:21.995987+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T00:26:04.917 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:26:05.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.020+0000 7f2f9d44d640 1 -- 172.21.15.27:0/126218260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f98107720 msgr2=0x7f2f98107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.020+0000 7f2f9d44d640 1 --2- 172.21.15.27:0/126218260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f98107720 0x7f2f98107b00 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f2f88009930 tx=0x7f2f8802a420 comp rx=0 tx=0).stop 2024-04-25T00:26:05.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.020+0000 7f2f9d44d640 1 -- 172.21.15.27:0/126218260 shutdown_connections 2024-04-25T00:26:05.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.020+0000 7f2f9d44d640 1 --2- 172.21.15.27:0/126218260 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f981020d0 0x7f2f9810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.020+0000 7f2f9d44d640 1 --2- 172.21.15.27:0/126218260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f98101710 0x7f2f98101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.020+0000 7f2f9d44d640 1 --2- 172.21.15.27:0/126218260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f98107720 0x7f2f98107b00 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.020+0000 7f2f9d44d640 1 -- 172.21.15.27:0/126218260 >> 172.21.15.27:0/126218260 conn(0x7f2f980fd180 msgr2=0x7f2f980ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:05.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.020+0000 7f2f9d44d640 1 -- 172.21.15.27:0/126218260 shutdown_connections 2024-04-25T00:26:05.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.020+0000 7f2f9d44d640 1 -- 172.21.15.27:0/126218260 wait complete. 2024-04-25T00:26:05.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.021+0000 7f2f9d44d640 1 Processor -- start 2024-04-25T00:26:05.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.021+0000 7f2f9d44d640 1 -- start start 2024-04-25T00:26:05.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.021+0000 7f2f9d44d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f98101710 0x7f2f98198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.021+0000 7f2f9d44d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f981020d0 0x7f2f981985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.021+0000 7f2f9d44d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f98107720 0x7f2f9819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.021+0000 7f2f9d44d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2f98111b30 con 0x7f2f981020d0 2024-04-25T00:26:05.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.021+0000 7f2f9d44d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2f981119b0 con 0x7f2f98107720 2024-04-25T00:26:05.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.021+0000 7f2f9d44d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2f98111cb0 con 0x7f2f98101710 2024-04-25T00:26:05.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.021+0000 7f2f8ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f981020d0 0x7f2f981985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.021+0000 7f2f8ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f981020d0 0x7f2f981985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54614/0 (socket says 172.21.15.27:54614) 2024-04-25T00:26:05.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.021+0000 7f2f8ffff640 1 -- 172.21.15.27:0/1088497183 learned_addr learned my addr 172.21.15.27:0/1088497183 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:05.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.021+0000 7f2f96ffd640 1 --2- 172.21.15.27:0/1088497183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f98101710 0x7f2f98198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.021+0000 7f2f977fe640 1 --2- 172.21.15.27:0/1088497183 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f98107720 0x7f2f9819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.022+0000 7f2f8ffff640 1 -- 172.21.15.27:0/1088497183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f98101710 msgr2=0x7f2f98198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.022+0000 7f2f8ffff640 1 --2- 172.21.15.27:0/1088497183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f98101710 0x7f2f98198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.022+0000 7f2f8ffff640 1 -- 172.21.15.27:0/1088497183 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f98107720 msgr2=0x7f2f9819c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.022+0000 7f2f8ffff640 1 --2- 172.21.15.27:0/1088497183 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f98107720 0x7f2f9819c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.022+0000 7f2f8ffff640 1 -- 172.21.15.27:0/1088497183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9819d060 con 0x7f2f981020d0 2024-04-25T00:26:05.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.022+0000 7f2f96ffd640 1 --2- 172.21.15.27:0/1088497183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f98101710 0x7f2f98198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:26:05.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.022+0000 7f2f977fe640 1 --2- 172.21.15.27:0/1088497183 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f98107720 0x7f2f9819c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:26:05.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.022+0000 7f2f8ffff640 1 --2- 172.21.15.27:0/1088497183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f981020d0 0x7f2f981985b0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f2f8000d770 tx=0x7f2f8000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:05.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.022+0000 7f2f94ff9640 1 -- 172.21.15.27:0/1088497183 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f80017070 con 0x7f2f981020d0 2024-04-25T00:26:05.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.022+0000 7f2f94ff9640 1 -- 172.21.15.27:0/1088497183 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f80005500 con 0x7f2f981020d0 2024-04-25T00:26:05.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.022+0000 7f2f9d44d640 1 -- 172.21.15.27:0/1088497183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f9819d350 con 0x7f2f981020d0 2024-04-25T00:26:05.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.022+0000 7f2f94ff9640 1 -- 172.21.15.27:0/1088497183 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f80004e00 con 0x7f2f981020d0 2024-04-25T00:26:05.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.022+0000 7f2f9d44d640 1 -- 172.21.15.27:0/1088497183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f981a4be0 con 0x7f2f981020d0 2024-04-25T00:26:05.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.023+0000 7f2f9d44d640 1 -- 172.21.15.27:0/1088497183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f980689b0 con 0x7f2f981020d0 2024-04-25T00:26:05.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.024+0000 7f2f94ff9640 1 -- 172.21.15.27:0/1088497183 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2f80004fa0 con 0x7f2f981020d0 2024-04-25T00:26:05.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.025+0000 7f2f94ff9640 1 --2- 172.21.15.27:0/1088497183 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2f840a2d00 0x7f2f840a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.025+0000 7f2f96ffd640 1 --2- 172.21.15.27:0/1088497183 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2f840a2d00 0x7f2f840a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.025+0000 7f2f94ff9640 1 -- 172.21.15.27:0/1088497183 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f2f800cf2f0 con 0x7f2f981020d0 2024-04-25T00:26:05.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.025+0000 7f2f96ffd640 1 --2- 172.21.15.27:0/1088497183 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2f840a2d00 0x7f2f840a51c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2f88000c00 tx=0x7f2f88005a20 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:05.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.029+0000 7f2f94ff9640 1 -- 172.21.15.27:0/1088497183 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2f8009bba0 con 0x7f2f981020d0 2024-04-25T00:26:05.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.213+0000 7f2f9d44d640 1 -- 172.21.15.27:0/1088497183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2f981096f0 con 0x7f2f981020d0 2024-04-25T00:26:05.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.214+0000 7f2f94ff9640 1 -- 172.21.15.27:0/1088497183 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f2f8009b4a0 con 0x7f2f981020d0 2024-04-25T00:26:05.215 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T00:26:05.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.216+0000 7f2f9d44d640 1 -- 172.21.15.27:0/1088497183 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2f840a2d00 msgr2=0x7f2f840a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.216+0000 7f2f9d44d640 1 --2- 172.21.15.27:0/1088497183 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2f840a2d00 0x7f2f840a51c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2f88000c00 tx=0x7f2f88005a20 comp rx=0 tx=0).stop 2024-04-25T00:26:05.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.216+0000 7f2f9d44d640 1 -- 172.21.15.27:0/1088497183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f981020d0 msgr2=0x7f2f981985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.216+0000 7f2f9d44d640 1 --2- 172.21.15.27:0/1088497183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f981020d0 0x7f2f981985b0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f2f8000d770 tx=0x7f2f8000dc30 comp rx=0 tx=0).stop 2024-04-25T00:26:05.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.216+0000 7f2f9d44d640 1 -- 172.21.15.27:0/1088497183 shutdown_connections 2024-04-25T00:26:05.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.216+0000 7f2f9d44d640 1 --2- 172.21.15.27:0/1088497183 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2f840a2d00 0x7f2f840a51c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.216+0000 7f2f9d44d640 1 --2- 172.21.15.27:0/1088497183 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f98107720 0x7f2f9819c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.216+0000 7f2f9d44d640 1 --2- 172.21.15.27:0/1088497183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f981020d0 0x7f2f981985b0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.216+0000 7f2f9d44d640 1 --2- 172.21.15.27:0/1088497183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f98101710 0x7f2f98198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.216+0000 7f2f9d44d640 1 -- 172.21.15.27:0/1088497183 >> 172.21.15.27:0/1088497183 conn(0x7f2f980fd180 msgr2=0x7f2f980fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:05.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.216+0000 7f2f9d44d640 1 -- 172.21.15.27:0/1088497183 shutdown_connections 2024-04-25T00:26:05.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.216+0000 7f2f9d44d640 1 -- 172.21.15.27:0/1088497183 wait complete. 2024-04-25T00:26:05.228 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T00:26:05.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:04 smithi139 ceph-mon[32833]: pgmap v312: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T00:26:05.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:04 smithi153 ceph-mon[32896]: pgmap v312: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T00:26:05.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.332+0000 7fd732bc4640 1 -- 172.21.15.27:0/2880952944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd72c100620 msgr2=0x7fd72c100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.332+0000 7fd732bc4640 1 --2- 172.21.15.27:0/2880952944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd72c100620 0x7fd72c100aa0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fd720009930 tx=0x7fd72002a420 comp rx=0 tx=0).stop 2024-04-25T00:26:05.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.332+0000 7fd732bc4640 1 -- 172.21.15.27:0/2880952944 shutdown_connections 2024-04-25T00:26:05.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.332+0000 7fd732bc4640 1 --2- 172.21.15.27:0/2880952944 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd72c100fe0 0x7fd72c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.332+0000 7fd732bc4640 1 --2- 172.21.15.27:0/2880952944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd72c100620 0x7fd72c100aa0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.332+0000 7fd732bc4640 1 --2- 172.21.15.27:0/2880952944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd72c106630 0x7fd72c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.332+0000 7fd732bc4640 1 -- 172.21.15.27:0/2880952944 >> 172.21.15.27:0/2880952944 conn(0x7fd72c0776a0 msgr2=0x7fd72c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:05.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.332+0000 7fd732bc4640 1 -- 172.21.15.27:0/2880952944 shutdown_connections 2024-04-25T00:26:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.332+0000 7fd732bc4640 1 -- 172.21.15.27:0/2880952944 wait complete. 2024-04-25T00:26:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.333+0000 7fd732bc4640 1 Processor -- start 2024-04-25T00:26:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.333+0000 7fd732bc4640 1 -- start start 2024-04-25T00:26:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.333+0000 7fd732bc4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd72c100620 0x7fd72c1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.333+0000 7fd732bc4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd72c100fe0 0x7fd72c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.333+0000 7fd732bc4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd72c106630 0x7fd72c19c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.333+0000 7fd732bc4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c111b40 con 0x7fd72c100620 2024-04-25T00:26:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.333+0000 7fd732bc4640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c1119c0 con 0x7fd72c106630 2024-04-25T00:26:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.333+0000 7fd732bc4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c111cc0 con 0x7fd72c100fe0 2024-04-25T00:26:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.333+0000 7fd730960640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd72c100620 0x7fd72c1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.333+0000 7fd730960640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd72c100620 0x7fd72c1980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54630/0 (socket says 172.21.15.27:54630) 2024-04-25T00:26:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.333+0000 7fd730960640 1 -- 172.21.15.27:0/2357566177 learned_addr learned my addr 172.21.15.27:0/2357566177 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:05.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.334+0000 7fd72bfff640 1 --2- 172.21.15.27:0/2357566177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd72c100fe0 0x7fd72c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.334+0000 7fd731161640 1 --2- 172.21.15.27:0/2357566177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd72c106630 0x7fd72c19c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.334+0000 7fd730960640 1 -- 172.21.15.27:0/2357566177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd72c100fe0 msgr2=0x7fd72c1985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.334+0000 7fd730960640 1 --2- 172.21.15.27:0/2357566177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd72c100fe0 0x7fd72c1985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.334+0000 7fd730960640 1 -- 172.21.15.27:0/2357566177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd72c106630 msgr2=0x7fd72c19c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.334+0000 7fd730960640 1 --2- 172.21.15.27:0/2357566177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd72c106630 0x7fd72c19c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.334+0000 7fd730960640 1 -- 172.21.15.27:0/2357566177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c19d090 con 0x7fd72c100620 2024-04-25T00:26:05.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.334+0000 7fd731161640 1 --2- 172.21.15.27:0/2357566177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd72c106630 0x7fd72c19c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:26:05.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.334+0000 7fd730960640 1 --2- 172.21.15.27:0/2357566177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd72c100620 0x7fd72c1980a0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00d950 tx=0x7fd71c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:05.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.334+0000 7fd729ffb640 1 -- 172.21.15.27:0/2357566177 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c017030 con 0x7fd72c100620 2024-04-25T00:26:05.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.334+0000 7fd729ffb640 1 -- 172.21.15.27:0/2357566177 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd71c004020 con 0x7fd72c100620 2024-04-25T00:26:05.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.334+0000 7fd732bc4640 1 -- 172.21.15.27:0/2357566177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd72c19d380 con 0x7fd72c100620 2024-04-25T00:26:05.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.334+0000 7fd729ffb640 1 -- 172.21.15.27:0/2357566177 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c0047a0 con 0x7fd72c100620 2024-04-25T00:26:05.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.334+0000 7fd732bc4640 1 -- 172.21.15.27:0/2357566177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd72c1a4c10 con 0x7fd72c100620 2024-04-25T00:26:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.335+0000 7fd732bc4640 1 -- 172.21.15.27:0/2357566177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd72c0689b0 con 0x7fd72c100620 2024-04-25T00:26:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.336+0000 7fd729ffb640 1 -- 172.21.15.27:0/2357566177 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd71c0041d0 con 0x7fd72c100620 2024-04-25T00:26:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.337+0000 7fd729ffb640 1 --2- 172.21.15.27:0/2357566177 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd7140a2d00 0x7fd7140a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.337+0000 7fd72bfff640 1 --2- 172.21.15.27:0/2357566177 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd7140a2d00 0x7fd7140a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.337+0000 7fd729ffb640 1 -- 172.21.15.27:0/2357566177 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fd71c0cf1d0 con 0x7fd72c100620 2024-04-25T00:26:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.338+0000 7fd72bfff640 1 --2- 172.21.15.27:0/2357566177 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd7140a2d00 0x7fd7140a51c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd720000c00 tx=0x7fd720005a20 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:05.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.342+0000 7fd729ffb640 1 -- 172.21.15.27:0/2357566177 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd71c09ba80 con 0x7fd72c100620 2024-04-25T00:26:05.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.527+0000 7fd732bc4640 1 -- 172.21.15.27:0/2357566177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd72c108be0 con 0x7fd72c100620 2024-04-25T00:26:05.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.528+0000 7fd729ffb640 1 -- 172.21.15.27:0/2357566177 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fd71c09b380 con 0x7fd72c100620 2024-04-25T00:26:05.529 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:26:05.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.530+0000 7fd732bc4640 1 -- 172.21.15.27:0/2357566177 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd7140a2d00 msgr2=0x7fd7140a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.530+0000 7fd732bc4640 1 --2- 172.21.15.27:0/2357566177 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd7140a2d00 0x7fd7140a51c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd720000c00 tx=0x7fd720005a20 comp rx=0 tx=0).stop 2024-04-25T00:26:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.530+0000 7fd732bc4640 1 -- 172.21.15.27:0/2357566177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd72c100620 msgr2=0x7fd72c1980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.530+0000 7fd732bc4640 1 --2- 172.21.15.27:0/2357566177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd72c100620 0x7fd72c1980a0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00d950 tx=0x7fd71c00de10 comp rx=0 tx=0).stop 2024-04-25T00:26:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.530+0000 7fd732bc4640 1 -- 172.21.15.27:0/2357566177 shutdown_connections 2024-04-25T00:26:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.530+0000 7fd732bc4640 1 --2- 172.21.15.27:0/2357566177 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd7140a2d00 0x7fd7140a51c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.530+0000 7fd732bc4640 1 --2- 172.21.15.27:0/2357566177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd72c106630 0x7fd72c19c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.530+0000 7fd732bc4640 1 --2- 172.21.15.27:0/2357566177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd72c100fe0 0x7fd72c1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.530+0000 7fd732bc4640 1 --2- 172.21.15.27:0/2357566177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd72c100620 0x7fd72c1980a0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.530+0000 7fd732bc4640 1 -- 172.21.15.27:0/2357566177 >> 172.21.15.27:0/2357566177 conn(0x7fd72c0776a0 msgr2=0x7fd72c103290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.530+0000 7fd732bc4640 1 -- 172.21.15.27:0/2357566177 shutdown_connections 2024-04-25T00:26:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.530+0000 7fd732bc4640 1 -- 172.21.15.27:0/2357566177 wait complete. 2024-04-25T00:26:05.542 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T00:26:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.647+0000 7fbacfb14640 1 -- 172.21.15.27:0/2905231286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac81012e0 msgr2=0x7fbac810d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.647+0000 7fbacfb14640 1 --2- 172.21.15.27:0/2905231286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac81012e0 0x7fbac810d4b0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fbac0009cd0 tx=0x7fbac002a8e0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.647+0000 7fbacfb14640 1 -- 172.21.15.27:0/2905231286 shutdown_connections 2024-04-25T00:26:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.647+0000 7fbacfb14640 1 --2- 172.21.15.27:0/2905231286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac81012e0 0x7fbac810d4b0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.647+0000 7fbacfb14640 1 --2- 172.21.15.27:0/2905231286 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbac8100920 0x7fbac8100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.647+0000 7fbacfb14640 1 --2- 172.21.15.27:0/2905231286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbac8106810 0x7fbac8106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.647+0000 7fbacfb14640 1 -- 172.21.15.27:0/2905231286 >> 172.21.15.27:0/2905231286 conn(0x7fbac8077620 msgr2=0x7fbac8077a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.647+0000 7fbacfb14640 1 -- 172.21.15.27:0/2905231286 shutdown_connections 2024-04-25T00:26:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.647+0000 7fbacfb14640 1 -- 172.21.15.27:0/2905231286 wait complete. 2024-04-25T00:26:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.648+0000 7fbacfb14640 1 Processor -- start 2024-04-25T00:26:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.648+0000 7fbacfb14640 1 -- start start 2024-04-25T00:26:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.648+0000 7fbacfb14640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac8100920 0x7fbac8198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.648+0000 7fbacfb14640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbac81012e0 0x7fbac8199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.648+0000 7fbacfb14640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbac8106810 0x7fbac81a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.648+0000 7fbacfb14640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbac8110b30 con 0x7fbac8100920 2024-04-25T00:26:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.648+0000 7fbacfb14640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbac81109b0 con 0x7fbac8106810 2024-04-25T00:26:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.648+0000 7fbacfb14640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbac8110cb0 con 0x7fbac81012e0 2024-04-25T00:26:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.648+0000 7fbacd8b0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac8100920 0x7fbac8198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.648+0000 7fbacd8b0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac8100920 0x7fbac8198f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54642/0 (socket says 172.21.15.27:54642) 2024-04-25T00:26:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.648+0000 7fbacd8b0640 1 -- 172.21.15.27:0/1283040265 learned_addr learned my addr 172.21.15.27:0/1283040265 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.648+0000 7fbacd8b0640 1 -- 172.21.15.27:0/1283040265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbac81012e0 msgr2=0x7fbac8199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.648+0000 7fbacd8b0640 1 --2- 172.21.15.27:0/1283040265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbac81012e0 0x7fbac8199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.648+0000 7fbacd8b0640 1 -- 172.21.15.27:0/1283040265 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbac8106810 msgr2=0x7fbac81a3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.648+0000 7fbacd8b0640 1 --2- 172.21.15.27:0/1283040265 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbac8106810 0x7fbac81a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.648+0000 7fbacd8b0640 1 -- 172.21.15.27:0/1283040265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac81a44d0 con 0x7fbac8100920 2024-04-25T00:26:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.649+0000 7fbacd8b0640 1 --2- 172.21.15.27:0/1283040265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac8100920 0x7fbac8198f10 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fbac400d770 tx=0x7fbac400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.649+0000 7fbabeffd640 1 -- 172.21.15.27:0/1283040265 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac4017070 con 0x7fbac8100920 2024-04-25T00:26:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.649+0000 7fbabeffd640 1 -- 172.21.15.27:0/1283040265 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbac4005500 con 0x7fbac8100920 2024-04-25T00:26:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.649+0000 7fbacfb14640 1 -- 172.21.15.27:0/1283040265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbac81a46a0 con 0x7fbac8100920 2024-04-25T00:26:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.649+0000 7fbabeffd640 1 -- 172.21.15.27:0/1283040265 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac4004e00 con 0x7fbac8100920 2024-04-25T00:26:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.649+0000 7fbacfb14640 1 -- 172.21.15.27:0/1283040265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbac81a4b10 con 0x7fbac8100920 2024-04-25T00:26:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.650+0000 7fbacfb14640 1 -- 172.21.15.27:0/1283040265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbac80689b0 con 0x7fbac8100920 2024-04-25T00:26:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.651+0000 7fbabeffd640 1 -- 172.21.15.27:0/1283040265 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fbac4004fa0 con 0x7fbac8100920 2024-04-25T00:26:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.652+0000 7fbabeffd640 1 --2- 172.21.15.27:0/1283040265 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fbab40a2d00 0x7fbab40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.652+0000 7fbacd0af640 1 --2- 172.21.15.27:0/1283040265 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fbab40a2d00 0x7fbab40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.652+0000 7fbabeffd640 1 -- 172.21.15.27:0/1283040265 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fbac40cf290 con 0x7fbac8100920 2024-04-25T00:26:05.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.652+0000 7fbacd0af640 1 --2- 172.21.15.27:0/1283040265 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fbab40a2d00 0x7fbab40a51c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbab8005fd0 tx=0x7fbab8005c70 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:05.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.656+0000 7fbabeffd640 1 -- 172.21.15.27:0/1283040265 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbac409bb40 con 0x7fbac8100920 2024-04-25T00:26:05.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.841+0000 7fbacfb14640 1 -- 172.21.15.27:0/1283040265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbac8108570 con 0x7fbac8100920 2024-04-25T00:26:05.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.841+0000 7fbabeffd640 1 -- 172.21.15.27:0/1283040265 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fbac409b440 con 0x7fbac8100920 2024-04-25T00:26:05.843 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T00:26:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.843+0000 7fbacfb14640 1 -- 172.21.15.27:0/1283040265 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fbab40a2d00 msgr2=0x7fbab40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.843+0000 7fbacfb14640 1 --2- 172.21.15.27:0/1283040265 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fbab40a2d00 0x7fbab40a51c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbab8005fd0 tx=0x7fbab8005c70 comp rx=0 tx=0).stop 2024-04-25T00:26:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.843+0000 7fbacfb14640 1 -- 172.21.15.27:0/1283040265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac8100920 msgr2=0x7fbac8198f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.843+0000 7fbacfb14640 1 --2- 172.21.15.27:0/1283040265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac8100920 0x7fbac8198f10 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fbac400d770 tx=0x7fbac400dc30 comp rx=0 tx=0).stop 2024-04-25T00:26:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.843+0000 7fbacfb14640 1 -- 172.21.15.27:0/1283040265 shutdown_connections 2024-04-25T00:26:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.843+0000 7fbacfb14640 1 --2- 172.21.15.27:0/1283040265 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fbab40a2d00 0x7fbab40a51c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.843+0000 7fbacfb14640 1 --2- 172.21.15.27:0/1283040265 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbac8106810 0x7fbac81a3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.843+0000 7fbacfb14640 1 --2- 172.21.15.27:0/1283040265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbac81012e0 0x7fbac8199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.843+0000 7fbacfb14640 1 --2- 172.21.15.27:0/1283040265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac8100920 0x7fbac8198f10 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.843+0000 7fbacfb14640 1 -- 172.21.15.27:0/1283040265 >> 172.21.15.27:0/1283040265 conn(0x7fbac8077620 msgr2=0x7fbac80ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:05.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.844+0000 7fbacfb14640 1 -- 172.21.15.27:0/1283040265 shutdown_connections 2024-04-25T00:26:05.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.844+0000 7fbacfb14640 1 -- 172.21.15.27:0/1283040265 wait complete. 2024-04-25T00:26:05.856 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T00:26:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.960+0000 7fb649f90640 1 -- 172.21.15.27:0/1325249170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb644069dc0 msgr2=0x7fb64406a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.960+0000 7fb649f90640 1 --2- 172.21.15.27:0/1325249170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb644069dc0 0x7fb64406a240 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fb638009930 tx=0x7fb63802a420 comp rx=0 tx=0).stop 2024-04-25T00:26:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.960+0000 7fb649f90640 1 -- 172.21.15.27:0/1325249170 shutdown_connections 2024-04-25T00:26:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.960+0000 7fb649f90640 1 --2- 172.21.15.27:0/1325249170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb64406a780 0x7fb64410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.960+0000 7fb649f90640 1 --2- 172.21.15.27:0/1325249170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb644069dc0 0x7fb64406a240 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.960+0000 7fb649f90640 1 --2- 172.21.15.27:0/1325249170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb644107730 0x7fb644107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.960+0000 7fb649f90640 1 -- 172.21.15.27:0/1325249170 >> 172.21.15.27:0/1325249170 conn(0x7fb6440776a0 msgr2=0x7fb644077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.960+0000 7fb649f90640 1 -- 172.21.15.27:0/1325249170 shutdown_connections 2024-04-25T00:26:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.960+0000 7fb649f90640 1 -- 172.21.15.27:0/1325249170 wait complete. 2024-04-25T00:26:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.961+0000 7fb649f90640 1 Processor -- start 2024-04-25T00:26:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.961+0000 7fb649f90640 1 -- start start 2024-04-25T00:26:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.961+0000 7fb649f90640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb644069dc0 0x7fb644100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.961+0000 7fb649f90640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb64406a780 0x7fb644100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.961+0000 7fb649f90640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb644107730 0x7fb6441010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.961+0000 7fb649f90640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb644111b00 con 0x7fb64406a780 2024-04-25T00:26:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.961+0000 7fb649f90640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb644111980 con 0x7fb644107730 2024-04-25T00:26:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.961+0000 7fb649f90640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb644111c80 con 0x7fb644069dc0 2024-04-25T00:26:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.961+0000 7fb642ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb64406a780 0x7fb644100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.962+0000 7fb642ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb64406a780 0x7fb644100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54662/0 (socket says 172.21.15.27:54662) 2024-04-25T00:26:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.962+0000 7fb642ffd640 1 -- 172.21.15.27:0/259351867 learned_addr learned my addr 172.21.15.27:0/259351867 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:05.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.962+0000 7fb6437fe640 1 --2- 172.21.15.27:0/259351867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb644069dc0 0x7fb644100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.962+0000 7fb643fff640 1 --2- 172.21.15.27:0/259351867 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb644107730 0x7fb6441010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.962+0000 7fb642ffd640 1 -- 172.21.15.27:0/259351867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb644069dc0 msgr2=0x7fb644100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.962+0000 7fb642ffd640 1 --2- 172.21.15.27:0/259351867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb644069dc0 0x7fb644100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.962+0000 7fb642ffd640 1 -- 172.21.15.27:0/259351867 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb644107730 msgr2=0x7fb6441010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.962+0000 7fb642ffd640 1 --2- 172.21.15.27:0/259351867 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb644107730 0x7fb6441010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.962+0000 7fb642ffd640 1 -- 172.21.15.27:0/259351867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6441a6560 con 0x7fb64406a780 2024-04-25T00:26:05.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.962+0000 7fb642ffd640 1 --2- 172.21.15.27:0/259351867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb64406a780 0x7fb644100bb0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fb638009230 tx=0x7fb638004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:05.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.962+0000 7fb640ff9640 1 -- 172.21.15.27:0/259351867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63803f030 con 0x7fb64406a780 2024-04-25T00:26:05.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.962+0000 7fb649f90640 1 -- 172.21.15.27:0/259351867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6441a67f0 con 0x7fb64406a780 2024-04-25T00:26:05.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.962+0000 7fb640ff9640 1 -- 172.21.15.27:0/259351867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb638003ad0 con 0x7fb64406a780 2024-04-25T00:26:05.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.962+0000 7fb640ff9640 1 -- 172.21.15.27:0/259351867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb638035070 con 0x7fb64406a780 2024-04-25T00:26:05.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.962+0000 7fb649f90640 1 -- 172.21.15.27:0/259351867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6441a6d30 con 0x7fb64406a780 2024-04-25T00:26:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.963+0000 7fb649f90640 1 -- 172.21.15.27:0/259351867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6440689b0 con 0x7fb64406a780 2024-04-25T00:26:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.965+0000 7fb640ff9640 1 -- 172.21.15.27:0/259351867 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb638042030 con 0x7fb64406a780 2024-04-25T00:26:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.966+0000 7fb640ff9640 1 --2- 172.21.15.27:0/259351867 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb62c0a2d00 0x7fb62c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.966+0000 7fb640ff9640 1 -- 172.21.15.27:0/259351867 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fb6380efd50 con 0x7fb64406a780 2024-04-25T00:26:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.966+0000 7fb6437fe640 1 --2- 172.21.15.27:0/259351867 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb62c0a2d00 0x7fb62c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.966+0000 7fb6437fe640 1 --2- 172.21.15.27:0/259351867 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb62c0a2d00 0x7fb62c0a51c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb634005d00 tx=0x7fb63400b410 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:05.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:05.969+0000 7fb640ff9640 1 -- 172.21.15.27:0/259351867 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb6380bc600 con 0x7fb64406a780 2024-04-25T00:26:06.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.153+0000 7fb649f90640 1 -- 172.21.15.27:0/259351867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb644109cb0 con 0x7fb64406a780 2024-04-25T00:26:06.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.154+0000 7fb640ff9640 1 -- 172.21.15.27:0/259351867 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fb6380bbf00 con 0x7fb64406a780 2024-04-25T00:26:06.155 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:26:06.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.156+0000 7fb649f90640 1 -- 172.21.15.27:0/259351867 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb62c0a2d00 msgr2=0x7fb62c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:06.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.156+0000 7fb649f90640 1 --2- 172.21.15.27:0/259351867 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb62c0a2d00 0x7fb62c0a51c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb634005d00 tx=0x7fb63400b410 comp rx=0 tx=0).stop 2024-04-25T00:26:06.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.156+0000 7fb649f90640 1 -- 172.21.15.27:0/259351867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb64406a780 msgr2=0x7fb644100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:06.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.156+0000 7fb649f90640 1 --2- 172.21.15.27:0/259351867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb64406a780 0x7fb644100bb0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fb638009230 tx=0x7fb638004270 comp rx=0 tx=0).stop 2024-04-25T00:26:06.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.156+0000 7fb649f90640 1 -- 172.21.15.27:0/259351867 shutdown_connections 2024-04-25T00:26:06.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.156+0000 7fb649f90640 1 --2- 172.21.15.27:0/259351867 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fb62c0a2d00 0x7fb62c0a51c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:06.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.156+0000 7fb649f90640 1 --2- 172.21.15.27:0/259351867 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb644107730 0x7fb6441010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:06.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.156+0000 7fb649f90640 1 --2- 172.21.15.27:0/259351867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb64406a780 0x7fb644100bb0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:06.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.156+0000 7fb649f90640 1 --2- 172.21.15.27:0/259351867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb644069dc0 0x7fb644100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:06.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.156+0000 7fb649f90640 1 -- 172.21.15.27:0/259351867 >> 172.21.15.27:0/259351867 conn(0x7fb6440776a0 msgr2=0x7fb6440fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:06.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.156+0000 7fb649f90640 1 -- 172.21.15.27:0/259351867 shutdown_connections 2024-04-25T00:26:06.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.156+0000 7fb649f90640 1 -- 172.21.15.27:0/259351867 wait complete. 2024-04-25T00:26:06.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:05 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2269064935' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:26:06.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:05 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1088497183' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:26:06.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:05 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2357566177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:26:06.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:05 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1283040265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:26:06.168 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-25T00:26:06.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:05 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2269064935' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:26:06.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:05 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1088497183' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:26:06.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:05 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2357566177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:26:06.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:05 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1283040265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:26:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.270+0000 7fd34b25d640 1 -- 172.21.15.27:0/4163571849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd344107720 msgr2=0x7fd344107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.270+0000 7fd34b25d640 1 --2- 172.21.15.27:0/4163571849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd344107720 0x7fd344107b00 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fd32c009930 tx=0x7fd32c02a420 comp rx=0 tx=0).stop 2024-04-25T00:26:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.270+0000 7fd34b25d640 1 -- 172.21.15.27:0/4163571849 shutdown_connections 2024-04-25T00:26:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.270+0000 7fd34b25d640 1 --2- 172.21.15.27:0/4163571849 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3441020d0 0x7fd34410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.270+0000 7fd34b25d640 1 --2- 172.21.15.27:0/4163571849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd344101710 0x7fd344101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.270+0000 7fd34b25d640 1 --2- 172.21.15.27:0/4163571849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd344107720 0x7fd344107b00 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.270+0000 7fd34b25d640 1 -- 172.21.15.27:0/4163571849 >> 172.21.15.27:0/4163571849 conn(0x7fd344078b00 msgr2=0x7fd344078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.270+0000 7fd34b25d640 1 -- 172.21.15.27:0/4163571849 shutdown_connections 2024-04-25T00:26:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.270+0000 7fd34b25d640 1 -- 172.21.15.27:0/4163571849 wait complete. 2024-04-25T00:26:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd34b25d640 1 Processor -- start 2024-04-25T00:26:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd34b25d640 1 -- start start 2024-04-25T00:26:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd34b25d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd344101710 0x7fd344198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd34b25d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3441020d0 0x7fd344198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd34b25d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd344107720 0x7fd34419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd34b25d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd344111b30 con 0x7fd3441020d0 2024-04-25T00:26:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd34b25d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd3441119b0 con 0x7fd344101710 2024-04-25T00:26:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd34b25d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd344111cb0 con 0x7fd344107720 2024-04-25T00:26:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd343fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3441020d0 0x7fd344198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd343fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3441020d0 0x7fd344198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54682/0 (socket says 172.21.15.27:54682) 2024-04-25T00:26:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd343fff640 1 -- 172.21.15.27:0/3795245309 learned_addr learned my addr 172.21.15.27:0/3795245309 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd348ff9640 1 --2- 172.21.15.27:0/3795245309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd344101710 0x7fd344198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd3497fa640 1 --2- 172.21.15.27:0/3795245309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd344107720 0x7fd34419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd343fff640 1 -- 172.21.15.27:0/3795245309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd344107720 msgr2=0x7fd34419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd343fff640 1 --2- 172.21.15.27:0/3795245309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd344107720 0x7fd34419c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd343fff640 1 -- 172.21.15.27:0/3795245309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd344101710 msgr2=0x7fd344198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd343fff640 1 --2- 172.21.15.27:0/3795245309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd344101710 0x7fd344198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd343fff640 1 -- 172.21.15.27:0/3795245309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd34419cfd0 con 0x7fd3441020d0 2024-04-25T00:26:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd348ff9640 1 --2- 172.21.15.27:0/3795245309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd344101710 0x7fd344198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:26:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.271+0000 7fd343fff640 1 --2- 172.21.15.27:0/3795245309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3441020d0 0x7fd344198570 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fd33400c970 tx=0x7fd33400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.272+0000 7fd341ffb640 1 -- 172.21.15.27:0/3795245309 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd334007bb0 con 0x7fd3441020d0 2024-04-25T00:26:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.272+0000 7fd34b25d640 1 -- 172.21.15.27:0/3795245309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd34419d2c0 con 0x7fd3441020d0 2024-04-25T00:26:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.272+0000 7fd341ffb640 1 -- 172.21.15.27:0/3795245309 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd334007d50 con 0x7fd3441020d0 2024-04-25T00:26:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.272+0000 7fd341ffb640 1 -- 172.21.15.27:0/3795245309 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd33401d060 con 0x7fd3441020d0 2024-04-25T00:26:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.272+0000 7fd34b25d640 1 -- 172.21.15.27:0/3795245309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3441a4b50 con 0x7fd3441020d0 2024-04-25T00:26:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.273+0000 7fd34b25d640 1 -- 172.21.15.27:0/3795245309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3440689b0 con 0x7fd3441020d0 2024-04-25T00:26:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.274+0000 7fd341ffb640 1 -- 172.21.15.27:0/3795245309 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd334004f40 con 0x7fd3441020d0 2024-04-25T00:26:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.274+0000 7fd341ffb640 1 --2- 172.21.15.27:0/3795245309 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd3300a2d00 0x7fd3300a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.274+0000 7fd348ff9640 1 --2- 172.21.15.27:0/3795245309 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd3300a2d00 0x7fd3300a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.275+0000 7fd341ffb640 1 -- 172.21.15.27:0/3795245309 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7fd3340cf290 con 0x7fd3441020d0 2024-04-25T00:26:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.275+0000 7fd348ff9640 1 --2- 172.21.15.27:0/3795245309 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd3300a2d00 0x7fd3300a51c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd32c000c00 tx=0x7fd32c005a20 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:06.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:05 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2269064935' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:26:06.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:05 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1088497183' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:26:06.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:05 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2357566177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:26:06.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:05 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1283040265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:26:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.279+0000 7fd341ffb640 1 -- 172.21.15.27:0/3795245309 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd334014030 con 0x7fd3441020d0 2024-04-25T00:26:06.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.513+0000 7fd34b25d640 1 -- 172.21.15.27:0/3795245309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fd344109ce0 con 0x7fd3441020d0 2024-04-25T00:26:06.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.542+0000 7fd341ffb640 1 -- 172.21.15.27:0/3795245309 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+740903 (secure 0 0 0) 0x7fd33409bb40 con 0x7fd3441020d0 2024-04-25T00:26:06.553 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:26:06.553 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:26:06.553 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 26, 2024-04-25T00:26:06.553 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.553 INFO:teuthology.orchestra.run.smithi027.stdout: "active_gid": 15564, 2024-04-25T00:26:06.553 INFO:teuthology.orchestra.run.smithi027.stdout: "active_name": "x", 2024-04-25T00:26:06.554 INFO:teuthology.orchestra.run.smithi027.stdout: "active_addrs": { 2024-04-25T00:26:06.554 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T00:26:06.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.554 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T00:26:06.554 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6800", 2024-04-25T00:26:06.554 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1076858552 2024-04-25T00:26:06.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.554 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-04-25T00:26:06.554 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6801", 2024-04-25T00:26:06.554 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1076858552 2024-04-25T00:26:06.554 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.554 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T00:26:06.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.555 INFO:teuthology.orchestra.run.smithi027.stdout: "active_addr": "172.21.15.27:6801/1076858552", 2024-04-25T00:26:06.555 INFO:teuthology.orchestra.run.smithi027.stdout: "active_change": "2024-04-25T00:15:45.035134+0000", 2024-04-25T00:26:06.555 INFO:teuthology.orchestra.run.smithi027.stdout: "active_mgr_features": 4540701547738038271, 2024-04-25T00:26:06.555 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-04-25T00:26:06.555 INFO:teuthology.orchestra.run.smithi027.stdout: "standbys": [ 2024-04-25T00:26:06.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.555 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 25171, 2024-04-25T00:26:06.555 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "z", 2024-04-25T00:26:06.555 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_features": 4540701547738038271, 2024-04-25T00:26:06.555 INFO:teuthology.orchestra.run.smithi027.stdout: "available_modules": [ 2024-04-25T00:26:06.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.555 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "alerts", 2024-04-25T00:26:06.555 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.555 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.556 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.556 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T00:26:06.556 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T00:26:06.556 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.556 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.556 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.556 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T00:26:06.556 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.556 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.557 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.557 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T00:26:06.557 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.557 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.557 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.557 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.557 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.557 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.558 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.558 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.558 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.558 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.558 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.558 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.558 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.558 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.558 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.558 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.558 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.559 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.559 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.559 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.559 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.559 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.559 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.559 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.559 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.559 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.560 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.560 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.560 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.560 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.560 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.560 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.561 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.561 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.561 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.561 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.561 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.561 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.561 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.561 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.562 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.562 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.562 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.562 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.562 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.562 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.562 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.563 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.563 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.563 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.563 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.563 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.564 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.564 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.564 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.564 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.564 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.565 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.565 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.565 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.565 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.565 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.565 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.565 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.566 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.566 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.566 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.566 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.566 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.566 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_destination": { 2024-04-25T00:26:06.566 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_destination", 2024-04-25T00:26:06.566 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.566 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.566 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.566 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.566 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.566 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.567 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.567 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email address to send alerts to", 2024-04-25T00:26:06.567 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.567 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.567 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.567 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_from_name": { 2024-04-25T00:26:06.567 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_from_name", 2024-04-25T00:26:06.567 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.567 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.567 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.567 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "Ceph", 2024-04-25T00:26:06.568 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.568 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.568 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.568 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email From: name", 2024-04-25T00:26:06.568 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.568 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.568 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.568 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_host": { 2024-04-25T00:26:06.568 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_host", 2024-04-25T00:26:06.568 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.568 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.568 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.568 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.568 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.569 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.569 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.569 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP server", 2024-04-25T00:26:06.569 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.569 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.569 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.569 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_password": { 2024-04-25T00:26:06.569 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_password", 2024-04-25T00:26:06.569 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.569 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.569 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.570 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.570 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.570 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.570 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.570 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Password to authenticate with", 2024-04-25T00:26:06.570 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.570 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.570 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.570 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_port": { 2024-04-25T00:26:06.570 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_port", 2024-04-25T00:26:06.570 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.570 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.570 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.570 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "465", 2024-04-25T00:26:06.571 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.571 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.571 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.571 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP port", 2024-04-25T00:26:06.571 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.571 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.571 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.571 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_sender": { 2024-04-25T00:26:06.571 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_sender", 2024-04-25T00:26:06.571 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.571 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.571 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.571 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.572 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.572 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.572 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.572 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP envelope sender", 2024-04-25T00:26:06.572 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.572 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.572 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.572 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_ssl": { 2024-04-25T00:26:06.572 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_ssl", 2024-04-25T00:26:06.572 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.572 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.572 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.572 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.573 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.573 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.573 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.573 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T00:26:06.573 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.573 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.573 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.573 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_user": { 2024-04-25T00:26:06.573 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_user", 2024-04-25T00:26:06.573 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.573 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.573 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.573 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.573 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.574 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.574 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.574 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "User to authenticate as", 2024-04-25T00:26:06.574 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.574 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.574 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.574 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.574 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.574 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "balancer", 2024-04-25T00:26:06.574 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.574 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.574 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.575 INFO:teuthology.orchestra.run.smithi027.stdout: "active": { 2024-04-25T00:26:06.575 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "active", 2024-04-25T00:26:06.575 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.575 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.575 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.575 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.575 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.575 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.575 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.575 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T00:26:06.575 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.575 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.575 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.576 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_time": { 2024-04-25T00:26:06.576 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_time", 2024-04-25T00:26:06.576 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.576 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.576 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.576 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0000", 2024-04-25T00:26:06.576 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.576 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.576 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.576 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T00:26:06.576 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T00:26:06.576 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.576 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.576 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_weekday": { 2024-04-25T00:26:06.577 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_weekday", 2024-04-25T00:26:06.577 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T00:26:06.577 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.577 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.577 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T00:26:06.577 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T00:26:06.577 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T00:26:06.577 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.577 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T00:26:06.577 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T00:26:06.577 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.577 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.577 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_max_iterations": { 2024-04-25T00:26:06.578 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_max_iterations", 2024-04-25T00:26:06.578 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T00:26:06.578 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.578 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.578 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "25", 2024-04-25T00:26:06.578 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T00:26:06.578 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "250", 2024-04-25T00:26:06.578 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.578 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T00:26:06.578 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.578 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.578 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.578 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_metrics": { 2024-04-25T00:26:06.578 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_metrics", 2024-04-25T00:26:06.579 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.579 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.579 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.579 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T00:26:06.579 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.579 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.579 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.579 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T00:26:06.579 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T00:26:06.579 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.579 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.579 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_step": { 2024-04-25T00:26:06.580 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_step", 2024-04-25T00:26:06.580 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:06.580 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.580 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.580 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.5", 2024-04-25T00:26:06.580 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0.001", 2024-04-25T00:26:06.580 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "0.999", 2024-04-25T00:26:06.580 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.580 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "aggressiveness of optimization", 2024-04-25T00:26:06.580 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T00:26:06.580 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.580 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.580 INFO:teuthology.orchestra.run.smithi027.stdout: "end_time": { 2024-04-25T00:26:06.581 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_time", 2024-04-25T00:26:06.581 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.581 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.581 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.581 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2359", 2024-04-25T00:26:06.581 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.581 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.581 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.581 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T00:26:06.581 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T00:26:06.581 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.582 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.582 INFO:teuthology.orchestra.run.smithi027.stdout: "end_weekday": { 2024-04-25T00:26:06.582 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_weekday", 2024-04-25T00:26:06.582 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T00:26:06.582 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.583 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.583 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T00:26:06.583 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T00:26:06.583 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T00:26:06.583 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.583 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T00:26:06.583 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T00:26:06.584 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.584 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.584 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.584 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.584 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.584 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.584 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.584 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.585 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.585 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.585 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.585 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.585 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.585 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.585 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.585 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.585 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.585 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.585 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.585 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.585 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.585 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.586 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.586 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.586 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.586 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.586 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.586 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.586 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.586 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.587 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.587 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.587 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.587 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.587 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.587 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.587 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.588 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.588 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.588 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.588 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.588 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.588 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.588 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.588 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.588 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.588 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.588 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.588 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.588 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.588 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.588 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.589 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.589 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.589 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.589 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.589 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.589 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.589 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.589 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.589 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.589 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.589 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.589 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.589 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.590 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.590 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.590 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.590 INFO:teuthology.orchestra.run.smithi027.stdout: "min_score": { 2024-04-25T00:26:06.590 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_score", 2024-04-25T00:26:06.590 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:06.590 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.590 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.590 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T00:26:06.590 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.590 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.590 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.590 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T00:26:06.591 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.591 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.591 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.591 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T00:26:06.591 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T00:26:06.591 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.591 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.591 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.591 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "upmap", 2024-04-25T00:26:06.591 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.591 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.591 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.592 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-compat", 2024-04-25T00:26:06.592 INFO:teuthology.orchestra.run.smithi027.stdout: "none", 2024-04-25T00:26:06.592 INFO:teuthology.orchestra.run.smithi027.stdout: "read", 2024-04-25T00:26:06.592 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap", 2024-04-25T00:26:06.592 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap-read" 2024-04-25T00:26:06.592 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.592 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Balancer mode", 2024-04-25T00:26:06.592 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.592 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.592 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.592 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_ids": { 2024-04-25T00:26:06.592 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_ids", 2024-04-25T00:26:06.592 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.592 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.593 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.593 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.593 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.593 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.593 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.593 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T00:26:06.593 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.593 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.593 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.593 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T00:26:06.593 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T00:26:06.593 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.593 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T00:26:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T00:26:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_deviation": { 2024-04-25T00:26:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_deviation", 2024-04-25T00:26:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T00:26:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T00:26:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T00:26:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T00:26:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_optimizations": { 2024-04-25T00:26:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_optimizations", 2024-04-25T00:26:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T00:26:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.596 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:06.596 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.596 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.596 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.596 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T00:26:06.596 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.596 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.596 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.596 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.596 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.596 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm", 2024-04-25T00:26:06.596 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.597 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.597 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.597 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_down_multiplier": { 2024-04-25T00:26:06.597 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_down_multiplier", 2024-04-25T00:26:06.597 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:06.597 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.597 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.597 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T00:26:06.597 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.597 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.597 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.597 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T00:26:06.597 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.597 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.597 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.598 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_refresh_rate": { 2024-04-25T00:26:06.598 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_refresh_rate", 2024-04-25T00:26:06.598 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.598 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.598 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.598 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "20", 2024-04-25T00:26:06.598 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.598 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.598 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.598 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T00:26:06.598 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.598 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.598 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.599 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_starting_port": { 2024-04-25T00:26:06.599 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_starting_port", 2024-04-25T00:26:06.599 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.599 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.599 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.599 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4721", 2024-04-25T00:26:06.599 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.599 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.599 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.599 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T00:26:06.600 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.600 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.600 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.600 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_ptrace": { 2024-04-25T00:26:06.600 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_ptrace", 2024-04-25T00:26:06.600 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.600 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.600 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.600 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.600 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.600 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.600 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.601 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T00:26:06.601 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T00:26:06.601 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.601 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.601 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_interval": { 2024-04-25T00:26:06.601 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_interval", 2024-04-25T00:26:06.601 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.601 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.601 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.601 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T00:26:06.601 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.601 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.601 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.602 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T00:26:06.602 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.602 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.602 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.602 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_memory_target_ratio": { 2024-04-25T00:26:06.602 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T00:26:06.602 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:06.602 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.602 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.602 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.7", 2024-04-25T00:26:06.602 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.602 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.603 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.603 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T00:26:06.603 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.603 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.603 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.603 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm_log_destination": { 2024-04-25T00:26:06.603 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm_log_destination", 2024-04-25T00:26:06.603 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.603 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.603 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.603 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.603 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.603 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.603 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.604 INFO:teuthology.orchestra.run.smithi027.stdout: "file", 2024-04-25T00:26:06.604 INFO:teuthology.orchestra.run.smithi027.stdout: "file,syslog", 2024-04-25T00:26:06.604 INFO:teuthology.orchestra.run.smithi027.stdout: "syslog" 2024-04-25T00:26:06.604 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.604 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T00:26:06.604 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.604 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.604 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.604 INFO:teuthology.orchestra.run.smithi027.stdout: "cgroups_split": { 2024-04-25T00:26:06.604 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cgroups_split", 2024-04-25T00:26:06.604 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.604 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.604 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.605 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.605 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.605 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.605 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.605 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T00:26:06.605 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.605 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.605 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.605 INFO:teuthology.orchestra.run.smithi027.stdout: "config_checks_enabled": { 2024-04-25T00:26:06.605 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_checks_enabled", 2024-04-25T00:26:06.605 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.605 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.605 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.605 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.606 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.606 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.606 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.606 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T00:26:06.606 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.606 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.606 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.606 INFO:teuthology.orchestra.run.smithi027.stdout: "config_dashboard": { 2024-04-25T00:26:06.606 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_dashboard", 2024-04-25T00:26:06.606 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.606 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.606 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.607 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.607 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.607 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.607 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.607 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T00:26:06.607 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.607 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.607 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.607 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_alertmanager": { 2024-04-25T00:26:06.607 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_alertmanager", 2024-04-25T00:26:06.607 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.607 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.607 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.607 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T00:26:06.607 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.608 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.608 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.608 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T00:26:06.608 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.608 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.608 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.608 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_base": { 2024-04-25T00:26:06.608 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_base", 2024-04-25T00:26:06.608 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.608 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.608 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.609 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T00:26:06.609 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.609 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.609 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.609 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Container image name, without the tag", 2024-04-25T00:26:06.609 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.609 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.609 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.609 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_elasticsearch": { 2024-04-25T00:26:06.609 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_elasticsearch", 2024-04-25T00:26:06.609 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.609 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.609 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.609 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T00:26:06.609 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.610 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.610 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.610 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "elasticsearch container image", 2024-04-25T00:26:06.610 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.610 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.610 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.610 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_grafana": { 2024-04-25T00:26:06.610 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_grafana", 2024-04-25T00:26:06.610 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.610 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.610 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.611 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T00:26:06.611 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.611 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.611 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.611 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T00:26:06.611 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.611 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.611 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.611 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_haproxy": { 2024-04-25T00:26:06.611 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_haproxy", 2024-04-25T00:26:06.611 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.611 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.611 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.611 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T00:26:06.612 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.612 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.612 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.612 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "HAproxy container image", 2024-04-25T00:26:06.612 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.612 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.612 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.612 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_agent": { 2024-04-25T00:26:06.612 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_agent", 2024-04-25T00:26:06.612 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.612 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.612 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.612 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T00:26:06.613 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.613 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.613 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.613 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger agent container image", 2024-04-25T00:26:06.613 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.613 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.613 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.613 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_collector": { 2024-04-25T00:26:06.613 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_collector", 2024-04-25T00:26:06.613 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.613 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.614 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.614 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T00:26:06.614 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.614 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.614 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.614 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger collector container image", 2024-04-25T00:26:06.614 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.614 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.614 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.614 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_query": { 2024-04-25T00:26:06.614 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_query", 2024-04-25T00:26:06.615 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.615 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.615 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.615 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T00:26:06.615 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.615 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.615 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.615 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger query container image", 2024-04-25T00:26:06.615 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.615 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.615 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.615 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_keepalived": { 2024-04-25T00:26:06.615 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_keepalived", 2024-04-25T00:26:06.615 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.616 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.616 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.616 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T00:26:06.616 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.616 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.616 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.616 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Keepalived container image", 2024-04-25T00:26:06.616 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.616 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.616 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.616 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_loki": { 2024-04-25T00:26:06.616 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_loki", 2024-04-25T00:26:06.616 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.616 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.617 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.617 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T00:26:06.617 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.617 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.617 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.617 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Loki container image", 2024-04-25T00:26:06.617 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.617 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.617 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.617 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_node_exporter": { 2024-04-25T00:26:06.617 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_node_exporter", 2024-04-25T00:26:06.617 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.617 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.618 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.618 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T00:26:06.618 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.618 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.618 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.618 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T00:26:06.618 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.618 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.618 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.618 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_nvmeof": { 2024-04-25T00:26:06.618 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_nvmeof", 2024-04-25T00:26:06.618 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.619 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.619 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.619 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T00:26:06.619 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.619 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.619 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.619 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Nvme-of container image", 2024-04-25T00:26:06.619 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.619 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.619 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.619 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_prometheus": { 2024-04-25T00:26:06.619 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_prometheus", 2024-04-25T00:26:06.619 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.620 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.620 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.620 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T00:26:06.620 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.620 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.620 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.620 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T00:26:06.620 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.620 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.620 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.620 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_promtail": { 2024-04-25T00:26:06.620 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_promtail", 2024-04-25T00:26:06.620 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.620 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.621 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.621 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T00:26:06.621 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.621 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.621 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.621 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Promtail container image", 2024-04-25T00:26:06.621 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.621 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.621 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.621 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_samba": { 2024-04-25T00:26:06.621 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_samba", 2024-04-25T00:26:06.621 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.621 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.621 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.622 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T00:26:06.622 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.622 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.622 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.622 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Samba/SMB container image", 2024-04-25T00:26:06.622 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.622 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.622 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.622 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_snmp_gateway": { 2024-04-25T00:26:06.622 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_snmp_gateway", 2024-04-25T00:26:06.622 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.622 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.623 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.623 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T00:26:06.623 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.623 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.623 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.623 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SNMP Gateway container image", 2024-04-25T00:26:06.623 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.623 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.623 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.623 INFO:teuthology.orchestra.run.smithi027.stdout: "container_init": { 2024-04-25T00:26:06.623 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_init", 2024-04-25T00:26:06.623 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.624 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.624 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.624 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.624 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.624 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.624 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.624 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T00:26:06.624 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.624 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.624 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.624 INFO:teuthology.orchestra.run.smithi027.stdout: "daemon_cache_timeout": { 2024-04-25T00:26:06.624 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "daemon_cache_timeout", 2024-04-25T00:26:06.625 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.625 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.625 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.625 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T00:26:06.625 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.625 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.625 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.625 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T00:26:06.625 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.625 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.625 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.625 INFO:teuthology.orchestra.run.smithi027.stdout: "default_cephadm_command_timeout": { 2024-04-25T00:26:06.625 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T00:26:06.625 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.626 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.626 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.626 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "900", 2024-04-25T00:26:06.626 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.626 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.626 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.626 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T00:26:06.626 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.626 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.626 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.626 INFO:teuthology.orchestra.run.smithi027.stdout: "default_registry": { 2024-04-25T00:26:06.626 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_registry", 2024-04-25T00:26:06.627 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.627 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.627 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.627 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io", 2024-04-25T00:26:06.627 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.627 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.627 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.627 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T00:26:06.627 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.627 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.627 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.627 INFO:teuthology.orchestra.run.smithi027.stdout: "device_cache_timeout": { 2024-04-25T00:26:06.628 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_cache_timeout", 2024-04-25T00:26:06.628 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.628 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.628 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.628 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1800", 2024-04-25T00:26:06.628 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.628 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.628 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.628 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache device inventory", 2024-04-25T00:26:06.628 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.628 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.628 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.628 INFO:teuthology.orchestra.run.smithi027.stdout: "device_enhanced_scan": { 2024-04-25T00:26:06.629 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_enhanced_scan", 2024-04-25T00:26:06.629 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.629 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.629 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.629 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.629 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.629 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.629 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.629 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T00:26:06.630 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.630 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.630 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.630 INFO:teuthology.orchestra.run.smithi027.stdout: "facts_cache_timeout": { 2024-04-25T00:26:06.630 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "facts_cache_timeout", 2024-04-25T00:26:06.631 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.631 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.631 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.631 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T00:26:06.631 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.631 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.632 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.632 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache host facts data", 2024-04-25T00:26:06.632 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.632 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.632 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.632 INFO:teuthology.orchestra.run.smithi027.stdout: "grafana_dashboards_path": { 2024-04-25T00:26:06.632 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "grafana_dashboards_path", 2024-04-25T00:26:06.632 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.632 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.632 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.632 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T00:26:06.633 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.633 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.633 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.633 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T00:26:06.633 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.633 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.633 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.633 INFO:teuthology.orchestra.run.smithi027.stdout: "host_check_interval": { 2024-04-25T00:26:06.634 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "host_check_interval", 2024-04-25T00:26:06.634 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.634 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.634 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.634 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T00:26:06.634 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.634 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.634 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.634 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to perform a host check", 2024-04-25T00:26:06.634 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "hw_monitoring": { 2024-04-25T00:26:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hw_monitoring", 2024-04-25T00:26:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T00:26:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "inventory_list_all": { 2024-04-25T00:26:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "inventory_list_all", 2024-04-25T00:26:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T00:26:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.638 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.638 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.638 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.638 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.638 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.638 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.638 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.638 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.638 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.638 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.639 INFO:teuthology.orchestra.run.smithi027.stdout: "log_refresh_metadata": { 2024-04-25T00:26:06.639 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_refresh_metadata", 2024-04-25T00:26:06.639 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.639 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.639 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.640 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.640 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.640 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.640 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.640 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T00:26:06.640 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.640 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.641 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.641 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.641 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.641 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.641 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.641 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.641 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.641 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.641 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.641 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.641 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T00:26:06.642 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.642 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.642 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.642 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.642 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.642 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.642 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.642 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.642 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.642 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.643 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.643 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.643 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.643 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.643 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.643 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.643 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.643 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.643 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.643 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.643 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.643 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.644 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.644 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.644 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.644 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.644 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.644 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.644 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.644 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.644 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.645 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.645 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.645 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.645 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.645 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.646 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T00:26:06.646 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T00:26:06.646 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.646 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.646 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.646 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.646 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.647 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.647 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.647 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T00:26:06.647 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.647 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.647 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.647 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T00:26:06.647 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T00:26:06.648 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.648 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.648 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.648 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "*", 2024-04-25T00:26:06.648 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.648 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.648 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.648 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T00:26:06.648 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.648 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.648 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.648 INFO:teuthology.orchestra.run.smithi027.stdout: "max_count_per_host": { 2024-04-25T00:26:06.648 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_count_per_host", 2024-04-25T00:26:06.649 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.649 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.649 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.649 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:06.649 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.649 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.649 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.649 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of daemons per service per host", 2024-04-25T00:26:06.649 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.649 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.649 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.649 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd_draining_count": { 2024-04-25T00:26:06.649 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_osd_draining_count", 2024-04-25T00:26:06.650 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.650 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.650 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.650 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:06.650 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.650 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.650 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.650 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T00:26:06.650 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.650 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.651 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.651 INFO:teuthology.orchestra.run.smithi027.stdout: "migration_current": { 2024-04-25T00:26:06.651 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "migration_current", 2024-04-25T00:26:06.651 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.651 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.651 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.651 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.651 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.651 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.651 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.651 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "internal - do not modify", 2024-04-25T00:26:06.651 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.651 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.652 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.652 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T00:26:06.652 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T00:26:06.652 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.652 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.652 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.652 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "root", 2024-04-25T00:26:06.652 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.652 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.652 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.652 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm-package", 2024-04-25T00:26:06.653 INFO:teuthology.orchestra.run.smithi027.stdout: "root" 2024-04-25T00:26:06.653 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.653 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T00:26:06.653 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.653 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.653 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.653 INFO:teuthology.orchestra.run.smithi027.stdout: "oob_default_addr": { 2024-04-25T00:26:06.653 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "oob_default_addr", 2024-04-25T00:26:06.653 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.653 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.653 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.653 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "169.254.1.1", 2024-04-25T00:26:06.653 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.653 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.654 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.654 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T00:26:06.654 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.654 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.654 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.655 INFO:teuthology.orchestra.run.smithi027.stdout: "prometheus_alerts_path": { 2024-04-25T00:26:06.655 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus_alerts_path", 2024-04-25T00:26:06.655 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.655 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.655 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.655 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T00:26:06.655 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.655 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.655 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.655 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T00:26:06.655 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.655 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.655 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.656 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_insecure": { 2024-04-25T00:26:06.656 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_insecure", 2024-04-25T00:26:06.656 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.656 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.656 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.656 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.656 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.656 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.656 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.656 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T00:26:06.656 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.656 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.656 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.656 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_password": { 2024-04-25T00:26:06.657 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_password", 2024-04-25T00:26:06.657 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.657 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.657 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.657 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.657 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.657 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.657 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.657 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T00:26:06.657 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.657 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.657 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.658 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_url": { 2024-04-25T00:26:06.658 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_url", 2024-04-25T00:26:06.658 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.658 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.658 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.658 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.658 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.658 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.658 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.658 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T00:26:06.658 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.658 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.658 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.659 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_username": { 2024-04-25T00:26:06.659 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_username", 2024-04-25T00:26:06.659 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.659 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.659 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.659 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.659 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.659 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.659 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.659 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T00:26:06.660 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.660 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.660 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.660 INFO:teuthology.orchestra.run.smithi027.stdout: "secure_monitoring_stack": { 2024-04-25T00:26:06.660 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "secure_monitoring_stack", 2024-04-25T00:26:06.660 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.660 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.660 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.660 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.660 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.660 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.660 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.660 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T00:26:06.660 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.661 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.661 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.661 INFO:teuthology.orchestra.run.smithi027.stdout: "service_discovery_port": { 2024-04-25T00:26:06.661 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "service_discovery_port", 2024-04-25T00:26:06.661 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.661 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.661 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.661 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8765", 2024-04-25T00:26:06.661 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.661 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.661 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.661 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "cephadm service discovery port", 2024-04-25T00:26:06.661 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.661 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.661 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.662 INFO:teuthology.orchestra.run.smithi027.stdout: "ssh_config_file": { 2024-04-25T00:26:06.662 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssh_config_file", 2024-04-25T00:26:06.662 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.662 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.662 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.662 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.662 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.662 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.662 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.662 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T00:26:06.662 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.662 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.662 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.663 INFO:teuthology.orchestra.run.smithi027.stdout: "use_agent": { 2024-04-25T00:26:06.663 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_agent", 2024-04-25T00:26:06.663 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.663 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.663 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.663 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.663 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.663 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.663 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.663 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T00:26:06.663 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.663 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.663 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.664 INFO:teuthology.orchestra.run.smithi027.stdout: "use_repo_digest": { 2024-04-25T00:26:06.664 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_repo_digest", 2024-04-25T00:26:06.664 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.664 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.664 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.664 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.664 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.664 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.664 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.664 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T00:26:06.664 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.664 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.664 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.664 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_failed_host_check": { 2024-04-25T00:26:06.664 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_failed_host_check", 2024-04-25T00:26:06.665 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.665 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.665 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.665 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.665 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.665 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.665 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.665 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T00:26:06.665 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.665 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.665 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.665 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_daemons": { 2024-04-25T00:26:06.665 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_daemons", 2024-04-25T00:26:06.665 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.666 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.666 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.666 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.666 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.666 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.666 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.666 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T00:26:06.666 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.666 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.666 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.666 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_hosts": { 2024-04-25T00:26:06.666 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_hosts", 2024-04-25T00:26:06.666 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.666 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.667 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.667 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.667 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.667 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.667 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.667 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T00:26:06.667 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.667 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.667 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.667 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.667 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.667 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crash", 2024-04-25T00:26:06.667 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.668 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.668 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.668 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.668 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.668 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.668 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.668 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.668 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.668 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.668 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.668 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.668 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.669 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.669 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.669 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.669 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.669 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.669 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.669 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.669 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.669 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.669 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.670 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.670 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.670 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.670 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.670 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.670 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.670 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.670 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.670 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.670 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.670 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.671 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.671 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.671 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.671 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.671 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.671 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.671 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.671 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.671 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.671 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.671 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.671 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.671 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.671 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.672 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.672 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.672 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.672 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.672 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.672 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.672 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.672 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.672 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.672 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.672 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.672 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.672 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.673 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.673 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.673 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.673 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.673 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.673 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.673 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.673 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.673 INFO:teuthology.orchestra.run.smithi027.stdout: "retain_interval": { 2024-04-25T00:26:06.673 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retain_interval", 2024-04-25T00:26:06.673 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.673 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.673 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.673 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "31536000", 2024-04-25T00:26:06.674 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.674 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.674 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.674 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T00:26:06.674 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.674 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.674 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.674 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_recent_interval": { 2024-04-25T00:26:06.674 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_recent_interval", 2024-04-25T00:26:06.674 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.674 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.674 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.674 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1209600", 2024-04-25T00:26:06.675 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.675 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.675 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.675 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T00:26:06.675 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.675 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.675 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.675 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.675 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.675 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dashboard", 2024-04-25T00:26:06.675 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.675 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.675 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.675 INFO:teuthology.orchestra.run.smithi027.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T00:26:06.676 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T00:26:06.676 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.676 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.676 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.676 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:06.676 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.676 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.676 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.676 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.676 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.676 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.676 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.677 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T00:26:06.677 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T00:26:06.677 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.677 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.677 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.677 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.677 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.677 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.677 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.677 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.677 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.677 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.677 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.678 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T00:26:06.678 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T00:26:06.678 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.678 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.678 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.678 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.678 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.678 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.678 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.678 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.678 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.678 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.678 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.679 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_ENABLED": { 2024-04-25T00:26:06.679 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T00:26:06.679 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.679 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.679 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.679 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.679 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.679 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.679 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.679 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.679 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.679 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.679 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.679 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T00:26:06.680 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T00:26:06.680 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.680 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.680 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.680 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.680 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.680 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.680 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.680 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.680 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.680 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.680 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.680 INFO:teuthology.orchestra.run.smithi027.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T00:26:06.680 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T00:26:06.680 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.681 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.681 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.681 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.681 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.681 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.681 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.681 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.681 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.681 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.681 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.681 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T00:26:06.681 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T00:26:06.682 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.682 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.682 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.682 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.682 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.682 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.682 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.682 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.682 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.682 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.682 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.682 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T00:26:06.683 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T00:26:06.683 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.683 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.683 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.683 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.683 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.683 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.683 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.683 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.683 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.683 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.683 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.683 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T00:26:06.684 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T00:26:06.684 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.684 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.684 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.684 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.684 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.684 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.684 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.684 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.684 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.684 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.684 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.684 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T00:26:06.684 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T00:26:06.685 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.685 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.685 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.685 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.685 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.685 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.685 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.685 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.685 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.685 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.685 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.685 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T00:26:06.685 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T00:26:06.685 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T00:26:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T00:26:06.687 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.687 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.687 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.687 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.687 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.687 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.687 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.687 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.687 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.687 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.687 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.688 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T00:26:06.688 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T00:26:06.688 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.688 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.688 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.688 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.688 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.688 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.688 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.688 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.688 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.688 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.688 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.688 INFO:teuthology.orchestra.run.smithi027.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T00:26:06.688 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T00:26:06.689 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.689 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.689 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.689 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.689 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.689 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.689 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.689 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.689 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.689 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.689 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.689 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T00:26:06.689 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T00:26:06.689 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.690 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.690 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.690 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T00:26:06.690 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.690 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.690 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.690 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.690 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.690 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.690 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.690 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T00:26:06.690 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T00:26:06.690 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.691 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.691 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.691 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.691 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.691 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.691 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.691 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.691 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.691 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.691 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.691 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_URL": { 2024-04-25T00:26:06.691 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_URL", 2024-04-25T00:26:06.691 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.691 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.691 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.692 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.692 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.692 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.692 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.692 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.692 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.692 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.692 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.692 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T00:26:06.692 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T00:26:06.692 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.692 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.692 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.692 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T00:26:06.692 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.693 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.693 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.693 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.693 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.693 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.693 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.693 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T00:26:06.693 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T00:26:06.693 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.693 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.693 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.693 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.693 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.693 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.693 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.694 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.694 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.694 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.694 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.694 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T00:26:06.694 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T00:26:06.694 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.694 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.694 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.694 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.694 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.694 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.695 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.695 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.695 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.695 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.695 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.695 INFO:teuthology.orchestra.run.smithi027.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T00:26:06.695 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T00:26:06.695 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.695 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.695 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.695 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.695 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.695 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.696 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.696 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.696 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.696 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.696 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.696 INFO:teuthology.orchestra.run.smithi027.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T00:26:06.696 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T00:26:06.696 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.696 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.696 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.696 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.696 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.696 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.696 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.696 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.697 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.697 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.697 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.697 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T00:26:06.697 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T00:26:06.697 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.697 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.697 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.697 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.697 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.697 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.697 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.697 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.697 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.697 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.698 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.698 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T00:26:06.698 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T00:26:06.698 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.698 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.698 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.698 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.698 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.698 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.698 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.698 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.698 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.698 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.699 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.699 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T00:26:06.699 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T00:26:06.699 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.699 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.699 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.699 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.699 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.699 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.699 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.699 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.699 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.699 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.699 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.700 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T00:26:06.700 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T00:26:06.700 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.700 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.700 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.700 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.700 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.700 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.700 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.700 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.700 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.700 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.700 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.700 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T00:26:06.701 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T00:26:06.701 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.701 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.701 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.701 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.701 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.701 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.701 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.701 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.701 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.701 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.701 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.701 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T00:26:06.701 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T00:26:06.701 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.702 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.702 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.702 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.702 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.702 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.702 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.702 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.702 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.702 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.702 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.702 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T00:26:06.702 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T00:26:06.702 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.703 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.703 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.703 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.703 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.703 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.703 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.703 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.703 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.703 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.703 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.703 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T00:26:06.703 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T00:26:06.703 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.703 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.703 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.704 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.704 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.704 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.704 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.704 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.704 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.704 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.704 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.704 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T00:26:06.704 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T00:26:06.704 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.704 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.704 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.705 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.705 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.705 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.705 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.705 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.705 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.705 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.705 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.705 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T00:26:06.705 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T00:26:06.705 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.705 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.705 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.705 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.706 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.706 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.706 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.706 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.706 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.706 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.706 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.706 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T00:26:06.706 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T00:26:06.706 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.706 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.707 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.707 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T00:26:06.707 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.707 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.707 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.707 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.707 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.707 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.707 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.707 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T00:26:06.707 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T00:26:06.707 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.707 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.707 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.708 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:06.708 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.708 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.708 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.708 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.708 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.708 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.708 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.708 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T00:26:06.708 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T00:26:06.708 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.708 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.708 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.708 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8", 2024-04-25T00:26:06.709 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.709 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.709 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.709 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.709 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.709 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.709 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.709 INFO:teuthology.orchestra.run.smithi027.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T00:26:06.709 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T00:26:06.709 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.710 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.710 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.710 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "45", 2024-04-25T00:26:06.710 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.710 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.710 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.710 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.710 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.710 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.710 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.710 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T00:26:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T00:26:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T00:26:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T00:26:06.712 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.712 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.712 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.712 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T00:26:06.712 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.712 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.712 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.712 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.712 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.712 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.713 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.713 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T00:26:06.713 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T00:26:06.713 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.713 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.713 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.713 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.713 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.713 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.713 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.713 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.713 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.714 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.714 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.714 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T00:26:06.714 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T00:26:06.714 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.714 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.714 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.714 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.714 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.714 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.714 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.714 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.715 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.715 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.715 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.715 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T00:26:06.715 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T00:26:06.715 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.715 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.715 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.715 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T00:26:06.715 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.715 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.715 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.715 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.716 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.716 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.716 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.716 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T00:26:06.716 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T00:26:06.716 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.716 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.716 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.716 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:06.716 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.716 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.716 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.716 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.716 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.717 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.717 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.717 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T00:26:06.717 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T00:26:06.717 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.717 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.717 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.717 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T00:26:06.717 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.717 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.717 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.717 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.717 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.717 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.717 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.718 INFO:teuthology.orchestra.run.smithi027.stdout: "cross_origin_url": { 2024-04-25T00:26:06.718 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cross_origin_url", 2024-04-25T00:26:06.718 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.718 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.718 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.718 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.718 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.718 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.718 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.718 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.718 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.718 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.718 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.719 INFO:teuthology.orchestra.run.smithi027.stdout: "crt_file": { 2024-04-25T00:26:06.719 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crt_file", 2024-04-25T00:26:06.719 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.719 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.719 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.719 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.719 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.719 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.719 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.719 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.719 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.719 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.719 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.719 INFO:teuthology.orchestra.run.smithi027.stdout: "debug": { 2024-04-25T00:26:06.720 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "debug", 2024-04-25T00:26:06.720 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.720 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.720 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.720 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.720 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.720 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.720 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.720 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable/disable debug options", 2024-04-25T00:26:06.720 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.720 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.720 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.720 INFO:teuthology.orchestra.run.smithi027.stdout: "jwt_token_ttl": { 2024-04-25T00:26:06.720 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "jwt_token_ttl", 2024-04-25T00:26:06.721 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.721 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.721 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.721 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "28800", 2024-04-25T00:26:06.721 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.721 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.721 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.721 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.721 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.721 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.721 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.721 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T00:26:06.721 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T00:26:06.721 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.721 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.721 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.722 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.722 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.722 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.722 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.722 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.722 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.722 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.722 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.722 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.722 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.722 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.723 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.723 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.723 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.723 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.723 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.723 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.723 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.723 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.723 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.723 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.723 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.723 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.723 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.723 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.723 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.724 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.724 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.724 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.724 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.724 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.724 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.724 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.724 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.724 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.724 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.724 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.724 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.724 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.724 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.724 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.725 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.725 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.725 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.725 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.725 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.725 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.725 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.725 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.725 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.725 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.725 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.725 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.725 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.725 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.725 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.726 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.726 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.726 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.726 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.726 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.726 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.726 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.726 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.726 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.726 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.726 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.726 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.726 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.726 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.727 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.727 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.727 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.727 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.727 INFO:teuthology.orchestra.run.smithi027.stdout: "motd": { 2024-04-25T00:26:06.727 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "motd", 2024-04-25T00:26:06.727 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.728 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.728 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.728 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.728 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.728 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.728 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.729 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "The message of the day", 2024-04-25T00:26:06.729 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.729 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.729 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.729 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect_resolve_ip_addr": { 2024-04-25T00:26:06.730 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T00:26:06.730 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.730 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.730 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.730 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.730 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.731 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.731 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.731 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.731 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.731 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.731 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.732 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.732 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T00:26:06.732 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T00:26:06.732 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.732 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.732 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.733 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T00:26:06.733 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.733 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.733 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.733 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.733 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.734 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.734 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.734 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T00:26:06.734 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T00:26:06.735 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.735 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.735 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.735 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8080", 2024-04-25T00:26:06.735 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.736 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.736 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.736 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.736 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.736 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.736 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.737 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.737 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T00:26:06.737 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T00:26:06.737 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.737 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.737 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.738 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.738 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.738 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.738 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.738 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.738 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.739 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.739 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.739 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl_server_port": { 2024-04-25T00:26:06.739 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl_server_port", 2024-04-25T00:26:06.739 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.740 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.740 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.740 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8443", 2024-04-25T00:26:06.740 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.740 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.741 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.741 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.741 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.741 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.741 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.742 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T00:26:06.742 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T00:26:06.742 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.742 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.742 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.743 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "redirect", 2024-04-25T00:26:06.743 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.743 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.743 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.743 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.743 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect" 2024-04-25T00:26:06.743 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.743 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.743 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.743 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.743 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.744 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T00:26:06.744 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T00:26:06.744 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.744 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.744 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.744 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T00:26:06.744 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T00:26:06.744 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T00:26:06.744 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.744 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.744 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.744 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.744 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.745 INFO:teuthology.orchestra.run.smithi027.stdout: "url_prefix": { 2024-04-25T00:26:06.745 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url_prefix", 2024-04-25T00:26:06.745 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.745 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.745 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.745 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.745 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.745 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.745 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.745 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.745 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.745 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.746 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.746 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.746 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.746 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.747 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "devicehealth", 2024-04-25T00:26:06.747 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.747 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.747 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.747 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_monitoring": { 2024-04-25T00:26:06.747 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_monitoring", 2024-04-25T00:26:06.748 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.748 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.748 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.748 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.748 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.748 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.748 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.749 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "monitor device health metrics", 2024-04-25T00:26:06.749 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.749 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.749 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.749 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.749 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.749 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.749 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.749 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.749 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.749 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.749 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.749 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.749 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.750 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.750 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.750 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.750 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.750 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.750 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.750 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.750 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.750 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.750 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.750 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.750 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.750 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.750 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.751 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.751 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.751 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.751 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.751 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.751 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.751 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.751 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.751 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.751 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.751 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.751 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.751 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.751 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.752 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.752 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.752 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.752 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.753 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.753 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.753 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.753 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.754 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.754 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.754 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.754 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.754 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.754 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.754 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.754 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.754 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.754 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.754 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.754 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.754 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.755 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.755 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.755 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.755 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.755 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.755 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.755 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.755 INFO:teuthology.orchestra.run.smithi027.stdout: "mark_out_threshold": { 2024-04-25T00:26:06.755 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mark_out_threshold", 2024-04-25T00:26:06.755 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.755 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.755 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.755 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2419200", 2024-04-25T00:26:06.755 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.756 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.756 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.756 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T00:26:06.756 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.756 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.756 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.756 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.756 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": { 2024-04-25T00:26:06.756 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_name", 2024-04-25T00:26:06.756 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.756 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.756 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.756 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "device_health_metrics", 2024-04-25T00:26:06.756 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.756 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.756 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.757 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T00:26:06.757 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.757 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.757 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.757 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.757 INFO:teuthology.orchestra.run.smithi027.stdout: "retention_period": { 2024-04-25T00:26:06.757 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retention_period", 2024-04-25T00:26:06.757 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.757 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.757 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.757 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15552000", 2024-04-25T00:26:06.757 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.757 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.757 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.757 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain device health metrics", 2024-04-25T00:26:06.758 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.758 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.758 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.758 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_frequency": { 2024-04-25T00:26:06.758 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_frequency", 2024-04-25T00:26:06.758 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.758 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.758 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.758 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T00:26:06.758 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.758 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.759 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.759 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T00:26:06.759 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.759 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.759 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.759 INFO:teuthology.orchestra.run.smithi027.stdout: "self_heal": { 2024-04-25T00:26:06.759 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "self_heal", 2024-04-25T00:26:06.759 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.759 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.759 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.759 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.759 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.760 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.760 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.760 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T00:26:06.760 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.760 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.760 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.761 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T00:26:06.761 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T00:26:06.761 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.761 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.761 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.761 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T00:26:06.761 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.761 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.761 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.761 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T00:26:06.761 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.761 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.761 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.762 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_threshold": { 2024-04-25T00:26:06.762 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_threshold", 2024-04-25T00:26:06.762 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.762 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.762 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.762 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7257600", 2024-04-25T00:26:06.762 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.762 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.762 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.762 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T00:26:06.762 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.762 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.762 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.763 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.763 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.763 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.763 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "diskprediction_local", 2024-04-25T00:26:06.763 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.763 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.763 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.763 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.763 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.763 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.763 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.763 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.763 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.764 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.764 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.764 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.764 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.764 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.764 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.764 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.764 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.764 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.764 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.764 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.764 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.764 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.765 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.765 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.765 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.765 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.765 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.765 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.765 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.765 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.765 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.765 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.765 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.765 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.766 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.766 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.766 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.766 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.766 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.766 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.766 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.766 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.766 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.766 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.766 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.766 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.766 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.766 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.767 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.767 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.767 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.767 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.767 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.767 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.767 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.768 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.768 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.768 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.768 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.768 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.768 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.769 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.769 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.769 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.769 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.769 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.769 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.770 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.770 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.770 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.770 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.770 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.770 INFO:teuthology.orchestra.run.smithi027.stdout: "predict_interval": { 2024-04-25T00:26:06.770 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predict_interval", 2024-04-25T00:26:06.770 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.770 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.770 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.771 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T00:26:06.771 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.771 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.771 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.771 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.771 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.771 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.771 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.771 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.771 INFO:teuthology.orchestra.run.smithi027.stdout: "predictor_model": { 2024-04-25T00:26:06.771 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predictor_model", 2024-04-25T00:26:06.771 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.771 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.771 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.772 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "prophetstor", 2024-04-25T00:26:06.772 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.772 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.772 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.772 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.772 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.772 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.772 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.772 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.772 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T00:26:06.772 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T00:26:06.772 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.772 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.772 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.772 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T00:26:06.773 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.773 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.773 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.773 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.773 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.773 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.773 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.773 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.773 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.773 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.773 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.773 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "influx", 2024-04-25T00:26:06.773 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": false, 2024-04-25T00:26:06.773 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "influxdb python module not found", 2024-04-25T00:26:06.773 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.774 INFO:teuthology.orchestra.run.smithi027.stdout: "batch_size": { 2024-04-25T00:26:06.774 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "batch_size", 2024-04-25T00:26:06.774 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.774 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.774 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.774 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5000", 2024-04-25T00:26:06.774 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.775 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.775 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.775 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T00:26:06.775 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.775 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.775 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.775 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.775 INFO:teuthology.orchestra.run.smithi027.stdout: "database": { 2024-04-25T00:26:06.776 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "database", 2024-04-25T00:26:06.776 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.776 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.776 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.776 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "ceph", 2024-04-25T00:26:06.776 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.776 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.776 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.776 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T00:26:06.776 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.776 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.776 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.776 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.776 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": { 2024-04-25T00:26:06.777 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hostname", 2024-04-25T00:26:06.777 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.777 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.777 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.777 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.777 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.777 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.777 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.777 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server hostname", 2024-04-25T00:26:06.777 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.777 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.777 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.778 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T00:26:06.778 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T00:26:06.778 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.778 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.778 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.778 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "30", 2024-04-25T00:26:06.778 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "5", 2024-04-25T00:26:06.778 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.778 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.778 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T00:26:06.778 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.778 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.779 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.779 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.779 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.779 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.779 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.779 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.779 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.779 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.779 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.779 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.779 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.779 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.779 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.779 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.780 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.780 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.780 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.780 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.780 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.780 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.780 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.780 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.780 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.780 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.780 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.780 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.780 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.781 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.781 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.781 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.781 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.781 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.781 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.781 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.781 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.781 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.781 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.781 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.781 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.781 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.781 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.781 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.782 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.782 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.782 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.782 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.782 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.782 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.782 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.782 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.782 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.782 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.782 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.782 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.782 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.783 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.783 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.783 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.783 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.783 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.783 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.783 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.783 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.783 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.783 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.783 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.783 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.783 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.783 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.783 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.784 INFO:teuthology.orchestra.run.smithi027.stdout: "password": { 2024-04-25T00:26:06.784 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "password", 2024-04-25T00:26:06.784 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.784 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.784 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.784 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.784 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.784 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.784 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.784 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "password of InfluxDB server user", 2024-04-25T00:26:06.784 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.784 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.784 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.785 INFO:teuthology.orchestra.run.smithi027.stdout: "port": { 2024-04-25T00:26:06.785 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "port", 2024-04-25T00:26:06.785 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.785 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.785 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.785 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8086", 2024-04-25T00:26:06.785 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.785 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.785 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.785 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server port", 2024-04-25T00:26:06.785 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.785 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.785 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.786 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T00:26:06.786 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T00:26:06.786 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.786 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.786 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.786 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "false", 2024-04-25T00:26:06.786 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.786 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.786 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.786 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T00:26:06.786 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.786 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.786 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.787 INFO:teuthology.orchestra.run.smithi027.stdout: "threads": { 2024-04-25T00:26:06.787 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threads", 2024-04-25T00:26:06.787 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.787 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.787 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.787 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T00:26:06.787 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T00:26:06.787 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "32", 2024-04-25T00:26:06.787 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.787 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T00:26:06.787 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.787 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.787 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.788 INFO:teuthology.orchestra.run.smithi027.stdout: "username": { 2024-04-25T00:26:06.788 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "username", 2024-04-25T00:26:06.788 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.788 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.788 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.788 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.788 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.788 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.788 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.788 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "username of InfluxDB server user", 2024-04-25T00:26:06.788 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.788 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.788 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.788 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.789 INFO:teuthology.orchestra.run.smithi027.stdout: "verify_ssl": { 2024-04-25T00:26:06.789 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "verify_ssl", 2024-04-25T00:26:06.789 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.789 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.789 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.789 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "true", 2024-04-25T00:26:06.789 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.789 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.789 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.789 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T00:26:06.789 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.789 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.789 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.789 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.790 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.790 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.790 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.790 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "insights", 2024-04-25T00:26:06.790 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.790 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.790 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.790 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.790 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.790 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.790 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.790 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.790 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.790 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.791 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.791 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.791 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.791 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.791 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.791 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.791 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.791 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.791 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.791 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.791 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.791 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.791 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.791 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.791 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.792 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.792 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.792 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.792 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.792 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.792 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.792 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.792 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.792 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.792 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.792 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.792 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.792 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.792 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.793 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.793 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.793 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.793 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.793 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.793 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.793 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.793 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.793 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.793 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.793 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.793 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.793 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.793 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.794 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.794 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.794 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.794 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.794 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.794 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.794 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.794 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.794 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.794 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.794 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.794 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.794 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.795 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.795 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.795 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.795 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.795 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.795 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.795 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.795 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.795 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "iostat", 2024-04-25T00:26:06.795 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.795 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.795 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.795 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.796 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.796 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.796 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.796 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.796 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.796 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.796 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.796 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.796 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.796 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.796 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.796 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.796 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.797 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.797 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.797 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.797 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.797 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.797 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.797 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.797 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.797 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.797 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.797 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.797 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.797 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.797 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.798 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.798 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.798 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.798 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.798 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.798 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.798 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.798 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.798 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.798 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.798 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.798 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.798 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.798 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.799 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.799 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.799 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.799 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.799 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.799 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.799 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.799 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.799 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.799 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.799 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.799 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.799 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.799 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.800 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.800 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.800 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.800 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.800 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.800 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.800 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.800 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.800 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.800 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.800 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.800 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.800 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.800 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.801 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.801 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "k8sevents", 2024-04-25T00:26:06.801 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.801 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.801 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.801 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_event_retention_days": { 2024-04-25T00:26:06.801 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ceph_event_retention_days", 2024-04-25T00:26:06.801 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.801 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.801 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.801 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7", 2024-04-25T00:26:06.801 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.801 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.802 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.802 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T00:26:06.802 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.802 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.802 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.802 INFO:teuthology.orchestra.run.smithi027.stdout: "config_check_secs": { 2024-04-25T00:26:06.802 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_check_secs", 2024-04-25T00:26:06.802 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.802 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.802 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.803 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:06.803 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "10", 2024-04-25T00:26:06.803 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.803 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.803 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T00:26:06.803 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.803 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.803 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.803 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.803 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.803 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.803 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.803 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.804 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.804 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.804 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.804 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.804 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.804 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.804 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.804 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.804 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.804 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.804 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.804 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.804 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.804 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.805 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.805 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.805 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.805 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.805 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.805 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.805 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.805 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.805 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.805 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.805 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.805 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.805 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.806 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.806 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.806 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.806 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.806 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.806 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.806 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.806 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.806 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.806 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.806 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.806 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.807 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.807 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.807 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.807 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.807 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.807 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.807 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.807 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.807 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.807 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.807 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.807 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.807 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.807 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.808 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.808 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.808 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.808 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.808 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.808 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.808 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.808 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.808 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.808 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.808 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.808 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.808 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "localpool", 2024-04-25T00:26:06.808 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.809 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.809 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.809 INFO:teuthology.orchestra.run.smithi027.stdout: "failure_domain": { 2024-04-25T00:26:06.809 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "failure_domain", 2024-04-25T00:26:06.809 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.809 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.809 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.809 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "host", 2024-04-25T00:26:06.809 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.809 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.809 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.810 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "failure domain for any created local pool", 2024-04-25T00:26:06.810 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T00:26:06.810 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.810 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.810 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.810 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.810 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.810 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.810 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.810 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.810 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.810 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.810 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.810 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.811 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.811 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.811 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.811 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.811 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.811 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.811 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.811 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.811 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.811 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.811 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.811 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.812 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.812 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.812 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.812 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.812 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.812 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.812 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.812 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.812 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.812 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.812 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.812 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.812 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.812 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.813 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.813 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.813 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.813 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.813 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.813 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.813 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.813 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.813 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.813 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.813 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.813 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.813 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.814 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.814 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.814 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.814 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.814 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.814 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.814 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.814 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.814 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.814 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.814 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.814 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.814 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.814 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.815 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.815 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.815 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.815 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.815 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.815 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": { 2024-04-25T00:26:06.815 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_size", 2024-04-25T00:26:06.815 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.815 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.815 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.815 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.815 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.816 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.816 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.816 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default min_size for any created local pool", 2024-04-25T00:26:06.816 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T00:26:06.816 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.816 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.816 INFO:teuthology.orchestra.run.smithi027.stdout: "num_rep": { 2024-04-25T00:26:06.816 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "num_rep", 2024-04-25T00:26:06.816 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.816 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.816 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.816 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3", 2024-04-25T00:26:06.816 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.817 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.817 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.817 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default replica count for any created local pool", 2024-04-25T00:26:06.817 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.817 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.817 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.817 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.817 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": { 2024-04-25T00:26:06.817 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_num", 2024-04-25T00:26:06.817 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.817 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.817 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.817 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "128", 2024-04-25T00:26:06.817 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.818 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.818 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.818 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T00:26:06.818 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.818 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.818 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.818 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.818 INFO:teuthology.orchestra.run.smithi027.stdout: "prefix": { 2024-04-25T00:26:06.818 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prefix", 2024-04-25T00:26:06.818 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.818 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.818 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.818 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.818 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.818 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.819 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.819 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name prefix for any created local pool", 2024-04-25T00:26:06.819 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.819 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.819 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.819 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.819 INFO:teuthology.orchestra.run.smithi027.stdout: "subtree": { 2024-04-25T00:26:06.819 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subtree", 2024-04-25T00:26:06.819 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.819 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.819 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.819 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "rack", 2024-04-25T00:26:06.819 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.819 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.819 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.820 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T00:26:06.820 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T00:26:06.820 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.820 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.820 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.820 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.820 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.820 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.820 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mds_autoscaler", 2024-04-25T00:26:06.820 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.820 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.820 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.820 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.820 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.820 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.820 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.821 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.821 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.821 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.821 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.821 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.821 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.821 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.821 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.821 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.821 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.821 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.821 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.821 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.821 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.821 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.821 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.822 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.822 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.822 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.822 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.822 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.822 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.822 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.822 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.822 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.822 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.822 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.822 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.822 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.822 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.823 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.823 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.823 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.823 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.823 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.823 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.823 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.823 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.823 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.823 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.823 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.823 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.823 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.824 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.824 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.824 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.824 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.824 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.824 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.824 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.824 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.824 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.824 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.824 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.824 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.824 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.824 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.824 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.824 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.825 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.825 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.825 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.825 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.825 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.825 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.825 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.825 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.825 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.825 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirroring", 2024-04-25T00:26:06.825 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.825 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.825 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.825 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.825 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.826 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.826 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.826 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.826 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.826 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.826 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.826 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.826 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.826 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.826 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.826 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.826 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.826 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.826 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.827 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.827 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.827 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.827 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.827 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.827 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.827 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.827 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.827 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.827 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.827 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.827 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.827 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.827 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.828 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.828 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.828 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.828 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.828 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.828 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.828 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.828 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.828 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.828 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.828 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.828 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.828 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.829 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.829 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.829 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.829 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.829 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.829 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.829 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.829 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.829 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.829 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.829 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.829 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.829 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.829 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.829 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.830 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.830 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.830 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.830 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.830 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.830 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.830 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.830 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.830 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.830 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.830 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.830 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.830 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.830 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "nfs", 2024-04-25T00:26:06.830 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.831 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.831 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.831 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.831 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.831 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.831 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.831 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.831 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.831 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.831 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.831 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.831 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.831 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.831 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.831 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.832 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.832 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.832 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.832 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.832 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.832 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.832 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.832 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.832 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.832 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.832 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.832 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.832 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.832 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.832 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.833 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.833 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.833 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.833 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.833 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.833 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.833 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.833 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.833 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.833 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.833 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.833 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.833 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.833 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.833 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.833 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.834 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.834 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.834 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.834 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.834 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.834 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.834 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.834 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.834 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.834 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.834 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.834 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.834 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.834 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.835 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.835 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.835 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.835 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.835 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.835 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.835 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.835 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.835 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.835 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.835 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.835 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.835 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.835 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T00:26:06.835 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.836 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.836 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.836 INFO:teuthology.orchestra.run.smithi027.stdout: "fail_fs": { 2024-04-25T00:26:06.836 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "fail_fs", 2024-04-25T00:26:06.836 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.836 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.836 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.836 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.836 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.836 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.836 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.836 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T00:26:06.836 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.836 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.836 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.837 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.837 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.837 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.837 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.837 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.837 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.837 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.837 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.837 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.837 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.837 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.837 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.837 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.838 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.838 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.838 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.838 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.838 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.838 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.838 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.838 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.838 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.838 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.838 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.838 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.838 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.838 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.838 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.839 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.839 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.839 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.839 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.839 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.839 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.839 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.839 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.839 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.839 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.839 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.839 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.839 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.839 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.839 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.839 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.840 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.840 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.840 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.840 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.840 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.840 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.840 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.840 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.840 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.840 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.840 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.840 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.840 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.840 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.841 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.841 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.841 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.841 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.841 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.841 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.841 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.841 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.841 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.841 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.841 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator": { 2024-04-25T00:26:06.841 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T00:26:06.841 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.841 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.842 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.842 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.842 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.842 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.842 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.842 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm", 2024-04-25T00:26:06.842 INFO:teuthology.orchestra.run.smithi027.stdout: "rook", 2024-04-25T00:26:06.842 INFO:teuthology.orchestra.run.smithi027.stdout: "test_orchestrator" 2024-04-25T00:26:06.842 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.842 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Orchestrator backend", 2024-04-25T00:26:06.842 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.842 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.842 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.842 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.843 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.843 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.843 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_perf_query", 2024-04-25T00:26:06.843 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.843 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.843 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.843 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.843 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.843 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.843 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.843 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.844 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.844 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.844 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.844 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.844 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.844 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.844 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.844 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.844 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.844 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.844 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.844 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.844 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.844 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.845 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.845 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.845 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.845 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.845 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.845 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.845 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.845 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.845 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.845 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.845 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.845 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.845 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.846 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.846 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.846 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.846 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.846 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.846 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.846 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.846 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.846 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.846 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.846 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.846 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.846 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.846 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.847 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.847 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.847 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.847 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.847 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.847 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.847 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.847 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.847 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.847 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.847 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.847 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.848 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.848 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.848 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.848 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.848 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.848 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.848 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.848 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.848 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.848 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.848 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.848 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.848 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_support", 2024-04-25T00:26:06.848 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.848 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.849 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.849 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.849 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.849 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.849 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.849 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.849 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.849 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.849 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.849 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.849 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.849 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.849 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.849 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.849 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.849 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.850 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.850 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.850 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.850 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.850 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.850 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.850 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.850 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.850 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.850 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.850 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.850 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.850 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.851 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.851 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.851 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.851 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.851 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.851 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.851 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.851 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.851 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.851 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.851 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.851 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.851 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.851 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.851 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.851 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.852 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.852 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.852 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.852 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.852 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.852 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.852 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.852 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.852 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.852 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.852 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.852 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.852 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.852 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.852 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.852 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.853 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.853 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.853 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.853 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.853 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.853 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.853 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.853 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.853 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.853 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.853 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.853 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_autoscaler", 2024-04-25T00:26:06.853 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.853 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.853 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.854 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.854 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.854 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.854 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.854 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.854 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.854 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.854 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.854 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.854 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.854 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.854 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.854 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.854 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.854 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.855 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.855 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.855 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.855 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.855 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.855 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.855 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.855 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.855 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.855 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.855 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.855 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.855 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.855 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.855 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.856 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.856 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.856 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.856 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.856 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.856 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.856 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.856 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.856 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.856 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.856 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.856 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.856 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.856 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.856 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.857 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.857 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.857 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.857 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.857 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.857 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.857 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.857 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.857 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.857 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.857 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.857 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.857 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.857 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.857 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.858 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.858 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.858 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.858 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.858 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.858 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.858 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.858 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.858 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T00:26:06.858 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T00:26:06.858 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.858 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.859 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.859 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T00:26:06.859 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.859 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.859 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.859 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.859 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.859 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.859 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.859 INFO:teuthology.orchestra.run.smithi027.stdout: "threshold": { 2024-04-25T00:26:06.860 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threshold", 2024-04-25T00:26:06.860 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:06.860 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.860 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.860 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T00:26:06.860 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1.0", 2024-04-25T00:26:06.860 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.860 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.860 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "scaling threshold", 2024-04-25T00:26:06.860 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T00:26:06.860 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.860 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.860 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.860 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.861 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.861 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "progress", 2024-04-25T00:26:06.861 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.861 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.861 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.861 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_pg_recovery_event": { 2024-04-25T00:26:06.861 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_pg_recovery_event", 2024-04-25T00:26:06.862 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.862 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.862 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.862 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.862 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.862 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.862 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.863 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T00:26:06.863 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.863 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.863 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.863 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T00:26:06.864 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T00:26:06.864 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.864 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.864 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.864 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.864 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.864 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.865 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.865 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.865 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.865 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.865 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.865 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.866 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.866 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.866 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.866 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.866 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.866 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.866 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.867 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.867 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.867 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.867 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.867 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.867 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.868 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.868 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.868 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.868 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.868 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.868 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.869 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.869 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.869 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.869 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.869 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.869 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.869 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.870 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.870 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.870 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.870 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.870 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.871 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.871 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.871 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.871 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.871 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.871 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.871 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.871 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.872 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.872 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.872 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.872 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.872 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.872 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.872 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.873 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.873 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.873 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.873 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.873 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.873 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.874 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.874 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.874 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.874 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.874 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.875 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.875 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.875 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.875 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.875 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.875 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.875 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.876 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.876 INFO:teuthology.orchestra.run.smithi027.stdout: "max_completed_events": { 2024-04-25T00:26:06.876 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_completed_events", 2024-04-25T00:26:06.876 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.876 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.877 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.877 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "50", 2024-04-25T00:26:06.877 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.877 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.877 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.877 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "number of past completed events to remember", 2024-04-25T00:26:06.877 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.878 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.878 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.878 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T00:26:06.878 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T00:26:06.878 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.879 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.879 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.879 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T00:26:06.879 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.879 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.881 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.881 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long the module is going to sleep", 2024-04-25T00:26:06.882 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.882 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.882 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.882 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.882 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.883 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.883 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus", 2024-04-25T00:26:06.883 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.883 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.883 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.883 INFO:teuthology.orchestra.run.smithi027.stdout: "cache": { 2024-04-25T00:26:06.884 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cache", 2024-04-25T00:26:06.884 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.884 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.884 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.884 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.884 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.885 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.885 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.885 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.885 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.885 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.885 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.885 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.886 INFO:teuthology.orchestra.run.smithi027.stdout: "exclude_perf_counters": { 2024-04-25T00:26:06.886 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "exclude_perf_counters", 2024-04-25T00:26:06.886 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.886 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.886 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.886 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.886 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.886 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.887 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.887 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T00:26:06.887 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T00:26:06.887 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.887 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.887 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.887 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.887 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.888 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.888 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.888 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.888 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.888 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.888 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.888 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.888 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.889 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.889 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.889 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.889 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.889 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.889 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.889 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.890 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.890 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.890 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.890 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.890 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.890 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.890 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.891 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.891 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.891 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.891 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.891 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.891 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.891 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.891 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.891 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.891 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.891 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.891 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.891 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.891 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.891 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.892 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.892 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.892 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.892 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.892 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.892 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.892 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.892 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.892 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.892 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.892 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.892 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.892 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.892 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.893 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.893 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.893 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.893 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.893 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.893 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.893 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.893 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.893 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.893 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.893 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.893 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.893 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.893 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.894 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools": { 2024-04-25T00:26:06.894 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools", 2024-04-25T00:26:06.894 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.894 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.894 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.894 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.894 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.894 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.894 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.894 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.894 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.894 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.894 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.895 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T00:26:06.895 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T00:26:06.895 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.895 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.895 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.895 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "300", 2024-04-25T00:26:06.895 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.895 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.895 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.895 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.895 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.895 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.895 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.896 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_interval": { 2024-04-25T00:26:06.896 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_interval", 2024-04-25T00:26:06.896 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:06.896 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.896 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.896 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15.0", 2024-04-25T00:26:06.896 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.896 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.896 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.896 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.896 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.896 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.896 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.897 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T00:26:06.897 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T00:26:06.897 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.897 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.897 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.897 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T00:26:06.897 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.897 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.897 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.897 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T00:26:06.897 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.897 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.897 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.897 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.897 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T00:26:06.898 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T00:26:06.898 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.898 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.898 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.898 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "9283", 2024-04-25T00:26:06.898 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.898 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.898 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.898 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T00:26:06.898 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.898 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.898 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.899 INFO:teuthology.orchestra.run.smithi027.stdout: "stale_cache_strategy": { 2024-04-25T00:26:06.899 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stale_cache_strategy", 2024-04-25T00:26:06.899 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.899 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.899 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.899 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "log", 2024-04-25T00:26:06.899 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.899 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.899 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.899 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.899 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.899 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.899 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.900 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.900 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T00:26:06.900 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T00:26:06.900 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.900 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.900 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.900 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "default", 2024-04-25T00:26:06.900 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.900 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.900 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.900 INFO:teuthology.orchestra.run.smithi027.stdout: "default", 2024-04-25T00:26:06.900 INFO:teuthology.orchestra.run.smithi027.stdout: "error" 2024-04-25T00:26:06.900 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.900 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.901 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.901 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.901 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.901 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.901 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T00:26:06.901 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T00:26:06.901 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.901 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.901 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.901 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T00:26:06.901 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T00:26:06.901 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T00:26:06.901 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.901 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.901 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.902 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.902 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.902 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.902 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.902 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.902 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.902 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_support", 2024-04-25T00:26:06.902 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.902 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.902 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.904 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.904 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.904 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.904 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.904 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.904 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.904 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.904 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.904 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.904 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.904 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.904 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.904 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.904 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.904 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.905 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.905 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.905 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.905 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.905 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.905 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.905 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.905 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.905 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.905 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.905 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.905 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.905 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.905 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.906 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.906 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.906 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.906 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.906 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.906 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.906 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.906 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.906 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.906 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.906 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.906 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.906 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.907 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.907 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.907 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.907 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.907 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.907 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.907 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.907 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.907 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.907 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.907 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.907 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_snap_create": { 2024-04-25T00:26:06.907 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_snap_create", 2024-04-25T00:26:06.907 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.907 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.908 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.908 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:06.908 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.908 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.908 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.908 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.908 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.908 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.908 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.908 INFO:teuthology.orchestra.run.smithi027.stdout: "mirror_snapshot_schedule": { 2024-04-25T00:26:06.908 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T00:26:06.908 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.908 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.908 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.908 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.909 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.909 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.909 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.909 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.909 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.909 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.909 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.909 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.909 INFO:teuthology.orchestra.run.smithi027.stdout: "trash_purge_schedule": { 2024-04-25T00:26:06.909 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "trash_purge_schedule", 2024-04-25T00:26:06.909 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.909 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.909 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.909 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.909 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.910 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.910 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.910 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.910 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.910 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.910 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.910 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.910 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.910 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.910 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "restful", 2024-04-25T00:26:06.910 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.910 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.910 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.910 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_auth": { 2024-04-25T00:26:06.911 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_auth", 2024-04-25T00:26:06.911 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.911 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.911 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.911 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.911 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.911 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.911 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.911 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.911 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.911 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.911 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.911 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T00:26:06.911 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T00:26:06.912 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.912 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.912 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.912 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.912 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.912 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.912 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.912 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.912 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.912 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.912 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.913 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.913 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.913 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.913 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.913 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.913 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.913 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.913 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.913 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.913 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.913 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.913 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.913 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.913 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.913 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.914 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.914 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.914 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.914 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.914 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.914 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.914 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.914 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.914 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.914 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.915 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.915 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.915 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.915 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.915 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.915 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.915 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.915 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.915 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.915 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.915 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.915 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.915 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.915 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.916 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.916 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.916 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.916 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.916 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.916 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.916 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.916 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.916 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.916 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.916 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.916 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.916 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.916 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.916 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.917 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.917 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.917 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.917 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.917 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.917 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.917 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.917 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.917 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.917 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.917 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.917 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.917 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.917 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.917 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.918 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.918 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T00:26:06.918 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T00:26:06.918 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.918 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.918 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.918 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.918 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.918 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.918 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.918 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.918 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.918 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.918 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.919 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.919 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T00:26:06.919 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T00:26:06.919 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.919 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.919 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.919 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.919 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.919 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.919 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.919 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.919 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.919 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.919 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.920 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.920 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.920 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.920 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.920 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rgw", 2024-04-25T00:26:06.920 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.920 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.920 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.920 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.920 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.920 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.920 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.920 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.920 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.920 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.921 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.921 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.921 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.921 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.921 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.921 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.921 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.921 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.921 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.921 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.921 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.921 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.921 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.921 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.921 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.922 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.922 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.922 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.922 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.922 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.922 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.922 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.922 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.922 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.922 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.922 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.922 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.922 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.923 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.923 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.923 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.923 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.923 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.923 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.923 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.923 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.923 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.923 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.923 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.923 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.923 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.923 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.924 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.924 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.924 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.924 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.924 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.924 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.924 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.924 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.924 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.924 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.924 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.924 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.924 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.924 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.925 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.925 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.925 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.925 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.925 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.925 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.925 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.925 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.925 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rook", 2024-04-25T00:26:06.925 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.925 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.925 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.925 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.925 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.925 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.926 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.926 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.926 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.926 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.926 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.926 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.926 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.926 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.926 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.926 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.926 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.926 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.926 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.927 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.927 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.927 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.927 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.927 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.927 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.927 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.927 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.927 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.927 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.927 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.927 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.927 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.927 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.927 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.928 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.928 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.928 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.928 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.928 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.928 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.928 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.928 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.928 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.928 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.928 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.928 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.928 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.929 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.929 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.929 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.929 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.929 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.929 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.929 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.929 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.929 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.929 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.929 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.929 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.929 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.930 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.930 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.930 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.930 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.930 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.930 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.930 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.930 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.930 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.930 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.930 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.930 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.930 INFO:teuthology.orchestra.run.smithi027.stdout: "storage_class": { 2024-04-25T00:26:06.930 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "storage_class", 2024-04-25T00:26:06.931 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.931 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.931 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.931 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "local", 2024-04-25T00:26:06.931 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.931 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.931 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.931 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T00:26:06.931 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.931 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.931 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.931 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.931 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.932 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.932 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.932 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "selftest", 2024-04-25T00:26:06.932 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.932 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.932 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.932 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.932 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.932 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.932 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.932 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.932 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.932 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.932 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.932 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.933 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.933 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.933 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.933 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.933 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.933 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.933 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.933 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.933 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.933 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.933 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.933 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.933 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.933 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.934 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.934 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.934 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.934 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.934 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.934 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.934 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.934 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.934 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.934 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.934 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.934 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.934 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.935 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.935 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.935 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.935 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.935 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.935 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.935 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.935 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.935 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.935 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.935 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.935 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.935 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.935 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.935 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.936 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.936 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.936 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.936 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.936 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.936 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.936 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.936 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.936 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.936 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.936 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.936 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.936 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.937 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.937 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.937 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.937 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.937 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.937 INFO:teuthology.orchestra.run.smithi027.stdout: "roption1": { 2024-04-25T00:26:06.937 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption1", 2024-04-25T00:26:06.937 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.937 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.937 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.937 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.937 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.937 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.937 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.937 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.938 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.938 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.938 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.938 INFO:teuthology.orchestra.run.smithi027.stdout: "roption2": { 2024-04-25T00:26:06.938 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption2", 2024-04-25T00:26:06.938 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.938 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.938 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.938 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "xyz", 2024-04-25T00:26:06.938 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.938 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.938 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.938 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.939 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.939 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.939 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.939 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption1": { 2024-04-25T00:26:06.939 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption1", 2024-04-25T00:26:06.939 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.939 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.939 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.939 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.939 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.939 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.939 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.939 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.939 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.940 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.940 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.940 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption2": { 2024-04-25T00:26:06.940 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption2", 2024-04-25T00:26:06.940 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.940 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.940 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.940 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.940 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.940 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.940 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.940 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.941 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.941 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.941 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.941 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption3": { 2024-04-25T00:26:06.941 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption3", 2024-04-25T00:26:06.941 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:06.941 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.941 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.941 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.941 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.941 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.941 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.941 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.942 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.942 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.942 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.942 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.942 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption4": { 2024-04-25T00:26:06.942 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption4", 2024-04-25T00:26:06.942 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.942 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.942 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.942 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.942 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.942 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.942 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.943 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.943 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.943 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.943 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.943 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption5": { 2024-04-25T00:26:06.943 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption5", 2024-04-25T00:26:06.943 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.943 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.943 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.943 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.944 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.944 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.944 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.944 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.944 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.944 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.944 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.944 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption6": { 2024-04-25T00:26:06.944 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption6", 2024-04-25T00:26:06.944 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.944 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.944 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.944 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.944 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.945 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.945 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.945 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.945 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.945 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.945 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.945 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption7": { 2024-04-25T00:26:06.945 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption7", 2024-04-25T00:26:06.945 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.945 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.945 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.945 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.945 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T00:26:06.946 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "42", 2024-04-25T00:26:06.946 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.946 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.946 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.946 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.946 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.946 INFO:teuthology.orchestra.run.smithi027.stdout: "testkey": { 2024-04-25T00:26:06.946 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testkey", 2024-04-25T00:26:06.946 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.946 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.946 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.946 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.947 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.947 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.947 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.947 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.947 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.947 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.947 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.947 INFO:teuthology.orchestra.run.smithi027.stdout: "testlkey": { 2024-04-25T00:26:06.947 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testlkey", 2024-04-25T00:26:06.947 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.947 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.947 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.947 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.948 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.948 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.948 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.948 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.948 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.948 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.948 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.948 INFO:teuthology.orchestra.run.smithi027.stdout: "testnewline": { 2024-04-25T00:26:06.948 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testnewline", 2024-04-25T00:26:06.948 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.948 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.948 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.948 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.948 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.949 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.949 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.949 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.949 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.949 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.949 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.949 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.949 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.949 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-04-25T00:26:06.949 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.949 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.949 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.949 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_m_granularity": { 2024-04-25T00:26:06.950 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_m_granularity", 2024-04-25T00:26:06.950 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.950 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.950 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.950 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.950 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.950 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.950 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.950 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T00:26:06.950 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.950 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.950 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.950 INFO:teuthology.orchestra.run.smithi027.stdout: "dump_on_update": { 2024-04-25T00:26:06.951 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dump_on_update", 2024-04-25T00:26:06.951 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.951 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.951 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.951 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.951 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.951 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.951 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.951 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "dump database to debug log on update", 2024-04-25T00:26:06.951 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.951 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.951 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.951 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.951 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.951 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.952 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.952 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.952 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.952 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.952 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.952 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.952 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.952 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.952 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.952 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.952 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.952 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.952 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.952 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.952 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.953 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.953 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.953 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.953 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.953 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.953 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.953 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.953 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.953 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.953 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.953 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.953 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.953 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.953 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.954 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.954 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.954 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.954 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.954 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.954 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.954 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.954 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.954 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.954 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.954 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.954 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.955 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.955 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.955 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.955 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.955 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.955 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.955 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.955 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.955 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.955 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.955 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.956 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.956 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.956 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.956 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.956 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.956 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.956 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.956 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.956 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.956 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.956 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.956 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.956 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.956 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.957 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stats", 2024-04-25T00:26:06.957 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.957 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.957 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.957 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.957 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.957 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.957 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.957 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.957 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.957 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.957 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.957 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.957 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.958 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.958 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.958 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.958 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.958 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.958 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.958 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.958 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.958 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.958 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.958 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.958 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.958 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.959 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.959 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.959 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.959 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.959 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.959 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.959 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.959 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.959 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.959 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.960 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.960 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.960 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.960 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.960 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.960 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.960 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.960 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.960 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.960 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.960 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.960 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.960 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.961 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.961 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.961 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.961 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.961 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.961 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.961 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.961 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.961 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.961 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.961 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.961 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.961 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.962 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.962 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.962 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.962 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.962 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.962 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.962 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.962 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.962 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.962 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "status", 2024-04-25T00:26:06.962 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.963 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.963 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.963 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.963 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.963 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.963 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.963 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.963 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.963 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.963 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.963 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.963 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.963 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.963 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.963 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.964 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.964 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.964 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.964 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.964 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.964 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.964 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.964 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.964 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.964 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.964 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.964 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.964 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.965 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.965 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.965 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.965 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.965 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.965 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.965 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.965 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.965 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.965 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.965 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.965 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.965 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.966 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.966 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.966 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.966 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.966 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.966 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.966 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.966 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.966 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.966 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.966 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.966 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.966 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.966 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.967 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.967 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.967 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.967 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.967 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.967 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.967 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.967 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.967 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.967 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.967 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.967 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.967 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.967 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.968 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.968 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telegraf", 2024-04-25T00:26:06.968 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.968 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.968 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.968 INFO:teuthology.orchestra.run.smithi027.stdout: "address": { 2024-04-25T00:26:06.968 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "address", 2024-04-25T00:26:06.968 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.968 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.968 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.968 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T00:26:06.968 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.969 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.969 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.969 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.969 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.969 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.969 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.969 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T00:26:06.969 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T00:26:06.969 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:06.969 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.969 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.969 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15", 2024-04-25T00:26:06.969 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.970 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.970 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.970 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.970 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.970 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.970 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.970 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:06.970 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:06.970 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.970 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.970 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.970 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.970 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.971 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.971 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.971 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.971 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.971 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.971 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.971 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.971 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.971 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.971 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.971 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.972 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.972 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.972 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:06.972 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:06.972 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.972 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.972 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.972 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.972 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.972 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.972 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.972 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.972 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.973 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.973 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.973 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:06.973 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:06.973 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.973 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.973 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.973 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:06.973 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.973 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.973 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:06.973 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:06.973 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.975 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.975 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:06.975 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.975 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.975 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.975 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.975 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.975 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.975 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.975 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.975 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.976 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:06.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:06.976 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telemetry", 2024-04-25T00:26:06.976 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:06.976 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:06.977 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:06.977 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_basic": { 2024-04-25T00:26:06.977 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_basic", 2024-04-25T00:26:06.977 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.977 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.978 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.978 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.978 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.978 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.978 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.978 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T00:26:06.979 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.979 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.979 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.979 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_crash": { 2024-04-25T00:26:06.980 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_crash", 2024-04-25T00:26:06.980 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.980 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.980 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.980 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.980 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.981 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.981 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.981 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T00:26:06.981 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.981 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.982 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.982 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_device": { 2024-04-25T00:26:06.982 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_device", 2024-04-25T00:26:06.982 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.982 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.983 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.983 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:06.983 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.983 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.983 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.983 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T00:26:06.984 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.984 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.984 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.984 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_ident": { 2024-04-25T00:26:06.984 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_ident", 2024-04-25T00:26:06.985 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.985 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.985 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.985 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.985 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.985 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.986 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.986 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T00:26:06.986 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.986 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.986 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.987 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_perf": { 2024-04-25T00:26:06.987 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_perf", 2024-04-25T00:26:06.987 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.987 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.987 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.988 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.988 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.988 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.988 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.988 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T00:26:06.988 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.989 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.989 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.989 INFO:teuthology.orchestra.run.smithi027.stdout: "contact": { 2024-04-25T00:26:06.989 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "contact", 2024-04-25T00:26:06.989 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.990 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.990 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.990 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.990 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.990 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.990 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.991 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.991 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.991 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.991 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.991 INFO:teuthology.orchestra.run.smithi027.stdout: "description": { 2024-04-25T00:26:06.992 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "description", 2024-04-25T00:26:06.992 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.992 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.992 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.992 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:06.993 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.993 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.993 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.993 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.993 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.993 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.994 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.994 INFO:teuthology.orchestra.run.smithi027.stdout: "device_url": { 2024-04-25T00:26:06.994 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_url", 2024-04-25T00:26:06.994 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:06.994 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.995 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.995 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T00:26:06.995 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.995 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.995 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.996 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.996 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.996 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.996 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.997 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T00:26:06.997 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T00:26:06.997 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:06.997 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.997 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.997 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:06.997 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:06.997 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.998 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.998 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.998 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.998 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.998 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.998 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T00:26:06.998 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T00:26:06.998 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.998 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.998 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.998 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "24", 2024-04-25T00:26:06.998 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "8", 2024-04-25T00:26:06.999 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:06.999 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:06.999 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:06.999 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:06.999 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:06.999 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:06.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:06.999 INFO:teuthology.orchestra.run.smithi027.stdout: "last_opt_revision": { 2024-04-25T00:26:06.999 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "last_opt_revision", 2024-04-25T00:26:06.999 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:06.999 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:06.999 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:06.999 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1", 2024-04-25T00:26:06.999 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.000 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.000 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.000 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.000 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.000 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.000 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.001 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard": { 2024-04-25T00:26:07.001 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard", 2024-04-25T00:26:07.001 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.001 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.001 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.001 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.001 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.001 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.001 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.001 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.001 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.001 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.001 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.002 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard_description": { 2024-04-25T00:26:07.002 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard_description", 2024-04-25T00:26:07.002 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.002 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.002 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.002 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.002 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.002 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.002 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.002 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.003 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.003 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.003 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.003 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.003 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.003 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.003 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.003 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.003 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.003 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.008 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.008 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.008 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.008 INFO:teuthology.orchestra.run.smithi027.stdout: "organization": { 2024-04-25T00:26:07.008 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "organization", 2024-04-25T00:26:07.008 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.008 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.008 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.008 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.008 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.008 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.008 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.008 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.009 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.009 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.009 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.009 INFO:teuthology.orchestra.run.smithi027.stdout: "proxy": { 2024-04-25T00:26:07.009 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "proxy", 2024-04-25T00:26:07.009 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.009 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.009 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.009 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.009 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.009 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.009 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.009 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.009 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.010 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.010 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.010 INFO:teuthology.orchestra.run.smithi027.stdout: "url": { 2024-04-25T00:26:07.010 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url", 2024-04-25T00:26:07.010 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.010 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.010 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.010 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T00:26:07.010 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.010 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.010 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.010 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.011 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.011 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.011 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.011 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.011 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.011 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "test_orchestrator", 2024-04-25T00:26:07.011 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.011 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.011 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.011 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.011 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.011 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.012 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.012 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.012 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.012 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.012 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.012 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.012 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.012 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.012 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.012 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.012 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.012 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.012 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.012 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.013 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.013 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.013 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.013 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.013 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.013 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.013 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.013 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.013 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.013 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.013 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.013 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.013 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.014 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.014 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.014 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.014 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.014 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.014 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.014 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.014 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.014 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.014 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.014 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.014 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.014 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.015 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.015 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.015 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.015 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.015 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.015 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.015 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.015 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.015 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.015 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.015 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.016 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.016 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.016 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.016 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.016 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.016 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.016 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.016 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.016 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.016 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.016 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.016 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.016 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.016 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.017 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "volumes", 2024-04-25T00:26:07.017 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.017 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.017 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.017 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.017 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.017 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.017 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.017 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.017 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.017 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.017 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.018 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.018 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.018 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.018 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.018 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.018 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.018 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.018 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.018 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.018 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.018 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.018 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.019 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.019 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.019 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.019 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.019 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.019 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.019 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.019 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.019 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.019 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.019 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.019 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.019 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.019 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.020 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.020 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.020 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.020 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.020 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.020 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.020 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.020 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.020 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.020 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.020 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.020 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.020 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.020 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.021 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.021 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.021 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.021 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.021 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.021 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.021 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.021 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.021 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.021 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.021 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.021 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.021 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.021 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.022 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.022 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.022 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.022 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.022 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_clones": { 2024-04-25T00:26:07.022 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_clones", 2024-04-25T00:26:07.022 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.022 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.022 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.022 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4", 2024-04-25T00:26:07.022 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.022 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.022 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.023 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T00:26:07.023 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.023 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.023 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.023 INFO:teuthology.orchestra.run.smithi027.stdout: "periodic_async_work": { 2024-04-25T00:26:07.023 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "periodic_async_work", 2024-04-25T00:26:07.023 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.023 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.023 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.023 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.023 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.023 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.023 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.023 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Periodically check for async work", 2024-04-25T00:26:07.024 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.024 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.024 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.024 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_delay": { 2024-04-25T00:26:07.024 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_delay", 2024-04-25T00:26:07.024 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.024 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.024 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.024 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T00:26:07.024 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.024 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.024 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.025 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T00:26:07.025 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.025 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.025 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.025 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_no_wait": { 2024-04-25T00:26:07.025 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T00:26:07.025 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.025 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.025 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.025 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.025 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.025 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.025 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.025 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T00:26:07.026 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.026 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.026 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.026 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.026 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.026 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix", 2024-04-25T00:26:07.026 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.026 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.026 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.026 INFO:teuthology.orchestra.run.smithi027.stdout: "discovery_interval": { 2024-04-25T00:26:07.026 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "discovery_interval", 2024-04-25T00:26:07.026 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T00:26:07.027 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.027 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.027 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "100", 2024-04-25T00:26:07.027 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.027 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.027 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.027 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.027 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.027 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.027 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.027 INFO:teuthology.orchestra.run.smithi027.stdout: "identifier": { 2024-04-25T00:26:07.027 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "identifier", 2024-04-25T00:26:07.027 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.028 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.028 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.028 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.028 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.028 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.028 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.028 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.028 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.028 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.028 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.028 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T00:26:07.028 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T00:26:07.028 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.028 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.029 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.029 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T00:26:07.029 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.029 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.029 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.029 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.029 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.029 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.029 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.029 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.029 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.029 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.029 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.029 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.029 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.030 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.030 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.030 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.030 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.030 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.030 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.030 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.030 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.030 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.030 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.030 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.030 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.030 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.030 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.031 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.031 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.031 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.031 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.031 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.031 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.031 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.031 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.031 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.031 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.031 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.031 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.031 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.032 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.032 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.032 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.032 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.032 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.032 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.032 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.032 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.032 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.032 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.032 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.032 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.032 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.032 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.033 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.033 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.033 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.033 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.033 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.033 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.033 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.033 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.033 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.033 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.033 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.033 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.033 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.033 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.034 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.034 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.034 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.034 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.034 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.034 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_host": { 2024-04-25T00:26:07.034 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_host", 2024-04-25T00:26:07.034 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.034 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.034 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.034 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.034 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.035 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.035 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.035 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.035 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.035 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.035 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.035 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_port": { 2024-04-25T00:26:07.035 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_port", 2024-04-25T00:26:07.035 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.035 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.035 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.035 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10051", 2024-04-25T00:26:07.036 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.036 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.036 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.036 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.036 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.036 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.036 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.036 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_sender": { 2024-04-25T00:26:07.037 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_sender", 2024-04-25T00:26:07.037 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.037 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.037 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.037 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T00:26:07.037 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.037 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.037 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.037 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.037 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.037 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T00:26:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 25241, 2024-04-25T00:26:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "y", 2024-04-25T00:26:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_features": 4540701547738038271, 2024-04-25T00:26:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: "available_modules": [ 2024-04-25T00:26:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "alerts", 2024-04-25T00:26:07.039 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.039 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.039 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.039 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T00:26:07.039 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T00:26:07.039 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.039 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.039 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.039 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T00:26:07.039 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.039 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.039 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.039 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T00:26:07.039 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.039 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.040 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.040 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.040 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.040 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.040 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.040 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.040 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.040 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.040 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.040 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.040 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.040 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.040 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.041 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.041 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.041 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.041 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.041 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.041 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.041 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.041 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.041 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.041 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.041 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.041 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.041 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.042 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.042 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.042 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.042 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.042 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.042 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.042 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.042 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.042 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.042 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.042 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.042 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.043 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.043 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.043 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.043 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.043 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.043 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.043 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.043 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.043 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.043 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.043 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.043 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.043 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.043 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.044 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.044 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.044 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.044 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.044 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.044 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.044 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.044 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.044 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.044 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.044 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.044 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.045 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.045 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.045 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.045 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_destination": { 2024-04-25T00:26:07.045 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_destination", 2024-04-25T00:26:07.045 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.045 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.045 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.045 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.045 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.045 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.045 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.045 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email address to send alerts to", 2024-04-25T00:26:07.045 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.046 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.046 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.046 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_from_name": { 2024-04-25T00:26:07.046 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_from_name", 2024-04-25T00:26:07.046 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.046 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.046 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.046 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "Ceph", 2024-04-25T00:26:07.046 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.046 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.046 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.046 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email From: name", 2024-04-25T00:26:07.046 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.047 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.047 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.047 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_host": { 2024-04-25T00:26:07.047 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_host", 2024-04-25T00:26:07.047 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.047 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.047 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.047 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.047 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.047 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.047 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.047 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP server", 2024-04-25T00:26:07.047 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.047 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.048 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.048 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_password": { 2024-04-25T00:26:07.048 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_password", 2024-04-25T00:26:07.048 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.048 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.048 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.048 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.048 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.048 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.048 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.048 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Password to authenticate with", 2024-04-25T00:26:07.048 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.048 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.048 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.049 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_port": { 2024-04-25T00:26:07.049 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_port", 2024-04-25T00:26:07.049 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.049 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.049 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.049 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "465", 2024-04-25T00:26:07.049 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.049 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.049 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.049 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP port", 2024-04-25T00:26:07.049 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.049 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.049 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.049 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_sender": { 2024-04-25T00:26:07.050 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_sender", 2024-04-25T00:26:07.050 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.050 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.050 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.050 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.050 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.050 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.050 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.050 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP envelope sender", 2024-04-25T00:26:07.050 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.050 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.050 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.050 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_ssl": { 2024-04-25T00:26:07.051 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_ssl", 2024-04-25T00:26:07.051 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.051 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.051 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.051 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.051 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.051 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.051 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.051 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T00:26:07.051 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.051 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.051 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.052 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_user": { 2024-04-25T00:26:07.052 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_user", 2024-04-25T00:26:07.052 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.052 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.052 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.052 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.052 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.052 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.052 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.052 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "User to authenticate as", 2024-04-25T00:26:07.052 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.052 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.052 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.052 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.052 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.053 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "balancer", 2024-04-25T00:26:07.053 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.053 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.053 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.053 INFO:teuthology.orchestra.run.smithi027.stdout: "active": { 2024-04-25T00:26:07.053 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "active", 2024-04-25T00:26:07.053 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.053 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.053 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.053 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.053 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.053 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.053 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.054 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T00:26:07.054 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.054 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.054 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.054 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_time": { 2024-04-25T00:26:07.054 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_time", 2024-04-25T00:26:07.054 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.054 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.054 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.054 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0000", 2024-04-25T00:26:07.054 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.054 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.054 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.055 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T00:26:07.055 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T00:26:07.055 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.055 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.055 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_weekday": { 2024-04-25T00:26:07.055 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_weekday", 2024-04-25T00:26:07.055 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T00:26:07.055 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.055 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.055 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T00:26:07.055 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T00:26:07.055 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T00:26:07.055 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.056 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T00:26:07.056 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T00:26:07.056 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.056 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.056 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_max_iterations": { 2024-04-25T00:26:07.056 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_max_iterations", 2024-04-25T00:26:07.056 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T00:26:07.056 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.056 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.056 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "25", 2024-04-25T00:26:07.056 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T00:26:07.056 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "250", 2024-04-25T00:26:07.056 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.057 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T00:26:07.057 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.057 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.057 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.057 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_metrics": { 2024-04-25T00:26:07.057 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_metrics", 2024-04-25T00:26:07.057 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.057 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.057 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.057 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T00:26:07.057 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.057 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.057 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.058 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T00:26:07.058 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T00:26:07.058 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.058 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.058 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_step": { 2024-04-25T00:26:07.058 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_step", 2024-04-25T00:26:07.058 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:07.058 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.058 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.058 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.5", 2024-04-25T00:26:07.058 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0.001", 2024-04-25T00:26:07.059 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "0.999", 2024-04-25T00:26:07.059 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.059 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "aggressiveness of optimization", 2024-04-25T00:26:07.059 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T00:26:07.059 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.059 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.059 INFO:teuthology.orchestra.run.smithi027.stdout: "end_time": { 2024-04-25T00:26:07.059 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_time", 2024-04-25T00:26:07.059 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.059 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.060 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.060 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2359", 2024-04-25T00:26:07.060 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.060 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.060 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.060 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T00:26:07.060 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T00:26:07.060 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.060 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.060 INFO:teuthology.orchestra.run.smithi027.stdout: "end_weekday": { 2024-04-25T00:26:07.060 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_weekday", 2024-04-25T00:26:07.060 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T00:26:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T00:26:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T00:26:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T00:26:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T00:26:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T00:26:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.062 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.062 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.062 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.062 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.062 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.062 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.062 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.062 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.062 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.062 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.062 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.062 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.062 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.063 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.063 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.063 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.063 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.063 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.063 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.063 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.063 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.063 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.063 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.063 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.063 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.063 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.063 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.064 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.064 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.064 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.064 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.064 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.064 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.064 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.064 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.064 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.064 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.064 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.064 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.065 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.065 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.065 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.065 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.065 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.065 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.065 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.065 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.065 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.065 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.065 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.065 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.066 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.066 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.066 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.066 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.066 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.066 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.066 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.066 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.066 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.066 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.066 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.066 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.067 INFO:teuthology.orchestra.run.smithi027.stdout: "min_score": { 2024-04-25T00:26:07.067 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_score", 2024-04-25T00:26:07.067 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:07.067 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.067 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.067 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T00:26:07.067 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.067 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.067 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.067 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T00:26:07.067 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.067 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.067 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.068 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T00:26:07.068 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T00:26:07.068 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.068 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.068 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.068 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "upmap", 2024-04-25T00:26:07.068 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.068 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.068 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.068 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-compat", 2024-04-25T00:26:07.068 INFO:teuthology.orchestra.run.smithi027.stdout: "none", 2024-04-25T00:26:07.068 INFO:teuthology.orchestra.run.smithi027.stdout: "read", 2024-04-25T00:26:07.068 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap", 2024-04-25T00:26:07.068 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap-read" 2024-04-25T00:26:07.069 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.069 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Balancer mode", 2024-04-25T00:26:07.069 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.069 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.069 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.069 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_ids": { 2024-04-25T00:26:07.069 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_ids", 2024-04-25T00:26:07.069 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.069 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.069 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.070 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.070 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.070 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.070 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.070 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T00:26:07.070 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.070 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.070 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.070 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T00:26:07.071 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T00:26:07.071 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.071 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.071 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.071 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T00:26:07.071 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.071 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.071 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.071 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T00:26:07.071 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.072 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.072 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.072 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_deviation": { 2024-04-25T00:26:07.072 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_deviation", 2024-04-25T00:26:07.072 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.072 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.072 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.072 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T00:26:07.072 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T00:26:07.072 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.072 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.072 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T00:26:07.072 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T00:26:07.073 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.073 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.073 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.073 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_optimizations": { 2024-04-25T00:26:07.073 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_optimizations", 2024-04-25T00:26:07.073 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T00:26:07.073 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.073 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.073 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:07.073 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.073 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.073 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.073 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T00:26:07.073 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.073 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.074 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.074 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.074 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.074 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.074 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm", 2024-04-25T00:26:07.074 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.074 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.074 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.074 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_down_multiplier": { 2024-04-25T00:26:07.074 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_down_multiplier", 2024-04-25T00:26:07.074 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:07.075 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.075 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.075 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T00:26:07.075 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.075 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.075 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.075 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T00:26:07.075 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.075 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.075 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.075 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_refresh_rate": { 2024-04-25T00:26:07.075 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_refresh_rate", 2024-04-25T00:26:07.075 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.075 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.076 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.076 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "20", 2024-04-25T00:26:07.076 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.076 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.076 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.076 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T00:26:07.076 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.076 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.076 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.076 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_starting_port": { 2024-04-25T00:26:07.076 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_starting_port", 2024-04-25T00:26:07.076 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.076 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.076 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.076 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4721", 2024-04-25T00:26:07.077 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.077 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.077 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.077 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T00:26:07.077 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.077 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.077 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.077 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_ptrace": { 2024-04-25T00:26:07.077 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_ptrace", 2024-04-25T00:26:07.077 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.077 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.077 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.077 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.077 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.078 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.078 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.078 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T00:26:07.078 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T00:26:07.078 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.078 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.078 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_interval": { 2024-04-25T00:26:07.078 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_interval", 2024-04-25T00:26:07.078 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.078 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.078 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.079 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T00:26:07.079 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.079 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.079 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.079 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T00:26:07.079 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.079 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.079 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.079 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_memory_target_ratio": { 2024-04-25T00:26:07.079 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T00:26:07.079 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:07.079 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.079 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.079 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.7", 2024-04-25T00:26:07.079 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.080 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.080 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.080 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T00:26:07.080 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.080 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.080 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.080 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm_log_destination": { 2024-04-25T00:26:07.080 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm_log_destination", 2024-04-25T00:26:07.080 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.080 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.080 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.080 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.080 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.080 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.080 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.081 INFO:teuthology.orchestra.run.smithi027.stdout: "file", 2024-04-25T00:26:07.081 INFO:teuthology.orchestra.run.smithi027.stdout: "file,syslog", 2024-04-25T00:26:07.081 INFO:teuthology.orchestra.run.smithi027.stdout: "syslog" 2024-04-25T00:26:07.081 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.081 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T00:26:07.081 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.081 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.081 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.081 INFO:teuthology.orchestra.run.smithi027.stdout: "cgroups_split": { 2024-04-25T00:26:07.081 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cgroups_split", 2024-04-25T00:26:07.081 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.081 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.081 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.081 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.081 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.082 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.082 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.082 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T00:26:07.082 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.082 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.082 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.082 INFO:teuthology.orchestra.run.smithi027.stdout: "config_checks_enabled": { 2024-04-25T00:26:07.082 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_checks_enabled", 2024-04-25T00:26:07.082 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.082 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.083 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.083 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.083 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.083 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.083 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.083 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T00:26:07.083 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.083 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.083 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.084 INFO:teuthology.orchestra.run.smithi027.stdout: "config_dashboard": { 2024-04-25T00:26:07.084 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_dashboard", 2024-04-25T00:26:07.084 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.084 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.084 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.084 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.084 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.084 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.084 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.084 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T00:26:07.084 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.084 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.084 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.085 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_alertmanager": { 2024-04-25T00:26:07.085 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_alertmanager", 2024-04-25T00:26:07.085 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.085 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.085 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.085 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T00:26:07.085 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.085 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.085 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.085 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T00:26:07.085 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.085 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.085 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.086 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_base": { 2024-04-25T00:26:07.086 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_base", 2024-04-25T00:26:07.086 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.086 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.086 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.086 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T00:26:07.086 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.086 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.086 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.086 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Container image name, without the tag", 2024-04-25T00:26:07.086 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.086 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.086 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.087 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_elasticsearch": { 2024-04-25T00:26:07.087 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_elasticsearch", 2024-04-25T00:26:07.087 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.087 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.087 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.087 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T00:26:07.087 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.087 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.087 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.087 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "elasticsearch container image", 2024-04-25T00:26:07.087 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.087 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.088 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.088 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_grafana": { 2024-04-25T00:26:07.088 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_grafana", 2024-04-25T00:26:07.088 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.088 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.088 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.088 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T00:26:07.088 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.088 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.088 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.089 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T00:26:07.089 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.089 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.089 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.089 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_haproxy": { 2024-04-25T00:26:07.089 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_haproxy", 2024-04-25T00:26:07.089 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.089 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.089 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.089 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T00:26:07.089 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.089 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.089 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.090 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "HAproxy container image", 2024-04-25T00:26:07.090 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.090 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.090 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.090 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_agent": { 2024-04-25T00:26:07.090 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_agent", 2024-04-25T00:26:07.090 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.090 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.090 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.090 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T00:26:07.090 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.091 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.091 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.091 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger agent container image", 2024-04-25T00:26:07.091 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.091 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.091 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.091 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_collector": { 2024-04-25T00:26:07.091 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_collector", 2024-04-25T00:26:07.091 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.091 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.091 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.091 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T00:26:07.091 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.092 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.092 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.092 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger collector container image", 2024-04-25T00:26:07.092 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.092 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.092 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.092 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_query": { 2024-04-25T00:26:07.092 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_query", 2024-04-25T00:26:07.092 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.092 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.092 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.092 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T00:26:07.093 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.093 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.093 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.093 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger query container image", 2024-04-25T00:26:07.093 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.093 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.093 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.093 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_keepalived": { 2024-04-25T00:26:07.093 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_keepalived", 2024-04-25T00:26:07.093 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.093 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.093 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.093 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T00:26:07.094 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.094 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.094 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.094 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Keepalived container image", 2024-04-25T00:26:07.094 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.094 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.094 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.094 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_loki": { 2024-04-25T00:26:07.094 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_loki", 2024-04-25T00:26:07.094 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.094 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.094 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.094 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T00:26:07.095 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.095 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.095 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.095 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Loki container image", 2024-04-25T00:26:07.095 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.095 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.095 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.095 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_node_exporter": { 2024-04-25T00:26:07.095 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_node_exporter", 2024-04-25T00:26:07.095 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.095 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.095 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.095 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T00:26:07.096 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.096 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.096 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.096 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T00:26:07.096 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.096 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.096 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.096 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_nvmeof": { 2024-04-25T00:26:07.096 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_nvmeof", 2024-04-25T00:26:07.096 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.096 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.096 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.096 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T00:26:07.096 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.097 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.097 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.097 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Nvme-of container image", 2024-04-25T00:26:07.097 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.097 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.097 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.097 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_prometheus": { 2024-04-25T00:26:07.097 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_prometheus", 2024-04-25T00:26:07.097 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.097 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.097 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.097 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T00:26:07.097 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.098 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.098 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.098 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T00:26:07.098 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.098 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.098 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.098 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_promtail": { 2024-04-25T00:26:07.098 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_promtail", 2024-04-25T00:26:07.098 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.098 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.099 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.099 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T00:26:07.099 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.099 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.099 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.099 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Promtail container image", 2024-04-25T00:26:07.099 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.099 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.099 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.099 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_samba": { 2024-04-25T00:26:07.099 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_samba", 2024-04-25T00:26:07.099 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.099 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.100 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.100 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T00:26:07.100 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.100 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.100 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.100 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Samba/SMB container image", 2024-04-25T00:26:07.100 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.100 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.100 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.100 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_snmp_gateway": { 2024-04-25T00:26:07.100 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_snmp_gateway", 2024-04-25T00:26:07.100 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.100 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.100 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.101 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T00:26:07.101 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.101 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.101 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.101 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SNMP Gateway container image", 2024-04-25T00:26:07.101 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.101 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.101 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.102 INFO:teuthology.orchestra.run.smithi027.stdout: "container_init": { 2024-04-25T00:26:07.102 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_init", 2024-04-25T00:26:07.102 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.102 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.102 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.102 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.102 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.102 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.102 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.102 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T00:26:07.102 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.102 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.103 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.103 INFO:teuthology.orchestra.run.smithi027.stdout: "daemon_cache_timeout": { 2024-04-25T00:26:07.103 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "daemon_cache_timeout", 2024-04-25T00:26:07.103 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.103 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.103 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.103 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T00:26:07.103 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.103 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.104 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.104 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T00:26:07.104 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.104 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.104 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.104 INFO:teuthology.orchestra.run.smithi027.stdout: "default_cephadm_command_timeout": { 2024-04-25T00:26:07.104 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T00:26:07.104 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.104 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.104 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.105 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "900", 2024-04-25T00:26:07.105 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.105 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.105 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.105 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T00:26:07.105 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.105 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.105 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.105 INFO:teuthology.orchestra.run.smithi027.stdout: "default_registry": { 2024-04-25T00:26:07.105 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_registry", 2024-04-25T00:26:07.105 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.105 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.105 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.105 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io", 2024-04-25T00:26:07.106 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.106 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.106 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.106 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T00:26:07.106 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.106 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.106 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.106 INFO:teuthology.orchestra.run.smithi027.stdout: "device_cache_timeout": { 2024-04-25T00:26:07.106 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_cache_timeout", 2024-04-25T00:26:07.106 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.106 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.106 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.107 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1800", 2024-04-25T00:26:07.107 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.107 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.107 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.107 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache device inventory", 2024-04-25T00:26:07.107 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.107 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.107 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.107 INFO:teuthology.orchestra.run.smithi027.stdout: "device_enhanced_scan": { 2024-04-25T00:26:07.107 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_enhanced_scan", 2024-04-25T00:26:07.107 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.107 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.107 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.107 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.108 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.108 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.108 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.108 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T00:26:07.108 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.108 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.108 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.108 INFO:teuthology.orchestra.run.smithi027.stdout: "facts_cache_timeout": { 2024-04-25T00:26:07.108 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "facts_cache_timeout", 2024-04-25T00:26:07.108 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.108 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.108 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.108 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T00:26:07.108 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.109 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.109 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.109 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache host facts data", 2024-04-25T00:26:07.109 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.109 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.109 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.109 INFO:teuthology.orchestra.run.smithi027.stdout: "grafana_dashboards_path": { 2024-04-25T00:26:07.109 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "grafana_dashboards_path", 2024-04-25T00:26:07.109 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.109 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.109 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.109 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T00:26:07.109 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.109 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.110 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.110 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T00:26:07.110 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.110 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.110 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.110 INFO:teuthology.orchestra.run.smithi027.stdout: "host_check_interval": { 2024-04-25T00:26:07.110 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "host_check_interval", 2024-04-25T00:26:07.110 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.110 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.110 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.110 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T00:26:07.110 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.110 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.111 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.111 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to perform a host check", 2024-04-25T00:26:07.111 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.111 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.111 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.111 INFO:teuthology.orchestra.run.smithi027.stdout: "hw_monitoring": { 2024-04-25T00:26:07.111 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hw_monitoring", 2024-04-25T00:26:07.111 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.111 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.111 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.111 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.111 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.111 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.112 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.112 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T00:26:07.112 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.112 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.112 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.112 INFO:teuthology.orchestra.run.smithi027.stdout: "inventory_list_all": { 2024-04-25T00:26:07.112 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "inventory_list_all", 2024-04-25T00:26:07.112 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.112 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.112 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.112 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.112 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.112 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.113 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.113 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T00:26:07.113 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.113 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.113 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.113 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.113 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.113 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.113 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.113 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.113 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.113 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.113 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.113 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.114 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.114 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.114 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.114 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.114 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.114 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.114 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.114 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.114 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.114 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.114 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.115 INFO:teuthology.orchestra.run.smithi027.stdout: "log_refresh_metadata": { 2024-04-25T00:26:07.115 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_refresh_metadata", 2024-04-25T00:26:07.115 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.115 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.115 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.115 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.115 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.115 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.115 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.115 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T00:26:07.115 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.115 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.115 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.116 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.116 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.116 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.116 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.116 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.116 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.116 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.116 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.116 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.116 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T00:26:07.116 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.116 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.116 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.117 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.117 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.117 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.117 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.117 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.117 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.117 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.117 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.117 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.117 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.117 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.117 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.117 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.118 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.118 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.118 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.118 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.118 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.118 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.118 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.118 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.118 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.118 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.118 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.118 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.119 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.119 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.119 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.119 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.119 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.119 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.119 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.119 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.119 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T00:26:07.119 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T00:26:07.119 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.119 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.119 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.120 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.120 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.120 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.120 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.120 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T00:26:07.120 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.120 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.120 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.120 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T00:26:07.120 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T00:26:07.120 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.120 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.120 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.120 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "*", 2024-04-25T00:26:07.120 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.121 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.121 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.121 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T00:26:07.121 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.121 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.121 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.121 INFO:teuthology.orchestra.run.smithi027.stdout: "max_count_per_host": { 2024-04-25T00:26:07.121 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_count_per_host", 2024-04-25T00:26:07.121 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.121 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.121 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.121 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:07.121 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.121 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.121 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.122 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of daemons per service per host", 2024-04-25T00:26:07.122 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.122 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.122 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.122 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd_draining_count": { 2024-04-25T00:26:07.122 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_osd_draining_count", 2024-04-25T00:26:07.122 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.122 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.122 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.122 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:07.122 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.122 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.122 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.123 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T00:26:07.123 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.123 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.123 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.123 INFO:teuthology.orchestra.run.smithi027.stdout: "migration_current": { 2024-04-25T00:26:07.123 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "migration_current", 2024-04-25T00:26:07.123 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.123 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.123 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.123 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.123 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.123 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.123 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.123 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "internal - do not modify", 2024-04-25T00:26:07.123 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.124 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.124 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.124 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T00:26:07.124 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T00:26:07.124 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.124 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.124 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.124 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "root", 2024-04-25T00:26:07.124 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.124 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.124 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.124 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm-package", 2024-04-25T00:26:07.124 INFO:teuthology.orchestra.run.smithi027.stdout: "root" 2024-04-25T00:26:07.124 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.124 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T00:26:07.125 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.125 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.125 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.125 INFO:teuthology.orchestra.run.smithi027.stdout: "oob_default_addr": { 2024-04-25T00:26:07.125 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "oob_default_addr", 2024-04-25T00:26:07.125 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.125 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.125 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.125 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "169.254.1.1", 2024-04-25T00:26:07.125 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.125 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.125 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.125 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T00:26:07.126 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.126 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.126 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.126 INFO:teuthology.orchestra.run.smithi027.stdout: "prometheus_alerts_path": { 2024-04-25T00:26:07.126 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus_alerts_path", 2024-04-25T00:26:07.126 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.126 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.126 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.126 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T00:26:07.126 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.126 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T00:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_insecure": { 2024-04-25T00:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_insecure", 2024-04-25T00:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T00:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_password": { 2024-04-25T00:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_password", 2024-04-25T00:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.130 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.130 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.130 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T00:26:07.130 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.130 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.130 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.131 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_url": { 2024-04-25T00:26:07.131 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_url", 2024-04-25T00:26:07.131 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.131 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.131 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.131 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.131 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.131 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.132 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.132 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T00:26:07.132 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.132 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.132 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.132 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_username": { 2024-04-25T00:26:07.132 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_username", 2024-04-25T00:26:07.132 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.132 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.133 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.133 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.133 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.133 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.133 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.133 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T00:26:07.133 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.133 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.134 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.134 INFO:teuthology.orchestra.run.smithi027.stdout: "secure_monitoring_stack": { 2024-04-25T00:26:07.134 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "secure_monitoring_stack", 2024-04-25T00:26:07.134 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.134 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.134 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.134 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.135 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.135 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.135 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.135 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T00:26:07.135 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.135 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.135 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.135 INFO:teuthology.orchestra.run.smithi027.stdout: "service_discovery_port": { 2024-04-25T00:26:07.136 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "service_discovery_port", 2024-04-25T00:26:07.136 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.136 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.136 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.136 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8765", 2024-04-25T00:26:07.136 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.136 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.136 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.137 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "cephadm service discovery port", 2024-04-25T00:26:07.137 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.137 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.137 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.137 INFO:teuthology.orchestra.run.smithi027.stdout: "ssh_config_file": { 2024-04-25T00:26:07.137 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssh_config_file", 2024-04-25T00:26:07.137 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.138 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.138 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.138 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.138 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.138 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.138 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.138 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T00:26:07.138 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.139 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.139 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.139 INFO:teuthology.orchestra.run.smithi027.stdout: "use_agent": { 2024-04-25T00:26:07.139 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_agent", 2024-04-25T00:26:07.139 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.139 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.139 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.139 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.139 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.139 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.139 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.139 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T00:26:07.140 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.140 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.140 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.140 INFO:teuthology.orchestra.run.smithi027.stdout: "use_repo_digest": { 2024-04-25T00:26:07.140 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_repo_digest", 2024-04-25T00:26:07.140 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.140 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.140 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.140 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.140 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.140 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.140 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T00:26:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_failed_host_check": { 2024-04-25T00:26:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_failed_host_check", 2024-04-25T00:26:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.142 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.142 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T00:26:07.142 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.142 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.142 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.142 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_daemons": { 2024-04-25T00:26:07.142 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_daemons", 2024-04-25T00:26:07.142 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.142 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.142 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.142 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.142 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.142 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.143 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.143 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T00:26:07.143 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.143 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.143 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.143 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_hosts": { 2024-04-25T00:26:07.143 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_hosts", 2024-04-25T00:26:07.143 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.143 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.143 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.143 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.143 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.143 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.143 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.144 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T00:26:07.144 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.144 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.144 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.144 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.144 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.144 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crash", 2024-04-25T00:26:07.144 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.144 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.144 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.144 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.144 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.144 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.145 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.145 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.145 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.145 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.145 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.145 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.145 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.145 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.145 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.145 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.146 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.146 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.146 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.146 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.146 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.146 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.146 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.146 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.146 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.146 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.146 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.146 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.146 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.147 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.147 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.147 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.147 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.147 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.147 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.147 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.147 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.147 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.147 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.147 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.147 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.147 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.148 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.148 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.148 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.148 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.148 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.148 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.148 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.148 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.148 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.148 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.148 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.148 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.148 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.148 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.149 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.149 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.149 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.149 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.149 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.149 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.149 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.149 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.149 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.149 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.149 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.149 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.149 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.150 INFO:teuthology.orchestra.run.smithi027.stdout: "retain_interval": { 2024-04-25T00:26:07.150 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retain_interval", 2024-04-25T00:26:07.150 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.150 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.150 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.150 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "31536000", 2024-04-25T00:26:07.150 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.150 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.150 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.150 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T00:26:07.150 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.150 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.150 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.151 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_recent_interval": { 2024-04-25T00:26:07.151 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_recent_interval", 2024-04-25T00:26:07.151 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.151 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.151 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.151 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1209600", 2024-04-25T00:26:07.151 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.151 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.151 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.151 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T00:26:07.151 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.151 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.151 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.151 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.151 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dashboard", 2024-04-25T00:26:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T00:26:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T00:26:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.153 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.153 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.153 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T00:26:07.153 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T00:26:07.153 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.153 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.153 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.153 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.153 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.153 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.153 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.153 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.153 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.153 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.153 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.154 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T00:26:07.154 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T00:26:07.154 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.154 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.154 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.154 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.154 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.154 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.154 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.154 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.154 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.154 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.154 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.155 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_ENABLED": { 2024-04-25T00:26:07.155 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T00:26:07.155 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.155 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.155 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.155 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.155 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.155 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.155 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.155 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.155 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.155 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.155 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.155 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T00:26:07.155 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T00:26:07.156 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.156 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.156 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.156 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.156 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.156 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.156 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.156 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.156 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.156 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.156 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.156 INFO:teuthology.orchestra.run.smithi027.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T00:26:07.156 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T00:26:07.156 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.157 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.157 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.157 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.157 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.157 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.157 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.157 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.157 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.157 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.157 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.157 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T00:26:07.157 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T00:26:07.157 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.157 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.157 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.158 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.158 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.158 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.158 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.158 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.158 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.158 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.158 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.158 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T00:26:07.158 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T00:26:07.158 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.158 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.159 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.159 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.159 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.159 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.159 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.159 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.159 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.159 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.159 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.159 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T00:26:07.159 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T00:26:07.159 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.159 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.159 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.159 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.160 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.160 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.160 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.160 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.160 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.160 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.160 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.160 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T00:26:07.160 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T00:26:07.160 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.160 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.160 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.161 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.161 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.161 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.161 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.161 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.161 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.161 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.161 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.161 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T00:26:07.161 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T00:26:07.162 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.162 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.162 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.162 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.162 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.162 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.162 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.162 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.162 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.162 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.162 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.162 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T00:26:07.163 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T00:26:07.163 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.163 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.163 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.163 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.163 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.163 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.163 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.163 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.163 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.163 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.163 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.163 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T00:26:07.164 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T00:26:07.164 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.164 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.164 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.164 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.164 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.164 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.164 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.164 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.164 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.164 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.164 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.164 INFO:teuthology.orchestra.run.smithi027.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T00:26:07.165 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T00:26:07.165 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.165 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.165 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.165 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.165 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.165 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.165 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.165 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.165 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.165 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.165 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.165 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T00:26:07.165 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T00:26:07.166 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.166 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.166 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.166 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T00:26:07.166 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.166 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.166 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.166 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.166 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.166 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.166 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.166 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T00:26:07.167 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T00:26:07.167 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.167 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.167 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.167 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.167 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.167 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.167 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.167 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.167 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.167 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.167 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.167 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_URL": { 2024-04-25T00:26:07.167 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_URL", 2024-04-25T00:26:07.168 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.168 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.168 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.168 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.168 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.168 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.168 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.168 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.168 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.168 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.168 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.168 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T00:26:07.168 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T00:26:07.168 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.169 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.169 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.169 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T00:26:07.169 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.169 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.169 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.169 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.169 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.169 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.169 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.169 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T00:26:07.169 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T00:26:07.169 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.169 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.169 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.170 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.170 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.170 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.170 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.170 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.170 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.170 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.170 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.170 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T00:26:07.170 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T00:26:07.170 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.170 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.170 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.170 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.171 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.171 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.171 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.171 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.171 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.171 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.171 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.171 INFO:teuthology.orchestra.run.smithi027.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T00:26:07.171 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T00:26:07.171 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.171 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.171 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.171 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.172 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.172 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.172 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.172 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.172 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.172 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.172 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.172 INFO:teuthology.orchestra.run.smithi027.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T00:26:07.172 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T00:26:07.172 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.172 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.172 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.173 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.173 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.173 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.173 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.173 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.173 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.173 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.173 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.173 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T00:26:07.173 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T00:26:07.173 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.173 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.173 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.173 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.173 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.174 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.174 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.174 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.174 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.174 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.174 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.174 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T00:26:07.174 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T00:26:07.174 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.174 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.174 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.174 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.174 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.174 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.175 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.175 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.175 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.175 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.175 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.175 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T00:26:07.175 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T00:26:07.175 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.175 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.175 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.175 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.175 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.175 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.175 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.176 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.176 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.176 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.176 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.176 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T00:26:07.176 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T00:26:07.176 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.176 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.176 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.176 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.176 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.176 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.176 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.176 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.176 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.177 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.177 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.177 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T00:26:07.177 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T00:26:07.177 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.177 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.177 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.177 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.177 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.177 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.177 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.177 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.178 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.178 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.178 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.178 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T00:26:07.178 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T00:26:07.178 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.178 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.178 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.178 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.178 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.178 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.178 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.178 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.179 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.179 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.179 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.179 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T00:26:07.179 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T00:26:07.179 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.179 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.179 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.179 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.179 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.179 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.179 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.179 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.179 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.180 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.180 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.180 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T00:26:07.180 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T00:26:07.180 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.180 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.180 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.180 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.180 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.180 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.180 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.180 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.180 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.180 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.180 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.181 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T00:26:07.181 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T00:26:07.181 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.181 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.181 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.181 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.181 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.181 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.181 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.181 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.181 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.181 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.181 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.182 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T00:26:07.182 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T00:26:07.182 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.182 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.182 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.182 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.182 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.182 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.182 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.182 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.182 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.182 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.182 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.183 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T00:26:07.183 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T00:26:07.183 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.183 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.183 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.183 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T00:26:07.183 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.183 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.183 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.183 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.183 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.183 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.183 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.184 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T00:26:07.184 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T00:26:07.184 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.184 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.184 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.184 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:07.184 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.184 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.184 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.184 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.184 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.184 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.184 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.184 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T00:26:07.184 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T00:26:07.184 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.185 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.185 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.185 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8", 2024-04-25T00:26:07.185 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.185 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.185 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.185 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.185 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.185 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.185 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.185 INFO:teuthology.orchestra.run.smithi027.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T00:26:07.185 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T00:26:07.185 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.186 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.186 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.186 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "45", 2024-04-25T00:26:07.186 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.186 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.186 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.186 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.186 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.186 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.186 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.186 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T00:26:07.186 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T00:26:07.186 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.186 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.187 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.187 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.187 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.187 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.187 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.187 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.187 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.187 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.187 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.187 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T00:26:07.187 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T00:26:07.187 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.187 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.187 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.188 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T00:26:07.188 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.188 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.188 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.188 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.188 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.188 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.188 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.188 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T00:26:07.188 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T00:26:07.188 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.188 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.188 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.188 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.189 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.189 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.189 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.189 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.189 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.189 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.189 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.189 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T00:26:07.189 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T00:26:07.189 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.189 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.189 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.189 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.189 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.189 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.190 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.190 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.190 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.190 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.190 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.190 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T00:26:07.190 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T00:26:07.190 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.190 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.190 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.190 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T00:26:07.190 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.190 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.190 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.191 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.191 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.191 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.191 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.191 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T00:26:07.191 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T00:26:07.191 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.191 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.191 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.191 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:07.191 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.191 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.191 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.191 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.191 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.191 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.192 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.192 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T00:26:07.192 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T00:26:07.192 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.192 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.192 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.192 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T00:26:07.192 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.192 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.192 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.192 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.192 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.192 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.192 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "cross_origin_url": { 2024-04-25T00:26:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cross_origin_url", 2024-04-25T00:26:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "crt_file": { 2024-04-25T00:26:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crt_file", 2024-04-25T00:26:07.194 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.194 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.194 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.194 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.194 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.194 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.194 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.194 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.194 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.194 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.194 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.194 INFO:teuthology.orchestra.run.smithi027.stdout: "debug": { 2024-04-25T00:26:07.194 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "debug", 2024-04-25T00:26:07.194 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.195 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.195 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.195 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.195 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.195 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.195 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.195 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable/disable debug options", 2024-04-25T00:26:07.195 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.195 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.195 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.195 INFO:teuthology.orchestra.run.smithi027.stdout: "jwt_token_ttl": { 2024-04-25T00:26:07.195 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "jwt_token_ttl", 2024-04-25T00:26:07.195 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.195 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.196 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.196 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "28800", 2024-04-25T00:26:07.196 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.196 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.196 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.196 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.196 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.196 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.196 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.196 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T00:26:07.196 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T00:26:07.196 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.196 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.196 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.196 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.197 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.197 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.197 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.197 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.197 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.197 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.197 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.197 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.197 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.197 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.197 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.197 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.197 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.197 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.198 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.198 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.198 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.198 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.198 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.198 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.198 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.198 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.198 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.198 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.198 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.198 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.199 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.199 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.199 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.199 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.199 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.199 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.199 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.199 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.199 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.199 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.199 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.199 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.199 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.199 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.200 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.200 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.200 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.200 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.200 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.200 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.200 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.200 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.200 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.200 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.200 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.200 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.200 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.200 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.200 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.201 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.201 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.201 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.201 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.201 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.201 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.201 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.201 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.201 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.201 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.201 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.201 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.201 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.201 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.201 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.201 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.202 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.202 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.202 INFO:teuthology.orchestra.run.smithi027.stdout: "motd": { 2024-04-25T00:26:07.202 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "motd", 2024-04-25T00:26:07.202 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.202 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.202 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.202 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.202 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.202 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.202 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.202 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "The message of the day", 2024-04-25T00:26:07.202 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.202 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.203 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.203 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect_resolve_ip_addr": { 2024-04-25T00:26:07.203 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T00:26:07.203 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.203 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.203 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.203 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.203 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.203 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.203 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.203 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.203 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.204 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.204 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.204 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T00:26:07.204 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T00:26:07.204 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.204 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.204 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.204 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T00:26:07.204 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.204 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.204 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.204 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.204 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.204 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.204 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.205 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T00:26:07.205 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T00:26:07.205 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.205 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.205 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.205 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8080", 2024-04-25T00:26:07.205 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.205 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.205 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.205 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.205 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.206 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.206 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.206 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T00:26:07.206 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T00:26:07.206 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.206 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.206 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.206 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.206 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.206 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.206 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.206 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.206 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.206 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.206 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.207 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl_server_port": { 2024-04-25T00:26:07.207 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl_server_port", 2024-04-25T00:26:07.207 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.207 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.207 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.207 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8443", 2024-04-25T00:26:07.207 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.207 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.207 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.207 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.207 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.207 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.207 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.207 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T00:26:07.208 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T00:26:07.208 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.208 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.208 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.208 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "redirect", 2024-04-25T00:26:07.208 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.208 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.208 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.208 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.208 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect" 2024-04-25T00:26:07.208 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.208 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.208 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.208 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.208 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.209 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T00:26:07.209 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T00:26:07.209 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.209 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.209 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.209 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T00:26:07.209 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T00:26:07.209 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T00:26:07.209 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.209 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.209 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.209 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.209 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.209 INFO:teuthology.orchestra.run.smithi027.stdout: "url_prefix": { 2024-04-25T00:26:07.210 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url_prefix", 2024-04-25T00:26:07.210 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.210 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.210 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.210 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.210 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.210 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.210 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.210 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.210 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.210 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.210 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.210 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.210 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.210 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.211 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "devicehealth", 2024-04-25T00:26:07.211 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.211 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.211 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.211 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_monitoring": { 2024-04-25T00:26:07.211 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_monitoring", 2024-04-25T00:26:07.211 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.211 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.211 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.211 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.211 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.211 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.211 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.211 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "monitor device health metrics", 2024-04-25T00:26:07.211 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.212 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.212 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.212 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.212 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.212 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.212 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.212 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.212 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.212 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.212 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.212 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.212 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.212 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.212 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.213 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.213 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.213 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.213 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.213 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.213 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.213 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.213 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.213 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.213 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.213 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.213 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.213 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.213 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.214 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.214 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.214 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.214 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.214 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.214 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.214 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.214 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.214 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.214 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.214 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.214 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.214 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.214 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.214 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.215 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.215 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.215 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.215 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.215 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.215 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.215 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.215 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.215 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.215 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.215 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.215 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.215 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.215 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.215 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.216 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.216 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.216 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.216 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.216 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.216 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.216 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.216 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.216 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.216 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.216 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.216 INFO:teuthology.orchestra.run.smithi027.stdout: "mark_out_threshold": { 2024-04-25T00:26:07.216 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mark_out_threshold", 2024-04-25T00:26:07.216 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.216 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.216 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.217 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2419200", 2024-04-25T00:26:07.217 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.217 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.217 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.217 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T00:26:07.217 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.217 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.217 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.217 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": { 2024-04-25T00:26:07.217 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_name", 2024-04-25T00:26:07.217 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.217 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.217 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.217 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "device_health_metrics", 2024-04-25T00:26:07.217 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.218 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.218 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.218 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T00:26:07.218 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.218 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.218 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.218 INFO:teuthology.orchestra.run.smithi027.stdout: "retention_period": { 2024-04-25T00:26:07.218 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retention_period", 2024-04-25T00:26:07.218 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.218 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.218 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.218 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15552000", 2024-04-25T00:26:07.218 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.218 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.218 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.219 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain device health metrics", 2024-04-25T00:26:07.219 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.219 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.219 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.219 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_frequency": { 2024-04-25T00:26:07.219 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_frequency", 2024-04-25T00:26:07.219 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.219 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.219 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.219 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T00:26:07.219 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.219 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.219 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.219 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T00:26:07.219 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.220 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.220 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.220 INFO:teuthology.orchestra.run.smithi027.stdout: "self_heal": { 2024-04-25T00:26:07.220 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "self_heal", 2024-04-25T00:26:07.220 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.220 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.220 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.220 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.220 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.220 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.220 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.220 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T00:26:07.220 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.220 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.221 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.221 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.221 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T00:26:07.221 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T00:26:07.221 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.221 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.221 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.221 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T00:26:07.221 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.221 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.221 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.221 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T00:26:07.221 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.221 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.221 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.221 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.222 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_threshold": { 2024-04-25T00:26:07.222 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_threshold", 2024-04-25T00:26:07.222 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.222 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.222 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.222 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7257600", 2024-04-25T00:26:07.222 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.222 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.222 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.222 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T00:26:07.222 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.222 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.222 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.222 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.222 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.223 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "diskprediction_local", 2024-04-25T00:26:07.223 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.223 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.223 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.223 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.223 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.223 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.223 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.223 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.223 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.223 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.224 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.224 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.224 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.224 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.224 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.224 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.224 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.224 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.224 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.224 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.224 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.224 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.224 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.224 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.224 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.225 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.225 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.225 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.225 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.225 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.225 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.225 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.225 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.225 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.225 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.225 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.225 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.225 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.225 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.225 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.226 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.226 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.226 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.226 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.226 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.226 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.226 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.226 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.226 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.226 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.226 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.226 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.226 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.226 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.226 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.227 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.227 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.227 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.227 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.227 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.227 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.227 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.227 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.227 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.227 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.227 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.227 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.227 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.227 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.227 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.228 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.228 INFO:teuthology.orchestra.run.smithi027.stdout: "predict_interval": { 2024-04-25T00:26:07.228 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predict_interval", 2024-04-25T00:26:07.228 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.228 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.228 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.228 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T00:26:07.228 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.228 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.228 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.228 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.228 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.228 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.229 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.229 INFO:teuthology.orchestra.run.smithi027.stdout: "predictor_model": { 2024-04-25T00:26:07.229 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predictor_model", 2024-04-25T00:26:07.229 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.229 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.229 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.229 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "prophetstor", 2024-04-25T00:26:07.229 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.229 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.229 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.229 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.229 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.230 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.230 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.230 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T00:26:07.230 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T00:26:07.230 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.230 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.230 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.230 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T00:26:07.230 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.230 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.230 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.230 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.230 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.231 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.231 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.231 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.231 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.231 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.231 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "influx", 2024-04-25T00:26:07.231 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": false, 2024-04-25T00:26:07.231 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "influxdb python module not found", 2024-04-25T00:26:07.231 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.231 INFO:teuthology.orchestra.run.smithi027.stdout: "batch_size": { 2024-04-25T00:26:07.231 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "batch_size", 2024-04-25T00:26:07.231 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.231 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.232 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.232 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5000", 2024-04-25T00:26:07.232 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.232 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.232 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.232 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T00:26:07.232 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.232 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.232 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.232 INFO:teuthology.orchestra.run.smithi027.stdout: "database": { 2024-04-25T00:26:07.232 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "database", 2024-04-25T00:26:07.232 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.232 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.232 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.232 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "ceph", 2024-04-25T00:26:07.233 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.233 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.233 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.233 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T00:26:07.233 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.233 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.233 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.233 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.233 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": { 2024-04-25T00:26:07.233 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hostname", 2024-04-25T00:26:07.233 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.233 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.233 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.233 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.233 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.234 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.234 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.234 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server hostname", 2024-04-25T00:26:07.234 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.234 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.234 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.234 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T00:26:07.234 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T00:26:07.234 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.234 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.234 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.234 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "30", 2024-04-25T00:26:07.235 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "5", 2024-04-25T00:26:07.235 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.235 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.235 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T00:26:07.235 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.235 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.235 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.235 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.235 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.235 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.235 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.235 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.235 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.235 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.235 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.236 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.236 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.236 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.236 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.236 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.236 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.236 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.236 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.236 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.236 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.236 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.236 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.236 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.236 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.236 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.237 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.237 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.237 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.237 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.237 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.237 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.237 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.237 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.237 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.237 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.237 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.237 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.237 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.237 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.237 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.238 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.238 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.238 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.238 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.238 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.238 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.238 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.238 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.238 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.238 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.238 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.238 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.238 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.238 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.239 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.239 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.239 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.239 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.239 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.239 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.239 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.239 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.239 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.239 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.239 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.239 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.239 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.239 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.239 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.240 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.240 INFO:teuthology.orchestra.run.smithi027.stdout: "password": { 2024-04-25T00:26:07.240 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "password", 2024-04-25T00:26:07.240 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.240 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.240 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.240 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.240 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.240 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.240 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.240 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "password of InfluxDB server user", 2024-04-25T00:26:07.240 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.240 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.240 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.240 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.240 INFO:teuthology.orchestra.run.smithi027.stdout: "port": { 2024-04-25T00:26:07.241 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "port", 2024-04-25T00:26:07.241 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.241 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.241 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.241 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8086", 2024-04-25T00:26:07.241 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.241 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.241 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.241 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server port", 2024-04-25T00:26:07.241 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.241 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.241 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.242 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T00:26:07.242 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T00:26:07.242 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.242 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.242 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.242 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "false", 2024-04-25T00:26:07.242 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.242 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.242 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.242 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T00:26:07.242 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.243 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.243 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.243 INFO:teuthology.orchestra.run.smithi027.stdout: "threads": { 2024-04-25T00:26:07.243 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threads", 2024-04-25T00:26:07.243 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.243 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.243 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.243 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T00:26:07.243 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T00:26:07.243 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "32", 2024-04-25T00:26:07.243 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.243 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T00:26:07.243 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.244 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.244 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.244 INFO:teuthology.orchestra.run.smithi027.stdout: "username": { 2024-04-25T00:26:07.244 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "username", 2024-04-25T00:26:07.244 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.244 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.244 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.244 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.244 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.244 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.244 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.244 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "username of InfluxDB server user", 2024-04-25T00:26:07.244 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.245 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.245 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.245 INFO:teuthology.orchestra.run.smithi027.stdout: "verify_ssl": { 2024-04-25T00:26:07.245 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "verify_ssl", 2024-04-25T00:26:07.245 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.245 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.245 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.245 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "true", 2024-04-25T00:26:07.245 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.245 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.245 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.245 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T00:26:07.245 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.245 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.246 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.246 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.246 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.246 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.246 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.246 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "insights", 2024-04-25T00:26:07.246 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.246 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.246 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.247 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.247 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.247 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.247 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.247 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.247 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.247 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.247 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.247 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.247 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.247 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.247 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.247 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.247 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.248 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.248 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.248 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.248 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.248 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.248 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.248 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.248 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.248 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.248 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.248 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.248 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.248 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.249 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.249 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.249 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.249 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.249 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.249 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.249 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.249 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.249 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.249 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.249 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.249 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.249 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.249 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.250 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.250 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.250 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.250 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.250 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.250 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.250 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.250 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.250 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.250 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.250 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.250 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.250 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.250 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.251 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.251 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.251 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.251 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.251 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.251 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.251 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.251 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.251 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.251 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.251 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.251 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.252 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.252 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.252 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.252 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "iostat", 2024-04-25T00:26:07.252 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.252 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.252 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.252 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.252 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.253 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.253 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.253 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.253 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.253 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.253 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.253 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.253 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.253 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.253 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.253 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.253 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.253 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.253 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.254 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.254 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.254 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.254 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.254 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.254 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.254 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.254 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.254 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.254 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.254 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.254 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.254 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.255 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.255 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.255 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.255 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.255 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.255 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.255 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.255 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.255 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.255 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.255 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.255 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.255 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.256 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.256 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.256 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.256 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.256 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.256 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.256 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.256 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.256 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.256 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.256 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.256 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.256 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.256 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.256 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.256 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.257 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.257 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.257 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.257 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.257 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.257 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.257 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.257 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.257 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.257 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.257 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.257 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "k8sevents", 2024-04-25T00:26:07.257 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.257 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.258 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.258 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_event_retention_days": { 2024-04-25T00:26:07.258 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ceph_event_retention_days", 2024-04-25T00:26:07.258 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.258 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.258 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.258 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7", 2024-04-25T00:26:07.258 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.258 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.258 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.258 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T00:26:07.258 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.259 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.259 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.259 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.259 INFO:teuthology.orchestra.run.smithi027.stdout: "config_check_secs": { 2024-04-25T00:26:07.259 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_check_secs", 2024-04-25T00:26:07.259 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.259 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.259 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.259 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:07.259 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "10", 2024-04-25T00:26:07.259 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.259 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.259 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T00:26:07.259 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.260 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.260 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.260 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.260 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.260 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.260 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.260 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.260 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.260 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.260 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.260 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.260 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.260 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.260 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.261 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.261 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.261 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.261 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.261 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.261 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.261 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.261 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.261 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.261 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.261 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.261 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.261 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.261 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.261 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.262 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.262 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.262 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.262 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.262 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.262 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.262 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.262 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.262 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.262 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.262 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.263 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.263 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.263 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.263 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.263 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.263 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.263 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.263 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.263 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.263 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.263 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.263 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.263 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.263 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.263 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.264 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.264 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.264 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.264 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.264 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.264 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.264 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.264 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.264 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.264 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.264 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.264 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.264 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.264 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.265 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.265 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "localpool", 2024-04-25T00:26:07.265 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.265 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.265 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.265 INFO:teuthology.orchestra.run.smithi027.stdout: "failure_domain": { 2024-04-25T00:26:07.265 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "failure_domain", 2024-04-25T00:26:07.265 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.265 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.265 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.265 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "host", 2024-04-25T00:26:07.265 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.265 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.266 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.266 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "failure domain for any created local pool", 2024-04-25T00:26:07.266 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T00:26:07.266 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.266 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.266 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.266 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.266 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.266 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.266 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.266 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.266 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.266 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.266 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.267 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.267 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.267 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.267 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.267 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.267 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.267 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.267 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.267 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.267 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.267 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.267 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.267 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.267 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.267 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.268 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.268 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.268 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.268 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.268 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.268 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.268 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.268 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.268 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.268 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.268 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.268 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.268 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.268 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.269 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.269 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.269 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.269 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.269 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.269 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.269 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.270 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.270 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.270 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.270 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.270 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.271 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.271 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.271 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.271 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.271 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.272 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.272 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.272 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.272 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.272 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.272 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.273 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.273 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.273 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.273 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.273 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.274 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": { 2024-04-25T00:26:07.274 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_size", 2024-04-25T00:26:07.274 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.274 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.274 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.275 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.275 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.275 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.275 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.275 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default min_size for any created local pool", 2024-04-25T00:26:07.275 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T00:26:07.276 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.276 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.276 INFO:teuthology.orchestra.run.smithi027.stdout: "num_rep": { 2024-04-25T00:26:07.276 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "num_rep", 2024-04-25T00:26:07.277 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.277 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.277 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.277 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3", 2024-04-25T00:26:07.277 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.277 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.278 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.278 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default replica count for any created local pool", 2024-04-25T00:26:07.278 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.278 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.278 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.278 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.279 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": { 2024-04-25T00:26:07.279 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_num", 2024-04-25T00:26:07.279 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.279 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.279 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.280 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "128", 2024-04-25T00:26:07.280 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.280 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.280 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.281 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T00:26:07.281 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.281 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.281 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.281 INFO:teuthology.orchestra.run.smithi027.stdout: "prefix": { 2024-04-25T00:26:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prefix", 2024-04-25T00:26:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.283 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.283 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.283 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name prefix for any created local pool", 2024-04-25T00:26:07.283 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.283 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.284 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.284 INFO:teuthology.orchestra.run.smithi027.stdout: "subtree": { 2024-04-25T00:26:07.284 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subtree", 2024-04-25T00:26:07.284 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.284 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.285 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.285 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "rack", 2024-04-25T00:26:07.285 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.285 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.285 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.285 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T00:26:07.286 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T00:26:07.286 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.286 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.286 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.286 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.287 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.287 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mds_autoscaler", 2024-04-25T00:26:07.287 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.287 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.287 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.288 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.288 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.288 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.288 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.288 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.289 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.289 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.289 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.289 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.289 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.289 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.290 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.290 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.290 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.290 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.290 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.290 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.291 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.291 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.291 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.291 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.291 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.292 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.292 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.292 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.292 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.292 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.292 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.293 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.293 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.293 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.293 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.293 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.293 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.294 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.294 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.294 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.294 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.294 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.295 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.295 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.295 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.295 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.295 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.295 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.296 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.296 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.296 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.296 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.296 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.296 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.297 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.297 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.297 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.297 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.297 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.297 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.298 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.298 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.298 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.298 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.298 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.299 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.299 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.299 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.299 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.299 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.299 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.300 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.300 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.300 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.300 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.300 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.301 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirroring", 2024-04-25T00:26:07.301 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.301 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.301 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.301 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.301 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.301 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.302 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.302 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.302 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.302 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.303 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.303 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.303 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.303 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.303 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.303 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.303 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.303 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.303 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.303 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.303 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.303 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.304 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.304 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.304 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.304 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.304 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.304 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.304 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.304 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.304 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.304 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.304 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.304 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.304 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.304 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.305 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.305 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.305 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.305 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.305 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.305 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.305 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.305 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.305 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.305 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.305 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.305 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.306 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.306 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.306 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.306 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.306 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.306 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.306 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.306 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.306 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.306 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.306 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.307 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.307 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.307 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.307 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.307 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.307 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.307 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.307 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.307 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.307 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.307 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.307 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.308 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.308 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.308 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.308 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.308 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.308 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "nfs", 2024-04-25T00:26:07.308 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.308 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.308 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.308 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.308 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.309 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.309 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.309 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.309 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.309 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.309 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.309 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.309 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.309 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.309 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.309 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.309 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.310 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.310 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.310 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.310 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.310 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.310 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.310 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.310 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.310 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.310 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.310 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.311 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.311 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.311 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.311 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.311 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.311 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.311 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.311 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.311 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.311 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.312 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.312 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.312 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.312 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.312 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.312 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.312 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.312 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.312 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.312 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.312 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.312 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.313 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.313 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.313 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.313 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.313 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.313 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.313 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.313 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.313 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.313 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.313 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.313 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.314 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.314 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.314 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.314 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.314 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.314 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.314 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.314 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.314 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.314 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.314 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.314 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.315 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.315 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.315 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T00:26:07.315 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.315 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.315 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.315 INFO:teuthology.orchestra.run.smithi027.stdout: "fail_fs": { 2024-04-25T00:26:07.315 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "fail_fs", 2024-04-25T00:26:07.315 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.315 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.315 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.316 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.316 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.316 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.316 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.316 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T00:26:07.316 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.316 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.316 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.316 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.316 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.316 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.316 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.316 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.317 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.317 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.317 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.317 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.317 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.317 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.317 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.317 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.317 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.317 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.317 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.317 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.318 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.318 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.318 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.318 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.318 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.318 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.318 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.318 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.318 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.318 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.318 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.318 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.319 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.319 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.319 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.319 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.319 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.319 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.319 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.319 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.319 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.319 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.319 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.319 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.319 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.320 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.320 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.320 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.320 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.320 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.320 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.320 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.320 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.320 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.320 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.320 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.320 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.320 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.321 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.321 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.321 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.321 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.321 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.321 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.321 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.321 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.321 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.321 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.321 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.321 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.321 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.322 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.322 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.322 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.322 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator": { 2024-04-25T00:26:07.322 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T00:26:07.322 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.322 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.322 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.322 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.322 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.322 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.323 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.323 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm", 2024-04-25T00:26:07.323 INFO:teuthology.orchestra.run.smithi027.stdout: "rook", 2024-04-25T00:26:07.323 INFO:teuthology.orchestra.run.smithi027.stdout: "test_orchestrator" 2024-04-25T00:26:07.323 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.323 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Orchestrator backend", 2024-04-25T00:26:07.323 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.323 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.323 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.323 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.323 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.323 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.323 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.324 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_perf_query", 2024-04-25T00:26:07.324 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.324 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.324 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.324 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.324 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.324 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.324 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.324 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.324 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.324 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.324 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.325 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.325 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.325 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.325 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.325 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.325 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.325 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.325 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.325 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.325 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.325 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.325 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.326 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.326 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.326 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.326 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.326 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.326 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.326 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.326 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.326 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.326 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.326 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.326 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.327 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.327 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.327 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.327 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.327 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.327 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.327 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.327 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.327 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.327 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.327 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.327 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.328 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.328 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.328 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.328 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.328 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.328 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.328 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.328 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.328 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.328 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.328 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.328 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.328 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.329 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.329 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.329 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.329 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.329 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.329 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.329 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.329 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.329 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.329 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.329 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.329 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.330 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.330 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.330 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.330 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.330 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_support", 2024-04-25T00:26:07.330 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.330 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.330 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.330 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.330 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.330 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.331 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.331 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.331 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.331 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.331 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.331 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.331 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.331 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.331 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.331 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.331 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.331 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.332 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.332 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.332 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.332 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.332 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.332 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.332 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.332 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.332 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.332 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.332 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.332 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.333 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.333 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.333 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.333 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.333 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.333 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.333 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.333 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.333 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.333 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.333 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.333 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.333 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.334 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.334 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.334 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.334 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.334 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.334 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.334 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.334 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.334 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.334 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.334 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.334 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.335 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.335 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.335 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.335 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.335 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.335 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.335 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.335 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.335 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.335 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.335 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.335 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.335 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.336 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.336 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.336 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.336 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.336 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.336 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.336 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.336 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.336 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_autoscaler", 2024-04-25T00:26:07.337 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.337 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.337 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.337 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.337 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.337 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.337 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.338 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.339 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.340 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.340 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.340 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.340 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.340 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.340 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.340 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.340 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.340 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.340 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.340 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.340 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.340 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.341 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.341 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.341 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.341 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.341 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.341 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.341 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.341 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.341 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.341 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.341 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.341 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.342 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.342 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.342 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.342 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.342 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.342 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.342 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.342 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.342 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.342 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.342 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.343 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.343 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.343 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.343 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.343 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.343 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T00:26:07.343 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T00:26:07.343 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.343 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.343 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.343 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T00:26:07.344 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.344 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.344 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.344 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.344 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.344 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.344 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.344 INFO:teuthology.orchestra.run.smithi027.stdout: "threshold": { 2024-04-25T00:26:07.344 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threshold", 2024-04-25T00:26:07.344 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:07.344 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.345 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.345 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T00:26:07.345 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1.0", 2024-04-25T00:26:07.345 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.345 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.345 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "scaling threshold", 2024-04-25T00:26:07.345 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T00:26:07.345 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.345 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.345 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.345 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.346 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.346 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.346 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "progress", 2024-04-25T00:26:07.346 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.346 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.346 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.346 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_pg_recovery_event": { 2024-04-25T00:26:07.346 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_pg_recovery_event", 2024-04-25T00:26:07.346 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.346 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.346 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.346 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.347 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.347 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.347 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.347 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T00:26:07.347 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.347 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.347 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.347 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T00:26:07.347 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T00:26:07.347 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.347 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.348 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.348 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.348 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.348 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.348 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.348 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.348 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.348 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.348 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.348 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.348 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.349 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.349 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.349 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.349 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.349 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.349 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.349 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.349 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.349 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.349 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.349 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.349 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.349 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.350 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.350 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.350 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.350 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.350 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.350 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.350 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.350 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.350 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.350 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.350 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.351 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.351 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.351 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.351 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.351 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.351 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.351 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.351 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.351 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.351 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.351 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.352 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.352 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.352 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.352 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.352 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.352 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.352 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.352 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.352 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.352 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.352 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.352 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.353 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.353 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.353 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.353 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.353 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.353 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.353 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.353 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.353 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.353 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.353 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.353 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.353 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.354 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.354 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.354 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.354 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.354 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.354 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.354 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.354 INFO:teuthology.orchestra.run.smithi027.stdout: "max_completed_events": { 2024-04-25T00:26:07.354 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_completed_events", 2024-04-25T00:26:07.354 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.354 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.354 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.355 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "50", 2024-04-25T00:26:07.355 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.355 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.355 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.355 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "number of past completed events to remember", 2024-04-25T00:26:07.355 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.355 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.355 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.355 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.355 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T00:26:07.355 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T00:26:07.355 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.355 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.356 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.356 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T00:26:07.356 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.356 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.356 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.356 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long the module is going to sleep", 2024-04-25T00:26:07.356 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.356 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.356 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.356 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.356 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.356 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.356 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.356 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus", 2024-04-25T00:26:07.357 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.357 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.357 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.357 INFO:teuthology.orchestra.run.smithi027.stdout: "cache": { 2024-04-25T00:26:07.357 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cache", 2024-04-25T00:26:07.357 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.357 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.357 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.357 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.357 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.357 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.357 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.358 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.358 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.358 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.358 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.358 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.358 INFO:teuthology.orchestra.run.smithi027.stdout: "exclude_perf_counters": { 2024-04-25T00:26:07.358 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "exclude_perf_counters", 2024-04-25T00:26:07.358 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.358 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.358 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.358 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.358 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.359 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.359 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.359 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T00:26:07.359 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T00:26:07.359 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.359 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.359 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.359 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.359 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.359 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.359 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.359 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.360 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.360 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.360 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.360 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.360 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.360 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.360 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.360 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.360 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.360 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.360 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.360 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.360 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.361 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.361 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.361 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.361 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.361 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.361 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.361 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.361 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.361 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.361 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.361 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.361 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.361 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.362 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.362 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.362 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.362 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.362 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.362 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.362 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.362 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.362 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.362 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.362 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.363 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.363 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.363 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.363 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.363 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.363 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.363 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.363 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.363 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.363 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.363 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.364 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.364 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.364 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.364 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.364 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.364 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.364 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.364 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.364 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.364 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.364 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.364 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.364 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.365 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.365 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.365 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.365 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.365 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools": { 2024-04-25T00:26:07.365 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools", 2024-04-25T00:26:07.365 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.365 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.365 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.365 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.365 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.365 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.365 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.366 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.366 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.366 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.366 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.366 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.366 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T00:26:07.366 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T00:26:07.366 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.366 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.366 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.366 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "300", 2024-04-25T00:26:07.366 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.367 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.367 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.367 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.367 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.367 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.367 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.367 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.367 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_interval": { 2024-04-25T00:26:07.367 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_interval", 2024-04-25T00:26:07.367 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:07.367 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.367 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15.0", 2024-04-25T00:26:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T00:26:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T00:26:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T00:26:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T00:26:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T00:26:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T00:26:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "9283", 2024-04-25T00:26:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T00:26:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "stale_cache_strategy": { 2024-04-25T00:26:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stale_cache_strategy", 2024-04-25T00:26:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "log", 2024-04-25T00:26:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T00:26:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T00:26:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "default", 2024-04-25T00:26:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "default", 2024-04-25T00:26:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "error" 2024-04-25T00:26:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T00:26:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T00:26:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T00:26:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T00:26:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T00:26:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_support", 2024-04-25T00:26:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.375 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.375 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.375 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.375 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.375 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.375 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.375 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.375 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.375 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.375 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.375 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.375 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.375 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.375 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.376 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.376 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.376 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.376 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.376 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.376 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.376 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.376 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.376 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.376 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.376 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.376 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.377 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.377 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.377 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.377 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.377 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.377 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.377 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.377 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.377 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.377 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.377 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.377 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.377 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.378 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.378 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.378 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.378 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.378 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.378 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.378 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.378 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.378 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.378 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.378 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.379 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.379 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.379 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.379 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.379 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.379 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.379 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.379 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.379 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.379 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.379 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.379 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.379 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.380 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.380 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.380 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.380 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.380 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.380 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.380 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.380 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.380 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_snap_create": { 2024-04-25T00:26:07.380 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_snap_create", 2024-04-25T00:26:07.380 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.380 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.381 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.381 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:07.381 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.381 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.381 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.381 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.381 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.381 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.381 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.381 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.381 INFO:teuthology.orchestra.run.smithi027.stdout: "mirror_snapshot_schedule": { 2024-04-25T00:26:07.381 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T00:26:07.381 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.382 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.382 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.382 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.382 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.382 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.382 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.382 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.382 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.382 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.382 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.382 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.382 INFO:teuthology.orchestra.run.smithi027.stdout: "trash_purge_schedule": { 2024-04-25T00:26:07.382 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "trash_purge_schedule", 2024-04-25T00:26:07.383 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.383 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.383 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.383 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.383 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.383 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.383 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.383 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.383 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.383 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.383 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.383 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.384 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.384 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.384 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.384 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "restful", 2024-04-25T00:26:07.384 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.384 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.384 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.384 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_auth": { 2024-04-25T00:26:07.384 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_auth", 2024-04-25T00:26:07.384 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.384 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.384 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.384 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.385 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.385 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.385 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.385 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.385 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.385 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.385 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.385 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.385 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T00:26:07.385 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T00:26:07.385 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.385 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.386 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.386 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.386 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.386 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.386 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.386 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.386 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.386 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.386 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.386 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.387 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.387 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.387 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.387 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.387 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.387 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.387 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.387 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.387 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.387 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.387 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.387 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.387 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.388 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.388 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.388 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.388 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.388 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.388 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.388 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.388 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.388 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.388 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.388 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.388 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.388 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.389 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.389 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.389 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.389 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.389 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.389 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.389 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.389 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.389 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.389 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.389 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.389 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.390 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.390 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.390 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.390 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.390 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.390 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.390 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.390 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.390 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.390 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.390 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.390 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.391 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.391 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.391 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.391 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.391 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.391 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.391 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.391 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.391 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.391 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.391 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.391 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.391 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.392 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.392 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.392 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.392 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.392 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.392 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T00:26:07.392 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T00:26:07.392 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.392 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.392 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.392 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.392 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.393 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.393 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.393 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.393 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.393 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.393 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.393 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T00:26:07.393 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T00:26:07.393 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.393 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.393 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.394 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.394 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.394 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.394 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.394 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.394 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.394 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.394 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.394 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.394 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.394 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.395 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.395 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rgw", 2024-04-25T00:26:07.395 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.395 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.395 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.395 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.395 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.395 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.395 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.395 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.395 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.395 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.396 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.396 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.396 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.396 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.396 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.396 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.396 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.396 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.396 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.396 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.396 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.396 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.397 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.397 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.397 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.397 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.397 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.397 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.397 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.397 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.397 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.397 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.397 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.397 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.397 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.398 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.398 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.398 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.398 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.398 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.398 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.398 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.398 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.398 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.398 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.398 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.399 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.399 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.399 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.399 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.399 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.399 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.399 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.399 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.399 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.399 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.399 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.399 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.400 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.400 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.400 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.400 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.400 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.400 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.400 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.400 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.400 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.400 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.400 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.400 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.400 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.401 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.401 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.401 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rook", 2024-04-25T00:26:07.401 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.401 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.401 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.401 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.401 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.401 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.401 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.401 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.401 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.401 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.402 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.402 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.402 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.402 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.402 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.402 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.402 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.402 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.402 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.402 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.402 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.402 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.403 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.403 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.403 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.403 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.403 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.403 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.403 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.403 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.403 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.403 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.403 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.404 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.404 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.404 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.404 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.404 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.404 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.404 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.404 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.404 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.404 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.404 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.405 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.405 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.405 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.405 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.405 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.405 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.405 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.405 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.405 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.405 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.405 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.405 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.406 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.406 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.406 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.406 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.406 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.406 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.406 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.406 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.406 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.406 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.406 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.406 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.407 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.407 INFO:teuthology.orchestra.run.smithi027.stdout: "storage_class": { 2024-04-25T00:26:07.407 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "storage_class", 2024-04-25T00:26:07.407 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.407 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.407 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.407 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "local", 2024-04-25T00:26:07.407 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.407 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.407 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.407 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T00:26:07.407 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.407 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.408 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.408 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.408 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.408 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.408 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.408 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "selftest", 2024-04-25T00:26:07.408 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.408 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.408 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.408 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.408 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.408 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.408 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.409 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.409 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.409 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.409 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.409 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.409 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.409 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.409 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.409 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.409 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.409 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.409 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.410 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.410 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.410 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.410 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.410 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.410 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.410 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.410 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.410 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.410 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.410 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.410 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.411 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.411 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.411 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.411 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.411 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.411 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.411 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.411 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.411 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.411 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.411 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.411 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.411 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.412 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.412 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.412 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.412 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.412 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.412 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.412 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.412 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.412 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.412 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.412 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.412 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.412 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.413 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.413 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.413 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.413 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.413 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.413 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.413 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.413 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.413 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.413 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.413 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.413 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.414 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.414 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.414 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.414 INFO:teuthology.orchestra.run.smithi027.stdout: "roption1": { 2024-04-25T00:26:07.414 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption1", 2024-04-25T00:26:07.414 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.414 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.414 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.414 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.414 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.414 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.414 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.415 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.415 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.415 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.415 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.415 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.415 INFO:teuthology.orchestra.run.smithi027.stdout: "roption2": { 2024-04-25T00:26:07.415 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption2", 2024-04-25T00:26:07.415 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.415 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.415 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.415 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "xyz", 2024-04-25T00:26:07.415 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.416 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.416 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.416 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.416 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.416 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.416 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.416 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.416 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption1": { 2024-04-25T00:26:07.416 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption1", 2024-04-25T00:26:07.416 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.416 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.416 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.416 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.417 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.417 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.417 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.417 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.417 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.417 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.417 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.417 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.417 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption2": { 2024-04-25T00:26:07.417 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption2", 2024-04-25T00:26:07.417 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.417 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.417 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.418 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.418 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.418 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.418 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.418 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.418 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.418 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.418 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.418 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.418 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption3": { 2024-04-25T00:26:07.418 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption3", 2024-04-25T00:26:07.418 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:07.419 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.419 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.419 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.419 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.419 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.419 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.419 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.419 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.419 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.419 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.420 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption4": { 2024-04-25T00:26:07.420 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption4", 2024-04-25T00:26:07.420 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.420 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.420 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.420 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.420 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.420 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.420 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.420 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.420 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.420 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.421 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.421 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.421 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption5": { 2024-04-25T00:26:07.421 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption5", 2024-04-25T00:26:07.421 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.421 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.421 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.421 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.421 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.421 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.421 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.421 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.421 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.422 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.422 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.422 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.422 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption6": { 2024-04-25T00:26:07.422 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption6", 2024-04-25T00:26:07.422 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.422 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.422 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.422 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.422 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.422 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.422 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.423 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.423 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.423 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.423 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.423 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption7": { 2024-04-25T00:26:07.423 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption7", 2024-04-25T00:26:07.423 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.423 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.423 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.423 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.423 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T00:26:07.424 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "42", 2024-04-25T00:26:07.424 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.424 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.424 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.424 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.424 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.424 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.424 INFO:teuthology.orchestra.run.smithi027.stdout: "testkey": { 2024-04-25T00:26:07.424 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testkey", 2024-04-25T00:26:07.424 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.424 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.424 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.425 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.425 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.425 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.425 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.425 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.425 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.425 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.425 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.425 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.425 INFO:teuthology.orchestra.run.smithi027.stdout: "testlkey": { 2024-04-25T00:26:07.425 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testlkey", 2024-04-25T00:26:07.425 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.426 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.426 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.426 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.426 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.426 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.426 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.426 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.426 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.426 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.426 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.426 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.427 INFO:teuthology.orchestra.run.smithi027.stdout: "testnewline": { 2024-04-25T00:26:07.427 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testnewline", 2024-04-25T00:26:07.427 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.427 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.427 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.427 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.427 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.427 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.427 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.427 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.427 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.427 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.428 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.428 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.428 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.428 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.428 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-04-25T00:26:07.428 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.428 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.428 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.428 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_m_granularity": { 2024-04-25T00:26:07.428 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_m_granularity", 2024-04-25T00:26:07.428 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.429 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.429 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.429 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.429 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.429 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.429 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.429 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T00:26:07.429 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.429 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.429 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.429 INFO:teuthology.orchestra.run.smithi027.stdout: "dump_on_update": { 2024-04-25T00:26:07.430 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dump_on_update", 2024-04-25T00:26:07.430 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.430 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.430 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.430 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.430 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.430 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.430 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.430 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "dump database to debug log on update", 2024-04-25T00:26:07.430 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.430 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.430 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.431 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.431 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.431 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.431 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.431 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.431 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.431 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.431 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.431 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.431 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.431 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.431 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.432 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.432 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.432 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.432 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.432 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.432 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.432 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.432 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.432 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.432 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.432 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.433 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.433 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.433 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.433 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.433 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.433 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.433 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.433 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.433 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.433 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.433 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.434 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.434 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.434 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.434 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.434 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.434 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.434 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.434 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.434 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.434 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.434 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.434 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.435 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.435 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.435 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.435 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.435 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.435 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.435 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.435 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.435 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.435 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.435 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.436 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.436 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.436 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.436 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.436 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.436 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.436 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.436 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.436 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.436 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.436 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.437 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.437 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stats", 2024-04-25T00:26:07.437 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.437 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.437 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.437 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.437 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.437 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.437 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.437 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.437 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.438 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.438 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.438 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.438 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.438 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.438 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.438 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.438 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.438 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.438 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.438 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.438 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.439 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.439 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.439 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.439 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.439 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.439 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.439 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.439 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.439 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.439 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.439 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.440 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.440 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.440 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.440 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.440 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.440 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.440 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.440 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.440 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.440 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.440 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.441 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.441 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.441 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.441 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.441 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.441 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.441 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.441 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.441 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.441 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.441 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.441 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.441 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.442 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.442 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.442 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.442 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.442 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.442 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.442 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.442 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.442 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.442 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.442 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.443 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.443 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.443 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.443 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.443 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "status", 2024-04-25T00:26:07.443 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.443 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.443 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.443 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.443 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.443 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.444 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.444 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.444 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.444 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.444 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.444 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.444 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.444 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.444 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.444 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.444 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.445 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.445 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.445 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.445 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.445 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.445 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.445 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.445 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.445 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.445 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.445 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.445 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.445 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.446 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.446 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.446 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.446 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.446 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.446 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.446 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.446 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.446 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.446 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.446 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.447 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.447 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.447 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.447 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.447 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.447 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.447 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.447 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.447 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.447 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.447 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.447 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.447 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.448 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.448 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.448 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.448 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.448 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.448 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.448 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.448 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.448 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.448 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.448 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.448 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.449 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.449 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.449 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.449 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.449 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.449 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.449 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telegraf", 2024-04-25T00:26:07.449 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.449 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.449 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.449 INFO:teuthology.orchestra.run.smithi027.stdout: "address": { 2024-04-25T00:26:07.450 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "address", 2024-04-25T00:26:07.450 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.450 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.450 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.450 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T00:26:07.450 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.450 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.450 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.450 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.450 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.450 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.450 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.451 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.451 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T00:26:07.451 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T00:26:07.451 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.451 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.451 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.451 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15", 2024-04-25T00:26:07.451 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.451 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.451 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.451 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.452 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.452 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.452 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.452 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.452 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.452 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.452 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.452 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.452 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.452 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.453 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.453 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.453 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.453 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.453 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.453 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.453 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.453 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.453 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.453 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.453 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.453 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.454 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.454 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.454 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.454 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.454 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.454 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.454 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.454 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.454 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.454 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.455 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.455 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.455 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.455 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.455 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.455 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.455 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.455 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.455 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.455 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.455 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.456 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.456 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.456 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.456 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.456 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.456 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.456 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.456 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.456 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.456 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.456 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.457 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.457 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.457 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.457 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.457 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.457 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.457 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.457 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.457 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.457 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.457 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.458 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.458 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.458 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.458 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.458 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.458 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.458 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.458 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.458 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telemetry", 2024-04-25T00:26:07.458 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.458 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.459 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.459 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_basic": { 2024-04-25T00:26:07.459 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_basic", 2024-04-25T00:26:07.459 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.459 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.459 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.459 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.459 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.459 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.459 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.459 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T00:26:07.459 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.460 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.460 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.460 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_crash": { 2024-04-25T00:26:07.460 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_crash", 2024-04-25T00:26:07.460 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.460 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.460 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.460 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.460 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.460 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.460 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.461 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T00:26:07.461 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.461 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.461 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.461 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.461 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_device": { 2024-04-25T00:26:07.461 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_device", 2024-04-25T00:26:07.461 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.461 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.461 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.461 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.461 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.462 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.462 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.462 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T00:26:07.462 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.462 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.462 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.462 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_ident": { 2024-04-25T00:26:07.462 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_ident", 2024-04-25T00:26:07.462 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.462 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.462 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.463 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.463 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.463 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.463 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.463 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T00:26:07.463 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.463 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.463 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.463 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_perf": { 2024-04-25T00:26:07.463 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_perf", 2024-04-25T00:26:07.463 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.464 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.464 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.464 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.464 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.464 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.464 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.464 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T00:26:07.464 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.464 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.464 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.464 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.465 INFO:teuthology.orchestra.run.smithi027.stdout: "contact": { 2024-04-25T00:26:07.465 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "contact", 2024-04-25T00:26:07.465 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.465 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.465 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.465 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.465 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.465 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.465 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.465 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.465 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.466 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.466 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.466 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.466 INFO:teuthology.orchestra.run.smithi027.stdout: "description": { 2024-04-25T00:26:07.466 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "description", 2024-04-25T00:26:07.466 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.466 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.466 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.466 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.466 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.466 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.466 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.467 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.467 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.467 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.467 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.467 INFO:teuthology.orchestra.run.smithi027.stdout: "device_url": { 2024-04-25T00:26:07.467 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_url", 2024-04-25T00:26:07.467 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.467 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.467 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.467 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T00:26:07.467 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.467 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.468 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.468 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.468 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.468 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.468 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.468 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.468 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T00:26:07.468 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T00:26:07.468 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.468 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.468 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.468 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.469 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.469 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.469 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.469 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.469 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.469 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.469 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.469 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T00:26:07.469 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T00:26:07.469 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.469 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.469 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.470 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "24", 2024-04-25T00:26:07.470 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "8", 2024-04-25T00:26:07.470 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.470 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.470 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.470 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.470 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.470 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.470 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.470 INFO:teuthology.orchestra.run.smithi027.stdout: "last_opt_revision": { 2024-04-25T00:26:07.470 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "last_opt_revision", 2024-04-25T00:26:07.471 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.471 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.471 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.471 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1", 2024-04-25T00:26:07.471 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.471 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.471 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.471 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.471 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.471 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.471 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.472 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard": { 2024-04-25T00:26:07.472 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard", 2024-04-25T00:26:07.472 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.472 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.472 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.472 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.472 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.472 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.472 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.472 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.472 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.472 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.473 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.473 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard_description": { 2024-04-25T00:26:07.473 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard_description", 2024-04-25T00:26:07.473 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.473 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.473 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.473 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.473 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.473 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.473 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.473 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.474 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.474 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.474 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.474 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.474 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.474 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.474 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.474 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.474 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.474 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.474 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.475 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.475 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.475 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.475 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.475 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.475 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.475 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.475 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.475 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.475 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.475 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.475 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.476 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.476 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.476 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.476 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.476 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.476 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.476 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.476 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.476 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.476 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.476 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.476 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.477 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.477 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.477 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.477 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.477 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.477 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.477 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.477 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.477 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.477 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.477 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.478 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.478 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.478 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.478 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.478 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.478 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.478 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.478 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.478 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.478 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.478 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.479 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.479 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.479 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.479 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.479 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.479 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.479 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.479 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.479 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.479 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.479 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.479 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.480 INFO:teuthology.orchestra.run.smithi027.stdout: "organization": { 2024-04-25T00:26:07.480 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "organization", 2024-04-25T00:26:07.480 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.480 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.480 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.480 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.480 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.480 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.480 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.480 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.480 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.480 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.481 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.481 INFO:teuthology.orchestra.run.smithi027.stdout: "proxy": { 2024-04-25T00:26:07.481 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "proxy", 2024-04-25T00:26:07.481 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.481 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.481 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.481 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.481 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.481 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.481 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.481 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.482 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.482 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.482 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.482 INFO:teuthology.orchestra.run.smithi027.stdout: "url": { 2024-04-25T00:26:07.482 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url", 2024-04-25T00:26:07.482 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.482 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.482 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.482 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T00:26:07.482 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.482 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.483 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.483 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.483 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.483 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.483 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.483 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.483 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.483 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "test_orchestrator", 2024-04-25T00:26:07.483 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.483 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.484 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.484 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.484 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.484 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.484 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.484 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.484 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.484 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.484 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.484 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.484 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.485 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.485 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.485 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.485 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.485 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.485 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.485 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.485 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.485 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.485 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.485 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.486 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.486 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.486 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.486 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.486 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.486 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.486 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.486 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.486 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.486 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.486 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.486 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.487 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.487 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.487 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.487 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.487 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.487 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.487 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.487 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.487 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.487 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.487 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.488 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.488 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.488 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.488 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.488 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.488 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.488 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.488 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.488 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.488 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.488 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.488 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.489 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.489 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.489 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.489 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.489 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.489 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.489 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.489 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.489 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.489 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.489 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.489 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.490 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "volumes", 2024-04-25T00:26:07.490 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.490 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.490 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.490 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.490 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.490 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.490 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.490 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.490 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.490 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.491 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.491 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.491 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.491 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.491 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.491 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.491 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.491 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.491 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.491 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.491 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.491 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.491 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.492 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.492 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.492 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.492 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.492 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.492 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.492 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.492 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.492 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.492 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.492 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.492 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.492 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.493 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.493 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.493 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.493 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.493 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.493 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.493 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.493 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.493 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.493 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.493 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.494 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.494 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.494 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.494 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.494 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.494 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.494 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.494 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.494 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.494 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.494 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.494 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.495 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.495 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.495 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.495 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.495 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.495 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.495 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.495 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.495 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.495 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.495 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_clones": { 2024-04-25T00:26:07.496 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_clones", 2024-04-25T00:26:07.496 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.496 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.496 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.496 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4", 2024-04-25T00:26:07.496 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.496 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.496 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.496 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T00:26:07.496 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.496 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.496 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.497 INFO:teuthology.orchestra.run.smithi027.stdout: "periodic_async_work": { 2024-04-25T00:26:07.497 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "periodic_async_work", 2024-04-25T00:26:07.497 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.497 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.497 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.497 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.497 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.497 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.497 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.497 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Periodically check for async work", 2024-04-25T00:26:07.497 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.498 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.498 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.498 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_delay": { 2024-04-25T00:26:07.498 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_delay", 2024-04-25T00:26:07.498 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.498 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.498 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.498 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T00:26:07.498 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.498 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.498 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.499 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T00:26:07.499 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.499 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.499 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.499 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_no_wait": { 2024-04-25T00:26:07.499 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T00:26:07.499 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.499 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.499 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.499 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.499 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.500 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.500 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.500 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T00:26:07.500 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.500 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.500 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.500 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.500 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.500 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix", 2024-04-25T00:26:07.500 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.501 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.501 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.501 INFO:teuthology.orchestra.run.smithi027.stdout: "discovery_interval": { 2024-04-25T00:26:07.501 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "discovery_interval", 2024-04-25T00:26:07.501 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T00:26:07.501 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.501 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.501 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "100", 2024-04-25T00:26:07.501 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.501 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.501 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.501 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.502 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.502 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.502 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.502 INFO:teuthology.orchestra.run.smithi027.stdout: "identifier": { 2024-04-25T00:26:07.502 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "identifier", 2024-04-25T00:26:07.502 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.502 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.502 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.502 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.503 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.503 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.503 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.503 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.503 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.503 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.503 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.503 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T00:26:07.503 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T00:26:07.503 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.503 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.503 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.504 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T00:26:07.504 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.504 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.504 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.504 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.504 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.504 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.504 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.504 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.504 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.504 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.505 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.505 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.505 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.505 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.505 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.505 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.505 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.505 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.505 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.505 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.505 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.505 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.506 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.506 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.506 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.506 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.506 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.506 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.506 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.506 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.506 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.506 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.506 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.507 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.507 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.507 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.507 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.507 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.507 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.507 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.507 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.507 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.507 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.507 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.507 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.508 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.508 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.508 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.508 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.508 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.508 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.508 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.508 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.508 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.508 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.508 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.508 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.509 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.509 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.509 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.509 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.509 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.509 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.509 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.509 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.509 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.509 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.509 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.509 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.510 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.510 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.510 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.510 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.510 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_host": { 2024-04-25T00:26:07.510 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_host", 2024-04-25T00:26:07.510 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.510 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.510 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.511 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.511 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.511 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.511 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.511 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.511 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.511 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.511 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.511 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_port": { 2024-04-25T00:26:07.511 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_port", 2024-04-25T00:26:07.512 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.512 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.512 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.512 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10051", 2024-04-25T00:26:07.512 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.512 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.512 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.512 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.512 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.512 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.512 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.513 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_sender": { 2024-04-25T00:26:07.513 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_sender", 2024-04-25T00:26:07.513 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.513 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.513 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.513 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T00:26:07.513 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.513 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.513 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.513 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.513 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.513 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.514 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.514 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.514 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.514 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.514 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T00:26:07.514 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.514 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.514 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-04-25T00:26:07.514 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm", 2024-04-25T00:26:07.514 INFO:teuthology.orchestra.run.smithi027.stdout: "dashboard", 2024-04-25T00:26:07.514 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-04-25T00:26:07.514 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-04-25T00:26:07.515 INFO:teuthology.orchestra.run.smithi027.stdout: "restful", 2024-04-25T00:26:07.515 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_schedule" 2024-04-25T00:26:07.515 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.515 INFO:teuthology.orchestra.run.smithi027.stdout: "available_modules": [ 2024-04-25T00:26:07.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.515 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "alerts", 2024-04-25T00:26:07.515 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.515 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.515 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.515 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T00:26:07.515 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T00:26:07.515 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.515 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.515 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.516 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T00:26:07.516 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.516 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.516 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.516 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T00:26:07.516 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.516 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.516 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.516 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.516 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.516 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.516 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.517 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.517 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.517 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.517 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.517 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.517 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.517 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.517 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.517 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.517 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.517 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.517 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.517 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.517 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.518 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.518 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.518 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.518 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.518 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.518 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.518 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.518 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.518 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.518 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.518 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.519 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.519 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.519 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.519 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.519 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.519 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.519 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.519 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.519 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.519 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.519 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.520 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.520 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.520 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.520 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.520 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.520 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.520 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.520 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.520 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.520 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.520 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.521 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.521 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.521 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.521 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.521 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.521 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.521 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.521 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.521 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.521 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.521 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.521 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.522 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.522 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.522 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.522 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_destination": { 2024-04-25T00:26:07.522 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_destination", 2024-04-25T00:26:07.522 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.522 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.522 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.522 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.522 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.523 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.523 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.523 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email address to send alerts to", 2024-04-25T00:26:07.523 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.523 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.523 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.523 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_from_name": { 2024-04-25T00:26:07.523 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_from_name", 2024-04-25T00:26:07.523 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.523 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.523 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.524 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "Ceph", 2024-04-25T00:26:07.524 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.524 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.524 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.524 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email From: name", 2024-04-25T00:26:07.524 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.524 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.524 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.524 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_host": { 2024-04-25T00:26:07.524 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_host", 2024-04-25T00:26:07.524 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.525 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.525 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.525 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.525 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.525 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.525 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.525 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP server", 2024-04-25T00:26:07.525 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.525 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.525 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.525 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_password": { 2024-04-25T00:26:07.526 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_password", 2024-04-25T00:26:07.526 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.526 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.526 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.526 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.526 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.526 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.526 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.526 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Password to authenticate with", 2024-04-25T00:26:07.526 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.526 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.526 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.527 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_port": { 2024-04-25T00:26:07.527 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_port", 2024-04-25T00:26:07.527 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.527 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.527 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.527 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "465", 2024-04-25T00:26:07.527 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.527 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.527 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.527 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP port", 2024-04-25T00:26:07.527 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.528 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.528 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.528 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_sender": { 2024-04-25T00:26:07.528 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_sender", 2024-04-25T00:26:07.528 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.528 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.528 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.528 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.528 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.528 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.528 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.529 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP envelope sender", 2024-04-25T00:26:07.529 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.529 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.529 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.529 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_ssl": { 2024-04-25T00:26:07.529 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_ssl", 2024-04-25T00:26:07.529 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.529 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.529 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.529 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.529 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.529 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.530 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.530 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T00:26:07.530 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.530 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.530 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.530 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_user": { 2024-04-25T00:26:07.530 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_user", 2024-04-25T00:26:07.530 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.530 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.530 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.530 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.531 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.531 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.531 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.531 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "User to authenticate as", 2024-04-25T00:26:07.531 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.531 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.531 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.531 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.531 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.531 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "balancer", 2024-04-25T00:26:07.531 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.532 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.532 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.532 INFO:teuthology.orchestra.run.smithi027.stdout: "active": { 2024-04-25T00:26:07.532 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "active", 2024-04-25T00:26:07.532 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.532 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.532 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.532 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.532 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.532 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.532 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.532 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T00:26:07.533 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.533 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.533 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.533 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_time": { 2024-04-25T00:26:07.533 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_time", 2024-04-25T00:26:07.533 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.533 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.533 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.533 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0000", 2024-04-25T00:26:07.533 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.533 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.534 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.534 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T00:26:07.534 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T00:26:07.534 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.534 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.534 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_weekday": { 2024-04-25T00:26:07.534 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_weekday", 2024-04-25T00:26:07.534 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T00:26:07.534 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.534 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.534 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T00:26:07.535 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T00:26:07.535 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T00:26:07.535 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.535 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T00:26:07.535 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T00:26:07.535 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.535 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.535 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_max_iterations": { 2024-04-25T00:26:07.535 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_max_iterations", 2024-04-25T00:26:07.535 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T00:26:07.536 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.536 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.536 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "25", 2024-04-25T00:26:07.536 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T00:26:07.536 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "250", 2024-04-25T00:26:07.536 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.536 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T00:26:07.536 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.536 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.536 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.536 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_metrics": { 2024-04-25T00:26:07.537 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_metrics", 2024-04-25T00:26:07.537 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.537 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.537 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.537 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T00:26:07.537 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.537 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.537 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.537 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T00:26:07.537 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T00:26:07.537 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.537 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.538 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_step": { 2024-04-25T00:26:07.538 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_step", 2024-04-25T00:26:07.538 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:07.538 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.538 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.538 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.5", 2024-04-25T00:26:07.538 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0.001", 2024-04-25T00:26:07.538 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "0.999", 2024-04-25T00:26:07.538 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.538 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "aggressiveness of optimization", 2024-04-25T00:26:07.538 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T00:26:07.539 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.539 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.539 INFO:teuthology.orchestra.run.smithi027.stdout: "end_time": { 2024-04-25T00:26:07.539 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_time", 2024-04-25T00:26:07.539 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.539 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.539 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.539 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2359", 2024-04-25T00:26:07.539 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.539 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.540 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.540 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T00:26:07.540 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T00:26:07.540 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.540 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.540 INFO:teuthology.orchestra.run.smithi027.stdout: "end_weekday": { 2024-04-25T00:26:07.540 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_weekday", 2024-04-25T00:26:07.540 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T00:26:07.540 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.540 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.540 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T00:26:07.540 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T00:26:07.541 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T00:26:07.541 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.541 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T00:26:07.541 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T00:26:07.541 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.541 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.541 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.541 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.541 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.541 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.541 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.542 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.542 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.542 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.542 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.542 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.542 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.542 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.542 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.542 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.542 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.542 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.543 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.543 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.543 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.543 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.543 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.543 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.543 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.543 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.543 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.543 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.543 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.543 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.544 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.544 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.544 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.544 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.544 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.544 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.544 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.544 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.544 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.544 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.545 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.545 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.545 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.545 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.545 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.545 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.545 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.545 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.545 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.545 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.545 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.545 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.545 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.546 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.546 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.546 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.546 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.546 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.546 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.546 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.546 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.546 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.546 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.547 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.547 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.547 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.547 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.547 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.547 INFO:teuthology.orchestra.run.smithi027.stdout: "min_score": { 2024-04-25T00:26:07.547 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_score", 2024-04-25T00:26:07.547 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:07.547 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.547 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.547 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T00:26:07.548 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.548 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.548 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.548 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T00:26:07.548 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.548 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.548 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.548 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T00:26:07.548 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T00:26:07.548 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.548 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.549 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.549 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "upmap", 2024-04-25T00:26:07.549 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.549 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.549 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.549 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-compat", 2024-04-25T00:26:07.549 INFO:teuthology.orchestra.run.smithi027.stdout: "none", 2024-04-25T00:26:07.549 INFO:teuthology.orchestra.run.smithi027.stdout: "read", 2024-04-25T00:26:07.549 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap", 2024-04-25T00:26:07.549 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap-read" 2024-04-25T00:26:07.549 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.549 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Balancer mode", 2024-04-25T00:26:07.550 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.550 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.550 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.550 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_ids": { 2024-04-25T00:26:07.550 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_ids", 2024-04-25T00:26:07.550 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.550 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.550 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.550 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.550 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.550 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.551 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.551 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T00:26:07.551 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.551 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.551 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.551 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T00:26:07.551 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T00:26:07.551 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.551 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.551 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.551 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T00:26:07.552 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.552 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.552 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.552 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T00:26:07.552 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.552 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.552 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.552 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_deviation": { 2024-04-25T00:26:07.552 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_deviation", 2024-04-25T00:26:07.552 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.552 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.553 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.553 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T00:26:07.553 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T00:26:07.553 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.553 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.553 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T00:26:07.553 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T00:26:07.553 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.553 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.553 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_optimizations": { 2024-04-25T00:26:07.553 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_optimizations", 2024-04-25T00:26:07.554 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T00:26:07.554 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.554 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.554 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:07.554 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.554 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.554 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.554 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T00:26:07.554 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.554 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.554 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.555 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.555 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.555 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm", 2024-04-25T00:26:07.555 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.555 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.555 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.555 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_down_multiplier": { 2024-04-25T00:26:07.555 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_down_multiplier", 2024-04-25T00:26:07.555 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:07.555 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.556 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.556 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T00:26:07.556 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.556 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.556 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.556 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T00:26:07.556 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.556 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.556 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.556 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_refresh_rate": { 2024-04-25T00:26:07.557 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_refresh_rate", 2024-04-25T00:26:07.557 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.557 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.557 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.557 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "20", 2024-04-25T00:26:07.557 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.557 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.557 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.557 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T00:26:07.557 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.557 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.557 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.558 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_starting_port": { 2024-04-25T00:26:07.558 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_starting_port", 2024-04-25T00:26:07.558 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.558 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.558 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.558 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4721", 2024-04-25T00:26:07.558 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.558 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.558 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.558 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T00:26:07.558 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.559 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.559 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.559 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_ptrace": { 2024-04-25T00:26:07.559 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_ptrace", 2024-04-25T00:26:07.559 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.559 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.559 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.559 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.559 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.559 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.559 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.560 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T00:26:07.560 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T00:26:07.560 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.560 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.560 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_interval": { 2024-04-25T00:26:07.560 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_interval", 2024-04-25T00:26:07.560 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.560 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.560 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.560 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T00:26:07.560 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.560 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.561 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.561 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T00:26:07.561 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.561 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.561 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.561 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_memory_target_ratio": { 2024-04-25T00:26:07.561 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T00:26:07.561 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:07.561 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.561 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.561 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.7", 2024-04-25T00:26:07.561 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.562 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.562 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.562 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T00:26:07.562 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.562 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.562 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.562 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm_log_destination": { 2024-04-25T00:26:07.562 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm_log_destination", 2024-04-25T00:26:07.562 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.562 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.562 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.563 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.563 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.563 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.563 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.563 INFO:teuthology.orchestra.run.smithi027.stdout: "file", 2024-04-25T00:26:07.563 INFO:teuthology.orchestra.run.smithi027.stdout: "file,syslog", 2024-04-25T00:26:07.563 INFO:teuthology.orchestra.run.smithi027.stdout: "syslog" 2024-04-25T00:26:07.563 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.563 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T00:26:07.563 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.563 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.564 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.564 INFO:teuthology.orchestra.run.smithi027.stdout: "cgroups_split": { 2024-04-25T00:26:07.564 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cgroups_split", 2024-04-25T00:26:07.564 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.564 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.564 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.564 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.564 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.564 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.564 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.564 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T00:26:07.564 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.565 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.565 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.565 INFO:teuthology.orchestra.run.smithi027.stdout: "config_checks_enabled": { 2024-04-25T00:26:07.565 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_checks_enabled", 2024-04-25T00:26:07.565 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.565 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.565 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.565 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.565 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.565 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.565 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.566 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T00:26:07.566 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.566 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.566 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.566 INFO:teuthology.orchestra.run.smithi027.stdout: "config_dashboard": { 2024-04-25T00:26:07.566 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_dashboard", 2024-04-25T00:26:07.566 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.566 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.566 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.566 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.567 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.567 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.567 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.567 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T00:26:07.567 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.567 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.567 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.567 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_alertmanager": { 2024-04-25T00:26:07.567 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_alertmanager", 2024-04-25T00:26:07.567 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.567 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.568 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.568 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T00:26:07.568 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.568 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.568 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.568 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T00:26:07.568 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.568 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.568 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.568 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_base": { 2024-04-25T00:26:07.568 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_base", 2024-04-25T00:26:07.569 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.569 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.569 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.569 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T00:26:07.569 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.569 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.569 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.569 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Container image name, without the tag", 2024-04-25T00:26:07.569 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.569 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.569 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.570 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_elasticsearch": { 2024-04-25T00:26:07.570 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_elasticsearch", 2024-04-25T00:26:07.570 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.570 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.570 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.570 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T00:26:07.570 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.570 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.570 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.570 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "elasticsearch container image", 2024-04-25T00:26:07.570 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.571 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.571 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.571 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_grafana": { 2024-04-25T00:26:07.571 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_grafana", 2024-04-25T00:26:07.571 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.571 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.571 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.571 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T00:26:07.571 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.571 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.572 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.572 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T00:26:07.572 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.572 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.572 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.572 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_haproxy": { 2024-04-25T00:26:07.572 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_haproxy", 2024-04-25T00:26:07.572 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.572 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.572 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.572 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T00:26:07.573 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.573 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.573 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.573 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "HAproxy container image", 2024-04-25T00:26:07.573 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.573 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.573 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.573 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_agent": { 2024-04-25T00:26:07.573 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_agent", 2024-04-25T00:26:07.573 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.573 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.573 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.574 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T00:26:07.574 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.574 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.574 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.574 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger agent container image", 2024-04-25T00:26:07.574 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.574 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.574 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.574 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_collector": { 2024-04-25T00:26:07.575 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_collector", 2024-04-25T00:26:07.575 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.575 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.575 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.575 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T00:26:07.575 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.575 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.575 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.575 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger collector container image", 2024-04-25T00:26:07.575 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.575 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.575 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.576 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_query": { 2024-04-25T00:26:07.576 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_query", 2024-04-25T00:26:07.576 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.576 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.576 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.576 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T00:26:07.576 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.576 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.576 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.576 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger query container image", 2024-04-25T00:26:07.576 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.577 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.577 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.577 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_keepalived": { 2024-04-25T00:26:07.577 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_keepalived", 2024-04-25T00:26:07.577 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.577 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.577 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.577 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T00:26:07.577 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.577 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.577 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.577 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Keepalived container image", 2024-04-25T00:26:07.578 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.578 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.578 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.578 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_loki": { 2024-04-25T00:26:07.578 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_loki", 2024-04-25T00:26:07.578 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.578 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.578 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.578 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T00:26:07.578 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.578 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.579 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.579 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Loki container image", 2024-04-25T00:26:07.579 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.579 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.579 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.579 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_node_exporter": { 2024-04-25T00:26:07.579 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_node_exporter", 2024-04-25T00:26:07.579 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.579 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.579 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.579 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T00:26:07.579 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.579 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.580 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.580 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T00:26:07.580 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.580 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.580 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.580 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_nvmeof": { 2024-04-25T00:26:07.580 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_nvmeof", 2024-04-25T00:26:07.580 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.580 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.580 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.580 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T00:26:07.580 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.581 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.581 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.581 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Nvme-of container image", 2024-04-25T00:26:07.581 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.581 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.581 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.581 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_prometheus": { 2024-04-25T00:26:07.581 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_prometheus", 2024-04-25T00:26:07.581 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.581 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.581 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.581 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T00:26:07.582 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.582 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.582 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.582 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T00:26:07.582 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.582 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.582 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.582 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_promtail": { 2024-04-25T00:26:07.582 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_promtail", 2024-04-25T00:26:07.582 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.583 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.583 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.583 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T00:26:07.583 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.583 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.583 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.583 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Promtail container image", 2024-04-25T00:26:07.583 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.583 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.583 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.584 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_samba": { 2024-04-25T00:26:07.584 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_samba", 2024-04-25T00:26:07.584 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.584 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.584 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.584 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T00:26:07.584 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.584 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.584 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.584 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Samba/SMB container image", 2024-04-25T00:26:07.584 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.584 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.584 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.585 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_snmp_gateway": { 2024-04-25T00:26:07.585 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_snmp_gateway", 2024-04-25T00:26:07.585 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.585 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.585 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.585 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T00:26:07.585 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.585 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.585 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.585 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SNMP Gateway container image", 2024-04-25T00:26:07.585 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.585 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.586 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.586 INFO:teuthology.orchestra.run.smithi027.stdout: "container_init": { 2024-04-25T00:26:07.586 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_init", 2024-04-25T00:26:07.586 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.586 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.586 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.586 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.586 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.586 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.586 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.587 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T00:26:07.587 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.587 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.587 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.587 INFO:teuthology.orchestra.run.smithi027.stdout: "daemon_cache_timeout": { 2024-04-25T00:26:07.587 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "daemon_cache_timeout", 2024-04-25T00:26:07.587 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.587 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.587 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.587 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T00:26:07.587 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.588 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.588 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.588 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T00:26:07.588 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.588 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.588 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.588 INFO:teuthology.orchestra.run.smithi027.stdout: "default_cephadm_command_timeout": { 2024-04-25T00:26:07.588 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T00:26:07.588 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.588 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.588 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.589 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "900", 2024-04-25T00:26:07.589 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.589 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.589 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.589 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T00:26:07.589 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.589 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.589 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.589 INFO:teuthology.orchestra.run.smithi027.stdout: "default_registry": { 2024-04-25T00:26:07.589 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_registry", 2024-04-25T00:26:07.590 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.590 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.590 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.590 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io", 2024-04-25T00:26:07.590 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.590 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.590 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.590 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T00:26:07.590 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.590 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.590 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.591 INFO:teuthology.orchestra.run.smithi027.stdout: "device_cache_timeout": { 2024-04-25T00:26:07.591 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_cache_timeout", 2024-04-25T00:26:07.591 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.591 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.591 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.591 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1800", 2024-04-25T00:26:07.591 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.591 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.591 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.591 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache device inventory", 2024-04-25T00:26:07.591 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.591 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.592 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.592 INFO:teuthology.orchestra.run.smithi027.stdout: "device_enhanced_scan": { 2024-04-25T00:26:07.592 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_enhanced_scan", 2024-04-25T00:26:07.592 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.592 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.592 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.592 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.592 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.592 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.592 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.592 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T00:26:07.593 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.593 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.593 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.593 INFO:teuthology.orchestra.run.smithi027.stdout: "facts_cache_timeout": { 2024-04-25T00:26:07.593 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "facts_cache_timeout", 2024-04-25T00:26:07.593 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.593 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.593 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.593 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T00:26:07.593 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.593 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.594 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.594 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache host facts data", 2024-04-25T00:26:07.594 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.594 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.594 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.594 INFO:teuthology.orchestra.run.smithi027.stdout: "grafana_dashboards_path": { 2024-04-25T00:26:07.594 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "grafana_dashboards_path", 2024-04-25T00:26:07.594 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.594 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.595 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.595 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T00:26:07.595 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.595 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.595 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.595 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T00:26:07.595 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.595 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.595 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.595 INFO:teuthology.orchestra.run.smithi027.stdout: "host_check_interval": { 2024-04-25T00:26:07.595 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "host_check_interval", 2024-04-25T00:26:07.596 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.596 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.596 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.596 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T00:26:07.596 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.596 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.596 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.596 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to perform a host check", 2024-04-25T00:26:07.596 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.596 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.596 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.596 INFO:teuthology.orchestra.run.smithi027.stdout: "hw_monitoring": { 2024-04-25T00:26:07.596 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hw_monitoring", 2024-04-25T00:26:07.597 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.597 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.597 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.597 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.597 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.597 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.597 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.597 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T00:26:07.597 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.597 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.597 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.598 INFO:teuthology.orchestra.run.smithi027.stdout: "inventory_list_all": { 2024-04-25T00:26:07.598 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "inventory_list_all", 2024-04-25T00:26:07.598 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.598 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.598 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.598 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.598 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.598 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.598 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.598 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T00:26:07.599 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.599 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.599 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.599 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.599 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.599 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.599 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.599 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.599 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.599 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.599 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.600 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.600 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.600 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.600 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.600 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.600 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.600 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.600 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.600 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.600 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.600 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.600 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.601 INFO:teuthology.orchestra.run.smithi027.stdout: "log_refresh_metadata": { 2024-04-25T00:26:07.601 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_refresh_metadata", 2024-04-25T00:26:07.601 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.601 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.601 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.601 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.601 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.601 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.601 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.601 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T00:26:07.602 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.602 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.602 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.602 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.602 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.602 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.602 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.602 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.602 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.602 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T00:26:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.604 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.604 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.604 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.604 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.604 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.604 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.604 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.604 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.604 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.604 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.604 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.605 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.605 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.605 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.605 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.605 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.605 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.605 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.605 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.605 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.605 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.605 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.606 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.606 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.606 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.606 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.606 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.606 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.606 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.606 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T00:26:07.606 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T00:26:07.606 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.606 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.607 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.607 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.607 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.607 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.607 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.607 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T00:26:07.607 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.607 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.607 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.607 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T00:26:07.608 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T00:26:07.608 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.608 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.608 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.608 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "*", 2024-04-25T00:26:07.608 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.608 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.608 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.608 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T00:26:07.608 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.608 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.608 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.609 INFO:teuthology.orchestra.run.smithi027.stdout: "max_count_per_host": { 2024-04-25T00:26:07.609 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_count_per_host", 2024-04-25T00:26:07.609 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.609 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.609 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.609 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:07.609 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.609 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.609 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.609 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of daemons per service per host", 2024-04-25T00:26:07.609 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.609 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.609 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.610 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd_draining_count": { 2024-04-25T00:26:07.610 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_osd_draining_count", 2024-04-25T00:26:07.610 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.610 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.610 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.610 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:07.610 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.610 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.610 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.610 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T00:26:07.611 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.611 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.611 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.611 INFO:teuthology.orchestra.run.smithi027.stdout: "migration_current": { 2024-04-25T00:26:07.611 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "migration_current", 2024-04-25T00:26:07.611 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.611 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.611 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.611 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.611 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.611 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.612 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.612 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "internal - do not modify", 2024-04-25T00:26:07.612 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.612 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.612 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.612 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T00:26:07.612 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T00:26:07.612 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.612 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.612 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.612 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "root", 2024-04-25T00:26:07.612 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.613 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.613 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.613 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm-package", 2024-04-25T00:26:07.613 INFO:teuthology.orchestra.run.smithi027.stdout: "root" 2024-04-25T00:26:07.613 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.613 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T00:26:07.613 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.613 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.613 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.613 INFO:teuthology.orchestra.run.smithi027.stdout: "oob_default_addr": { 2024-04-25T00:26:07.614 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "oob_default_addr", 2024-04-25T00:26:07.614 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.614 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.614 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.614 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "169.254.1.1", 2024-04-25T00:26:07.614 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.614 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.614 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.614 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T00:26:07.614 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.615 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.615 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.615 INFO:teuthology.orchestra.run.smithi027.stdout: "prometheus_alerts_path": { 2024-04-25T00:26:07.615 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus_alerts_path", 2024-04-25T00:26:07.615 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.615 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.615 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.615 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T00:26:07.615 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.615 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.615 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.616 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T00:26:07.616 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.616 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.616 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.616 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_insecure": { 2024-04-25T00:26:07.616 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_insecure", 2024-04-25T00:26:07.616 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.616 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.616 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.616 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.616 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.617 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.617 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.617 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T00:26:07.617 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.617 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.617 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.617 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_password": { 2024-04-25T00:26:07.617 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_password", 2024-04-25T00:26:07.617 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.617 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.617 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.618 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.618 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.618 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.618 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.618 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T00:26:07.618 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.618 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.618 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.618 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_url": { 2024-04-25T00:26:07.618 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_url", 2024-04-25T00:26:07.618 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.619 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.619 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.619 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.619 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.619 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.619 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.619 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T00:26:07.619 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.619 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.619 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.619 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_username": { 2024-04-25T00:26:07.619 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_username", 2024-04-25T00:26:07.619 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.620 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.620 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.620 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.620 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.620 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.620 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.620 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T00:26:07.620 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.620 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.620 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.620 INFO:teuthology.orchestra.run.smithi027.stdout: "secure_monitoring_stack": { 2024-04-25T00:26:07.620 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "secure_monitoring_stack", 2024-04-25T00:26:07.621 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.621 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.621 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.621 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.621 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.621 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.621 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.621 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T00:26:07.621 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.621 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.621 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.621 INFO:teuthology.orchestra.run.smithi027.stdout: "service_discovery_port": { 2024-04-25T00:26:07.622 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "service_discovery_port", 2024-04-25T00:26:07.622 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.622 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.622 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.622 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8765", 2024-04-25T00:26:07.622 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.622 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.622 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.622 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "cephadm service discovery port", 2024-04-25T00:26:07.622 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.622 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.622 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.623 INFO:teuthology.orchestra.run.smithi027.stdout: "ssh_config_file": { 2024-04-25T00:26:07.623 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssh_config_file", 2024-04-25T00:26:07.623 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.623 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.623 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.623 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.623 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.623 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.623 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.623 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T00:26:07.623 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.623 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.624 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.624 INFO:teuthology.orchestra.run.smithi027.stdout: "use_agent": { 2024-04-25T00:26:07.624 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_agent", 2024-04-25T00:26:07.624 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.624 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.624 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.624 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.624 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.624 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.624 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.624 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T00:26:07.624 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.625 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.625 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.625 INFO:teuthology.orchestra.run.smithi027.stdout: "use_repo_digest": { 2024-04-25T00:26:07.625 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_repo_digest", 2024-04-25T00:26:07.625 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.625 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.625 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.625 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.625 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.625 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.625 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.625 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T00:26:07.626 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.626 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.626 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.626 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_failed_host_check": { 2024-04-25T00:26:07.626 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_failed_host_check", 2024-04-25T00:26:07.626 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.626 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.626 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.626 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.626 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.626 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.626 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.627 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T00:26:07.627 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.627 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.627 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.627 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_daemons": { 2024-04-25T00:26:07.627 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_daemons", 2024-04-25T00:26:07.627 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.627 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.627 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.627 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.627 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.627 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.628 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.628 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T00:26:07.628 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.628 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.628 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.628 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_hosts": { 2024-04-25T00:26:07.628 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_hosts", 2024-04-25T00:26:07.628 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.628 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.628 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.628 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.629 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.629 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.629 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.629 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T00:26:07.629 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.629 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.629 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.629 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.629 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.629 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crash", 2024-04-25T00:26:07.630 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.630 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.630 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.630 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.630 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.630 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.630 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.630 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.630 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.630 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.630 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.630 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.631 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.631 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.631 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.631 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.631 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.631 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.631 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.631 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.631 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.631 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.631 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.632 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.632 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.632 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.632 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.632 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.632 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.632 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.632 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.632 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.632 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.632 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.632 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.632 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.633 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.633 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.633 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.633 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.633 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.633 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.633 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.633 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.633 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.633 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.633 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.634 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.634 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.634 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.634 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.634 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.634 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.634 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.634 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.634 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.634 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.634 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.635 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.635 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.635 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.635 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.635 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.635 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.635 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.635 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.635 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.635 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.635 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.636 INFO:teuthology.orchestra.run.smithi027.stdout: "retain_interval": { 2024-04-25T00:26:07.636 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retain_interval", 2024-04-25T00:26:07.636 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.636 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.636 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.636 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "31536000", 2024-04-25T00:26:07.636 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.636 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.636 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.636 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T00:26:07.636 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.636 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.637 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.637 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_recent_interval": { 2024-04-25T00:26:07.637 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_recent_interval", 2024-04-25T00:26:07.637 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.637 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.637 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.637 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1209600", 2024-04-25T00:26:07.637 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.637 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.637 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.637 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T00:26:07.637 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.638 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.638 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.638 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.638 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.638 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.638 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dashboard", 2024-04-25T00:26:07.638 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.638 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.638 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.638 INFO:teuthology.orchestra.run.smithi027.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T00:26:07.639 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T00:26:07.639 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.639 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.639 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.639 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:07.639 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.639 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.639 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.639 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.639 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.639 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.639 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.640 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T00:26:07.640 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T00:26:07.640 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.640 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.640 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.640 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.640 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.640 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.640 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.640 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.640 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.641 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.641 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.641 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T00:26:07.641 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T00:26:07.641 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.641 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.641 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.641 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.641 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.641 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.641 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.642 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.642 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.642 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.642 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.642 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_ENABLED": { 2024-04-25T00:26:07.642 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T00:26:07.642 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.642 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.642 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.642 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.643 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.643 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.643 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.643 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.643 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.643 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.643 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.643 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T00:26:07.643 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T00:26:07.643 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.643 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.644 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.644 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.644 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.644 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.644 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.644 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.644 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.644 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.644 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.644 INFO:teuthology.orchestra.run.smithi027.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T00:26:07.644 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T00:26:07.645 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.645 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.645 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.645 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.645 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.645 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.645 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.645 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.645 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.645 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.645 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.646 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T00:26:07.646 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T00:26:07.646 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.646 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.646 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.646 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.646 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.646 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.646 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.646 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.646 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.647 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.647 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.647 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T00:26:07.647 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T00:26:07.647 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.647 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.647 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.647 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.647 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.647 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.647 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.648 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.648 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.648 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.648 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.648 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T00:26:07.648 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T00:26:07.648 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.648 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.648 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.648 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.648 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.648 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.649 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.649 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.649 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.649 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.649 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.649 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T00:26:07.649 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T00:26:07.649 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.649 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.649 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.650 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.650 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.650 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.650 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.650 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.650 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.650 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.650 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.650 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T00:26:07.650 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T00:26:07.650 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.651 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.651 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.651 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.651 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.651 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.651 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.651 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.651 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.651 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.651 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.652 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T00:26:07.652 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T00:26:07.652 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.652 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.652 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.652 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.652 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.652 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.652 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.652 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.652 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.653 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.653 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.653 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T00:26:07.653 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T00:26:07.653 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.653 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.653 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.653 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.653 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.653 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.653 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.654 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.654 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.654 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.654 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.654 INFO:teuthology.orchestra.run.smithi027.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T00:26:07.654 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T00:26:07.654 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.654 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.654 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.654 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.654 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.655 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.655 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.655 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.655 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.655 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.655 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.655 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T00:26:07.655 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T00:26:07.655 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.655 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.655 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.655 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T00:26:07.656 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.656 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.656 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.656 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.656 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.656 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.656 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.656 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T00:26:07.656 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T00:26:07.656 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.656 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.656 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.657 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.657 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.657 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.657 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.657 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.657 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.657 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.657 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.657 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_URL": { 2024-04-25T00:26:07.657 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_URL", 2024-04-25T00:26:07.657 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.658 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.658 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.658 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.658 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.658 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.658 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.658 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.658 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.658 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.658 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.658 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T00:26:07.659 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T00:26:07.659 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.659 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.659 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.659 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T00:26:07.659 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.659 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.659 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.659 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.659 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.659 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.659 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.659 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T00:26:07.660 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T00:26:07.660 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.660 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.660 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.660 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.660 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.660 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.660 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.660 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.660 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.660 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.660 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.661 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T00:26:07.661 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T00:26:07.661 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.661 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.661 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.661 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.661 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.661 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.661 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.661 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.661 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.662 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.662 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.662 INFO:teuthology.orchestra.run.smithi027.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T00:26:07.662 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T00:26:07.662 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.662 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.662 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.662 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.662 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.662 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.662 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.663 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.663 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.663 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.663 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.663 INFO:teuthology.orchestra.run.smithi027.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T00:26:07.663 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T00:26:07.663 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.663 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.663 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.663 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.664 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.664 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.664 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.664 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.664 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.664 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.664 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.664 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T00:26:07.664 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T00:26:07.664 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.664 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.665 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.665 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.665 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.665 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.665 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.665 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.665 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.665 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.665 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.665 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T00:26:07.665 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T00:26:07.666 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.666 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.666 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.666 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.666 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.666 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.666 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.666 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.666 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.666 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.666 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.667 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T00:26:07.667 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T00:26:07.667 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.667 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.667 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.667 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.667 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.667 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.667 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.667 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.667 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.667 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.668 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.668 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T00:26:07.668 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T00:26:07.668 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.668 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.668 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.668 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.668 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.668 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.668 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.668 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.669 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.669 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.669 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.669 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T00:26:07.669 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T00:26:07.669 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.669 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.669 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.669 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.669 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.669 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.669 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.670 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.670 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.670 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.670 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.670 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T00:26:07.670 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T00:26:07.670 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.670 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.670 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.670 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.670 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.671 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.671 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.671 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.671 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.671 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.671 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.671 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T00:26:07.671 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T00:26:07.671 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.671 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.671 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.671 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.672 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.672 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.672 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.672 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.672 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.672 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.672 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.672 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T00:26:07.672 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T00:26:07.672 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.672 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.673 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.673 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.673 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.673 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.673 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.673 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.673 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.673 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.673 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.673 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T00:26:07.674 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T00:26:07.674 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.674 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.674 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.674 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.674 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.674 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.674 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.674 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.674 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.674 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.674 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.675 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T00:26:07.675 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T00:26:07.675 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.675 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.675 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.675 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.675 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.675 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.675 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.675 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.675 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.675 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.675 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.676 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T00:26:07.676 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T00:26:07.676 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.676 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.676 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.676 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T00:26:07.676 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.676 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.676 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.676 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.677 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.677 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.677 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.677 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T00:26:07.677 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T00:26:07.677 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.677 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.677 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.677 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:07.677 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.677 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.678 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.678 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.678 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.678 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.678 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.678 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T00:26:07.678 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T00:26:07.678 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.678 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.678 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.679 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8", 2024-04-25T00:26:07.679 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.679 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.679 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.679 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.679 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.679 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.679 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.679 INFO:teuthology.orchestra.run.smithi027.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T00:26:07.679 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T00:26:07.679 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.680 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.680 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.680 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "45", 2024-04-25T00:26:07.680 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.680 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.680 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.680 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.680 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.680 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.680 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.681 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T00:26:07.681 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T00:26:07.681 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.681 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.681 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.681 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.681 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.681 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.681 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.681 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.681 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.681 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.682 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.682 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T00:26:07.682 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T00:26:07.682 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.682 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.682 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.682 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T00:26:07.682 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.682 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.682 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.683 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.683 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.683 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.683 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.683 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T00:26:07.683 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T00:26:07.683 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.683 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.683 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.683 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.683 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.684 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.684 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.684 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.684 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.684 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.684 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.684 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T00:26:07.684 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T00:26:07.684 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.684 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.684 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.685 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.685 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.685 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.685 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.685 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.685 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.685 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.685 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.685 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T00:26:07.685 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T00:26:07.685 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.686 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.686 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.686 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T00:26:07.686 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.686 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.686 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.686 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.686 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.686 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.686 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.686 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T00:26:07.687 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T00:26:07.687 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.687 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.687 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.687 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:07.687 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.687 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.687 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.687 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.687 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.687 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.687 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.688 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T00:26:07.688 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T00:26:07.688 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.688 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.688 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.688 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T00:26:07.688 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.688 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.688 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.688 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.688 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.689 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.689 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.689 INFO:teuthology.orchestra.run.smithi027.stdout: "cross_origin_url": { 2024-04-25T00:26:07.689 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cross_origin_url", 2024-04-25T00:26:07.689 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.689 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.689 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.689 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.689 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.689 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.689 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.690 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.690 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.690 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.690 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.690 INFO:teuthology.orchestra.run.smithi027.stdout: "crt_file": { 2024-04-25T00:26:07.690 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crt_file", 2024-04-25T00:26:07.690 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.690 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.690 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.690 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.690 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.690 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.691 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.691 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.691 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.691 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.691 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.691 INFO:teuthology.orchestra.run.smithi027.stdout: "debug": { 2024-04-25T00:26:07.691 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "debug", 2024-04-25T00:26:07.691 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.691 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.691 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.691 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.692 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.692 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.692 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.692 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable/disable debug options", 2024-04-25T00:26:07.692 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.692 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.692 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.692 INFO:teuthology.orchestra.run.smithi027.stdout: "jwt_token_ttl": { 2024-04-25T00:26:07.692 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "jwt_token_ttl", 2024-04-25T00:26:07.692 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.692 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.693 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.693 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "28800", 2024-04-25T00:26:07.693 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.693 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.693 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.693 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.693 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.693 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.693 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.693 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T00:26:07.693 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T00:26:07.693 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.694 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.694 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.694 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.694 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.694 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.694 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.694 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.694 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.694 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.694 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.695 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.695 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.695 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.695 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.695 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.695 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.695 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.695 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.695 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.695 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.695 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.695 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.696 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.696 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.696 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.696 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.696 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.696 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.696 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.696 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.696 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.696 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.696 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.697 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.697 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.697 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.697 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.697 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.697 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.697 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.697 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.697 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.697 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.697 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.697 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.697 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.698 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.698 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.698 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.698 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.698 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.698 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.698 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.698 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.698 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.698 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.698 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.698 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.699 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.699 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.699 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.699 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.699 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.699 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.699 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.699 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.699 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.699 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.699 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.699 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.699 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.700 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.700 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.700 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.700 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.700 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.700 INFO:teuthology.orchestra.run.smithi027.stdout: "motd": { 2024-04-25T00:26:07.700 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "motd", 2024-04-25T00:26:07.700 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.700 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.700 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.700 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.700 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.700 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.701 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.701 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "The message of the day", 2024-04-25T00:26:07.701 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.701 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.701 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.701 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect_resolve_ip_addr": { 2024-04-25T00:26:07.701 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T00:26:07.701 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.701 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.701 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.702 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.702 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.702 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.702 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.702 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.702 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.702 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.702 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.702 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T00:26:07.702 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T00:26:07.703 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.703 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.703 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.703 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T00:26:07.703 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.703 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.703 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.703 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.703 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.703 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.703 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.704 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T00:26:07.704 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T00:26:07.704 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.704 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.704 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.704 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8080", 2024-04-25T00:26:07.704 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.704 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.704 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.704 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.704 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.704 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.705 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.705 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T00:26:07.705 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T00:26:07.705 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.705 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.705 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.705 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.705 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.705 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.705 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.705 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.706 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.706 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.706 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.706 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl_server_port": { 2024-04-25T00:26:07.706 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl_server_port", 2024-04-25T00:26:07.706 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.706 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.706 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.706 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8443", 2024-04-25T00:26:07.706 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.706 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.707 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.707 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.707 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.707 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.707 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.707 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T00:26:07.707 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T00:26:07.707 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.707 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.709 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.709 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "redirect", 2024-04-25T00:26:07.709 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.709 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.709 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.709 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.709 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect" 2024-04-25T00:26:07.709 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.710 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.710 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.710 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.710 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.710 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T00:26:07.710 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T00:26:07.710 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.710 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.710 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.710 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T00:26:07.711 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T00:26:07.711 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T00:26:07.711 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.711 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.711 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.711 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.711 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.711 INFO:teuthology.orchestra.run.smithi027.stdout: "url_prefix": { 2024-04-25T00:26:07.711 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url_prefix", 2024-04-25T00:26:07.711 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.711 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.711 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.712 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.712 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.712 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.712 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.712 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.712 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.712 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.712 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.712 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.712 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.713 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "devicehealth", 2024-04-25T00:26:07.713 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.713 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.713 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.713 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_monitoring": { 2024-04-25T00:26:07.713 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_monitoring", 2024-04-25T00:26:07.713 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.713 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.713 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.713 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.713 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.713 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.713 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.714 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "monitor device health metrics", 2024-04-25T00:26:07.714 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.714 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.714 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.714 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.714 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.714 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.714 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.714 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.714 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.714 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.715 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.715 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.715 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.715 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.715 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.715 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.715 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.715 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.715 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.715 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.715 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.715 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.716 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.716 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.716 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.716 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.716 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.716 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.716 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.716 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.716 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.716 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.716 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.717 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.717 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.717 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.717 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.717 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.717 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.717 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.717 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.717 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.717 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.717 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.717 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.718 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.718 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.718 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.718 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.718 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.718 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.718 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.718 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.718 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.718 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.718 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.719 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.719 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.719 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.719 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.719 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.719 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.719 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.719 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.719 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.719 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.719 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.719 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.719 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.720 INFO:teuthology.orchestra.run.smithi027.stdout: "mark_out_threshold": { 2024-04-25T00:26:07.720 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mark_out_threshold", 2024-04-25T00:26:07.720 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.720 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.720 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.720 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2419200", 2024-04-25T00:26:07.720 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.720 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.720 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.720 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T00:26:07.720 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.720 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.720 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.721 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": { 2024-04-25T00:26:07.721 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_name", 2024-04-25T00:26:07.721 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.721 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.721 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.721 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "device_health_metrics", 2024-04-25T00:26:07.721 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.721 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.721 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.721 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T00:26:07.721 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.722 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.722 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.722 INFO:teuthology.orchestra.run.smithi027.stdout: "retention_period": { 2024-04-25T00:26:07.722 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retention_period", 2024-04-25T00:26:07.722 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.722 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.722 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.722 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15552000", 2024-04-25T00:26:07.722 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.722 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.722 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.723 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain device health metrics", 2024-04-25T00:26:07.723 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.723 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.723 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.723 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_frequency": { 2024-04-25T00:26:07.723 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_frequency", 2024-04-25T00:26:07.723 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.723 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.723 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.723 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T00:26:07.723 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.724 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.724 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.724 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T00:26:07.724 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.724 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.724 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.724 INFO:teuthology.orchestra.run.smithi027.stdout: "self_heal": { 2024-04-25T00:26:07.724 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "self_heal", 2024-04-25T00:26:07.724 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.724 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.724 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.725 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.725 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.725 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.725 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.725 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T00:26:07.725 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.725 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.725 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.725 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T00:26:07.725 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T00:26:07.725 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.725 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.726 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.726 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T00:26:07.726 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.726 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.726 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.726 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T00:26:07.726 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.726 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.726 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.726 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_threshold": { 2024-04-25T00:26:07.726 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_threshold", 2024-04-25T00:26:07.727 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.727 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.727 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.727 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7257600", 2024-04-25T00:26:07.727 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.727 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.727 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.727 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T00:26:07.727 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.727 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.727 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.727 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.727 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.728 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.728 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "diskprediction_local", 2024-04-25T00:26:07.728 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.728 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.728 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.728 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.728 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.728 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.728 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.728 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.728 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.729 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.729 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.729 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.729 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.729 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.729 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.729 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.729 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.729 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.729 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.729 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.729 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.730 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.730 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.730 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.730 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.730 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.730 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.730 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.730 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.730 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.730 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.730 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.730 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.731 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.731 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.731 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.731 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.731 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.731 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.731 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.731 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.731 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.731 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.731 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.731 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.732 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.732 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.732 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.732 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.732 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.732 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.732 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.732 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.732 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.732 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.732 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.733 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.733 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.733 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.733 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.733 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.733 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.733 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.733 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.733 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.733 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.733 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.733 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.733 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.734 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.734 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.734 INFO:teuthology.orchestra.run.smithi027.stdout: "predict_interval": { 2024-04-25T00:26:07.734 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predict_interval", 2024-04-25T00:26:07.734 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.734 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.734 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.735 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T00:26:07.735 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.735 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.735 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.735 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.735 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.735 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.735 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.735 INFO:teuthology.orchestra.run.smithi027.stdout: "predictor_model": { 2024-04-25T00:26:07.735 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predictor_model", 2024-04-25T00:26:07.735 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.736 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.736 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.736 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "prophetstor", 2024-04-25T00:26:07.736 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.736 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.736 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.736 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.736 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.736 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.736 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.736 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T00:26:07.737 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T00:26:07.737 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.737 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.737 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.737 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T00:26:07.737 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.737 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.737 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.737 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.737 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.737 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.737 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.738 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.738 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.738 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.738 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "influx", 2024-04-25T00:26:07.738 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": false, 2024-04-25T00:26:07.738 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "influxdb python module not found", 2024-04-25T00:26:07.738 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.738 INFO:teuthology.orchestra.run.smithi027.stdout: "batch_size": { 2024-04-25T00:26:07.738 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "batch_size", 2024-04-25T00:26:07.738 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.738 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.739 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.739 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5000", 2024-04-25T00:26:07.739 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.739 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.739 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.739 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T00:26:07.739 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.739 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.739 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.739 INFO:teuthology.orchestra.run.smithi027.stdout: "database": { 2024-04-25T00:26:07.739 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "database", 2024-04-25T00:26:07.740 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.740 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.740 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.740 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "ceph", 2024-04-25T00:26:07.740 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.740 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.740 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.740 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T00:26:07.740 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.740 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.740 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.741 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": { 2024-04-25T00:26:07.741 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hostname", 2024-04-25T00:26:07.741 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.741 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.741 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.741 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.741 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.741 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.741 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.741 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server hostname", 2024-04-25T00:26:07.741 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.741 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.741 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T00:26:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T00:26:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "30", 2024-04-25T00:26:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "5", 2024-04-25T00:26:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T00:26:07.742 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.743 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.743 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.743 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.743 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.743 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.743 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.743 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.743 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.743 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.743 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.743 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.744 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.744 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.744 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.744 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.744 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.744 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.744 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.744 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.744 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.744 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.744 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.744 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.745 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.745 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.745 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.745 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.745 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.745 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.745 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.745 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.745 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.745 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.745 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.745 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.746 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.746 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.746 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.746 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.746 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.746 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.746 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.746 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.746 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.746 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.746 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.747 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.747 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.747 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.747 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.747 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.747 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.747 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.747 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.747 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.747 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.747 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.747 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.748 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.748 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.748 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.748 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.748 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.748 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.748 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.748 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.748 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.748 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.748 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.748 INFO:teuthology.orchestra.run.smithi027.stdout: "password": { 2024-04-25T00:26:07.749 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "password", 2024-04-25T00:26:07.749 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.749 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.749 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.749 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.749 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.749 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.749 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.749 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "password of InfluxDB server user", 2024-04-25T00:26:07.749 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.749 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.749 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.750 INFO:teuthology.orchestra.run.smithi027.stdout: "port": { 2024-04-25T00:26:07.750 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "port", 2024-04-25T00:26:07.750 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.750 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.750 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.750 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8086", 2024-04-25T00:26:07.750 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.750 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.750 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.750 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server port", 2024-04-25T00:26:07.750 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.751 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.751 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.751 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T00:26:07.751 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T00:26:07.751 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.751 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.751 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.751 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "false", 2024-04-25T00:26:07.751 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.751 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.751 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.751 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T00:26:07.752 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.752 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.752 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.752 INFO:teuthology.orchestra.run.smithi027.stdout: "threads": { 2024-04-25T00:26:07.752 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threads", 2024-04-25T00:26:07.752 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.752 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.752 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.752 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T00:26:07.752 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T00:26:07.752 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "32", 2024-04-25T00:26:07.753 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.753 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T00:26:07.753 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.753 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.753 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.753 INFO:teuthology.orchestra.run.smithi027.stdout: "username": { 2024-04-25T00:26:07.753 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "username", 2024-04-25T00:26:07.753 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.753 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.753 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.754 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.754 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.754 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.754 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.754 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "username of InfluxDB server user", 2024-04-25T00:26:07.754 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.754 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.754 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.754 INFO:teuthology.orchestra.run.smithi027.stdout: "verify_ssl": { 2024-04-25T00:26:07.754 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "verify_ssl", 2024-04-25T00:26:07.754 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.755 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.755 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.755 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "true", 2024-04-25T00:26:07.755 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.755 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.755 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.755 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T00:26:07.755 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.755 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.755 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.755 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.755 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.756 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.756 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.756 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "insights", 2024-04-25T00:26:07.756 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.756 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.756 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.756 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.756 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.756 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.756 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.756 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.756 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.756 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.757 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.757 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.757 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.757 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.757 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.757 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.757 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.757 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.757 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.757 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.757 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.757 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.758 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.758 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.758 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.758 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.758 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.758 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.758 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.758 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.758 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.758 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.758 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.759 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.759 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.759 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.759 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.759 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.759 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.759 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.759 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.759 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.759 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.759 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.760 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.760 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.760 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.760 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.760 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.760 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.760 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.760 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.760 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.760 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.760 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.760 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.761 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.761 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.761 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.761 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.761 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.761 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.761 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.761 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.761 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.761 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.761 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.762 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.762 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.762 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.762 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.762 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.762 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "iostat", 2024-04-25T00:26:07.762 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.762 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.762 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.762 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.762 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.763 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.763 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.763 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.763 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.763 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.763 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.763 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.763 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.763 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.763 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.763 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.763 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.764 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.764 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.764 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.764 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.764 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.764 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.764 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.764 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.764 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.764 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.765 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.765 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.765 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.765 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.765 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.765 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.765 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.765 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.765 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.765 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.765 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.765 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.766 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.766 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.766 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.766 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.766 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.766 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.766 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.766 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.766 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.766 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.766 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.767 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.767 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.767 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.767 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.767 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.767 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.767 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.767 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.767 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.767 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.767 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.767 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.768 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.768 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.768 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.768 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.768 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.768 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.768 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.768 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.768 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.768 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.768 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.768 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.768 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "k8sevents", 2024-04-25T00:26:07.769 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.769 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.769 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.769 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_event_retention_days": { 2024-04-25T00:26:07.769 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ceph_event_retention_days", 2024-04-25T00:26:07.769 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.769 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.769 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.769 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7", 2024-04-25T00:26:07.769 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.769 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.769 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.770 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T00:26:07.770 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.770 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.770 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.770 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.770 INFO:teuthology.orchestra.run.smithi027.stdout: "config_check_secs": { 2024-04-25T00:26:07.770 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_check_secs", 2024-04-25T00:26:07.770 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.770 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.770 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.770 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:07.770 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "10", 2024-04-25T00:26:07.771 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.771 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.771 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T00:26:07.771 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.771 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.771 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.771 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.771 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.771 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.771 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.771 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.771 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.771 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.772 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.772 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.772 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.772 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.772 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.772 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.772 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.772 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.772 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.772 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.772 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.772 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.772 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.773 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.773 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.773 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.773 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.773 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.773 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.773 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.773 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.773 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.773 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.773 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.774 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.774 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.774 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.774 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.774 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.774 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.774 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.774 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.774 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.774 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.774 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.774 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.775 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.775 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.775 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.775 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.775 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.775 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.775 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.775 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.775 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.775 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.775 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.775 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.775 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.776 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.776 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.776 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.776 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.776 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.776 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.776 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.776 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.776 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.776 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.776 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.776 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.777 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.777 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.777 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.777 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.777 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "localpool", 2024-04-25T00:26:07.777 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.777 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.777 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.777 INFO:teuthology.orchestra.run.smithi027.stdout: "failure_domain": { 2024-04-25T00:26:07.777 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "failure_domain", 2024-04-25T00:26:07.777 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.778 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.778 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.778 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "host", 2024-04-25T00:26:07.778 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.778 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.778 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.778 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "failure domain for any created local pool", 2024-04-25T00:26:07.778 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T00:26:07.778 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.778 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.778 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.778 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.779 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.779 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.779 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.779 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.779 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.779 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.779 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.779 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.779 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.779 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.779 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.779 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.779 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.780 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.780 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.780 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.780 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.780 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.780 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.780 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.780 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.780 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.780 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.780 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.781 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.781 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.781 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.781 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.781 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.781 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.781 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.781 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.781 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.781 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.781 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.781 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.781 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.782 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.782 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.782 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.782 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.782 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.782 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.782 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.782 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.782 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.782 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.782 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.782 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.782 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.783 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.783 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.783 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.783 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.783 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.783 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.783 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.783 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.783 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.783 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.783 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.783 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.784 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.784 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.784 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.784 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.784 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": { 2024-04-25T00:26:07.784 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_size", 2024-04-25T00:26:07.784 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.784 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.784 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.784 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.785 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.785 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.785 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.785 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default min_size for any created local pool", 2024-04-25T00:26:07.785 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T00:26:07.785 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.785 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.786 INFO:teuthology.orchestra.run.smithi027.stdout: "num_rep": { 2024-04-25T00:26:07.786 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "num_rep", 2024-04-25T00:26:07.786 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.786 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.786 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.786 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3", 2024-04-25T00:26:07.786 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.786 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.786 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.786 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default replica count for any created local pool", 2024-04-25T00:26:07.786 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.787 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.787 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.787 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": { 2024-04-25T00:26:07.787 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_num", 2024-04-25T00:26:07.787 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.787 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.787 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.787 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "128", 2024-04-25T00:26:07.787 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.787 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.787 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.788 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T00:26:07.788 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.788 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.788 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.788 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.788 INFO:teuthology.orchestra.run.smithi027.stdout: "prefix": { 2024-04-25T00:26:07.788 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prefix", 2024-04-25T00:26:07.788 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.788 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.788 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.788 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.788 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.788 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.789 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.789 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name prefix for any created local pool", 2024-04-25T00:26:07.789 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.789 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.789 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.789 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.789 INFO:teuthology.orchestra.run.smithi027.stdout: "subtree": { 2024-04-25T00:26:07.789 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subtree", 2024-04-25T00:26:07.789 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.789 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.789 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.789 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "rack", 2024-04-25T00:26:07.789 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.790 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.790 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.790 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T00:26:07.790 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T00:26:07.790 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.790 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.790 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.790 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.790 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.790 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.790 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mds_autoscaler", 2024-04-25T00:26:07.790 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.791 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.791 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.791 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.791 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.791 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.791 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.791 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.791 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.791 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.791 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.791 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.791 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.791 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.792 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.792 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.792 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.792 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.792 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.792 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.792 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.792 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.792 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.792 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.792 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.792 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.793 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.793 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.793 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.793 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.793 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.793 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.793 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.793 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.793 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.793 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.793 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.793 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.794 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.794 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.794 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.794 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.794 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.794 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.794 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.794 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.794 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.794 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.794 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.794 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.794 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.795 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.795 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.795 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.795 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.795 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.795 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.795 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.795 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.795 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.795 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.795 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.796 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.796 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.796 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.796 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.796 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.796 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.796 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.796 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.796 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.796 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.796 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.796 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.796 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirroring", 2024-04-25T00:26:07.797 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.797 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.797 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.797 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.797 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.797 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.797 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.797 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.797 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.797 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.797 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.797 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.798 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.798 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.798 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.798 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.798 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.798 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.798 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.798 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.798 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.798 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.798 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.799 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.799 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.799 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.799 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.799 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.799 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.799 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.799 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.799 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.799 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.799 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.799 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.800 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.800 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.800 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.800 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.800 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.800 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.800 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.800 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.800 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.800 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.800 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.800 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.801 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.801 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.801 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.801 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.801 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.801 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.801 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.801 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.801 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.801 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.801 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.802 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.802 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.802 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.802 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.802 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.802 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.802 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.802 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.802 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.802 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.802 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.802 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.803 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.803 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.803 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "nfs", 2024-04-25T00:26:07.803 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.803 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.803 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.803 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.803 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.803 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.803 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.803 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.803 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.804 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.804 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.804 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.804 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.804 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.804 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.804 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.804 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.804 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.804 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.804 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.804 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.805 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.805 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.805 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.805 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.805 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.805 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.805 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.805 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.805 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.805 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.805 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.806 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.806 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.806 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.806 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.806 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.806 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.806 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.806 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.806 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.806 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.806 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.806 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.807 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.807 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.807 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.807 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.807 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.807 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.807 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.807 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.807 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.807 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.807 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.807 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.808 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.808 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.808 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.808 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.808 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.808 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.808 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.808 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.808 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.808 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.808 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.809 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.809 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.809 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.809 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.809 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.809 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T00:26:07.809 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.809 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.809 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.809 INFO:teuthology.orchestra.run.smithi027.stdout: "fail_fs": { 2024-04-25T00:26:07.809 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "fail_fs", 2024-04-25T00:26:07.809 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.810 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.810 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.810 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.810 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.810 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.810 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.810 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T00:26:07.810 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.810 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.810 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.810 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.810 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.811 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.811 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.811 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.811 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.811 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.811 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.811 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.811 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.811 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.811 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.811 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.811 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.812 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.812 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.812 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.812 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.812 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.812 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.812 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.812 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.812 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.812 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.812 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.812 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.813 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.813 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.813 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.813 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.813 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.813 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.813 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.813 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.813 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.813 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.813 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.813 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.814 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.814 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.814 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.814 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.814 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.814 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.814 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.814 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.814 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.814 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.814 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.814 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.815 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.815 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.815 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.815 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.815 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.815 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.815 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.815 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.815 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.815 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.815 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.815 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.816 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.816 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.816 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.816 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.816 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.816 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator": { 2024-04-25T00:26:07.816 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T00:26:07.816 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.816 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.816 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.816 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.817 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.817 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.817 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.817 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm", 2024-04-25T00:26:07.817 INFO:teuthology.orchestra.run.smithi027.stdout: "rook", 2024-04-25T00:26:07.817 INFO:teuthology.orchestra.run.smithi027.stdout: "test_orchestrator" 2024-04-25T00:26:07.817 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.817 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Orchestrator backend", 2024-04-25T00:26:07.817 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.817 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.817 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.817 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.818 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.818 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.818 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.818 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_perf_query", 2024-04-25T00:26:07.818 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.818 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.818 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.818 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.818 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.818 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.818 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.819 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.819 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.819 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.819 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.819 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.819 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.819 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.819 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.819 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.819 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.819 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.819 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.820 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.820 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.820 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.820 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.820 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.820 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.820 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.820 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.820 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.820 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.820 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.820 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.821 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.821 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.821 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.821 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.821 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.821 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.821 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.821 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.821 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.821 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.822 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.822 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.822 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.822 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.822 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.822 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.822 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.822 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.822 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.822 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.822 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.822 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.823 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.823 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.823 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.823 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.823 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.823 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.823 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.823 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.823 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.823 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.824 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.824 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.824 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.824 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.824 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.824 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.824 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.824 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.824 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.824 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.824 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.824 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_support", 2024-04-25T00:26:07.824 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.825 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.825 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.825 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.825 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.825 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.825 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.825 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.825 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.825 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.825 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.825 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.825 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.825 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.826 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.826 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.826 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.826 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.826 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.826 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.826 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.826 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.826 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.826 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.826 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.827 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.827 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.827 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.827 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.827 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.827 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.827 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.827 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.827 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.827 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.827 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.827 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.828 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.828 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.828 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.828 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.828 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.828 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.828 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.828 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.828 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.828 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.828 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.829 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.829 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.829 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.829 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.829 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.829 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.829 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.829 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.829 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.829 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.829 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.829 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.830 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.830 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.830 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.830 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.830 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.830 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.830 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.830 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.830 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.830 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.830 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.830 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.831 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.831 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.831 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_autoscaler", 2024-04-25T00:26:07.831 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.831 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.831 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.831 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.831 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.831 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.831 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.831 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.831 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.831 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.832 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.832 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.832 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.832 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.832 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.832 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.832 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.832 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.832 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.832 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.832 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.832 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.832 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.833 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.833 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.833 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.833 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.833 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.833 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.833 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.833 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.833 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.833 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.833 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.833 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.833 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.834 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.834 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.834 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.834 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.834 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.834 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.834 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.834 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.834 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.834 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.835 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.835 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.835 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.835 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.835 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.835 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.835 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.835 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.835 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.835 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.835 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.836 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.836 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.836 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.836 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.836 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.836 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.836 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.836 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.836 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.836 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.836 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.837 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.837 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.837 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.837 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T00:26:07.837 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T00:26:07.837 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.837 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.837 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.837 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T00:26:07.837 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.837 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.837 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.837 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.838 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.838 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.838 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.838 INFO:teuthology.orchestra.run.smithi027.stdout: "threshold": { 2024-04-25T00:26:07.838 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threshold", 2024-04-25T00:26:07.838 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:07.838 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.838 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.838 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T00:26:07.838 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1.0", 2024-04-25T00:26:07.838 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.839 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.839 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "scaling threshold", 2024-04-25T00:26:07.839 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T00:26:07.839 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.839 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.839 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.839 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.839 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.839 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.839 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "progress", 2024-04-25T00:26:07.839 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.840 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.840 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.840 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_pg_recovery_event": { 2024-04-25T00:26:07.840 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_pg_recovery_event", 2024-04-25T00:26:07.840 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.840 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.840 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.840 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.840 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.840 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.840 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.840 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T00:26:07.840 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.841 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.841 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.841 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T00:26:07.841 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T00:26:07.841 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.841 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.841 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.841 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.841 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.841 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.841 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.841 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.841 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.842 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.842 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.842 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.842 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.842 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.842 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.842 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.842 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.842 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.842 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.842 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.842 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.843 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.843 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.843 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.843 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.843 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.843 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.843 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.843 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.843 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.843 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.843 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.843 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.844 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.844 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.844 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.844 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.844 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.844 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.844 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.844 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.844 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.844 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.844 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.844 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.844 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.845 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.845 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.845 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.845 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.845 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.845 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.845 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.845 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.845 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.845 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.845 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.845 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.845 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.845 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.846 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.846 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.846 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.846 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.846 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.846 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.846 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.846 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.846 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.846 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.846 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.847 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.847 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.847 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.847 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.847 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.847 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.847 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.847 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.847 INFO:teuthology.orchestra.run.smithi027.stdout: "max_completed_events": { 2024-04-25T00:26:07.847 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_completed_events", 2024-04-25T00:26:07.847 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.847 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.847 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.848 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "50", 2024-04-25T00:26:07.848 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.848 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.848 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.848 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "number of past completed events to remember", 2024-04-25T00:26:07.848 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.848 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.848 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.848 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T00:26:07.848 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T00:26:07.848 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.848 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.849 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.849 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T00:26:07.849 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.849 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.849 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.849 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long the module is going to sleep", 2024-04-25T00:26:07.849 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.849 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.849 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.849 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.849 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.850 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.850 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus", 2024-04-25T00:26:07.850 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.850 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.850 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.850 INFO:teuthology.orchestra.run.smithi027.stdout: "cache": { 2024-04-25T00:26:07.850 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cache", 2024-04-25T00:26:07.850 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.850 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.850 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.850 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.850 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.851 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.851 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.851 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.851 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.851 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.851 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.851 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.851 INFO:teuthology.orchestra.run.smithi027.stdout: "exclude_perf_counters": { 2024-04-25T00:26:07.851 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "exclude_perf_counters", 2024-04-25T00:26:07.851 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.851 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.851 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.852 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.852 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.852 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.852 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.852 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T00:26:07.852 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T00:26:07.852 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.852 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.852 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.852 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.854 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.854 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.854 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.854 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.854 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.854 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.854 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.854 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.854 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.854 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.854 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.855 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.855 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.855 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.855 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.855 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.855 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.855 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.855 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.855 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.855 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.855 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.856 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.856 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.856 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.856 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.856 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.856 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.856 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.856 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.856 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.856 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.856 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.856 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.856 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.857 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.857 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.857 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.857 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.857 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.857 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.857 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.857 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.857 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.857 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.857 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.857 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.857 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.857 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.858 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.858 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.858 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.858 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.858 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools": { 2024-04-25T00:26:07.858 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools", 2024-04-25T00:26:07.858 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.858 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.858 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.858 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.859 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.859 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.859 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.859 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.859 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.859 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.859 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.859 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T00:26:07.859 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T00:26:07.859 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.859 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.860 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.860 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "300", 2024-04-25T00:26:07.860 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.860 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.860 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.860 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.860 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.860 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.860 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.860 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.860 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_interval": { 2024-04-25T00:26:07.860 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_interval", 2024-04-25T00:26:07.860 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:07.861 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.861 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.861 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15.0", 2024-04-25T00:26:07.861 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.861 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.861 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.861 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.861 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.861 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.861 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.861 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T00:26:07.861 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T00:26:07.861 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.862 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.862 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.862 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T00:26:07.862 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.862 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.862 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.862 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T00:26:07.862 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.862 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.862 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.862 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.862 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T00:26:07.863 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T00:26:07.863 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.863 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.863 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.863 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "9283", 2024-04-25T00:26:07.863 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.863 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.863 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.863 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T00:26:07.863 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.863 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.863 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.864 INFO:teuthology.orchestra.run.smithi027.stdout: "stale_cache_strategy": { 2024-04-25T00:26:07.864 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stale_cache_strategy", 2024-04-25T00:26:07.864 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.864 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.864 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.864 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "log", 2024-04-25T00:26:07.864 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.864 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.864 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.864 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.864 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.864 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.864 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.865 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T00:26:07.865 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T00:26:07.865 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.865 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.865 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.865 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "default", 2024-04-25T00:26:07.865 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.865 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.865 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.865 INFO:teuthology.orchestra.run.smithi027.stdout: "default", 2024-04-25T00:26:07.865 INFO:teuthology.orchestra.run.smithi027.stdout: "error" 2024-04-25T00:26:07.865 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.865 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.865 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.866 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.866 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.866 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T00:26:07.866 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T00:26:07.866 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.866 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.866 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.866 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T00:26:07.866 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T00:26:07.866 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T00:26:07.867 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.867 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.867 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.867 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.867 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.867 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.867 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.867 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.867 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.867 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_support", 2024-04-25T00:26:07.867 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.867 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.867 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.867 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.867 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.868 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.868 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.868 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.868 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.868 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.868 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.868 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.868 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.868 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.868 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.868 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.868 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.868 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.869 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.869 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.869 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.869 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.869 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.869 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.869 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.869 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.869 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.869 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.869 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.869 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.870 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.870 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.870 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.870 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.870 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.870 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.870 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.870 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.870 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.870 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.871 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.871 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.871 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.871 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.871 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.871 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.871 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.871 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.871 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.871 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.871 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.871 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.871 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.872 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.872 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.872 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.872 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.872 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.872 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.872 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.872 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.872 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.872 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.872 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.872 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.872 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.873 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.873 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.873 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.873 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_snap_create": { 2024-04-25T00:26:07.873 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_snap_create", 2024-04-25T00:26:07.873 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.873 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.873 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.873 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T00:26:07.873 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.874 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.874 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.874 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.874 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.874 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.874 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.874 INFO:teuthology.orchestra.run.smithi027.stdout: "mirror_snapshot_schedule": { 2024-04-25T00:26:07.874 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T00:26:07.874 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.874 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.874 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.875 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.875 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.875 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.875 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.875 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.875 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.875 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.875 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.875 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.875 INFO:teuthology.orchestra.run.smithi027.stdout: "trash_purge_schedule": { 2024-04-25T00:26:07.875 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "trash_purge_schedule", 2024-04-25T00:26:07.875 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.875 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.875 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.876 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.876 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.876 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.876 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.876 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.876 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.876 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.876 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.876 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.876 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.876 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.876 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "restful", 2024-04-25T00:26:07.876 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.877 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.877 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.877 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_auth": { 2024-04-25T00:26:07.877 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_auth", 2024-04-25T00:26:07.877 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.877 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.877 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.877 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.877 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.877 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.877 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.877 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.878 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.878 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.878 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.878 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T00:26:07.878 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T00:26:07.878 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.878 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.878 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.878 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.878 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.878 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.879 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.879 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.879 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.879 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.879 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.879 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.879 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.879 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.879 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.879 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.879 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.879 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.879 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.880 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.880 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.880 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.880 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.880 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.880 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.880 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.880 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.880 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.880 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.880 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.880 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.880 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.881 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.881 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.881 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.881 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.881 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.881 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.881 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.881 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.881 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.881 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.881 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.881 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.881 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.882 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.882 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.882 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.882 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.882 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.882 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.882 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.882 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.882 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.882 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.882 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.882 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.882 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.883 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.883 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.883 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.883 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.883 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.883 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.883 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.883 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.883 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.883 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.883 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.883 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.884 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.884 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.884 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.884 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.884 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.884 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.884 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.884 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.884 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T00:26:07.884 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T00:26:07.884 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.884 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.885 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.885 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.885 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.885 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.885 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.885 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.885 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.885 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.885 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.885 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.885 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T00:26:07.885 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T00:26:07.886 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.886 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.886 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.886 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.886 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.886 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.886 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.886 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.886 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.886 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.886 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.886 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.887 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.887 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.887 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rgw", 2024-04-25T00:26:07.887 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.887 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.887 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.887 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.887 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.887 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.887 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.887 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.887 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.887 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.888 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.888 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.888 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.888 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.888 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.888 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.888 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.888 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.888 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.888 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.888 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.888 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.889 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.889 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.889 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.889 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.889 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.889 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.889 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.889 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.889 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.889 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.889 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.889 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.890 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.890 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.890 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.890 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.890 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.890 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.890 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.890 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.890 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.890 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.890 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.891 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.891 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.891 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.891 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.891 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.891 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.891 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.891 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.891 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.891 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.891 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.891 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.891 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.892 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.892 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.892 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.892 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.892 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.892 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.892 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.892 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.892 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.892 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.892 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.892 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.892 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.893 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.893 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.893 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.893 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rook", 2024-04-25T00:26:07.893 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.893 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.893 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.893 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.893 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.893 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.893 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.893 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.894 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.894 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.894 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.894 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.894 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.894 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.894 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.894 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.894 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.894 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.894 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.894 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.895 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.895 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.895 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.895 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.895 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.895 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.895 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.895 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.895 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.895 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.895 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.895 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.896 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.896 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.896 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.896 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.896 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.896 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.896 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.896 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.896 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.896 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.896 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.897 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.897 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.897 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.897 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.897 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.897 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.897 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.897 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.897 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.897 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.897 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.898 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.898 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.898 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.898 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.898 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.898 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.898 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.898 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.898 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.898 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.898 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.899 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.899 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.899 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.899 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.899 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.899 INFO:teuthology.orchestra.run.smithi027.stdout: "storage_class": { 2024-04-25T00:26:07.899 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "storage_class", 2024-04-25T00:26:07.899 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.899 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.899 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.899 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "local", 2024-04-25T00:26:07.899 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.900 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.900 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.900 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T00:26:07.900 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.900 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.900 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.900 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.900 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.900 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.900 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.900 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "selftest", 2024-04-25T00:26:07.900 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.901 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.901 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.901 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.901 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.901 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.901 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.901 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.901 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.901 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.901 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.901 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.901 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.902 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.902 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.902 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.902 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.902 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.902 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.902 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.902 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.902 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.902 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.902 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.902 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.902 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.903 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.903 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.903 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.903 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.903 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.903 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.903 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.903 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.903 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.903 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.903 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.903 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.904 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.904 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.904 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.904 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.904 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.904 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.904 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.904 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.904 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.904 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.904 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.904 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.905 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.905 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.905 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.905 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.905 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.905 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.905 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.905 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.905 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.905 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.905 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.906 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.906 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.906 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.906 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.906 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.906 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.906 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.906 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.906 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.906 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.906 INFO:teuthology.orchestra.run.smithi027.stdout: "roption1": { 2024-04-25T00:26:07.906 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption1", 2024-04-25T00:26:07.907 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.907 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.907 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.907 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.907 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.907 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.907 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.907 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.907 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.907 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.907 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.908 INFO:teuthology.orchestra.run.smithi027.stdout: "roption2": { 2024-04-25T00:26:07.908 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption2", 2024-04-25T00:26:07.908 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.908 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.908 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.908 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "xyz", 2024-04-25T00:26:07.908 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.908 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.908 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.908 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.908 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.908 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.908 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.909 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.909 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption1": { 2024-04-25T00:26:07.909 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption1", 2024-04-25T00:26:07.909 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.909 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.909 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.909 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.909 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.909 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.909 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.909 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.909 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.910 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.910 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.910 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption2": { 2024-04-25T00:26:07.910 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption2", 2024-04-25T00:26:07.910 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.910 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.910 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.910 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.910 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.910 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.910 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.911 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.911 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.911 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.911 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.911 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption3": { 2024-04-25T00:26:07.911 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption3", 2024-04-25T00:26:07.911 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T00:26:07.911 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.911 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.911 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.911 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.911 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.912 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.912 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.912 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.912 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.912 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.912 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption4": { 2024-04-25T00:26:07.912 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption4", 2024-04-25T00:26:07.912 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.912 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.912 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.912 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.912 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.913 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.913 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.913 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.913 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.913 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.913 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.913 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.913 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption5": { 2024-04-25T00:26:07.913 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption5", 2024-04-25T00:26:07.913 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.913 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.913 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.914 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.914 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.914 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.914 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.914 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.914 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.914 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.914 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.914 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.914 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption6": { 2024-04-25T00:26:07.914 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption6", 2024-04-25T00:26:07.915 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.915 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.915 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.915 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.915 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.915 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.915 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.915 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.915 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.915 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.915 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.915 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.916 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption7": { 2024-04-25T00:26:07.916 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption7", 2024-04-25T00:26:07.916 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.916 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.916 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.916 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.916 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T00:26:07.916 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "42", 2024-04-25T00:26:07.916 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.916 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.916 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.916 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.916 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.917 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.917 INFO:teuthology.orchestra.run.smithi027.stdout: "testkey": { 2024-04-25T00:26:07.917 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testkey", 2024-04-25T00:26:07.917 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.917 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.917 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.917 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.917 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.917 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.917 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.917 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.917 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.918 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.918 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.918 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.918 INFO:teuthology.orchestra.run.smithi027.stdout: "testlkey": { 2024-04-25T00:26:07.918 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testlkey", 2024-04-25T00:26:07.918 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.918 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.918 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.918 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.918 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.918 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.918 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.919 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.919 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.919 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.919 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.919 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.919 INFO:teuthology.orchestra.run.smithi027.stdout: "testnewline": { 2024-04-25T00:26:07.919 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testnewline", 2024-04-25T00:26:07.919 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.919 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.919 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.919 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.919 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.919 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.920 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.920 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.920 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.920 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.920 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.920 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.920 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.920 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.920 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.920 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-04-25T00:26:07.920 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.920 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.921 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.921 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_m_granularity": { 2024-04-25T00:26:07.921 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_m_granularity", 2024-04-25T00:26:07.921 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.921 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.921 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.921 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.921 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.921 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.921 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.921 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T00:26:07.921 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.922 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.922 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.922 INFO:teuthology.orchestra.run.smithi027.stdout: "dump_on_update": { 2024-04-25T00:26:07.922 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dump_on_update", 2024-04-25T00:26:07.922 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.922 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.922 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.922 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.922 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.922 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.922 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.923 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "dump database to debug log on update", 2024-04-25T00:26:07.923 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.923 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.923 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.923 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.923 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.923 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.923 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.923 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.923 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.923 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.924 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.924 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.924 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.924 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.924 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.924 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.924 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.924 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.924 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.924 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.924 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.924 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.924 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.925 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.925 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.925 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.925 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.925 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.925 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.925 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.925 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.925 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.925 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.925 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.925 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.926 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.926 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.926 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.926 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.926 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.926 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.926 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.926 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.926 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.926 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.927 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.927 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.927 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.927 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.927 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.927 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.927 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.927 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.927 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.927 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.927 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.927 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.928 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.928 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.928 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.928 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.928 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.928 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.928 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.928 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.928 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.928 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.928 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.928 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.928 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.929 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.929 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.929 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.929 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.929 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stats", 2024-04-25T00:26:07.929 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.929 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.929 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.929 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.929 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.929 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.929 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.930 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.930 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.930 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.930 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.930 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.930 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.930 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.930 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.930 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.930 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.930 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.930 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.931 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.931 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.931 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.931 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.931 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.931 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.931 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.931 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.931 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.931 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.931 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.931 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.932 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.932 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.932 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.932 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.932 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.932 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.932 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.932 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.932 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.932 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.932 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.932 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.933 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.933 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.933 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.933 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.933 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.933 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.933 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.933 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.933 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.933 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.933 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.933 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.933 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.934 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.934 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.934 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.934 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.934 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.934 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.934 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.934 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.934 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.934 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.934 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.935 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.935 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.935 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.935 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.935 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.935 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.935 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.935 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.935 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "status", 2024-04-25T00:26:07.935 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.935 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.935 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.936 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.936 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.936 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.936 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.936 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.936 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.936 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.936 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.936 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.936 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.936 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.936 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.937 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.937 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.937 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.937 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.937 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.937 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.937 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.937 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.937 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.937 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.937 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.937 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.937 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.938 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.938 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.938 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.938 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.938 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.938 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.938 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.938 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.938 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.938 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.939 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.939 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.939 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.939 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.939 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.939 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.939 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.939 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.939 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.939 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.939 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.939 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.939 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.940 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.940 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.940 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.940 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.940 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.940 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.940 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.940 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.940 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.940 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.940 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.940 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.941 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.941 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.941 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.941 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.941 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.941 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.941 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.941 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.941 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.941 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.941 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telegraf", 2024-04-25T00:26:07.942 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.942 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.942 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.942 INFO:teuthology.orchestra.run.smithi027.stdout: "address": { 2024-04-25T00:26:07.942 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "address", 2024-04-25T00:26:07.942 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.942 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.942 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.942 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T00:26:07.942 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.942 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.942 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.943 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.943 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.943 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.943 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.943 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T00:26:07.943 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T00:26:07.943 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.943 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.943 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.943 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15", 2024-04-25T00:26:07.943 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.944 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.944 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.944 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.944 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.944 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.944 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.944 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.944 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.944 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.944 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.944 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.944 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.945 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.945 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.945 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.945 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.945 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.945 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.945 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.945 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.945 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.945 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.945 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.945 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.945 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.946 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.946 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.946 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.946 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.946 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.946 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.946 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.946 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.946 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.946 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.946 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.947 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.947 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.947 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.947 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.947 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.947 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.947 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.947 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.947 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.947 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.947 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.948 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.948 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.948 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.948 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.948 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.948 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.948 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.948 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.948 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.948 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.948 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.948 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.949 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.949 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.949 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.949 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.949 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.949 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.949 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.949 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.949 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.949 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.950 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.950 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.950 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.950 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.950 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.950 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telemetry", 2024-04-25T00:26:07.950 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.950 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.950 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.950 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_basic": { 2024-04-25T00:26:07.951 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_basic", 2024-04-25T00:26:07.951 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.951 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.951 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.951 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.951 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.951 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.951 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.951 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T00:26:07.951 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.951 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.951 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.952 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_crash": { 2024-04-25T00:26:07.952 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_crash", 2024-04-25T00:26:07.952 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.952 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.952 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.952 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.952 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.952 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.952 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.952 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T00:26:07.952 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.953 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.953 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.953 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_device": { 2024-04-25T00:26:07.953 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_device", 2024-04-25T00:26:07.953 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.953 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.953 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.953 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.953 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.953 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.953 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.954 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T00:26:07.954 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.954 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.954 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.954 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_ident": { 2024-04-25T00:26:07.954 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_ident", 2024-04-25T00:26:07.954 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.954 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.954 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.954 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.954 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.955 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.955 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.955 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T00:26:07.955 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.955 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.955 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.955 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_perf": { 2024-04-25T00:26:07.955 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_perf", 2024-04-25T00:26:07.955 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.955 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.955 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.956 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.956 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.956 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.956 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.956 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T00:26:07.956 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.956 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.956 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.956 INFO:teuthology.orchestra.run.smithi027.stdout: "contact": { 2024-04-25T00:26:07.956 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "contact", 2024-04-25T00:26:07.956 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.956 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.957 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.957 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.957 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.957 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.957 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.957 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.957 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.957 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.957 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.957 INFO:teuthology.orchestra.run.smithi027.stdout: "description": { 2024-04-25T00:26:07.957 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "description", 2024-04-25T00:26:07.957 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.957 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.958 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.958 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.958 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.958 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.958 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.958 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.958 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.958 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.958 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.958 INFO:teuthology.orchestra.run.smithi027.stdout: "device_url": { 2024-04-25T00:26:07.958 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_url", 2024-04-25T00:26:07.958 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.959 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.959 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.959 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T00:26:07.959 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.959 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.959 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.959 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.959 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.959 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.959 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.960 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T00:26:07.960 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T00:26:07.960 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.960 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.960 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.960 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.960 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.960 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.960 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.960 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.960 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.960 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.960 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.961 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T00:26:07.961 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T00:26:07.961 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.961 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.961 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.961 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "24", 2024-04-25T00:26:07.961 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "8", 2024-04-25T00:26:07.961 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.961 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.961 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.961 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.962 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.962 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.962 INFO:teuthology.orchestra.run.smithi027.stdout: "last_opt_revision": { 2024-04-25T00:26:07.962 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "last_opt_revision", 2024-04-25T00:26:07.962 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.962 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.962 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.962 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1", 2024-04-25T00:26:07.962 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.962 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.962 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.963 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.963 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.963 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.963 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.963 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard": { 2024-04-25T00:26:07.963 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard", 2024-04-25T00:26:07.963 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.963 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.963 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.963 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.963 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.964 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.964 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.964 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.964 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.964 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.964 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.964 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard_description": { 2024-04-25T00:26:07.964 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard_description", 2024-04-25T00:26:07.964 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.964 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.964 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.965 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.965 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.965 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.965 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.965 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.965 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.965 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.965 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.965 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.965 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.965 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.966 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.966 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.966 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.966 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.966 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.966 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.966 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.966 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.966 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.966 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.966 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.966 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.966 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.967 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.967 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.967 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.967 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.967 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.967 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.967 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.967 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.967 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.967 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.967 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.967 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.968 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.968 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.968 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.968 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.968 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.968 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.968 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.968 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.968 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.968 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.968 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.969 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.969 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.969 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.969 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.969 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.969 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.969 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.969 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.969 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.969 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.969 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.969 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.969 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.970 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.970 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.970 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.970 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.970 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.970 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.970 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.970 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.970 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.970 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.970 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.971 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.971 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.971 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.971 INFO:teuthology.orchestra.run.smithi027.stdout: "organization": { 2024-04-25T00:26:07.971 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "organization", 2024-04-25T00:26:07.971 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.971 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.971 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.971 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.971 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.971 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.971 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.972 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.972 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.972 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.972 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.972 INFO:teuthology.orchestra.run.smithi027.stdout: "proxy": { 2024-04-25T00:26:07.972 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "proxy", 2024-04-25T00:26:07.972 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.972 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.972 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.972 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.972 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.972 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.973 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.973 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.973 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.973 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.973 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.973 INFO:teuthology.orchestra.run.smithi027.stdout: "url": { 2024-04-25T00:26:07.973 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url", 2024-04-25T00:26:07.973 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.973 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.973 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.973 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T00:26:07.974 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.974 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.974 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.974 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.974 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.974 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.974 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.974 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.974 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.974 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "test_orchestrator", 2024-04-25T00:26:07.975 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.975 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.975 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.975 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.975 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.975 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.975 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.975 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.975 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.975 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.975 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.975 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.975 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.976 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.976 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.976 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.976 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.976 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.976 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.976 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.976 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.976 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.976 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.976 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.976 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.977 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.977 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.977 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.977 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.977 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.977 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.977 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.977 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.977 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.977 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.977 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.977 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.978 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.978 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.978 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.978 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.978 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.978 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.978 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.978 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.978 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.978 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.978 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.978 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.979 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.979 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.979 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.979 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.979 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.979 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.979 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.979 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.979 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.979 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.979 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.980 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.980 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.980 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.980 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.980 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.980 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.980 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.980 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.980 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.980 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.980 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.981 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "volumes", 2024-04-25T00:26:07.981 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.981 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.981 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.981 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.981 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.981 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.981 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.981 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.981 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.981 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.981 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.982 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.982 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.982 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.982 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.982 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.982 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.982 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.982 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.982 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.982 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.982 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.982 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.983 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.983 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.983 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.983 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.983 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.983 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.983 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.983 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.983 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.983 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.983 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.983 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.984 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.984 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.984 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.984 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.984 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.984 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.984 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.984 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.984 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.984 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.984 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.984 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.985 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.985 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.985 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.985 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.985 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.985 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.985 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.985 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.985 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.985 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:07.985 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:07.985 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.986 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.986 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.986 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.986 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.986 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.986 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.986 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.986 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.986 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.986 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.986 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_clones": { 2024-04-25T00:26:07.987 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_clones", 2024-04-25T00:26:07.987 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.987 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.987 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.987 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4", 2024-04-25T00:26:07.987 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.987 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.987 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.987 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T00:26:07.987 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.987 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.988 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.988 INFO:teuthology.orchestra.run.smithi027.stdout: "periodic_async_work": { 2024-04-25T00:26:07.988 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "periodic_async_work", 2024-04-25T00:26:07.988 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.988 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.988 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.988 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.988 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.988 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.988 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.988 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Periodically check for async work", 2024-04-25T00:26:07.989 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.989 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.989 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.989 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_delay": { 2024-04-25T00:26:07.989 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_delay", 2024-04-25T00:26:07.989 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:07.989 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.989 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.989 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T00:26:07.989 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.989 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.989 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.990 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T00:26:07.990 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.990 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.990 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.990 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_no_wait": { 2024-04-25T00:26:07.990 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T00:26:07.990 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.990 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.990 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.990 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T00:26:07.991 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.991 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.991 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.991 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T00:26:07.991 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.991 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.991 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.991 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.991 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:07.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:07.991 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix", 2024-04-25T00:26:07.991 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T00:26:07.992 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T00:26:07.992 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T00:26:07.992 INFO:teuthology.orchestra.run.smithi027.stdout: "discovery_interval": { 2024-04-25T00:26:07.992 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "discovery_interval", 2024-04-25T00:26:07.992 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T00:26:07.992 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.992 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.992 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "100", 2024-04-25T00:26:07.992 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.992 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.992 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.992 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.992 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.992 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.993 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.993 INFO:teuthology.orchestra.run.smithi027.stdout: "identifier": { 2024-04-25T00:26:07.993 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "identifier", 2024-04-25T00:26:07.993 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.993 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.993 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.993 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.993 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.993 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.993 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.994 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.994 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.994 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.994 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.994 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T00:26:07.994 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T00:26:07.994 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T00:26:07.994 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.994 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:07.994 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T00:26:07.994 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.995 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.995 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.995 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.995 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.995 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.995 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.995 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T00:26:07.995 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T00:26:07.995 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.995 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.995 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.996 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:07.996 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.996 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.996 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.996 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.996 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.996 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.996 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.996 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.996 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.996 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.996 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.997 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.997 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.997 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.997 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T00:26:07.997 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T00:26:07.997 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:07.997 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.997 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.997 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:07.997 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.997 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.998 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:07.998 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.998 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.998 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.998 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:07.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:07.998 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T00:26:07.998 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T00:26:07.998 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:07.998 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:07.998 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:07.998 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T00:26:07.998 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:07.999 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:07.999 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T00:26:07.999 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T00:26:07.999 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T00:26:07.999 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T00:26:07.999 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T00:26:07.999 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T00:26:07.999 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T00:26:07.999 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:07.999 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:07.999 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:07.999 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:07.999 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:08.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:08.000 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T00:26:08.000 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T00:26:08.000 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T00:26:08.000 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:08.000 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T00:26:08.000 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T00:26:08.000 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:08.000 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:08.000 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:08.000 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:08.000 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:08.000 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:08.001 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:08.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:08.001 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_host": { 2024-04-25T00:26:08.001 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_host", 2024-04-25T00:26:08.001 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:08.001 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:08.001 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:08.001 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T00:26:08.001 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:08.001 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:08.001 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:08.001 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:08.001 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:08.002 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:08.002 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:08.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:08.002 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_port": { 2024-04-25T00:26:08.002 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_port", 2024-04-25T00:26:08.002 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T00:26:08.002 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:08.002 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:08.002 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10051", 2024-04-25T00:26:08.002 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:08.002 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:08.002 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:08.003 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:08.003 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:08.003 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:08.003 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:08.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:08.003 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_sender": { 2024-04-25T00:26:08.003 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_sender", 2024-04-25T00:26:08.003 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T00:26:08.003 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T00:26:08.003 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T00:26:08.003 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T00:26:08.003 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T00:26:08.004 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T00:26:08.004 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T00:26:08.004 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T00:26:08.004 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T00:26:08.004 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T00:26:08.004 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T00:26:08.004 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:08.004 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:08.004 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:08.004 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:08.004 INFO:teuthology.orchestra.run.smithi027.stdout: "services": { 2024-04-25T00:26:08.004 INFO:teuthology.orchestra.run.smithi027.stdout: "dashboard": "https://172.21.15.27:8443/" 2024-04-25T00:26:08.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:08.005 INFO:teuthology.orchestra.run.smithi027.stdout: "always_on_modules": { 2024-04-25T00:26:08.005 INFO:teuthology.orchestra.run.smithi027.stdout: "octopus": [ 2024-04-25T00:26:08.005 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-04-25T00:26:08.005 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-25T00:26:08.005 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-04-25T00:26:08.005 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-04-25T00:26:08.005 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-04-25T00:26:08.005 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-04-25T00:26:08.005 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-04-25T00:26:08.005 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-04-25T00:26:08.005 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-04-25T00:26:08.006 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-04-25T00:26:08.006 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:08.006 INFO:teuthology.orchestra.run.smithi027.stdout: "pacific": [ 2024-04-25T00:26:08.006 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-04-25T00:26:08.006 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-25T00:26:08.006 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-04-25T00:26:08.006 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-04-25T00:26:08.006 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-04-25T00:26:08.006 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-04-25T00:26:08.006 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-04-25T00:26:08.006 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-04-25T00:26:08.006 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-04-25T00:26:08.007 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-04-25T00:26:08.007 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:08.007 INFO:teuthology.orchestra.run.smithi027.stdout: "quincy": [ 2024-04-25T00:26:08.007 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-04-25T00:26:08.007 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-25T00:26:08.007 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-04-25T00:26:08.007 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-04-25T00:26:08.007 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-04-25T00:26:08.007 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-04-25T00:26:08.007 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-04-25T00:26:08.007 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-04-25T00:26:08.007 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-04-25T00:26:08.007 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-04-25T00:26:08.008 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:08.008 INFO:teuthology.orchestra.run.smithi027.stdout: "reef": [ 2024-04-25T00:26:08.008 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-04-25T00:26:08.008 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-25T00:26:08.008 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-04-25T00:26:08.008 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-04-25T00:26:08.008 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-04-25T00:26:08.008 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-04-25T00:26:08.008 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-04-25T00:26:08.008 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-04-25T00:26:08.008 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-04-25T00:26:08.008 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-04-25T00:26:08.009 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T00:26:08.009 INFO:teuthology.orchestra.run.smithi027.stdout: "squid": [ 2024-04-25T00:26:08.009 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-04-25T00:26:08.009 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-25T00:26:08.009 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-04-25T00:26:08.009 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-04-25T00:26:08.009 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-04-25T00:26:08.009 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-04-25T00:26:08.009 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-04-25T00:26:08.009 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-04-25T00:26:08.009 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-04-25T00:26:08.010 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-04-25T00:26:08.010 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T00:26:08.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:08.010 INFO:teuthology.orchestra.run.smithi027.stdout: "last_failure_osd_epoch": 94, 2024-04-25T00:26:08.010 INFO:teuthology.orchestra.run.smithi027.stdout: "active_clients": [ 2024-04-25T00:26:08.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:08.010 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "devicehealth", 2024-04-25T00:26:08.010 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T00:26:08.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:08.010 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T00:26:08.010 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-04-25T00:26:08.010 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3371282088 2024-04-25T00:26:08.011 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:08.011 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T00:26:08.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:08.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:08.011 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "libcephsqlite", 2024-04-25T00:26:08.011 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T00:26:08.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:08.011 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T00:26:08.011 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-04-25T00:26:08.011 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4076884317 2024-04-25T00:26:08.011 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:08.011 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T00:26:08.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:08.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:08.012 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_support", 2024-04-25T00:26:08.012 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T00:26:08.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:08.012 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T00:26:08.012 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-04-25T00:26:08.012 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3893477778 2024-04-25T00:26:08.012 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:08.012 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T00:26:08.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:08.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:08.012 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-04-25T00:26:08.013 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T00:26:08.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:08.013 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T00:26:08.013 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-04-25T00:26:08.013 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2089487408 2024-04-25T00:26:08.013 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:08.013 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T00:26:08.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:08.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:08.013 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-04-25T00:26:08.013 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T00:26:08.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:08.013 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T00:26:08.014 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-04-25T00:26:08.014 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3801544940 2024-04-25T00:26:08.014 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:08.014 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T00:26:08.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:08.014 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:08.014 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "volumes", 2024-04-25T00:26:08.014 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T00:26:08.014 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T00:26:08.014 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T00:26:08.014 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-04-25T00:26:08.014 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2973653738 2024-04-25T00:26:08.015 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:08.015 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T00:26:08.015 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:08.015 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T00:26:08.015 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:26:08.016 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:06 smithi153 ceph-mon[32896]: pgmap v313: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-04-25T00:26:08.017 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:06 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/259351867' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:26:08.017 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:06 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3795245309' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T00:26:08.017 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:06 smithi139 ceph-mon[32833]: pgmap v313: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-04-25T00:26:08.017 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:06 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/259351867' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:26:08.017 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:06 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3795245309' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T00:26:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.547+0000 7fd34b25d640 1 -- 172.21.15.27:0/3795245309 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd3300a2d00 msgr2=0x7fd3300a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.547+0000 7fd34b25d640 1 --2- 172.21.15.27:0/3795245309 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd3300a2d00 0x7fd3300a51c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd32c000c00 tx=0x7fd32c005a20 comp rx=0 tx=0).stop 2024-04-25T00:26:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.547+0000 7fd34b25d640 1 -- 172.21.15.27:0/3795245309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3441020d0 msgr2=0x7fd344198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.547+0000 7fd34b25d640 1 --2- 172.21.15.27:0/3795245309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3441020d0 0x7fd344198570 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fd33400c970 tx=0x7fd33400ce30 comp rx=0 tx=0).stop 2024-04-25T00:26:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.547+0000 7fd34b25d640 1 -- 172.21.15.27:0/3795245309 shutdown_connections 2024-04-25T00:26:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.547+0000 7fd34b25d640 1 --2- 172.21.15.27:0/3795245309 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fd3300a2d00 0x7fd3300a51c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.547+0000 7fd34b25d640 1 --2- 172.21.15.27:0/3795245309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd344107720 0x7fd34419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.547+0000 7fd34b25d640 1 --2- 172.21.15.27:0/3795245309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3441020d0 0x7fd344198570 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:08.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.547+0000 7fd34b25d640 1 --2- 172.21.15.27:0/3795245309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd344101710 0x7fd344198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:08.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.547+0000 7fd34b25d640 1 -- 172.21.15.27:0/3795245309 >> 172.21.15.27:0/3795245309 conn(0x7fd344078b00 msgr2=0x7fd344104160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:08.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.547+0000 7fd34b25d640 1 -- 172.21.15.27:0/3795245309 shutdown_connections 2024-04-25T00:26:08.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:06.547+0000 7fd34b25d640 1 -- 172.21.15.27:0/3795245309 wait complete. 2024-04-25T00:26:08.025 DEBUG:tasks.check_counter:Getting stats from a 2024-04-25T00:26:08.025 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-25T00:26:08.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:06 smithi027 ceph-mon[29208]: pgmap v313: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-04-25T00:26:08.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:06 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/259351867' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:26:08.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:06 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3795245309' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T00:26:08.345 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:26:08.989 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:08 smithi027 ceph-mon[29208]: pgmap v314: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-04-25T00:26:09.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:08 smithi139 ceph-mon[32833]: pgmap v314: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-04-25T00:26:09.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:08 smithi153 ceph-mon[32896]: pgmap v314: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-04-25T00:26:10.075 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:09 smithi027 ceph-mon[29208]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T00:26:10.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:09 smithi139 ceph-mon[32833]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T00:26:10.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:09 smithi153 ceph-mon[32896]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T00:26:10.687 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:26:10.687 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T00:26:10.687 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 1051, 2024-04-25T00:26:10.687 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 965, 2024-04-25T00:26:10.687 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 2345136, 2024-04-25T00:26:10.687 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 367318, 2024-04-25T00:26:10.687 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 5, 2024-04-25T00:26:10.687 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 1, 2024-04-25T00:26:10.687 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.136452232, 2024-04-25T00:26:10.687 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.040525037, 2024-04-25T00:26:10.687 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.066343238, 2024-04-25T00:26:10.687 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.032371406, 2024-04-25T00:26:10.688 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:10.688 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 965, 2024-04-25T00:26:10.688 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.059077434, 2024-04-25T00:26:10.688 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000061220 2024-04-25T00:26:10.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.688 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:10.688 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.688 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.688 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.688 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 2096160, 2024-04-25T00:26:10.688 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 131680 2024-04-25T00:26:10.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.688 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T00:26:10.689 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 628, 2024-04-25T00:26:10.689 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 786, 2024-04-25T00:26:10.689 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 154968, 2024-04-25T00:26:10.689 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 638816, 2024-04-25T00:26:10.689 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 7, 2024-04-25T00:26:10.689 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 5, 2024-04-25T00:26:10.689 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.073557228, 2024-04-25T00:26:10.689 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.023850722, 2024-04-25T00:26:10.689 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.016255688, 2024-04-25T00:26:10.689 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.025201132, 2024-04-25T00:26:10.689 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:10.689 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 783, 2024-04-25T00:26:10.689 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.052396936, 2024-04-25T00:26:10.689 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000066918 2024-04-25T00:26:10.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.690 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:10.690 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.690 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.690 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.690 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T00:26:10.690 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 439680 2024-04-25T00:26:10.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.690 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T00:26:10.690 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 754, 2024-04-25T00:26:10.690 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 756, 2024-04-25T00:26:10.690 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 233543, 2024-04-25T00:26:10.691 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 239773, 2024-04-25T00:26:10.691 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 4, 2024-04-25T00:26:10.691 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 3, 2024-04-25T00:26:10.691 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.068765503, 2024-04-25T00:26:10.691 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.021132336, 2024-04-25T00:26:10.691 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.018122796, 2024-04-25T00:26:10.691 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.021882177, 2024-04-25T00:26:10.691 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:10.691 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 754, 2024-04-25T00:26:10.691 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.032037247, 2024-04-25T00:26:10.691 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000042489 2024-04-25T00:26:10.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.691 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:10.691 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.692 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.692 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.692 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T00:26:10.692 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T00:26:10.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.692 INFO:teuthology.orchestra.run.smithi027.stdout: "cct": { 2024-04-25T00:26:10.692 INFO:teuthology.orchestra.run.smithi027.stdout: "total_workers": 1, 2024-04-25T00:26:10.692 INFO:teuthology.orchestra.run.smithi027.stdout: "unhealthy_workers": 0 2024-04-25T00:26:10.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.692 INFO:teuthology.orchestra.run.smithi027.stdout: "finisher-MDSRank": { 2024-04-25T00:26:10.692 INFO:teuthology.orchestra.run.smithi027.stdout: "queue_len": 0, 2024-04-25T00:26:10.692 INFO:teuthology.orchestra.run.smithi027.stdout: "complete_latency": { 2024-04-25T00:26:10.692 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 3447, 2024-04-25T00:26:10.693 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.126285359, 2024-04-25T00:26:10.693 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000036636 2024-04-25T00:26:10.693 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:10.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.693 INFO:teuthology.orchestra.run.smithi027.stdout: "finisher-PurgeQueue": { 2024-04-25T00:26:10.693 INFO:teuthology.orchestra.run.smithi027.stdout: "queue_len": 0, 2024-04-25T00:26:10.693 INFO:teuthology.orchestra.run.smithi027.stdout: "complete_latency": { 2024-04-25T00:26:10.693 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.693 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.693 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.693 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:10.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.693 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-25T00:26:10.693 INFO:teuthology.orchestra.run.smithi027.stdout: "request": 0, 2024-04-25T00:26:10.694 INFO:teuthology.orchestra.run.smithi027.stdout: "reply": 0, 2024-04-25T00:26:10.694 INFO:teuthology.orchestra.run.smithi027.stdout: "reply_latency": { 2024-04-25T00:26:10.694 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.694 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.694 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.694 INFO:teuthology.orchestra.run.smithi027.stdout: "slow_reply": 0, 2024-04-25T00:26:10.694 INFO:teuthology.orchestra.run.smithi027.stdout: "forward": 0, 2024-04-25T00:26:10.694 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_fetch_complete": 0, 2024-04-25T00:26:10.694 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_fetch_keys": 0, 2024-04-25T00:26:10.694 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_commit": 0, 2024-04-25T00:26:10.694 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_split": 0, 2024-04-25T00:26:10.694 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_merge": 0, 2024-04-25T00:26:10.695 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes": 3, 2024-04-25T00:26:10.695 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_top": 0, 2024-04-25T00:26:10.695 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_bottom": 3, 2024-04-25T00:26:10.695 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_pin_tail": 0, 2024-04-25T00:26:10.695 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_pinned": 3, 2024-04-25T00:26:10.695 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_expired": 0, 2024-04-25T00:26:10.695 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_with_caps": 0, 2024-04-25T00:26:10.695 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": 0, 2024-04-25T00:26:10.695 INFO:teuthology.orchestra.run.smithi027.stdout: "subtrees": 1, 2024-04-25T00:26:10.695 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse": 0, 2024-04-25T00:26:10.695 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_hit": 0, 2024-04-25T00:26:10.695 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_forward": 0, 2024-04-25T00:26:10.695 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_discover": 0, 2024-04-25T00:26:10.696 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_dir_fetch": 0, 2024-04-25T00:26:10.696 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_remote_ino": 0, 2024-04-25T00:26:10.696 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_lock": 0, 2024-04-25T00:26:10.696 INFO:teuthology.orchestra.run.smithi027.stdout: "load_cent": 0, 2024-04-25T00:26:10.696 INFO:teuthology.orchestra.run.smithi027.stdout: "q": 0, 2024-04-25T00:26:10.696 INFO:teuthology.orchestra.run.smithi027.stdout: "exported": 0, 2024-04-25T00:26:10.696 INFO:teuthology.orchestra.run.smithi027.stdout: "exported_inodes": 0, 2024-04-25T00:26:10.696 INFO:teuthology.orchestra.run.smithi027.stdout: "imported": 0, 2024-04-25T00:26:10.696 INFO:teuthology.orchestra.run.smithi027.stdout: "imported_inodes": 0, 2024-04-25T00:26:10.696 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_dir_fetch": 0, 2024-04-25T00:26:10.696 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_backtrace_fetch": 0, 2024-04-25T00:26:10.696 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_peer_discover": 0, 2024-04-25T00:26:10.696 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rfiles": 0, 2024-04-25T00:26:10.696 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rbytes": 0, 2024-04-25T00:26:10.697 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rsnaps": 0, 2024-04-25T00:26:10.697 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T00:26:10.697 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_set_tag": 0, 2024-04-25T00:26:10.697 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T00:26:10.697 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_inotable_repaired": 0, 2024-04-25T00:26:10.697 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dir_inodes": 0, 2024-04-25T00:26:10.697 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T00:26:10.697 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T00:26:10.697 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_file_inodes": 0, 2024-04-25T00:26:10.697 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_inode_file_caps": 0, 2024-04-25T00:26:10.697 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T00:26:10.697 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_grant": 0, 2024-04-25T00:26:10.697 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T00:26:10.697 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T00:26:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T00:26:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_caps": 0, 2024-04-25T00:26:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_caps_dirty": 0, 2024-04-25T00:26:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_cap_release": 0, 2024-04-25T00:26:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: "process_request_cap_release": 0 2024-04-25T00:26:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache": { 2024-04-25T00:26:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_update": 0, 2024-04-25T00:26:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_update_receipt": 0, 2024-04-25T00:26:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_try_discover": 0, 2024-04-25T00:26:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_send_discover": 0, 2024-04-25T00:26:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_handle_discover": 0, 2024-04-25T00:26:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays": 0, 2024-04-25T00:26:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays_delayed": 0, 2024-04-25T00:26:10.699 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays_enqueuing": 0, 2024-04-25T00:26:10.699 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_created": 0, 2024-04-25T00:26:10.699 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_enqueued": 0, 2024-04-25T00:26:10.699 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_reintegrated": 0, 2024-04-25T00:26:10.699 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_migrated": 0, 2024-04-25T00:26:10.699 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_processing": 0, 2024-04-25T00:26:10.699 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_enqueued": 0, 2024-04-25T00:26:10.699 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_prioritized": 0, 2024-04-25T00:26:10.699 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_started": 0, 2024-04-25T00:26:10.699 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_completed": 0, 2024-04-25T00:26:10.699 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_quiesce_path": 0, 2024-04-25T00:26:10.699 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_quiesce_inode": 0, 2024-04-25T00:26:10.699 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T00:26:10.699 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_exportdir": 0, 2024-04-25T00:26:10.699 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_flush": 0, 2024-04-25T00:26:10.700 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_fragmentdir": 0, 2024-04-25T00:26:10.700 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_fragstats": 0, 2024-04-25T00:26:10.700 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_inodestats": 0 2024-04-25T00:26:10.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.700 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log": { 2024-04-25T00:26:10.700 INFO:teuthology.orchestra.run.smithi027.stdout: "evlrg": 0, 2024-04-25T00:26:10.700 INFO:teuthology.orchestra.run.smithi027.stdout: "evadd": 0, 2024-04-25T00:26:10.700 INFO:teuthology.orchestra.run.smithi027.stdout: "evex": 0, 2024-04-25T00:26:10.700 INFO:teuthology.orchestra.run.smithi027.stdout: "evtrm": 0, 2024-04-25T00:26:10.700 INFO:teuthology.orchestra.run.smithi027.stdout: "ev": 21, 2024-04-25T00:26:10.700 INFO:teuthology.orchestra.run.smithi027.stdout: "evexg": 0, 2024-04-25T00:26:10.700 INFO:teuthology.orchestra.run.smithi027.stdout: "evexd": 0, 2024-04-25T00:26:10.700 INFO:teuthology.orchestra.run.smithi027.stdout: "segadd": 0, 2024-04-25T00:26:10.700 INFO:teuthology.orchestra.run.smithi027.stdout: "segex": 0, 2024-04-25T00:26:10.700 INFO:teuthology.orchestra.run.smithi027.stdout: "segtrm": 0, 2024-04-25T00:26:10.700 INFO:teuthology.orchestra.run.smithi027.stdout: "seg": 2, 2024-04-25T00:26:10.701 INFO:teuthology.orchestra.run.smithi027.stdout: "segmjr": 2, 2024-04-25T00:26:10.701 INFO:teuthology.orchestra.run.smithi027.stdout: "segexg": 0, 2024-04-25T00:26:10.701 INFO:teuthology.orchestra.run.smithi027.stdout: "segexd": 0, 2024-04-25T00:26:10.701 INFO:teuthology.orchestra.run.smithi027.stdout: "expos": 4194304, 2024-04-25T00:26:10.701 INFO:teuthology.orchestra.run.smithi027.stdout: "wrpos": 4230450, 2024-04-25T00:26:10.701 INFO:teuthology.orchestra.run.smithi027.stdout: "rdpos": 4227595, 2024-04-25T00:26:10.701 INFO:teuthology.orchestra.run.smithi027.stdout: "jlat": { 2024-04-25T00:26:10.701 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.701 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.701 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.701 INFO:teuthology.orchestra.run.smithi027.stdout: "replayed": 21 2024-04-25T00:26:10.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.701 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_mem": { 2024-04-25T00:26:10.701 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 4, 2024-04-25T00:26:10.701 INFO:teuthology.orchestra.run.smithi027.stdout: "ino+": 20, 2024-04-25T00:26:10.702 INFO:teuthology.orchestra.run.smithi027.stdout: "ino-": 16, 2024-04-25T00:26:10.702 INFO:teuthology.orchestra.run.smithi027.stdout: "dir": 2, 2024-04-25T00:26:10.702 INFO:teuthology.orchestra.run.smithi027.stdout: "dir+": 15, 2024-04-25T00:26:10.702 INFO:teuthology.orchestra.run.smithi027.stdout: "dir-": 13, 2024-04-25T00:26:10.702 INFO:teuthology.orchestra.run.smithi027.stdout: "dn": 3, 2024-04-25T00:26:10.702 INFO:teuthology.orchestra.run.smithi027.stdout: "dn+": 21, 2024-04-25T00:26:10.702 INFO:teuthology.orchestra.run.smithi027.stdout: "dn-": 18, 2024-04-25T00:26:10.702 INFO:teuthology.orchestra.run.smithi027.stdout: "cap": 0, 2024-04-25T00:26:10.702 INFO:teuthology.orchestra.run.smithi027.stdout: "cap+": 0, 2024-04-25T00:26:10.702 INFO:teuthology.orchestra.run.smithi027.stdout: "cap-": 0, 2024-04-25T00:26:10.702 INFO:teuthology.orchestra.run.smithi027.stdout: "rss": 53448, 2024-04-25T00:26:10.702 INFO:teuthology.orchestra.run.smithi027.stdout: "heap": 215420 2024-04-25T00:26:10.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.702 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_server": { 2024-04-25T00:26:10.703 INFO:teuthology.orchestra.run.smithi027.stdout: "dispatch_client_request": 0, 2024-04-25T00:26:10.703 INFO:teuthology.orchestra.run.smithi027.stdout: "dispatch_server_request": 0, 2024-04-25T00:26:10.703 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_request": 0, 2024-04-25T00:26:10.703 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_session": 0, 2024-04-25T00:26:10.703 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_peer_request": 0, 2024-04-25T00:26:10.703 INFO:teuthology.orchestra.run.smithi027.stdout: "req_create_latency": { 2024-04-25T00:26:10.703 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.703 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.703 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.703 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getattr_latency": { 2024-04-25T00:26:10.703 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.703 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.703 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getfilelock_latency": { 2024-04-25T00:26:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "req_link_latency": { 2024-04-25T00:26:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookup_latency": { 2024-04-25T00:26:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.704 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.705 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.705 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookuphash_latency": { 2024-04-25T00:26:10.705 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.705 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.705 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.705 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupino_latency": { 2024-04-25T00:26:10.705 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.705 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.705 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.705 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupname_latency": { 2024-04-25T00:26:10.705 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.706 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.706 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.706 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupparent_latency": { 2024-04-25T00:26:10.706 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.706 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.706 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.706 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupsnap_latency": { 2024-04-25T00:26:10.706 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.706 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.706 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.706 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lssnap_latency": { 2024-04-25T00:26:10.707 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.707 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.707 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.707 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mkdir_latency": { 2024-04-25T00:26:10.707 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.707 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.708 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.708 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mknod_latency": { 2024-04-25T00:26:10.708 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.708 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.708 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.708 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mksnap_latency": { 2024-04-25T00:26:10.708 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.708 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.708 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.708 INFO:teuthology.orchestra.run.smithi027.stdout: "req_open_latency": { 2024-04-25T00:26:10.709 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.709 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.709 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.709 INFO:teuthology.orchestra.run.smithi027.stdout: "req_readdir_latency": { 2024-04-25T00:26:10.709 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.709 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.709 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.709 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rename_latency": { 2024-04-25T00:26:10.709 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.709 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.709 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.710 INFO:teuthology.orchestra.run.smithi027.stdout: "req_renamesnap_latency": { 2024-04-25T00:26:10.710 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.710 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.710 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.710 INFO:teuthology.orchestra.run.smithi027.stdout: "req_snapdiff_latency": { 2024-04-25T00:26:10.710 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.710 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.710 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.710 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmdir_latency": { 2024-04-25T00:26:10.710 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.710 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.711 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.711 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmsnap_latency": { 2024-04-25T00:26:10.711 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.711 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.711 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.711 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmxattr_latency": { 2024-04-25T00:26:10.711 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.711 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.711 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.711 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setattr_latency": { 2024-04-25T00:26:10.711 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.711 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.712 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.712 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setdirlayout_latency": { 2024-04-25T00:26:10.712 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.712 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.712 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.712 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setfilelock_latency": { 2024-04-25T00:26:10.712 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.712 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.712 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.712 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setlayout_latency": { 2024-04-25T00:26:10.712 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.713 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.713 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.713 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setxattr_latency": { 2024-04-25T00:26:10.713 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.713 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.713 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.713 INFO:teuthology.orchestra.run.smithi027.stdout: "req_symlink_latency": { 2024-04-25T00:26:10.713 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.713 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.713 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.713 INFO:teuthology.orchestra.run.smithi027.stdout: "req_unlink_latency": { 2024-04-25T00:26:10.714 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.714 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.714 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.714 INFO:teuthology.orchestra.run.smithi027.stdout: "cap_revoke_eviction": 0, 2024-04-25T00:26:10.714 INFO:teuthology.orchestra.run.smithi027.stdout: "cap_acquisition_throttle": 0, 2024-04-25T00:26:10.714 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getvxattr_latency": { 2024-04-25T00:26:10.714 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.714 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.714 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.714 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:10.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.714 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_sessions": { 2024-04-25T00:26:10.715 INFO:teuthology.orchestra.run.smithi027.stdout: "session_count": 1, 2024-04-25T00:26:10.715 INFO:teuthology.orchestra.run.smithi027.stdout: "session_add": 4, 2024-04-25T00:26:10.715 INFO:teuthology.orchestra.run.smithi027.stdout: "session_remove": 3, 2024-04-25T00:26:10.715 INFO:teuthology.orchestra.run.smithi027.stdout: "sessions_open": 4, 2024-04-25T00:26:10.715 INFO:teuthology.orchestra.run.smithi027.stdout: "sessions_stale": 0, 2024-04-25T00:26:10.715 INFO:teuthology.orchestra.run.smithi027.stdout: "total_load": 0, 2024-04-25T00:26:10.715 INFO:teuthology.orchestra.run.smithi027.stdout: "average_load": 0, 2024-04-25T00:26:10.715 INFO:teuthology.orchestra.run.smithi027.stdout: "avg_session_uptime": 5353, 2024-04-25T00:26:10.715 INFO:teuthology.orchestra.run.smithi027.stdout: "mdthresh_evicted": 0 2024-04-25T00:26:10.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.715 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool": { 2024-04-25T00:26:10.715 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_bytes": 0, 2024-04-25T00:26:10.715 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_items": 0, 2024-04-25T00:26:10.715 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T00:26:10.716 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_items": 0, 2024-04-25T00:26:10.716 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T00:26:10.716 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_items": 0, 2024-04-25T00:26:10.716 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T00:26:10.716 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T00:26:10.716 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T00:26:10.716 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T00:26:10.716 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T00:26:10.716 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_items": 0, 2024-04-25T00:26:10.716 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T00:26:10.716 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T00:26:10.716 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_bytes": 0, 2024-04-25T00:26:10.716 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_items": 0, 2024-04-25T00:26:10.716 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_bytes": 0, 2024-04-25T00:26:10.717 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_items": 0, 2024-04-25T00:26:10.717 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T00:26:10.717 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T00:26:10.717 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T00:26:10.717 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T00:26:10.717 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T00:26:10.717 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_items": 0, 2024-04-25T00:26:10.717 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_bytes": 0, 2024-04-25T00:26:10.717 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_items": 0, 2024-04-25T00:26:10.717 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T00:26:10.717 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T00:26:10.717 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_bytes": 0, 2024-04-25T00:26:10.717 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_items": 0, 2024-04-25T00:26:10.718 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_bytes": 0, 2024-04-25T00:26:10.718 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_items": 0, 2024-04-25T00:26:10.718 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T00:26:10.718 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_items": 0, 2024-04-25T00:26:10.718 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T00:26:10.718 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_items": 0, 2024-04-25T00:26:10.718 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_bytes": 189473, 2024-04-25T00:26:10.718 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_items": 53, 2024-04-25T00:26:10.718 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_bytes": 0, 2024-04-25T00:26:10.718 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_items": 0, 2024-04-25T00:26:10.718 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bytes": 0, 2024-04-25T00:26:10.718 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_items": 0, 2024-04-25T00:26:10.718 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_bytes": 0, 2024-04-25T00:26:10.719 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_items": 0, 2024-04-25T00:26:10.719 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_bytes": 0, 2024-04-25T00:26:10.719 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_items": 0, 2024-04-25T00:26:10.719 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_bytes": 9752, 2024-04-25T00:26:10.719 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_items": 188, 2024-04-25T00:26:10.719 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T00:26:10.719 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_items": 0, 2024-04-25T00:26:10.719 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_bytes": 0, 2024-04-25T00:26:10.719 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_items": 0, 2024-04-25T00:26:10.719 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_bytes": 15776, 2024-04-25T00:26:10.719 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_items": 44, 2024-04-25T00:26:10.719 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_bytes": 0, 2024-04-25T00:26:10.719 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_items": 0, 2024-04-25T00:26:10.720 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_bytes": 0, 2024-04-25T00:26:10.720 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_items": 0 2024-04-25T00:26:10.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.720 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter": { 2024-04-25T00:26:10.720 INFO:teuthology.orchestra.run.smithi027.stdout: "op_active": 0, 2024-04-25T00:26:10.720 INFO:teuthology.orchestra.run.smithi027.stdout: "op_laggy": 0, 2024-04-25T00:26:10.720 INFO:teuthology.orchestra.run.smithi027.stdout: "op_send": 2121, 2024-04-25T00:26:10.720 INFO:teuthology.orchestra.run.smithi027.stdout: "op_send_bytes": 21, 2024-04-25T00:26:10.720 INFO:teuthology.orchestra.run.smithi027.stdout: "op_resend": 3, 2024-04-25T00:26:10.720 INFO:teuthology.orchestra.run.smithi027.stdout: "op_reply": 2121, 2024-04-25T00:26:10.720 INFO:teuthology.orchestra.run.smithi027.stdout: "op_latency": { 2024-04-25T00:26:10.720 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 2121, 2024-04-25T00:26:10.720 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 2.857928458, 2024-04-25T00:26:10.720 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.001347443 2024-04-25T00:26:10.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.721 INFO:teuthology.orchestra.run.smithi027.stdout: "op_inflight": 0, 2024-04-25T00:26:10.721 INFO:teuthology.orchestra.run.smithi027.stdout: "oplen_avg": { 2024-04-25T00:26:10.721 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 2121, 2024-04-25T00:26:10.721 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 2122 2024-04-25T00:26:10.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.721 INFO:teuthology.orchestra.run.smithi027.stdout: "op": 2121, 2024-04-25T00:26:10.721 INFO:teuthology.orchestra.run.smithi027.stdout: "op_r": 2121, 2024-04-25T00:26:10.721 INFO:teuthology.orchestra.run.smithi027.stdout: "op_w": 0, 2024-04-25T00:26:10.721 INFO:teuthology.orchestra.run.smithi027.stdout: "op_rmw": 0, 2024-04-25T00:26:10.721 INFO:teuthology.orchestra.run.smithi027.stdout: "op_pg": 0, 2024-04-25T00:26:10.721 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_stat": 1364, 2024-04-25T00:26:10.721 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_create": 0, 2024-04-25T00:26:10.721 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_read": 755, 2024-04-25T00:26:10.722 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_write": 0, 2024-04-25T00:26:10.722 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_writefull": 0, 2024-04-25T00:26:10.722 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_writesame": 0, 2024-04-25T00:26:10.722 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_append": 0, 2024-04-25T00:26:10.722 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_zero": 0, 2024-04-25T00:26:10.722 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_truncate": 0, 2024-04-25T00:26:10.722 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_delete": 0, 2024-04-25T00:26:10.722 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_mapext": 0, 2024-04-25T00:26:10.722 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_sparse_read": 0, 2024-04-25T00:26:10.722 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_clonerange": 0, 2024-04-25T00:26:10.722 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_getxattr": 1, 2024-04-25T00:26:10.722 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_setxattr": 0, 2024-04-25T00:26:10.722 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_cmpxattr": 0, 2024-04-25T00:26:10.722 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_rmxattr": 0, 2024-04-25T00:26:10.723 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_resetxattrs": 0, 2024-04-25T00:26:10.723 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_call": 0, 2024-04-25T00:26:10.723 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_watch": 0, 2024-04-25T00:26:10.723 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_notify": 0, 2024-04-25T00:26:10.723 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T00:26:10.723 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_pgls": 0, 2024-04-25T00:26:10.723 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_pgls_filter": 0, 2024-04-25T00:26:10.723 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_other": 0, 2024-04-25T00:26:10.723 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_active": 0, 2024-04-25T00:26:10.723 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_send": 0, 2024-04-25T00:26:10.723 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_resend": 0, 2024-04-25T00:26:10.723 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_ping": 0, 2024-04-25T00:26:10.723 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_active": 0, 2024-04-25T00:26:10.723 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_send": 0, 2024-04-25T00:26:10.724 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_resend": 0, 2024-04-25T00:26:10.724 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_active": 0, 2024-04-25T00:26:10.724 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_send": 0, 2024-04-25T00:26:10.724 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_resend": 0, 2024-04-25T00:26:10.724 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_active": 0, 2024-04-25T00:26:10.724 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_send": 0, 2024-04-25T00:26:10.724 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_resend": 0, 2024-04-25T00:26:10.724 INFO:teuthology.orchestra.run.smithi027.stdout: "command_active": 0, 2024-04-25T00:26:10.724 INFO:teuthology.orchestra.run.smithi027.stdout: "command_send": 0, 2024-04-25T00:26:10.724 INFO:teuthology.orchestra.run.smithi027.stdout: "command_resend": 0, 2024-04-25T00:26:10.724 INFO:teuthology.orchestra.run.smithi027.stdout: "map_epoch": 95, 2024-04-25T00:26:10.724 INFO:teuthology.orchestra.run.smithi027.stdout: "map_full": 0, 2024-04-25T00:26:10.724 INFO:teuthology.orchestra.run.smithi027.stdout: "map_inc": 6, 2024-04-25T00:26:10.724 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_sessions": 5, 2024-04-25T00:26:10.725 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_session_open": 5, 2024-04-25T00:26:10.725 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_session_close": 0, 2024-04-25T00:26:10.725 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_laggy": 0, 2024-04-25T00:26:10.725 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_wr": 0, 2024-04-25T00:26:10.725 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_rd": 2, 2024-04-25T00:26:10.725 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_del": 0 2024-04-25T00:26:10.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.725 INFO:teuthology.orchestra.run.smithi027.stdout: "oft": { 2024-04-25T00:26:10.725 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_objs": 0, 2024-04-25T00:26:10.725 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_kv_pairs": 0, 2024-04-25T00:26:10.725 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_updates": 0, 2024-04-25T00:26:10.725 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_removes": 0 2024-04-25T00:26:10.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.725 INFO:teuthology.orchestra.run.smithi027.stdout: "osd-slow-ops": { 2024-04-25T00:26:10.726 INFO:teuthology.orchestra.run.smithi027.stdout: "slow_ops_count": 0 2024-04-25T00:26:10.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.726 INFO:teuthology.orchestra.run.smithi027.stdout: "purge_queue": { 2024-04-25T00:26:10.726 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_ops": 0, 2024-04-25T00:26:10.726 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T00:26:10.726 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing": 0, 2024-04-25T00:26:10.726 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_high_water": 0, 2024-04-25T00:26:10.726 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executed_ops": 0, 2024-04-25T00:26:10.726 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executed": 0, 2024-04-25T00:26:10.726 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_item_in_journal": 0 2024-04-25T00:26:10.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.726 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T00:26:10.726 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-25T00:26:10.726 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-25T00:26:10.727 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-25T00:26:10.727 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 2429, 2024-04-25T00:26:10.727 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 2497015, 2024-04-25T00:26:10.727 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:10.727 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 2429, 2024-04-25T00:26:10.727 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-25T00:26:10.727 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-25T00:26:10.727 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 2429, 2024-04-25T00:26:10.727 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 2497015, 2024-04-25T00:26:10.727 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-25T00:26:10.727 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.727 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.727 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.727 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:10.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.728 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-objecter_bytes": { 2024-04-25T00:26:10.728 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-25T00:26:10.728 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-25T00:26:10.728 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-25T00:26:10.728 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-25T00:26:10.728 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-25T00:26:10.728 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:10.728 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:10.728 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 2121, 2024-04-25T00:26:10.728 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 36151, 2024-04-25T00:26:10.728 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 2121, 2024-04-25T00:26:10.728 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 36151, 2024-04-25T00:26:10.728 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-25T00:26:10.728 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.729 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.729 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.729 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:10.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.729 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-objecter_ops": { 2024-04-25T00:26:10.729 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-25T00:26:10.729 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1024, 2024-04-25T00:26:10.729 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-25T00:26:10.729 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-25T00:26:10.729 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-25T00:26:10.729 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:10.729 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:10.729 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 2121, 2024-04-25T00:26:10.729 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 2121, 2024-04-25T00:26:10.729 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 2121, 2024-04-25T00:26:10.730 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 2121, 2024-04-25T00:26:10.730 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-25T00:26:10.730 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.730 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.730 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.730 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:10.730 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.730 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-write_buf_throttle": { 2024-04-25T00:26:10.730 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-25T00:26:10.730 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 3758096384, 2024-04-25T00:26:10.730 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-25T00:26:10.730 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-25T00:26:10.730 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-25T00:26:10.730 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:10.731 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:10.731 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-25T00:26:10.731 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-25T00:26:10.731 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 0, 2024-04-25T00:26:10.731 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 0, 2024-04-25T00:26:10.731 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-25T00:26:10.731 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.731 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.731 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.731 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:10.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:10.731 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-write_buf_throttle-0x55ca95af86e0": { 2024-04-25T00:26:10.731 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-25T00:26:10.731 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 3758096384, 2024-04-25T00:26:10.732 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-25T00:26:10.732 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-25T00:26:10.732 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-25T00:26:10.732 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:10.732 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:10.732 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-25T00:26:10.732 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-25T00:26:10.732 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 0, 2024-04-25T00:26:10.732 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 0, 2024-04-25T00:26:10.732 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-25T00:26:10.732 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:10.732 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:10.732 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:10.732 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:10.733 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:10.733 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:26:10.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:10 smithi027 ceph-mon[29208]: pgmap v316: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 33 op/s 2024-04-25T00:26:11.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:10 smithi139 ceph-mon[32833]: pgmap v316: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 33 op/s 2024-04-25T00:26:11.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:10 smithi153 ceph-mon[32896]: pgmap v316: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 33 op/s 2024-04-25T00:26:11.475 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-04-25T00:26:11.475 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-04-25T00:26:11.475 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-04-25T00:26:11.475 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-04-25T00:26:11.476 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T00:26:11.476 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-04-25T00:26:11.476 DEBUG:tasks.check_counter:Getting stats from d 2024-04-25T00:26:11.476 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-25T00:26:11.732 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:26:12.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:12 smithi027 ceph-mon[29208]: pgmap v317: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-25T00:26:13.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:12 smithi139 ceph-mon[32833]: pgmap v317: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-25T00:26:13.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:12 smithi153 ceph-mon[32896]: pgmap v317: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-25T00:26:13.657 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:26:13.658 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T00:26:13.658 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 4, 2024-04-25T00:26:13.658 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 4, 2024-04-25T00:26:13.658 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 3024, 2024-04-25T00:26:13.658 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 448, 2024-04-25T00:26:13.658 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 4, 2024-04-25T00:26:13.658 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 0, 2024-04-25T00:26:13.658 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.001673422, 2024-04-25T00:26:13.658 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.000040356, 2024-04-25T00:26:13.658 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.000625890, 2024-04-25T00:26:13.658 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T00:26:13.658 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:13.658 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 3, 2024-04-25T00:26:13.659 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000719792, 2024-04-25T00:26:13.659 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000239930 2024-04-25T00:26:13.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:13.659 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:13.659 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:13.659 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:13.659 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:13.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:13.659 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-25T00:26:13.659 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T00:26:13.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:13.659 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T00:26:13.659 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 2, 2024-04-25T00:26:13.659 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 165, 2024-04-25T00:26:13.660 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 320, 2024-04-25T00:26:13.660 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 184448, 2024-04-25T00:26:13.660 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 5, 2024-04-25T00:26:13.660 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 3, 2024-04-25T00:26:13.660 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.013047016, 2024-04-25T00:26:13.660 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.009018437, 2024-04-25T00:26:13.660 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.001801275, 2024-04-25T00:26:13.660 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T00:26:13.660 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:13.660 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 162, 2024-04-25T00:26:13.660 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.016649403, 2024-04-25T00:26:13.660 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000102774 2024-04-25T00:26:13.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:13.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:13.661 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:13.661 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:13.661 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:13.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:13.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T00:26:13.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 184448 2024-04-25T00:26:13.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:13.661 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T00:26:13.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 227, 2024-04-25T00:26:13.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 216, 2024-04-25T00:26:13.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 1582352, 2024-04-25T00:26:13.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 132544, 2024-04-25T00:26:13.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 2, 2024-04-25T00:26:13.662 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 1, 2024-04-25T00:26:13.662 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.045049186, 2024-04-25T00:26:13.662 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.012740529, 2024-04-25T00:26:13.662 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.037612513, 2024-04-25T00:26:13.662 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.004825674, 2024-04-25T00:26:13.662 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:13.662 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 215, 2024-04-25T00:26:13.662 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.014378911, 2024-04-25T00:26:13.662 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000066878 2024-04-25T00:26:13.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:13.662 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:13.662 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:13.662 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:13.662 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:13.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:13.663 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 1582352, 2024-04-25T00:26:13.663 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 132544 2024-04-25T00:26:13.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:13.663 INFO:teuthology.orchestra.run.smithi027.stdout: "cct": { 2024-04-25T00:26:13.663 INFO:teuthology.orchestra.run.smithi027.stdout: "total_workers": 0, 2024-04-25T00:26:13.663 INFO:teuthology.orchestra.run.smithi027.stdout: "unhealthy_workers": 0 2024-04-25T00:26:13.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:13.663 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool": { 2024-04-25T00:26:13.663 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_bytes": 0, 2024-04-25T00:26:13.663 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_items": 0, 2024-04-25T00:26:13.663 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T00:26:13.663 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_items": 0, 2024-04-25T00:26:13.664 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T00:26:13.664 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_items": 0, 2024-04-25T00:26:13.664 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T00:26:13.664 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T00:26:13.664 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T00:26:13.664 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T00:26:13.664 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T00:26:13.664 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_items": 0, 2024-04-25T00:26:13.664 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T00:26:13.664 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T00:26:13.664 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_bytes": 0, 2024-04-25T00:26:13.664 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_items": 0, 2024-04-25T00:26:13.664 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_bytes": 0, 2024-04-25T00:26:13.664 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_items": 0, 2024-04-25T00:26:13.664 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T00:26:13.665 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T00:26:13.665 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T00:26:13.665 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T00:26:13.665 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T00:26:13.665 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_items": 0, 2024-04-25T00:26:13.665 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_bytes": 0, 2024-04-25T00:26:13.665 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_items": 0, 2024-04-25T00:26:13.665 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T00:26:13.665 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T00:26:13.665 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_bytes": 0, 2024-04-25T00:26:13.665 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_items": 0, 2024-04-25T00:26:13.665 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_bytes": 0, 2024-04-25T00:26:13.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_items": 0, 2024-04-25T00:26:13.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T00:26:13.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_items": 0, 2024-04-25T00:26:13.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T00:26:13.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_items": 0, 2024-04-25T00:26:13.666 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_bytes": 148994, 2024-04-25T00:26:13.666 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_items": 31, 2024-04-25T00:26:13.666 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_bytes": 0, 2024-04-25T00:26:13.666 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_items": 0, 2024-04-25T00:26:13.666 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bytes": 0, 2024-04-25T00:26:13.666 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_items": 0, 2024-04-25T00:26:13.666 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_bytes": 0, 2024-04-25T00:26:13.666 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_items": 0, 2024-04-25T00:26:13.666 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_bytes": 0, 2024-04-25T00:26:13.667 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_items": 0, 2024-04-25T00:26:13.667 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_bytes": 0, 2024-04-25T00:26:13.667 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_items": 0, 2024-04-25T00:26:13.667 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T00:26:13.667 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_items": 0, 2024-04-25T00:26:13.667 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_bytes": 0, 2024-04-25T00:26:13.667 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_items": 0, 2024-04-25T00:26:13.667 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_bytes": 928, 2024-04-25T00:26:13.667 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_items": 2, 2024-04-25T00:26:13.667 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_bytes": 0, 2024-04-25T00:26:13.667 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_items": 0, 2024-04-25T00:26:13.667 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_bytes": 0, 2024-04-25T00:26:13.667 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_items": 0 2024-04-25T00:26:13.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:13.667 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T00:26:13.668 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-25T00:26:13.668 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-25T00:26:13.668 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-25T00:26:13.668 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 229, 2024-04-25T00:26:13.668 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 1550370, 2024-04-25T00:26:13.668 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:13.668 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 229, 2024-04-25T00:26:13.668 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-25T00:26:13.668 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-25T00:26:13.668 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 229, 2024-04-25T00:26:13.668 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 1550370, 2024-04-25T00:26:13.668 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-25T00:26:13.669 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:13.669 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:13.669 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:13.669 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:13.669 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:13.669 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:26:14.298 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-25T00:26:14.299 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-25T00:26:14.299 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-25T00:26:14.299 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-25T00:26:14.299 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-25T00:26:14.299 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-04-25T00:26:14.299 DEBUG:tasks.check_counter:Getting stats from g 2024-04-25T00:26:14.299 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-25T00:26:14.549 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:26:14.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:14 smithi027 ceph-mon[29208]: pgmap v318: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-25T00:26:14.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:14 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:26:15.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:14 smithi139 ceph-mon[32833]: pgmap v318: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-25T00:26:15.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:14 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:26:15.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:14 smithi153 ceph-mon[32896]: pgmap v318: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-25T00:26:15.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:14 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T00:26:16.323 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:26:16.323 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T00:26:16.323 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 752, 2024-04-25T00:26:16.323 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 726, 2024-04-25T00:26:16.323 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 2244544, 2024-04-25T00:26:16.323 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 280668, 2024-04-25T00:26:16.323 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 5, 2024-04-25T00:26:16.323 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 1, 2024-04-25T00:26:16.323 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.101035623, 2024-04-25T00:26:16.324 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.035665101, 2024-04-25T00:26:16.324 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.053552946, 2024-04-25T00:26:16.324 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.018341985, 2024-04-25T00:26:16.324 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:16.324 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 726, 2024-04-25T00:26:16.324 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.035708577, 2024-04-25T00:26:16.324 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000049185 2024-04-25T00:26:16.324 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.324 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:16.324 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.324 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.324 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.324 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.324 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 2096704, 2024-04-25T00:26:16.325 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 136128 2024-04-25T00:26:16.325 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.325 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T00:26:16.325 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 781, 2024-04-25T00:26:16.325 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 941, 2024-04-25T00:26:16.325 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 272907, 2024-04-25T00:26:16.325 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 691043, 2024-04-25T00:26:16.325 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 7, 2024-04-25T00:26:16.325 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 5, 2024-04-25T00:26:16.325 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.068409561, 2024-04-25T00:26:16.325 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.023471276, 2024-04-25T00:26:16.325 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.017963599, 2024-04-25T00:26:16.325 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.020915426, 2024-04-25T00:26:16.325 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:16.325 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 938, 2024-04-25T00:26:16.326 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.044013418, 2024-04-25T00:26:16.326 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000046922 2024-04-25T00:26:16.326 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.326 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:16.326 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.326 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.326 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.326 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.326 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T00:26:16.326 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 443264 2024-04-25T00:26:16.326 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.326 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T00:26:16.327 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 4, 2024-04-25T00:26:16.327 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 6, 2024-04-25T00:26:16.327 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 1629, 2024-04-25T00:26:16.327 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 1284, 2024-04-25T00:26:16.327 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 5, 2024-04-25T00:26:16.327 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 4, 2024-04-25T00:26:16.327 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.001587529, 2024-04-25T00:26:16.327 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.000080825, 2024-04-25T00:26:16.327 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.001297964, 2024-04-25T00:26:16.327 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.000097354, 2024-04-25T00:26:16.327 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:16.327 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 4, 2024-04-25T00:26:16.327 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.002664598, 2024-04-25T00:26:16.327 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000666149 2024-04-25T00:26:16.328 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.328 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:16.328 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.328 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.328 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.328 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.328 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T00:26:16.328 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T00:26:16.328 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.328 INFO:teuthology.orchestra.run.smithi027.stdout: "cct": { 2024-04-25T00:26:16.328 INFO:teuthology.orchestra.run.smithi027.stdout: "total_workers": 1, 2024-04-25T00:26:16.328 INFO:teuthology.orchestra.run.smithi027.stdout: "unhealthy_workers": 0 2024-04-25T00:26:16.328 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.328 INFO:teuthology.orchestra.run.smithi027.stdout: "finisher-MDSRank": { 2024-04-25T00:26:16.329 INFO:teuthology.orchestra.run.smithi027.stdout: "queue_len": 0, 2024-04-25T00:26:16.329 INFO:teuthology.orchestra.run.smithi027.stdout: "complete_latency": { 2024-04-25T00:26:16.329 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 2105, 2024-04-25T00:26:16.329 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.050913682, 2024-04-25T00:26:16.329 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000024187 2024-04-25T00:26:16.329 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:16.329 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.329 INFO:teuthology.orchestra.run.smithi027.stdout: "finisher-PurgeQueue": { 2024-04-25T00:26:16.329 INFO:teuthology.orchestra.run.smithi027.stdout: "queue_len": 0, 2024-04-25T00:26:16.329 INFO:teuthology.orchestra.run.smithi027.stdout: "complete_latency": { 2024-04-25T00:26:16.330 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.330 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.330 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.330 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:16.330 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.330 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-25T00:26:16.330 INFO:teuthology.orchestra.run.smithi027.stdout: "request": 0, 2024-04-25T00:26:16.330 INFO:teuthology.orchestra.run.smithi027.stdout: "reply": 0, 2024-04-25T00:26:16.330 INFO:teuthology.orchestra.run.smithi027.stdout: "reply_latency": { 2024-04-25T00:26:16.330 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.330 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.330 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.330 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.331 INFO:teuthology.orchestra.run.smithi027.stdout: "slow_reply": 0, 2024-04-25T00:26:16.331 INFO:teuthology.orchestra.run.smithi027.stdout: "forward": 0, 2024-04-25T00:26:16.331 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_fetch_complete": 0, 2024-04-25T00:26:16.331 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_fetch_keys": 0, 2024-04-25T00:26:16.331 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_commit": 0, 2024-04-25T00:26:16.331 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_split": 0, 2024-04-25T00:26:16.331 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_merge": 0, 2024-04-25T00:26:16.331 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes": 4, 2024-04-25T00:26:16.331 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_top": 0, 2024-04-25T00:26:16.331 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_bottom": 4, 2024-04-25T00:26:16.331 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_pin_tail": 0, 2024-04-25T00:26:16.331 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_pinned": 4, 2024-04-25T00:26:16.331 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_expired": 0, 2024-04-25T00:26:16.331 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_with_caps": 0, 2024-04-25T00:26:16.332 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": 0, 2024-04-25T00:26:16.332 INFO:teuthology.orchestra.run.smithi027.stdout: "subtrees": 1, 2024-04-25T00:26:16.332 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse": 0, 2024-04-25T00:26:16.332 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_hit": 0, 2024-04-25T00:26:16.332 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_forward": 0, 2024-04-25T00:26:16.332 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_discover": 0, 2024-04-25T00:26:16.332 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_dir_fetch": 0, 2024-04-25T00:26:16.332 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_remote_ino": 0, 2024-04-25T00:26:16.332 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_lock": 0, 2024-04-25T00:26:16.332 INFO:teuthology.orchestra.run.smithi027.stdout: "load_cent": 0, 2024-04-25T00:26:16.332 INFO:teuthology.orchestra.run.smithi027.stdout: "q": 0, 2024-04-25T00:26:16.332 INFO:teuthology.orchestra.run.smithi027.stdout: "exported": 0, 2024-04-25T00:26:16.332 INFO:teuthology.orchestra.run.smithi027.stdout: "exported_inodes": 0, 2024-04-25T00:26:16.332 INFO:teuthology.orchestra.run.smithi027.stdout: "imported": 0, 2024-04-25T00:26:16.332 INFO:teuthology.orchestra.run.smithi027.stdout: "imported_inodes": 0, 2024-04-25T00:26:16.333 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_dir_fetch": 0, 2024-04-25T00:26:16.333 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_backtrace_fetch": 0, 2024-04-25T00:26:16.333 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_peer_discover": 0, 2024-04-25T00:26:16.333 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rfiles": 0, 2024-04-25T00:26:16.333 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rbytes": 0, 2024-04-25T00:26:16.333 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rsnaps": 0, 2024-04-25T00:26:16.333 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T00:26:16.333 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_set_tag": 0, 2024-04-25T00:26:16.333 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T00:26:16.333 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_inotable_repaired": 0, 2024-04-25T00:26:16.333 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dir_inodes": 0, 2024-04-25T00:26:16.333 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T00:26:16.333 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T00:26:16.333 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_file_inodes": 0, 2024-04-25T00:26:16.334 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_inode_file_caps": 0, 2024-04-25T00:26:16.334 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T00:26:16.334 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_grant": 0, 2024-04-25T00:26:16.334 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T00:26:16.334 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T00:26:16.334 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T00:26:16.334 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_caps": 0, 2024-04-25T00:26:16.334 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_caps_dirty": 0, 2024-04-25T00:26:16.334 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_cap_release": 0, 2024-04-25T00:26:16.334 INFO:teuthology.orchestra.run.smithi027.stdout: "process_request_cap_release": 0 2024-04-25T00:26:16.335 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.335 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache": { 2024-04-25T00:26:16.335 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_update": 0, 2024-04-25T00:26:16.335 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_update_receipt": 0, 2024-04-25T00:26:16.335 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_try_discover": 0, 2024-04-25T00:26:16.335 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_send_discover": 0, 2024-04-25T00:26:16.335 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_handle_discover": 0, 2024-04-25T00:26:16.335 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays": 0, 2024-04-25T00:26:16.335 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays_delayed": 0, 2024-04-25T00:26:16.335 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays_enqueuing": 0, 2024-04-25T00:26:16.335 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_created": 0, 2024-04-25T00:26:16.335 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_enqueued": 0, 2024-04-25T00:26:16.336 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_reintegrated": 0, 2024-04-25T00:26:16.336 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_migrated": 0, 2024-04-25T00:26:16.336 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_processing": 0, 2024-04-25T00:26:16.336 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_enqueued": 0, 2024-04-25T00:26:16.336 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_prioritized": 0, 2024-04-25T00:26:16.336 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_started": 0, 2024-04-25T00:26:16.336 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_completed": 0, 2024-04-25T00:26:16.336 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_quiesce_path": 0, 2024-04-25T00:26:16.336 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_quiesce_inode": 0, 2024-04-25T00:26:16.336 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T00:26:16.336 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_exportdir": 0, 2024-04-25T00:26:16.336 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_flush": 0, 2024-04-25T00:26:16.336 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_fragmentdir": 0, 2024-04-25T00:26:16.336 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_fragstats": 0, 2024-04-25T00:26:16.337 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_inodestats": 0 2024-04-25T00:26:16.337 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.337 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log": { 2024-04-25T00:26:16.337 INFO:teuthology.orchestra.run.smithi027.stdout: "evlrg": 0, 2024-04-25T00:26:16.337 INFO:teuthology.orchestra.run.smithi027.stdout: "evadd": 0, 2024-04-25T00:26:16.337 INFO:teuthology.orchestra.run.smithi027.stdout: "evex": 0, 2024-04-25T00:26:16.337 INFO:teuthology.orchestra.run.smithi027.stdout: "evtrm": 0, 2024-04-25T00:26:16.337 INFO:teuthology.orchestra.run.smithi027.stdout: "ev": 34, 2024-04-25T00:26:16.337 INFO:teuthology.orchestra.run.smithi027.stdout: "evexg": 0, 2024-04-25T00:26:16.337 INFO:teuthology.orchestra.run.smithi027.stdout: "evexd": 0, 2024-04-25T00:26:16.337 INFO:teuthology.orchestra.run.smithi027.stdout: "segadd": 0, 2024-04-25T00:26:16.337 INFO:teuthology.orchestra.run.smithi027.stdout: "segex": 0, 2024-04-25T00:26:16.337 INFO:teuthology.orchestra.run.smithi027.stdout: "segtrm": 0, 2024-04-25T00:26:16.337 INFO:teuthology.orchestra.run.smithi027.stdout: "seg": 2, 2024-04-25T00:26:16.337 INFO:teuthology.orchestra.run.smithi027.stdout: "segmjr": 2, 2024-04-25T00:26:16.338 INFO:teuthology.orchestra.run.smithi027.stdout: "segexg": 0, 2024-04-25T00:26:16.338 INFO:teuthology.orchestra.run.smithi027.stdout: "segexd": 0, 2024-04-25T00:26:16.338 INFO:teuthology.orchestra.run.smithi027.stdout: "expos": 4194304, 2024-04-25T00:26:16.338 INFO:teuthology.orchestra.run.smithi027.stdout: "wrpos": 4268153, 2024-04-25T00:26:16.338 INFO:teuthology.orchestra.run.smithi027.stdout: "rdpos": 4265548, 2024-04-25T00:26:16.338 INFO:teuthology.orchestra.run.smithi027.stdout: "jlat": { 2024-04-25T00:26:16.338 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.338 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.338 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.338 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.338 INFO:teuthology.orchestra.run.smithi027.stdout: "replayed": 34 2024-04-25T00:26:16.338 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.339 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_mem": { 2024-04-25T00:26:16.339 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 4, 2024-04-25T00:26:16.339 INFO:teuthology.orchestra.run.smithi027.stdout: "ino+": 27, 2024-04-25T00:26:16.339 INFO:teuthology.orchestra.run.smithi027.stdout: "ino-": 23, 2024-04-25T00:26:16.339 INFO:teuthology.orchestra.run.smithi027.stdout: "dir": 2, 2024-04-25T00:26:16.339 INFO:teuthology.orchestra.run.smithi027.stdout: "dir+": 20, 2024-04-25T00:26:16.339 INFO:teuthology.orchestra.run.smithi027.stdout: "dir-": 18, 2024-04-25T00:26:16.340 INFO:teuthology.orchestra.run.smithi027.stdout: "dn": 4, 2024-04-25T00:26:16.340 INFO:teuthology.orchestra.run.smithi027.stdout: "dn+": 31, 2024-04-25T00:26:16.340 INFO:teuthology.orchestra.run.smithi027.stdout: "dn-": 27, 2024-04-25T00:26:16.340 INFO:teuthology.orchestra.run.smithi027.stdout: "cap": 0, 2024-04-25T00:26:16.340 INFO:teuthology.orchestra.run.smithi027.stdout: "cap+": 0, 2024-04-25T00:26:16.340 INFO:teuthology.orchestra.run.smithi027.stdout: "cap-": 0, 2024-04-25T00:26:16.340 INFO:teuthology.orchestra.run.smithi027.stdout: "rss": 54256, 2024-04-25T00:26:16.340 INFO:teuthology.orchestra.run.smithi027.stdout: "heap": 215420 2024-04-25T00:26:16.340 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.340 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_server": { 2024-04-25T00:26:16.340 INFO:teuthology.orchestra.run.smithi027.stdout: "dispatch_client_request": 0, 2024-04-25T00:26:16.341 INFO:teuthology.orchestra.run.smithi027.stdout: "dispatch_server_request": 0, 2024-04-25T00:26:16.341 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_request": 0, 2024-04-25T00:26:16.341 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_session": 0, 2024-04-25T00:26:16.341 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_peer_request": 0, 2024-04-25T00:26:16.341 INFO:teuthology.orchestra.run.smithi027.stdout: "req_create_latency": { 2024-04-25T00:26:16.341 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.341 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.341 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.341 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getattr_latency": { 2024-04-25T00:26:16.341 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.341 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.341 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.341 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getfilelock_latency": { 2024-04-25T00:26:16.342 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.342 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.342 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.342 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.342 INFO:teuthology.orchestra.run.smithi027.stdout: "req_link_latency": { 2024-04-25T00:26:16.342 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.342 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.342 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.342 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.342 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookup_latency": { 2024-04-25T00:26:16.342 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.342 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.342 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.343 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookuphash_latency": { 2024-04-25T00:26:16.343 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.343 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.343 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.343 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupino_latency": { 2024-04-25T00:26:16.343 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.343 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.343 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.343 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupname_latency": { 2024-04-25T00:26:16.343 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.344 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.344 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.344 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupparent_latency": { 2024-04-25T00:26:16.344 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.344 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.344 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.344 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupsnap_latency": { 2024-04-25T00:26:16.344 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.344 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.344 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.345 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lssnap_latency": { 2024-04-25T00:26:16.345 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.345 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.345 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.345 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.345 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mkdir_latency": { 2024-04-25T00:26:16.345 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.346 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.346 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.346 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.346 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mknod_latency": { 2024-04-25T00:26:16.346 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.346 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.346 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.346 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.346 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mksnap_latency": { 2024-04-25T00:26:16.346 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.347 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.347 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.347 INFO:teuthology.orchestra.run.smithi027.stdout: "req_open_latency": { 2024-04-25T00:26:16.347 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.347 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.347 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.347 INFO:teuthology.orchestra.run.smithi027.stdout: "req_readdir_latency": { 2024-04-25T00:26:16.347 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.347 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.347 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.347 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rename_latency": { 2024-04-25T00:26:16.347 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.348 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.348 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.348 INFO:teuthology.orchestra.run.smithi027.stdout: "req_renamesnap_latency": { 2024-04-25T00:26:16.348 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.348 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.348 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.348 INFO:teuthology.orchestra.run.smithi027.stdout: "req_snapdiff_latency": { 2024-04-25T00:26:16.348 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.348 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.348 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.348 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmdir_latency": { 2024-04-25T00:26:16.348 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.349 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.349 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.349 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.349 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmsnap_latency": { 2024-04-25T00:26:16.349 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.349 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.349 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.349 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.349 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmxattr_latency": { 2024-04-25T00:26:16.349 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.349 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.349 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.349 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.349 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setattr_latency": { 2024-04-25T00:26:16.350 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.350 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.350 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.350 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.350 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setdirlayout_latency": { 2024-04-25T00:26:16.350 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.350 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.350 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.350 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.350 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setfilelock_latency": { 2024-04-25T00:26:16.350 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.350 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.350 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.350 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.351 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setlayout_latency": { 2024-04-25T00:26:16.351 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.351 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.351 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.351 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setxattr_latency": { 2024-04-25T00:26:16.351 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.351 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.351 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.351 INFO:teuthology.orchestra.run.smithi027.stdout: "req_symlink_latency": { 2024-04-25T00:26:16.351 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.351 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.351 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.352 INFO:teuthology.orchestra.run.smithi027.stdout: "req_unlink_latency": { 2024-04-25T00:26:16.352 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.352 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.352 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.352 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.352 INFO:teuthology.orchestra.run.smithi027.stdout: "cap_revoke_eviction": 0, 2024-04-25T00:26:16.352 INFO:teuthology.orchestra.run.smithi027.stdout: "cap_acquisition_throttle": 0, 2024-04-25T00:26:16.352 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getvxattr_latency": { 2024-04-25T00:26:16.352 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.352 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.352 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.352 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:16.352 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.352 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_sessions": { 2024-04-25T00:26:16.353 INFO:teuthology.orchestra.run.smithi027.stdout: "session_count": 1, 2024-04-25T00:26:16.353 INFO:teuthology.orchestra.run.smithi027.stdout: "session_add": 3, 2024-04-25T00:26:16.353 INFO:teuthology.orchestra.run.smithi027.stdout: "session_remove": 2, 2024-04-25T00:26:16.353 INFO:teuthology.orchestra.run.smithi027.stdout: "sessions_open": 3, 2024-04-25T00:26:16.353 INFO:teuthology.orchestra.run.smithi027.stdout: "sessions_stale": 0, 2024-04-25T00:26:16.353 INFO:teuthology.orchestra.run.smithi027.stdout: "total_load": 0, 2024-04-25T00:26:16.353 INFO:teuthology.orchestra.run.smithi027.stdout: "average_load": 0, 2024-04-25T00:26:16.353 INFO:teuthology.orchestra.run.smithi027.stdout: "avg_session_uptime": 4173, 2024-04-25T00:26:16.353 INFO:teuthology.orchestra.run.smithi027.stdout: "mdthresh_evicted": 0 2024-04-25T00:26:16.353 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.353 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool": { 2024-04-25T00:26:16.353 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_bytes": 0, 2024-04-25T00:26:16.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_items": 0, 2024-04-25T00:26:16.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T00:26:16.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_items": 0, 2024-04-25T00:26:16.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T00:26:16.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_items": 0, 2024-04-25T00:26:16.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T00:26:16.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T00:26:16.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T00:26:16.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T00:26:16.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T00:26:16.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_items": 0, 2024-04-25T00:26:16.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T00:26:16.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T00:26:16.354 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_bytes": 0, 2024-04-25T00:26:16.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_items": 0, 2024-04-25T00:26:16.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_bytes": 0, 2024-04-25T00:26:16.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_items": 0, 2024-04-25T00:26:16.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T00:26:16.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T00:26:16.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T00:26:16.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T00:26:16.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T00:26:16.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_items": 0, 2024-04-25T00:26:16.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_bytes": 0, 2024-04-25T00:26:16.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_items": 0, 2024-04-25T00:26:16.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T00:26:16.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T00:26:16.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_bytes": 0, 2024-04-25T00:26:16.355 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_items": 0, 2024-04-25T00:26:16.356 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_bytes": 0, 2024-04-25T00:26:16.356 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_items": 0, 2024-04-25T00:26:16.356 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T00:26:16.356 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_items": 0, 2024-04-25T00:26:16.356 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T00:26:16.356 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_items": 0, 2024-04-25T00:26:16.356 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_bytes": 197530, 2024-04-25T00:26:16.356 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_items": 57, 2024-04-25T00:26:16.356 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_bytes": 0, 2024-04-25T00:26:16.356 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_items": 0, 2024-04-25T00:26:16.356 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bytes": 0, 2024-04-25T00:26:16.357 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_items": 0, 2024-04-25T00:26:16.357 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_bytes": 0, 2024-04-25T00:26:16.357 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_items": 0, 2024-04-25T00:26:16.357 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_bytes": 0, 2024-04-25T00:26:16.357 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_items": 0, 2024-04-25T00:26:16.357 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_bytes": 9752, 2024-04-25T00:26:16.357 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_items": 188, 2024-04-25T00:26:16.357 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T00:26:16.357 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_items": 0, 2024-04-25T00:26:16.357 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_bytes": 0, 2024-04-25T00:26:16.357 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_items": 0, 2024-04-25T00:26:16.358 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_bytes": 16480, 2024-04-25T00:26:16.358 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_items": 48, 2024-04-25T00:26:16.358 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_bytes": 0, 2024-04-25T00:26:16.358 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_items": 0, 2024-04-25T00:26:16.358 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_bytes": 0, 2024-04-25T00:26:16.358 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_items": 0 2024-04-25T00:26:16.358 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.358 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter": { 2024-04-25T00:26:16.358 INFO:teuthology.orchestra.run.smithi027.stdout: "op_active": 0, 2024-04-25T00:26:16.358 INFO:teuthology.orchestra.run.smithi027.stdout: "op_laggy": 0, 2024-04-25T00:26:16.358 INFO:teuthology.orchestra.run.smithi027.stdout: "op_send": 1223, 2024-04-25T00:26:16.359 INFO:teuthology.orchestra.run.smithi027.stdout: "op_send_bytes": 21, 2024-04-25T00:26:16.359 INFO:teuthology.orchestra.run.smithi027.stdout: "op_resend": 3, 2024-04-25T00:26:16.359 INFO:teuthology.orchestra.run.smithi027.stdout: "op_reply": 1223, 2024-04-25T00:26:16.359 INFO:teuthology.orchestra.run.smithi027.stdout: "op_latency": { 2024-04-25T00:26:16.359 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 1223, 2024-04-25T00:26:16.359 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 309.565241726, 2024-04-25T00:26:16.359 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.253119576 2024-04-25T00:26:16.359 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.359 INFO:teuthology.orchestra.run.smithi027.stdout: "op_inflight": 0, 2024-04-25T00:26:16.359 INFO:teuthology.orchestra.run.smithi027.stdout: "oplen_avg": { 2024-04-25T00:26:16.359 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 1223, 2024-04-25T00:26:16.359 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 1224 2024-04-25T00:26:16.359 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.359 INFO:teuthology.orchestra.run.smithi027.stdout: "op": 1223, 2024-04-25T00:26:16.359 INFO:teuthology.orchestra.run.smithi027.stdout: "op_r": 1223, 2024-04-25T00:26:16.360 INFO:teuthology.orchestra.run.smithi027.stdout: "op_w": 0, 2024-04-25T00:26:16.360 INFO:teuthology.orchestra.run.smithi027.stdout: "op_rmw": 0, 2024-04-25T00:26:16.360 INFO:teuthology.orchestra.run.smithi027.stdout: "op_pg": 0, 2024-04-25T00:26:16.360 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_stat": 764, 2024-04-25T00:26:16.360 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_create": 0, 2024-04-25T00:26:16.360 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_read": 457, 2024-04-25T00:26:16.360 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_write": 0, 2024-04-25T00:26:16.360 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_writefull": 0, 2024-04-25T00:26:16.360 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_writesame": 0, 2024-04-25T00:26:16.360 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_append": 0, 2024-04-25T00:26:16.360 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_zero": 0, 2024-04-25T00:26:16.361 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_truncate": 0, 2024-04-25T00:26:16.361 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_delete": 0, 2024-04-25T00:26:16.361 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_mapext": 0, 2024-04-25T00:26:16.361 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_sparse_read": 0, 2024-04-25T00:26:16.361 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_clonerange": 0, 2024-04-25T00:26:16.361 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_getxattr": 1, 2024-04-25T00:26:16.361 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_setxattr": 0, 2024-04-25T00:26:16.361 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_cmpxattr": 0, 2024-04-25T00:26:16.361 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_rmxattr": 0, 2024-04-25T00:26:16.361 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_resetxattrs": 0, 2024-04-25T00:26:16.361 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_call": 0, 2024-04-25T00:26:16.361 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_watch": 0, 2024-04-25T00:26:16.361 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_notify": 0, 2024-04-25T00:26:16.361 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T00:26:16.361 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_pgls": 0, 2024-04-25T00:26:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_pgls_filter": 0, 2024-04-25T00:26:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_other": 0, 2024-04-25T00:26:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_active": 0, 2024-04-25T00:26:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_send": 0, 2024-04-25T00:26:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_resend": 0, 2024-04-25T00:26:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_ping": 0, 2024-04-25T00:26:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_active": 0, 2024-04-25T00:26:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_send": 0, 2024-04-25T00:26:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_resend": 0, 2024-04-25T00:26:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_active": 0, 2024-04-25T00:26:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_send": 0, 2024-04-25T00:26:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_resend": 0, 2024-04-25T00:26:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_active": 0, 2024-04-25T00:26:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_send": 0, 2024-04-25T00:26:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_resend": 0, 2024-04-25T00:26:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: "command_active": 0, 2024-04-25T00:26:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: "command_send": 0, 2024-04-25T00:26:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: "command_resend": 0, 2024-04-25T00:26:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: "map_epoch": 95, 2024-04-25T00:26:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: "map_full": 0, 2024-04-25T00:26:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: "map_inc": 6, 2024-04-25T00:26:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_sessions": 6, 2024-04-25T00:26:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_session_open": 6, 2024-04-25T00:26:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_session_close": 0, 2024-04-25T00:26:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_laggy": 0, 2024-04-25T00:26:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_wr": 0, 2024-04-25T00:26:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_rd": 2, 2024-04-25T00:26:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_del": 0 2024-04-25T00:26:16.364 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.364 INFO:teuthology.orchestra.run.smithi027.stdout: "oft": { 2024-04-25T00:26:16.364 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_objs": 0, 2024-04-25T00:26:16.364 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_kv_pairs": 0, 2024-04-25T00:26:16.364 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_updates": 0, 2024-04-25T00:26:16.364 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_removes": 0 2024-04-25T00:26:16.364 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.364 INFO:teuthology.orchestra.run.smithi027.stdout: "osd-slow-ops": { 2024-04-25T00:26:16.364 INFO:teuthology.orchestra.run.smithi027.stdout: "slow_ops_count": 0 2024-04-25T00:26:16.364 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.364 INFO:teuthology.orchestra.run.smithi027.stdout: "purge_queue": { 2024-04-25T00:26:16.364 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_ops": 0, 2024-04-25T00:26:16.364 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T00:26:16.364 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing": 0, 2024-04-25T00:26:16.365 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_high_water": 0, 2024-04-25T00:26:16.365 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executed_ops": 0, 2024-04-25T00:26:16.365 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executed": 0, 2024-04-25T00:26:16.365 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_item_in_journal": 0 2024-04-25T00:26:16.365 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.365 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T00:26:16.365 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-25T00:26:16.365 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-25T00:26:16.365 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-25T00:26:16.365 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 1533, 2024-04-25T00:26:16.365 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 2362984, 2024-04-25T00:26:16.365 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:16.365 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 1533, 2024-04-25T00:26:16.365 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-25T00:26:16.365 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-25T00:26:16.366 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 1533, 2024-04-25T00:26:16.366 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 2362984, 2024-04-25T00:26:16.366 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-25T00:26:16.366 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.366 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.366 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.366 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:16.366 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.366 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-objecter_bytes": { 2024-04-25T00:26:16.366 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-25T00:26:16.366 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-25T00:26:16.366 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-25T00:26:16.366 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-25T00:26:16.367 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-25T00:26:16.367 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:16.367 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:16.367 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 1223, 2024-04-25T00:26:16.367 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 73854, 2024-04-25T00:26:16.367 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 1223, 2024-04-25T00:26:16.367 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 73854, 2024-04-25T00:26:16.368 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-25T00:26:16.368 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.368 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.368 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.368 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:16.368 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.368 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-objecter_ops": { 2024-04-25T00:26:16.368 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-25T00:26:16.368 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1024, 2024-04-25T00:26:16.368 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-25T00:26:16.368 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-25T00:26:16.368 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-25T00:26:16.368 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:16.368 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:16.369 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 1223, 2024-04-25T00:26:16.369 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 1223, 2024-04-25T00:26:16.369 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 1223, 2024-04-25T00:26:16.369 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 1223, 2024-04-25T00:26:16.369 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-25T00:26:16.369 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.369 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.369 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.369 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:16.369 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.369 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-write_buf_throttle": { 2024-04-25T00:26:16.369 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-25T00:26:16.369 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 3758096384, 2024-04-25T00:26:16.370 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-25T00:26:16.370 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-25T00:26:16.370 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-25T00:26:16.370 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:16.370 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:16.370 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-25T00:26:16.370 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-25T00:26:16.370 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 0, 2024-04-25T00:26:16.370 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 0, 2024-04-25T00:26:16.370 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-25T00:26:16.370 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.370 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.370 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.370 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:16.371 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:16.371 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-write_buf_throttle-0x5577f752c6e0": { 2024-04-25T00:26:16.371 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-25T00:26:16.371 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 3758096384, 2024-04-25T00:26:16.371 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-25T00:26:16.371 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-25T00:26:16.371 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-25T00:26:16.371 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:16.371 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:16.371 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-25T00:26:16.371 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-25T00:26:16.371 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 0, 2024-04-25T00:26:16.371 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 0, 2024-04-25T00:26:16.371 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-25T00:26:16.371 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:16.372 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:16.372 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:16.372 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:16.372 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:16.372 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:26:17.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:16 smithi027 ceph-mon[29208]: pgmap v319: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T00:26:17.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:16 smithi139 ceph-mon[32833]: pgmap v319: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T00:26:17.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:16 smithi153 ceph-mon[32896]: pgmap v319: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T00:26:17.737 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-04-25T00:26:17.737 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-04-25T00:26:17.737 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-04-25T00:26:17.737 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-04-25T00:26:17.737 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T00:26:17.737 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=0 2024-04-25T00:26:17.738 DEBUG:tasks.check_counter:Getting stats from j 2024-04-25T00:26:17.738 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-25T00:26:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T00:26:18.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:17 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:18.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:17 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:18.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:17 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:18.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:17 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:18.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:17 smithi027 ceph-mon[29208]: pgmap v320: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T00:26:18.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:17 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:17 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:17 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.140 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:17 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.140 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:17 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-25T00:26:18.140 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:17 smithi027 ceph-mon[29208]: Unable to set osd_memory_target on smithi153 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-25T00:26:18.140 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:17 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.140 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:17 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.140 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:17 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.140 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:17 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.140 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:17 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi139 to 176.2M 2024-04-25T00:26:18.140 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:17 smithi027 ceph-mon[29208]: Unable to set osd_memory_target on smithi139 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T00:26:18.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:17 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:18.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:17 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:18.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:17 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:18.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:17 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:18.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:17 smithi139 ceph-mon[32833]: pgmap v320: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T00:26:18.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:17 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:17 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:17 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:17 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:17 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-25T00:26:18.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:17 smithi139 ceph-mon[32833]: Unable to set osd_memory_target on smithi153 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-25T00:26:18.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:17 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:17 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:17 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:17 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:17 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi139 to 176.2M 2024-04-25T00:26:18.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:17 smithi139 ceph-mon[32833]: Unable to set osd_memory_target on smithi139 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T00:26:18.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:17 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:18.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:17 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:18.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:17 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:18.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:17 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:18.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:17 smithi153 ceph-mon[32896]: pgmap v320: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T00:26:18.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:17 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:17 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:17 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:17 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:17 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-25T00:26:18.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:17 smithi153 ceph-mon[32896]: Unable to set osd_memory_target on smithi153 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-25T00:26:18.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:17 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:17 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:17 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:17 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:18.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:17 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi139 to 176.2M 2024-04-25T00:26:18.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:17 smithi153 ceph-mon[32896]: Unable to set osd_memory_target on smithi139 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T00:26:19.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:19 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:19.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:19 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:19.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:19 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:19.753 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:19 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:19.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:19 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:19.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:19 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:19.849 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T00:26:19.849 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T00:26:19.849 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 230, 2024-04-25T00:26:19.850 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 219, 2024-04-25T00:26:19.850 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 1583168, 2024-04-25T00:26:19.850 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 134464, 2024-04-25T00:26:19.850 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 4, 2024-04-25T00:26:19.850 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 0, 2024-04-25T00:26:19.850 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.044924354, 2024-04-25T00:26:19.850 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.013884196, 2024-04-25T00:26:19.850 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.035932714, 2024-04-25T00:26:19.850 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.004827390, 2024-04-25T00:26:19.850 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:19.850 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 218, 2024-04-25T00:26:19.850 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.013616981, 2024-04-25T00:26:19.850 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000062463 2024-04-25T00:26:19.851 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:19.851 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:19.851 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:19.851 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:19.851 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:19.851 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:19.851 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 1583168, 2024-04-25T00:26:19.851 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 134464 2024-04-25T00:26:19.851 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:19.851 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T00:26:19.851 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 6, 2024-04-25T00:26:19.851 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 169, 2024-04-25T00:26:19.851 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 3344, 2024-04-25T00:26:19.851 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 185248, 2024-04-25T00:26:19.852 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 5, 2024-04-25T00:26:19.852 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 3, 2024-04-25T00:26:19.852 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.015132802, 2024-04-25T00:26:19.852 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.010011108, 2024-04-25T00:26:19.852 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.002699474, 2024-04-25T00:26:19.852 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T00:26:19.852 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:19.852 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 167, 2024-04-25T00:26:19.852 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.014094616, 2024-04-25T00:26:19.852 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000084398 2024-04-25T00:26:19.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:19.852 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:19.852 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:19.852 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:19.852 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:19.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:19.853 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-25T00:26:19.853 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 185248 2024-04-25T00:26:19.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:19.853 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T00:26:19.853 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 0, 2024-04-25T00:26:19.853 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 2, 2024-04-25T00:26:19.853 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 0, 2024-04-25T00:26:19.853 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 0, 2024-04-25T00:26:19.853 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 2, 2024-04-25T00:26:19.854 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 1, 2024-04-25T00:26:19.854 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.000338501, 2024-04-25T00:26:19.854 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T00:26:19.854 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.000261916, 2024-04-25T00:26:19.854 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T00:26:19.854 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:19.854 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:19.854 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:19.854 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:19.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:19.854 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:19.854 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:19.854 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:19.855 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:19.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:19.855 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T00:26:19.855 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T00:26:19.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:19.855 INFO:teuthology.orchestra.run.smithi027.stdout: "cct": { 2024-04-25T00:26:19.855 INFO:teuthology.orchestra.run.smithi027.stdout: "total_workers": 0, 2024-04-25T00:26:19.855 INFO:teuthology.orchestra.run.smithi027.stdout: "unhealthy_workers": 0 2024-04-25T00:26:19.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:19.855 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool": { 2024-04-25T00:26:19.855 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_bytes": 0, 2024-04-25T00:26:19.855 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_items": 0, 2024-04-25T00:26:19.855 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T00:26:19.855 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_items": 0, 2024-04-25T00:26:19.855 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T00:26:19.856 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_items": 0, 2024-04-25T00:26:19.856 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T00:26:19.856 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T00:26:19.856 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T00:26:19.856 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T00:26:19.856 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T00:26:19.856 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_items": 0, 2024-04-25T00:26:19.856 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T00:26:19.856 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T00:26:19.856 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_bytes": 0, 2024-04-25T00:26:19.856 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_items": 0, 2024-04-25T00:26:19.856 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_bytes": 0, 2024-04-25T00:26:19.856 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_items": 0, 2024-04-25T00:26:19.856 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T00:26:19.857 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T00:26:19.857 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T00:26:19.857 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T00:26:19.857 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T00:26:19.857 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_items": 0, 2024-04-25T00:26:19.857 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_bytes": 0, 2024-04-25T00:26:19.857 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_items": 0, 2024-04-25T00:26:19.857 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T00:26:19.857 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T00:26:19.857 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_bytes": 0, 2024-04-25T00:26:19.857 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_items": 0, 2024-04-25T00:26:19.857 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_bytes": 0, 2024-04-25T00:26:19.857 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_items": 0, 2024-04-25T00:26:19.858 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T00:26:19.858 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_items": 0, 2024-04-25T00:26:19.858 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T00:26:19.858 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_items": 0, 2024-04-25T00:26:19.858 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_bytes": 149017, 2024-04-25T00:26:19.858 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_items": 31, 2024-04-25T00:26:19.858 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_bytes": 0, 2024-04-25T00:26:19.858 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_items": 0, 2024-04-25T00:26:19.858 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bytes": 0, 2024-04-25T00:26:19.858 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_items": 0, 2024-04-25T00:26:19.858 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_bytes": 0, 2024-04-25T00:26:19.858 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_items": 0, 2024-04-25T00:26:19.858 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_bytes": 0, 2024-04-25T00:26:19.859 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_items": 0, 2024-04-25T00:26:19.859 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_bytes": 0, 2024-04-25T00:26:19.859 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_items": 0, 2024-04-25T00:26:19.859 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T00:26:19.859 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_items": 0, 2024-04-25T00:26:19.859 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_bytes": 0, 2024-04-25T00:26:19.859 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_items": 0, 2024-04-25T00:26:19.859 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_bytes": 928, 2024-04-25T00:26:19.859 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_items": 2, 2024-04-25T00:26:19.859 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_bytes": 0, 2024-04-25T00:26:19.859 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_items": 0, 2024-04-25T00:26:19.859 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_bytes": 0, 2024-04-25T00:26:19.859 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_items": 0 2024-04-25T00:26:19.860 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T00:26:19.860 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T00:26:19.860 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-25T00:26:19.860 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-25T00:26:19.860 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-25T00:26:19.860 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 232, 2024-04-25T00:26:19.860 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 1550760, 2024-04-25T00:26:19.860 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:19.860 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 232, 2024-04-25T00:26:19.860 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-25T00:26:19.860 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-25T00:26:19.860 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 232, 2024-04-25T00:26:19.860 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 1550760, 2024-04-25T00:26:19.860 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-25T00:26:19.860 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T00:26:19.861 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T00:26:19.861 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T00:26:19.861 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:19.861 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T00:26:19.861 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T00:26:20.550 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-25T00:26:20.550 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-25T00:26:20.550 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-25T00:26:20.550 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-25T00:26:20.550 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-25T00:26:20.550 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-04-25T00:26:20.550 DEBUG:tasks.check_counter:Getting stats from b 2024-04-25T00:26:20.551 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-25T00:26:20.609 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:20 smithi139 ceph-mon[32833]: pgmap v321: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 197 B/s wr, 10 op/s 2024-04-25T00:26:20.609 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:20 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:20.609 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:20 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:20.609 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:20 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:20.609 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:20 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:20.609 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:20 smithi139 ceph-mon[32833]: Adjusting osd_memory_target on smithi027 to 176.2M 2024-04-25T00:26:20.609 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:20 smithi139 ceph-mon[32833]: Unable to set osd_memory_target on smithi027 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-04-25T00:26:20.609 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:20 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:26:20.609 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:20 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:26:20.609 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:20 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:20.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:20 smithi027 ceph-mon[29208]: pgmap v321: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 197 B/s wr, 10 op/s 2024-04-25T00:26:20.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:20 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:20.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:20 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:20.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:20 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:20.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:20 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:20.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:20 smithi027 ceph-mon[29208]: Adjusting osd_memory_target on smithi027 to 176.2M 2024-04-25T00:26:20.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:20 smithi027 ceph-mon[29208]: Unable to set osd_memory_target on smithi027 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-04-25T00:26:20.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:20 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:26:20.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:20 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:26:20.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:20 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:20.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:20 smithi153 ceph-mon[32896]: pgmap v321: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 197 B/s wr, 10 op/s 2024-04-25T00:26:20.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:20 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:20.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:20 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:20.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:20 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:20.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:20 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T00:26:20.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:20 smithi153 ceph-mon[32896]: Adjusting osd_memory_target on smithi027 to 176.2M 2024-04-25T00:26:20.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:20 smithi153 ceph-mon[32896]: Unable to set osd_memory_target on smithi027 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-04-25T00:26:20.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:20 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T00:26:20.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:20 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T00:26:20.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:20 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' 2024-04-25T00:26:20.858 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T00:26:22.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:21 smithi139 ceph-mon[32833]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T00:26:22.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:21 smithi153 ceph-mon[32896]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T00:26:22.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:21 smithi027 ceph-mon[29208]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T00:26:22.858 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:22 smithi139 ceph-mon[32833]: pgmap v323: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T00:26:22.859 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-04-25T00:26:22.859 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T00:26:22.859 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 5912, 2024-04-25T00:26:22.859 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 5784, 2024-04-25T00:26:22.859 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 2106823, 2024-04-25T00:26:22.859 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 27006507, 2024-04-25T00:26:22.859 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 27, 2024-04-25T00:26:22.859 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 4, 2024-04-25T00:26:22.860 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.612683927, 2024-04-25T00:26:22.860 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.172399306, 2024-04-25T00:26:22.860 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.167538518, 2024-04-25T00:26:22.860 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.251106672, 2024-04-25T00:26:22.860 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:22.860 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 5788, 2024-04-25T00:26:22.860 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.417803462, 2024-04-25T00:26:22.860 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000072184 2024-04-25T00:26:22.860 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.860 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:22.860 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 3830, 2024-04-25T00:26:22.860 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000646845, 2024-04-25T00:26:22.860 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000168 2024-04-25T00:26:22.861 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.861 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T00:26:22.861 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 321888 2024-04-25T00:26:22.861 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.861 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T00:26:22.861 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 31033, 2024-04-25T00:26:22.861 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 35029, 2024-04-25T00:26:22.861 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 13506771, 2024-04-25T00:26:22.861 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 28694090, 2024-04-25T00:26:22.861 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 15, 2024-04-25T00:26:22.861 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 5, 2024-04-25T00:26:22.861 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 2.154789661, 2024-04-25T00:26:22.861 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.790353305, 2024-04-25T00:26:22.861 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 2.252964527, 2024-04-25T00:26:22.861 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.183928546, 2024-04-25T00:26:22.862 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:22.862 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 35031, 2024-04-25T00:26:22.862 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 2.080940457, 2024-04-25T00:26:22.862 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000059402 2024-04-25T00:26:22.862 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.862 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:22.862 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 58700, 2024-04-25T00:26:22.862 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.010121555, 2024-04-25T00:26:22.862 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000172 2024-04-25T00:26:22.862 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.862 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 12384592, 2024-04-25T00:26:22.862 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 15561936 2024-04-25T00:26:22.862 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.863 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T00:26:22.863 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 5995, 2024-04-25T00:26:22.863 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 5165, 2024-04-25T00:26:22.863 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 2056737, 2024-04-25T00:26:22.863 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 17330117, 2024-04-25T00:26:22.863 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 100, 2024-04-25T00:26:22.863 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 11, 2024-04-25T00:26:22.863 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.614268096, 2024-04-25T00:26:22.863 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.199760833, 2024-04-25T00:26:22.863 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.194714836, 2024-04-25T00:26:22.863 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.197174864, 2024-04-25T00:26:22.863 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:22.863 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 5154, 2024-04-25T00:26:22.863 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.921931762, 2024-04-25T00:26:22.864 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000178876 2024-04-25T00:26:22.864 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.864 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:22.864 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2860, 2024-04-25T00:26:22.864 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000532201, 2024-04-25T00:26:22.864 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000186 2024-04-25T00:26:22.864 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.864 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 398624, 2024-04-25T00:26:22.864 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 192800 2024-04-25T00:26:22.864 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.864 INFO:teuthology.orchestra.run.smithi139.stdout: "cct": { 2024-04-25T00:26:22.864 INFO:teuthology.orchestra.run.smithi139.stdout: "total_workers": 1, 2024-04-25T00:26:22.864 INFO:teuthology.orchestra.run.smithi139.stdout: "unhealthy_workers": 0 2024-04-25T00:26:22.864 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.864 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-MDSRank": { 2024-04-25T00:26:22.865 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-04-25T00:26:22.865 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-04-25T00:26:22.865 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 4389, 2024-04-25T00:26:22.865 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 7.098120934, 2024-04-25T00:26:22.865 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.001617252 2024-04-25T00:26:22.865 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:22.865 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.865 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-PurgeQueue": { 2024-04-25T00:26:22.865 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-04-25T00:26:22.865 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-04-25T00:26:22.865 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 5777, 2024-04-25T00:26:22.865 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.284720762, 2024-04-25T00:26:22.865 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000049285 2024-04-25T00:26:22.865 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:22.866 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.866 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": { 2024-04-25T00:26:22.866 INFO:teuthology.orchestra.run.smithi139.stdout: "request": 2569, 2024-04-25T00:26:22.866 INFO:teuthology.orchestra.run.smithi139.stdout: "reply": 2560, 2024-04-25T00:26:22.866 INFO:teuthology.orchestra.run.smithi139.stdout: "reply_latency": { 2024-04-25T00:26:22.866 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2560, 2024-04-25T00:26:22.866 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 302.455013580, 2024-04-25T00:26:22.866 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.118146489 2024-04-25T00:26:22.866 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.866 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_reply": 1, 2024-04-25T00:26:22.866 INFO:teuthology.orchestra.run.smithi139.stdout: "forward": 9, 2024-04-25T00:26:22.866 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_complete": 0, 2024-04-25T00:26:22.866 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_keys": 0, 2024-04-25T00:26:22.866 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_commit": 21, 2024-04-25T00:26:22.867 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_split": 1, 2024-04-25T00:26:22.867 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_merge": 0, 2024-04-25T00:26:22.867 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes": 1111, 2024-04-25T00:26:22.867 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_top": 2, 2024-04-25T00:26:22.867 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_bottom": 1029, 2024-04-25T00:26:22.867 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pin_tail": 80, 2024-04-25T00:26:22.867 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pinned": 1109, 2024-04-25T00:26:22.867 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_expired": 28, 2024-04-25T00:26:22.867 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_with_caps": 3, 2024-04-25T00:26:22.867 INFO:teuthology.orchestra.run.smithi139.stdout: "caps": 3, 2024-04-25T00:26:22.867 INFO:teuthology.orchestra.run.smithi139.stdout: "subtrees": 4, 2024-04-25T00:26:22.867 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse": 3341, 2024-04-25T00:26:22.867 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_hit": 3111, 2024-04-25T00:26:22.867 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_forward": 0, 2024-04-25T00:26:22.867 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_discover": 0, 2024-04-25T00:26:22.868 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_dir_fetch": 0, 2024-04-25T00:26:22.868 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_remote_ino": 0, 2024-04-25T00:26:22.868 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_lock": 51, 2024-04-25T00:26:22.868 INFO:teuthology.orchestra.run.smithi139.stdout: "load_cent": 0, 2024-04-25T00:26:22.868 INFO:teuthology.orchestra.run.smithi139.stdout: "q": 0, 2024-04-25T00:26:22.868 INFO:teuthology.orchestra.run.smithi139.stdout: "exported": 9, 2024-04-25T00:26:22.868 INFO:teuthology.orchestra.run.smithi139.stdout: "exported_inodes": 19, 2024-04-25T00:26:22.868 INFO:teuthology.orchestra.run.smithi139.stdout: "imported": 9, 2024-04-25T00:26:22.868 INFO:teuthology.orchestra.run.smithi139.stdout: "imported_inodes": 29, 2024-04-25T00:26:22.868 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_dir_fetch": 0, 2024-04-25T00:26:22.868 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_backtrace_fetch": 900, 2024-04-25T00:26:22.868 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_peer_discover": 0, 2024-04-25T00:26:22.868 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rfiles": 4, 2024-04-25T00:26:22.868 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rbytes": 234, 2024-04-25T00:26:22.869 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rsnaps": 6, 2024-04-25T00:26:22.869 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_fetch": 900, 2024-04-25T00:26:22.869 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_set_tag": 0, 2024-04-25T00:26:22.869 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T00:26:22.869 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_inotable_repaired": 0, 2024-04-25T00:26:22.869 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_inodes": 238, 2024-04-25T00:26:22.869 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_base_inodes": 10, 2024-04-25T00:26:22.869 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dirfrag_rstats": 228, 2024-04-25T00:26:22.869 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_file_inodes": 662, 2024-04-25T00:26:22.869 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_inode_file_caps": 19, 2024-04-25T00:26:22.869 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_revoke": 345, 2024-04-25T00:26:22.869 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_grant": 3964, 2024-04-25T00:26:22.869 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_trunc": 6, 2024-04-25T00:26:22.869 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T00:26:22.870 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T00:26:22.870 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps": 24245, 2024-04-25T00:26:22.870 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps_dirty": 23104, 2024-04-25T00:26:22.870 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_cap_release": 101, 2024-04-25T00:26:22.870 INFO:teuthology.orchestra.run.smithi139.stdout: "process_request_cap_release": 1334 2024-04-25T00:26:22.870 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.870 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_cache": { 2024-04-25T00:26:22.870 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update": 0, 2024-04-25T00:26:22.870 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update_receipt": 0, 2024-04-25T00:26:22.870 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_try_discover": 0, 2024-04-25T00:26:22.870 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_send_discover": 0, 2024-04-25T00:26:22.870 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_handle_discover": 18, 2024-04-25T00:26:22.870 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays": 83, 2024-04-25T00:26:22.870 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_delayed": 0, 2024-04-25T00:26:22.871 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_enqueuing": 0, 2024-04-25T00:26:22.871 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_created": 924, 2024-04-25T00:26:22.871 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_enqueued": 780, 2024-04-25T00:26:22.871 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_reintegrated": 61, 2024-04-25T00:26:22.871 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_migrated": 0, 2024-04-25T00:26:22.871 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_processing": 0, 2024-04-25T00:26:22.871 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_enqueued": 0, 2024-04-25T00:26:22.871 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_prioritized": 0, 2024-04-25T00:26:22.871 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_started": 0, 2024-04-25T00:26:22.871 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_completed": 0, 2024-04-25T00:26:22.871 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_path": 0, 2024-04-25T00:26:22.872 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_inode": 0, 2024-04-25T00:26:22.872 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_enqueue_scrub": 10, 2024-04-25T00:26:22.872 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_exportdir": 9, 2024-04-25T00:26:22.872 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_flush": 0, 2024-04-25T00:26:22.872 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragmentdir": 1, 2024-04-25T00:26:22.872 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragstats": 0, 2024-04-25T00:26:22.872 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_inodestats": 0 2024-04-25T00:26:22.872 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.872 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_log": { 2024-04-25T00:26:22.872 INFO:teuthology.orchestra.run.smithi139.stdout: "evlrg": 0, 2024-04-25T00:26:22.872 INFO:teuthology.orchestra.run.smithi139.stdout: "evadd": 26845, 2024-04-25T00:26:22.872 INFO:teuthology.orchestra.run.smithi139.stdout: "evex": 0, 2024-04-25T00:26:22.872 INFO:teuthology.orchestra.run.smithi139.stdout: "evtrm": 0, 2024-04-25T00:26:22.872 INFO:teuthology.orchestra.run.smithi139.stdout: "ev": 26845, 2024-04-25T00:26:22.873 INFO:teuthology.orchestra.run.smithi139.stdout: "evexg": 0, 2024-04-25T00:26:22.873 INFO:teuthology.orchestra.run.smithi139.stdout: "evexd": 0, 2024-04-25T00:26:22.873 INFO:teuthology.orchestra.run.smithi139.stdout: "segadd": 31, 2024-04-25T00:26:22.873 INFO:teuthology.orchestra.run.smithi139.stdout: "segex": 0, 2024-04-25T00:26:22.873 INFO:teuthology.orchestra.run.smithi139.stdout: "segtrm": 0, 2024-04-25T00:26:22.873 INFO:teuthology.orchestra.run.smithi139.stdout: "seg": 31, 2024-04-25T00:26:22.873 INFO:teuthology.orchestra.run.smithi139.stdout: "segmjr": 4, 2024-04-25T00:26:22.873 INFO:teuthology.orchestra.run.smithi139.stdout: "segexg": 0, 2024-04-25T00:26:22.873 INFO:teuthology.orchestra.run.smithi139.stdout: "segexd": 0, 2024-04-25T00:26:22.873 INFO:teuthology.orchestra.run.smithi139.stdout: "expos": 4194304, 2024-04-25T00:26:22.873 INFO:teuthology.orchestra.run.smithi139.stdout: "wrpos": 62296858, 2024-04-25T00:26:22.873 INFO:teuthology.orchestra.run.smithi139.stdout: "rdpos": 0, 2024-04-25T00:26:22.873 INFO:teuthology.orchestra.run.smithi139.stdout: "jlat": { 2024-04-25T00:26:22.874 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2073, 2024-04-25T00:26:22.874 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 1553.717098485, 2024-04-25T00:26:22.874 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.749501735 2024-04-25T00:26:22.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.874 INFO:teuthology.orchestra.run.smithi139.stdout: "replayed": 0 2024-04-25T00:26:22.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.874 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_mem": { 2024-04-25T00:26:22.874 INFO:teuthology.orchestra.run.smithi139.stdout: "ino": 427, 2024-04-25T00:26:22.874 INFO:teuthology.orchestra.run.smithi139.stdout: "ino+": 1227, 2024-04-25T00:26:22.874 INFO:teuthology.orchestra.run.smithi139.stdout: "ino-": 800, 2024-04-25T00:26:22.874 INFO:teuthology.orchestra.run.smithi139.stdout: "dir": 108, 2024-04-25T00:26:22.874 INFO:teuthology.orchestra.run.smithi139.stdout: "dir+": 125, 2024-04-25T00:26:22.874 INFO:teuthology.orchestra.run.smithi139.stdout: "dir-": 17, 2024-04-25T00:26:22.875 INFO:teuthology.orchestra.run.smithi139.stdout: "dn": 1111, 2024-04-25T00:26:22.875 INFO:teuthology.orchestra.run.smithi139.stdout: "dn+": 2034, 2024-04-25T00:26:22.875 INFO:teuthology.orchestra.run.smithi139.stdout: "dn-": 923, 2024-04-25T00:26:22.875 INFO:teuthology.orchestra.run.smithi139.stdout: "cap": 3, 2024-04-25T00:26:22.875 INFO:teuthology.orchestra.run.smithi139.stdout: "cap+": 891, 2024-04-25T00:26:22.875 INFO:teuthology.orchestra.run.smithi139.stdout: "cap-": 888, 2024-04-25T00:26:22.875 INFO:teuthology.orchestra.run.smithi139.stdout: "rss": 88704, 2024-04-25T00:26:22.875 INFO:teuthology.orchestra.run.smithi139.stdout: "heap": 231804 2024-04-25T00:26:22.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.875 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_server": { 2024-04-25T00:26:22.875 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_client_request": 3308, 2024-04-25T00:26:22.875 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_server_request": 19, 2024-04-25T00:26:22.876 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_request": 2569, 2024-04-25T00:26:22.876 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_session": 49, 2024-04-25T00:26:22.876 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_peer_request": 56, 2024-04-25T00:26:22.876 INFO:teuthology.orchestra.run.smithi139.stdout: "req_create_latency": { 2024-04-25T00:26:22.876 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 765, 2024-04-25T00:26:22.876 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.850624888, 2024-04-25T00:26:22.876 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.001111927 2024-04-25T00:26:22.876 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.876 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getattr_latency": { 2024-04-25T00:26:22.876 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 147, 2024-04-25T00:26:22.876 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.051604335, 2024-04-25T00:26:22.876 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000351049 2024-04-25T00:26:22.876 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.876 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getfilelock_latency": { 2024-04-25T00:26:22.877 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:22.877 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:22.877 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:22.877 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.877 INFO:teuthology.orchestra.run.smithi139.stdout: "req_link_latency": { 2024-04-25T00:26:22.877 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 61, 2024-04-25T00:26:22.877 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.054463537, 2024-04-25T00:26:22.877 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000892844 2024-04-25T00:26:22.877 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.877 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookup_latency": { 2024-04-25T00:26:22.877 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 213, 2024-04-25T00:26:22.877 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.066851167, 2024-04-25T00:26:22.877 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000313855 2024-04-25T00:26:22.877 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.878 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookuphash_latency": { 2024-04-25T00:26:22.878 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:22.878 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:22.878 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:22.878 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.878 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupino_latency": { 2024-04-25T00:26:22.878 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1, 2024-04-25T00:26:22.878 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000423891, 2024-04-25T00:26:22.878 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000423891 2024-04-25T00:26:22.878 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.878 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupname_latency": { 2024-04-25T00:26:22.878 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:22.878 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:22.878 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:22.879 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.879 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupparent_latency": { 2024-04-25T00:26:22.879 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:22.879 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:22.879 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:22.879 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.879 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupsnap_latency": { 2024-04-25T00:26:22.879 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:22.879 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:22.879 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:22.879 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.879 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lssnap_latency": { 2024-04-25T00:26:22.880 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 7, 2024-04-25T00:26:22.880 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.001247211, 2024-04-25T00:26:22.880 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000178173 2024-04-25T00:26:22.880 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.880 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mkdir_latency": { 2024-04-25T00:26:22.880 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 105, 2024-04-25T00:26:22.880 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.103845438, 2024-04-25T00:26:22.880 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000989004 2024-04-25T00:26:22.880 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.881 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mknod_latency": { 2024-04-25T00:26:22.881 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:22.881 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:22.881 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:22.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.881 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mksnap_latency": { 2024-04-25T00:26:22.881 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 7, 2024-04-25T00:26:22.881 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 286.209403040, 2024-04-25T00:26:22.881 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 40.887057577 2024-04-25T00:26:22.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.881 INFO:teuthology.orchestra.run.smithi139.stdout: "req_open_latency": { 2024-04-25T00:26:22.881 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 49, 2024-04-25T00:26:22.881 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.009797553, 2024-04-25T00:26:22.881 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000199950 2024-04-25T00:26:22.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.882 INFO:teuthology.orchestra.run.smithi139.stdout: "req_readdir_latency": { 2024-04-25T00:26:22.882 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 166, 2024-04-25T00:26:22.882 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 1.044508364, 2024-04-25T00:26:22.882 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.006292219 2024-04-25T00:26:22.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.882 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rename_latency": { 2024-04-25T00:26:22.882 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 113, 2024-04-25T00:26:22.882 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 10.975338082, 2024-04-25T00:26:22.882 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.097126885 2024-04-25T00:26:22.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.882 INFO:teuthology.orchestra.run.smithi139.stdout: "req_renamesnap_latency": { 2024-04-25T00:26:22.882 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:22.882 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:22.882 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:22.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.883 INFO:teuthology.orchestra.run.smithi139.stdout: "req_snapdiff_latency": { 2024-04-25T00:26:22.883 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:22.883 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:22.883 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:22.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.883 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmdir_latency": { 2024-04-25T00:26:22.883 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 99, 2024-04-25T00:26:22.883 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 1.786016227, 2024-04-25T00:26:22.883 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.018040567 2024-04-25T00:26:22.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.883 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmsnap_latency": { 2024-04-25T00:26:22.883 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1, 2024-04-25T00:26:22.883 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.005179192, 2024-04-25T00:26:22.884 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.005179192 2024-04-25T00:26:22.884 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.884 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmxattr_latency": { 2024-04-25T00:26:22.884 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:22.884 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:22.884 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:22.884 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.884 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setattr_latency": { 2024-04-25T00:26:22.884 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 8, 2024-04-25T00:26:22.884 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.031581750, 2024-04-25T00:26:22.884 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.003947718 2024-04-25T00:26:22.884 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.884 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setdirlayout_latency": { 2024-04-25T00:26:22.884 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:22.884 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setfilelock_latency": { 2024-04-25T00:26:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 4, 2024-04-25T00:26:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.001082901, 2024-04-25T00:26:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000270725 2024-04-25T00:26:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setlayout_latency": { 2024-04-25T00:26:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setxattr_latency": { 2024-04-25T00:26:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 9, 2024-04-25T00:26:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.021074905, 2024-04-25T00:26:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.002341656 2024-04-25T00:26:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: "req_symlink_latency": { 2024-04-25T00:26:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 13, 2024-04-25T00:26:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.008120976, 2024-04-25T00:26:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000624690 2024-04-25T00:26:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: "req_unlink_latency": { 2024-04-25T00:26:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 792, 2024-04-25T00:26:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 1.233850123, 2024-04-25T00:26:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.001557891 2024-04-25T00:26:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_revoke_eviction": 0, 2024-04-25T00:26:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_acquisition_throttle": 0, 2024-04-25T00:26:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getvxattr_latency": { 2024-04-25T00:26:22.887 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:22.887 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:22.887 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:22.887 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:22.887 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.887 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_sessions": { 2024-04-25T00:26:22.887 INFO:teuthology.orchestra.run.smithi139.stdout: "session_count": 2, 2024-04-25T00:26:22.887 INFO:teuthology.orchestra.run.smithi139.stdout: "session_add": 7, 2024-04-25T00:26:22.887 INFO:teuthology.orchestra.run.smithi139.stdout: "session_remove": 5, 2024-04-25T00:26:22.887 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_open": 2, 2024-04-25T00:26:22.887 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_stale": 0, 2024-04-25T00:26:22.887 INFO:teuthology.orchestra.run.smithi139.stdout: "total_load": 96, 2024-04-25T00:26:22.887 INFO:teuthology.orchestra.run.smithi139.stdout: "average_load": 48, 2024-04-25T00:26:22.888 INFO:teuthology.orchestra.run.smithi139.stdout: "avg_session_uptime": 626, 2024-04-25T00:26:22.888 INFO:teuthology.orchestra.run.smithi139.stdout: "mdthresh_evicted": 0 2024-04-25T00:26:22.888 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.888 INFO:teuthology.orchestra.run.smithi139.stdout: "mempool": { 2024-04-25T00:26:22.888 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_bytes": 0, 2024-04-25T00:26:22.888 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_items": 0, 2024-04-25T00:26:22.888 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T00:26:22.888 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_items": 0, 2024-04-25T00:26:22.888 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T00:26:22.888 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_items": 0, 2024-04-25T00:26:22.888 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T00:26:22.888 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T00:26:22.888 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T00:26:22.888 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T00:26:22.889 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T00:26:22.889 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_items": 0, 2024-04-25T00:26:22.889 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T00:26:22.889 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T00:26:22.889 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_bytes": 0, 2024-04-25T00:26:22.889 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_items": 0, 2024-04-25T00:26:22.889 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_bytes": 0, 2024-04-25T00:26:22.889 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_items": 0, 2024-04-25T00:26:22.889 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T00:26:22.889 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T00:26:22.889 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T00:26:22.889 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T00:26:22.889 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T00:26:22.889 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_items": 0, 2024-04-25T00:26:22.890 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_bytes": 0, 2024-04-25T00:26:22.890 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_items": 0, 2024-04-25T00:26:22.890 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T00:26:22.890 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T00:26:22.890 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_bytes": 0, 2024-04-25T00:26:22.890 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_items": 0, 2024-04-25T00:26:22.890 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_bytes": 0, 2024-04-25T00:26:22.890 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_items": 0, 2024-04-25T00:26:22.890 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T00:26:22.890 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_items": 0, 2024-04-25T00:26:22.890 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T00:26:22.890 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_items": 0, 2024-04-25T00:26:22.890 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_bytes": 996475, 2024-04-25T00:26:22.890 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_items": 712, 2024-04-25T00:26:22.891 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_bytes": 1232, 2024-04-25T00:26:22.891 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_items": 14, 2024-04-25T00:26:22.891 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_bytes": 0, 2024-04-25T00:26:22.891 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_items": 0, 2024-04-25T00:26:22.891 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_bytes": 0, 2024-04-25T00:26:22.891 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_items": 0, 2024-04-25T00:26:22.891 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_bytes": 0, 2024-04-25T00:26:22.891 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_items": 0, 2024-04-25T00:26:22.891 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_bytes": 9752, 2024-04-25T00:26:22.891 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_items": 188, 2024-04-25T00:26:22.891 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T00:26:22.891 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_items": 0, 2024-04-25T00:26:22.891 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_bytes": 0, 2024-04-25T00:26:22.891 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_items": 0, 2024-04-25T00:26:22.891 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_bytes": 2085573, 2024-04-25T00:26:22.891 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_items": 40209, 2024-04-25T00:26:22.892 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_bytes": 0, 2024-04-25T00:26:22.892 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_items": 0, 2024-04-25T00:26:22.892 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_bytes": 0, 2024-04-25T00:26:22.892 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_items": 0 2024-04-25T00:26:22.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.892 INFO:teuthology.orchestra.run.smithi139.stdout: "objecter": { 2024-04-25T00:26:22.892 INFO:teuthology.orchestra.run.smithi139.stdout: "op_active": 0, 2024-04-25T00:26:22.892 INFO:teuthology.orchestra.run.smithi139.stdout: "op_laggy": 0, 2024-04-25T00:26:22.892 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send": 9709, 2024-04-25T00:26:22.892 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send_bytes": 58508794, 2024-04-25T00:26:22.892 INFO:teuthology.orchestra.run.smithi139.stdout: "op_resend": 35, 2024-04-25T00:26:22.892 INFO:teuthology.orchestra.run.smithi139.stdout: "op_reply": 9709, 2024-04-25T00:26:22.892 INFO:teuthology.orchestra.run.smithi139.stdout: "op_latency": { 2024-04-25T00:26:22.892 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 9709, 2024-04-25T00:26:22.892 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 3237.599704446, 2024-04-25T00:26:22.892 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.333463766 2024-04-25T00:26:22.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.893 INFO:teuthology.orchestra.run.smithi139.stdout: "op_inflight": 0, 2024-04-25T00:26:22.893 INFO:teuthology.orchestra.run.smithi139.stdout: "oplen_avg": { 2024-04-25T00:26:22.893 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 9709, 2024-04-25T00:26:22.893 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 10083 2024-04-25T00:26:22.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.893 INFO:teuthology.orchestra.run.smithi139.stdout: "op": 9709, 2024-04-25T00:26:22.893 INFO:teuthology.orchestra.run.smithi139.stdout: "op_r": 937, 2024-04-25T00:26:22.893 INFO:teuthology.orchestra.run.smithi139.stdout: "op_w": 8772, 2024-04-25T00:26:22.893 INFO:teuthology.orchestra.run.smithi139.stdout: "op_rmw": 0, 2024-04-25T00:26:22.893 INFO:teuthology.orchestra.run.smithi139.stdout: "op_pg": 0, 2024-04-25T00:26:22.893 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_stat": 0, 2024-04-25T00:26:22.893 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_create": 11, 2024-04-25T00:26:22.893 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_read": 27, 2024-04-25T00:26:22.893 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_write": 2090, 2024-04-25T00:26:22.893 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writefull": 51, 2024-04-25T00:26:22.894 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writesame": 0, 2024-04-25T00:26:22.894 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_append": 0, 2024-04-25T00:26:22.894 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_zero": 0, 2024-04-25T00:26:22.894 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_truncate": 1, 2024-04-25T00:26:22.894 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_delete": 6324, 2024-04-25T00:26:22.894 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_mapext": 0, 2024-04-25T00:26:22.894 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_sparse_read": 0, 2024-04-25T00:26:22.894 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_clonerange": 0, 2024-04-25T00:26:22.894 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_getxattr": 910, 2024-04-25T00:26:22.894 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_setxattr": 22, 2024-04-25T00:26:22.894 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_cmpxattr": 0, 2024-04-25T00:26:22.894 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_rmxattr": 0, 2024-04-25T00:26:22.894 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_resetxattrs": 0, 2024-04-25T00:26:22.895 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_call": 0, 2024-04-25T00:26:22.895 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_watch": 0, 2024-04-25T00:26:22.895 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_notify": 0, 2024-04-25T00:26:22.895 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T00:26:22.895 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls": 0, 2024-04-25T00:26:22.895 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls_filter": 0, 2024-04-25T00:26:22.895 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_other": 6, 2024-04-25T00:26:22.895 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_active": 0, 2024-04-25T00:26:22.895 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_send": 0, 2024-04-25T00:26:22.895 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_resend": 0, 2024-04-25T00:26:22.895 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_ping": 0, 2024-04-25T00:26:22.895 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_active": 0, 2024-04-25T00:26:22.895 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_send": 0, 2024-04-25T00:26:22.895 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_resend": 0, 2024-04-25T00:26:22.895 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_active": 0, 2024-04-25T00:26:22.895 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_send": 0, 2024-04-25T00:26:22.896 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_resend": 0, 2024-04-25T00:26:22.896 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_active": 0, 2024-04-25T00:26:22.896 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_send": 0, 2024-04-25T00:26:22.896 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_resend": 0, 2024-04-25T00:26:22.896 INFO:teuthology.orchestra.run.smithi139.stdout: "command_active": 0, 2024-04-25T00:26:22.896 INFO:teuthology.orchestra.run.smithi139.stdout: "command_send": 0, 2024-04-25T00:26:22.896 INFO:teuthology.orchestra.run.smithi139.stdout: "command_resend": 0, 2024-04-25T00:26:22.896 INFO:teuthology.orchestra.run.smithi139.stdout: "map_epoch": 96, 2024-04-25T00:26:22.896 INFO:teuthology.orchestra.run.smithi139.stdout: "map_full": 0, 2024-04-25T00:26:22.896 INFO:teuthology.orchestra.run.smithi139.stdout: "map_inc": 12, 2024-04-25T00:26:22.896 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_sessions": 12, 2024-04-25T00:26:22.896 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_open": 12, 2024-04-25T00:26:22.896 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_close": 0, 2024-04-25T00:26:22.896 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_laggy": 0, 2024-04-25T00:26:22.896 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_wr": 473, 2024-04-25T00:26:22.897 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_rd": 0, 2024-04-25T00:26:22.897 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_del": 168 2024-04-25T00:26:22.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.897 INFO:teuthology.orchestra.run.smithi139.stdout: "oft": { 2024-04-25T00:26:22.897 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_objs": 1, 2024-04-25T00:26:22.897 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_kv_pairs": 0, 2024-04-25T00:26:22.897 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_updates": 0, 2024-04-25T00:26:22.897 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_removes": 0 2024-04-25T00:26:22.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.897 INFO:teuthology.orchestra.run.smithi139.stdout: "osd-slow-ops": { 2024-04-25T00:26:22.897 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_ops_count": 1 2024-04-25T00:26:22.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.897 INFO:teuthology.orchestra.run.smithi139.stdout: "purge_queue": { 2024-04-25T00:26:22.897 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops": 0, 2024-04-25T00:26:22.898 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops_high_water": 52, 2024-04-25T00:26:22.898 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing": 0, 2024-04-25T00:26:22.898 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_high_water": 9, 2024-04-25T00:26:22.898 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed_ops": 5799, 2024-04-25T00:26:22.898 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed": 780, 2024-04-25T00:26:22.898 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_item_in_journal": 0 2024-04-25T00:26:22.898 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.898 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T00:26:22.898 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:22.898 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-04-25T00:26:22.898 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:22.898 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 42936, 2024-04-25T00:26:22.898 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 12558346, 2024-04-25T00:26:22.898 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:22.898 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 42936, 2024-04-25T00:26:22.899 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T00:26:22.899 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T00:26:22.899 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 42936, 2024-04-25T00:26:22.899 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 12558346, 2024-04-25T00:26:22.899 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:22.899 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:22.899 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:22.899 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:22.899 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:22.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.899 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_bytes": { 2024-04-25T00:26:22.899 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:22.899 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-04-25T00:26:22.899 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:22.899 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T00:26:22.900 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T00:26:22.900 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:22.900 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:22.900 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 9709, 2024-04-25T00:26:22.900 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 58598558, 2024-04-25T00:26:22.900 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 9709, 2024-04-25T00:26:22.900 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 58598558, 2024-04-25T00:26:22.900 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:22.900 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:22.900 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:22.900 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:22.900 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:22.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.900 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_ops": { 2024-04-25T00:26:22.901 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:22.901 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 1024, 2024-04-25T00:26:22.901 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:22.901 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T00:26:22.901 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T00:26:22.901 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:22.901 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:22.901 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 9709, 2024-04-25T00:26:22.901 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 9709, 2024-04-25T00:26:22.901 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 9709, 2024-04-25T00:26:22.901 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 9709, 2024-04-25T00:26:22.901 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:22.901 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:22.901 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:22.901 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:22.901 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:22.902 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.902 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle": { 2024-04-25T00:26:22.902 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:22.902 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-04-25T00:26:22.902 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:22.902 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 780, 2024-04-25T00:26:22.902 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 89764, 2024-04-25T00:26:22.902 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:22.902 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 780, 2024-04-25T00:26:22.902 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T00:26:22.902 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T00:26:22.902 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 17, 2024-04-25T00:26:22.902 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 89764, 2024-04-25T00:26:22.902 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:22.902 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:22.903 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:22.903 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:22.903 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:22.903 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:22.903 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle-0x55d80265a500": { 2024-04-25T00:26:22.903 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:22.903 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-04-25T00:26:22.903 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:22.903 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 26845, 2024-04-25T00:26:22.903 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 58102554, 2024-04-25T00:26:22.903 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:22.903 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 26845, 2024-04-25T00:26:22.903 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T00:26:22.903 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T00:26:22.903 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 2073, 2024-04-25T00:26:22.904 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 58102554, 2024-04-25T00:26:22.904 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:22.904 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:22.904 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:22.904 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:22.904 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:22.904 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:22.904 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-04-25T00:26:23.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:22 smithi153 ceph-mon[32896]: pgmap v323: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T00:26:23.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:22 smithi027 ceph-mon[29208]: pgmap v323: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T00:26:23.672 INFO:tasks.check_counter:Daemon mds.b mds.exported=9 2024-04-25T00:26:23.672 INFO:tasks.check_counter:Daemon mds.b mds.imported=9 2024-04-25T00:26:23.673 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-25T00:26:23.673 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=7 2024-04-25T00:26:23.673 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=1 2024-04-25T00:26:23.673 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=1 2024-04-25T00:26:23.673 DEBUG:tasks.check_counter:Getting stats from e 2024-04-25T00:26:23.673 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-25T00:26:23.919 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T00:26:25.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:24 smithi153 ceph-mon[32896]: pgmap v324: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T00:26:25.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:24 smithi027 ceph-mon[29208]: pgmap v324: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T00:26:25.212 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:24 smithi139 ceph-mon[32833]: pgmap v324: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T00:26:25.783 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-04-25T00:26:25.783 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T00:26:25.783 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 773, 2024-04-25T00:26:25.783 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 773, 2024-04-25T00:26:25.783 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 233843, 2024-04-25T00:26:25.784 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 245054, 2024-04-25T00:26:25.784 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 6, 2024-04-25T00:26:25.784 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 2, 2024-04-25T00:26:25.784 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.066480232, 2024-04-25T00:26:25.784 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.024141995, 2024-04-25T00:26:25.784 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.016150636, 2024-04-25T00:26:25.784 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.019014987, 2024-04-25T00:26:25.784 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:25.784 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 772, 2024-04-25T00:26:25.784 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.045322329, 2024-04-25T00:26:25.784 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000058707 2024-04-25T00:26:25.784 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.784 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:25.785 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.785 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.785 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.785 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.785 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-25T00:26:25.785 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T00:26:25.785 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.785 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T00:26:25.785 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 675, 2024-04-25T00:26:25.785 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 839, 2024-04-25T00:26:25.785 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 165900, 2024-04-25T00:26:25.785 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 659603, 2024-04-25T00:26:25.785 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 7, 2024-04-25T00:26:25.785 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 5, 2024-04-25T00:26:25.785 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.069262141, 2024-04-25T00:26:25.786 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.021721798, 2024-04-25T00:26:25.786 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.015130560, 2024-04-25T00:26:25.786 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.024607630, 2024-04-25T00:26:25.786 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:25.786 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 836, 2024-04-25T00:26:25.786 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.054638088, 2024-04-25T00:26:25.786 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000065356 2024-04-25T00:26:25.786 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.786 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:25.786 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.786 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.786 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.786 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.787 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T00:26:25.787 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 445584 2024-04-25T00:26:25.787 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.787 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T00:26:25.787 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 1075, 2024-04-25T00:26:25.787 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 986, 2024-04-25T00:26:25.787 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 2374928, 2024-04-25T00:26:25.787 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 376618, 2024-04-25T00:26:25.787 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 3, 2024-04-25T00:26:25.787 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 2, 2024-04-25T00:26:25.787 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.134407547, 2024-04-25T00:26:25.787 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.037736965, 2024-04-25T00:26:25.787 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.066150210, 2024-04-25T00:26:25.787 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.034701152, 2024-04-25T00:26:25.788 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:25.788 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 985, 2024-04-25T00:26:25.788 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.068337992, 2024-04-25T00:26:25.788 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000069378 2024-04-25T00:26:25.788 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.788 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:25.788 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.788 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.788 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.788 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.788 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 2121248, 2024-04-25T00:26:25.788 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 136528 2024-04-25T00:26:25.788 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.788 INFO:teuthology.orchestra.run.smithi139.stdout: "cct": { 2024-04-25T00:26:25.789 INFO:teuthology.orchestra.run.smithi139.stdout: "total_workers": 1, 2024-04-25T00:26:25.789 INFO:teuthology.orchestra.run.smithi139.stdout: "unhealthy_workers": 0 2024-04-25T00:26:25.789 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.789 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-MDSRank": { 2024-04-25T00:26:25.789 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-04-25T00:26:25.789 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-04-25T00:26:25.789 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 3679, 2024-04-25T00:26:25.789 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.113121093, 2024-04-25T00:26:25.789 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000030747 2024-04-25T00:26:25.789 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:25.789 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.789 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-PurgeQueue": { 2024-04-25T00:26:25.789 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-04-25T00:26:25.790 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-04-25T00:26:25.790 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.790 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.790 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.790 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:25.790 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.790 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": { 2024-04-25T00:26:25.790 INFO:teuthology.orchestra.run.smithi139.stdout: "request": 0, 2024-04-25T00:26:25.790 INFO:teuthology.orchestra.run.smithi139.stdout: "reply": 0, 2024-04-25T00:26:25.790 INFO:teuthology.orchestra.run.smithi139.stdout: "reply_latency": { 2024-04-25T00:26:25.790 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.790 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.790 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.790 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.791 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_reply": 0, 2024-04-25T00:26:25.791 INFO:teuthology.orchestra.run.smithi139.stdout: "forward": 0, 2024-04-25T00:26:25.791 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_complete": 0, 2024-04-25T00:26:25.791 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_keys": 0, 2024-04-25T00:26:25.791 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_commit": 0, 2024-04-25T00:26:25.791 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_split": 0, 2024-04-25T00:26:25.791 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_merge": 0, 2024-04-25T00:26:25.791 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes": 2, 2024-04-25T00:26:25.791 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_top": 0, 2024-04-25T00:26:25.791 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_bottom": 2, 2024-04-25T00:26:25.791 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pin_tail": 0, 2024-04-25T00:26:25.791 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pinned": 2, 2024-04-25T00:26:25.791 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_expired": 0, 2024-04-25T00:26:25.792 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_with_caps": 0, 2024-04-25T00:26:25.792 INFO:teuthology.orchestra.run.smithi139.stdout: "caps": 0, 2024-04-25T00:26:25.792 INFO:teuthology.orchestra.run.smithi139.stdout: "subtrees": 1, 2024-04-25T00:26:25.792 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse": 0, 2024-04-25T00:26:25.792 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_hit": 0, 2024-04-25T00:26:25.792 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_forward": 0, 2024-04-25T00:26:25.792 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_discover": 0, 2024-04-25T00:26:25.792 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_dir_fetch": 0, 2024-04-25T00:26:25.792 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_remote_ino": 0, 2024-04-25T00:26:25.792 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_lock": 0, 2024-04-25T00:26:25.792 INFO:teuthology.orchestra.run.smithi139.stdout: "load_cent": 0, 2024-04-25T00:26:25.792 INFO:teuthology.orchestra.run.smithi139.stdout: "q": 0, 2024-04-25T00:26:25.792 INFO:teuthology.orchestra.run.smithi139.stdout: "exported": 0, 2024-04-25T00:26:25.792 INFO:teuthology.orchestra.run.smithi139.stdout: "exported_inodes": 0, 2024-04-25T00:26:25.793 INFO:teuthology.orchestra.run.smithi139.stdout: "imported": 0, 2024-04-25T00:26:25.793 INFO:teuthology.orchestra.run.smithi139.stdout: "imported_inodes": 0, 2024-04-25T00:26:25.793 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_dir_fetch": 0, 2024-04-25T00:26:25.793 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_backtrace_fetch": 0, 2024-04-25T00:26:25.793 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_peer_discover": 0, 2024-04-25T00:26:25.793 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rfiles": 0, 2024-04-25T00:26:25.793 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rbytes": 0, 2024-04-25T00:26:25.793 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rsnaps": 0, 2024-04-25T00:26:25.793 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T00:26:25.793 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_set_tag": 0, 2024-04-25T00:26:25.793 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T00:26:25.793 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_inotable_repaired": 0, 2024-04-25T00:26:25.793 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_inodes": 0, 2024-04-25T00:26:25.793 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T00:26:25.793 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T00:26:25.794 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_file_inodes": 0, 2024-04-25T00:26:25.794 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_inode_file_caps": 0, 2024-04-25T00:26:25.794 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T00:26:25.794 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_grant": 0, 2024-04-25T00:26:25.794 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T00:26:25.794 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T00:26:25.794 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T00:26:25.794 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps": 0, 2024-04-25T00:26:25.794 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps_dirty": 0, 2024-04-25T00:26:25.794 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_cap_release": 0, 2024-04-25T00:26:25.794 INFO:teuthology.orchestra.run.smithi139.stdout: "process_request_cap_release": 0 2024-04-25T00:26:25.794 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.794 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_cache": { 2024-04-25T00:26:25.794 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update": 0, 2024-04-25T00:26:25.795 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update_receipt": 0, 2024-04-25T00:26:25.795 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_try_discover": 0, 2024-04-25T00:26:25.795 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_send_discover": 0, 2024-04-25T00:26:25.795 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_handle_discover": 0, 2024-04-25T00:26:25.795 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays": 0, 2024-04-25T00:26:25.795 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_delayed": 0, 2024-04-25T00:26:25.795 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_enqueuing": 0, 2024-04-25T00:26:25.795 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_created": 0, 2024-04-25T00:26:25.795 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_enqueued": 0, 2024-04-25T00:26:25.795 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_reintegrated": 0, 2024-04-25T00:26:25.795 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_migrated": 0, 2024-04-25T00:26:25.795 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_processing": 0, 2024-04-25T00:26:25.795 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_enqueued": 0, 2024-04-25T00:26:25.795 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_prioritized": 0, 2024-04-25T00:26:25.796 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_started": 0, 2024-04-25T00:26:25.796 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_completed": 0, 2024-04-25T00:26:25.796 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_path": 0, 2024-04-25T00:26:25.796 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_inode": 0, 2024-04-25T00:26:25.796 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T00:26:25.796 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_exportdir": 0, 2024-04-25T00:26:25.796 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_flush": 0, 2024-04-25T00:26:25.796 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragmentdir": 0, 2024-04-25T00:26:25.796 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragstats": 0, 2024-04-25T00:26:25.796 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_inodestats": 0 2024-04-25T00:26:25.796 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.796 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_log": { 2024-04-25T00:26:25.796 INFO:teuthology.orchestra.run.smithi139.stdout: "evlrg": 0, 2024-04-25T00:26:25.796 INFO:teuthology.orchestra.run.smithi139.stdout: "evadd": 0, 2024-04-25T00:26:25.797 INFO:teuthology.orchestra.run.smithi139.stdout: "evex": 0, 2024-04-25T00:26:25.797 INFO:teuthology.orchestra.run.smithi139.stdout: "evtrm": 0, 2024-04-25T00:26:25.797 INFO:teuthology.orchestra.run.smithi139.stdout: "ev": 16, 2024-04-25T00:26:25.797 INFO:teuthology.orchestra.run.smithi139.stdout: "evexg": 0, 2024-04-25T00:26:25.797 INFO:teuthology.orchestra.run.smithi139.stdout: "evexd": 0, 2024-04-25T00:26:25.797 INFO:teuthology.orchestra.run.smithi139.stdout: "segadd": 0, 2024-04-25T00:26:25.797 INFO:teuthology.orchestra.run.smithi139.stdout: "segex": 0, 2024-04-25T00:26:25.797 INFO:teuthology.orchestra.run.smithi139.stdout: "segtrm": 0, 2024-04-25T00:26:25.797 INFO:teuthology.orchestra.run.smithi139.stdout: "seg": 2, 2024-04-25T00:26:25.797 INFO:teuthology.orchestra.run.smithi139.stdout: "segmjr": 2, 2024-04-25T00:26:25.797 INFO:teuthology.orchestra.run.smithi139.stdout: "segexg": 0, 2024-04-25T00:26:25.797 INFO:teuthology.orchestra.run.smithi139.stdout: "segexd": 0, 2024-04-25T00:26:25.797 INFO:teuthology.orchestra.run.smithi139.stdout: "expos": 4194304, 2024-04-25T00:26:25.797 INFO:teuthology.orchestra.run.smithi139.stdout: "wrpos": 4223182, 2024-04-25T00:26:25.798 INFO:teuthology.orchestra.run.smithi139.stdout: "rdpos": 4220577, 2024-04-25T00:26:25.798 INFO:teuthology.orchestra.run.smithi139.stdout: "jlat": { 2024-04-25T00:26:25.798 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.798 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.798 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.798 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.798 INFO:teuthology.orchestra.run.smithi139.stdout: "replayed": 16 2024-04-25T00:26:25.798 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.798 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_mem": { 2024-04-25T00:26:25.798 INFO:teuthology.orchestra.run.smithi139.stdout: "ino": 4, 2024-04-25T00:26:25.798 INFO:teuthology.orchestra.run.smithi139.stdout: "ino+": 18, 2024-04-25T00:26:25.798 INFO:teuthology.orchestra.run.smithi139.stdout: "ino-": 14, 2024-04-25T00:26:25.798 INFO:teuthology.orchestra.run.smithi139.stdout: "dir": 2, 2024-04-25T00:26:25.798 INFO:teuthology.orchestra.run.smithi139.stdout: "dir+": 15, 2024-04-25T00:26:25.799 INFO:teuthology.orchestra.run.smithi139.stdout: "dir-": 13, 2024-04-25T00:26:25.799 INFO:teuthology.orchestra.run.smithi139.stdout: "dn": 2, 2024-04-25T00:26:25.799 INFO:teuthology.orchestra.run.smithi139.stdout: "dn+": 17, 2024-04-25T00:26:25.799 INFO:teuthology.orchestra.run.smithi139.stdout: "dn-": 15, 2024-04-25T00:26:25.799 INFO:teuthology.orchestra.run.smithi139.stdout: "cap": 0, 2024-04-25T00:26:25.799 INFO:teuthology.orchestra.run.smithi139.stdout: "cap+": 0, 2024-04-25T00:26:25.799 INFO:teuthology.orchestra.run.smithi139.stdout: "cap-": 0, 2024-04-25T00:26:25.799 INFO:teuthology.orchestra.run.smithi139.stdout: "rss": 50084, 2024-04-25T00:26:25.799 INFO:teuthology.orchestra.run.smithi139.stdout: "heap": 215420 2024-04-25T00:26:25.799 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.799 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_server": { 2024-04-25T00:26:25.799 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_client_request": 0, 2024-04-25T00:26:25.799 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_server_request": 0, 2024-04-25T00:26:25.799 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_request": 0, 2024-04-25T00:26:25.799 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_session": 0, 2024-04-25T00:26:25.800 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_peer_request": 0, 2024-04-25T00:26:25.800 INFO:teuthology.orchestra.run.smithi139.stdout: "req_create_latency": { 2024-04-25T00:26:25.800 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.800 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.800 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.800 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.800 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getattr_latency": { 2024-04-25T00:26:25.800 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.800 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.800 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.800 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.800 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getfilelock_latency": { 2024-04-25T00:26:25.800 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.800 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.801 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.801 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.801 INFO:teuthology.orchestra.run.smithi139.stdout: "req_link_latency": { 2024-04-25T00:26:25.801 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.801 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.801 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.801 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.801 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookup_latency": { 2024-04-25T00:26:25.801 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.801 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.801 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.801 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.801 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookuphash_latency": { 2024-04-25T00:26:25.801 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupino_latency": { 2024-04-25T00:26:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupname_latency": { 2024-04-25T00:26:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupparent_latency": { 2024-04-25T00:26:25.803 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.803 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.803 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.803 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.803 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupsnap_latency": { 2024-04-25T00:26:25.803 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.803 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.803 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.803 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.803 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lssnap_latency": { 2024-04-25T00:26:25.803 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.803 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.803 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.804 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.804 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mkdir_latency": { 2024-04-25T00:26:25.804 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.804 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.804 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mknod_latency": { 2024-04-25T00:26:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mksnap_latency": { 2024-04-25T00:26:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: "req_open_latency": { 2024-04-25T00:26:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.806 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.806 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.806 INFO:teuthology.orchestra.run.smithi139.stdout: "req_readdir_latency": { 2024-04-25T00:26:25.806 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.806 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.806 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.806 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.806 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rename_latency": { 2024-04-25T00:26:25.806 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.806 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.806 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.806 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.806 INFO:teuthology.orchestra.run.smithi139.stdout: "req_renamesnap_latency": { 2024-04-25T00:26:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: "req_snapdiff_latency": { 2024-04-25T00:26:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmdir_latency": { 2024-04-25T00:26:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.808 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmsnap_latency": { 2024-04-25T00:26:25.808 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.808 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.808 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.808 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.808 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmxattr_latency": { 2024-04-25T00:26:25.808 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.808 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.808 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.808 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.808 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setattr_latency": { 2024-04-25T00:26:25.808 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setdirlayout_latency": { 2024-04-25T00:26:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setfilelock_latency": { 2024-04-25T00:26:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setlayout_latency": { 2024-04-25T00:26:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setxattr_latency": { 2024-04-25T00:26:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: "req_symlink_latency": { 2024-04-25T00:26:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: "req_unlink_latency": { 2024-04-25T00:26:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_revoke_eviction": 0, 2024-04-25T00:26:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_acquisition_throttle": 0, 2024-04-25T00:26:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getvxattr_latency": { 2024-04-25T00:26:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_sessions": { 2024-04-25T00:26:25.812 INFO:teuthology.orchestra.run.smithi139.stdout: "session_count": 2, 2024-04-25T00:26:25.812 INFO:teuthology.orchestra.run.smithi139.stdout: "session_add": 4, 2024-04-25T00:26:25.812 INFO:teuthology.orchestra.run.smithi139.stdout: "session_remove": 2, 2024-04-25T00:26:25.812 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_open": 3, 2024-04-25T00:26:25.812 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_stale": 0, 2024-04-25T00:26:25.812 INFO:teuthology.orchestra.run.smithi139.stdout: "total_load": 0, 2024-04-25T00:26:25.812 INFO:teuthology.orchestra.run.smithi139.stdout: "average_load": 0, 2024-04-25T00:26:25.812 INFO:teuthology.orchestra.run.smithi139.stdout: "avg_session_uptime": 2535, 2024-04-25T00:26:25.812 INFO:teuthology.orchestra.run.smithi139.stdout: "mdthresh_evicted": 0 2024-04-25T00:26:25.812 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.812 INFO:teuthology.orchestra.run.smithi139.stdout: "mempool": { 2024-04-25T00:26:25.812 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_bytes": 0, 2024-04-25T00:26:25.812 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_items": 0, 2024-04-25T00:26:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T00:26:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_items": 0, 2024-04-25T00:26:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T00:26:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_items": 0, 2024-04-25T00:26:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T00:26:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T00:26:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T00:26:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T00:26:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T00:26:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_items": 0, 2024-04-25T00:26:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T00:26:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T00:26:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_bytes": 0, 2024-04-25T00:26:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_items": 0, 2024-04-25T00:26:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_bytes": 0, 2024-04-25T00:26:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_items": 0, 2024-04-25T00:26:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T00:26:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T00:26:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T00:26:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T00:26:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T00:26:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_items": 0, 2024-04-25T00:26:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_bytes": 0, 2024-04-25T00:26:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_items": 0, 2024-04-25T00:26:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T00:26:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T00:26:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_bytes": 0, 2024-04-25T00:26:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_items": 0, 2024-04-25T00:26:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_bytes": 0, 2024-04-25T00:26:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_items": 0, 2024-04-25T00:26:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T00:26:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_items": 0, 2024-04-25T00:26:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T00:26:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_items": 0, 2024-04-25T00:26:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_bytes": 189383, 2024-04-25T00:26:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_items": 50, 2024-04-25T00:26:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_bytes": 0, 2024-04-25T00:26:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_items": 0, 2024-04-25T00:26:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_bytes": 0, 2024-04-25T00:26:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_items": 0, 2024-04-25T00:26:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_bytes": 0, 2024-04-25T00:26:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_items": 0, 2024-04-25T00:26:25.816 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_bytes": 0, 2024-04-25T00:26:25.816 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_items": 0, 2024-04-25T00:26:25.816 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_bytes": 9512, 2024-04-25T00:26:25.816 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_items": 182, 2024-04-25T00:26:25.816 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T00:26:25.816 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_items": 0, 2024-04-25T00:26:25.816 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_bytes": 0, 2024-04-25T00:26:25.816 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_items": 0, 2024-04-25T00:26:25.816 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_bytes": 15072, 2024-04-25T00:26:25.816 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_items": 40, 2024-04-25T00:26:25.816 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_bytes": 0, 2024-04-25T00:26:25.816 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_items": 0, 2024-04-25T00:26:25.816 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_bytes": 0, 2024-04-25T00:26:25.816 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_items": 0 2024-04-25T00:26:25.817 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.817 INFO:teuthology.orchestra.run.smithi139.stdout: "objecter": { 2024-04-25T00:26:25.817 INFO:teuthology.orchestra.run.smithi139.stdout: "op_active": 0, 2024-04-25T00:26:25.817 INFO:teuthology.orchestra.run.smithi139.stdout: "op_laggy": 0, 2024-04-25T00:26:25.817 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send": 2197, 2024-04-25T00:26:25.817 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send_bytes": 21, 2024-04-25T00:26:25.817 INFO:teuthology.orchestra.run.smithi139.stdout: "op_resend": 3, 2024-04-25T00:26:25.817 INFO:teuthology.orchestra.run.smithi139.stdout: "op_reply": 2197, 2024-04-25T00:26:25.817 INFO:teuthology.orchestra.run.smithi139.stdout: "op_latency": { 2024-04-25T00:26:25.817 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2197, 2024-04-25T00:26:25.817 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 5.546883093, 2024-04-25T00:26:25.817 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.002524753 2024-04-25T00:26:25.817 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.817 INFO:teuthology.orchestra.run.smithi139.stdout: "op_inflight": 0, 2024-04-25T00:26:25.817 INFO:teuthology.orchestra.run.smithi139.stdout: "oplen_avg": { 2024-04-25T00:26:25.818 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2197, 2024-04-25T00:26:25.818 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 2198 2024-04-25T00:26:25.818 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.818 INFO:teuthology.orchestra.run.smithi139.stdout: "op": 2197, 2024-04-25T00:26:25.818 INFO:teuthology.orchestra.run.smithi139.stdout: "op_r": 2197, 2024-04-25T00:26:25.818 INFO:teuthology.orchestra.run.smithi139.stdout: "op_w": 0, 2024-04-25T00:26:25.818 INFO:teuthology.orchestra.run.smithi139.stdout: "op_rmw": 0, 2024-04-25T00:26:25.818 INFO:teuthology.orchestra.run.smithi139.stdout: "op_pg": 0, 2024-04-25T00:26:25.818 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_stat": 1426, 2024-04-25T00:26:25.818 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_create": 0, 2024-04-25T00:26:25.818 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_read": 769, 2024-04-25T00:26:25.818 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_write": 0, 2024-04-25T00:26:25.818 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writefull": 0, 2024-04-25T00:26:25.819 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writesame": 0, 2024-04-25T00:26:25.819 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_append": 0, 2024-04-25T00:26:25.819 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_zero": 0, 2024-04-25T00:26:25.819 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_truncate": 0, 2024-04-25T00:26:25.819 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_delete": 0, 2024-04-25T00:26:25.819 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_mapext": 0, 2024-04-25T00:26:25.819 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_sparse_read": 0, 2024-04-25T00:26:25.819 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_clonerange": 0, 2024-04-25T00:26:25.819 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_getxattr": 1, 2024-04-25T00:26:25.819 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_setxattr": 0, 2024-04-25T00:26:25.819 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_cmpxattr": 0, 2024-04-25T00:26:25.819 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_rmxattr": 0, 2024-04-25T00:26:25.819 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_resetxattrs": 0, 2024-04-25T00:26:25.819 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_call": 0, 2024-04-25T00:26:25.819 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_watch": 0, 2024-04-25T00:26:25.820 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_notify": 0, 2024-04-25T00:26:25.820 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T00:26:25.820 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls": 0, 2024-04-25T00:26:25.820 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls_filter": 0, 2024-04-25T00:26:25.820 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_other": 0, 2024-04-25T00:26:25.820 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_active": 0, 2024-04-25T00:26:25.820 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_send": 0, 2024-04-25T00:26:25.820 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_resend": 0, 2024-04-25T00:26:25.820 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_ping": 0, 2024-04-25T00:26:25.820 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_active": 0, 2024-04-25T00:26:25.820 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_send": 0, 2024-04-25T00:26:25.820 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_resend": 0, 2024-04-25T00:26:25.820 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_active": 0, 2024-04-25T00:26:25.820 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_send": 0, 2024-04-25T00:26:25.821 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_resend": 0, 2024-04-25T00:26:25.821 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_active": 0, 2024-04-25T00:26:25.821 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_send": 0, 2024-04-25T00:26:25.821 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_resend": 0, 2024-04-25T00:26:25.821 INFO:teuthology.orchestra.run.smithi139.stdout: "command_active": 0, 2024-04-25T00:26:25.821 INFO:teuthology.orchestra.run.smithi139.stdout: "command_send": 0, 2024-04-25T00:26:25.821 INFO:teuthology.orchestra.run.smithi139.stdout: "command_resend": 0, 2024-04-25T00:26:25.821 INFO:teuthology.orchestra.run.smithi139.stdout: "map_epoch": 96, 2024-04-25T00:26:25.821 INFO:teuthology.orchestra.run.smithi139.stdout: "map_full": 0, 2024-04-25T00:26:25.821 INFO:teuthology.orchestra.run.smithi139.stdout: "map_inc": 7, 2024-04-25T00:26:25.821 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_sessions": 5, 2024-04-25T00:26:25.821 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_open": 5, 2024-04-25T00:26:25.821 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_close": 0, 2024-04-25T00:26:25.821 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_laggy": 0, 2024-04-25T00:26:25.822 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_wr": 0, 2024-04-25T00:26:25.822 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_rd": 2, 2024-04-25T00:26:25.822 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_del": 0 2024-04-25T00:26:25.822 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.822 INFO:teuthology.orchestra.run.smithi139.stdout: "oft": { 2024-04-25T00:26:25.822 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_objs": 0, 2024-04-25T00:26:25.822 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_kv_pairs": 0, 2024-04-25T00:26:25.822 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_updates": 0, 2024-04-25T00:26:25.822 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_removes": 0 2024-04-25T00:26:25.822 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.822 INFO:teuthology.orchestra.run.smithi139.stdout: "osd-slow-ops": { 2024-04-25T00:26:25.822 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_ops_count": 0 2024-04-25T00:26:25.822 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.822 INFO:teuthology.orchestra.run.smithi139.stdout: "purge_queue": { 2024-04-25T00:26:25.823 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops": 0, 2024-04-25T00:26:25.823 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T00:26:25.823 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing": 0, 2024-04-25T00:26:25.823 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_high_water": 0, 2024-04-25T00:26:25.823 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed_ops": 0, 2024-04-25T00:26:25.823 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed": 0, 2024-04-25T00:26:25.823 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_item_in_journal": 0 2024-04-25T00:26:25.823 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.823 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T00:26:25.823 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:25.823 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-04-25T00:26:25.823 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:25.823 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 2519, 2024-04-25T00:26:25.823 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 2529250, 2024-04-25T00:26:25.824 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:25.824 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 2519, 2024-04-25T00:26:25.824 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T00:26:25.824 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T00:26:25.824 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 2519, 2024-04-25T00:26:25.824 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 2529250, 2024-04-25T00:26:25.824 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:25.824 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.824 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.824 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.824 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:25.824 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.824 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_bytes": { 2024-04-25T00:26:25.824 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:25.825 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-04-25T00:26:25.825 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:25.825 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T00:26:25.825 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T00:26:25.825 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:25.825 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:25.825 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 2197, 2024-04-25T00:26:25.825 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 28883, 2024-04-25T00:26:25.825 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 2197, 2024-04-25T00:26:25.825 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 28883, 2024-04-25T00:26:25.825 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:25.825 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.825 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.825 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.826 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:25.826 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.826 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_ops": { 2024-04-25T00:26:25.826 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:25.826 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 1024, 2024-04-25T00:26:25.826 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:25.826 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T00:26:25.826 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T00:26:25.826 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:25.826 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:25.826 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 2197, 2024-04-25T00:26:25.826 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 2197, 2024-04-25T00:26:25.826 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 2197, 2024-04-25T00:26:25.826 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 2197, 2024-04-25T00:26:25.827 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:25.827 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.827 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.827 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.827 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:25.827 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.827 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle": { 2024-04-25T00:26:25.827 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:25.827 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-04-25T00:26:25.827 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:25.827 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T00:26:25.827 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T00:26:25.827 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:25.827 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:25.828 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T00:26:25.828 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T00:26:25.828 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 0, 2024-04-25T00:26:25.828 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 0, 2024-04-25T00:26:25.828 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:25.828 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.828 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.828 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.828 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:25.828 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:25.828 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle-0x559615c646e0": { 2024-04-25T00:26:25.828 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:25.828 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-04-25T00:26:25.828 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:25.829 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T00:26:25.829 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T00:26:25.829 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:25.829 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:25.829 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T00:26:25.829 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T00:26:25.829 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 0, 2024-04-25T00:26:25.829 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 0, 2024-04-25T00:26:25.829 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:25.829 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:25.829 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:25.829 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:25.829 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:25.830 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:25.830 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-04-25T00:26:26.726 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-25T00:26:26.726 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-25T00:26:26.726 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-04-25T00:26:26.726 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-25T00:26:26.726 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T00:26:26.726 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-04-25T00:26:26.726 DEBUG:tasks.check_counter:Getting stats from h 2024-04-25T00:26:26.727 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-25T00:26:26.972 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T00:26:27.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:26 smithi139 ceph-mon[32833]: pgmap v325: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T00:26:27.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:26 smithi153 ceph-mon[32896]: pgmap v325: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T00:26:27.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:26 smithi027 ceph-mon[29208]: pgmap v325: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T00:26:28.687 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-04-25T00:26:28.687 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T00:26:28.687 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 1238, 2024-04-25T00:26:28.687 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 1147, 2024-04-25T00:26:28.687 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 10782302, 2024-04-25T00:26:28.687 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 429998, 2024-04-25T00:26:28.687 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 6, 2024-04-25T00:26:28.688 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 2, 2024-04-25T00:26:28.688 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.135427312, 2024-04-25T00:26:28.688 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.044641424, 2024-04-25T00:26:28.688 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.064557110, 2024-04-25T00:26:28.688 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.030080239, 2024-04-25T00:26:28.688 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:28.688 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1147, 2024-04-25T00:26:28.688 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.044888582, 2024-04-25T00:26:28.688 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000039135 2024-04-25T00:26:28.688 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.688 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:28.688 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.688 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.688 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.688 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.689 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 2123888, 2024-04-25T00:26:28.689 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 139024 2024-04-25T00:26:28.689 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.689 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T00:26:28.689 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 16, 2024-04-25T00:26:28.689 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 239, 2024-04-25T00:26:28.689 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 16781652, 2024-04-25T00:26:28.689 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 457877, 2024-04-25T00:26:28.689 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 8, 2024-04-25T00:26:28.689 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 6, 2024-04-25T00:26:28.689 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.027506806, 2024-04-25T00:26:28.689 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.007568373, 2024-04-25T00:26:28.689 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.017016955, 2024-04-25T00:26:28.690 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.000491259, 2024-04-25T00:26:28.690 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:28.690 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 236, 2024-04-25T00:26:28.690 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.023941612, 2024-04-25T00:26:28.690 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000101447 2024-04-25T00:26:28.690 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.690 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:28.690 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.690 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.690 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.690 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.690 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T00:26:28.690 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 448816 2024-04-25T00:26:28.691 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.691 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T00:26:28.691 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 526, 2024-04-25T00:26:28.691 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 528, 2024-04-25T00:26:28.691 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 33070232, 2024-04-25T00:26:28.691 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 167275, 2024-04-25T00:26:28.691 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 6, 2024-04-25T00:26:28.691 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 5, 2024-04-25T00:26:28.691 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.066035111, 2024-04-25T00:26:28.691 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.011929709, 2024-04-25T00:26:28.691 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.035634945, 2024-04-25T00:26:28.691 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.014668910, 2024-04-25T00:26:28.691 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:28.691 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 526, 2024-04-25T00:26:28.692 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.018700025, 2024-04-25T00:26:28.692 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000035551 2024-04-25T00:26:28.692 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.692 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:28.692 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.692 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.692 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.692 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.692 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T00:26:28.692 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T00:26:28.692 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.692 INFO:teuthology.orchestra.run.smithi139.stdout: "cct": { 2024-04-25T00:26:28.692 INFO:teuthology.orchestra.run.smithi139.stdout: "total_workers": 1, 2024-04-25T00:26:28.692 INFO:teuthology.orchestra.run.smithi139.stdout: "unhealthy_workers": 0 2024-04-25T00:26:28.693 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.693 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-MDSRank": { 2024-04-25T00:26:28.693 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-04-25T00:26:28.693 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-04-25T00:26:28.693 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2432, 2024-04-25T00:26:28.693 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.058455440, 2024-04-25T00:26:28.693 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000024035 2024-04-25T00:26:28.693 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:28.693 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.693 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-PurgeQueue": { 2024-04-25T00:26:28.693 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-04-25T00:26:28.693 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-04-25T00:26:28.693 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.693 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.694 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.694 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:28.694 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.694 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": { 2024-04-25T00:26:28.694 INFO:teuthology.orchestra.run.smithi139.stdout: "request": 0, 2024-04-25T00:26:28.694 INFO:teuthology.orchestra.run.smithi139.stdout: "reply": 0, 2024-04-25T00:26:28.694 INFO:teuthology.orchestra.run.smithi139.stdout: "reply_latency": { 2024-04-25T00:26:28.694 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.694 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.694 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.694 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.694 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_reply": 0, 2024-04-25T00:26:28.694 INFO:teuthology.orchestra.run.smithi139.stdout: "forward": 0, 2024-04-25T00:26:28.695 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_complete": 0, 2024-04-25T00:26:28.695 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_keys": 0, 2024-04-25T00:26:28.695 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_commit": 0, 2024-04-25T00:26:28.695 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_split": 0, 2024-04-25T00:26:28.695 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_merge": 0, 2024-04-25T00:26:28.695 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes": 1884, 2024-04-25T00:26:28.695 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_top": 0, 2024-04-25T00:26:28.695 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_bottom": 1884, 2024-04-25T00:26:28.695 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pin_tail": 0, 2024-04-25T00:26:28.695 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pinned": 1884, 2024-04-25T00:26:28.695 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_expired": 0, 2024-04-25T00:26:28.695 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_with_caps": 0, 2024-04-25T00:26:28.695 INFO:teuthology.orchestra.run.smithi139.stdout: "caps": 0, 2024-04-25T00:26:28.695 INFO:teuthology.orchestra.run.smithi139.stdout: "subtrees": 2, 2024-04-25T00:26:28.695 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse": 0, 2024-04-25T00:26:28.696 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_hit": 0, 2024-04-25T00:26:28.696 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_forward": 0, 2024-04-25T00:26:28.696 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_discover": 0, 2024-04-25T00:26:28.696 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_dir_fetch": 0, 2024-04-25T00:26:28.696 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_remote_ino": 0, 2024-04-25T00:26:28.696 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_lock": 0, 2024-04-25T00:26:28.696 INFO:teuthology.orchestra.run.smithi139.stdout: "load_cent": 0, 2024-04-25T00:26:28.696 INFO:teuthology.orchestra.run.smithi139.stdout: "q": 0, 2024-04-25T00:26:28.696 INFO:teuthology.orchestra.run.smithi139.stdout: "exported": 0, 2024-04-25T00:26:28.696 INFO:teuthology.orchestra.run.smithi139.stdout: "exported_inodes": 0, 2024-04-25T00:26:28.696 INFO:teuthology.orchestra.run.smithi139.stdout: "imported": 0, 2024-04-25T00:26:28.696 INFO:teuthology.orchestra.run.smithi139.stdout: "imported_inodes": 0, 2024-04-25T00:26:28.696 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_dir_fetch": 0, 2024-04-25T00:26:28.696 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_backtrace_fetch": 0, 2024-04-25T00:26:28.696 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_peer_discover": 0, 2024-04-25T00:26:28.697 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rfiles": 4, 2024-04-25T00:26:28.697 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rbytes": 234, 2024-04-25T00:26:28.697 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rsnaps": 6, 2024-04-25T00:26:28.697 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T00:26:28.697 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_set_tag": 0, 2024-04-25T00:26:28.697 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T00:26:28.697 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_inotable_repaired": 0, 2024-04-25T00:26:28.697 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_inodes": 0, 2024-04-25T00:26:28.697 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T00:26:28.697 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T00:26:28.697 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_file_inodes": 0, 2024-04-25T00:26:28.697 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_inode_file_caps": 0, 2024-04-25T00:26:28.697 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T00:26:28.697 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_grant": 0, 2024-04-25T00:26:28.697 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T00:26:28.698 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T00:26:28.698 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T00:26:28.698 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps": 0, 2024-04-25T00:26:28.698 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps_dirty": 0, 2024-04-25T00:26:28.698 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_cap_release": 0, 2024-04-25T00:26:28.698 INFO:teuthology.orchestra.run.smithi139.stdout: "process_request_cap_release": 0 2024-04-25T00:26:28.698 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.698 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_cache": { 2024-04-25T00:26:28.698 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update": 0, 2024-04-25T00:26:28.698 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update_receipt": 0, 2024-04-25T00:26:28.698 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_try_discover": 0, 2024-04-25T00:26:28.698 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_send_discover": 0, 2024-04-25T00:26:28.698 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_handle_discover": 0, 2024-04-25T00:26:28.699 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays": 0, 2024-04-25T00:26:28.699 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_delayed": 0, 2024-04-25T00:26:28.699 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_enqueuing": 0, 2024-04-25T00:26:28.699 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_created": 0, 2024-04-25T00:26:28.699 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_enqueued": 0, 2024-04-25T00:26:28.699 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_reintegrated": 0, 2024-04-25T00:26:28.699 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_migrated": 0, 2024-04-25T00:26:28.699 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_processing": 0, 2024-04-25T00:26:28.699 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_enqueued": 0, 2024-04-25T00:26:28.699 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_prioritized": 0, 2024-04-25T00:26:28.699 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_started": 0, 2024-04-25T00:26:28.699 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_completed": 0, 2024-04-25T00:26:28.699 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_path": 0, 2024-04-25T00:26:28.699 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_inode": 0, 2024-04-25T00:26:28.700 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T00:26:28.700 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_exportdir": 0, 2024-04-25T00:26:28.700 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_flush": 0, 2024-04-25T00:26:28.700 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragmentdir": 0, 2024-04-25T00:26:28.700 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragstats": 0, 2024-04-25T00:26:28.700 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_inodestats": 0 2024-04-25T00:26:28.700 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.700 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_log": { 2024-04-25T00:26:28.700 INFO:teuthology.orchestra.run.smithi139.stdout: "evlrg": 0, 2024-04-25T00:26:28.700 INFO:teuthology.orchestra.run.smithi139.stdout: "evadd": 0, 2024-04-25T00:26:28.700 INFO:teuthology.orchestra.run.smithi139.stdout: "evex": 0, 2024-04-25T00:26:28.700 INFO:teuthology.orchestra.run.smithi139.stdout: "evtrm": 0, 2024-04-25T00:26:28.700 INFO:teuthology.orchestra.run.smithi139.stdout: "ev": 26845, 2024-04-25T00:26:28.700 INFO:teuthology.orchestra.run.smithi139.stdout: "evexg": 0, 2024-04-25T00:26:28.701 INFO:teuthology.orchestra.run.smithi139.stdout: "evexd": 0, 2024-04-25T00:26:28.701 INFO:teuthology.orchestra.run.smithi139.stdout: "segadd": 0, 2024-04-25T00:26:28.701 INFO:teuthology.orchestra.run.smithi139.stdout: "segex": 0, 2024-04-25T00:26:28.701 INFO:teuthology.orchestra.run.smithi139.stdout: "segtrm": 0, 2024-04-25T00:26:28.701 INFO:teuthology.orchestra.run.smithi139.stdout: "seg": 31, 2024-04-25T00:26:28.701 INFO:teuthology.orchestra.run.smithi139.stdout: "segmjr": 4, 2024-04-25T00:26:28.701 INFO:teuthology.orchestra.run.smithi139.stdout: "segexg": 0, 2024-04-25T00:26:28.701 INFO:teuthology.orchestra.run.smithi139.stdout: "segexd": 0, 2024-04-25T00:26:28.701 INFO:teuthology.orchestra.run.smithi139.stdout: "expos": 4194304, 2024-04-25T00:26:28.701 INFO:teuthology.orchestra.run.smithi139.stdout: "wrpos": 62296858, 2024-04-25T00:26:28.701 INFO:teuthology.orchestra.run.smithi139.stdout: "rdpos": 62296722, 2024-04-25T00:26:28.701 INFO:teuthology.orchestra.run.smithi139.stdout: "jlat": { 2024-04-25T00:26:28.701 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.701 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.702 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.702 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.702 INFO:teuthology.orchestra.run.smithi139.stdout: "replayed": 26845 2024-04-25T00:26:28.702 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.702 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_mem": { 2024-04-25T00:26:28.702 INFO:teuthology.orchestra.run.smithi139.stdout: "ino": 422, 2024-04-25T00:26:28.702 INFO:teuthology.orchestra.run.smithi139.stdout: "ino+": 1212, 2024-04-25T00:26:28.702 INFO:teuthology.orchestra.run.smithi139.stdout: "ino-": 790, 2024-04-25T00:26:28.702 INFO:teuthology.orchestra.run.smithi139.stdout: "dir": 103, 2024-04-25T00:26:28.702 INFO:teuthology.orchestra.run.smithi139.stdout: "dir+": 120, 2024-04-25T00:26:28.702 INFO:teuthology.orchestra.run.smithi139.stdout: "dir-": 17, 2024-04-25T00:26:28.702 INFO:teuthology.orchestra.run.smithi139.stdout: "dn": 1884, 2024-04-25T00:26:28.702 INFO:teuthology.orchestra.run.smithi139.stdout: "dn+": 1951, 2024-04-25T00:26:28.702 INFO:teuthology.orchestra.run.smithi139.stdout: "dn-": 67, 2024-04-25T00:26:28.703 INFO:teuthology.orchestra.run.smithi139.stdout: "cap": 0, 2024-04-25T00:26:28.703 INFO:teuthology.orchestra.run.smithi139.stdout: "cap+": 0, 2024-04-25T00:26:28.703 INFO:teuthology.orchestra.run.smithi139.stdout: "cap-": 0, 2024-04-25T00:26:28.703 INFO:teuthology.orchestra.run.smithi139.stdout: "rss": 57996, 2024-04-25T00:26:28.703 INFO:teuthology.orchestra.run.smithi139.stdout: "heap": 215420 2024-04-25T00:26:28.703 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.703 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_server": { 2024-04-25T00:26:28.703 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_client_request": 0, 2024-04-25T00:26:28.703 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_server_request": 0, 2024-04-25T00:26:28.703 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_request": 0, 2024-04-25T00:26:28.703 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_session": 0, 2024-04-25T00:26:28.703 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_peer_request": 0, 2024-04-25T00:26:28.703 INFO:teuthology.orchestra.run.smithi139.stdout: "req_create_latency": { 2024-04-25T00:26:28.704 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.704 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.704 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.704 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.704 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getattr_latency": { 2024-04-25T00:26:28.704 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.704 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.704 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.704 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.704 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getfilelock_latency": { 2024-04-25T00:26:28.704 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.704 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.704 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.704 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.704 INFO:teuthology.orchestra.run.smithi139.stdout: "req_link_latency": { 2024-04-25T00:26:28.705 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.705 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.705 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.705 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.705 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookup_latency": { 2024-04-25T00:26:28.705 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.705 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.705 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.705 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.705 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookuphash_latency": { 2024-04-25T00:26:28.705 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.705 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.705 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.705 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.706 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupino_latency": { 2024-04-25T00:26:28.706 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.706 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.706 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.706 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.706 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupname_latency": { 2024-04-25T00:26:28.706 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.706 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.706 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.706 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.706 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupparent_latency": { 2024-04-25T00:26:28.706 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.706 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.706 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.707 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.707 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupsnap_latency": { 2024-04-25T00:26:28.707 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.707 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.707 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.707 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.707 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lssnap_latency": { 2024-04-25T00:26:28.707 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.707 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.707 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.707 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.707 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mkdir_latency": { 2024-04-25T00:26:28.708 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.708 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.708 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.708 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.708 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mknod_latency": { 2024-04-25T00:26:28.708 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.708 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.708 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.708 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.709 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mksnap_latency": { 2024-04-25T00:26:28.709 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.709 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.709 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.709 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.709 INFO:teuthology.orchestra.run.smithi139.stdout: "req_open_latency": { 2024-04-25T00:26:28.709 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.709 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.709 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.709 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.709 INFO:teuthology.orchestra.run.smithi139.stdout: "req_readdir_latency": { 2024-04-25T00:26:28.709 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.709 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.710 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.710 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.710 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rename_latency": { 2024-04-25T00:26:28.710 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.710 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.710 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.710 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.710 INFO:teuthology.orchestra.run.smithi139.stdout: "req_renamesnap_latency": { 2024-04-25T00:26:28.710 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.710 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.710 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.710 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.711 INFO:teuthology.orchestra.run.smithi139.stdout: "req_snapdiff_latency": { 2024-04-25T00:26:28.711 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.711 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.711 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.711 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.711 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmdir_latency": { 2024-04-25T00:26:28.711 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.711 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.711 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.711 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.711 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmsnap_latency": { 2024-04-25T00:26:28.711 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.711 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.711 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.711 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.712 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmxattr_latency": { 2024-04-25T00:26:28.712 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.712 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.712 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.712 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.712 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setattr_latency": { 2024-04-25T00:26:28.712 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.712 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.712 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.712 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.712 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setdirlayout_latency": { 2024-04-25T00:26:28.712 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.712 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.712 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.713 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.713 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setfilelock_latency": { 2024-04-25T00:26:28.713 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.713 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.713 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.713 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.713 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setlayout_latency": { 2024-04-25T00:26:28.713 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.713 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.713 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.713 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.713 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setxattr_latency": { 2024-04-25T00:26:28.713 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.713 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.713 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.714 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.714 INFO:teuthology.orchestra.run.smithi139.stdout: "req_symlink_latency": { 2024-04-25T00:26:28.714 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.714 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.714 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.714 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.714 INFO:teuthology.orchestra.run.smithi139.stdout: "req_unlink_latency": { 2024-04-25T00:26:28.714 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.714 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.714 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.714 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.714 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_revoke_eviction": 0, 2024-04-25T00:26:28.714 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_acquisition_throttle": 0, 2024-04-25T00:26:28.715 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getvxattr_latency": { 2024-04-25T00:26:28.715 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.715 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.715 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.715 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:28.715 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.715 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_sessions": { 2024-04-25T00:26:28.715 INFO:teuthology.orchestra.run.smithi139.stdout: "session_count": 2, 2024-04-25T00:26:28.715 INFO:teuthology.orchestra.run.smithi139.stdout: "session_add": 7, 2024-04-25T00:26:28.715 INFO:teuthology.orchestra.run.smithi139.stdout: "session_remove": 5, 2024-04-25T00:26:28.715 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_open": 5, 2024-04-25T00:26:28.715 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_stale": 0, 2024-04-25T00:26:28.715 INFO:teuthology.orchestra.run.smithi139.stdout: "total_load": 0, 2024-04-25T00:26:28.715 INFO:teuthology.orchestra.run.smithi139.stdout: "average_load": 0, 2024-04-25T00:26:28.716 INFO:teuthology.orchestra.run.smithi139.stdout: "avg_session_uptime": 3804, 2024-04-25T00:26:28.716 INFO:teuthology.orchestra.run.smithi139.stdout: "mdthresh_evicted": 0 2024-04-25T00:26:28.716 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.716 INFO:teuthology.orchestra.run.smithi139.stdout: "mempool": { 2024-04-25T00:26:28.716 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_bytes": 0, 2024-04-25T00:26:28.716 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_items": 0, 2024-04-25T00:26:28.716 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T00:26:28.716 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_items": 0, 2024-04-25T00:26:28.716 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T00:26:28.716 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_items": 0, 2024-04-25T00:26:28.716 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T00:26:28.716 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T00:26:28.716 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T00:26:28.716 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T00:26:28.717 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T00:26:28.717 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_items": 0, 2024-04-25T00:26:28.717 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T00:26:28.717 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T00:26:28.717 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_bytes": 0, 2024-04-25T00:26:28.717 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_items": 0, 2024-04-25T00:26:28.717 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_bytes": 0, 2024-04-25T00:26:28.717 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_items": 0, 2024-04-25T00:26:28.717 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T00:26:28.717 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T00:26:28.717 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T00:26:28.717 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T00:26:28.717 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T00:26:28.717 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_items": 0, 2024-04-25T00:26:28.718 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_bytes": 0, 2024-04-25T00:26:28.718 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_items": 0, 2024-04-25T00:26:28.718 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T00:26:28.718 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T00:26:28.718 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_bytes": 0, 2024-04-25T00:26:28.718 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_items": 0, 2024-04-25T00:26:28.718 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_bytes": 0, 2024-04-25T00:26:28.718 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_items": 0, 2024-04-25T00:26:28.718 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T00:26:28.718 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_items": 0, 2024-04-25T00:26:28.718 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T00:26:28.718 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_items": 0, 2024-04-25T00:26:28.718 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_bytes": 241219, 2024-04-25T00:26:28.719 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_items": 555, 2024-04-25T00:26:28.719 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_bytes": 0, 2024-04-25T00:26:28.719 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_items": 0, 2024-04-25T00:26:28.719 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_bytes": 0, 2024-04-25T00:26:28.719 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_items": 0, 2024-04-25T00:26:28.719 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_bytes": 0, 2024-04-25T00:26:28.719 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_items": 0, 2024-04-25T00:26:28.719 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_bytes": 0, 2024-04-25T00:26:28.719 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_items": 0, 2024-04-25T00:26:28.719 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_bytes": 9512, 2024-04-25T00:26:28.719 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_items": 182, 2024-04-25T00:26:28.719 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T00:26:28.719 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_items": 0, 2024-04-25T00:26:28.719 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_bytes": 0, 2024-04-25T00:26:28.720 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_items": 0, 2024-04-25T00:26:28.720 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_bytes": 2566751, 2024-04-25T00:26:28.720 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_items": 38402, 2024-04-25T00:26:28.720 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_bytes": 0, 2024-04-25T00:26:28.720 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_items": 0, 2024-04-25T00:26:28.720 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_bytes": 0, 2024-04-25T00:26:28.720 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_items": 0 2024-04-25T00:26:28.720 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.720 INFO:teuthology.orchestra.run.smithi139.stdout: "objecter": { 2024-04-25T00:26:28.720 INFO:teuthology.orchestra.run.smithi139.stdout: "op_active": 0, 2024-04-25T00:26:28.720 INFO:teuthology.orchestra.run.smithi139.stdout: "op_laggy": 0, 2024-04-25T00:26:28.720 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send": 1455, 2024-04-25T00:26:28.720 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send_bytes": 26, 2024-04-25T00:26:28.720 INFO:teuthology.orchestra.run.smithi139.stdout: "op_resend": 4, 2024-04-25T00:26:28.720 INFO:teuthology.orchestra.run.smithi139.stdout: "op_reply": 1455, 2024-04-25T00:26:28.721 INFO:teuthology.orchestra.run.smithi139.stdout: "op_latency": { 2024-04-25T00:26:28.721 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1455, 2024-04-25T00:26:28.721 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 309.752429602, 2024-04-25T00:26:28.721 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.212888267 2024-04-25T00:26:28.721 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.721 INFO:teuthology.orchestra.run.smithi139.stdout: "op_inflight": 0, 2024-04-25T00:26:28.721 INFO:teuthology.orchestra.run.smithi139.stdout: "oplen_avg": { 2024-04-25T00:26:28.721 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1455, 2024-04-25T00:26:28.721 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 1456 2024-04-25T00:26:28.721 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.721 INFO:teuthology.orchestra.run.smithi139.stdout: "op": 1455, 2024-04-25T00:26:28.721 INFO:teuthology.orchestra.run.smithi139.stdout: "op_r": 1455, 2024-04-25T00:26:28.721 INFO:teuthology.orchestra.run.smithi139.stdout: "op_w": 0, 2024-04-25T00:26:28.721 INFO:teuthology.orchestra.run.smithi139.stdout: "op_rmw": 0, 2024-04-25T00:26:28.721 INFO:teuthology.orchestra.run.smithi139.stdout: "op_pg": 0, 2024-04-25T00:26:28.722 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_stat": 906, 2024-04-25T00:26:28.722 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_create": 0, 2024-04-25T00:26:28.722 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_read": 546, 2024-04-25T00:26:28.722 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_write": 0, 2024-04-25T00:26:28.722 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writefull": 0, 2024-04-25T00:26:28.722 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writesame": 0, 2024-04-25T00:26:28.722 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_append": 0, 2024-04-25T00:26:28.722 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_zero": 0, 2024-04-25T00:26:28.722 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_truncate": 0, 2024-04-25T00:26:28.722 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_delete": 0, 2024-04-25T00:26:28.722 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_mapext": 0, 2024-04-25T00:26:28.722 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_sparse_read": 0, 2024-04-25T00:26:28.722 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_clonerange": 0, 2024-04-25T00:26:28.722 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_getxattr": 2, 2024-04-25T00:26:28.723 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_setxattr": 0, 2024-04-25T00:26:28.723 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_cmpxattr": 0, 2024-04-25T00:26:28.723 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_rmxattr": 0, 2024-04-25T00:26:28.723 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_resetxattrs": 0, 2024-04-25T00:26:28.723 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_call": 0, 2024-04-25T00:26:28.723 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_watch": 0, 2024-04-25T00:26:28.723 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_notify": 0, 2024-04-25T00:26:28.723 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T00:26:28.723 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls": 0, 2024-04-25T00:26:28.723 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls_filter": 0, 2024-04-25T00:26:28.724 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_other": 0, 2024-04-25T00:26:28.724 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_active": 0, 2024-04-25T00:26:28.724 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_send": 0, 2024-04-25T00:26:28.724 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_resend": 0, 2024-04-25T00:26:28.724 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_ping": 0, 2024-04-25T00:26:28.724 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_active": 0, 2024-04-25T00:26:28.724 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_send": 0, 2024-04-25T00:26:28.724 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_resend": 0, 2024-04-25T00:26:28.724 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_active": 0, 2024-04-25T00:26:28.724 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_send": 0, 2024-04-25T00:26:28.725 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_resend": 0, 2024-04-25T00:26:28.725 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_active": 0, 2024-04-25T00:26:28.725 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_send": 0, 2024-04-25T00:26:28.725 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_resend": 0, 2024-04-25T00:26:28.725 INFO:teuthology.orchestra.run.smithi139.stdout: "command_active": 0, 2024-04-25T00:26:28.725 INFO:teuthology.orchestra.run.smithi139.stdout: "command_send": 0, 2024-04-25T00:26:28.725 INFO:teuthology.orchestra.run.smithi139.stdout: "command_resend": 0, 2024-04-25T00:26:28.725 INFO:teuthology.orchestra.run.smithi139.stdout: "map_epoch": 96, 2024-04-25T00:26:28.725 INFO:teuthology.orchestra.run.smithi139.stdout: "map_full": 0, 2024-04-25T00:26:28.725 INFO:teuthology.orchestra.run.smithi139.stdout: "map_inc": 7, 2024-04-25T00:26:28.725 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_sessions": 9, 2024-04-25T00:26:28.725 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_open": 9, 2024-04-25T00:26:28.725 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_close": 0, 2024-04-25T00:26:28.726 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_laggy": 0, 2024-04-25T00:26:28.726 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_wr": 0, 2024-04-25T00:26:28.726 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_rd": 2, 2024-04-25T00:26:28.726 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_del": 0 2024-04-25T00:26:28.726 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.726 INFO:teuthology.orchestra.run.smithi139.stdout: "oft": { 2024-04-25T00:26:28.726 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_objs": 0, 2024-04-25T00:26:28.726 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_kv_pairs": 0, 2024-04-25T00:26:28.726 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_updates": 0, 2024-04-25T00:26:28.726 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_removes": 0 2024-04-25T00:26:28.726 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.726 INFO:teuthology.orchestra.run.smithi139.stdout: "osd-slow-ops": { 2024-04-25T00:26:28.726 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_ops_count": 0 2024-04-25T00:26:28.727 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.727 INFO:teuthology.orchestra.run.smithi139.stdout: "purge_queue": { 2024-04-25T00:26:28.727 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops": 0, 2024-04-25T00:26:28.727 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T00:26:28.727 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing": 0, 2024-04-25T00:26:28.727 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_high_water": 0, 2024-04-25T00:26:28.727 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed_ops": 0, 2024-04-25T00:26:28.727 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed": 0, 2024-04-25T00:26:28.727 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_item_in_journal": 0 2024-04-25T00:26:28.727 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.727 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T00:26:28.727 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:28.727 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-04-25T00:26:28.727 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:28.728 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 1776, 2024-04-25T00:26:28.728 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 60455693, 2024-04-25T00:26:28.728 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:28.728 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 1776, 2024-04-25T00:26:28.728 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T00:26:28.728 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T00:26:28.728 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 1776, 2024-04-25T00:26:28.728 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 60455693, 2024-04-25T00:26:28.728 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:28.728 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.728 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.728 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.728 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:28.729 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.729 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_bytes": { 2024-04-25T00:26:28.729 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:28.729 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-04-25T00:26:28.729 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:28.729 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T00:26:28.729 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T00:26:28.729 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:28.729 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:28.729 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 1455, 2024-04-25T00:26:28.729 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 58102564, 2024-04-25T00:26:28.729 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 1455, 2024-04-25T00:26:28.729 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 58102564, 2024-04-25T00:26:28.730 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:28.730 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.730 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.730 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.730 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:28.730 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.730 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_ops": { 2024-04-25T00:26:28.730 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:28.730 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 1024, 2024-04-25T00:26:28.730 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:28.730 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T00:26:28.730 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T00:26:28.730 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:28.730 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:28.731 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 1455, 2024-04-25T00:26:28.731 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 1455, 2024-04-25T00:26:28.731 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 1455, 2024-04-25T00:26:28.731 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 1455, 2024-04-25T00:26:28.731 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:28.731 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.731 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.731 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.731 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:28.731 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.731 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle": { 2024-04-25T00:26:28.731 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:28.731 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-04-25T00:26:28.731 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:28.732 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T00:26:28.732 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T00:26:28.732 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:28.732 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:28.732 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T00:26:28.732 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T00:26:28.732 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 0, 2024-04-25T00:26:28.732 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 0, 2024-04-25T00:26:28.732 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:28.732 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.732 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.732 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.732 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:28.732 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:28.733 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle-0x55fe72ebe6e0": { 2024-04-25T00:26:28.733 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:28.733 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-04-25T00:26:28.733 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:28.733 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T00:26:28.733 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T00:26:28.733 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:28.733 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:28.733 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T00:26:28.733 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T00:26:28.733 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 0, 2024-04-25T00:26:28.733 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 0, 2024-04-25T00:26:28.733 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:28.733 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:28.734 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:28.734 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:28.734 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:28.734 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:28.734 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-04-25T00:26:29.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:28 smithi139 ceph-mon[32833]: pgmap v326: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T00:26:29.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:28 smithi153 ceph-mon[32896]: pgmap v326: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T00:26:29.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:28 smithi027 ceph-mon[29208]: pgmap v326: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T00:26:29.488 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-25T00:26:29.489 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-25T00:26:29.489 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-04-25T00:26:29.489 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-25T00:26:29.489 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T00:26:29.489 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-04-25T00:26:29.489 DEBUG:tasks.check_counter:Getting stats from k 2024-04-25T00:26:29.489 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-25T00:26:29.733 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T00:26:30.766 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:30 smithi139 ceph-mon[32833]: pgmap v327: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T00:26:31.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:30 smithi153 ceph-mon[32896]: pgmap v327: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T00:26:31.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:30 smithi027 ceph-mon[29208]: pgmap v327: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T00:26:31.302 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-04-25T00:26:31.303 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T00:26:31.303 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 1067, 2024-04-25T00:26:31.303 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 1157, 2024-04-25T00:26:31.303 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 543490, 2024-04-25T00:26:31.303 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 661823, 2024-04-25T00:26:31.303 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 12, 2024-04-25T00:26:31.303 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 4, 2024-04-25T00:26:31.303 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.096530607, 2024-04-25T00:26:31.303 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.041748531, 2024-04-25T00:26:31.303 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.053850358, 2024-04-25T00:26:31.303 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.005300327, 2024-04-25T00:26:31.303 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:31.304 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1155, 2024-04-25T00:26:31.304 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.058020696, 2024-04-25T00:26:31.304 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000050234 2024-04-25T00:26:31.304 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.304 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:31.304 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1972, 2024-04-25T00:26:31.304 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000413902, 2024-04-25T00:26:31.304 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000209 2024-04-25T00:26:31.304 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.304 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 928, 2024-04-25T00:26:31.304 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 1216 2024-04-25T00:26:31.304 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.304 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T00:26:31.305 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 68, 2024-04-25T00:26:31.305 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 136, 2024-04-25T00:26:31.305 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 17770, 2024-04-25T00:26:31.305 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 420174, 2024-04-25T00:26:31.305 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 33, 2024-04-25T00:26:31.305 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 4, 2024-04-25T00:26:31.305 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.028679379, 2024-04-25T00:26:31.305 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.009402386, 2024-04-25T00:26:31.305 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.015755989, 2024-04-25T00:26:31.305 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.000903636, 2024-04-25T00:26:31.305 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:31.305 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 134, 2024-04-25T00:26:31.305 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.026553907, 2024-04-25T00:26:31.306 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000198163 2024-04-25T00:26:31.306 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.306 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:31.306 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 99, 2024-04-25T00:26:31.306 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000018320, 2024-04-25T00:26:31.306 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000185 2024-04-25T00:26:31.306 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.306 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 11072, 2024-04-25T00:26:31.306 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 150496 2024-04-25T00:26:31.306 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.306 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T00:26:31.306 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 388, 2024-04-25T00:26:31.306 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 434, 2024-04-25T00:26:31.307 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 2122449, 2024-04-25T00:26:31.307 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 483770, 2024-04-25T00:26:31.307 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 6, 2024-04-25T00:26:31.307 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 5, 2024-04-25T00:26:31.307 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.060357176, 2024-04-25T00:26:31.307 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.018992747, 2024-04-25T00:26:31.307 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.045795781, 2024-04-25T00:26:31.307 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.007297418, 2024-04-25T00:26:31.307 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:31.307 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 433, 2024-04-25T00:26:31.307 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.041291800, 2024-04-25T00:26:31.307 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000095362 2024-04-25T00:26:31.307 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.307 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:31.308 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.308 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.308 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.308 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.308 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 2113664, 2024-04-25T00:26:31.308 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 467680 2024-04-25T00:26:31.308 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.308 INFO:teuthology.orchestra.run.smithi139.stdout: "cct": { 2024-04-25T00:26:31.308 INFO:teuthology.orchestra.run.smithi139.stdout: "total_workers": 1, 2024-04-25T00:26:31.308 INFO:teuthology.orchestra.run.smithi139.stdout: "unhealthy_workers": 0 2024-04-25T00:26:31.308 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.308 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-MDSRank": { 2024-04-25T00:26:31.308 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-04-25T00:26:31.309 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-04-25T00:26:31.309 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 105, 2024-04-25T00:26:31.309 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.009329680, 2024-04-25T00:26:31.309 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000088854 2024-04-25T00:26:31.309 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:31.309 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.309 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-PurgeQueue": { 2024-04-25T00:26:31.309 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-04-25T00:26:31.309 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-04-25T00:26:31.309 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 18, 2024-04-25T00:26:31.309 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000542247, 2024-04-25T00:26:31.309 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000030124 2024-04-25T00:26:31.309 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:31.309 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.309 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": { 2024-04-25T00:26:31.310 INFO:teuthology.orchestra.run.smithi139.stdout: "request": 8, 2024-04-25T00:26:31.310 INFO:teuthology.orchestra.run.smithi139.stdout: "reply": 7, 2024-04-25T00:26:31.310 INFO:teuthology.orchestra.run.smithi139.stdout: "reply_latency": { 2024-04-25T00:26:31.310 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 7, 2024-04-25T00:26:31.310 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.017845830, 2024-04-25T00:26:31.310 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.002549404 2024-04-25T00:26:31.310 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.310 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_reply": 0, 2024-04-25T00:26:31.310 INFO:teuthology.orchestra.run.smithi139.stdout: "forward": 1, 2024-04-25T00:26:31.310 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_complete": 0, 2024-04-25T00:26:31.310 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_keys": 0, 2024-04-25T00:26:31.310 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_commit": 11, 2024-04-25T00:26:31.310 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_split": 0, 2024-04-25T00:26:31.311 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_merge": 0, 2024-04-25T00:26:31.311 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes": 16, 2024-04-25T00:26:31.311 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_top": 1, 2024-04-25T00:26:31.311 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_bottom": 1, 2024-04-25T00:26:31.311 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pin_tail": 14, 2024-04-25T00:26:31.311 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pinned": 14, 2024-04-25T00:26:31.311 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_expired": 28, 2024-04-25T00:26:31.311 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_with_caps": 1, 2024-04-25T00:26:31.311 INFO:teuthology.orchestra.run.smithi139.stdout: "caps": 1, 2024-04-25T00:26:31.311 INFO:teuthology.orchestra.run.smithi139.stdout: "subtrees": 3, 2024-04-25T00:26:31.311 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse": 25, 2024-04-25T00:26:31.311 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_hit": 10, 2024-04-25T00:26:31.311 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_forward": 0, 2024-04-25T00:26:31.311 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_discover": 5, 2024-04-25T00:26:31.311 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_dir_fetch": 0, 2024-04-25T00:26:31.312 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_remote_ino": 0, 2024-04-25T00:26:31.312 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_lock": 0, 2024-04-25T00:26:31.312 INFO:teuthology.orchestra.run.smithi139.stdout: "load_cent": 0, 2024-04-25T00:26:31.312 INFO:teuthology.orchestra.run.smithi139.stdout: "q": 0, 2024-04-25T00:26:31.312 INFO:teuthology.orchestra.run.smithi139.stdout: "exported": 3, 2024-04-25T00:26:31.312 INFO:teuthology.orchestra.run.smithi139.stdout: "exported_inodes": 8, 2024-04-25T00:26:31.312 INFO:teuthology.orchestra.run.smithi139.stdout: "imported": 3, 2024-04-25T00:26:31.312 INFO:teuthology.orchestra.run.smithi139.stdout: "imported_inodes": 5, 2024-04-25T00:26:31.312 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_dir_fetch": 0, 2024-04-25T00:26:31.312 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_backtrace_fetch": 6, 2024-04-25T00:26:31.312 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_peer_discover": 0, 2024-04-25T00:26:31.312 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rfiles": 4, 2024-04-25T00:26:31.312 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rbytes": 234, 2024-04-25T00:26:31.312 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rsnaps": 6, 2024-04-25T00:26:31.313 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_fetch": 6, 2024-04-25T00:26:31.313 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_set_tag": 0, 2024-04-25T00:26:31.313 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T00:26:31.313 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_inotable_repaired": 0, 2024-04-25T00:26:31.313 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_inodes": 0, 2024-04-25T00:26:31.313 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T00:26:31.313 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T00:26:31.313 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_file_inodes": 6, 2024-04-25T00:26:31.313 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_inode_file_caps": 0, 2024-04-25T00:26:31.313 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_revoke": 5, 2024-04-25T00:26:31.313 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_grant": 9, 2024-04-25T00:26:31.313 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T00:26:31.313 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T00:26:31.313 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T00:26:31.314 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps": 6, 2024-04-25T00:26:31.314 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps_dirty": 1, 2024-04-25T00:26:31.314 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_cap_release": 1, 2024-04-25T00:26:31.314 INFO:teuthology.orchestra.run.smithi139.stdout: "process_request_cap_release": 7 2024-04-25T00:26:31.314 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.314 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_cache": { 2024-04-25T00:26:31.314 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update": 0, 2024-04-25T00:26:31.314 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update_receipt": 0, 2024-04-25T00:26:31.314 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_try_discover": 0, 2024-04-25T00:26:31.314 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_send_discover": 5, 2024-04-25T00:26:31.314 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_handle_discover": 0, 2024-04-25T00:26:31.314 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays": 0, 2024-04-25T00:26:31.314 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_delayed": 0, 2024-04-25T00:26:31.315 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_enqueuing": 0, 2024-04-25T00:26:31.315 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_created": 3, 2024-04-25T00:26:31.315 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_enqueued": 3, 2024-04-25T00:26:31.315 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_reintegrated": 0, 2024-04-25T00:26:31.315 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_migrated": 0, 2024-04-25T00:26:31.315 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_processing": 0, 2024-04-25T00:26:31.315 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_enqueued": 0, 2024-04-25T00:26:31.315 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_prioritized": 0, 2024-04-25T00:26:31.315 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_started": 0, 2024-04-25T00:26:31.315 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_completed": 0, 2024-04-25T00:26:31.315 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_path": 0, 2024-04-25T00:26:31.315 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_inode": 0, 2024-04-25T00:26:31.315 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T00:26:31.315 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_exportdir": 3, 2024-04-25T00:26:31.315 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_flush": 0, 2024-04-25T00:26:31.316 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragmentdir": 0, 2024-04-25T00:26:31.316 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragstats": 0, 2024-04-25T00:26:31.316 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_inodestats": 0 2024-04-25T00:26:31.316 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.316 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_log": { 2024-04-25T00:26:31.316 INFO:teuthology.orchestra.run.smithi139.stdout: "evlrg": 0, 2024-04-25T00:26:31.316 INFO:teuthology.orchestra.run.smithi139.stdout: "evadd": 34, 2024-04-25T00:26:31.316 INFO:teuthology.orchestra.run.smithi139.stdout: "evex": 0, 2024-04-25T00:26:31.316 INFO:teuthology.orchestra.run.smithi139.stdout: "evtrm": 0, 2024-04-25T00:26:31.316 INFO:teuthology.orchestra.run.smithi139.stdout: "ev": 34, 2024-04-25T00:26:31.316 INFO:teuthology.orchestra.run.smithi139.stdout: "evexg": 0, 2024-04-25T00:26:31.316 INFO:teuthology.orchestra.run.smithi139.stdout: "evexd": 0, 2024-04-25T00:26:31.316 INFO:teuthology.orchestra.run.smithi139.stdout: "segadd": 2, 2024-04-25T00:26:31.316 INFO:teuthology.orchestra.run.smithi139.stdout: "segex": 0, 2024-04-25T00:26:31.316 INFO:teuthology.orchestra.run.smithi139.stdout: "segtrm": 0, 2024-04-25T00:26:31.317 INFO:teuthology.orchestra.run.smithi139.stdout: "seg": 2, 2024-04-25T00:26:31.317 INFO:teuthology.orchestra.run.smithi139.stdout: "segmjr": 2, 2024-04-25T00:26:31.317 INFO:teuthology.orchestra.run.smithi139.stdout: "segexg": 0, 2024-04-25T00:26:31.317 INFO:teuthology.orchestra.run.smithi139.stdout: "segexd": 0, 2024-04-25T00:26:31.317 INFO:teuthology.orchestra.run.smithi139.stdout: "expos": 4194304, 2024-04-25T00:26:31.317 INFO:teuthology.orchestra.run.smithi139.stdout: "wrpos": 4268153, 2024-04-25T00:26:31.317 INFO:teuthology.orchestra.run.smithi139.stdout: "rdpos": 0, 2024-04-25T00:26:31.317 INFO:teuthology.orchestra.run.smithi139.stdout: "jlat": { 2024-04-25T00:26:31.317 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 26, 2024-04-25T00:26:31.317 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.203115977, 2024-04-25T00:26:31.317 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.007812152 2024-04-25T00:26:31.317 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.317 INFO:teuthology.orchestra.run.smithi139.stdout: "replayed": 0 2024-04-25T00:26:31.317 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.317 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_mem": { 2024-04-25T00:26:31.317 INFO:teuthology.orchestra.run.smithi139.stdout: "ino": 20, 2024-04-25T00:26:31.318 INFO:teuthology.orchestra.run.smithi139.stdout: "ino+": 36, 2024-04-25T00:26:31.318 INFO:teuthology.orchestra.run.smithi139.stdout: "ino-": 16, 2024-04-25T00:26:31.318 INFO:teuthology.orchestra.run.smithi139.stdout: "dir": 19, 2024-04-25T00:26:31.318 INFO:teuthology.orchestra.run.smithi139.stdout: "dir+": 30, 2024-04-25T00:26:31.318 INFO:teuthology.orchestra.run.smithi139.stdout: "dir-": 11, 2024-04-25T00:26:31.318 INFO:teuthology.orchestra.run.smithi139.stdout: "dn": 16, 2024-04-25T00:26:31.318 INFO:teuthology.orchestra.run.smithi139.stdout: "dn+": 47, 2024-04-25T00:26:31.318 INFO:teuthology.orchestra.run.smithi139.stdout: "dn-": 31, 2024-04-25T00:26:31.318 INFO:teuthology.orchestra.run.smithi139.stdout: "cap": 1, 2024-04-25T00:26:31.318 INFO:teuthology.orchestra.run.smithi139.stdout: "cap+": 7, 2024-04-25T00:26:31.318 INFO:teuthology.orchestra.run.smithi139.stdout: "cap-": 6, 2024-04-25T00:26:31.318 INFO:teuthology.orchestra.run.smithi139.stdout: "rss": 54004, 2024-04-25T00:26:31.318 INFO:teuthology.orchestra.run.smithi139.stdout: "heap": 223612 2024-04-25T00:26:31.318 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.319 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_server": { 2024-04-25T00:26:31.319 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_client_request": 18, 2024-04-25T00:26:31.319 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_server_request": 3, 2024-04-25T00:26:31.319 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_request": 8, 2024-04-25T00:26:31.319 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_session": 17, 2024-04-25T00:26:31.319 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_peer_request": 12, 2024-04-25T00:26:31.319 INFO:teuthology.orchestra.run.smithi139.stdout: "req_create_latency": { 2024-04-25T00:26:31.319 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.319 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.319 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.319 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.319 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getattr_latency": { 2024-04-25T00:26:31.319 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1, 2024-04-25T00:26:31.319 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000691473, 2024-04-25T00:26:31.319 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000691473 2024-04-25T00:26:31.319 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.320 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getfilelock_latency": { 2024-04-25T00:26:31.320 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.320 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.320 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.320 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.320 INFO:teuthology.orchestra.run.smithi139.stdout: "req_link_latency": { 2024-04-25T00:26:31.320 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.320 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.320 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.320 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.320 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookup_latency": { 2024-04-25T00:26:31.320 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.320 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.321 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.321 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookuphash_latency": { 2024-04-25T00:26:31.321 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.321 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.321 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.321 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupino_latency": { 2024-04-25T00:26:31.321 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.321 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.321 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.321 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupname_latency": { 2024-04-25T00:26:31.321 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.321 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.322 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.322 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.322 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupparent_latency": { 2024-04-25T00:26:31.322 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.322 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.322 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.322 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.322 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupsnap_latency": { 2024-04-25T00:26:31.322 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.322 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.322 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.322 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.322 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lssnap_latency": { 2024-04-25T00:26:31.322 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.322 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.323 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.323 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.323 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mkdir_latency": { 2024-04-25T00:26:31.323 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.323 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.323 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.324 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.324 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mknod_latency": { 2024-04-25T00:26:31.324 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.324 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.324 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.324 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.324 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mksnap_latency": { 2024-04-25T00:26:31.324 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.324 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.324 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.324 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.324 INFO:teuthology.orchestra.run.smithi139.stdout: "req_open_latency": { 2024-04-25T00:26:31.324 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.325 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.325 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.325 INFO:teuthology.orchestra.run.smithi139.stdout: "req_readdir_latency": { 2024-04-25T00:26:31.325 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 3, 2024-04-25T00:26:31.325 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.001174775, 2024-04-25T00:26:31.325 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000391591 2024-04-25T00:26:31.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.325 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rename_latency": { 2024-04-25T00:26:31.325 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.325 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.325 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.326 INFO:teuthology.orchestra.run.smithi139.stdout: "req_renamesnap_latency": { 2024-04-25T00:26:31.326 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.326 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.326 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.326 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.326 INFO:teuthology.orchestra.run.smithi139.stdout: "req_snapdiff_latency": { 2024-04-25T00:26:31.326 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.326 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.326 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.326 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.326 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmdir_latency": { 2024-04-25T00:26:31.327 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.327 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.327 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.327 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.327 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmsnap_latency": { 2024-04-25T00:26:31.327 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.327 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.327 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.327 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.327 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmxattr_latency": { 2024-04-25T00:26:31.327 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.327 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.327 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.327 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.328 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setattr_latency": { 2024-04-25T00:26:31.328 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.328 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.328 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.328 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.328 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setdirlayout_latency": { 2024-04-25T00:26:31.328 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.328 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.328 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.328 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.328 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setfilelock_latency": { 2024-04-25T00:26:31.328 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.328 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.328 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.329 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.329 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setlayout_latency": { 2024-04-25T00:26:31.329 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.329 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.329 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.329 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.329 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setxattr_latency": { 2024-04-25T00:26:31.329 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.329 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.329 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.329 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.329 INFO:teuthology.orchestra.run.smithi139.stdout: "req_symlink_latency": { 2024-04-25T00:26:31.329 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.329 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.329 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.330 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.330 INFO:teuthology.orchestra.run.smithi139.stdout: "req_unlink_latency": { 2024-04-25T00:26:31.330 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 3, 2024-04-25T00:26:31.330 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.015979582, 2024-04-25T00:26:31.330 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.005326527 2024-04-25T00:26:31.330 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.330 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_revoke_eviction": 0, 2024-04-25T00:26:31.330 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_acquisition_throttle": 0, 2024-04-25T00:26:31.330 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getvxattr_latency": { 2024-04-25T00:26:31.330 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.330 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.330 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.330 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:31.330 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.330 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_sessions": { 2024-04-25T00:26:31.331 INFO:teuthology.orchestra.run.smithi139.stdout: "session_count": 1, 2024-04-25T00:26:31.331 INFO:teuthology.orchestra.run.smithi139.stdout: "session_add": 3, 2024-04-25T00:26:31.331 INFO:teuthology.orchestra.run.smithi139.stdout: "session_remove": 2, 2024-04-25T00:26:31.331 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_open": 1, 2024-04-25T00:26:31.331 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_stale": 0, 2024-04-25T00:26:31.331 INFO:teuthology.orchestra.run.smithi139.stdout: "total_load": 3, 2024-04-25T00:26:31.331 INFO:teuthology.orchestra.run.smithi139.stdout: "average_load": 3, 2024-04-25T00:26:31.331 INFO:teuthology.orchestra.run.smithi139.stdout: "avg_session_uptime": 646, 2024-04-25T00:26:31.331 INFO:teuthology.orchestra.run.smithi139.stdout: "mdthresh_evicted": 0 2024-04-25T00:26:31.331 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.331 INFO:teuthology.orchestra.run.smithi139.stdout: "mempool": { 2024-04-25T00:26:31.331 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_bytes": 0, 2024-04-25T00:26:31.331 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_items": 0, 2024-04-25T00:26:31.331 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T00:26:31.331 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_items": 0, 2024-04-25T00:26:31.332 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T00:26:31.332 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_items": 0, 2024-04-25T00:26:31.332 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T00:26:31.332 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T00:26:31.332 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T00:26:31.332 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T00:26:31.332 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T00:26:31.332 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_items": 0, 2024-04-25T00:26:31.332 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T00:26:31.332 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T00:26:31.332 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_bytes": 0, 2024-04-25T00:26:31.332 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_items": 0, 2024-04-25T00:26:31.332 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_bytes": 0, 2024-04-25T00:26:31.333 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_items": 0, 2024-04-25T00:26:31.333 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T00:26:31.333 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T00:26:31.333 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T00:26:31.333 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T00:26:31.333 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T00:26:31.333 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_items": 0, 2024-04-25T00:26:31.333 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_bytes": 0, 2024-04-25T00:26:31.333 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_items": 0, 2024-04-25T00:26:31.333 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T00:26:31.333 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T00:26:31.333 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_bytes": 0, 2024-04-25T00:26:31.333 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_items": 0, 2024-04-25T00:26:31.334 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_bytes": 0, 2024-04-25T00:26:31.334 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_items": 0, 2024-04-25T00:26:31.334 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T00:26:31.334 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_items": 0, 2024-04-25T00:26:31.334 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T00:26:31.334 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_items": 0, 2024-04-25T00:26:31.334 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_bytes": 313623, 2024-04-25T00:26:31.334 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_items": 108, 2024-04-25T00:26:31.334 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_bytes": 0, 2024-04-25T00:26:31.334 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_items": 0, 2024-04-25T00:26:31.334 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_bytes": 0, 2024-04-25T00:26:31.334 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_items": 0, 2024-04-25T00:26:31.334 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_bytes": 0, 2024-04-25T00:26:31.335 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_items": 0, 2024-04-25T00:26:31.335 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_bytes": 0, 2024-04-25T00:26:31.335 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_items": 0, 2024-04-25T00:26:31.335 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_bytes": 9512, 2024-04-25T00:26:31.335 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_items": 182, 2024-04-25T00:26:31.335 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T00:26:31.335 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_items": 0, 2024-04-25T00:26:31.335 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_bytes": 0, 2024-04-25T00:26:31.335 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_items": 0, 2024-04-25T00:26:31.335 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_bytes": 86045, 2024-04-25T00:26:31.335 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_items": 347, 2024-04-25T00:26:31.335 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_bytes": 0, 2024-04-25T00:26:31.335 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_items": 0, 2024-04-25T00:26:31.336 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_bytes": 0, 2024-04-25T00:26:31.336 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_items": 0 2024-04-25T00:26:31.336 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.336 INFO:teuthology.orchestra.run.smithi139.stdout: "objecter": { 2024-04-25T00:26:31.336 INFO:teuthology.orchestra.run.smithi139.stdout: "op_active": 0, 2024-04-25T00:26:31.336 INFO:teuthology.orchestra.run.smithi139.stdout: "op_laggy": 0, 2024-04-25T00:26:31.336 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send": 95, 2024-04-25T00:26:31.336 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send_bytes": 87555, 2024-04-25T00:26:31.336 INFO:teuthology.orchestra.run.smithi139.stdout: "op_resend": 33, 2024-04-25T00:26:31.336 INFO:teuthology.orchestra.run.smithi139.stdout: "op_reply": 95, 2024-04-25T00:26:31.336 INFO:teuthology.orchestra.run.smithi139.stdout: "op_latency": { 2024-04-25T00:26:31.336 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 95, 2024-04-25T00:26:31.336 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.695985609, 2024-04-25T00:26:31.336 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.007326164 2024-04-25T00:26:31.336 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.337 INFO:teuthology.orchestra.run.smithi139.stdout: "op_inflight": 0, 2024-04-25T00:26:31.337 INFO:teuthology.orchestra.run.smithi139.stdout: "oplen_avg": { 2024-04-25T00:26:31.337 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 95, 2024-04-25T00:26:31.337 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 128 2024-04-25T00:26:31.337 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.337 INFO:teuthology.orchestra.run.smithi139.stdout: "op": 95, 2024-04-25T00:26:31.337 INFO:teuthology.orchestra.run.smithi139.stdout: "op_r": 7, 2024-04-25T00:26:31.337 INFO:teuthology.orchestra.run.smithi139.stdout: "op_w": 88, 2024-04-25T00:26:31.337 INFO:teuthology.orchestra.run.smithi139.stdout: "op_rmw": 0, 2024-04-25T00:26:31.337 INFO:teuthology.orchestra.run.smithi139.stdout: "op_pg": 0, 2024-04-25T00:26:31.337 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_stat": 0, 2024-04-25T00:26:31.337 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_create": 10, 2024-04-25T00:26:31.337 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_read": 1, 2024-04-25T00:26:31.337 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_write": 27, 2024-04-25T00:26:31.337 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writefull": 11, 2024-04-25T00:26:31.338 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writesame": 0, 2024-04-25T00:26:31.338 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_append": 0, 2024-04-25T00:26:31.338 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_zero": 0, 2024-04-25T00:26:31.338 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_truncate": 0, 2024-04-25T00:26:31.338 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_delete": 18, 2024-04-25T00:26:31.338 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_mapext": 0, 2024-04-25T00:26:31.338 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_sparse_read": 0, 2024-04-25T00:26:31.338 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_clonerange": 0, 2024-04-25T00:26:31.338 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_getxattr": 6, 2024-04-25T00:26:31.338 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_setxattr": 20, 2024-04-25T00:26:31.338 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_cmpxattr": 0, 2024-04-25T00:26:31.338 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_rmxattr": 0, 2024-04-25T00:26:31.338 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_resetxattrs": 0, 2024-04-25T00:26:31.339 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_call": 0, 2024-04-25T00:26:31.339 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_watch": 0, 2024-04-25T00:26:31.339 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_notify": 0, 2024-04-25T00:26:31.339 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T00:26:31.339 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls": 0, 2024-04-25T00:26:31.339 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls_filter": 0, 2024-04-25T00:26:31.339 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_other": 0, 2024-04-25T00:26:31.339 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_active": 0, 2024-04-25T00:26:31.339 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_send": 0, 2024-04-25T00:26:31.339 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_resend": 0, 2024-04-25T00:26:31.339 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_ping": 0, 2024-04-25T00:26:31.339 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_active": 0, 2024-04-25T00:26:31.339 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_send": 0, 2024-04-25T00:26:31.339 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_resend": 0, 2024-04-25T00:26:31.339 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_active": 0, 2024-04-25T00:26:31.340 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_send": 0, 2024-04-25T00:26:31.340 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_resend": 0, 2024-04-25T00:26:31.340 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_active": 0, 2024-04-25T00:26:31.340 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_send": 0, 2024-04-25T00:26:31.340 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_resend": 0, 2024-04-25T00:26:31.340 INFO:teuthology.orchestra.run.smithi139.stdout: "command_active": 0, 2024-04-25T00:26:31.340 INFO:teuthology.orchestra.run.smithi139.stdout: "command_send": 0, 2024-04-25T00:26:31.340 INFO:teuthology.orchestra.run.smithi139.stdout: "command_resend": 0, 2024-04-25T00:26:31.340 INFO:teuthology.orchestra.run.smithi139.stdout: "map_epoch": 96, 2024-04-25T00:26:31.340 INFO:teuthology.orchestra.run.smithi139.stdout: "map_full": 0, 2024-04-25T00:26:31.340 INFO:teuthology.orchestra.run.smithi139.stdout: "map_inc": 7, 2024-04-25T00:26:31.340 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_sessions": 11, 2024-04-25T00:26:31.340 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_open": 11, 2024-04-25T00:26:31.340 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_close": 0, 2024-04-25T00:26:31.341 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_laggy": 0, 2024-04-25T00:26:31.341 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_wr": 31, 2024-04-25T00:26:31.341 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_rd": 0, 2024-04-25T00:26:31.341 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_del": 4 2024-04-25T00:26:31.341 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.341 INFO:teuthology.orchestra.run.smithi139.stdout: "oft": { 2024-04-25T00:26:31.341 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_objs": 1, 2024-04-25T00:26:31.341 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_kv_pairs": 0, 2024-04-25T00:26:31.341 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_updates": 0, 2024-04-25T00:26:31.341 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_removes": 0 2024-04-25T00:26:31.341 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.341 INFO:teuthology.orchestra.run.smithi139.stdout: "osd-slow-ops": { 2024-04-25T00:26:31.341 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_ops_count": 0 2024-04-25T00:26:31.341 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.342 INFO:teuthology.orchestra.run.smithi139.stdout: "purge_queue": { 2024-04-25T00:26:31.342 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops": 0, 2024-04-25T00:26:31.342 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops_high_water": 6, 2024-04-25T00:26:31.342 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing": 0, 2024-04-25T00:26:31.342 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_high_water": 3, 2024-04-25T00:26:31.342 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed_ops": 6, 2024-04-25T00:26:31.342 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed": 3, 2024-04-25T00:26:31.342 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_item_in_journal": 0 2024-04-25T00:26:31.342 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.342 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T00:26:31.342 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:31.342 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-04-25T00:26:31.342 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:31.343 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 1519, 2024-04-25T00:26:31.343 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 2525264, 2024-04-25T00:26:31.343 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:31.343 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 1519, 2024-04-25T00:26:31.343 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T00:26:31.343 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T00:26:31.343 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 1519, 2024-04-25T00:26:31.343 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 2525264, 2024-04-25T00:26:31.343 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:31.343 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.343 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.343 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.343 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:31.343 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.344 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_bytes": { 2024-04-25T00:26:31.344 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:31.344 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-04-25T00:26:31.344 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:31.344 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T00:26:31.344 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T00:26:31.344 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:31.344 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:31.344 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 95, 2024-04-25T00:26:31.344 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 87930, 2024-04-25T00:26:31.344 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 95, 2024-04-25T00:26:31.344 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 87930, 2024-04-25T00:26:31.344 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:31.344 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.345 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.345 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.345 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:31.345 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.345 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_ops": { 2024-04-25T00:26:31.345 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:31.345 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 1024, 2024-04-25T00:26:31.345 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:31.345 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T00:26:31.345 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T00:26:31.345 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:31.345 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:31.345 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 95, 2024-04-25T00:26:31.345 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 95, 2024-04-25T00:26:31.345 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 95, 2024-04-25T00:26:31.346 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 95, 2024-04-25T00:26:31.346 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:31.346 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.346 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.346 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.346 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:31.346 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.346 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle": { 2024-04-25T00:26:31.346 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:31.346 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-04-25T00:26:31.346 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:31.346 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 3, 2024-04-25T00:26:31.346 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 375, 2024-04-25T00:26:31.346 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:31.347 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 3, 2024-04-25T00:26:31.347 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T00:26:31.347 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T00:26:31.347 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 1, 2024-04-25T00:26:31.347 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 375, 2024-04-25T00:26:31.347 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:31.347 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.347 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.347 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.347 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:31.347 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T00:26:31.347 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle-0x55ca38078500": { 2024-04-25T00:26:31.347 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T00:26:31.347 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-04-25T00:26:31.347 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T00:26:31.347 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 34, 2024-04-25T00:26:31.348 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 73849, 2024-04-25T00:26:31.348 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:31.348 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 34, 2024-04-25T00:26:31.348 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T00:26:31.348 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T00:26:31.348 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 26, 2024-04-25T00:26:31.348 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 73849, 2024-04-25T00:26:31.348 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T00:26:31.348 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T00:26:31.348 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T00:26:31.348 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T00:26:31.348 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:31.348 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T00:26:31.348 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-04-25T00:26:32.144 INFO:tasks.check_counter:Daemon mds.k mds.exported=3 2024-04-25T00:26:32.145 INFO:tasks.check_counter:Daemon mds.k mds.imported=3 2024-04-25T00:26:32.145 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=6 2024-04-25T00:26:32.145 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-04-25T00:26:32.145 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T00:26:32.145 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=0 2024-04-25T00:26:32.145 DEBUG:tasks.check_counter:Getting stats from c 2024-04-25T00:26:32.145 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-25T00:26:32.442 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T00:26:32.857 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:32 smithi153 ceph-mon[32896]: pgmap v328: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T00:26:33.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:32 smithi027 ceph-mon[29208]: pgmap v328: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T00:26:33.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:32 smithi139 ceph-mon[32833]: pgmap v328: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T00:26:34.419 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-25T00:26:34.420 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T00:26:34.420 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 408, 2024-04-25T00:26:34.420 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 328, 2024-04-25T00:26:34.420 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 2142244, 2024-04-25T00:26:34.420 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 283834, 2024-04-25T00:26:34.420 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 19, 2024-04-25T00:26:34.420 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 5, 2024-04-25T00:26:34.420 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.061678174, 2024-04-25T00:26:34.420 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.015670064, 2024-04-25T00:26:34.420 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.049303554, 2024-04-25T00:26:34.420 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.006557837, 2024-04-25T00:26:34.420 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:34.421 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 323, 2024-04-25T00:26:34.421 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.027355888, 2024-04-25T00:26:34.421 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000084693 2024-04-25T00:26:34.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.421 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:34.421 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 39, 2024-04-25T00:26:34.421 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000005616, 2024-04-25T00:26:34.421 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000144 2024-04-25T00:26:34.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.421 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 2133680, 2024-04-25T00:26:34.421 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 149840 2024-04-25T00:26:34.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.422 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T00:26:34.422 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 50, 2024-04-25T00:26:34.422 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 221, 2024-04-25T00:26:34.422 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 13216, 2024-04-25T00:26:34.422 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 611203, 2024-04-25T00:26:34.422 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 21, 2024-04-25T00:26:34.422 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 5, 2024-04-25T00:26:34.422 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.023067070, 2024-04-25T00:26:34.423 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.008407471, 2024-04-25T00:26:34.423 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.011449346, 2024-04-25T00:26:34.423 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000706726, 2024-04-25T00:26:34.423 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:34.423 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 219, 2024-04-25T00:26:34.423 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.028151315, 2024-04-25T00:26:34.423 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000128544 2024-04-25T00:26:34.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.424 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:34.424 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 38, 2024-04-25T00:26:34.424 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000008297, 2024-04-25T00:26:34.424 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000218 2024-04-25T00:26:34.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.424 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 5072, 2024-04-25T00:26:34.425 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 453168 2024-04-25T00:26:34.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.425 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T00:26:34.425 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 897, 2024-04-25T00:26:34.425 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 1058, 2024-04-25T00:26:34.425 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 476698, 2024-04-25T00:26:34.425 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 680574, 2024-04-25T00:26:34.425 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 11, 2024-04-25T00:26:34.425 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 4, 2024-04-25T00:26:34.426 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.092547045, 2024-04-25T00:26:34.426 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.034954893, 2024-04-25T00:26:34.426 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.059982781, 2024-04-25T00:26:34.426 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.005034402, 2024-04-25T00:26:34.426 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:34.426 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1060, 2024-04-25T00:26:34.426 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.063104492, 2024-04-25T00:26:34.427 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000059532 2024-04-25T00:26:34.427 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.427 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:34.427 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1757, 2024-04-25T00:26:34.427 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000446174, 2024-04-25T00:26:34.427 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000253 2024-04-25T00:26:34.427 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.427 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 7424, 2024-04-25T00:26:34.427 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 9424 2024-04-25T00:26:34.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.428 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-04-25T00:26:34.428 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-04-25T00:26:34.428 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-04-25T00:26:34.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.428 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-04-25T00:26:34.428 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-25T00:26:34.428 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-25T00:26:34.428 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 73, 2024-04-25T00:26:34.429 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.004716657, 2024-04-25T00:26:34.429 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000064611 2024-04-25T00:26:34.429 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:34.429 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.429 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-04-25T00:26:34.429 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-25T00:26:34.429 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-25T00:26:34.429 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 15, 2024-04-25T00:26:34.429 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000393573, 2024-04-25T00:26:34.429 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000026238 2024-04-25T00:26:34.430 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:34.430 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.430 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-25T00:26:34.430 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 24, 2024-04-25T00:26:34.430 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 6, 2024-04-25T00:26:34.430 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-04-25T00:26:34.430 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 6, 2024-04-25T00:26:34.431 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.003530357, 2024-04-25T00:26:34.431 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000588392 2024-04-25T00:26:34.431 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.431 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-04-25T00:26:34.431 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 18, 2024-04-25T00:26:34.431 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-04-25T00:26:34.431 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-04-25T00:26:34.431 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 11, 2024-04-25T00:26:34.431 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-04-25T00:26:34.432 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-04-25T00:26:34.432 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 14, 2024-04-25T00:26:34.432 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 1, 2024-04-25T00:26:34.432 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 2, 2024-04-25T00:26:34.432 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 11, 2024-04-25T00:26:34.432 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 11, 2024-04-25T00:26:34.432 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 17, 2024-04-25T00:26:34.433 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 0, 2024-04-25T00:26:34.433 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 0, 2024-04-25T00:26:34.433 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 3, 2024-04-25T00:26:34.433 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 34, 2024-04-25T00:26:34.433 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 11, 2024-04-25T00:26:34.433 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 15, 2024-04-25T00:26:34.433 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 3, 2024-04-25T00:26:34.433 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-04-25T00:26:34.433 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-04-25T00:26:34.434 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-04-25T00:26:34.434 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-04-25T00:26:34.434 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-04-25T00:26:34.434 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 1, 2024-04-25T00:26:34.434 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 3, 2024-04-25T00:26:34.434 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 1, 2024-04-25T00:26:34.434 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 2, 2024-04-25T00:26:34.435 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-04-25T00:26:34.435 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 2, 2024-04-25T00:26:34.435 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-04-25T00:26:34.435 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 4, 2024-04-25T00:26:34.435 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 234, 2024-04-25T00:26:34.435 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 6, 2024-04-25T00:26:34.435 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 2, 2024-04-25T00:26:34.435 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-04-25T00:26:34.436 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T00:26:34.436 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-04-25T00:26:34.436 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-04-25T00:26:34.436 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T00:26:34.436 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T00:26:34.436 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 2, 2024-04-25T00:26:34.436 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-04-25T00:26:34.436 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 1, 2024-04-25T00:26:34.437 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 2, 2024-04-25T00:26:34.437 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T00:26:34.437 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T00:26:34.437 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T00:26:34.437 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 1, 2024-04-25T00:26:34.437 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-04-25T00:26:34.437 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 1, 2024-04-25T00:26:34.437 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 4 2024-04-25T00:26:34.438 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.438 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-04-25T00:26:34.438 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-04-25T00:26:34.438 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-04-25T00:26:34.438 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-04-25T00:26:34.438 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 4, 2024-04-25T00:26:34.438 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-04-25T00:26:34.439 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-04-25T00:26:34.439 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-04-25T00:26:34.439 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-04-25T00:26:34.439 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 1, 2024-04-25T00:26:34.439 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 1, 2024-04-25T00:26:34.439 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-04-25T00:26:34.439 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-04-25T00:26:34.439 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-04-25T00:26:34.440 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-04-25T00:26:34.440 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-04-25T00:26:34.440 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-04-25T00:26:34.440 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-04-25T00:26:34.440 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-04-25T00:26:34.440 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-04-25T00:26:34.440 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T00:26:34.441 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 1, 2024-04-25T00:26:34.441 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-04-25T00:26:34.441 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-04-25T00:26:34.441 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-04-25T00:26:34.441 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-04-25T00:26:34.441 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.441 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-04-25T00:26:34.441 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-04-25T00:26:34.442 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 16, 2024-04-25T00:26:34.442 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-04-25T00:26:34.442 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-04-25T00:26:34.442 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 16, 2024-04-25T00:26:34.442 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-04-25T00:26:34.442 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-04-25T00:26:34.442 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 2, 2024-04-25T00:26:34.442 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-04-25T00:26:34.443 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-04-25T00:26:34.443 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-04-25T00:26:34.443 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 2, 2024-04-25T00:26:34.443 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-04-25T00:26:34.443 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-04-25T00:26:34.443 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-04-25T00:26:34.443 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 4223182, 2024-04-25T00:26:34.444 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 0, 2024-04-25T00:26:34.444 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-04-25T00:26:34.444 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 14, 2024-04-25T00:26:34.444 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.125475910, 2024-04-25T00:26:34.444 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.008962565 2024-04-25T00:26:34.444 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.444 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 0 2024-04-25T00:26:34.444 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.445 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-04-25T00:26:34.445 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 18, 2024-04-25T00:26:34.445 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 28, 2024-04-25T00:26:34.445 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 10, 2024-04-25T00:26:34.445 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 17, 2024-04-25T00:26:34.445 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 26, 2024-04-25T00:26:34.445 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 9, 2024-04-25T00:26:34.445 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 14, 2024-04-25T00:26:34.446 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 32, 2024-04-25T00:26:34.446 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 18, 2024-04-25T00:26:34.446 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 0, 2024-04-25T00:26:34.446 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 6, 2024-04-25T00:26:34.446 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 6, 2024-04-25T00:26:34.446 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 53556, 2024-04-25T00:26:34.446 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 223612 2024-04-25T00:26:34.446 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.446 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-04-25T00:26:34.447 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 29, 2024-04-25T00:26:34.447 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 1, 2024-04-25T00:26:34.447 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 24, 2024-04-25T00:26:34.447 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 25, 2024-04-25T00:26:34.447 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 4, 2024-04-25T00:26:34.447 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-04-25T00:26:34.447 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.447 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.448 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.448 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.448 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-04-25T00:26:34.448 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-04-25T00:26:34.448 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000428746, 2024-04-25T00:26:34.448 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000214373 2024-04-25T00:26:34.448 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.448 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-04-25T00:26:34.448 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.449 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.449 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.449 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.449 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-04-25T00:26:34.449 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.449 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.449 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.450 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.450 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-04-25T00:26:34.450 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-04-25T00:26:34.450 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000847629, 2024-04-25T00:26:34.450 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000423814 2024-04-25T00:26:34.450 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.450 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-04-25T00:26:34.450 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.451 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.451 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.451 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.451 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-04-25T00:26:34.451 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.451 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.451 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.451 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.451 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-04-25T00:26:34.452 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.452 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.452 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.452 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.452 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-04-25T00:26:34.452 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.452 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.452 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.453 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.453 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-04-25T00:26:34.453 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.453 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.453 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.453 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.453 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-04-25T00:26:34.453 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.454 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.454 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.454 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.454 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-04-25T00:26:34.455 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.455 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.455 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.455 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.456 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-04-25T00:26:34.456 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.456 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.456 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.456 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.456 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-04-25T00:26:34.456 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.456 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.457 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.457 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.457 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-04-25T00:26:34.457 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.457 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.457 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.457 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.457 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-04-25T00:26:34.458 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1, 2024-04-25T00:26:34.458 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000299394, 2024-04-25T00:26:34.458 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000299394 2024-04-25T00:26:34.458 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.458 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-04-25T00:26:34.458 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.458 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.458 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.459 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.459 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-04-25T00:26:34.459 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.459 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.459 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.459 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.459 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-04-25T00:26:34.459 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.460 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.460 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.460 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.460 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-04-25T00:26:34.460 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.460 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.460 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.460 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.461 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-04-25T00:26:34.461 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.461 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.461 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.461 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.461 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-04-25T00:26:34.461 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.461 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.462 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.462 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.462 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-04-25T00:26:34.462 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.462 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.462 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.462 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.463 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-04-25T00:26:34.463 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.463 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.463 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.463 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.463 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-04-25T00:26:34.463 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.463 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.463 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.464 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.464 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-04-25T00:26:34.464 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.464 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.464 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.464 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.464 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-04-25T00:26:34.464 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.464 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.465 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.465 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.465 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-04-25T00:26:34.465 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.465 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.465 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.465 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.466 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-04-25T00:26:34.466 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1, 2024-04-25T00:26:34.466 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.001954588, 2024-04-25T00:26:34.466 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001954588 2024-04-25T00:26:34.466 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.467 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-04-25T00:26:34.467 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-04-25T00:26:34.467 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-04-25T00:26:34.467 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.467 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.467 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.467 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:34.468 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.468 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-04-25T00:26:34.468 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 2, 2024-04-25T00:26:34.468 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 4, 2024-04-25T00:26:34.468 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 2, 2024-04-25T00:26:34.468 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 2, 2024-04-25T00:26:34.469 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-04-25T00:26:34.469 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 1, 2024-04-25T00:26:34.469 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 0, 2024-04-25T00:26:34.469 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 636, 2024-04-25T00:26:34.469 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-04-25T00:26:34.469 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.469 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-04-25T00:26:34.469 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-04-25T00:26:34.469 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-04-25T00:26:34.470 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T00:26:34.470 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-04-25T00:26:34.470 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T00:26:34.470 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-04-25T00:26:34.470 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T00:26:34.470 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T00:26:34.470 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T00:26:34.470 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T00:26:34.470 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T00:26:34.470 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-04-25T00:26:34.470 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T00:26:34.470 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T00:26:34.471 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-04-25T00:26:34.471 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-04-25T00:26:34.471 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-04-25T00:26:34.471 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-04-25T00:26:34.471 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T00:26:34.471 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T00:26:34.471 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T00:26:34.471 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T00:26:34.471 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T00:26:34.471 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-04-25T00:26:34.471 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-04-25T00:26:34.471 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-04-25T00:26:34.471 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T00:26:34.472 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T00:26:34.472 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-04-25T00:26:34.472 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-04-25T00:26:34.472 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-04-25T00:26:34.472 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-04-25T00:26:34.472 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T00:26:34.472 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-04-25T00:26:34.472 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T00:26:34.472 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-04-25T00:26:34.472 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 266887, 2024-04-25T00:26:34.472 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 107, 2024-04-25T00:26:34.472 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-04-25T00:26:34.472 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-04-25T00:26:34.472 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-04-25T00:26:34.472 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-04-25T00:26:34.473 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-04-25T00:26:34.473 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-04-25T00:26:34.473 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-04-25T00:26:34.473 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-04-25T00:26:34.473 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 9512, 2024-04-25T00:26:34.473 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 182, 2024-04-25T00:26:34.473 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T00:26:34.473 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-04-25T00:26:34.473 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-04-25T00:26:34.473 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-04-25T00:26:34.473 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 77376, 2024-04-25T00:26:34.473 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 293, 2024-04-25T00:26:34.473 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-04-25T00:26:34.473 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-04-25T00:26:34.474 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-04-25T00:26:34.474 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-04-25T00:26:34.474 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.474 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-04-25T00:26:34.474 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-04-25T00:26:34.474 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-04-25T00:26:34.474 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 73, 2024-04-25T00:26:34.474 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 41526, 2024-04-25T00:26:34.474 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 33, 2024-04-25T00:26:34.474 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 73, 2024-04-25T00:26:34.474 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-04-25T00:26:34.474 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 73, 2024-04-25T00:26:34.475 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.655988485, 2024-04-25T00:26:34.475 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.008986143 2024-04-25T00:26:34.475 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.475 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-04-25T00:26:34.475 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-04-25T00:26:34.475 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 73, 2024-04-25T00:26:34.475 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 104 2024-04-25T00:26:34.475 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.475 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 73, 2024-04-25T00:26:34.475 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 3, 2024-04-25T00:26:34.475 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 70, 2024-04-25T00:26:34.475 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-04-25T00:26:34.475 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-04-25T00:26:34.475 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 0, 2024-04-25T00:26:34.475 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 10, 2024-04-25T00:26:34.476 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 1, 2024-04-25T00:26:34.476 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 15, 2024-04-25T00:26:34.476 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 11, 2024-04-25T00:26:34.476 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-04-25T00:26:34.476 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-04-25T00:26:34.476 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-04-25T00:26:34.476 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-04-25T00:26:34.476 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 14, 2024-04-25T00:26:34.476 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-04-25T00:26:34.476 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-04-25T00:26:34.476 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-04-25T00:26:34.476 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 2, 2024-04-25T00:26:34.476 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 20, 2024-04-25T00:26:34.476 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-04-25T00:26:34.476 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-04-25T00:26:34.477 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-04-25T00:26:34.477 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-04-25T00:26:34.477 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-04-25T00:26:34.477 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-04-25T00:26:34.477 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T00:26:34.477 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-04-25T00:26:34.477 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-04-25T00:26:34.477 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-04-25T00:26:34.477 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-04-25T00:26:34.477 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-04-25T00:26:34.477 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-04-25T00:26:34.477 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-04-25T00:26:34.477 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-04-25T00:26:34.477 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-04-25T00:26:34.477 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-04-25T00:26:34.477 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-04-25T00:26:34.477 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-04-25T00:26:34.478 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-04-25T00:26:34.478 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-04-25T00:26:34.478 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-04-25T00:26:34.478 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-04-25T00:26:34.478 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-04-25T00:26:34.478 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-04-25T00:26:34.478 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-04-25T00:26:34.478 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 96, 2024-04-25T00:26:34.478 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-04-25T00:26:34.478 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 7, 2024-04-25T00:26:34.478 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 10, 2024-04-25T00:26:34.478 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 10, 2024-04-25T00:26:34.478 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-04-25T00:26:34.479 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-04-25T00:26:34.479 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 27, 2024-04-25T00:26:34.479 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 0, 2024-04-25T00:26:34.479 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 4 2024-04-25T00:26:34.479 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.479 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-04-25T00:26:34.479 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 1, 2024-04-25T00:26:34.479 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-04-25T00:26:34.479 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-04-25T00:26:34.479 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-04-25T00:26:34.480 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.480 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-04-25T00:26:34.480 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-04-25T00:26:34.480 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.480 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-04-25T00:26:34.480 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-04-25T00:26:34.480 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 2, 2024-04-25T00:26:34.480 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-04-25T00:26:34.480 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 1, 2024-04-25T00:26:34.480 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 2, 2024-04-25T00:26:34.480 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 1, 2024-04-25T00:26:34.480 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-04-25T00:26:34.480 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.480 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T00:26:34.480 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:34.480 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-25T00:26:34.481 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:34.481 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 1351, 2024-04-25T00:26:34.481 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 2487988, 2024-04-25T00:26:34.481 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:34.481 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 1351, 2024-04-25T00:26:34.481 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T00:26:34.481 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T00:26:34.481 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 1351, 2024-04-25T00:26:34.481 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2487988, 2024-04-25T00:26:34.481 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:34.481 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.481 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.481 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.481 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:34.481 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-04-25T00:26:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-25T00:26:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T00:26:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T00:26:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 73, 2024-04-25T00:26:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 41651, 2024-04-25T00:26:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 73, 2024-04-25T00:26:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 41651, 2024-04-25T00:26:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-04-25T00:26:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-04-25T00:26:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T00:26:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T00:26:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 73, 2024-04-25T00:26:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 73, 2024-04-25T00:26:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 73, 2024-04-25T00:26:34.484 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 73, 2024-04-25T00:26:34.484 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:34.484 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.484 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.484 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.484 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:34.484 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.484 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-04-25T00:26:34.484 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:34.484 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-25T00:26:34.484 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:34.484 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 1, 2024-04-25T00:26:34.484 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 125, 2024-04-25T00:26:34.484 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:34.485 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 1, 2024-04-25T00:26:34.485 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T00:26:34.485 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T00:26:34.485 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 1, 2024-04-25T00:26:34.485 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 125, 2024-04-25T00:26:34.485 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:34.485 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.485 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.485 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.485 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:34.485 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:34.485 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x560fd47c8500": { 2024-04-25T00:26:34.485 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:34.485 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-25T00:26:34.485 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:34.485 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 16, 2024-04-25T00:26:34.486 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 28878, 2024-04-25T00:26:34.486 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:34.486 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 16, 2024-04-25T00:26:34.486 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T00:26:34.486 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T00:26:34.486 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 14, 2024-04-25T00:26:34.486 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 28878, 2024-04-25T00:26:34.486 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:34.486 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:34.486 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:34.486 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:34.486 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:34.486 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:34.486 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-25T00:26:35.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:34 smithi027 ceph-mon[29208]: pgmap v329: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:26:35.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:34 smithi139 ceph-mon[32833]: pgmap v329: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:26:35.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:34 smithi153 ceph-mon[32896]: pgmap v329: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:26:35.409 INFO:tasks.check_counter:Daemon mds.c mds.exported=1 2024-04-25T00:26:35.409 INFO:tasks.check_counter:Daemon mds.c mds.imported=1 2024-04-25T00:26:35.410 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-25T00:26:35.410 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-25T00:26:35.410 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T00:26:35.410 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-04-25T00:26:35.410 DEBUG:tasks.check_counter:Getting stats from f 2024-04-25T00:26:35.410 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-25T00:26:35.661 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T00:26:36.857 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:36 smithi153 ceph-mon[32896]: pgmap v330: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:26:37.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:36 smithi139 ceph-mon[32833]: pgmap v330: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:26:37.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:36 smithi027 ceph-mon[29208]: pgmap v330: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:26:37.606 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-25T00:26:37.606 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T00:26:37.606 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 2155, 2024-04-25T00:26:37.606 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 2157, 2024-04-25T00:26:37.606 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 719417, 2024-04-25T00:26:37.606 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 685297, 2024-04-25T00:26:37.606 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 6, 2024-04-25T00:26:37.606 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-04-25T00:26:37.607 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.182108542, 2024-04-25T00:26:37.607 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.056636446, 2024-04-25T00:26:37.607 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.046501722, 2024-04-25T00:26:37.607 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.065584939, 2024-04-25T00:26:37.607 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:37.607 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2155, 2024-04-25T00:26:37.607 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.083650758, 2024-04-25T00:26:37.607 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000038817 2024-04-25T00:26:37.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.607 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:37.607 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.607 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.607 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.608 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T00:26:37.608 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T00:26:37.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.608 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T00:26:37.608 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 4, 2024-04-25T00:26:37.608 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 171, 2024-04-25T00:26:37.608 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 876, 2024-04-25T00:26:37.608 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 453274, 2024-04-25T00:26:37.608 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 6, 2024-04-25T00:26:37.608 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 4, 2024-04-25T00:26:37.608 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.010609109, 2024-04-25T00:26:37.608 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.006422131, 2024-04-25T00:26:37.608 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.002385561, 2024-04-25T00:26:37.608 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000062401, 2024-04-25T00:26:37.609 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:37.609 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 168, 2024-04-25T00:26:37.609 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.014376838, 2024-04-25T00:26:37.609 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000085576 2024-04-25T00:26:37.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.609 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:37.609 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.609 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.609 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.609 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T00:26:37.609 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 452576 2024-04-25T00:26:37.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.609 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T00:26:37.610 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 327, 2024-04-25T00:26:37.610 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 236, 2024-04-25T00:26:37.610 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 2125084, 2024-04-25T00:26:37.610 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 138574, 2024-04-25T00:26:37.610 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 3, 2024-04-25T00:26:37.610 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-04-25T00:26:37.610 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.050192894, 2024-04-25T00:26:37.610 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.013243190, 2024-04-25T00:26:37.610 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.044316264, 2024-04-25T00:26:37.610 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.005702335, 2024-04-25T00:26:37.610 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:37.610 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 236, 2024-04-25T00:26:37.610 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.014787455, 2024-04-25T00:26:37.611 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000062658 2024-04-25T00:26:37.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.611 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:37.611 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.611 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.611 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.611 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 2124816, 2024-04-25T00:26:37.611 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 138256 2024-04-25T00:26:37.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.611 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-04-25T00:26:37.611 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-04-25T00:26:37.611 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-04-25T00:26:37.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.612 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-04-25T00:26:37.612 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-25T00:26:37.612 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-25T00:26:37.612 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 3609, 2024-04-25T00:26:37.612 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.100421924, 2024-04-25T00:26:37.612 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000027825 2024-04-25T00:26:37.612 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:37.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.612 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-04-25T00:26:37.612 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-25T00:26:37.612 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-25T00:26:37.612 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.612 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.612 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.613 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:37.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.613 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-25T00:26:37.613 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 0, 2024-04-25T00:26:37.613 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 0, 2024-04-25T00:26:37.613 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-04-25T00:26:37.613 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.613 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.613 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.613 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-04-25T00:26:37.613 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 0, 2024-04-25T00:26:37.613 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-04-25T00:26:37.613 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-04-25T00:26:37.614 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 0, 2024-04-25T00:26:37.614 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-04-25T00:26:37.614 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-04-25T00:26:37.614 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 5, 2024-04-25T00:26:37.614 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 0, 2024-04-25T00:26:37.614 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 5, 2024-04-25T00:26:37.614 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-04-25T00:26:37.614 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 5, 2024-04-25T00:26:37.614 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 0, 2024-04-25T00:26:37.614 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 0, 2024-04-25T00:26:37.614 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 0, 2024-04-25T00:26:37.614 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 1, 2024-04-25T00:26:37.614 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 0, 2024-04-25T00:26:37.615 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 0, 2024-04-25T00:26:37.615 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 0, 2024-04-25T00:26:37.615 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 0, 2024-04-25T00:26:37.615 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-04-25T00:26:37.615 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-04-25T00:26:37.615 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-04-25T00:26:37.615 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-04-25T00:26:37.615 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-04-25T00:26:37.615 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-04-25T00:26:37.615 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-04-25T00:26:37.615 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-04-25T00:26:37.615 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-04-25T00:26:37.615 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-04-25T00:26:37.615 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 0, 2024-04-25T00:26:37.616 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-04-25T00:26:37.616 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 0, 2024-04-25T00:26:37.616 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 0, 2024-04-25T00:26:37.616 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 0, 2024-04-25T00:26:37.616 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T00:26:37.616 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-04-25T00:26:37.616 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T00:26:37.616 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-04-25T00:26:37.616 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-04-25T00:26:37.616 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T00:26:37.616 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T00:26:37.616 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 0, 2024-04-25T00:26:37.616 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-04-25T00:26:37.616 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T00:26:37.617 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 0, 2024-04-25T00:26:37.617 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T00:26:37.617 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T00:26:37.617 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T00:26:37.617 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 0, 2024-04-25T00:26:37.617 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-04-25T00:26:37.617 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 0, 2024-04-25T00:26:37.617 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 0 2024-04-25T00:26:37.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.617 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-04-25T00:26:37.617 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-04-25T00:26:37.617 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-04-25T00:26:37.617 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-04-25T00:26:37.617 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 0, 2024-04-25T00:26:37.617 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-04-25T00:26:37.618 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-04-25T00:26:37.618 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-04-25T00:26:37.618 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-04-25T00:26:37.618 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 0, 2024-04-25T00:26:37.618 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 0, 2024-04-25T00:26:37.618 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-04-25T00:26:37.618 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-04-25T00:26:37.618 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-04-25T00:26:37.618 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-04-25T00:26:37.618 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-04-25T00:26:37.618 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-04-25T00:26:37.618 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-04-25T00:26:37.618 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-04-25T00:26:37.618 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-04-25T00:26:37.619 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T00:26:37.619 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-04-25T00:26:37.619 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-04-25T00:26:37.619 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-04-25T00:26:37.619 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-04-25T00:26:37.619 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-04-25T00:26:37.619 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.619 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-04-25T00:26:37.619 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-04-25T00:26:37.619 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 0, 2024-04-25T00:26:37.619 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-04-25T00:26:37.619 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-04-25T00:26:37.619 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 36, 2024-04-25T00:26:37.619 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-04-25T00:26:37.619 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-04-25T00:26:37.619 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 0, 2024-04-25T00:26:37.620 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-04-25T00:26:37.620 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-04-25T00:26:37.620 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-04-25T00:26:37.620 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 2, 2024-04-25T00:26:37.620 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-04-25T00:26:37.620 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-04-25T00:26:37.620 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-04-25T00:26:37.620 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 4302408, 2024-04-25T00:26:37.620 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 4299803, 2024-04-25T00:26:37.620 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-04-25T00:26:37.620 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.620 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.620 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.621 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.621 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 36 2024-04-25T00:26:37.621 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.621 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-04-25T00:26:37.621 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 4, 2024-04-25T00:26:37.621 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 28, 2024-04-25T00:26:37.621 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 24, 2024-04-25T00:26:37.621 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 2, 2024-04-25T00:26:37.621 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 19, 2024-04-25T00:26:37.621 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 17, 2024-04-25T00:26:37.621 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 5, 2024-04-25T00:26:37.621 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 36, 2024-04-25T00:26:37.621 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 31, 2024-04-25T00:26:37.621 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 0, 2024-04-25T00:26:37.622 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 0, 2024-04-25T00:26:37.622 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 0, 2024-04-25T00:26:37.622 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 51064, 2024-04-25T00:26:37.622 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 215420 2024-04-25T00:26:37.622 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.622 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-04-25T00:26:37.622 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 0, 2024-04-25T00:26:37.622 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-04-25T00:26:37.622 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 0, 2024-04-25T00:26:37.622 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 0, 2024-04-25T00:26:37.622 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 0, 2024-04-25T00:26:37.622 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-04-25T00:26:37.622 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.622 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.623 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.623 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.623 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-04-25T00:26:37.623 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.623 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.623 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.623 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.623 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-04-25T00:26:37.623 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.623 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.623 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.623 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.623 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-04-25T00:26:37.623 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.623 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.624 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.624 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-04-25T00:26:37.624 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.624 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.624 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.624 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-04-25T00:26:37.624 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.624 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.624 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.624 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-04-25T00:26:37.624 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.624 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.624 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.625 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-04-25T00:26:37.625 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.625 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.625 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.625 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-04-25T00:26:37.625 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.625 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.625 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.625 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-04-25T00:26:37.625 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.625 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.625 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.626 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-04-25T00:26:37.626 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.626 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.626 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.626 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-04-25T00:26:37.626 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.626 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.626 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.626 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-04-25T00:26:37.626 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.626 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.627 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.627 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-04-25T00:26:37.627 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.627 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.627 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.627 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-04-25T00:26:37.627 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.627 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.627 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.627 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-04-25T00:26:37.628 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.628 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.628 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.628 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-04-25T00:26:37.628 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.628 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.628 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.628 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-04-25T00:26:37.628 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.628 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.628 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.628 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-04-25T00:26:37.629 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.629 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.629 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.629 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-04-25T00:26:37.629 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.629 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.629 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.629 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-04-25T00:26:37.629 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.629 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.629 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.629 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-04-25T00:26:37.630 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.630 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.630 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.630 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-04-25T00:26:37.630 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.630 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.630 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.630 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-04-25T00:26:37.630 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.630 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.630 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.631 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-04-25T00:26:37.631 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.631 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.631 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.631 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-04-25T00:26:37.631 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.631 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.631 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.631 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-04-25T00:26:37.631 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.631 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.631 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.632 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-04-25T00:26:37.632 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.632 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.632 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.632 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-04-25T00:26:37.632 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.632 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.632 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.632 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-04-25T00:26:37.632 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-04-25T00:26:37.632 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-04-25T00:26:37.632 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.632 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.633 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.633 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:37.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.633 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-04-25T00:26:37.633 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 2, 2024-04-25T00:26:37.633 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 2, 2024-04-25T00:26:37.633 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 0, 2024-04-25T00:26:37.633 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 2, 2024-04-25T00:26:37.633 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-04-25T00:26:37.633 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 0, 2024-04-25T00:26:37.633 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 0, 2024-04-25T00:26:37.633 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 1911, 2024-04-25T00:26:37.633 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-04-25T00:26:37.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.633 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-04-25T00:26:37.634 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-04-25T00:26:37.634 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-04-25T00:26:37.634 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T00:26:37.634 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-04-25T00:26:37.634 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T00:26:37.634 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-04-25T00:26:37.634 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T00:26:37.634 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T00:26:37.634 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T00:26:37.634 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T00:26:37.634 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T00:26:37.634 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-04-25T00:26:37.634 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T00:26:37.635 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T00:26:37.635 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-04-25T00:26:37.635 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-04-25T00:26:37.635 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-04-25T00:26:37.635 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-04-25T00:26:37.635 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T00:26:37.635 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T00:26:37.635 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T00:26:37.635 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T00:26:37.635 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T00:26:37.635 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-04-25T00:26:37.635 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-04-25T00:26:37.635 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-04-25T00:26:37.635 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T00:26:37.635 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T00:26:37.635 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-04-25T00:26:37.635 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-04-25T00:26:37.636 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-04-25T00:26:37.636 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-04-25T00:26:37.636 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T00:26:37.636 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-04-25T00:26:37.636 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T00:26:37.636 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-04-25T00:26:37.636 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 181353, 2024-04-25T00:26:37.636 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 48, 2024-04-25T00:26:37.636 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-04-25T00:26:37.636 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-04-25T00:26:37.636 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-04-25T00:26:37.636 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-04-25T00:26:37.636 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-04-25T00:26:37.636 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-04-25T00:26:37.636 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-04-25T00:26:37.636 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-04-25T00:26:37.636 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 9512, 2024-04-25T00:26:37.637 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 182, 2024-04-25T00:26:37.637 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T00:26:37.637 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-04-25T00:26:37.637 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-04-25T00:26:37.637 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-04-25T00:26:37.637 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 17184, 2024-04-25T00:26:37.637 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 52, 2024-04-25T00:26:37.637 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-04-25T00:26:37.637 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-04-25T00:26:37.637 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-04-25T00:26:37.637 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-04-25T00:26:37.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.637 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-04-25T00:26:37.637 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-04-25T00:26:37.637 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-04-25T00:26:37.637 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 2158, 2024-04-25T00:26:37.638 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 21, 2024-04-25T00:26:37.638 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 3, 2024-04-25T00:26:37.638 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 2158, 2024-04-25T00:26:37.638 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-04-25T00:26:37.638 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2158, 2024-04-25T00:26:37.638 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 23.756577187, 2024-04-25T00:26:37.638 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.011008608 2024-04-25T00:26:37.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.638 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-04-25T00:26:37.638 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-04-25T00:26:37.638 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2158, 2024-04-25T00:26:37.638 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 2159 2024-04-25T00:26:37.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.638 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 2158, 2024-04-25T00:26:37.638 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 2158, 2024-04-25T00:26:37.639 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 0, 2024-04-25T00:26:37.639 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-04-25T00:26:37.639 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-04-25T00:26:37.639 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 1384, 2024-04-25T00:26:37.639 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 0, 2024-04-25T00:26:37.639 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 772, 2024-04-25T00:26:37.639 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 0, 2024-04-25T00:26:37.639 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 0, 2024-04-25T00:26:37.639 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-04-25T00:26:37.639 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-04-25T00:26:37.639 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-04-25T00:26:37.639 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-04-25T00:26:37.639 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 0, 2024-04-25T00:26:37.639 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-04-25T00:26:37.639 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-04-25T00:26:37.639 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-04-25T00:26:37.640 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 1, 2024-04-25T00:26:37.640 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 0, 2024-04-25T00:26:37.640 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-04-25T00:26:37.640 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-04-25T00:26:37.640 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-04-25T00:26:37.640 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-04-25T00:26:37.640 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-04-25T00:26:37.640 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-04-25T00:26:37.640 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T00:26:37.640 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-04-25T00:26:37.640 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-04-25T00:26:37.640 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-04-25T00:26:37.640 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-04-25T00:26:37.640 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-04-25T00:26:37.640 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-04-25T00:26:37.640 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-04-25T00:26:37.641 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-04-25T00:26:37.641 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-04-25T00:26:37.641 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-04-25T00:26:37.641 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-04-25T00:26:37.641 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-04-25T00:26:37.641 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-04-25T00:26:37.641 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-04-25T00:26:37.641 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-04-25T00:26:37.641 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-04-25T00:26:37.641 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-04-25T00:26:37.641 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-04-25T00:26:37.641 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-04-25T00:26:37.641 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 96, 2024-04-25T00:26:37.641 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-04-25T00:26:37.641 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 7, 2024-04-25T00:26:37.641 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 4, 2024-04-25T00:26:37.642 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 4, 2024-04-25T00:26:37.642 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-04-25T00:26:37.642 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-04-25T00:26:37.642 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 0, 2024-04-25T00:26:37.642 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 2, 2024-04-25T00:26:37.642 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 0 2024-04-25T00:26:37.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.642 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-04-25T00:26:37.642 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 0, 2024-04-25T00:26:37.642 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-04-25T00:26:37.642 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-04-25T00:26:37.642 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-04-25T00:26:37.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.642 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-04-25T00:26:37.643 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-04-25T00:26:37.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.643 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-04-25T00:26:37.643 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-04-25T00:26:37.643 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T00:26:37.643 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-04-25T00:26:37.643 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 0, 2024-04-25T00:26:37.643 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 0, 2024-04-25T00:26:37.643 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 0, 2024-04-25T00:26:37.643 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-04-25T00:26:37.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.643 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T00:26:37.643 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:37.643 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-25T00:26:37.644 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:37.644 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 2482, 2024-04-25T00:26:37.644 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 2603182, 2024-04-25T00:26:37.644 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:37.644 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 2482, 2024-04-25T00:26:37.644 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T00:26:37.644 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T00:26:37.644 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2482, 2024-04-25T00:26:37.644 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2603182, 2024-04-25T00:26:37.644 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:37.644 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.644 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.644 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.645 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:37.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.645 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-04-25T00:26:37.645 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:37.645 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-25T00:26:37.645 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:37.645 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T00:26:37.645 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T00:26:37.645 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:37.645 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:37.645 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2158, 2024-04-25T00:26:37.645 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 108109, 2024-04-25T00:26:37.645 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2158, 2024-04-25T00:26:37.645 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 108109, 2024-04-25T00:26:37.645 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:37.646 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.646 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.646 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.646 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:37.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.646 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-04-25T00:26:37.646 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:37.646 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-04-25T00:26:37.646 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:37.646 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T00:26:37.646 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T00:26:37.646 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:37.646 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:37.647 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2158, 2024-04-25T00:26:37.647 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 2158, 2024-04-25T00:26:37.647 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2158, 2024-04-25T00:26:37.647 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2158, 2024-04-25T00:26:37.647 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:37.647 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.647 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.647 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.647 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:37.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.647 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-04-25T00:26:37.647 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:37.647 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-25T00:26:37.647 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T00:26:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T00:26:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T00:26:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T00:26:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-04-25T00:26:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-04-25T00:26:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x5603501b86e0": { 2024-04-25T00:26:37.649 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:37.649 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-25T00:26:37.649 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:37.649 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T00:26:37.649 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T00:26:37.649 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:37.649 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:37.649 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T00:26:37.649 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T00:26:37.649 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-04-25T00:26:37.649 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-04-25T00:26:37.649 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:37.649 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:37.649 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:37.650 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:37.650 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:37.650 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:37.650 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-25T00:26:38.375 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-25T00:26:38.375 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-25T00:26:38.375 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-04-25T00:26:38.375 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-25T00:26:38.375 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T00:26:38.375 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-04-25T00:26:38.375 DEBUG:tasks.check_counter:Getting stats from i 2024-04-25T00:26:38.376 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-25T00:26:38.615 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T00:26:39.016 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:38 smithi153 ceph-mon[32896]: pgmap v331: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-25T00:26:39.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:38 smithi027 ceph-mon[29208]: pgmap v331: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-25T00:26:39.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:38 smithi139 ceph-mon[32833]: pgmap v331: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-25T00:26:40.396 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-25T00:26:40.397 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T00:26:40.397 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 1115, 2024-04-25T00:26:40.397 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 1212, 2024-04-25T00:26:40.397 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 556942, 2024-04-25T00:26:40.397 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 753553, 2024-04-25T00:26:40.397 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 9, 2024-04-25T00:26:40.397 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 4, 2024-04-25T00:26:40.397 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.104973317, 2024-04-25T00:26:40.397 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.038305593, 2024-04-25T00:26:40.397 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.065836689, 2024-04-25T00:26:40.397 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.007816980, 2024-04-25T00:26:40.397 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:40.397 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1211, 2024-04-25T00:26:40.397 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.068254850, 2024-04-25T00:26:40.398 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000056362 2024-04-25T00:26:40.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.398 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:40.398 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2040, 2024-04-25T00:26:40.398 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000527471, 2024-04-25T00:26:40.398 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000258 2024-04-25T00:26:40.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.398 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-25T00:26:40.398 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T00:26:40.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.398 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T00:26:40.398 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 42, 2024-04-25T00:26:40.398 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 223, 2024-04-25T00:26:40.398 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 9805, 2024-04-25T00:26:40.399 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 539389, 2024-04-25T00:26:40.399 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 17, 2024-04-25T00:26:40.399 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 5, 2024-04-25T00:26:40.399 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.017259020, 2024-04-25T00:26:40.399 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.007280934, 2024-04-25T00:26:40.399 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.007605740, 2024-04-25T00:26:40.399 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000579006, 2024-04-25T00:26:40.399 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:40.399 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 220, 2024-04-25T00:26:40.399 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.030462247, 2024-04-25T00:26:40.399 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000138464 2024-04-25T00:26:40.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.399 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:40.400 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 49, 2024-04-25T00:26:40.400 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000009576, 2024-04-25T00:26:40.400 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000195 2024-04-25T00:26:40.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.400 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3616, 2024-04-25T00:26:40.400 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 460032 2024-04-25T00:26:40.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.400 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T00:26:40.400 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 418, 2024-04-25T00:26:40.400 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 352, 2024-04-25T00:26:40.400 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 2149416, 2024-04-25T00:26:40.400 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 387624, 2024-04-25T00:26:40.400 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 24, 2024-04-25T00:26:40.400 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 8, 2024-04-25T00:26:40.401 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.067043399, 2024-04-25T00:26:40.401 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.018064400, 2024-04-25T00:26:40.401 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.053240704, 2024-04-25T00:26:40.401 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.006832780, 2024-04-25T00:26:40.401 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:40.401 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 351, 2024-04-25T00:26:40.401 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.021473045, 2024-04-25T00:26:40.401 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000061176 2024-04-25T00:26:40.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.401 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:40.401 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 92, 2024-04-25T00:26:40.401 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000016746, 2024-04-25T00:26:40.401 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000182 2024-04-25T00:26:40.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.401 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 2142096, 2024-04-25T00:26:40.402 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 164464 2024-04-25T00:26:40.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.402 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-04-25T00:26:40.402 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-04-25T00:26:40.402 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-04-25T00:26:40.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.402 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-04-25T00:26:40.402 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-25T00:26:40.402 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-25T00:26:40.402 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 105, 2024-04-25T00:26:40.402 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.011648497, 2024-04-25T00:26:40.402 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000110938 2024-04-25T00:26:40.402 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:40.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.403 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-04-25T00:26:40.403 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-25T00:26:40.403 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-25T00:26:40.403 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 19, 2024-04-25T00:26:40.403 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000743116, 2024-04-25T00:26:40.403 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000039111 2024-04-25T00:26:40.403 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:40.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.403 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-25T00:26:40.403 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 23, 2024-04-25T00:26:40.403 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 11, 2024-04-25T00:26:40.403 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-04-25T00:26:40.403 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 11, 2024-04-25T00:26:40.403 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.011820531, 2024-04-25T00:26:40.404 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001074593 2024-04-25T00:26:40.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.404 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-04-25T00:26:40.404 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 12, 2024-04-25T00:26:40.404 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-04-25T00:26:40.404 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-04-25T00:26:40.404 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 11, 2024-04-25T00:26:40.404 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-04-25T00:26:40.404 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-04-25T00:26:40.404 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 12, 2024-04-25T00:26:40.404 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 1, 2024-04-25T00:26:40.404 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 1, 2024-04-25T00:26:40.404 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 10, 2024-04-25T00:26:40.404 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 10, 2024-04-25T00:26:40.405 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 34, 2024-04-25T00:26:40.405 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 1, 2024-04-25T00:26:40.405 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 1, 2024-04-25T00:26:40.405 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 3, 2024-04-25T00:26:40.405 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 47, 2024-04-25T00:26:40.405 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 18, 2024-04-25T00:26:40.405 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 10, 2024-04-25T00:26:40.405 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 6, 2024-04-25T00:26:40.405 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-04-25T00:26:40.405 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-04-25T00:26:40.405 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-04-25T00:26:40.405 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-04-25T00:26:40.405 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-04-25T00:26:40.405 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 4, 2024-04-25T00:26:40.405 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 12, 2024-04-25T00:26:40.405 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 4, 2024-04-25T00:26:40.405 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 8, 2024-04-25T00:26:40.406 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-04-25T00:26:40.406 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 8, 2024-04-25T00:26:40.406 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-04-25T00:26:40.406 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 4, 2024-04-25T00:26:40.406 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 234, 2024-04-25T00:26:40.406 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 6, 2024-04-25T00:26:40.406 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 8, 2024-04-25T00:26:40.406 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-04-25T00:26:40.406 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T00:26:40.406 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-04-25T00:26:40.406 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-04-25T00:26:40.406 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T00:26:40.406 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T00:26:40.406 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 8, 2024-04-25T00:26:40.407 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-04-25T00:26:40.407 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 4, 2024-04-25T00:26:40.407 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 8, 2024-04-25T00:26:40.407 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T00:26:40.407 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T00:26:40.407 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T00:26:40.407 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 4, 2024-04-25T00:26:40.407 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-04-25T00:26:40.407 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 1, 2024-04-25T00:26:40.407 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 8 2024-04-25T00:26:40.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.407 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-04-25T00:26:40.407 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-04-25T00:26:40.407 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-04-25T00:26:40.407 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-04-25T00:26:40.407 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 5, 2024-04-25T00:26:40.408 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-04-25T00:26:40.408 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-04-25T00:26:40.408 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-04-25T00:26:40.408 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-04-25T00:26:40.408 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 4, 2024-04-25T00:26:40.408 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 4, 2024-04-25T00:26:40.408 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-04-25T00:26:40.408 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-04-25T00:26:40.408 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-04-25T00:26:40.408 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-04-25T00:26:40.408 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-04-25T00:26:40.408 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-04-25T00:26:40.408 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-04-25T00:26:40.408 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-04-25T00:26:40.408 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-04-25T00:26:40.408 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T00:26:40.409 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 4, 2024-04-25T00:26:40.409 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-04-25T00:26:40.409 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-04-25T00:26:40.409 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-04-25T00:26:40.409 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-04-25T00:26:40.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.409 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-04-25T00:26:40.409 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-04-25T00:26:40.409 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 36, 2024-04-25T00:26:40.409 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-04-25T00:26:40.409 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-04-25T00:26:40.409 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 36, 2024-04-25T00:26:40.409 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-04-25T00:26:40.409 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-04-25T00:26:40.409 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 2, 2024-04-25T00:26:40.410 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-04-25T00:26:40.410 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-04-25T00:26:40.410 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-04-25T00:26:40.410 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 2, 2024-04-25T00:26:40.410 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-04-25T00:26:40.410 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-04-25T00:26:40.410 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-04-25T00:26:40.410 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 4302408, 2024-04-25T00:26:40.410 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 0, 2024-04-25T00:26:40.410 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-04-25T00:26:40.410 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 25, 2024-04-25T00:26:40.410 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.107000911, 2024-04-25T00:26:40.410 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.004280036 2024-04-25T00:26:40.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.410 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 0 2024-04-25T00:26:40.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.411 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-04-25T00:26:40.411 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 16, 2024-04-25T00:26:40.411 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 37, 2024-04-25T00:26:40.411 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 21, 2024-04-25T00:26:40.411 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 15, 2024-04-25T00:26:40.411 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 29, 2024-04-25T00:26:40.411 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 14, 2024-04-25T00:26:40.411 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 12, 2024-04-25T00:26:40.411 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 50, 2024-04-25T00:26:40.411 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 38, 2024-04-25T00:26:40.411 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 1, 2024-04-25T00:26:40.411 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 9, 2024-04-25T00:26:40.411 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 8, 2024-04-25T00:26:40.412 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 54040, 2024-04-25T00:26:40.412 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 223612 2024-04-25T00:26:40.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.412 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-04-25T00:26:40.412 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 35, 2024-04-25T00:26:40.412 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 4, 2024-04-25T00:26:40.412 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 23, 2024-04-25T00:26:40.412 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 17, 2024-04-25T00:26:40.412 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 16, 2024-04-25T00:26:40.412 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-04-25T00:26:40.412 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.412 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.412 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.412 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-04-25T00:26:40.413 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 3, 2024-04-25T00:26:40.413 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000827338, 2024-04-25T00:26:40.413 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000275779 2024-04-25T00:26:40.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.413 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-04-25T00:26:40.413 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.413 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.413 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.413 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-04-25T00:26:40.413 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.413 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.413 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.413 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-04-25T00:26:40.414 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.414 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.414 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.414 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.414 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-04-25T00:26:40.414 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.414 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.414 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.414 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.414 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-04-25T00:26:40.414 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.414 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.414 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.414 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.415 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-04-25T00:26:40.415 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.415 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.415 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.415 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-04-25T00:26:40.415 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.415 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.415 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.415 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-04-25T00:26:40.415 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.415 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.415 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.415 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-04-25T00:26:40.416 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.416 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.416 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.416 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-04-25T00:26:40.416 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.416 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-04-25T00:26:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-04-25T00:26:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-04-25T00:26:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.418 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.418 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.418 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-04-25T00:26:40.418 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4, 2024-04-25T00:26:40.418 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.001398781, 2024-04-25T00:26:40.418 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000349695 2024-04-25T00:26:40.418 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.418 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-04-25T00:26:40.418 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.418 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.418 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.418 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.418 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-04-25T00:26:40.418 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.419 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.419 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.419 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.419 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-04-25T00:26:40.419 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.419 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.419 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.419 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.419 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-04-25T00:26:40.419 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.419 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.419 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.419 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.419 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-04-25T00:26:40.420 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.420 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.420 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.420 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-04-25T00:26:40.420 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.420 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.420 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.420 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-04-25T00:26:40.420 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.420 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.421 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.421 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-04-25T00:26:40.421 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.421 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.421 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.421 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-04-25T00:26:40.421 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.421 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.421 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.421 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-04-25T00:26:40.421 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.422 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.422 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.422 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-04-25T00:26:40.422 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.422 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.422 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.422 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-04-25T00:26:40.422 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.422 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.422 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.422 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-04-25T00:26:40.422 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4, 2024-04-25T00:26:40.423 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.009594412, 2024-04-25T00:26:40.423 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.002398603 2024-04-25T00:26:40.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.423 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-04-25T00:26:40.423 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-04-25T00:26:40.423 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-04-25T00:26:40.423 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.423 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.423 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.423 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:40.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.423 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-04-25T00:26:40.423 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 2, 2024-04-25T00:26:40.423 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 2, 2024-04-25T00:26:40.423 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 0, 2024-04-25T00:26:40.424 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 2, 2024-04-25T00:26:40.424 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-04-25T00:26:40.424 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 4, 2024-04-25T00:26:40.424 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 2, 2024-04-25T00:26:40.424 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 638, 2024-04-25T00:26:40.424 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-04-25T00:26:40.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.424 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-04-25T00:26:40.424 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-04-25T00:26:40.424 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-04-25T00:26:40.424 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T00:26:40.424 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-04-25T00:26:40.424 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T00:26:40.424 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-04-25T00:26:40.424 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T00:26:40.424 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T00:26:40.424 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T00:26:40.425 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T00:26:40.425 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T00:26:40.425 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-04-25T00:26:40.425 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T00:26:40.425 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T00:26:40.425 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-04-25T00:26:40.425 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-04-25T00:26:40.425 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-04-25T00:26:40.425 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-04-25T00:26:40.425 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T00:26:40.425 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T00:26:40.425 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T00:26:40.425 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T00:26:40.425 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T00:26:40.425 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-04-25T00:26:40.426 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-04-25T00:26:40.426 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-04-25T00:26:40.426 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T00:26:40.426 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T00:26:40.426 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-04-25T00:26:40.426 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-04-25T00:26:40.426 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-04-25T00:26:40.426 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-04-25T00:26:40.426 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T00:26:40.426 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-04-25T00:26:40.426 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T00:26:40.426 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-04-25T00:26:40.426 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 306311, 2024-04-25T00:26:40.426 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 117, 2024-04-25T00:26:40.426 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-04-25T00:26:40.427 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-04-25T00:26:40.427 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-04-25T00:26:40.427 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-04-25T00:26:40.427 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-04-25T00:26:40.427 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-04-25T00:26:40.427 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-04-25T00:26:40.427 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-04-25T00:26:40.427 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 9512, 2024-04-25T00:26:40.427 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 182, 2024-04-25T00:26:40.427 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T00:26:40.427 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-04-25T00:26:40.427 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-04-25T00:26:40.427 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-04-25T00:26:40.427 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 69320, 2024-04-25T00:26:40.428 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 260, 2024-04-25T00:26:40.428 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-04-25T00:26:40.428 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-04-25T00:26:40.428 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-04-25T00:26:40.428 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-04-25T00:26:40.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.428 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-04-25T00:26:40.428 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-04-25T00:26:40.428 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-04-25T00:26:40.428 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 94, 2024-04-25T00:26:40.428 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 121913, 2024-04-25T00:26:40.428 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 28, 2024-04-25T00:26:40.428 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 94, 2024-04-25T00:26:40.428 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-04-25T00:26:40.428 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 94, 2024-04-25T00:26:40.429 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.628988923, 2024-04-25T00:26:40.429 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.006691371 2024-04-25T00:26:40.429 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.429 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-04-25T00:26:40.429 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-04-25T00:26:40.429 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 94, 2024-04-25T00:26:40.429 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 124 2024-04-25T00:26:40.429 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.429 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 94, 2024-04-25T00:26:40.429 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 9, 2024-04-25T00:26:40.429 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 85, 2024-04-25T00:26:40.429 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-04-25T00:26:40.429 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-04-25T00:26:40.429 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 0, 2024-04-25T00:26:40.429 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 10, 2024-04-25T00:26:40.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 1, 2024-04-25T00:26:40.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 26, 2024-04-25T00:26:40.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 10, 2024-04-25T00:26:40.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-04-25T00:26:40.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-04-25T00:26:40.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-04-25T00:26:40.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-04-25T00:26:40.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 20, 2024-04-25T00:26:40.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-04-25T00:26:40.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-04-25T00:26:40.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-04-25T00:26:40.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 8, 2024-04-25T00:26:40.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 20, 2024-04-25T00:26:40.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-04-25T00:26:40.431 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-04-25T00:26:40.431 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-04-25T00:26:40.431 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-04-25T00:26:40.431 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-04-25T00:26:40.431 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-04-25T00:26:40.431 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T00:26:40.431 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-04-25T00:26:40.431 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-04-25T00:26:40.431 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-04-25T00:26:40.431 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-04-25T00:26:40.431 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-04-25T00:26:40.431 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-04-25T00:26:40.431 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-04-25T00:26:40.431 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-04-25T00:26:40.431 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-04-25T00:26:40.432 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-04-25T00:26:40.432 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-04-25T00:26:40.432 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-04-25T00:26:40.432 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-04-25T00:26:40.432 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-04-25T00:26:40.432 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-04-25T00:26:40.432 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-04-25T00:26:40.432 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-04-25T00:26:40.432 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-04-25T00:26:40.432 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-04-25T00:26:40.432 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 96, 2024-04-25T00:26:40.432 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-04-25T00:26:40.432 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 7, 2024-04-25T00:26:40.432 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 11, 2024-04-25T00:26:40.433 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 11, 2024-04-25T00:26:40.433 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-04-25T00:26:40.433 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-04-25T00:26:40.433 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 25, 2024-04-25T00:26:40.433 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 0, 2024-04-25T00:26:40.433 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 4 2024-04-25T00:26:40.433 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.433 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-04-25T00:26:40.433 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 1, 2024-04-25T00:26:40.433 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-04-25T00:26:40.433 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-04-25T00:26:40.433 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-04-25T00:26:40.433 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.433 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-04-25T00:26:40.433 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-04-25T00:26:40.434 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.434 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-04-25T00:26:40.434 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-04-25T00:26:40.434 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 8, 2024-04-25T00:26:40.434 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-04-25T00:26:40.434 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 4, 2024-04-25T00:26:40.434 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 8, 2024-04-25T00:26:40.434 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 4, 2024-04-25T00:26:40.434 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-04-25T00:26:40.434 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.434 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T00:26:40.434 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:40.434 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-25T00:26:40.435 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:40.435 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 1571, 2024-04-25T00:26:40.435 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 2552557, 2024-04-25T00:26:40.435 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:40.435 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 1571, 2024-04-25T00:26:40.435 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T00:26:40.435 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T00:26:40.435 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 1571, 2024-04-25T00:26:40.435 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2552557, 2024-04-25T00:26:40.435 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:40.435 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.435 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.435 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.435 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:40.435 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.436 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-04-25T00:26:40.436 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:40.436 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-25T00:26:40.436 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:40.436 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T00:26:40.436 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T00:26:40.436 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:40.436 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:40.436 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 94, 2024-04-25T00:26:40.436 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 122413, 2024-04-25T00:26:40.436 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 94, 2024-04-25T00:26:40.436 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 122413, 2024-04-25T00:26:40.436 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:40.436 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.436 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.437 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.437 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:40.437 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.437 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-04-25T00:26:40.437 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:40.437 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-04-25T00:26:40.437 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:40.437 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T00:26:40.437 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T00:26:40.437 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:40.437 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:40.437 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 94, 2024-04-25T00:26:40.437 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 94, 2024-04-25T00:26:40.437 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 94, 2024-04-25T00:26:40.438 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 94, 2024-04-25T00:26:40.438 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:40.438 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.438 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.438 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.438 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:40.438 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.438 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-04-25T00:26:40.438 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:40.438 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-25T00:26:40.438 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:40.438 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 4, 2024-04-25T00:26:40.438 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 500, 2024-04-25T00:26:40.438 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:40.438 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 4, 2024-04-25T00:26:40.439 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T00:26:40.439 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T00:26:40.439 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 1, 2024-04-25T00:26:40.439 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 500, 2024-04-25T00:26:40.439 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:40.439 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.439 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.439 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.439 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:40.439 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:40.439 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x558171d78500": { 2024-04-25T00:26:40.439 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:40.439 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-25T00:26:40.439 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:40.440 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 36, 2024-04-25T00:26:40.440 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 108104, 2024-04-25T00:26:40.440 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:40.440 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 36, 2024-04-25T00:26:40.440 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T00:26:40.440 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T00:26:40.440 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 25, 2024-04-25T00:26:40.440 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 108104, 2024-04-25T00:26:40.440 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:40.440 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:40.440 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:40.440 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:40.440 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:40.440 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:40.440 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-25T00:26:41.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:40 smithi027 ceph-mon[29208]: pgmap v332: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:26:41.201 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:40 smithi153 ceph-mon[32896]: pgmap v332: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:26:41.230 INFO:tasks.check_counter:Daemon mds.i mds.exported=4 2024-04-25T00:26:41.230 INFO:tasks.check_counter:Daemon mds.i mds.imported=4 2024-04-25T00:26:41.230 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-25T00:26:41.230 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-25T00:26:41.230 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T00:26:41.230 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-04-25T00:26:41.230 DEBUG:tasks.check_counter:Getting stats from l 2024-04-25T00:26:41.230 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 518857f8-0297-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-25T00:26:41.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:40 smithi139 ceph-mon[32833]: pgmap v332: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T00:26:41.467 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T00:26:42.915 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:42 smithi153 ceph-mon[32896]: pgmap v333: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T00:26:43.174 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-25T00:26:43.174 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T00:26:43.174 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 57, 2024-04-25T00:26:43.174 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 63, 2024-04-25T00:26:43.174 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 14044, 2024-04-25T00:26:43.174 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 126310, 2024-04-25T00:26:43.174 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 17, 2024-04-25T00:26:43.174 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-04-25T00:26:43.174 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.010888082, 2024-04-25T00:26:43.174 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.001413471, 2024-04-25T00:26:43.175 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.005073329, 2024-04-25T00:26:43.175 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.001087093, 2024-04-25T00:26:43.175 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:43.175 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 61, 2024-04-25T00:26:43.175 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.017586363, 2024-04-25T00:26:43.175 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000288301 2024-04-25T00:26:43.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.175 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:43.175 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 24, 2024-04-25T00:26:43.175 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000003915, 2024-04-25T00:26:43.175 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000163 2024-04-25T00:26:43.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.175 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1856, 2024-04-25T00:26:43.175 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 2432 2024-04-25T00:26:43.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.176 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T00:26:43.176 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 424, 2024-04-25T00:26:43.176 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 360, 2024-04-25T00:26:43.176 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 2142295, 2024-04-25T00:26:43.176 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 176823, 2024-04-25T00:26:43.176 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 7, 2024-04-25T00:26:43.176 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 6, 2024-04-25T00:26:43.176 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.067741799, 2024-04-25T00:26:43.176 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.018784771, 2024-04-25T00:26:43.176 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.054831637, 2024-04-25T00:26:43.176 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.007428652, 2024-04-25T00:26:43.176 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:43.176 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 359, 2024-04-25T00:26:43.176 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.028038110, 2024-04-25T00:26:43.177 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000078100 2024-04-25T00:26:43.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.177 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:43.177 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 91, 2024-04-25T00:26:43.177 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000023726, 2024-04-25T00:26:43.177 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000260 2024-04-25T00:26:43.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.177 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 2135376, 2024-04-25T00:26:43.177 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 164640 2024-04-25T00:26:43.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.177 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T00:26:43.177 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 865, 2024-04-25T00:26:43.177 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 1204, 2024-04-25T00:26:43.178 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 471365, 2024-04-25T00:26:43.178 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 1222541, 2024-04-25T00:26:43.178 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 27, 2024-04-25T00:26:43.178 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 5, 2024-04-25T00:26:43.178 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.102009921, 2024-04-25T00:26:43.178 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.039512004, 2024-04-25T00:26:43.178 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.062273568, 2024-04-25T00:26:43.178 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.004852098, 2024-04-25T00:26:43.178 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T00:26:43.178 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1202, 2024-04-25T00:26:43.178 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.076442564, 2024-04-25T00:26:43.178 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000063596 2024-04-25T00:26:43.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.179 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T00:26:43.179 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1696, 2024-04-25T00:26:43.179 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000424809, 2024-04-25T00:26:43.179 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000250 2024-04-25T00:26:43.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.179 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 5184, 2024-04-25T00:26:43.179 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 458800 2024-04-25T00:26:43.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.179 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-04-25T00:26:43.179 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-04-25T00:26:43.179 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-04-25T00:26:43.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.179 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-04-25T00:26:43.179 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-25T00:26:43.180 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-25T00:26:43.180 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 93, 2024-04-25T00:26:43.180 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.006982708, 2024-04-25T00:26:43.180 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000075082 2024-04-25T00:26:43.180 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:43.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.180 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-04-25T00:26:43.180 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-25T00:26:43.180 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-25T00:26:43.180 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 17, 2024-04-25T00:26:43.180 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000630982, 2024-04-25T00:26:43.180 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000037116 2024-04-25T00:26:43.180 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:43.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.181 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-25T00:26:43.181 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 30, 2024-04-25T00:26:43.181 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 8, 2024-04-25T00:26:43.181 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-04-25T00:26:43.181 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 8, 2024-04-25T00:26:43.181 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.007822495, 2024-04-25T00:26:43.181 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000977811 2024-04-25T00:26:43.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.181 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-04-25T00:26:43.181 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 22, 2024-04-25T00:26:43.182 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-04-25T00:26:43.182 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-04-25T00:26:43.182 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 11, 2024-04-25T00:26:43.182 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-04-25T00:26:43.182 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-04-25T00:26:43.182 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 12, 2024-04-25T00:26:43.182 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 0, 2024-04-25T00:26:43.182 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 2, 2024-04-25T00:26:43.182 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 10, 2024-04-25T00:26:43.182 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 10, 2024-04-25T00:26:43.182 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 23, 2024-04-25T00:26:43.182 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 1, 2024-04-25T00:26:43.182 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 1, 2024-04-25T00:26:43.182 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 3, 2024-04-25T00:26:43.183 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 40, 2024-04-25T00:26:43.183 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 10, 2024-04-25T00:26:43.183 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 21, 2024-04-25T00:26:43.183 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 3, 2024-04-25T00:26:43.183 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-04-25T00:26:43.183 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-04-25T00:26:43.183 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-04-25T00:26:43.183 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-04-25T00:26:43.183 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-04-25T00:26:43.183 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 1, 2024-04-25T00:26:43.183 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 6, 2024-04-25T00:26:43.183 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 1, 2024-04-25T00:26:43.183 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 4, 2024-04-25T00:26:43.184 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-04-25T00:26:43.184 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 4, 2024-04-25T00:26:43.184 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 1, 2024-04-25T00:26:43.184 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 4, 2024-04-25T00:26:43.184 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 234, 2024-04-25T00:26:43.184 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 6, 2024-04-25T00:26:43.184 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 4, 2024-04-25T00:26:43.184 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-04-25T00:26:43.184 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T00:26:43.184 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-04-25T00:26:43.184 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-04-25T00:26:43.184 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T00:26:43.184 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T00:26:43.184 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 4, 2024-04-25T00:26:43.185 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-04-25T00:26:43.185 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 1, 2024-04-25T00:26:43.185 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 3, 2024-04-25T00:26:43.185 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T00:26:43.185 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T00:26:43.185 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T00:26:43.185 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 1, 2024-04-25T00:26:43.185 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-04-25T00:26:43.185 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 1, 2024-04-25T00:26:43.185 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 4 2024-04-25T00:26:43.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.186 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-04-25T00:26:43.186 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-04-25T00:26:43.186 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-04-25T00:26:43.186 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-04-25T00:26:43.186 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 4, 2024-04-25T00:26:43.186 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-04-25T00:26:43.186 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-04-25T00:26:43.186 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-04-25T00:26:43.186 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-04-25T00:26:43.186 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 2, 2024-04-25T00:26:43.186 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 2, 2024-04-25T00:26:43.186 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-04-25T00:26:43.188 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-04-25T00:26:43.188 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T00:26:43.188 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 1, 2024-04-25T00:26:43.188 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-04-25T00:26:43.188 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-04-25T00:26:43.188 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-04-25T00:26:43.188 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-04-25T00:26:43.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.188 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-04-25T00:26:43.188 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-04-25T00:26:43.188 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 21, 2024-04-25T00:26:43.188 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-04-25T00:26:43.188 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-04-25T00:26:43.188 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 21, 2024-04-25T00:26:43.189 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-04-25T00:26:43.189 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-04-25T00:26:43.189 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 2, 2024-04-25T00:26:43.189 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-04-25T00:26:43.189 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-04-25T00:26:43.189 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-04-25T00:26:43.189 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 2, 2024-04-25T00:26:43.189 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-04-25T00:26:43.189 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-04-25T00:26:43.189 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-04-25T00:26:43.189 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 4230450, 2024-04-25T00:26:43.189 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 0, 2024-04-25T00:26:43.189 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-04-25T00:26:43.190 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 17, 2024-04-25T00:26:43.190 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.040124240, 2024-04-25T00:26:43.190 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.002360249 2024-04-25T00:26:43.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.190 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 0 2024-04-25T00:26:43.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.190 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-04-25T00:26:43.190 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 16, 2024-04-25T00:26:43.190 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 30, 2024-04-25T00:26:43.190 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 14, 2024-04-25T00:26:43.190 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 15, 2024-04-25T00:26:43.191 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 26, 2024-04-25T00:26:43.191 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 11, 2024-04-25T00:26:43.191 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 12, 2024-04-25T00:26:43.191 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 37, 2024-04-25T00:26:43.191 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 25, 2024-04-25T00:26:43.191 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 1, 2024-04-25T00:26:43.191 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 6, 2024-04-25T00:26:43.191 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 5, 2024-04-25T00:26:43.191 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 56372, 2024-04-25T00:26:43.191 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 223612 2024-04-25T00:26:43.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.191 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-04-25T00:26:43.191 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 36, 2024-04-25T00:26:43.191 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 1, 2024-04-25T00:26:43.192 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 30, 2024-04-25T00:26:43.192 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 21, 2024-04-25T00:26:43.192 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 5, 2024-04-25T00:26:43.192 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-04-25T00:26:43.192 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.192 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.192 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.192 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-04-25T00:26:43.193 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1, 2024-04-25T00:26:43.193 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000191565, 2024-04-25T00:26:43.193 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000191565 2024-04-25T00:26:43.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.193 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-04-25T00:26:43.193 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.193 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.193 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.193 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-04-25T00:26:43.193 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.193 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.193 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.194 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-04-25T00:26:43.194 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 3, 2024-04-25T00:26:43.194 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.001656732, 2024-04-25T00:26:43.194 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000552244 2024-04-25T00:26:43.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.194 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-04-25T00:26:43.194 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.194 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.194 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.194 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-04-25T00:26:43.194 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1, 2024-04-25T00:26:43.194 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000109707, 2024-04-25T00:26:43.194 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000109707 2024-04-25T00:26:43.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.195 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-04-25T00:26:43.195 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.195 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.195 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.195 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-04-25T00:26:43.195 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.195 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.195 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.195 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-04-25T00:26:43.195 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.195 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.195 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.196 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-04-25T00:26:43.196 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.196 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.196 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.196 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-04-25T00:26:43.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:42 smithi027 ceph-mon[29208]: pgmap v333: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T00:26:43.197 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.197 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.197 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.197 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-04-25T00:26:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-04-25T00:26:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-04-25T00:26:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.199 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-04-25T00:26:43.199 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1, 2024-04-25T00:26:43.199 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000327662, 2024-04-25T00:26:43.199 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000327662 2024-04-25T00:26:43.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.199 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-04-25T00:26:43.199 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.199 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.199 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.199 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-04-25T00:26:43.199 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.199 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.200 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.200 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-04-25T00:26:43.200 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.200 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.200 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.200 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-04-25T00:26:43.200 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.200 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.200 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.200 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-04-25T00:26:43.201 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.201 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.201 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.201 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-04-25T00:26:43.201 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.201 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.201 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.201 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-04-25T00:26:43.201 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.201 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.201 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.202 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-04-25T00:26:43.202 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.202 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.202 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.202 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-04-25T00:26:43.202 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.202 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.202 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.203 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-04-25T00:26:43.203 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.203 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.203 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.203 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-04-25T00:26:43.203 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.203 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.203 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.203 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-04-25T00:26:43.203 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.204 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.204 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.204 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-04-25T00:26:43.204 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-04-25T00:26:43.204 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.005536829, 2024-04-25T00:26:43.204 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.002768414 2024-04-25T00:26:43.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.204 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-04-25T00:26:43.204 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-04-25T00:26:43.204 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-04-25T00:26:43.204 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.204 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.205 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.205 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:43.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.205 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-04-25T00:26:43.205 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 1, 2024-04-25T00:26:43.205 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 4, 2024-04-25T00:26:43.205 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 3, 2024-04-25T00:26:43.205 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 1, 2024-04-25T00:26:43.205 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-04-25T00:26:43.205 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 3, 2024-04-25T00:26:43.205 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 3, 2024-04-25T00:26:43.205 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 646, 2024-04-25T00:26:43.206 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-04-25T00:26:43.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.206 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-04-25T00:26:43.206 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-04-25T00:26:43.206 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-04-25T00:26:43.206 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T00:26:43.206 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-04-25T00:26:43.206 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T00:26:43.206 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-04-25T00:26:43.206 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T00:26:43.206 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T00:26:43.206 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T00:26:43.206 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T00:26:43.206 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T00:26:43.207 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-04-25T00:26:43.207 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T00:26:43.207 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T00:26:43.207 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-04-25T00:26:43.207 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-04-25T00:26:43.207 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-04-25T00:26:43.207 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-04-25T00:26:43.207 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T00:26:43.207 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T00:26:43.207 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T00:26:43.207 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T00:26:43.207 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T00:26:43.207 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-04-25T00:26:43.207 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-04-25T00:26:43.208 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-04-25T00:26:43.208 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T00:26:43.208 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T00:26:43.208 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-04-25T00:26:43.208 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-04-25T00:26:43.208 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-04-25T00:26:43.208 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-04-25T00:26:43.208 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T00:26:43.208 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-04-25T00:26:43.208 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T00:26:43.208 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-04-25T00:26:43.208 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 274759, 2024-04-25T00:26:43.208 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 108, 2024-04-25T00:26:43.209 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-04-25T00:26:43.209 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-04-25T00:26:43.209 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-04-25T00:26:43.209 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-04-25T00:26:43.209 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-04-25T00:26:43.209 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-04-25T00:26:43.209 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-04-25T00:26:43.209 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-04-25T00:26:43.209 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 9512, 2024-04-25T00:26:43.209 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 182, 2024-04-25T00:26:43.210 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T00:26:43.210 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-04-25T00:26:43.210 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-04-25T00:26:43.210 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-04-25T00:26:43.210 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 69320, 2024-04-25T00:26:43.210 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 260, 2024-04-25T00:26:43.210 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-04-25T00:26:43.210 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-04-25T00:26:43.210 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-04-25T00:26:43.210 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-04-25T00:26:43.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.210 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-04-25T00:26:43.211 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-04-25T00:26:43.211 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-04-25T00:26:43.211 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 79, 2024-04-25T00:26:43.211 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 48948, 2024-04-25T00:26:43.211 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 33, 2024-04-25T00:26:43.211 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 79, 2024-04-25T00:26:43.211 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-04-25T00:26:43.211 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 79, 2024-04-25T00:26:43.211 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.668988246, 2024-04-25T00:26:43.211 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.008468205 2024-04-25T00:26:43.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.211 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-04-25T00:26:43.211 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-04-25T00:26:43.212 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 79, 2024-04-25T00:26:43.212 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 110 2024-04-25T00:26:43.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.212 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 79, 2024-04-25T00:26:43.212 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 5, 2024-04-25T00:26:43.212 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 74, 2024-04-25T00:26:43.212 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-04-25T00:26:43.212 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-04-25T00:26:43.212 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 0, 2024-04-25T00:26:43.212 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 10, 2024-04-25T00:26:43.212 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 1, 2024-04-25T00:26:43.212 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 18, 2024-04-25T00:26:43.212 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 10, 2024-04-25T00:26:43.212 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-04-25T00:26:43.213 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-04-25T00:26:43.213 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-04-25T00:26:43.213 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-04-25T00:26:43.213 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 16, 2024-04-25T00:26:43.213 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-04-25T00:26:43.213 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-04-25T00:26:43.213 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-04-25T00:26:43.213 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 4, 2024-04-25T00:26:43.213 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 20, 2024-04-25T00:26:43.213 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-04-25T00:26:43.213 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-04-25T00:26:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-04-25T00:26:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-04-25T00:26:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-04-25T00:26:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-04-25T00:26:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T00:26:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-04-25T00:26:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-04-25T00:26:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-04-25T00:26:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-04-25T00:26:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-04-25T00:26:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-04-25T00:26:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-04-25T00:26:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-04-25T00:26:43.214 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-04-25T00:26:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-04-25T00:26:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-04-25T00:26:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-04-25T00:26:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-04-25T00:26:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-04-25T00:26:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-04-25T00:26:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-04-25T00:26:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-04-25T00:26:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-04-25T00:26:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-04-25T00:26:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 96, 2024-04-25T00:26:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-04-25T00:26:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 7, 2024-04-25T00:26:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 10, 2024-04-25T00:26:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 10, 2024-04-25T00:26:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-04-25T00:26:43.215 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-04-25T00:26:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 27, 2024-04-25T00:26:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 0, 2024-04-25T00:26:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 4 2024-04-25T00:26:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-04-25T00:26:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 1, 2024-04-25T00:26:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-04-25T00:26:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-04-25T00:26:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-04-25T00:26:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-04-25T00:26:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-04-25T00:26:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-04-25T00:26:43.216 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-04-25T00:26:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 4, 2024-04-25T00:26:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-04-25T00:26:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 2, 2024-04-25T00:26:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 4, 2024-04-25T00:26:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 2, 2024-04-25T00:26:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-04-25T00:26:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T00:26:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-25T00:26:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 1342, 2024-04-25T00:26:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 2483964, 2024-04-25T00:26:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 1342, 2024-04-25T00:26:43.217 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T00:26:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T00:26:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 1342, 2024-04-25T00:26:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2483964, 2024-04-25T00:26:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-04-25T00:26:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-25T00:26:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T00:26:43.218 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T00:26:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 79, 2024-04-25T00:26:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 49198, 2024-04-25T00:26:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 79, 2024-04-25T00:26:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 49198, 2024-04-25T00:26:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.219 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-04-25T00:26:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-04-25T00:26:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T00:26:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T00:26:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T00:26:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 79, 2024-04-25T00:26:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 79, 2024-04-25T00:26:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 79, 2024-04-25T00:26:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 79, 2024-04-25T00:26:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:43.220 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-04-25T00:26:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-25T00:26:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 2, 2024-04-25T00:26:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 250, 2024-04-25T00:26:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 2, 2024-04-25T00:26:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T00:26:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T00:26:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 1, 2024-04-25T00:26:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 250, 2024-04-25T00:26:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T00:26:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x563858366500": { 2024-04-25T00:26:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T00:26:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-25T00:26:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T00:26:43.222 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 21, 2024-04-25T00:26:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 36146, 2024-04-25T00:26:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T00:26:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 21, 2024-04-25T00:26:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T00:26:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T00:26:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 17, 2024-04-25T00:26:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 36146, 2024-04-25T00:26:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T00:26:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T00:26:43.224 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T00:26:43.224 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T00:26:43.224 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:43.224 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T00:26:43.224 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-25T00:26:43.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:42 smithi139 ceph-mon[32833]: pgmap v333: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T00:26:43.966 INFO:tasks.check_counter:Daemon mds.l mds.exported=1 2024-04-25T00:26:43.966 INFO:tasks.check_counter:Daemon mds.l mds.imported=1 2024-04-25T00:26:43.966 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=6 2024-04-25T00:26:43.966 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-04-25T00:26:43.966 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T00:26:43.966 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-04-25T00:26:43.967 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-25T00:26:43.977 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T00:26:43.978 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:26:43.978 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T00:26:44.015 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T00:26:44.015 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T00:26:44.074 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:26:44.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.231+0000 7f74eb87b640 1 -- 172.21.15.27:0/607174387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e4100fe0 msgr2=0x7f74e410d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:44.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.231+0000 7f74eb87b640 1 --2- 172.21.15.27:0/607174387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e4100fe0 0x7f74e410d4f0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f74e0009cd0 tx=0x7f74e002a8e0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.231+0000 7f74eb87b640 1 -- 172.21.15.27:0/607174387 shutdown_connections 2024-04-25T00:26:44.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.231+0000 7f74eb87b640 1 --2- 172.21.15.27:0/607174387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e4100fe0 0x7f74e410d4f0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.231+0000 7f74eb87b640 1 --2- 172.21.15.27:0/607174387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74e4100620 0x7f74e4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.231+0000 7f74eb87b640 1 --2- 172.21.15.27:0/607174387 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74e4106630 0x7f74e4106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.231+0000 7f74eb87b640 1 -- 172.21.15.27:0/607174387 >> 172.21.15.27:0/607174387 conn(0x7f74e4077740 msgr2=0x7f74e4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:44.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.231+0000 7f74eb87b640 1 -- 172.21.15.27:0/607174387 shutdown_connections 2024-04-25T00:26:44.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.231+0000 7f74eb87b640 1 -- 172.21.15.27:0/607174387 wait complete. 2024-04-25T00:26:44.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.232+0000 7f74eb87b640 1 Processor -- start 2024-04-25T00:26:44.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.232+0000 7f74eb87b640 1 -- start start 2024-04-25T00:26:44.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.232+0000 7f74eb87b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74e4100620 0x7f74e4198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:44.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.232+0000 7f74eb87b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74e4100fe0 0x7f74e4198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:44.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.232+0000 7f74eb87b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e4106630 0x7f74e419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:44.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.232+0000 7f74eb87b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f74e4111b40 con 0x7f74e4106630 2024-04-25T00:26:44.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.232+0000 7f74eb87b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f74e41119c0 con 0x7f74e4100fe0 2024-04-25T00:26:44.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.232+0000 7f74eb87b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f74e4111cc0 con 0x7f74e4100620 2024-04-25T00:26:44.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.232+0000 7f74e9e18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e4106630 0x7f74e419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:44.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.232+0000 7f74e9e18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e4106630 0x7f74e419c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56674/0 (socket says 172.21.15.27:56674) 2024-04-25T00:26:44.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.232+0000 7f74e9e18640 1 -- 172.21.15.27:0/12988524 learned_addr learned my addr 172.21.15.27:0/12988524 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:44.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.232+0000 7f74e8e16640 1 --2- 172.21.15.27:0/12988524 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74e4100fe0 0x7f74e4198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:44.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.232+0000 7f74e9617640 1 --2- 172.21.15.27:0/12988524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74e4100620 0x7f74e4198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:44.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.232+0000 7f74e9e18640 1 -- 172.21.15.27:0/12988524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74e4100620 msgr2=0x7f74e4198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:44.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.232+0000 7f74e9e18640 1 --2- 172.21.15.27:0/12988524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74e4100620 0x7f74e4198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.233+0000 7f74e9e18640 1 -- 172.21.15.27:0/12988524 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74e4100fe0 msgr2=0x7f74e4198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:44.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.233+0000 7f74e9e18640 1 --2- 172.21.15.27:0/12988524 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74e4100fe0 0x7f74e4198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.233+0000 7f74e9e18640 1 -- 172.21.15.27:0/12988524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e419cfc0 con 0x7f74e4106630 2024-04-25T00:26:44.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.233+0000 7f74e9617640 1 --2- 172.21.15.27:0/12988524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74e4100620 0x7f74e4198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:26:44.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.233+0000 7f74e9e18640 1 --2- 172.21.15.27:0/12988524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e4106630 0x7f74e419c890 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f74e0005cc0 tx=0x7f74e0032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:44.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.233+0000 7f74da7fc640 1 -- 172.21.15.27:0/12988524 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e0004040 con 0x7f74e4106630 2024-04-25T00:26:44.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.233+0000 7f74eb87b640 1 -- 172.21.15.27:0/12988524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74e419d250 con 0x7f74e4106630 2024-04-25T00:26:44.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.233+0000 7f74da7fc640 1 -- 172.21.15.27:0/12988524 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74e00046c0 con 0x7f74e4106630 2024-04-25T00:26:44.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.233+0000 7f74da7fc640 1 -- 172.21.15.27:0/12988524 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e0030e00 con 0x7f74e4106630 2024-04-25T00:26:44.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.233+0000 7f74eb87b640 1 -- 172.21.15.27:0/12988524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74e41a4b40 con 0x7f74e4106630 2024-04-25T00:26:44.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.234+0000 7f74eb87b640 1 -- 172.21.15.27:0/12988524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74e4068560 con 0x7f74e4106630 2024-04-25T00:26:44.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.235+0000 7f74da7fc640 1 -- 172.21.15.27:0/12988524 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f74e00305a0 con 0x7f74e4106630 2024-04-25T00:26:44.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.236+0000 7f74da7fc640 1 --2- 172.21.15.27:0/12988524 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f74d00a2d00 0x7f74d00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:44.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.236+0000 7f74e9617640 1 --2- 172.21.15.27:0/12988524 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f74d00a2d00 0x7f74d00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:44.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.237+0000 7f74da7fc640 1 -- 172.21.15.27:0/12988524 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9496+0+0 (secure 0 0 0) 0x7f74e002e030 con 0x7f74e4106630 2024-04-25T00:26:44.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.237+0000 7f74e9617640 1 --2- 172.21.15.27:0/12988524 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f74d00a2d00 0x7f74d00a51c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f74dc005e00 tx=0x7f74dc005c70 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:44.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.241+0000 7f74da7fc640 1 -- 172.21.15.27:0/12988524 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f74e004b060 con 0x7f74e4106630 2024-04-25T00:26:44.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.427+0000 7f74eb87b640 1 -- 172.21.15.27:0/12988524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f74e4108c10 con 0x7f74e4106630 2024-04-25T00:26:44.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.429+0000 7f74da7fc640 1 -- 172.21.15.27:0/12988524 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+25290 (secure 0 0 0) 0x7f74e00bd870 con 0x7f74e4106630 2024-04-25T00:26:44.430 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:26:44.430 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":96,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:26:20.745957+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":9,"snap_epoch":95,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":9,"snap_epoch":95,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1885111026":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/2528869729":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/4085720546":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/236412562":"2024-04-26T00:15:45.034899+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/2161554778":"2024-04-26T00:15:45.034899+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6800/2161554778":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":2,"length":1},{"begin":9,"length":1}]},{"pool":4,"snaps":[{"begin":2,"length":1},{"begin":9,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:26:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.431+0000 7f74eb87b640 1 -- 172.21.15.27:0/12988524 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f74d00a2d00 msgr2=0x7f74d00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.431+0000 7f74eb87b640 1 --2- 172.21.15.27:0/12988524 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f74d00a2d00 0x7f74d00a51c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f74dc005e00 tx=0x7f74dc005c70 comp rx=0 tx=0).stop 2024-04-25T00:26:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.431+0000 7f74eb87b640 1 -- 172.21.15.27:0/12988524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e4106630 msgr2=0x7f74e419c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.431+0000 7f74eb87b640 1 --2- 172.21.15.27:0/12988524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e4106630 0x7f74e419c890 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f74e0005cc0 tx=0x7f74e0032f40 comp rx=0 tx=0).stop 2024-04-25T00:26:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.431+0000 7f74eb87b640 1 -- 172.21.15.27:0/12988524 shutdown_connections 2024-04-25T00:26:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.431+0000 7f74eb87b640 1 --2- 172.21.15.27:0/12988524 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f74d00a2d00 0x7f74d00a51c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.431+0000 7f74eb87b640 1 --2- 172.21.15.27:0/12988524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74e4106630 0x7f74e419c890 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.431+0000 7f74eb87b640 1 --2- 172.21.15.27:0/12988524 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74e4100fe0 0x7f74e4198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.431+0000 7f74eb87b640 1 --2- 172.21.15.27:0/12988524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74e4100620 0x7f74e4198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.431+0000 7f74eb87b640 1 -- 172.21.15.27:0/12988524 >> 172.21.15.27:0/12988524 conn(0x7f74e4077740 msgr2=0x7f74e4102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:44.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.431+0000 7f74eb87b640 1 -- 172.21.15.27:0/12988524 shutdown_connections 2024-04-25T00:26:44.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.431+0000 7f74eb87b640 1 -- 172.21.15.27:0/12988524 wait complete. 2024-04-25T00:26:44.443 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:08:34.006973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T00:13:17.337307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T00:13:18.253713+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 9, 'snap_epoch': 95, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T00:13:21.995987+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 9, 'snap_epoch': 95, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T00:26:44.444 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:26:44.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.547+0000 7f991fd14640 1 -- 172.21.15.27:0/1150360513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9918106630 msgr2=0x7f9918106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:44.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.547+0000 7f991fd14640 1 --2- 172.21.15.27:0/1150360513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9918106630 0x7f9918106a10 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f9910009cd0 tx=0x7f991002a8e0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.547+0000 7f991fd14640 1 -- 172.21.15.27:0/1150360513 shutdown_connections 2024-04-25T00:26:44.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.547+0000 7f991fd14640 1 --2- 172.21.15.27:0/1150360513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9918100fe0 0x7f991810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.547+0000 7f991fd14640 1 --2- 172.21.15.27:0/1150360513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9918100620 0x7f9918100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.547+0000 7f991fd14640 1 --2- 172.21.15.27:0/1150360513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9918106630 0x7f9918106a10 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.547+0000 7f991fd14640 1 -- 172.21.15.27:0/1150360513 >> 172.21.15.27:0/1150360513 conn(0x7f99180776a0 msgr2=0x7f9918077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:44.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.547+0000 7f991fd14640 1 -- 172.21.15.27:0/1150360513 shutdown_connections 2024-04-25T00:26:44.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.547+0000 7f991fd14640 1 -- 172.21.15.27:0/1150360513 wait complete. 2024-04-25T00:26:44.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.548+0000 7f991fd14640 1 Processor -- start 2024-04-25T00:26:44.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.548+0000 7f991fd14640 1 -- start start 2024-04-25T00:26:44.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.548+0000 7f991fd14640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9918100620 0x7f9918198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:44.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.548+0000 7f991fd14640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9918100fe0 0x7f9918198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:44.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.548+0000 7f991fd14640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9918106630 0x7f991819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:44.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.548+0000 7f991fd14640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9918111b90 con 0x7f9918100fe0 2024-04-25T00:26:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.548+0000 7f991fd14640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9918111a10 con 0x7f9918106630 2024-04-25T00:26:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.548+0000 7f991fd14640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9918111d10 con 0x7f9918100620 2024-04-25T00:26:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.548+0000 7f991d2af640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9918100fe0 0x7f9918198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.548+0000 7f991dab0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9918100620 0x7f9918198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.548+0000 7f991d2af640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9918100fe0 0x7f9918198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56694/0 (socket says 172.21.15.27:56694) 2024-04-25T00:26:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.548+0000 7f991d2af640 1 -- 172.21.15.27:0/1190990284 learned_addr learned my addr 172.21.15.27:0/1190990284 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.549+0000 7f991e2b1640 1 --2- 172.21.15.27:0/1190990284 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9918106630 0x7f991819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.549+0000 7f991dab0640 1 -- 172.21.15.27:0/1190990284 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9918106630 msgr2=0x7f991819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.549+0000 7f991dab0640 1 --2- 172.21.15.27:0/1190990284 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9918106630 0x7f991819c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.549+0000 7f991dab0640 1 -- 172.21.15.27:0/1190990284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9918100fe0 msgr2=0x7f9918198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.549+0000 7f991dab0640 1 --2- 172.21.15.27:0/1190990284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9918100fe0 0x7f9918198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.549+0000 7f991dab0640 1 -- 172.21.15.27:0/1190990284 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f991819cff0 con 0x7f9918100620 2024-04-25T00:26:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.549+0000 7f991d2af640 1 --2- 172.21.15.27:0/1190990284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9918100fe0 0x7f9918198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:26:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.549+0000 7f991dab0640 1 --2- 172.21.15.27:0/1190990284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9918100620 0x7f9918198050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9910000f80 tx=0x7f9910003970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:44.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.549+0000 7f9916ffd640 1 -- 172.21.15.27:0/1190990284 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9910003d60 con 0x7f9918100620 2024-04-25T00:26:44.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.549+0000 7f991fd14640 1 -- 172.21.15.27:0/1190990284 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f991819d280 con 0x7f9918100620 2024-04-25T00:26:44.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.549+0000 7f9916ffd640 1 -- 172.21.15.27:0/1190990284 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9910004520 con 0x7f9918100620 2024-04-25T00:26:44.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.549+0000 7f9916ffd640 1 -- 172.21.15.27:0/1190990284 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9910030d40 con 0x7f9918100620 2024-04-25T00:26:44.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.549+0000 7f991fd14640 1 -- 172.21.15.27:0/1190990284 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99181a4b70 con 0x7f9918100620 2024-04-25T00:26:44.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.550+0000 7f991fd14640 1 -- 172.21.15.27:0/1190990284 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99180689b0 con 0x7f9918100620 2024-04-25T00:26:44.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.551+0000 7f9916ffd640 1 -- 172.21.15.27:0/1190990284 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f99100046d0 con 0x7f9918100620 2024-04-25T00:26:44.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.552+0000 7f9916ffd640 1 --2- 172.21.15.27:0/1190990284 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f99000a2d00 0x7f99000a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:44.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.552+0000 7f991d2af640 1 --2- 172.21.15.27:0/1190990284 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f99000a2d00 0x7f99000a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:44.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.552+0000 7f9916ffd640 1 -- 172.21.15.27:0/1190990284 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9496+0+0 (secure 0 0 0) 0x7f991002e030 con 0x7f9918100620 2024-04-25T00:26:44.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.552+0000 7f991d2af640 1 --2- 172.21.15.27:0/1190990284 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f99000a2d00 0x7f99000a51c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9918077c10 tx=0x7f990c00a380 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:44.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.556+0000 7f9916ffd640 1 -- 172.21.15.27:0/1190990284 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f99100bd0b0 con 0x7f9918100620 2024-04-25T00:26:44.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.742+0000 7f991fd14640 1 -- 172.21.15.27:0/1190990284 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9918108be0 con 0x7f9918100620 2024-04-25T00:26:44.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.743+0000 7f9916ffd640 1 -- 172.21.15.27:0/1190990284 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7f99100bc9b0 con 0x7f9918100620 2024-04-25T00:26:44.744 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T00:26:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.745+0000 7f991fd14640 1 -- 172.21.15.27:0/1190990284 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f99000a2d00 msgr2=0x7f99000a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.745+0000 7f991fd14640 1 --2- 172.21.15.27:0/1190990284 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f99000a2d00 0x7f99000a51c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9918077c10 tx=0x7f990c00a380 comp rx=0 tx=0).stop 2024-04-25T00:26:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.745+0000 7f991fd14640 1 -- 172.21.15.27:0/1190990284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9918100620 msgr2=0x7f9918198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.745+0000 7f991fd14640 1 --2- 172.21.15.27:0/1190990284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9918100620 0x7f9918198050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9910000f80 tx=0x7f9910003970 comp rx=0 tx=0).stop 2024-04-25T00:26:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.745+0000 7f991fd14640 1 -- 172.21.15.27:0/1190990284 shutdown_connections 2024-04-25T00:26:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.745+0000 7f991fd14640 1 --2- 172.21.15.27:0/1190990284 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f99000a2d00 0x7f99000a51c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.745+0000 7f991fd14640 1 --2- 172.21.15.27:0/1190990284 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9918106630 0x7f991819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.745+0000 7f991fd14640 1 --2- 172.21.15.27:0/1190990284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9918100fe0 0x7f9918198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.745+0000 7f991fd14640 1 --2- 172.21.15.27:0/1190990284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9918100620 0x7f9918198050 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.745+0000 7f991fd14640 1 -- 172.21.15.27:0/1190990284 >> 172.21.15.27:0/1190990284 conn(0x7f99180776a0 msgr2=0x7f9918103130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.745+0000 7f991fd14640 1 -- 172.21.15.27:0/1190990284 shutdown_connections 2024-04-25T00:26:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.745+0000 7f991fd14640 1 -- 172.21.15.27:0/1190990284 wait complete. 2024-04-25T00:26:44.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T00:26:44.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.862+0000 7f50c1d6a640 1 -- 172.21.15.27:0/3090802036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50bc107730 msgr2=0x7f50bc107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.862+0000 7f50c1d6a640 1 --2- 172.21.15.27:0/3090802036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50bc107730 0x7f50bc107b10 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f50ac009930 tx=0x7f50ac02a420 comp rx=0 tx=0).stop 2024-04-25T00:26:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.862+0000 7f50c1d6a640 1 -- 172.21.15.27:0/3090802036 shutdown_connections 2024-04-25T00:26:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.862+0000 7f50c1d6a640 1 --2- 172.21.15.27:0/3090802036 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f50bc06a780 0x7f50bc10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.862+0000 7f50c1d6a640 1 --2- 172.21.15.27:0/3090802036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50bc069dc0 0x7f50bc06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.862+0000 7f50c1d6a640 1 --2- 172.21.15.27:0/3090802036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50bc107730 0x7f50bc107b10 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.862+0000 7f50c1d6a640 1 -- 172.21.15.27:0/3090802036 >> 172.21.15.27:0/3090802036 conn(0x7f50bc0776a0 msgr2=0x7f50bc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.862+0000 7f50c1d6a640 1 -- 172.21.15.27:0/3090802036 shutdown_connections 2024-04-25T00:26:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.862+0000 7f50c1d6a640 1 -- 172.21.15.27:0/3090802036 wait complete. 2024-04-25T00:26:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.863+0000 7f50c1d6a640 1 Processor -- start 2024-04-25T00:26:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.863+0000 7f50c1d6a640 1 -- start start 2024-04-25T00:26:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.863+0000 7f50c1d6a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f50bc069dc0 0x7f50bc100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.863+0000 7f50c1d6a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50bc06a780 0x7f50bc100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.863+0000 7f50c1d6a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50bc107730 0x7f50bc1010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.863+0000 7f50c1d6a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f50bc111b90 con 0x7f50bc06a780 2024-04-25T00:26:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.863+0000 7f50c1d6a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f50bc111a10 con 0x7f50bc069dc0 2024-04-25T00:26:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.863+0000 7f50c1d6a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f50bc111d10 con 0x7f50bc107730 2024-04-25T00:26:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.863+0000 7f50baffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50bc06a780 0x7f50bc100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.863+0000 7f50baffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50bc06a780 0x7f50bc100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56710/0 (socket says 172.21.15.27:56710) 2024-04-25T00:26:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.863+0000 7f50baffd640 1 -- 172.21.15.27:0/3691981290 learned_addr learned my addr 172.21.15.27:0/3691981290 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.863+0000 7f50bb7fe640 1 --2- 172.21.15.27:0/3691981290 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f50bc069dc0 0x7f50bc100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.863+0000 7f50bbfff640 1 --2- 172.21.15.27:0/3691981290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50bc107730 0x7f50bc1010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.863+0000 7f50baffd640 1 -- 172.21.15.27:0/3691981290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50bc107730 msgr2=0x7f50bc1010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.863+0000 7f50baffd640 1 --2- 172.21.15.27:0/3691981290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50bc107730 0x7f50bc1010f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.863+0000 7f50baffd640 1 -- 172.21.15.27:0/3691981290 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f50bc069dc0 msgr2=0x7f50bc100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.863+0000 7f50baffd640 1 --2- 172.21.15.27:0/3691981290 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f50bc069dc0 0x7f50bc100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:44.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.864+0000 7f50baffd640 1 -- 172.21.15.27:0/3691981290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50bc1a6490 con 0x7f50bc06a780 2024-04-25T00:26:44.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.864+0000 7f50bb7fe640 1 --2- 172.21.15.27:0/3691981290 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f50bc069dc0 0x7f50bc100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:26:44.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.864+0000 7f50baffd640 1 --2- 172.21.15.27:0/3691981290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50bc06a780 0x7f50bc100bb0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f50b000d770 tx=0x7f50b000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:44.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.864+0000 7f50b8ff9640 1 -- 172.21.15.27:0/3691981290 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b0017070 con 0x7f50bc06a780 2024-04-25T00:26:44.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.864+0000 7f50c1d6a640 1 -- 172.21.15.27:0/3691981290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50bc1a6780 con 0x7f50bc06a780 2024-04-25T00:26:44.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.864+0000 7f50b8ff9640 1 -- 172.21.15.27:0/3691981290 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50b0005500 con 0x7f50bc06a780 2024-04-25T00:26:44.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.864+0000 7f50b8ff9640 1 -- 172.21.15.27:0/3691981290 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b0004e00 con 0x7f50bc06a780 2024-04-25T00:26:44.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.864+0000 7f50c1d6a640 1 -- 172.21.15.27:0/3691981290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50bc1a6cd0 con 0x7f50bc06a780 2024-04-25T00:26:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.865+0000 7f50c1d6a640 1 -- 172.21.15.27:0/3691981290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50bc0689b0 con 0x7f50bc06a780 2024-04-25T00:26:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.866+0000 7f50b8ff9640 1 -- 172.21.15.27:0/3691981290 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f50b0004fa0 con 0x7f50bc06a780 2024-04-25T00:26:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.867+0000 7f50b8ff9640 1 --2- 172.21.15.27:0/3691981290 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f50a40a2d00 0x7f50a40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.867+0000 7f50bb7fe640 1 --2- 172.21.15.27:0/3691981290 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f50a40a2d00 0x7f50a40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.867+0000 7f50b8ff9640 1 -- 172.21.15.27:0/3691981290 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9496+0+0 (secure 0 0 0) 0x7f50b00cf300 con 0x7f50bc06a780 2024-04-25T00:26:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.867+0000 7f50bb7fe640 1 --2- 172.21.15.27:0/3691981290 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f50a40a2d00 0x7f50a40a51c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f50ac006010 tx=0x7f50ac005a20 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:44.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:44.871+0000 7f50b8ff9640 1 -- 172.21.15.27:0/3691981290 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f50b009bb50 con 0x7f50bc06a780 2024-04-25T00:26:45.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.059+0000 7f50c1d6a640 1 -- 172.21.15.27:0/3691981290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f50bc109c90 con 0x7f50bc06a780 2024-04-25T00:26:45.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.059+0000 7f50b8ff9640 1 -- 172.21.15.27:0/3691981290 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7f50b009b450 con 0x7f50bc06a780 2024-04-25T00:26:45.060 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:26:45.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.061+0000 7f50c1d6a640 1 -- 172.21.15.27:0/3691981290 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f50a40a2d00 msgr2=0x7f50a40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.061+0000 7f50c1d6a640 1 --2- 172.21.15.27:0/3691981290 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f50a40a2d00 0x7f50a40a51c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f50ac006010 tx=0x7f50ac005a20 comp rx=0 tx=0).stop 2024-04-25T00:26:45.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.061+0000 7f50c1d6a640 1 -- 172.21.15.27:0/3691981290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50bc06a780 msgr2=0x7f50bc100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.061+0000 7f50c1d6a640 1 --2- 172.21.15.27:0/3691981290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50bc06a780 0x7f50bc100bb0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f50b000d770 tx=0x7f50b000dc30 comp rx=0 tx=0).stop 2024-04-25T00:26:45.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.061+0000 7f50c1d6a640 1 -- 172.21.15.27:0/3691981290 shutdown_connections 2024-04-25T00:26:45.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.061+0000 7f50c1d6a640 1 --2- 172.21.15.27:0/3691981290 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f50a40a2d00 0x7f50a40a51c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.061+0000 7f50c1d6a640 1 --2- 172.21.15.27:0/3691981290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50bc107730 0x7f50bc1010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.061+0000 7f50c1d6a640 1 --2- 172.21.15.27:0/3691981290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50bc06a780 0x7f50bc100bb0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.061+0000 7f50c1d6a640 1 --2- 172.21.15.27:0/3691981290 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f50bc069dc0 0x7f50bc100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.061+0000 7f50c1d6a640 1 -- 172.21.15.27:0/3691981290 >> 172.21.15.27:0/3691981290 conn(0x7f50bc0776a0 msgr2=0x7f50bc0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:45.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.062+0000 7f50c1d6a640 1 -- 172.21.15.27:0/3691981290 shutdown_connections 2024-04-25T00:26:45.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.062+0000 7f50c1d6a640 1 -- 172.21.15.27:0/3691981290 wait complete. 2024-04-25T00:26:45.072 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:44 smithi027 ceph-mon[29208]: pgmap v334: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:26:45.072 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:44 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/12988524' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:26:45.073 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:44 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1190990284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:26:45.074 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T00:26:45.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.177+0000 7f132fcae640 1 -- 172.21.15.27:0/2707472288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1328100620 msgr2=0x7f1328100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.177+0000 7f132fcae640 1 --2- 172.21.15.27:0/2707472288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1328100620 0x7f1328100aa0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f1318009930 tx=0x7f131802a420 comp rx=0 tx=0).stop 2024-04-25T00:26:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.177+0000 7f132fcae640 1 -- 172.21.15.27:0/2707472288 shutdown_connections 2024-04-25T00:26:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.177+0000 7f132fcae640 1 --2- 172.21.15.27:0/2707472288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1328100fe0 0x7f132810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.177+0000 7f132fcae640 1 --2- 172.21.15.27:0/2707472288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1328100620 0x7f1328100aa0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.177+0000 7f132fcae640 1 --2- 172.21.15.27:0/2707472288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1328106630 0x7f1328106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.177+0000 7f132fcae640 1 -- 172.21.15.27:0/2707472288 >> 172.21.15.27:0/2707472288 conn(0x7f13280776a0 msgr2=0x7f1328077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.177+0000 7f132fcae640 1 -- 172.21.15.27:0/2707472288 shutdown_connections 2024-04-25T00:26:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.177+0000 7f132fcae640 1 -- 172.21.15.27:0/2707472288 wait complete. 2024-04-25T00:26:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.178+0000 7f132fcae640 1 Processor -- start 2024-04-25T00:26:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.178+0000 7f132fcae640 1 -- start start 2024-04-25T00:26:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.178+0000 7f132fcae640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1328100620 0x7f1328196e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.178+0000 7f132fcae640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1328100fe0 0x7f1328197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.178+0000 7f132fcae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1328106630 0x7f132819b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.178+0000 7f132fcae640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1328110a40 con 0x7f1328106630 2024-04-25T00:26:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.178+0000 7f132fcae640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f13281108c0 con 0x7f1328100fe0 2024-04-25T00:26:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.178+0000 7f132fcae640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1328110bc0 con 0x7f1328100620 2024-04-25T00:26:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.178+0000 7f132e24b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1328106630 0x7f132819b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.178+0000 7f132e24b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1328106630 0x7f132819b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56730/0 (socket says 172.21.15.27:56730) 2024-04-25T00:26:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.178+0000 7f132e24b640 1 -- 172.21.15.27:0/3542195441 learned_addr learned my addr 172.21.15.27:0/3542195441 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.178+0000 7f132da4a640 1 --2- 172.21.15.27:0/3542195441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1328100620 0x7f1328196e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.178+0000 7f132d249640 1 --2- 172.21.15.27:0/3542195441 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1328100fe0 0x7f1328197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.178+0000 7f132e24b640 1 -- 172.21.15.27:0/3542195441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1328100620 msgr2=0x7f1328196e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.178+0000 7f132e24b640 1 --2- 172.21.15.27:0/3542195441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1328100620 0x7f1328196e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.178+0000 7f132e24b640 1 -- 172.21.15.27:0/3542195441 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1328100fe0 msgr2=0x7f1328197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.178+0000 7f132e24b640 1 --2- 172.21.15.27:0/3542195441 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1328100fe0 0x7f1328197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.179+0000 7f132e24b640 1 -- 172.21.15.27:0/3542195441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132819bde0 con 0x7f1328106630 2024-04-25T00:26:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.179+0000 7f132da4a640 1 --2- 172.21.15.27:0/3542195441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1328100620 0x7f1328196e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:26:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.179+0000 7f132d249640 1 --2- 172.21.15.27:0/3542195441 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1328100fe0 0x7f1328197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:26:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.179+0000 7f132e24b640 1 --2- 172.21.15.27:0/3542195441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1328106630 0x7f132819b6b0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f132000cce0 tx=0x7f132000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.179+0000 7f131effd640 1 -- 172.21.15.27:0/3542195441 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132000f450 con 0x7f1328106630 2024-04-25T00:26:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.179+0000 7f131effd640 1 -- 172.21.15.27:0/3542195441 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f132000fad0 con 0x7f1328106630 2024-04-25T00:26:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.179+0000 7f131effd640 1 -- 172.21.15.27:0/3542195441 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1320018c10 con 0x7f1328106630 2024-04-25T00:26:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.179+0000 7f132fcae640 1 -- 172.21.15.27:0/3542195441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f132819c0d0 con 0x7f1328106630 2024-04-25T00:26:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.179+0000 7f132fcae640 1 -- 172.21.15.27:0/3542195441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13281a3960 con 0x7f1328106630 2024-04-25T00:26:45.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.180+0000 7f132fcae640 1 -- 172.21.15.27:0/3542195441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13280689b0 con 0x7f1328106630 2024-04-25T00:26:45.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.181+0000 7f131effd640 1 -- 172.21.15.27:0/3542195441 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f132000f5f0 con 0x7f1328106630 2024-04-25T00:26:45.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.182+0000 7f131effd640 1 --2- 172.21.15.27:0/3542195441 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f13140a2d00 0x7f13140a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.182+0000 7f132da4a640 1 --2- 172.21.15.27:0/3542195441 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f13140a2d00 0x7f13140a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.182+0000 7f131effd640 1 -- 172.21.15.27:0/3542195441 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9496+0+0 (secure 0 0 0) 0x7f13200d0000 con 0x7f1328106630 2024-04-25T00:26:45.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.183+0000 7f132da4a640 1 --2- 172.21.15.27:0/3542195441 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f13140a2d00 0x7f13140a51c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1324005e10 tx=0x7f1324005da0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:45.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.186+0000 7f131effd640 1 -- 172.21.15.27:0/3542195441 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1320014030 con 0x7f1328106630 2024-04-25T00:26:45.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:44 smithi139 ceph-mon[32833]: pgmap v334: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:26:45.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:44 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/12988524' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:26:45.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:44 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1190990284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:26:45.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:44 smithi153 ceph-mon[32896]: pgmap v334: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T00:26:45.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:44 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/12988524' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:26:45.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:44 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1190990284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T00:26:45.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.377+0000 7f132fcae640 1 -- 172.21.15.27:0/3542195441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f13281a3c00 con 0x7f1328106630 2024-04-25T00:26:45.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.378+0000 7f131effd640 1 -- 172.21.15.27:0/3542195441 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7f132009c8e0 con 0x7f1328106630 2024-04-25T00:26:45.380 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T00:26:45.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.380+0000 7f132fcae640 1 -- 172.21.15.27:0/3542195441 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f13140a2d00 msgr2=0x7f13140a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.380+0000 7f132fcae640 1 --2- 172.21.15.27:0/3542195441 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f13140a2d00 0x7f13140a51c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1324005e10 tx=0x7f1324005da0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.380+0000 7f132fcae640 1 -- 172.21.15.27:0/3542195441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1328106630 msgr2=0x7f132819b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.380+0000 7f132fcae640 1 --2- 172.21.15.27:0/3542195441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1328106630 0x7f132819b6b0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f132000cce0 tx=0x7f132000a620 comp rx=0 tx=0).stop 2024-04-25T00:26:45.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.380+0000 7f132fcae640 1 -- 172.21.15.27:0/3542195441 shutdown_connections 2024-04-25T00:26:45.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.380+0000 7f132fcae640 1 --2- 172.21.15.27:0/3542195441 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f13140a2d00 0x7f13140a51c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.380+0000 7f132fcae640 1 --2- 172.21.15.27:0/3542195441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1328106630 0x7f132819b6b0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.380+0000 7f132fcae640 1 --2- 172.21.15.27:0/3542195441 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1328100fe0 0x7f1328197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.380+0000 7f132fcae640 1 --2- 172.21.15.27:0/3542195441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1328100620 0x7f1328196e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.380+0000 7f132fcae640 1 -- 172.21.15.27:0/3542195441 >> 172.21.15.27:0/3542195441 conn(0x7f13280776a0 msgr2=0x7f13281030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:45.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.380+0000 7f132fcae640 1 -- 172.21.15.27:0/3542195441 shutdown_connections 2024-04-25T00:26:45.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.380+0000 7f132fcae640 1 -- 172.21.15.27:0/3542195441 wait complete. 2024-04-25T00:26:45.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T00:26:45.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.497+0000 7f88f7fc1640 1 -- 172.21.15.27:0/3941298961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f010bd00 msgr2=0x7f88f010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.497+0000 7f88f7fc1640 1 --2- 172.21.15.27:0/3941298961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f010bd00 0x7f88f010e1a0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f88e8009ca0 tx=0x7f88e802a9e0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.497+0000 7f88f7fc1640 1 -- 172.21.15.27:0/3941298961 shutdown_connections 2024-04-25T00:26:45.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.497+0000 7f88f7fc1640 1 --2- 172.21.15.27:0/3941298961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f010bd00 0x7f88f010e1a0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.497+0000 7f88f7fc1640 1 --2- 172.21.15.27:0/3941298961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f01032e0 0x7f88f010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.497+0000 7f88f7fc1640 1 --2- 172.21.15.27:0/3941298961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88f01029c0 0x7f88f0102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.497+0000 7f88f7fc1640 1 -- 172.21.15.27:0/3941298961 >> 172.21.15.27:0/3941298961 conn(0x7f88f0077740 msgr2=0x7f88f0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:45.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.497+0000 7f88f7fc1640 1 -- 172.21.15.27:0/3941298961 shutdown_connections 2024-04-25T00:26:45.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.497+0000 7f88f7fc1640 1 -- 172.21.15.27:0/3941298961 wait complete. 2024-04-25T00:26:45.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.498+0000 7f88f7fc1640 1 Processor -- start 2024-04-25T00:26:45.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.498+0000 7f88f7fc1640 1 -- start start 2024-04-25T00:26:45.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.498+0000 7f88f7fc1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f01029c0 0x7f88f0100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.498+0000 7f88f7fc1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f01032e0 0x7f88f0100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.498+0000 7f88f7fc1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88f010bd00 0x7f88f0101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.498+0000 7f88f7fc1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f88f0111b00 con 0x7f88f01029c0 2024-04-25T00:26:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.498+0000 7f88f7fc1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f88f0111980 con 0x7f88f010bd00 2024-04-25T00:26:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.498+0000 7f88f7fc1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f88f0111c80 con 0x7f88f01032e0 2024-04-25T00:26:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.498+0000 7f88f5d5d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f01029c0 0x7f88f0100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.498+0000 7f88f5d5d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f01029c0 0x7f88f0100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56750/0 (socket says 172.21.15.27:56750) 2024-04-25T00:26:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.498+0000 7f88f5d5d640 1 -- 172.21.15.27:0/4165582498 learned_addr learned my addr 172.21.15.27:0/4165582498 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.498+0000 7f88f555c640 1 --2- 172.21.15.27:0/4165582498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f01032e0 0x7f88f0100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.498+0000 7f88f655e640 1 --2- 172.21.15.27:0/4165582498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88f010bd00 0x7f88f0101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.499+0000 7f88f5d5d640 1 -- 172.21.15.27:0/4165582498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f01032e0 msgr2=0x7f88f0100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.499+0000 7f88f5d5d640 1 --2- 172.21.15.27:0/4165582498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f01032e0 0x7f88f0100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.499+0000 7f88f5d5d640 1 -- 172.21.15.27:0/4165582498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88f010bd00 msgr2=0x7f88f0101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.499+0000 7f88f5d5d640 1 --2- 172.21.15.27:0/4165582498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88f010bd00 0x7f88f0101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.499+0000 7f88f5d5d640 1 -- 172.21.15.27:0/4165582498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f01a64a0 con 0x7f88f01029c0 2024-04-25T00:26:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.499+0000 7f88f655e640 1 --2- 172.21.15.27:0/4165582498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88f010bd00 0x7f88f0101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:26:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.499+0000 7f88f5d5d640 1 --2- 172.21.15.27:0/4165582498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f01029c0 0x7f88f0100610 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00c890 tx=0x7f88ec00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.499+0000 7f88e6ffd640 1 -- 172.21.15.27:0/4165582498 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec004220 con 0x7f88f01029c0 2024-04-25T00:26:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.499+0000 7f88e6ffd640 1 -- 172.21.15.27:0/4165582498 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88ec0043c0 con 0x7f88f01029c0 2024-04-25T00:26:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.499+0000 7f88f7fc1640 1 -- 172.21.15.27:0/4165582498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f01a6790 con 0x7f88f01029c0 2024-04-25T00:26:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.499+0000 7f88e6ffd640 1 -- 172.21.15.27:0/4165582498 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec0049e0 con 0x7f88f01029c0 2024-04-25T00:26:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.499+0000 7f88f7fc1640 1 -- 172.21.15.27:0/4165582498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f01a6d30 con 0x7f88f01029c0 2024-04-25T00:26:45.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.500+0000 7f88f7fc1640 1 -- 172.21.15.27:0/4165582498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f0068560 con 0x7f88f01029c0 2024-04-25T00:26:45.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.501+0000 7f88e6ffd640 1 -- 172.21.15.27:0/4165582498 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f88ec0079b0 con 0x7f88f01029c0 2024-04-25T00:26:45.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.502+0000 7f88e6ffd640 1 --2- 172.21.15.27:0/4165582498 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f88dc0a2d80 0x7f88dc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.502+0000 7f88f555c640 1 --2- 172.21.15.27:0/4165582498 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f88dc0a2d80 0x7f88dc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.502+0000 7f88e6ffd640 1 -- 172.21.15.27:0/4165582498 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9496+0+0 (secure 0 0 0) 0x7f88ec0cf830 con 0x7f88f01029c0 2024-04-25T00:26:45.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.502+0000 7f88f555c640 1 --2- 172.21.15.27:0/4165582498 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f88dc0a2d80 0x7f88dc0a5240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f88f01020a0 tx=0x7f88e000b390 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:45.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.505+0000 7f88e6ffd640 1 -- 172.21.15.27:0/4165582498 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f88ec014030 con 0x7f88f01029c0 2024-04-25T00:26:45.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.691+0000 7f88f7fc1640 1 -- 172.21.15.27:0/4165582498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f88f0106e80 con 0x7f88f01029c0 2024-04-25T00:26:45.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.691+0000 7f88e6ffd640 1 -- 172.21.15.27:0/4165582498 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7f88ec09c100 con 0x7f88f01029c0 2024-04-25T00:26:45.692 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T00:26:45.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.693+0000 7f88f7fc1640 1 -- 172.21.15.27:0/4165582498 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f88dc0a2d80 msgr2=0x7f88dc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.693+0000 7f88f7fc1640 1 --2- 172.21.15.27:0/4165582498 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f88dc0a2d80 0x7f88dc0a5240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f88f01020a0 tx=0x7f88e000b390 comp rx=0 tx=0).stop 2024-04-25T00:26:45.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.693+0000 7f88f7fc1640 1 -- 172.21.15.27:0/4165582498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f01029c0 msgr2=0x7f88f0100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.693+0000 7f88f7fc1640 1 --2- 172.21.15.27:0/4165582498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f01029c0 0x7f88f0100610 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00c890 tx=0x7f88ec00cd50 comp rx=0 tx=0).stop 2024-04-25T00:26:45.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.693+0000 7f88f7fc1640 1 -- 172.21.15.27:0/4165582498 shutdown_connections 2024-04-25T00:26:45.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.693+0000 7f88f7fc1640 1 --2- 172.21.15.27:0/4165582498 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f88dc0a2d80 0x7f88dc0a5240 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.693+0000 7f88f7fc1640 1 --2- 172.21.15.27:0/4165582498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88f010bd00 0x7f88f0101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.693+0000 7f88f7fc1640 1 --2- 172.21.15.27:0/4165582498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f01032e0 0x7f88f0100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.693+0000 7f88f7fc1640 1 --2- 172.21.15.27:0/4165582498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f01029c0 0x7f88f0100610 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.693+0000 7f88f7fc1640 1 -- 172.21.15.27:0/4165582498 >> 172.21.15.27:0/4165582498 conn(0x7f88f0077740 msgr2=0x7f88f00fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:45.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.694+0000 7f88f7fc1640 1 -- 172.21.15.27:0/4165582498 shutdown_connections 2024-04-25T00:26:45.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.694+0000 7f88f7fc1640 1 -- 172.21.15.27:0/4165582498 wait complete. 2024-04-25T00:26:45.706 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T00:26:45.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.807+0000 7fc3ccea5640 1 -- 172.21.15.27:0/1207235790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c81032e0 msgr2=0x7fc3c810b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.807+0000 7fc3ccea5640 1 --2- 172.21.15.27:0/1207235790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c81032e0 0x7fc3c810b7c0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc009900 tx=0x7fc3bc02a510 comp rx=0 tx=0).stop 2024-04-25T00:26:45.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.808+0000 7fc3ccea5640 1 -- 172.21.15.27:0/1207235790 shutdown_connections 2024-04-25T00:26:45.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.808+0000 7fc3ccea5640 1 --2- 172.21.15.27:0/1207235790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3c810bd00 0x7fc3c810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.808+0000 7fc3ccea5640 1 --2- 172.21.15.27:0/1207235790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c81032e0 0x7fc3c810b7c0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.808+0000 7fc3ccea5640 1 --2- 172.21.15.27:0/1207235790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc3c81029c0 0x7fc3c8102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.808+0000 7fc3ccea5640 1 -- 172.21.15.27:0/1207235790 >> 172.21.15.27:0/1207235790 conn(0x7fc3c8077740 msgr2=0x7fc3c8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:45.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.808+0000 7fc3ccea5640 1 -- 172.21.15.27:0/1207235790 shutdown_connections 2024-04-25T00:26:45.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.808+0000 7fc3ccea5640 1 -- 172.21.15.27:0/1207235790 wait complete. 2024-04-25T00:26:45.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.808+0000 7fc3ccea5640 1 Processor -- start 2024-04-25T00:26:45.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.808+0000 7fc3ccea5640 1 -- start start 2024-04-25T00:26:45.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3ccea5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c81029c0 0x7fc3c8198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3ccea5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3c81032e0 0x7fc3c8198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3ccea5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc3c810bd00 0x7fc3c819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3ccea5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c8111b00 con 0x7fc3c81029c0 2024-04-25T00:26:45.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3ccea5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c8111980 con 0x7fc3c810bd00 2024-04-25T00:26:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3ccea5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c8111c80 con 0x7fc3c81032e0 2024-04-25T00:26:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3c659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c81029c0 0x7fc3c8198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3c659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c81029c0 0x7fc3c8198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56760/0 (socket says 172.21.15.27:56760) 2024-04-25T00:26:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3c659c640 1 -- 172.21.15.27:0/1051946932 learned_addr learned my addr 172.21.15.27:0/1051946932 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3c6d9d640 1 --2- 172.21.15.27:0/1051946932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc3c810bd00 0x7fc3c819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3c5d9b640 1 --2- 172.21.15.27:0/1051946932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3c81032e0 0x7fc3c8198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3c659c640 1 -- 172.21.15.27:0/1051946932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3c81032e0 msgr2=0x7fc3c8198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3c659c640 1 --2- 172.21.15.27:0/1051946932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3c81032e0 0x7fc3c8198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3c659c640 1 -- 172.21.15.27:0/1051946932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc3c810bd00 msgr2=0x7fc3c819c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3c659c640 1 --2- 172.21.15.27:0/1051946932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc3c810bd00 0x7fc3c819c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3c659c640 1 -- 172.21.15.27:0/1051946932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3c819d030 con 0x7fc3c81029c0 2024-04-25T00:26:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3c6d9d640 1 --2- 172.21.15.27:0/1051946932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc3c810bd00 0x7fc3c819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:26:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3c5d9b640 1 --2- 172.21.15.27:0/1051946932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3c81032e0 0x7fc3c8198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:26:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3c659c640 1 --2- 172.21.15.27:0/1051946932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c81029c0 0x7fc3c8198040 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fc3c000e9f0 tx=0x7fc3c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:45.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3b77fe640 1 -- 172.21.15.27:0/1051946932 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c000cd80 con 0x7fc3c81029c0 2024-04-25T00:26:45.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3ccea5640 1 -- 172.21.15.27:0/1051946932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3c819d320 con 0x7fc3c81029c0 2024-04-25T00:26:45.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3b77fe640 1 -- 172.21.15.27:0/1051946932 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3c0010910 con 0x7fc3c81029c0 2024-04-25T00:26:45.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.809+0000 7fc3b77fe640 1 -- 172.21.15.27:0/1051946932 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0018990 con 0x7fc3c81029c0 2024-04-25T00:26:45.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.810+0000 7fc3ccea5640 1 -- 172.21.15.27:0/1051946932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3c81a4bb0 con 0x7fc3c81029c0 2024-04-25T00:26:45.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.810+0000 7fc3ccea5640 1 -- 172.21.15.27:0/1051946932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3c8068560 con 0x7fc3c81029c0 2024-04-25T00:26:45.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.812+0000 7fc3b77fe640 1 -- 172.21.15.27:0/1051946932 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc3c0010ac0 con 0x7fc3c81029c0 2024-04-25T00:26:45.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.812+0000 7fc3b77fe640 1 --2- 172.21.15.27:0/1051946932 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc3b00a2d00 0x7fc3b00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.812+0000 7fc3c5d9b640 1 --2- 172.21.15.27:0/1051946932 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc3b00a2d00 0x7fc3b00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.812+0000 7fc3b77fe640 1 -- 172.21.15.27:0/1051946932 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9496+0+0 (secure 0 0 0) 0x7fc3c00cfc70 con 0x7fc3c81029c0 2024-04-25T00:26:45.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.813+0000 7fc3c5d9b640 1 --2- 172.21.15.27:0/1051946932 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc3b00a2d00 0x7fc3b00a51c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc000c00 tx=0x7fc3bc032c70 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:45.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:45.817+0000 7fc3b77fe640 1 -- 172.21.15.27:0/1051946932 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc3c009c4c0 con 0x7fc3c81029c0 2024-04-25T00:26:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.016+0000 7fc3ccea5640 1 -- 172.21.15.27:0/1051946932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc3c8199390 con 0x7fc3c81029c0 2024-04-25T00:26:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.016+0000 7fc3b77fe640 1 -- 172.21.15.27:0/1051946932 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v96) ==== 81+0+944 (secure 0 0 0) 0x7fc3c009bdc0 con 0x7fc3c81029c0 2024-04-25T00:26:46.017 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1885111026 2024-04-26T00:15:45.034899+0000 2024-04-25T00:26:46.017 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2528869729 2024-04-26T00:15:45.034899+0000 2024-04-25T00:26:46.018 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/4085720546 2024-04-26T00:15:45.034899+0000 2024-04-25T00:26:46.018 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3721121684 2024-04-26T00:06:09.484973+0000 2024-04-25T00:26:46.018 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/1186428290 2024-04-26T00:06:09.484973+0000 2024-04-25T00:26:46.018 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2012441698 2024-04-26T00:06:09.484973+0000 2024-04-25T00:26:46.018 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/236412562 2024-04-26T00:15:45.034899+0000 2024-04-25T00:26:46.018 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/3082151239 2024-04-26T00:05:42.617192+0000 2024-04-25T00:26:46.018 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/2161554778 2024-04-26T00:15:45.034899+0000 2024-04-25T00:26:46.018 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/3082151239 2024-04-26T00:05:42.617192+0000 2024-04-25T00:26:46.018 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/1186428290 2024-04-26T00:06:09.484973+0000 2024-04-25T00:26:46.018 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2172615181 2024-04-26T00:05:42.617192+0000 2024-04-25T00:26:46.018 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/316040314 2024-04-26T00:05:42.617192+0000 2024-04-25T00:26:46.018 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/2161554778 2024-04-26T00:15:45.034899+0000 2024-04-25T00:26:46.018 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1671770035 2024-04-26T00:05:42.617192+0000 2024-04-25T00:26:46.018 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1035759944 2024-04-26T00:06:09.484973+0000 2024-04-25T00:26:46.019 INFO:teuthology.orchestra.run.smithi027.stderr:listed 16 entries 2024-04-25T00:26:46.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.018+0000 7fc3ccea5640 1 -- 172.21.15.27:0/1051946932 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc3b00a2d00 msgr2=0x7fc3b00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:46.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.018+0000 7fc3ccea5640 1 --2- 172.21.15.27:0/1051946932 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc3b00a2d00 0x7fc3b00a51c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc000c00 tx=0x7fc3bc032c70 comp rx=0 tx=0).stop 2024-04-25T00:26:46.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.018+0000 7fc3ccea5640 1 -- 172.21.15.27:0/1051946932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c81029c0 msgr2=0x7fc3c8198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:46.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.018+0000 7fc3ccea5640 1 --2- 172.21.15.27:0/1051946932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c81029c0 0x7fc3c8198040 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fc3c000e9f0 tx=0x7fc3c000eeb0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.018+0000 7fc3ccea5640 1 -- 172.21.15.27:0/1051946932 shutdown_connections 2024-04-25T00:26:46.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.018+0000 7fc3ccea5640 1 --2- 172.21.15.27:0/1051946932 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fc3b00a2d00 0x7fc3b00a51c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.018+0000 7fc3ccea5640 1 --2- 172.21.15.27:0/1051946932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc3c810bd00 0x7fc3c819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.018+0000 7fc3ccea5640 1 --2- 172.21.15.27:0/1051946932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3c81032e0 0x7fc3c8198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.018+0000 7fc3ccea5640 1 --2- 172.21.15.27:0/1051946932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c81029c0 0x7fc3c8198040 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.018+0000 7fc3ccea5640 1 -- 172.21.15.27:0/1051946932 >> 172.21.15.27:0/1051946932 conn(0x7fc3c8077740 msgr2=0x7fc3c80fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:46.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.018+0000 7fc3ccea5640 1 -- 172.21.15.27:0/1051946932 shutdown_connections 2024-04-25T00:26:46.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.018+0000 7fc3ccea5640 1 -- 172.21.15.27:0/1051946932 wait complete. 2024-04-25T00:26:46.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:45 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3691981290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:26:46.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:45 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3542195441' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:26:46.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:45 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4165582498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:26:46.030 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T00:26:46.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:26:46.030 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T00:26:46.079 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T00:26:46.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:26:46.079 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T00:26:46.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:26:46.111 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-04-25T00:26:46.167 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T00:26:46.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:26:46.168 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T00:26:46.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:26:46.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-04-25T00:26:46.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:45 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3691981290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:26:46.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:45 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3542195441' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:26:46.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:45 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4165582498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:26:46.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:45 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3691981290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T00:26:46.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:45 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3542195441' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T00:26:46.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:45 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4165582498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T00:26:46.333 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-25T00:26:46.343 INFO:tasks.cephadm:Teardown begin 2024-04-25T00:26:46.343 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T00:26:46.343 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:46.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.445+0000 7fa39e0bd640 1 -- 172.21.15.27:0/1931355802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa398100620 msgr2=0x7fa398100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:46.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.445+0000 7fa39e0bd640 1 --2- 172.21.15.27:0/1931355802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa398100620 0x7fa398100aa0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fa38c009930 tx=0x7fa38c02a420 comp rx=0 tx=0).stop 2024-04-25T00:26:46.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.445+0000 7fa39e0bd640 1 -- 172.21.15.27:0/1931355802 shutdown_connections 2024-04-25T00:26:46.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.445+0000 7fa39e0bd640 1 --2- 172.21.15.27:0/1931355802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa398100fe0 0x7fa39810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.445+0000 7fa39e0bd640 1 --2- 172.21.15.27:0/1931355802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa398100620 0x7fa398100aa0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.445+0000 7fa39e0bd640 1 --2- 172.21.15.27:0/1931355802 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa398106630 0x7fa398106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.445+0000 7fa39e0bd640 1 -- 172.21.15.27:0/1931355802 >> 172.21.15.27:0/1931355802 conn(0x7fa398077740 msgr2=0x7fa398077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:46.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.445+0000 7fa39e0bd640 1 -- 172.21.15.27:0/1931355802 shutdown_connections 2024-04-25T00:26:46.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.445+0000 7fa39e0bd640 1 -- 172.21.15.27:0/1931355802 wait complete. 2024-04-25T00:26:46.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.446+0000 7fa39e0bd640 1 Processor -- start 2024-04-25T00:26:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.446+0000 7fa39e0bd640 1 -- start start 2024-04-25T00:26:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.446+0000 7fa39e0bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa398100620 0x7fa398102ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.446+0000 7fa39e0bd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa398100fe0 0x7fa3981030e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.446+0000 7fa39e0bd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa398106630 0x7fa398103620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.446+0000 7fa39e0bd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa398110a40 con 0x7fa398100fe0 2024-04-25T00:26:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.446+0000 7fa39e0bd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa3981108c0 con 0x7fa398106630 2024-04-25T00:26:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.446+0000 7fa39e0bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa398110bc0 con 0x7fa398100620 2024-04-25T00:26:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.446+0000 7fa396ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa398100fe0 0x7fa3981030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.447+0000 7fa396ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa398100fe0 0x7fa3981030e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56782/0 (socket says 172.21.15.27:56782) 2024-04-25T00:26:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.447+0000 7fa396ffd640 1 -- 172.21.15.27:0/468209686 learned_addr learned my addr 172.21.15.27:0/468209686 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.447+0000 7fa3977fe640 1 --2- 172.21.15.27:0/468209686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa398100620 0x7fa398102ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.447+0000 7fa397fff640 1 --2- 172.21.15.27:0/468209686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa398106630 0x7fa398103620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.447+0000 7fa396ffd640 1 -- 172.21.15.27:0/468209686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa398100620 msgr2=0x7fa398102ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.447+0000 7fa396ffd640 1 --2- 172.21.15.27:0/468209686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa398100620 0x7fa398102ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.447+0000 7fa396ffd640 1 -- 172.21.15.27:0/468209686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa398106630 msgr2=0x7fa398103620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.447+0000 7fa396ffd640 1 --2- 172.21.15.27:0/468209686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa398106630 0x7fa398103620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.447+0000 7fa396ffd640 1 -- 172.21.15.27:0/468209686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3981a64f0 con 0x7fa398100fe0 2024-04-25T00:26:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.447+0000 7fa397fff640 1 --2- 172.21.15.27:0/468209686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa398106630 0x7fa398103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:26:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.447+0000 7fa396ffd640 1 --2- 172.21.15.27:0/468209686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa398100fe0 0x7fa3981030e0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa38c009a50 tx=0x7fa38c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.447+0000 7fa394ff9640 1 -- 172.21.15.27:0/468209686 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa38c03f030 con 0x7fa398100fe0 2024-04-25T00:26:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.447+0000 7fa39e0bd640 1 -- 172.21.15.27:0/468209686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3981a6780 con 0x7fa398100fe0 2024-04-25T00:26:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.447+0000 7fa394ff9640 1 -- 172.21.15.27:0/468209686 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa38c003ad0 con 0x7fa398100fe0 2024-04-25T00:26:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.447+0000 7fa394ff9640 1 -- 172.21.15.27:0/468209686 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa38c035070 con 0x7fa398100fe0 2024-04-25T00:26:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.447+0000 7fa39e0bd640 1 -- 172.21.15.27:0/468209686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3981a6cc0 con 0x7fa398100fe0 2024-04-25T00:26:46.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.448+0000 7fa39e0bd640 1 -- 172.21.15.27:0/468209686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa398068560 con 0x7fa398100fe0 2024-04-25T00:26:46.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.449+0000 7fa394ff9640 1 -- 172.21.15.27:0/468209686 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa38c042030 con 0x7fa398100fe0 2024-04-25T00:26:46.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.450+0000 7fa394ff9640 1 --2- 172.21.15.27:0/468209686 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa3800a2d00 0x7fa3800a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:46.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.450+0000 7fa3977fe640 1 --2- 172.21.15.27:0/468209686 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa3800a2d00 0x7fa3800a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:46.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.450+0000 7fa394ff9640 1 -- 172.21.15.27:0/468209686 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9496+0+0 (secure 0 0 0) 0x7fa38c0f0060 con 0x7fa398100fe0 2024-04-25T00:26:46.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.451+0000 7fa3977fe640 1 --2- 172.21.15.27:0/468209686 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa3800a2d00 0x7fa3800a51c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa38800abe0 tx=0x7fa38800a2d0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:46.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.455+0000 7fa394ff9640 1 -- 172.21.15.27:0/468209686 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa38c0bc850 con 0x7fa398100fe0 2024-04-25T00:26:46.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.678+0000 7fa39e0bd640 1 -- 172.21.15.27:0/468209686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa398109bc0 con 0x7fa398100fe0 2024-04-25T00:26:46.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.678+0000 7fa394ff9640 1 -- 172.21.15.27:0/468209686 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+11952 (secure 0 0 0) 0x7fa38c0bc150 con 0x7fa398100fe0 2024-04-25T00:26:46.680 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:26:46.680 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":101,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:25:34.747257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":147,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":148,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":146,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24514,24430,24521,24428,24496]},"id":1}]} 2024-04-25T00:26:46.680 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-04-25T00:26:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.680+0000 7fa39e0bd640 1 -- 172.21.15.27:0/468209686 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa3800a2d00 msgr2=0x7fa3800a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.680+0000 7fa39e0bd640 1 --2- 172.21.15.27:0/468209686 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa3800a2d00 0x7fa3800a51c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa38800abe0 tx=0x7fa38800a2d0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.680+0000 7fa39e0bd640 1 -- 172.21.15.27:0/468209686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa398100fe0 msgr2=0x7fa3981030e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.680+0000 7fa39e0bd640 1 --2- 172.21.15.27:0/468209686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa398100fe0 0x7fa3981030e0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa38c009a50 tx=0x7fa38c004270 comp rx=0 tx=0).stop 2024-04-25T00:26:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.680+0000 7fa39e0bd640 1 -- 172.21.15.27:0/468209686 shutdown_connections 2024-04-25T00:26:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.680+0000 7fa39e0bd640 1 --2- 172.21.15.27:0/468209686 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa3800a2d00 0x7fa3800a51c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.680+0000 7fa39e0bd640 1 --2- 172.21.15.27:0/468209686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa398106630 0x7fa398103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.680+0000 7fa39e0bd640 1 --2- 172.21.15.27:0/468209686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa398100fe0 0x7fa3981030e0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.680+0000 7fa39e0bd640 1 --2- 172.21.15.27:0/468209686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa398100620 0x7fa398102ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.680+0000 7fa39e0bd640 1 -- 172.21.15.27:0/468209686 >> 172.21.15.27:0/468209686 conn(0x7fa398077740 msgr2=0x7fa398068b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.681+0000 7fa39e0bd640 1 -- 172.21.15.27:0/468209686 shutdown_connections 2024-04-25T00:26:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.681+0000 7fa39e0bd640 1 -- 172.21.15.27:0/468209686 wait complete. 2024-04-25T00:26:46.693 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.794+0000 7f9032553640 1 -- 172.21.15.27:0/2507795793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c103340 msgr2=0x7f902c10b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.794+0000 7f9032553640 1 --2- 172.21.15.27:0/2507795793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c103340 0x7f902c10b820 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f901c009900 tx=0x7f901c02a480 comp rx=0 tx=0).stop 2024-04-25T00:26:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.794+0000 7f9032553640 1 -- 172.21.15.27:0/2507795793 shutdown_connections 2024-04-25T00:26:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.794+0000 7f9032553640 1 --2- 172.21.15.27:0/2507795793 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f902c10bd60 0x7f902c10e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.794+0000 7f9032553640 1 --2- 172.21.15.27:0/2507795793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c103340 0x7f902c10b820 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.794+0000 7f9032553640 1 --2- 172.21.15.27:0/2507795793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f902c102a20 0x7f902c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.794+0000 7f9032553640 1 -- 172.21.15.27:0/2507795793 >> 172.21.15.27:0/2507795793 conn(0x7f902c0777a0 msgr2=0x7f902c077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.794+0000 7f9032553640 1 -- 172.21.15.27:0/2507795793 shutdown_connections 2024-04-25T00:26:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.794+0000 7f9032553640 1 -- 172.21.15.27:0/2507795793 wait complete. 2024-04-25T00:26:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.795+0000 7f9032553640 1 Processor -- start 2024-04-25T00:26:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.795+0000 7f9032553640 1 -- start start 2024-04-25T00:26:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.795+0000 7f9032553640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f902c102a20 0x7f902c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.795+0000 7f9032553640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f902c103340 0x7f902c1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.795+0000 7f9032553640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c10bd60 0x7f902c19c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.795+0000 7f9032553640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f902c111b30 con 0x7f902c10bd60 2024-04-25T00:26:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.795+0000 7f9032553640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f902c1119b0 con 0x7f902c102a20 2024-04-25T00:26:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.795+0000 7f9032553640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f902c111cb0 con 0x7f902c103340 2024-04-25T00:26:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.795+0000 7f9030af0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c10bd60 0x7f902c19c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.795+0000 7f902b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f902c103340 0x7f902c1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.795+0000 7f902b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f902c103340 0x7f902c1985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:53984/0 (socket says 172.21.15.27:53984) 2024-04-25T00:26:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.795+0000 7f902b7fe640 1 -- 172.21.15.27:0/3940705561 learned_addr learned my addr 172.21.15.27:0/3940705561 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.796+0000 7f902bfff640 1 --2- 172.21.15.27:0/3940705561 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f902c102a20 0x7f902c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.796+0000 7f9030af0640 1 -- 172.21.15.27:0/3940705561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f902c103340 msgr2=0x7f902c1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.796+0000 7f9030af0640 1 --2- 172.21.15.27:0/3940705561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f902c103340 0x7f902c1985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.796+0000 7f9030af0640 1 -- 172.21.15.27:0/3940705561 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f902c102a20 msgr2=0x7f902c198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.796+0000 7f9030af0640 1 --2- 172.21.15.27:0/3940705561 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f902c102a20 0x7f902c198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.796+0000 7f9030af0640 1 -- 172.21.15.27:0/3940705561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f902c19d030 con 0x7f902c10bd60 2024-04-25T00:26:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.796+0000 7f902b7fe640 1 --2- 172.21.15.27:0/3940705561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f902c103340 0x7f902c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:26:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.796+0000 7f9030af0640 1 --2- 172.21.15.27:0/3940705561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c10bd60 0x7f902c19c900 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f902400e960 tx=0x7f902400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.796+0000 7f90297fa640 1 -- 172.21.15.27:0/3940705561 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9024004d20 con 0x7f902c10bd60 2024-04-25T00:26:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.796+0000 7f9032553640 1 -- 172.21.15.27:0/3940705561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f902c19d320 con 0x7f902c10bd60 2024-04-25T00:26:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.796+0000 7f9032553640 1 -- 172.21.15.27:0/3940705561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f902c1a4bb0 con 0x7f902c10bd60 2024-04-25T00:26:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.796+0000 7f90297fa640 1 -- 172.21.15.27:0/3940705561 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9024005020 con 0x7f902c10bd60 2024-04-25T00:26:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.796+0000 7f90297fa640 1 -- 172.21.15.27:0/3940705561 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902401f060 con 0x7f902c10bd60 2024-04-25T00:26:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.797+0000 7f9032553640 1 -- 172.21.15.27:0/3940705561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f902c0689b0 con 0x7f902c10bd60 2024-04-25T00:26:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.798+0000 7f90297fa640 1 -- 172.21.15.27:0/3940705561 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9024005560 con 0x7f902c10bd60 2024-04-25T00:26:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.799+0000 7f90297fa640 1 --2- 172.21.15.27:0/3940705561 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f90140a2d00 0x7f90140a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.799+0000 7f902bfff640 1 --2- 172.21.15.27:0/3940705561 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f90140a2d00 0x7f90140a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.799+0000 7f90297fa640 1 -- 172.21.15.27:0/3940705561 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9496+0+0 (secure 0 0 0) 0x7f90240cf820 con 0x7f902c10bd60 2024-04-25T00:26:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.800+0000 7f902bfff640 1 --2- 172.21.15.27:0/3940705561 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f90140a2d00 0x7f90140a51c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f902000bbe0 tx=0x7f902000b390 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:46.804+0000 7f90297fa640 1 -- 172.21.15.27:0/3940705561 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9024016030 con 0x7f902c10bd60 2024-04-25T00:26:46.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:46 smithi027 ceph-mon[29208]: pgmap v335: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:26:46.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:46 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1051946932' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T00:26:46.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:46 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/468209686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:26:47.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.026+0000 7f9032553640 1 -- 172.21.15.27:0/3940705561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f902c106ee0 con 0x7f902c10bd60 2024-04-25T00:26:47.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.026+0000 7f90297fa640 1 -- 172.21.15.27:0/3940705561 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+11952 (secure 0 0 0) 0x7f902409ba20 con 0x7f902c10bd60 2024-04-25T00:26:47.028 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:26:47.028 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2280519367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2280519367},{"type":"v1","addr":"172.21.15.27:6835","nonce":2280519367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/590015015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":590015015},{"type":"v1","addr":"172.21.15.27:6837","nonce":590015015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":101,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T00:13:18.872210+0000","modified":"2024-04-25T00:25:34.747257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24521,"mds_1":24514,"mds_2":24496,"mds_3":24430,"mds_4":24428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":147,"addr":"172.21.15.27:6839/3286639641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":3286639641},{"type":"v1","addr":"172.21.15.27:6839","nonce":3286639641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14664":{"gid":14664,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6841/2271898034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":2271898034},{"type":"v1","addr":"172.21.15.27:6841","nonce":2271898034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.139:6833/3964352148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3964352148},{"type":"v1","addr":"172.21.15.139:6833","nonce":3964352148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":18,"addr":"172.21.15.153:6833/2700717364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2700717364},{"type":"v1","addr":"172.21.15.153:6833","nonce":2700717364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/3731284556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":3731284556},{"type":"v1","addr":"172.21.15.139:6835","nonce":3731284556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/584718807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":584718807},{"type":"v1","addr":"172.21.15.153:6835","nonce":584718807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":148,"addr":"172.21.15.139:6837/2477304919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":2477304919},{"type":"v1","addr":"172.21.15.139:6837","nonce":2477304919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/4066071541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":4066071541},{"type":"v1","addr":"172.21.15.153:6837","nonce":4066071541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/1144169732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":1144169732},{"type":"v1","addr":"172.21.15.153:6839","nonce":1144169732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":146,"addr":"172.21.15.139:6839/2728523692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2728523692},{"type":"v1","addr":"172.21.15.139:6839","nonce":2728523692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24514,24430,24521,24428,24496]},"id":1}]} 2024-04-25T00:26:47.029 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-04-25T00:26:47.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.028+0000 7f9032553640 1 -- 172.21.15.27:0/3940705561 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f90140a2d00 msgr2=0x7f90140a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:47.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.028+0000 7f9032553640 1 --2- 172.21.15.27:0/3940705561 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f90140a2d00 0x7f90140a51c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f902000bbe0 tx=0x7f902000b390 comp rx=0 tx=0).stop 2024-04-25T00:26:47.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.028+0000 7f9032553640 1 -- 172.21.15.27:0/3940705561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c10bd60 msgr2=0x7f902c19c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:47.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.028+0000 7f9032553640 1 --2- 172.21.15.27:0/3940705561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c10bd60 0x7f902c19c900 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f902400e960 tx=0x7f902400ee20 comp rx=0 tx=0).stop 2024-04-25T00:26:47.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.028+0000 7f9032553640 1 -- 172.21.15.27:0/3940705561 shutdown_connections 2024-04-25T00:26:47.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.028+0000 7f9032553640 1 --2- 172.21.15.27:0/3940705561 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f90140a2d00 0x7f90140a51c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.028+0000 7f9032553640 1 --2- 172.21.15.27:0/3940705561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c10bd60 0x7f902c19c900 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.028+0000 7f9032553640 1 --2- 172.21.15.27:0/3940705561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f902c103340 0x7f902c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.028+0000 7f9032553640 1 --2- 172.21.15.27:0/3940705561 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f902c102a20 0x7f902c198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.028+0000 7f9032553640 1 -- 172.21.15.27:0/3940705561 >> 172.21.15.27:0/3940705561 conn(0x7f902c0777a0 msgr2=0x7f902c0fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:47.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.028+0000 7f9032553640 1 -- 172.21.15.27:0/3940705561 shutdown_connections 2024-04-25T00:26:47.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.028+0000 7f9032553640 1 -- 172.21.15.27:0/3940705561 wait complete. 2024-04-25T00:26:47.040 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:26:47.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.141+0000 7f57d67a5640 1 -- 172.21.15.27:0/1271935157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d010bd60 msgr2=0x7f57d010e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:47.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.141+0000 7f57d67a5640 1 --2- 172.21.15.27:0/1271935157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d010bd60 0x7f57d010e1d0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f57c8009ca0 tx=0x7f57c802a950 comp rx=0 tx=0).stop 2024-04-25T00:26:47.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.141+0000 7f57d67a5640 1 -- 172.21.15.27:0/1271935157 shutdown_connections 2024-04-25T00:26:47.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.141+0000 7f57d67a5640 1 --2- 172.21.15.27:0/1271935157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d010bd60 0x7f57d010e1d0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.141+0000 7f57d67a5640 1 --2- 172.21.15.27:0/1271935157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d0103340 0x7f57d010b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.141+0000 7f57d67a5640 1 --2- 172.21.15.27:0/1271935157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57d0102a20 0x7f57d0102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.141+0000 7f57d67a5640 1 -- 172.21.15.27:0/1271935157 >> 172.21.15.27:0/1271935157 conn(0x7f57d00777a0 msgr2=0x7f57d0077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:47.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.141+0000 7f57d67a5640 1 -- 172.21.15.27:0/1271935157 shutdown_connections 2024-04-25T00:26:47.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.141+0000 7f57d67a5640 1 -- 172.21.15.27:0/1271935157 wait complete. 2024-04-25T00:26:47.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.141+0000 7f57d67a5640 1 Processor -- start 2024-04-25T00:26:47.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.142+0000 7f57d67a5640 1 -- start start 2024-04-25T00:26:47.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.142+0000 7f57d67a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d0102a20 0x7f57d0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:47.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.142+0000 7f57d67a5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57d0103340 0x7f57d0198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:47.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.142+0000 7f57d67a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d010bd60 0x7f57d019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:47.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.142+0000 7f57d67a5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f57d0111b30 con 0x7f57d010bd60 2024-04-25T00:26:47.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.142+0000 7f57d67a5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f57d01119b0 con 0x7f57d0103340 2024-04-25T00:26:47.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.142+0000 7f57d67a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f57d0111cb0 con 0x7f57d0102a20 2024-04-25T00:26:47.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.142+0000 7f57d4d42640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d010bd60 0x7f57d019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:47.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.142+0000 7f57d4d42640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d010bd60 0x7f57d019c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33990/0 (socket says 172.21.15.27:33990) 2024-04-25T00:26:47.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.142+0000 7f57d4d42640 1 -- 172.21.15.27:0/2873652157 learned_addr learned my addr 172.21.15.27:0/2873652157 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:47.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.142+0000 7f57cffff640 1 --2- 172.21.15.27:0/2873652157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d0102a20 0x7f57d0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:47.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.142+0000 7f57cf7fe640 1 --2- 172.21.15.27:0/2873652157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57d0103340 0x7f57d0198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:47.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.142+0000 7f57d4d42640 1 -- 172.21.15.27:0/2873652157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d0102a20 msgr2=0x7f57d0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:47.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.142+0000 7f57d4d42640 1 --2- 172.21.15.27:0/2873652157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d0102a20 0x7f57d0197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.142+0000 7f57d4d42640 1 -- 172.21.15.27:0/2873652157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57d0103340 msgr2=0x7f57d0198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:47.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.142+0000 7f57d4d42640 1 --2- 172.21.15.27:0/2873652157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57d0103340 0x7f57d0198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.142+0000 7f57d4d42640 1 -- 172.21.15.27:0/2873652157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57d019cf90 con 0x7f57d010bd60 2024-04-25T00:26:47.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.143+0000 7f57d4d42640 1 --2- 172.21.15.27:0/2873652157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d010bd60 0x7f57d019c860 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f57c8004af0 tx=0x7f57c802f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:47.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.143+0000 7f57cd7fa640 1 -- 172.21.15.27:0/2873652157 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c8034040 con 0x7f57d010bd60 2024-04-25T00:26:47.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.143+0000 7f57d67a5640 1 -- 172.21.15.27:0/2873652157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57d019d220 con 0x7f57d010bd60 2024-04-25T00:26:47.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.143+0000 7f57cd7fa640 1 -- 172.21.15.27:0/2873652157 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57c8030920 con 0x7f57d010bd60 2024-04-25T00:26:47.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.143+0000 7f57cd7fa640 1 -- 172.21.15.27:0/2873652157 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c80047b0 con 0x7f57d010bd60 2024-04-25T00:26:47.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.143+0000 7f57d67a5640 1 -- 172.21.15.27:0/2873652157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57d01a4b10 con 0x7f57d010bd60 2024-04-25T00:26:47.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.144+0000 7f57d67a5640 1 -- 172.21.15.27:0/2873652157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57d00689b0 con 0x7f57d010bd60 2024-04-25T00:26:47.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.145+0000 7f57cd7fa640 1 -- 172.21.15.27:0/2873652157 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f57c8037020 con 0x7f57d010bd60 2024-04-25T00:26:47.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.145+0000 7f57cd7fa640 1 --2- 172.21.15.27:0/2873652157 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f57b80a2d00 0x7f57b80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:47.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.146+0000 7f57cffff640 1 --2- 172.21.15.27:0/2873652157 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f57b80a2d00 0x7f57b80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:47.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.146+0000 7f57cd7fa640 1 -- 172.21.15.27:0/2873652157 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9496+0+0 (secure 0 0 0) 0x7f57c80f0720 con 0x7f57d010bd60 2024-04-25T00:26:47.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.146+0000 7f57cffff640 1 --2- 172.21.15.27:0/2873652157 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f57b80a2d00 0x7f57b80a51c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f57c00058c0 tx=0x7f57c000a2d0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:47.150 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:46 smithi153 ceph-mon[32896]: pgmap v335: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:26:47.150 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:46 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1051946932' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T00:26:47.150 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:46 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/468209686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:26:47.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.149+0000 7f57cd7fa640 1 -- 172.21.15.27:0/2873652157 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f57c8080740 con 0x7f57d010bd60 2024-04-25T00:26:47.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:46 smithi139 ceph-mon[32833]: pgmap v335: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T00:26:47.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:46 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1051946932' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T00:26:47.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:46 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/468209686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:26:47.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.336+0000 7f57d67a5640 1 -- 172.21.15.27:0/2873652157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f57d01a4d70 con 0x7f57d010bd60 2024-04-25T00:26:47.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.337+0000 7f57cd7fa640 1 -- 172.21.15.27:0/2873652157 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+25290 (secure 0 0 0) 0x7f57c80bd080 con 0x7f57d010bd60 2024-04-25T00:26:47.339 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T00:26:47.339 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":96,"fsid":"518857f8-0297-11ef-bc93-c7b262605968","created":"2024-04-25T00:05:10.376114+0000","modified":"2024-04-25T00:26:20.745957+0000","last_up_change":"2024-04-25T00:12:25.415573+0000","last_in_change":"2024-04-25T00:12:04.226987+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:08:34.006973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T00:13:17.337307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T00:13:18.253713+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":9,"snap_epoch":95,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T00:13:21.995987+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":9,"snap_epoch":95,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b523ee1d-7722-4c0d-ba18-82aa50b46b3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6803","nonce":238818528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6805","nonce":238818528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6809","nonce":238818528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":238818528},{"type":"v1","addr":"172.21.15.27:6807","nonce":238818528}]},"public_addr":"172.21.15.27:6803/238818528","cluster_addr":"172.21.15.27:6805/238818528","heartbeat_back_addr":"172.21.15.27:6809/238818528","heartbeat_front_addr":"172.21.15.27:6807/238818528","state":["exists","up"]},{"osd":1,"uuid":"1317fe63-aa4c-4687-8cdb-47f9531f654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6801","nonce":1346503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6803","nonce":1346503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6807","nonce":1346503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":1346503461},{"type":"v1","addr":"172.21.15.139:6805","nonce":1346503461}]},"public_addr":"172.21.15.139:6801/1346503461","cluster_addr":"172.21.15.139:6803/1346503461","heartbeat_back_addr":"172.21.15.139:6807/1346503461","heartbeat_front_addr":"172.21.15.139:6805/1346503461","state":["exists","up"]},{"osd":2,"uuid":"fbc82ce9-2c51-4c2d-91d0-84132845c085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6801","nonce":662156492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6803","nonce":662156492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6807","nonce":662156492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":662156492},{"type":"v1","addr":"172.21.15.153:6805","nonce":662156492}]},"public_addr":"172.21.15.153:6801/662156492","cluster_addr":"172.21.15.153:6803/662156492","heartbeat_back_addr":"172.21.15.153:6807/662156492","heartbeat_front_addr":"172.21.15.153:6805/662156492","state":["exists","up"]},{"osd":3,"uuid":"a5e90823-a44c-47e7-bef7-6274de4d59ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6811","nonce":2150456629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6813","nonce":2150456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6817","nonce":2150456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2150456629},{"type":"v1","addr":"172.21.15.27:6815","nonce":2150456629}]},"public_addr":"172.21.15.27:6811/2150456629","cluster_addr":"172.21.15.27:6813/2150456629","heartbeat_back_addr":"172.21.15.27:6817/2150456629","heartbeat_front_addr":"172.21.15.27:6815/2150456629","state":["exists","up"]},{"osd":4,"uuid":"be18fdcd-c36d-4176-8292-90b0aaa85cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6809","nonce":856291222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6811","nonce":856291222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6815","nonce":856291222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":856291222},{"type":"v1","addr":"172.21.15.139:6813","nonce":856291222}]},"public_addr":"172.21.15.139:6809/856291222","cluster_addr":"172.21.15.139:6811/856291222","heartbeat_back_addr":"172.21.15.139:6815/856291222","heartbeat_front_addr":"172.21.15.139:6813/856291222","state":["exists","up"]},{"osd":5,"uuid":"35ff5087-0d52-48f2-8417-a373b85a47e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6809","nonce":3798123772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6811","nonce":3798123772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6815","nonce":3798123772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3798123772},{"type":"v1","addr":"172.21.15.153:6813","nonce":3798123772}]},"public_addr":"172.21.15.153:6809/3798123772","cluster_addr":"172.21.15.153:6811/3798123772","heartbeat_back_addr":"172.21.15.153:6815/3798123772","heartbeat_front_addr":"172.21.15.153:6813/3798123772","state":["exists","up"]},{"osd":6,"uuid":"923eafd7-9819-4ef8-9008-0c5ff2a611c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6819","nonce":769569119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6821","nonce":769569119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6825","nonce":769569119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":769569119},{"type":"v1","addr":"172.21.15.27:6823","nonce":769569119}]},"public_addr":"172.21.15.27:6819/769569119","cluster_addr":"172.21.15.27:6821/769569119","heartbeat_back_addr":"172.21.15.27:6825/769569119","heartbeat_front_addr":"172.21.15.27:6823/769569119","state":["exists","up"]},{"osd":7,"uuid":"a71b99a8-c31e-4ac9-91fc-61735cd196b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6817","nonce":4023269486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6819","nonce":4023269486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6823","nonce":4023269486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4023269486},{"type":"v1","addr":"172.21.15.139:6821","nonce":4023269486}]},"public_addr":"172.21.15.139:6817/4023269486","cluster_addr":"172.21.15.139:6819/4023269486","heartbeat_back_addr":"172.21.15.139:6823/4023269486","heartbeat_front_addr":"172.21.15.139:6821/4023269486","state":["exists","up"]},{"osd":8,"uuid":"b73934bc-9fd5-4da3-8c9b-047a2d116d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6817","nonce":2251501067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6819","nonce":2251501067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6823","nonce":2251501067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2251501067},{"type":"v1","addr":"172.21.15.153:6821","nonce":2251501067}]},"public_addr":"172.21.15.153:6817/2251501067","cluster_addr":"172.21.15.153:6819/2251501067","heartbeat_back_addr":"172.21.15.153:6823/2251501067","heartbeat_front_addr":"172.21.15.153:6821/2251501067","state":["exists","up"]},{"osd":9,"uuid":"375ebd2b-ed51-4e03-b8bb-9725b30d81b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6827","nonce":2816562594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6829","nonce":2816562594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6833","nonce":2816562594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":2816562594},{"type":"v1","addr":"172.21.15.27:6831","nonce":2816562594}]},"public_addr":"172.21.15.27:6827/2816562594","cluster_addr":"172.21.15.27:6829/2816562594","heartbeat_back_addr":"172.21.15.27:6833/2816562594","heartbeat_front_addr":"172.21.15.27:6831/2816562594","state":["exists","up"]},{"osd":10,"uuid":"ea5bca6d-3675-453d-a0f2-cae08531fe5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6825","nonce":3154318186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6827","nonce":3154318186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6831","nonce":3154318186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":3154318186},{"type":"v1","addr":"172.21.15.139:6829","nonce":3154318186}]},"public_addr":"172.21.15.139:6825/3154318186","cluster_addr":"172.21.15.139:6827/3154318186","heartbeat_back_addr":"172.21.15.139:6831/3154318186","heartbeat_front_addr":"172.21.15.139:6829/3154318186","state":["exists","up"]},{"osd":11,"uuid":"5a15ea59-44bf-4e78-b60a-8cc5944ec151","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6825","nonce":2357170710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6827","nonce":2357170710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6831","nonce":2357170710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2357170710},{"type":"v1","addr":"172.21.15.153:6829","nonce":2357170710}]},"public_addr":"172.21.15.153:6825/2357170710","cluster_addr":"172.21.15.153:6827/2357170710","heartbeat_back_addr":"172.21.15.153:6831/2357170710","heartbeat_front_addr":"172.21.15.153:6829/2357170710","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:07:42.775020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:06.379049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:30.821131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:08:56.749944+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:22.158808+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:09:46.911258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:12.768889+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:10:38.309512+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:03.684640+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:30.294207+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:11:56.059107+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:12:22.908371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1885111026":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/2528869729":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/4085720546":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/3721121684":"2024-04-26T00:06:09.484973+0000","172.21.15.27:6800/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2012441698":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/236412562":"2024-04-26T00:15:45.034899+0000","172.21.15.27:6800/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/2161554778":"2024-04-26T00:15:45.034899+0000","172.21.15.27:6801/3082151239":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6801/1186428290":"2024-04-26T00:06:09.484973+0000","172.21.15.27:0/2172615181":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/316040314":"2024-04-26T00:05:42.617192+0000","172.21.15.27:6800/2161554778":"2024-04-26T00:15:45.034899+0000","172.21.15.27:0/1671770035":"2024-04-26T00:05:42.617192+0000","172.21.15.27:0/1035759944":"2024-04-26T00:06:09.484973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":2,"length":1},{"begin":9,"length":1}]},{"pool":4,"snaps":[{"begin":2,"length":1},{"begin":9,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:26:47.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.339+0000 7f57d67a5640 1 -- 172.21.15.27:0/2873652157 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f57b80a2d00 msgr2=0x7f57b80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:47.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.339+0000 7f57d67a5640 1 --2- 172.21.15.27:0/2873652157 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f57b80a2d00 0x7f57b80a51c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f57c00058c0 tx=0x7f57c000a2d0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.339+0000 7f57d67a5640 1 -- 172.21.15.27:0/2873652157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d010bd60 msgr2=0x7f57d019c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:47.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.339+0000 7f57d67a5640 1 --2- 172.21.15.27:0/2873652157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d010bd60 0x7f57d019c860 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f57c8004af0 tx=0x7f57c802f5b0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.340+0000 7f57d67a5640 1 -- 172.21.15.27:0/2873652157 shutdown_connections 2024-04-25T00:26:47.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.340+0000 7f57d67a5640 1 --2- 172.21.15.27:0/2873652157 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f57b80a2d00 0x7f57b80a51c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.340+0000 7f57d67a5640 1 --2- 172.21.15.27:0/2873652157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d010bd60 0x7f57d019c860 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.340+0000 7f57d67a5640 1 --2- 172.21.15.27:0/2873652157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57d0103340 0x7f57d0198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.340+0000 7f57d67a5640 1 --2- 172.21.15.27:0/2873652157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d0102a20 0x7f57d0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.340+0000 7f57d67a5640 1 -- 172.21.15.27:0/2873652157 >> 172.21.15.27:0/2873652157 conn(0x7f57d00777a0 msgr2=0x7f57d00fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:47.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.340+0000 7f57d67a5640 1 -- 172.21.15.27:0/2873652157 shutdown_connections 2024-04-25T00:26:47.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.340+0000 7f57d67a5640 1 -- 172.21.15.27:0/2873652157 wait complete. 2024-04-25T00:26:47.352 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T00:26:47.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.454+0000 7f0e20389640 1 -- 172.21.15.27:0/3525902210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e18101760 msgr2=0x7f0e18101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:47.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.454+0000 7f0e20389640 1 --2- 172.21.15.27:0/3525902210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e18101760 0x7f0e18101be0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009930 tx=0x7f0e0802a420 comp rx=0 tx=0).stop 2024-04-25T00:26:47.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.455+0000 7f0e20389640 1 -- 172.21.15.27:0/3525902210 shutdown_connections 2024-04-25T00:26:47.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.455+0000 7f0e20389640 1 --2- 172.21.15.27:0/3525902210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e18102120 0x7f0e1810e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.455+0000 7f0e20389640 1 --2- 172.21.15.27:0/3525902210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e18101760 0x7f0e18101be0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.455+0000 7f0e20389640 1 --2- 172.21.15.27:0/3525902210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e18107770 0x7f0e18107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.455+0000 7f0e20389640 1 -- 172.21.15.27:0/3525902210 >> 172.21.15.27:0/3525902210 conn(0x7f0e180fd250 msgr2=0x7f0e180ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:47.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.455+0000 7f0e20389640 1 -- 172.21.15.27:0/3525902210 shutdown_connections 2024-04-25T00:26:47.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.455+0000 7f0e20389640 1 -- 172.21.15.27:0/3525902210 wait complete. 2024-04-25T00:26:47.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.455+0000 7f0e20389640 1 Processor -- start 2024-04-25T00:26:47.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.455+0000 7f0e20389640 1 -- start start 2024-04-25T00:26:47.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.455+0000 7f0e20389640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e18101760 0x7f0e181980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:47.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.455+0000 7f0e20389640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e18102120 0x7f0e18198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:47.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.455+0000 7f0e20389640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e18107770 0x7f0e1819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:47.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.455+0000 7f0e20389640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0e18111b80 con 0x7f0e18102120 2024-04-25T00:26:47.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.455+0000 7f0e20389640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0e18111a00 con 0x7f0e18101760 2024-04-25T00:26:47.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.455+0000 7f0e20389640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0e18111d00 con 0x7f0e18107770 2024-04-25T00:26:47.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.456+0000 7f0e1d924640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e18102120 0x7f0e18198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:47.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.456+0000 7f0e1d924640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e18102120 0x7f0e18198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34000/0 (socket says 172.21.15.27:34000) 2024-04-25T00:26:47.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.456+0000 7f0e1d924640 1 -- 172.21.15.27:0/2418050378 learned_addr learned my addr 172.21.15.27:0/2418050378 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:47.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.456+0000 7f0e1e926640 1 --2- 172.21.15.27:0/2418050378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e18107770 0x7f0e1819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:47.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.456+0000 7f0e1e125640 1 --2- 172.21.15.27:0/2418050378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e18101760 0x7f0e181980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:47.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.456+0000 7f0e1d924640 1 -- 172.21.15.27:0/2418050378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e18107770 msgr2=0x7f0e1819c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:47.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.456+0000 7f0e1d924640 1 --2- 172.21.15.27:0/2418050378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e18107770 0x7f0e1819c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.456+0000 7f0e1d924640 1 -- 172.21.15.27:0/2418050378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e18101760 msgr2=0x7f0e181980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:47.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.456+0000 7f0e1d924640 1 --2- 172.21.15.27:0/2418050378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e18101760 0x7f0e181980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.456+0000 7f0e1d924640 1 -- 172.21.15.27:0/2418050378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e1819d080 con 0x7f0e18102120 2024-04-25T00:26:47.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.456+0000 7f0e1d924640 1 --2- 172.21.15.27:0/2418050378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e18102120 0x7f0e18198620 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009230 tx=0x7f0e08004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:47.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.456+0000 7f0e0f7fe640 1 -- 172.21.15.27:0/2418050378 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0803f030 con 0x7f0e18102120 2024-04-25T00:26:47.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.456+0000 7f0e20389640 1 -- 172.21.15.27:0/2418050378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e1819d310 con 0x7f0e18102120 2024-04-25T00:26:47.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.456+0000 7f0e0f7fe640 1 -- 172.21.15.27:0/2418050378 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e08003ad0 con 0x7f0e18102120 2024-04-25T00:26:47.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.456+0000 7f0e0f7fe640 1 -- 172.21.15.27:0/2418050378 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e08035070 con 0x7f0e18102120 2024-04-25T00:26:47.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.456+0000 7f0e20389640 1 -- 172.21.15.27:0/2418050378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e181a4c00 con 0x7f0e18102120 2024-04-25T00:26:47.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.457+0000 7f0e20389640 1 -- 172.21.15.27:0/2418050378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e180689b0 con 0x7f0e18102120 2024-04-25T00:26:47.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.458+0000 7f0e0f7fe640 1 -- 172.21.15.27:0/2418050378 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0e08042030 con 0x7f0e18102120 2024-04-25T00:26:47.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.459+0000 7f0e0f7fe640 1 --2- 172.21.15.27:0/2418050378 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0e040a2d00 0x7f0e040a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:47.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.459+0000 7f0e1e125640 1 --2- 172.21.15.27:0/2418050378 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0e040a2d00 0x7f0e040a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:47.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.459+0000 7f0e0f7fe640 1 -- 172.21.15.27:0/2418050378 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9496+0+0 (secure 0 0 0) 0x7f0e080efdd0 con 0x7f0e18102120 2024-04-25T00:26:47.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.460+0000 7f0e1e125640 1 --2- 172.21.15.27:0/2418050378 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0e040a2d00 0x7f0e040a51c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400bb60 tx=0x7f0e1400b390 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:47.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.464+0000 7f0e0f7fe640 1 -- 172.21.15.27:0/2418050378 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0e080bbfd0 con 0x7f0e18102120 2024-04-25T00:26:47.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.689+0000 7f0e20389640 1 -- 172.21.15.27:0/2418050378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0e18109cf0 con 0x7f0e18102120 2024-04-25T00:26:47.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.691+0000 7f0e0f7fe640 1 -- 172.21.15.27:0/2418050378 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v102) ==== 161+0+0 (secure 0 0 0) 0x7f0e080c02d0 con 0x7f0e18102120 2024-04-25T00:26:47.692 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T00:26:47.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.695+0000 7f0e20389640 1 -- 172.21.15.27:0/2418050378 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0e040a2d00 msgr2=0x7f0e040a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:47.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.695+0000 7f0e20389640 1 --2- 172.21.15.27:0/2418050378 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0e040a2d00 0x7f0e040a51c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400bb60 tx=0x7f0e1400b390 comp rx=0 tx=0).stop 2024-04-25T00:26:47.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.695+0000 7f0e20389640 1 -- 172.21.15.27:0/2418050378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e18102120 msgr2=0x7f0e18198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:47.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.695+0000 7f0e20389640 1 --2- 172.21.15.27:0/2418050378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e18102120 0x7f0e18198620 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009230 tx=0x7f0e08004270 comp rx=0 tx=0).stop 2024-04-25T00:26:47.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.695+0000 7f0e20389640 1 -- 172.21.15.27:0/2418050378 shutdown_connections 2024-04-25T00:26:47.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.695+0000 7f0e20389640 1 --2- 172.21.15.27:0/2418050378 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f0e040a2d00 0x7f0e040a51c0 secure :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400bb60 tx=0x7f0e1400b390 comp rx=0 tx=0).stop 2024-04-25T00:26:47.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.695+0000 7f0e20389640 1 --2- 172.21.15.27:0/2418050378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e18107770 0x7f0e1819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.695+0000 7f0e20389640 1 --2- 172.21.15.27:0/2418050378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e18102120 0x7f0e18198620 secure :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009230 tx=0x7f0e08004270 comp rx=0 tx=0).stop 2024-04-25T00:26:47.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.695+0000 7f0e20389640 1 --2- 172.21.15.27:0/2418050378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e18101760 0x7f0e181980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.695+0000 7f0e20389640 1 -- 172.21.15.27:0/2418050378 >> 172.21.15.27:0/2418050378 conn(0x7f0e180fd250 msgr2=0x7f0e180fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:47.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.696+0000 7f0e20389640 1 -- 172.21.15.27:0/2418050378 shutdown_connections 2024-04-25T00:26:47.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.696+0000 7f0e20389640 1 -- 172.21.15.27:0/2418050378 wait complete. 2024-04-25T00:26:47.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T00:26:47.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.829+0000 7febece7b640 1 -- 172.21.15.27:0/3367578233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febe8069dc0 msgr2=0x7febe806a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:47.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.829+0000 7febece7b640 1 --2- 172.21.15.27:0/3367578233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febe8069dc0 0x7febe806a240 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7febd0009930 tx=0x7febd002a420 comp rx=0 tx=0).stop 2024-04-25T00:26:47.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.829+0000 7febece7b640 1 -- 172.21.15.27:0/3367578233 shutdown_connections 2024-04-25T00:26:47.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.829+0000 7febece7b640 1 --2- 172.21.15.27:0/3367578233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7febe806a780 0x7febe810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.829+0000 7febece7b640 1 --2- 172.21.15.27:0/3367578233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febe8069dc0 0x7febe806a240 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.829+0000 7febece7b640 1 --2- 172.21.15.27:0/3367578233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febe8107730 0x7febe8107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.829+0000 7febece7b640 1 -- 172.21.15.27:0/3367578233 >> 172.21.15.27:0/3367578233 conn(0x7febe80776a0 msgr2=0x7febe8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:47.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.829+0000 7febece7b640 1 -- 172.21.15.27:0/3367578233 shutdown_connections 2024-04-25T00:26:47.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.829+0000 7febece7b640 1 -- 172.21.15.27:0/3367578233 wait complete. 2024-04-25T00:26:47.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febece7b640 1 Processor -- start 2024-04-25T00:26:47.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febece7b640 1 -- start start 2024-04-25T00:26:47.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febece7b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febe8069dc0 0x7febe8198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:47.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febece7b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7febe806a780 0x7febe8198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:47.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febece7b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febe8107730 0x7febe819c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:47.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febece7b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7febe8111b90 con 0x7febe8069dc0 2024-04-25T00:26:47.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febece7b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7febe8111a10 con 0x7febe806a780 2024-04-25T00:26:47.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febece7b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7febe8111d10 con 0x7febe8107730 2024-04-25T00:26:47.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febe659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febe8069dc0 0x7febe8198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:47.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febe659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febe8069dc0 0x7febe8198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34042/0 (socket says 172.21.15.27:34042) 2024-04-25T00:26:47.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febe659c640 1 -- 172.21.15.27:0/4237534477 learned_addr learned my addr 172.21.15.27:0/4237534477 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:47.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febe5d9b640 1 --2- 172.21.15.27:0/4237534477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7febe806a780 0x7febe8198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:47.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febe6d9d640 1 --2- 172.21.15.27:0/4237534477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febe8107730 0x7febe819c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:47.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febe659c640 1 -- 172.21.15.27:0/4237534477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febe8107730 msgr2=0x7febe819c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:47.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febe659c640 1 --2- 172.21.15.27:0/4237534477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febe8107730 0x7febe819c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febe659c640 1 -- 172.21.15.27:0/4237534477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7febe806a780 msgr2=0x7febe8198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:47.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febe659c640 1 --2- 172.21.15.27:0/4237534477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7febe806a780 0x7febe8198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.830+0000 7febe659c640 1 -- 172.21.15.27:0/4237534477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe819cfd0 con 0x7febe8069dc0 2024-04-25T00:26:47.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.831+0000 7febe659c640 1 --2- 172.21.15.27:0/4237534477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febe8069dc0 0x7febe8198030 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7febd800eb90 tx=0x7febd800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:47.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.831+0000 7febdf7fe640 1 -- 172.21.15.27:0/4237534477 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd8004d20 con 0x7febe8069dc0 2024-04-25T00:26:47.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.831+0000 7febdf7fe640 1 -- 172.21.15.27:0/4237534477 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febd800ee90 con 0x7febe8069dc0 2024-04-25T00:26:47.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.831+0000 7febece7b640 1 -- 172.21.15.27:0/4237534477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febe819d2c0 con 0x7febe8069dc0 2024-04-25T00:26:47.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.831+0000 7febdf7fe640 1 -- 172.21.15.27:0/4237534477 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd8010830 con 0x7febe8069dc0 2024-04-25T00:26:47.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.831+0000 7febece7b640 1 -- 172.21.15.27:0/4237534477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febe81a4b50 con 0x7febe8069dc0 2024-04-25T00:26:47.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.832+0000 7febece7b640 1 -- 172.21.15.27:0/4237534477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febe80689b0 con 0x7febe8069dc0 2024-04-25T00:26:47.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.834+0000 7febdf7fe640 1 -- 172.21.15.27:0/4237534477 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7febd8005000 con 0x7febe8069dc0 2024-04-25T00:26:47.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.834+0000 7febdf7fe640 1 --2- 172.21.15.27:0/4237534477 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7febd40a2d00 0x7febd40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:47.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.834+0000 7febe5d9b640 1 --2- 172.21.15.27:0/4237534477 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7febd40a2d00 0x7febd40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:47.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.834+0000 7febdf7fe640 1 -- 172.21.15.27:0/4237534477 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9838+0+0 (secure 0 0 0) 0x7febd80d01a0 con 0x7febe8069dc0 2024-04-25T00:26:47.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.835+0000 7febe5d9b640 1 --2- 172.21.15.27:0/4237534477 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7febd40a2d00 0x7febd40a51c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7febd0006010 tx=0x7febd0000bc0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:47.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:47.839+0000 7febdf7fe640 1 -- 172.21.15.27:0/4237534477 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7febd8053740 con 0x7febe8069dc0 2024-04-25T00:26:47.942 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:26:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-a[67853]: ignoring --setuser ceph since I am not root 2024-04-25T00:26:47.942 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:26:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-a[67853]: ignoring --setgroup ceph since I am not root 2024-04-25T00:26:47.943 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:26:47 smithi027 ceph-mds[67857]: -- 172.21.15.27:0/4292201109 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e8af363a40 con 0x55e8af337000 2024-04-25T00:26:47.943 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:26:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-a[67853]: starting mds.a at 2024-04-25T00:26:47.943 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:26:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-g[67504]: ignoring --setuser ceph since I am not root 2024-04-25T00:26:47.943 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:26:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-g[67504]: ignoring --setgroup ceph since I am not root 2024-04-25T00:26:47.943 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:26:47 smithi027 ceph-mds[67508]: -- 172.21.15.27:0/2281427968 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557611a1fa40 con 0x5576119f2c00 2024-04-25T00:26:47.943 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:26:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-g[67504]: starting mds.g at 2024-04-25T00:26:47.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:47 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-a[29204]: 2024-04-25T00:26:47.690+0000 7f1730402700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T00:26:47.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:47 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3940705561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:26:47.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:47 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2873652157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:26:47.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:47 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2418050378' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T00:26:47.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:47 smithi027 ceph-mon[29208]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T00:26:47.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:47 smithi027 ceph-mon[29208]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T00:26:47.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:47 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/2418050378' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T00:26:47.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:47 smithi027 ceph-mon[29208]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T00:26:47.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:47 smithi027 ceph-mon[29208]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-25T00:26:48.027 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-c[63662]: ignoring --setuser ceph since I am not root 2024-04-25T00:26:48.027 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-c[63662]: ignoring --setgroup ceph since I am not root 2024-04-25T00:26:48.027 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-mds[63666]: -- 172.21.15.153:0/2450522409 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5583143cda40 con 0x5583143a1000 2024-04-25T00:26:48.027 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-c[63662]: starting mds.c at 2024-04-25T00:26:48.027 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-i[63969]: ignoring --setuser ceph since I am not root 2024-04-25T00:26:48.027 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-i[63969]: ignoring --setgroup ceph since I am not root 2024-04-25T00:26:48.027 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-mds[63973]: -- 172.21.15.153:0/1687450489 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c7f44a5a40 con 0x55c7f4479000 2024-04-25T00:26:48.027 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-i[63969]: starting mds.i at 2024-04-25T00:26:48.028 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-l[63030]: ignoring --setuser ceph since I am not root 2024-04-25T00:26:48.028 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-l[63030]: ignoring --setgroup ceph since I am not root 2024-04-25T00:26:48.028 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-mds[63034]: -- 172.21.15.153:0/1999030990 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562c32491a40 con 0x562c32465000 2024-04-25T00:26:48.028 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-l[63030]: starting mds.l at 2024-04-25T00:26:48.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3940705561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:26:48.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2873652157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:26:48.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2418050378' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T00:26:48.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-mon[32896]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T00:26:48.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-mon[32896]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T00:26:48.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/2418050378' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T00:26:48.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-mon[32896]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T00:26:48.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-mon[32896]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-25T00:26:48.029 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-f[63355]: ignoring --setuser ceph since I am not root 2024-04-25T00:26:48.029 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-f[63355]: ignoring --setgroup ceph since I am not root 2024-04-25T00:26:48.029 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-mds[63359]: -- 172.21.15.153:0/3913308219 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d2f1e724e0 con 0x55d2f1e49000 2024-04-25T00:26:48.029 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-mds[63359]: -- 172.21.15.153:0/3913308219 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d2f1e75a40 con 0x55d2f1e49000 2024-04-25T00:26:48.029 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:26:47 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-f[63355]: starting mds.f at 2024-04-25T00:26:48.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.063+0000 7febece7b640 1 -- 172.21.15.27:0/4237534477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7febe802d170 con 0x7febe8069dc0 2024-04-25T00:26:48.254 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-b[62989]: ignoring --setuser ceph since I am not root 2024-04-25T00:26:48.254 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-b[62989]: ignoring --setgroup ceph since I am not root 2024-04-25T00:26:48.254 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-mds[62993]: -- 172.21.15.139:0/1674017441 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5637880604e0 con 0x563788036c00 2024-04-25T00:26:48.254 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-mds[62993]: -- 172.21.15.139:0/1674017441 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563788063a40 con 0x563788036c00 2024-04-25T00:26:48.255 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-b[62989]: starting mds.b at 2024-04-25T00:26:48.255 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-e[62368]: ignoring --setuser ceph since I am not root 2024-04-25T00:26:48.255 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-e[62368]: ignoring --setgroup ceph since I am not root 2024-04-25T00:26:48.255 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-mds[62372]: -- 172.21.15.139:0/2385075053 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5635cd49da40 con 0x5635cd471000 2024-04-25T00:26:48.255 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-e[62368]: starting mds.e at 2024-04-25T00:26:48.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3940705561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T00:26:48.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2873652157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T00:26:48.256 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2418050378' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T00:26:48.256 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-mon[32833]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T00:26:48.256 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-mon[32833]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T00:26:48.256 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/2418050378' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T00:26:48.256 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-mon[32833]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T00:26:48.256 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-mon[32833]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-25T00:26:48.256 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-h[62679]: ignoring --setuser ceph since I am not root 2024-04-25T00:26:48.256 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-h[62679]: ignoring --setgroup ceph since I am not root 2024-04-25T00:26:48.256 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-mds[62683]: -- 172.21.15.139:0/181357447 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ebd379fa40 con 0x55ebd3773000 2024-04-25T00:26:48.256 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-h[62679]: starting mds.h at 2024-04-25T00:26:48.257 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-k[62044]: ignoring --setuser ceph since I am not root 2024-04-25T00:26:48.257 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-k[62044]: ignoring --setgroup ceph since I am not root 2024-04-25T00:26:48.257 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-mds[62048]: -- 172.21.15.139:0/2817481137 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559225ec3a40 con 0x559225e96800 2024-04-25T00:26:48.257 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:26:47 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-k[62044]: starting mds.k at 2024-04-25T00:26:48.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.693+0000 7febdf7fe640 1 -- 172.21.15.27:0/4237534477 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v103) ==== 104+0+0 (secure 0 0 0) 0x7febd809c8a0 con 0x7febe8069dc0 2024-04-25T00:26:48.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.697+0000 7febece7b640 1 -- 172.21.15.27:0/4237534477 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7febd40a2d00 msgr2=0x7febd40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:48.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.697+0000 7febece7b640 1 --2- 172.21.15.27:0/4237534477 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7febd40a2d00 0x7febd40a51c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7febd0006010 tx=0x7febd0000bc0 comp rx=0 tx=0).stop 2024-04-25T00:26:48.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.697+0000 7febece7b640 1 -- 172.21.15.27:0/4237534477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febe8069dc0 msgr2=0x7febe8198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:48.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.697+0000 7febece7b640 1 --2- 172.21.15.27:0/4237534477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febe8069dc0 0x7febe8198030 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7febd800eb90 tx=0x7febd800c6a0 comp rx=0 tx=0).stop 2024-04-25T00:26:48.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.697+0000 7febece7b640 1 -- 172.21.15.27:0/4237534477 shutdown_connections 2024-04-25T00:26:48.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.697+0000 7febece7b640 1 --2- 172.21.15.27:0/4237534477 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7febd40a2d00 0x7febd40a51c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:48.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.697+0000 7febece7b640 1 --2- 172.21.15.27:0/4237534477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febe8107730 0x7febe819c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:48.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.697+0000 7febece7b640 1 --2- 172.21.15.27:0/4237534477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7febe806a780 0x7febe8198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:48.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.697+0000 7febece7b640 1 --2- 172.21.15.27:0/4237534477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febe8069dc0 0x7febe8198030 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:48.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.697+0000 7febece7b640 1 -- 172.21.15.27:0/4237534477 >> 172.21.15.27:0/4237534477 conn(0x7febe80776a0 msgr2=0x7febe80fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:48.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.697+0000 7febece7b640 1 -- 172.21.15.27:0/4237534477 shutdown_connections 2024-04-25T00:26:48.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.697+0000 7febece7b640 1 -- 172.21.15.27:0/4237534477 wait complete. 2024-04-25T00:26:48.716 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T00:26:48.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.829+0000 7fa4f8309640 1 -- 172.21.15.27:0/3633667950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f01020d0 msgr2=0x7fa4f010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:48.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.829+0000 7fa4f8309640 1 --2- 172.21.15.27:0/3633667950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f01020d0 0x7fa4f010e5e0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8009cd0 tx=0x7fa4e802a8e0 comp rx=0 tx=0).stop 2024-04-25T00:26:48.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.829+0000 7fa4f8309640 1 -- 172.21.15.27:0/3633667950 shutdown_connections 2024-04-25T00:26:48.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.829+0000 7fa4f8309640 1 --2- 172.21.15.27:0/3633667950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f01020d0 0x7fa4f010e5e0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:48.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.829+0000 7fa4f8309640 1 --2- 172.21.15.27:0/3633667950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4f0101710 0x7fa4f0101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:48.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.829+0000 7fa4f8309640 1 --2- 172.21.15.27:0/3633667950 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4f0107720 0x7fa4f0107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:48.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.829+0000 7fa4f8309640 1 -- 172.21.15.27:0/3633667950 >> 172.21.15.27:0/3633667950 conn(0x7fa4f00fd180 msgr2=0x7fa4f00ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:48.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.829+0000 7fa4f8309640 1 -- 172.21.15.27:0/3633667950 shutdown_connections 2024-04-25T00:26:48.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.829+0000 7fa4f8309640 1 -- 172.21.15.27:0/3633667950 wait complete. 2024-04-25T00:26:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.829+0000 7fa4f8309640 1 Processor -- start 2024-04-25T00:26:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.829+0000 7fa4f8309640 1 -- start start 2024-04-25T00:26:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f8309640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4f0101710 0x7fa4f0073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f8309640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f01020d0 0x7fa4f0073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f8309640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4f0107720 0x7fa4f00744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f8309640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f0111b30 con 0x7fa4f01020d0 2024-04-25T00:26:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f8309640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f01119b0 con 0x7fa4f0101710 2024-04-25T00:26:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f8309640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f0111cb0 con 0x7fa4f0107720 2024-04-25T00:26:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f58a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f01020d0 0x7fa4f0073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f58a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f01020d0 0x7fa4f0073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34048/0 (socket says 172.21.15.27:34048) 2024-04-25T00:26:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f58a4640 1 -- 172.21.15.27:0/3812719904 learned_addr learned my addr 172.21.15.27:0/3812719904 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f60a5640 1 --2- 172.21.15.27:0/3812719904 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4f0101710 0x7fa4f0073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f68a6640 1 --2- 172.21.15.27:0/3812719904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4f0107720 0x7fa4f00744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f58a4640 1 -- 172.21.15.27:0/3812719904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4f0107720 msgr2=0x7fa4f00744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f58a4640 1 --2- 172.21.15.27:0/3812719904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4f0107720 0x7fa4f00744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f58a4640 1 -- 172.21.15.27:0/3812719904 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4f0101710 msgr2=0x7fa4f0073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f58a4640 1 --2- 172.21.15.27:0/3812719904 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4f0101710 0x7fa4f0073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f58a4640 1 -- 172.21.15.27:0/3812719904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f01a6570 con 0x7fa4f01020d0 2024-04-25T00:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f58a4640 1 --2- 172.21.15.27:0/3812719904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f01020d0 0x7fa4f0073f80 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fa4e000e9f0 tx=0x7fa4e000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4e77fe640 1 -- 172.21.15.27:0/3812719904 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e000cd80 con 0x7fa4f01020d0 2024-04-25T00:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f8309640 1 -- 172.21.15.27:0/3812719904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4f01a6860 con 0x7fa4f01020d0 2024-04-25T00:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4e77fe640 1 -- 172.21.15.27:0/3812719904 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4e0005500 con 0x7fa4f01020d0 2024-04-25T00:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4e77fe640 1 -- 172.21.15.27:0/3812719904 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0010690 con 0x7fa4f01020d0 2024-04-25T00:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.830+0000 7fa4f8309640 1 -- 172.21.15.27:0/3812719904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4f01a6d60 con 0x7fa4f01020d0 2024-04-25T00:26:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.832+0000 7fa4f8309640 1 -- 172.21.15.27:0/3812719904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4f00689b0 con 0x7fa4f01020d0 2024-04-25T00:26:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.833+0000 7fa4e77fe640 1 -- 172.21.15.27:0/3812719904 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa4e00046c0 con 0x7fa4f01020d0 2024-04-25T00:26:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.833+0000 7fa4e77fe640 1 --2- 172.21.15.27:0/3812719904 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa4dc0a2d00 0x7fa4dc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.833+0000 7fa4f60a5640 1 --2- 172.21.15.27:0/3812719904 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa4dc0a2d00 0x7fa4dc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:48.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.833+0000 7fa4e77fe640 1 -- 172.21.15.27:0/3812719904 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9838+0+0 (secure 0 0 0) 0x7fa4e00cfed0 con 0x7fa4f01020d0 2024-04-25T00:26:48.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.834+0000 7fa4f60a5640 1 --2- 172.21.15.27:0/3812719904 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa4dc0a2d00 0x7fa4dc0a51c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec005fd0 tx=0x7fa4ec005850 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:48.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:48.838+0000 7fa4e77fe640 1 -- 172.21.15.27:0/3812719904 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa4e009c5d0 con 0x7fa4f01020d0 2024-04-25T00:26:49.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.023+0000 7fa4f8309640 1 -- 172.21.15.27:0/3812719904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa4f0109ca0 con 0x7fa4f01020d0 2024-04-25T00:26:49.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: pgmap v336: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T00:26:49.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4237534477' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T00:26:49.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T00:26:49.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T00:26:49.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: Cluster is now healthy 2024-04-25T00:26:49.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4237534477' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T00:26:49.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.153:6832/3481337436,v1:172.21.15.153:6834/3481337436] up:boot 2024-04-25T00:26:49.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.153:6836/1857491886,v1:172.21.15.153:6837/1857491886] up:boot 2024-04-25T00:26:49.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.153:6833/3647139630,v1:172.21.15.153:6835/3647139630] up:boot 2024-04-25T00:26:49.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.153:6838/1826514004,v1:172.21.15.153:6839/1826514004] up:boot 2024-04-25T00:26:49.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6838/2722731086,v1:172.21.15.27:6839/2722731086] up:boot 2024-04-25T00:26:49.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.27:6840/1103251288,v1:172.21.15.27:6841/1103251288] up:boot 2024-04-25T00:26:49.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6832/1210262889,v1:172.21.15.139:6836/1210262889] up:boot 2024-04-25T00:26:49.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6833/3844216734,v1:172.21.15.139:6837/3844216734] up:boot 2024-04-25T00:26:49.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6834/3388026486,v1:172.21.15.139:6839/3388026486] up:boot 2024-04-25T00:26:49.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: mds.? [v2:172.21.15.139:6835/3854463613,v1:172.21.15.139:6838/3854463613] up:boot 2024-04-25T00:26:49.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: fsmap 12 up:standby 2024-04-25T00:26:49.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T00:26:49.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T00:26:49.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T00:26:49.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T00:26:49.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T00:26:49.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T00:26:49.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T00:26:49.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T00:26:49.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T00:26:49.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:48 smithi027 ceph-mon[29208]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T00:26:49.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: pgmap v336: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T00:26:49.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4237534477' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T00:26:49.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T00:26:49.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T00:26:49.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: Cluster is now healthy 2024-04-25T00:26:49.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4237534477' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T00:26:49.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.153:6832/3481337436,v1:172.21.15.153:6834/3481337436] up:boot 2024-04-25T00:26:49.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.153:6836/1857491886,v1:172.21.15.153:6837/1857491886] up:boot 2024-04-25T00:26:49.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.153:6833/3647139630,v1:172.21.15.153:6835/3647139630] up:boot 2024-04-25T00:26:49.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.153:6838/1826514004,v1:172.21.15.153:6839/1826514004] up:boot 2024-04-25T00:26:49.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6838/2722731086,v1:172.21.15.27:6839/2722731086] up:boot 2024-04-25T00:26:49.254 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.27:6840/1103251288,v1:172.21.15.27:6841/1103251288] up:boot 2024-04-25T00:26:49.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6832/1210262889,v1:172.21.15.139:6836/1210262889] up:boot 2024-04-25T00:26:49.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6833/3844216734,v1:172.21.15.139:6837/3844216734] up:boot 2024-04-25T00:26:49.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6834/3388026486,v1:172.21.15.139:6839/3388026486] up:boot 2024-04-25T00:26:49.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: mds.? [v2:172.21.15.139:6835/3854463613,v1:172.21.15.139:6838/3854463613] up:boot 2024-04-25T00:26:49.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: fsmap 12 up:standby 2024-04-25T00:26:49.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T00:26:49.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T00:26:49.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T00:26:49.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T00:26:49.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T00:26:49.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T00:26:49.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T00:26:49.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T00:26:49.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T00:26:49.255 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:48 smithi139 ceph-mon[32833]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T00:26:49.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: pgmap v336: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T00:26:49.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4237534477' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T00:26:49.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T00:26:49.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T00:26:49.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: Cluster is now healthy 2024-04-25T00:26:49.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4237534477' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T00:26:49.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.153:6832/3481337436,v1:172.21.15.153:6834/3481337436] up:boot 2024-04-25T00:26:49.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.153:6836/1857491886,v1:172.21.15.153:6837/1857491886] up:boot 2024-04-25T00:26:49.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.153:6833/3647139630,v1:172.21.15.153:6835/3647139630] up:boot 2024-04-25T00:26:49.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.153:6838/1826514004,v1:172.21.15.153:6839/1826514004] up:boot 2024-04-25T00:26:49.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6838/2722731086,v1:172.21.15.27:6839/2722731086] up:boot 2024-04-25T00:26:49.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.27:6840/1103251288,v1:172.21.15.27:6841/1103251288] up:boot 2024-04-25T00:26:49.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6832/1210262889,v1:172.21.15.139:6836/1210262889] up:boot 2024-04-25T00:26:49.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6833/3844216734,v1:172.21.15.139:6837/3844216734] up:boot 2024-04-25T00:26:49.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6834/3388026486,v1:172.21.15.139:6839/3388026486] up:boot 2024-04-25T00:26:49.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: mds.? [v2:172.21.15.139:6835/3854463613,v1:172.21.15.139:6838/3854463613] up:boot 2024-04-25T00:26:49.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: fsmap 12 up:standby 2024-04-25T00:26:49.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T00:26:49.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T00:26:49.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T00:26:49.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T00:26:49.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T00:26:49.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T00:26:49.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T00:26:49.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T00:26:49.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T00:26:49.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:48 smithi153 ceph-mon[32896]: from='mgr.15564 172.21.15.27:0/691060593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T00:26:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.818+0000 7fa4e77fe640 1 -- 172.21.15.27:0/3812719904 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v98) ==== 181+0+0 (secure 0 0 0) 0x7fa4e009bed0 con 0x7fa4f01020d0 2024-04-25T00:26:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-04-25T00:26:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.822+0000 7fa4f8309640 1 -- 172.21.15.27:0/3812719904 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa4dc0a2d00 msgr2=0x7fa4dc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.822+0000 7fa4f8309640 1 --2- 172.21.15.27:0/3812719904 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa4dc0a2d00 0x7fa4dc0a51c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec005fd0 tx=0x7fa4ec005850 comp rx=0 tx=0).stop 2024-04-25T00:26:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.822+0000 7fa4f8309640 1 -- 172.21.15.27:0/3812719904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f01020d0 msgr2=0x7fa4f0073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.822+0000 7fa4f8309640 1 --2- 172.21.15.27:0/3812719904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f01020d0 0x7fa4f0073f80 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fa4e000e9f0 tx=0x7fa4e000edf0 comp rx=0 tx=0).stop 2024-04-25T00:26:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.822+0000 7fa4f8309640 1 -- 172.21.15.27:0/3812719904 shutdown_connections 2024-04-25T00:26:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.822+0000 7fa4f8309640 1 --2- 172.21.15.27:0/3812719904 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fa4dc0a2d00 0x7fa4dc0a51c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.822+0000 7fa4f8309640 1 --2- 172.21.15.27:0/3812719904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4f0107720 0x7fa4f00744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.822+0000 7fa4f8309640 1 --2- 172.21.15.27:0/3812719904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f01020d0 0x7fa4f0073f80 secure :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fa4e000e9f0 tx=0x7fa4e000edf0 comp rx=0 tx=0).stop 2024-04-25T00:26:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.822+0000 7fa4f8309640 1 --2- 172.21.15.27:0/3812719904 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4f0101710 0x7fa4f0073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.822+0000 7fa4f8309640 1 -- 172.21.15.27:0/3812719904 >> 172.21.15.27:0/3812719904 conn(0x7fa4f00fd180 msgr2=0x7fa4f00fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.823+0000 7fa4f8309640 1 -- 172.21.15.27:0/3812719904 shutdown_connections 2024-04-25T00:26:49.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.823+0000 7fa4f8309640 1 -- 172.21.15.27:0/3812719904 wait complete. 2024-04-25T00:26:49.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T00:26:49.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.952+0000 7f2c75f28640 1 -- 172.21.15.27:0/2762259191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c701020d0 msgr2=0x7f2c7010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:49.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.952+0000 7f2c75f28640 1 --2- 172.21.15.27:0/2762259191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c701020d0 0x7f2c7010e5e0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f2c68009cd0 tx=0x7f2c6802a8e0 comp rx=0 tx=0).stop 2024-04-25T00:26:49.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.952+0000 7f2c75f28640 1 -- 172.21.15.27:0/2762259191 shutdown_connections 2024-04-25T00:26:49.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.952+0000 7f2c75f28640 1 --2- 172.21.15.27:0/2762259191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c701020d0 0x7f2c7010e5e0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:49.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.952+0000 7f2c75f28640 1 --2- 172.21.15.27:0/2762259191 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c70101710 0x7f2c70101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:49.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.952+0000 7f2c75f28640 1 --2- 172.21.15.27:0/2762259191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c70107720 0x7f2c70107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:49.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.952+0000 7f2c75f28640 1 -- 172.21.15.27:0/2762259191 >> 172.21.15.27:0/2762259191 conn(0x7f2c700fd180 msgr2=0x7f2c700ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:49.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.952+0000 7f2c75f28640 1 -- 172.21.15.27:0/2762259191 shutdown_connections 2024-04-25T00:26:49.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.952+0000 7f2c75f28640 1 -- 172.21.15.27:0/2762259191 wait complete. 2024-04-25T00:26:49.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.953+0000 7f2c75f28640 1 Processor -- start 2024-04-25T00:26:49.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.953+0000 7f2c75f28640 1 -- start start 2024-04-25T00:26:49.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.953+0000 7f2c75f28640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c70101710 0x7f2c70073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:49.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.953+0000 7f2c75f28640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c701020d0 0x7f2c70073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:49.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.953+0000 7f2c75f28640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c70107720 0x7f2c700744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:49.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.953+0000 7f2c75f28640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2c70111b30 con 0x7f2c701020d0 2024-04-25T00:26:49.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.953+0000 7f2c75f28640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2c701119b0 con 0x7f2c70107720 2024-04-25T00:26:49.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.953+0000 7f2c75f28640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2c70111cb0 con 0x7f2c70101710 2024-04-25T00:26:49.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.953+0000 7f2c6effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c701020d0 0x7f2c70073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:49.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.953+0000 7f2c6effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c701020d0 0x7f2c70073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34056/0 (socket says 172.21.15.27:34056) 2024-04-25T00:26:49.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.953+0000 7f2c6effd640 1 -- 172.21.15.27:0/1530102662 learned_addr learned my addr 172.21.15.27:0/1530102662 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:49.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.954+0000 7f2c6ffff640 1 --2- 172.21.15.27:0/1530102662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c70107720 0x7f2c700744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:49.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.954+0000 7f2c6f7fe640 1 --2- 172.21.15.27:0/1530102662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c70101710 0x7f2c70073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:49.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.954+0000 7f2c6effd640 1 -- 172.21.15.27:0/1530102662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c70101710 msgr2=0x7f2c70073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:49.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.954+0000 7f2c6effd640 1 --2- 172.21.15.27:0/1530102662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c70101710 0x7f2c70073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:49.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.954+0000 7f2c6effd640 1 -- 172.21.15.27:0/1530102662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c70107720 msgr2=0x7f2c700744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:49.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.954+0000 7f2c6effd640 1 --2- 172.21.15.27:0/1530102662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c70107720 0x7f2c700744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:49.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.954+0000 7f2c6effd640 1 -- 172.21.15.27:0/1530102662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c701a6570 con 0x7f2c701020d0 2024-04-25T00:26:49.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.954+0000 7f2c6ffff640 1 --2- 172.21.15.27:0/1530102662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c70107720 0x7f2c700744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:26:49.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.954+0000 7f2c6f7fe640 1 --2- 172.21.15.27:0/1530102662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c70101710 0x7f2c70073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:26:49.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.954+0000 7f2c6effd640 1 --2- 172.21.15.27:0/1530102662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c701020d0 0x7f2c70073f80 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000c890 tx=0x7f2c6000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:49.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.954+0000 7f2c6cff9640 1 -- 172.21.15.27:0/1530102662 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60004220 con 0x7f2c701020d0 2024-04-25T00:26:49.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.954+0000 7f2c6cff9640 1 -- 172.21.15.27:0/1530102662 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c600043c0 con 0x7f2c701020d0 2024-04-25T00:26:49.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.954+0000 7f2c75f28640 1 -- 172.21.15.27:0/1530102662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c701a6860 con 0x7f2c701020d0 2024-04-25T00:26:49.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.954+0000 7f2c6cff9640 1 -- 172.21.15.27:0/1530102662 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c600049e0 con 0x7f2c701020d0 2024-04-25T00:26:49.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.954+0000 7f2c75f28640 1 -- 172.21.15.27:0/1530102662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c701a6db0 con 0x7f2c701020d0 2024-04-25T00:26:49.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.955+0000 7f2c75f28640 1 -- 172.21.15.27:0/1530102662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c700689b0 con 0x7f2c701020d0 2024-04-25T00:26:49.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.956+0000 7f2c6cff9640 1 -- 172.21.15.27:0/1530102662 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2c600079b0 con 0x7f2c701020d0 2024-04-25T00:26:49.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.957+0000 7f2c6cff9640 1 --2- 172.21.15.27:0/1530102662 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2c580a2d00 0x7f2c580a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:49.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.957+0000 7f2c6f7fe640 1 --2- 172.21.15.27:0/1530102662 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2c580a2d00 0x7f2c580a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:49.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.957+0000 7f2c6cff9640 1 -- 172.21.15.27:0/1530102662 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9342+0+0 (secure 0 0 0) 0x7f2c600cf890 con 0x7f2c701020d0 2024-04-25T00:26:49.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.957+0000 7f2c6f7fe640 1 --2- 172.21.15.27:0/1530102662 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2c580a2d00 0x7f2c580a51c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009930 tx=0x7f2c64008040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:49.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:49.961+0000 7f2c6cff9640 1 -- 172.21.15.27:0/1530102662 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2c60014030 con 0x7f2c701020d0 2024-04-25T00:26:50.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.145+0000 7f2c75f28640 1 -- 172.21.15.27:0/1530102662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2c7002d170 con 0x7f2c701020d0 2024-04-25T00:26:50.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:49 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3812719904' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T00:26:50.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:49 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3812719904' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T00:26:50.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:49 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3812719904' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T00:26:50.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.824+0000 7f2c6cff9640 1 -- 172.21.15.27:0/1530102662 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v99) ==== 169+0+0 (secure 0 0 0) 0x7f2c6009c180 con 0x7f2c701020d0 2024-04-25T00:26:50.826 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-04-25T00:26:50.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.829+0000 7f2c75f28640 1 -- 172.21.15.27:0/1530102662 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2c580a2d00 msgr2=0x7f2c580a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:50.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.829+0000 7f2c75f28640 1 --2- 172.21.15.27:0/1530102662 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2c580a2d00 0x7f2c580a51c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009930 tx=0x7f2c64008040 comp rx=0 tx=0).stop 2024-04-25T00:26:50.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.829+0000 7f2c75f28640 1 -- 172.21.15.27:0/1530102662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c701020d0 msgr2=0x7f2c70073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:50.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.829+0000 7f2c75f28640 1 --2- 172.21.15.27:0/1530102662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c701020d0 0x7f2c70073f80 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000c890 tx=0x7f2c6000cd50 comp rx=0 tx=0).stop 2024-04-25T00:26:50.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.830+0000 7f2c75f28640 1 -- 172.21.15.27:0/1530102662 shutdown_connections 2024-04-25T00:26:50.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.830+0000 7f2c75f28640 1 --2- 172.21.15.27:0/1530102662 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7f2c580a2d00 0x7f2c580a51c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:50.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.830+0000 7f2c75f28640 1 --2- 172.21.15.27:0/1530102662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c70107720 0x7f2c700744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:50.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.830+0000 7f2c75f28640 1 --2- 172.21.15.27:0/1530102662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c701020d0 0x7f2c70073f80 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:50.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.830+0000 7f2c75f28640 1 --2- 172.21.15.27:0/1530102662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c70101710 0x7f2c70073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:50.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.830+0000 7f2c75f28640 1 -- 172.21.15.27:0/1530102662 >> 172.21.15.27:0/1530102662 conn(0x7f2c700fd180 msgr2=0x7f2c700fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:50.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.831+0000 7f2c75f28640 1 -- 172.21.15.27:0/1530102662 shutdown_connections 2024-04-25T00:26:50.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.831+0000 7f2c75f28640 1 -- 172.21.15.27:0/1530102662 wait complete. 2024-04-25T00:26:50.848 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-25T00:26:50.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.959+0000 7fbaa5629640 1 -- 172.21.15.27:0/3299680104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa01029c0 msgr2=0x7fbaa0102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:50.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.959+0000 7fbaa5629640 1 --2- 172.21.15.27:0/3299680104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa01029c0 0x7fbaa0102da0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fba98009900 tx=0x7fba9802a510 comp rx=0 tx=0).stop 2024-04-25T00:26:50.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.959+0000 7fbaa5629640 1 -- 172.21.15.27:0/3299680104 shutdown_connections 2024-04-25T00:26:50.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.959+0000 7fbaa5629640 1 --2- 172.21.15.27:0/3299680104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbaa010bd00 0x7fbaa010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:50.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.959+0000 7fbaa5629640 1 --2- 172.21.15.27:0/3299680104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaa01032e0 0x7fbaa010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:50.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.959+0000 7fbaa5629640 1 --2- 172.21.15.27:0/3299680104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa01029c0 0x7fbaa0102da0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:50.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.959+0000 7fbaa5629640 1 -- 172.21.15.27:0/3299680104 >> 172.21.15.27:0/3299680104 conn(0x7fbaa0077740 msgr2=0x7fbaa0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:50.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.960+0000 7fbaa5629640 1 -- 172.21.15.27:0/3299680104 shutdown_connections 2024-04-25T00:26:50.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.960+0000 7fbaa5629640 1 -- 172.21.15.27:0/3299680104 wait complete. 2024-04-25T00:26:50.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.960+0000 7fbaa5629640 1 Processor -- start 2024-04-25T00:26:50.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.960+0000 7fbaa5629640 1 -- start start 2024-04-25T00:26:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.960+0000 7fbaa5629640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbaa01029c0 0x7fbaa0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.960+0000 7fbaa5629640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa01032e0 0x7fbaa0198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.960+0000 7fbaa5629640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaa010bd00 0x7fbaa019c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.960+0000 7fbaa5629640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa0111b00 con 0x7fbaa01032e0 2024-04-25T00:26:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.960+0000 7fbaa5629640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa0111980 con 0x7fbaa01029c0 2024-04-25T00:26:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.960+0000 7fbaa5629640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa0111c80 con 0x7fbaa010bd00 2024-04-25T00:26:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.961+0000 7fba9e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa01032e0 0x7fbaa0198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.961+0000 7fba9e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa01032e0 0x7fbaa0198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34080/0 (socket says 172.21.15.27:34080) 2024-04-25T00:26:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.961+0000 7fba9e7fc640 1 -- 172.21.15.27:0/4065697443 learned_addr learned my addr 172.21.15.27:0/4065697443 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T00:26:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.961+0000 7fba9effd640 1 --2- 172.21.15.27:0/4065697443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbaa01029c0 0x7fbaa0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.961+0000 7fba9f7fe640 1 --2- 172.21.15.27:0/4065697443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaa010bd00 0x7fbaa019c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.961+0000 7fba9e7fc640 1 -- 172.21.15.27:0/4065697443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaa010bd00 msgr2=0x7fbaa019c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.961+0000 7fba9e7fc640 1 --2- 172.21.15.27:0/4065697443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaa010bd00 0x7fbaa019c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.961+0000 7fba9e7fc640 1 -- 172.21.15.27:0/4065697443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbaa01029c0 msgr2=0x7fbaa0198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:50.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.961+0000 7fba9e7fc640 1 --2- 172.21.15.27:0/4065697443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbaa01029c0 0x7fbaa0198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:50.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.961+0000 7fba9e7fc640 1 -- 172.21.15.27:0/4065697443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa019d030 con 0x7fbaa01032e0 2024-04-25T00:26:50.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.961+0000 7fba9e7fc640 1 --2- 172.21.15.27:0/4065697443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa01032e0 0x7fbaa0198580 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fba9400d770 tx=0x7fba9400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:50.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.961+0000 7fba87fff640 1 -- 172.21.15.27:0/4065697443 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba94017070 con 0x7fbaa01032e0 2024-04-25T00:26:50.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.961+0000 7fbaa5629640 1 -- 172.21.15.27:0/4065697443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaa019d320 con 0x7fbaa01032e0 2024-04-25T00:26:50.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.961+0000 7fba87fff640 1 -- 172.21.15.27:0/4065697443 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba94005500 con 0x7fbaa01032e0 2024-04-25T00:26:50.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.961+0000 7fba87fff640 1 -- 172.21.15.27:0/4065697443 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba94004e00 con 0x7fbaa01032e0 2024-04-25T00:26:50.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.961+0000 7fbaa5629640 1 -- 172.21.15.27:0/4065697443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaa01a4bb0 con 0x7fbaa01032e0 2024-04-25T00:26:50.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.962+0000 7fbaa5629640 1 -- 172.21.15.27:0/4065697443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaa0068560 con 0x7fbaa01032e0 2024-04-25T00:26:50.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.963+0000 7fba87fff640 1 -- 172.21.15.27:0/4065697443 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fba94004fa0 con 0x7fbaa01032e0 2024-04-25T00:26:50.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.964+0000 7fba87fff640 1 --2- 172.21.15.27:0/4065697443 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fba880a2d00 0x7fba880a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:50.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.964+0000 7fba9effd640 1 --2- 172.21.15.27:0/4065697443 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fba880a2d00 0x7fba880a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:50.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.964+0000 7fba87fff640 1 -- 172.21.15.27:0/4065697443 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8754+0+0 (secure 0 0 0) 0x7fba940cefc0 con 0x7fbaa01032e0 2024-04-25T00:26:50.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.965+0000 7fba9effd640 1 --2- 172.21.15.27:0/4065697443 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fba880a2d00 0x7fba880a51c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fba98009230 tx=0x7fba980058d0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:50.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:50.969+0000 7fba87fff640 1 -- 172.21.15.27:0/4065697443 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fba9409baf0 con 0x7fbaa01032e0 2024-04-25T00:26:51.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:51.152+0000 7fbaa5629640 1 -- 172.21.15.27:0/4065697443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbaa0106eb0 con 0x7fbaa01032e0 2024-04-25T00:26:51.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:50 smithi027 ceph-mon[29208]: pgmap v338: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-25T00:26:51.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:50 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/3812719904' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T00:26:51.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:50 smithi027 ceph-mon[29208]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T00:26:51.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:50 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1530102662' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T00:26:51.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:50 smithi139 ceph-mon[32833]: pgmap v338: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-25T00:26:51.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:50 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/3812719904' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T00:26:51.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:50 smithi139 ceph-mon[32833]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T00:26:51.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:50 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1530102662' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T00:26:51.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:50 smithi153 ceph-mon[32896]: pgmap v338: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-25T00:26:51.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:50 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/3812719904' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T00:26:51.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:50 smithi153 ceph-mon[32896]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T00:26:51.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:50 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1530102662' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T00:26:51.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:51.830+0000 7fba87fff640 1 -- 172.21.15.27:0/4065697443 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v100) ==== 178+0+0 (secure 0 0 0) 0x7fba9409b3f0 con 0x7fbaa01032e0 2024-04-25T00:26:51.831 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data_ec' removed 2024-04-25T00:26:51.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:51.833+0000 7fbaa5629640 1 -- 172.21.15.27:0/4065697443 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fba880a2d00 msgr2=0x7fba880a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:51.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:51.833+0000 7fbaa5629640 1 --2- 172.21.15.27:0/4065697443 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fba880a2d00 0x7fba880a51c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fba98009230 tx=0x7fba980058d0 comp rx=0 tx=0).stop 2024-04-25T00:26:51.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:51.833+0000 7fbaa5629640 1 -- 172.21.15.27:0/4065697443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa01032e0 msgr2=0x7fbaa0198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:51.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:51.833+0000 7fbaa5629640 1 --2- 172.21.15.27:0/4065697443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa01032e0 0x7fbaa0198580 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fba9400d770 tx=0x7fba9400dc30 comp rx=0 tx=0).stop 2024-04-25T00:26:51.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:51.834+0000 7fbaa5629640 1 -- 172.21.15.27:0/4065697443 shutdown_connections 2024-04-25T00:26:51.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:51.834+0000 7fbaa5629640 1 --2- 172.21.15.27:0/4065697443 >> [v2:172.21.15.27:6800/1076858552,v1:172.21.15.27:6801/1076858552] conn(0x7fba880a2d00 0x7fba880a51c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:51.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:51.834+0000 7fbaa5629640 1 --2- 172.21.15.27:0/4065697443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaa010bd00 0x7fbaa019c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:51.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:51.834+0000 7fbaa5629640 1 --2- 172.21.15.27:0/4065697443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa01032e0 0x7fbaa0198580 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:51.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:51.834+0000 7fbaa5629640 1 --2- 172.21.15.27:0/4065697443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbaa01029c0 0x7fbaa0198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:51.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:51.834+0000 7fbaa5629640 1 -- 172.21.15.27:0/4065697443 >> 172.21.15.27:0/4065697443 conn(0x7fbaa0077740 msgr2=0x7fbaa00fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:51.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:51.834+0000 7fbaa5629640 1 -- 172.21.15.27:0/4065697443 shutdown_connections 2024-04-25T00:26:51.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T00:26:51.834+0000 7fbaa5629640 1 -- 172.21.15.27:0/4065697443 wait complete. 2024-04-25T00:26:51.856 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T00:26:51.894 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T00:26:51.926 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T00:26:51.956 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-25T00:26:51.956 DEBUG:teuthology.orchestra.run.smithi027:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T00:26:51.982 DEBUG:teuthology.orchestra.run.smithi139:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T00:26:51.999 DEBUG:teuthology.orchestra.run.smithi153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T00:26:52.022 INFO:tasks.cephadm:Stopping all daemons... 2024-04-25T00:26:52.023 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-25T00:26:52.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.a 2024-04-25T00:26:52.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:51 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/1530102662' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T00:26:52.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:51 smithi027 ceph-mon[29208]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T00:26:52.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:51 smithi027 ceph-mon[29208]: from='client.? 172.21.15.27:0/4065697443' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T00:26:52.193 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:52 smithi027 systemd[1]: Stopping Ceph mon.a for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:26:52.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:51 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/1530102662' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T00:26:52.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:51 smithi139 ceph-mon[32833]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T00:26:52.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:51 smithi139 ceph-mon[32833]: from='client.? 172.21.15.27:0/4065697443' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T00:26:52.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:51 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/1530102662' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T00:26:52.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:51 smithi153 ceph-mon[32896]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T00:26:52.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:51 smithi153 ceph-mon[32896]: from='client.? 172.21.15.27:0/4065697443' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T00:26:52.921 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:52 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-a[29204]: 2024-04-25T00:26:52.502+0000 7f1735c0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:26:52.921 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:52 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-a[29204]: 2024-04-25T00:26:52.502+0000 7f1735c0d700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-25T00:26:52.921 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:52 smithi027 podman[96104]: 2024-04-25 00:26:52.706749262 +0000 UTC m=+0.355546582 container died f663a30b3ae1a8b72665aae6baae930fb2ff028b8c889d424da825be51b63cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-25T00:26:53.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:53 smithi027 podman[96104]: 2024-04-25 00:26:53.131255374 +0000 UTC m=+0.780052703 container cleanup f663a30b3ae1a8b72665aae6baae930fb2ff028b8c889d424da825be51b63cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-25T00:26:53.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:53 smithi027 bash[96104]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-a 2024-04-25T00:26:53.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 00:26:53 smithi027 podman[96180]: 2024-04-25 00:26:53.341190492 +0000 UTC m=+0.633055304 container remove f663a30b3ae1a8b72665aae6baae930fb2ff028b8c889d424da825be51b63cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-d2c45d9) 2024-04-25T00:26:53.780 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.a.service' 2024-04-25T00:26:53.838 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:26:53.838 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-25T00:26:53.838 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-25T00:26:53.838 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.b 2024-04-25T00:26:54.253 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:53 smithi139 systemd[1]: Stopping Ceph mon.b for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:26:54.637 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:54 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-b[32829]: 2024-04-25T00:26:54.332+0000 7f6cf8f53700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:26:54.638 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:54 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-b[32829]: 2024-04-25T00:26:54.332+0000 7f6cf8f53700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-25T00:26:54.638 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:54 smithi139 podman[68507]: 2024-04-25 00:26:54.456969929 +0000 UTC m=+0.250962254 container died cb23fccae8acc7a35c7afd0e3837e82c5f9c73c7f8af24681dabf23273f60831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-25T00:26:55.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:54 smithi139 podman[68507]: 2024-04-25 00:26:54.812641454 +0000 UTC m=+0.606633777 container cleanup cb23fccae8acc7a35c7afd0e3837e82c5f9c73c7f8af24681dabf23273f60831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-25T00:26:55.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:54 smithi139 bash[68507]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-b 2024-04-25T00:26:55.503 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:55 smithi139 podman[68518]: 2024-04-25 00:26:55.097499641 +0000 UTC m=+0.633417690 container remove cb23fccae8acc7a35c7afd0e3837e82c5f9c73c7f8af24681dabf23273f60831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:26:55.637 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.b.service' 2024-04-25T00:26:56.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:55 smithi139 systemd[1]: ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.b.service: Deactivated successfully. 2024-04-25T00:26:56.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:55 smithi139 systemd[1]: Stopped Ceph mon.b for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:26:56.003 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 00:26:55 smithi139 systemd[1]: ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.b.service: Consumed 13.220s CPU time. 2024-04-25T00:26:56.155 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:26:56.155 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-25T00:26:56.155 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-25T00:26:56.155 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.c 2024-04-25T00:26:56.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:56 smithi153 systemd[1]: Stopping Ceph mon.c for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:26:58.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:58 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-c[32892]: 2024-04-25T00:26:58.408+0000 7fa866c1e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:26:58.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:58 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-c[32892]: 2024-04-25T00:26:58.408+0000 7fa866c1e700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-25T00:26:58.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:26:58 smithi153 podman[69406]: 2024-04-25 00:26:58.474419539 +0000 UTC m=+1.411670724 container died d2ab362c99894eec4a1a10c18d01b9a614ddd0a45c9d92d6e54d27e9d0e7853e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux ) 2024-04-25T00:27:01.193 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:27:00 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:27:00.738+0000 7fa587162700 -1 client.15600: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2024-04-25T00:27:03.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:27:03 smithi153 podman[69406]: 2024-04-25 00:27:03.148987945 +0000 UTC m=+6.086239124 container cleanup d2ab362c99894eec4a1a10c18d01b9a614ddd0a45c9d92d6e54d27e9d0e7853e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:27:03.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:27:03 smithi153 bash[69406]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-c 2024-04-25T00:27:06.671 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.c.service' 2024-04-25T00:27:06.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:27:06 smithi153 podman[69417]: 2024-04-25 00:27:06.511665585 +0000 UTC m=+8.032680418 container remove d2ab362c99894eec4a1a10c18d01b9a614ddd0a45c9d92d6e54d27e9d0e7853e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T00:27:06.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:27:06 smithi153 systemd[1]: ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.c.service: Deactivated successfully. 2024-04-25T00:27:06.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:27:06 smithi153 systemd[1]: Stopped Ceph mon.c for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:27:06.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 00:27:06 smithi153 systemd[1]: ceph-518857f8-0297-11ef-bc93-c7b262605968@mon.c.service: Consumed 13.506s CPU time. 2024-04-25T00:27:06.819 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:27:06.820 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-25T00:27:06.820 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-25T00:27:06.820 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mgr.x 2024-04-25T00:27:07.128 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:27:06 smithi027 systemd[1]: Stopping Ceph mgr.x for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:27:07.442 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:27:07 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:27:07.127+0000 7fa620a7a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:27:07.443 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:27:07 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x[29441]: 2024-04-25T00:27:07.128+0000 7fa620a7a700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-25T00:27:07.443 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:27:07 smithi027 podman[96306]: 2024-04-25 00:27:07.158539561 +0000 UTC m=+0.153453245 container died f7edeb76725eec46259d2c9af45d924ae873e77e218472c31ef5878a55f8b171 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T00:27:07.846 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:27:07 smithi027 podman[96306]: 2024-04-25 00:27:07.594829941 +0000 UTC m=+0.589743821 container cleanup f7edeb76725eec46259d2c9af45d924ae873e77e218472c31ef5878a55f8b171 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:27:07.847 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:27:07 smithi027 bash[96306]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x 2024-04-25T00:27:08.083 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-25T00:27:08.104 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:27:07 smithi027 podman[96320]: 2024-04-25 00:27:07.84662119 +0000 UTC m=+0.677613014 container remove f7edeb76725eec46259d2c9af45d924ae873e77e218472c31ef5878a55f8b171 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-25T00:27:08.104 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:27:08 smithi027 systemd[1]: ceph-518857f8-0297-11ef-bc93-c7b262605968@mgr.x.service: Deactivated successfully. 2024-04-25T00:27:08.104 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:27:08 smithi027 systemd[1]: Stopped Ceph mgr.x for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:27:08.104 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 00:27:08 smithi027 systemd[1]: ceph-518857f8-0297-11ef-bc93-c7b262605968@mgr.x.service: Consumed 1min 26.738s CPU time. 2024-04-25T00:27:08.119 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:27:08.119 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-25T00:27:08.120 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-25T00:27:08.120 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mgr.y 2024-04-25T00:27:08.503 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:27:08 smithi139 systemd[1]: Stopping Ceph mgr.y for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:27:10.003 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:27:09 smithi139 podman[68633]: 2024-04-25 00:27:09.707896092 +0000 UTC m=+1.294202148 container died 9ed73c881dff1fa6ff4094912e5fb39041d780b0abafd42f7951bc64c7aaaafd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9) 2024-04-25T00:27:14.003 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:27:13 smithi139 podman[68633]: 2024-04-25 00:27:13.676555234 +0000 UTC m=+5.262861287 container cleanup 9ed73c881dff1fa6ff4094912e5fb39041d780b0abafd42f7951bc64c7aaaafd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-04-25T00:27:14.003 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 00:27:13 smithi139 bash[68633]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-y 2024-04-25T00:27:15.376 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-25T00:27:15.411 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:27:15.412 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-25T00:27:15.412 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-25T00:27:15.412 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mgr.z 2024-04-25T00:27:15.777 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:27:15 smithi153 systemd[1]: Stopping Ceph mgr.z for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:27:16.678 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:27:16 smithi153 podman[69540]: 2024-04-25 00:27:16.391766588 +0000 UTC m=+0.794116426 container died 1b5a5bdb31ff66bac4e8a2916d0ae0489f884895ea1ef3482a4cd49432ef38b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-25T00:27:17.026 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:27:16 smithi153 podman[69540]: 2024-04-25 00:27:16.779434866 +0000 UTC m=+1.181784697 container cleanup 1b5a5bdb31ff66bac4e8a2916d0ae0489f884895ea1ef3482a4cd49432ef38b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T00:27:17.026 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 00:27:16 smithi153 bash[69540]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mgr-z 2024-04-25T00:27:17.252 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-25T00:27:17.286 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:27:17.287 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-25T00:27:17.287 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-25T00:27:17.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.0 2024-04-25T00:27:17.692 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:27:17 smithi027 systemd[1]: Stopping Ceph osd.0 for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:27:19.693 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:27:19 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-0[41329]: 2024-04-25T00:27:19.325+0000 7f7589e19700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:27:19.693 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:27:19 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-0[41329]: 2024-04-25T00:27:19.325+0000 7f7589e19700 -1 osd.0 100 *** Got signal Terminated *** 2024-04-25T00:27:19.693 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:27:19 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-0[41329]: 2024-04-25T00:27:19.325+0000 7f7589e19700 -1 osd.0 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T00:27:25.443 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:27:25 smithi027 podman[96442]: 2024-04-25 00:27:25.170028266 +0000 UTC m=+7.684788689 container died b1fa4815137358afe17404ba5d22fe5f5e6d84419922371e7d7980a7999ef366 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T00:27:29.086 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:27:28 smithi027 podman[96442]: 2024-04-25 00:27:28.834756987 +0000 UTC m=+11.349517401 container cleanup b1fa4815137358afe17404ba5d22fe5f5e6d84419922371e7d7980a7999ef366 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-0, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-25T00:27:29.086 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:27:28 smithi027 bash[96442]: ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-0 2024-04-25T00:27:29.442 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:27:29 smithi027 podman[96459]: 2024-04-25 00:27:29.086303433 +0000 UTC m=+3.913451138 container remove b1fa4815137358afe17404ba5d22fe5f5e6d84419922371e7d7980a7999ef366 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9) 2024-04-25T00:27:29.834 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:27:29 smithi027 podman[96548]: 2024-04-25 00:27:29.412684688 +0000 UTC m=+0.019555954 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:27:29.835 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:27:29 smithi027 podman[96548]: 2024-04-25 00:27:29.554174233 +0000 UTC m=+0.161045528 container create 155a00840375e17c7d03aba518665155fb06a1532b84986924a4120b5e11cf89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T00:27:30.192 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:27:29 smithi027 podman[96548]: 2024-04-25 00:27:29.948477941 +0000 UTC m=+0.555349193 container init 155a00840375e17c7d03aba518665155fb06a1532b84986924a4120b5e11cf89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T00:27:30.192 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:27:29 smithi027 podman[96548]: 2024-04-25 00:27:29.951777757 +0000 UTC m=+0.558649000 container start 155a00840375e17c7d03aba518665155fb06a1532b84986924a4120b5e11cf89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T00:27:30.193 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:27:30 smithi027 podman[96548]: 2024-04-25 00:27:30.056836475 +0000 UTC m=+0.663707731 container attach 155a00840375e17c7d03aba518665155fb06a1532b84986924a4120b5e11cf89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-25T00:27:30.860 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 00:27:30 smithi027 podman[96548]: 2024-04-25 00:27:30.502375891 +0000 UTC m=+1.109247140 container died 155a00840375e17c7d03aba518665155fb06a1532b84986924a4120b5e11cf89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-25T00:27:31.220 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.0.service' 2024-04-25T00:27:31.261 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:27:31.262 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-25T00:27:31.262 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-25T00:27:31.262 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.1 2024-04-25T00:27:31.582 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:27:31 smithi139 systemd[1]: Stopping Ceph osd.1 for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:27:31.937 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:27:31 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-1[37413]: 2024-04-25T00:27:31.581+0000 7f56f4de0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:27:31.937 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:27:31 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-1[37413]: 2024-04-25T00:27:31.581+0000 7f56f4de0700 -1 osd.1 100 *** Got signal Terminated *** 2024-04-25T00:27:31.937 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:27:31 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-1[37413]: 2024-04-25T00:27:31.581+0000 7f56f4de0700 -1 osd.1 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T00:27:37.504 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:27:37 smithi139 podman[68760]: 2024-04-25 00:27:37.231476199 +0000 UTC m=+5.765221908 container died 92c14b18aa89946d084bfe5a8e850201bd3278fc30d1499a5ee2c6b9394c5c6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-1, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-25T00:27:37.925 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:27:37 smithi139 podman[68760]: 2024-04-25 00:27:37.623413734 +0000 UTC m=+6.157159448 container cleanup 92c14b18aa89946d084bfe5a8e850201bd3278fc30d1499a5ee2c6b9394c5c6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-1, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-25T00:27:37.925 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:27:37 smithi139 bash[68760]: ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-1 2024-04-25T00:27:38.226 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:27:37 smithi139 podman[68791]: 2024-04-25 00:27:37.925272688 +0000 UTC m=+0.685138826 container remove 92c14b18aa89946d084bfe5a8e850201bd3278fc30d1499a5ee2c6b9394c5c6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:27:38.503 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:27:38 smithi139 podman[68882]: 2024-04-25 00:27:38.128987866 +0000 UTC m=+0.017791667 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:27:38.503 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:27:38 smithi139 podman[68882]: 2024-04-25 00:27:38.27629107 +0000 UTC m=+0.165094860 container create 62369f4bd73034823cd45fe7354b982c31dd70b6de225736b8d1231a3cdeb5b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-25T00:27:39.003 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:27:38 smithi139 podman[68882]: 2024-04-25 00:27:38.539603691 +0000 UTC m=+0.428407480 container init 62369f4bd73034823cd45fe7354b982c31dd70b6de225736b8d1231a3cdeb5b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-25T00:27:39.003 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:27:38 smithi139 podman[68882]: 2024-04-25 00:27:38.542604174 +0000 UTC m=+0.431407973 container start 62369f4bd73034823cd45fe7354b982c31dd70b6de225736b8d1231a3cdeb5b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:27:39.004 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:27:38 smithi139 podman[68882]: 2024-04-25 00:27:38.612122899 +0000 UTC m=+0.500926705 container attach 62369f4bd73034823cd45fe7354b982c31dd70b6de225736b8d1231a3cdeb5b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T00:27:39.364 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 00:27:39 smithi139 podman[68882]: 2024-04-25 00:27:39.076135753 +0000 UTC m=+0.964939544 container died 62369f4bd73034823cd45fe7354b982c31dd70b6de225736b8d1231a3cdeb5b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-25T00:27:39.659 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.1.service' 2024-04-25T00:27:39.698 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:27:39.698 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-25T00:27:39.698 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-25T00:27:39.698 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.2 2024-04-25T00:27:40.016 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:27:39 smithi153 systemd[1]: Stopping Ceph osd.2 for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:27:40.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:27:40 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-2[38088]: 2024-04-25T00:27:40.015+0000 7f4690279700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:27:40.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:27:40 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-2[38088]: 2024-04-25T00:27:40.015+0000 7f4690279700 -1 osd.2 100 *** Got signal Terminated *** 2024-04-25T00:27:40.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:27:40 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-2[38088]: 2024-04-25T00:27:40.015+0000 7f4690279700 -1 osd.2 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T00:27:46.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:27:45 smithi153 podman[69666]: 2024-04-25 00:27:45.776607973 +0000 UTC m=+5.885132421 container died 6782d8d0da7cca99a17d8d9484e59f839d3483a59c6dc85cd17ddb003529a3d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:27:46.434 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:27:46 smithi153 podman[69666]: 2024-04-25 00:27:46.191374708 +0000 UTC m=+6.299899155 container cleanup 6782d8d0da7cca99a17d8d9484e59f839d3483a59c6dc85cd17ddb003529a3d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T00:27:46.434 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:27:46 smithi153 bash[69666]: ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-2 2024-04-25T00:27:46.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:27:46 smithi153 podman[69695]: 2024-04-25 00:27:46.434391706 +0000 UTC m=+0.660821630 container remove 6782d8d0da7cca99a17d8d9484e59f839d3483a59c6dc85cd17ddb003529a3d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-25T00:27:47.148 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:27:46 smithi153 podman[69787]: 2024-04-25 00:27:46.772297558 +0000 UTC m=+0.018143987 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:27:47.148 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:27:46 smithi153 podman[69787]: 2024-04-25 00:27:46.885729354 +0000 UTC m=+0.131575785 container create ae8e44e47f04951706b596f02f26ce5908f1a44fde8590286353da0f86e103bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-25T00:27:47.148 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:27:47 smithi153 podman[69787]: 2024-04-25 00:27:47.147817317 +0000 UTC m=+0.393663757 container init ae8e44e47f04951706b596f02f26ce5908f1a44fde8590286353da0f86e103bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-25T00:27:47.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:27:47 smithi153 podman[69787]: 2024-04-25 00:27:47.151317645 +0000 UTC m=+0.397164077 container start ae8e44e47f04951706b596f02f26ce5908f1a44fde8590286353da0f86e103bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T00:27:47.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:27:47 smithi153 podman[69787]: 2024-04-25 00:27:47.28857395 +0000 UTC m=+0.534420391 container attach ae8e44e47f04951706b596f02f26ce5908f1a44fde8590286353da0f86e103bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T00:27:48.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 00:27:47 smithi153 podman[69787]: 2024-04-25 00:27:47.674456371 +0000 UTC m=+0.920302792 container died ae8e44e47f04951706b596f02f26ce5908f1a44fde8590286353da0f86e103bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T00:27:48.457 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.2.service' 2024-04-25T00:27:48.495 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:27:48.495 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-25T00:27:48.496 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-25T00:27:48.496 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.3 2024-04-25T00:27:48.815 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 00:27:48 smithi027 systemd[1]: Stopping Ceph osd.3 for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:27:49.192 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 00:27:48 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-3[47836]: 2024-04-25T00:27:48.813+0000 7fe7acaaf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:27:49.192 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 00:27:48 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-3[47836]: 2024-04-25T00:27:48.813+0000 7fe7acaaf700 -1 osd.3 100 *** Got signal Terminated *** 2024-04-25T00:27:49.193 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 00:27:48 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-3[47836]: 2024-04-25T00:27:48.813+0000 7fe7acaaf700 -1 osd.3 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T00:27:54.638 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 00:27:54 smithi027 podman[96678]: 2024-04-25 00:27:54.334155457 +0000 UTC m=+5.636023985 container died d2ad0646c6da4b0e7be96ea543509cb5ebbab8dcc97a7bba7667348fefe51c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T00:27:54.942 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 00:27:54 smithi027 podman[96678]: 2024-04-25 00:27:54.738235484 +0000 UTC m=+6.040104012 container cleanup d2ad0646c6da4b0e7be96ea543509cb5ebbab8dcc97a7bba7667348fefe51c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.license=GPLv2) 2024-04-25T00:27:54.942 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 00:27:54 smithi027 bash[96678]: ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-3 2024-04-25T00:27:55.421 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 00:27:54 smithi027 podman[96699]: 2024-04-25 00:27:54.997955154 +0000 UTC m=+0.666539896 container remove d2ad0646c6da4b0e7be96ea543509cb5ebbab8dcc97a7bba7667348fefe51c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-3, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS) 2024-04-25T00:27:55.671 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 00:27:55 smithi027 podman[96784]: 2024-04-25 00:27:55.324051698 +0000 UTC m=+0.018718877 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:27:55.671 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 00:27:55 smithi027 podman[96784]: 2024-04-25 00:27:55.457459726 +0000 UTC m=+0.152126901 container create c4ada851af6048f3ee9463b07a063d97bd6dc67dd902e79cee690e7f13aa325f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-25T00:27:55.671 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 00:27:55 smithi027 podman[96784]: 2024-04-25 00:27:55.668268778 +0000 UTC m=+0.362935946 container init c4ada851af6048f3ee9463b07a063d97bd6dc67dd902e79cee690e7f13aa325f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-25T00:27:55.942 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 00:27:55 smithi027 podman[96784]: 2024-04-25 00:27:55.671410907 +0000 UTC m=+0.366078075 container start c4ada851af6048f3ee9463b07a063d97bd6dc67dd902e79cee690e7f13aa325f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-3-deactivate, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-25T00:27:55.942 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 00:27:55 smithi027 podman[96784]: 2024-04-25 00:27:55.793564124 +0000 UTC m=+0.488231308 container attach c4ada851af6048f3ee9463b07a063d97bd6dc67dd902e79cee690e7f13aa325f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-25T00:27:56.263 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 00:27:55 smithi027 podman[96784]: 2024-04-25 00:27:55.972023891 +0000 UTC m=+0.666691069 container died c4ada851af6048f3ee9463b07a063d97bd6dc67dd902e79cee690e7f13aa325f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-3-deactivate, ceph=True, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-25T00:27:56.631 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.3.service' 2024-04-25T00:27:56.671 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:27:56.672 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-25T00:27:56.672 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-25T00:27:56.672 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.4 2024-04-25T00:27:57.003 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 00:27:56 smithi139 systemd[1]: Stopping Ceph osd.4 for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:27:57.503 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 00:27:57 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-4[44194]: 2024-04-25T00:27:57.010+0000 7f5afa446700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:27:57.503 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 00:27:57 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-4[44194]: 2024-04-25T00:27:57.010+0000 7f5afa446700 -1 osd.4 100 *** Got signal Terminated *** 2024-04-25T00:27:57.504 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 00:27:57 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-4[44194]: 2024-04-25T00:27:57.010+0000 7f5afa446700 -1 osd.4 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T00:28:03.112 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 00:28:02 smithi139 podman[68989]: 2024-04-25 00:28:02.837098181 +0000 UTC m=+5.968964721 container died b63c667fe116979f3012c5aa08f0181d927f3871485a558bcbe468e8bc4d84d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-4, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=) 2024-04-25T00:28:03.380 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 00:28:03 smithi139 podman[68989]: 2024-04-25 00:28:03.211366381 +0000 UTC m=+6.343232920 container cleanup b63c667fe116979f3012c5aa08f0181d927f3871485a558bcbe468e8bc4d84d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9) 2024-04-25T00:28:03.380 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 00:28:03 smithi139 bash[68989]: ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-4 2024-04-25T00:28:03.753 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 00:28:03 smithi139 podman[69006]: 2024-04-25 00:28:03.512798499 +0000 UTC m=+0.671467407 container remove b63c667fe116979f3012c5aa08f0181d927f3871485a558bcbe468e8bc4d84d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-4, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-d2c45d9) 2024-04-25T00:28:04.082 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 00:28:03 smithi139 podman[69105]: 2024-04-25 00:28:03.731389853 +0000 UTC m=+0.018988997 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:28:04.083 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 00:28:03 smithi139 podman[69105]: 2024-04-25 00:28:03.855361659 +0000 UTC m=+0.142960803 container create ac6af9bd469d928fd7e18013f2d85e4ef2fe1012ab777ae7fda5393dceb6e62e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:28:04.376 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 00:28:04 smithi139 podman[69105]: 2024-04-25 00:28:04.082947355 +0000 UTC m=+0.370546497 container init ac6af9bd469d928fd7e18013f2d85e4ef2fe1012ab777ae7fda5393dceb6e62e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T00:28:04.377 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 00:28:04 smithi139 podman[69105]: 2024-04-25 00:28:04.086127741 +0000 UTC m=+0.373726888 container start ac6af9bd469d928fd7e18013f2d85e4ef2fe1012ab777ae7fda5393dceb6e62e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T00:28:04.377 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 00:28:04 smithi139 podman[69105]: 2024-04-25 00:28:04.166377485 +0000 UTC m=+0.453976627 container attach ac6af9bd469d928fd7e18013f2d85e4ef2fe1012ab777ae7fda5393dceb6e62e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:28:04.727 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 00:28:04 smithi139 podman[69105]: 2024-04-25 00:28:04.37666433 +0000 UTC m=+0.664263480 container died ac6af9bd469d928fd7e18013f2d85e4ef2fe1012ab777ae7fda5393dceb6e62e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T00:28:05.078 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.4.service' 2024-04-25T00:28:05.118 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:28:05.118 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-25T00:28:05.119 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-25T00:28:05.119 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.5 2024-04-25T00:28:05.446 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 00:28:05 smithi153 systemd[1]: Stopping Ceph osd.5 for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:28:05.776 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 00:28:05 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-5[44710]: 2024-04-25T00:28:05.445+0000 7fca35175700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:28:05.776 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 00:28:05 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-5[44710]: 2024-04-25T00:28:05.445+0000 7fca35175700 -1 osd.5 100 *** Got signal Terminated *** 2024-04-25T00:28:05.776 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 00:28:05 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-5[44710]: 2024-04-25T00:28:05.445+0000 7fca35175700 -1 osd.5 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T00:28:11.596 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 00:28:11 smithi153 podman[69893]: 2024-04-25 00:28:11.29258132 +0000 UTC m=+5.980981646 container died 301fa508b3ee6682a9c40b0dba369cfa0d17255f95b4535463e7661494a2bb1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T00:28:11.963 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 00:28:11 smithi153 podman[69893]: 2024-04-25 00:28:11.704138359 +0000 UTC m=+6.392538694 container cleanup 301fa508b3ee6682a9c40b0dba369cfa0d17255f95b4535463e7661494a2bb1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T00:28:11.964 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 00:28:11 smithi153 bash[69893]: ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-5 2024-04-25T00:28:12.276 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 00:28:11 smithi153 podman[69913]: 2024-04-25 00:28:11.96400332 +0000 UTC m=+0.667900566 container remove 301fa508b3ee6682a9c40b0dba369cfa0d17255f95b4535463e7661494a2bb1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-25T00:28:12.627 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 00:28:12 smithi153 podman[70002]: 2024-04-25 00:28:12.273209675 +0000 UTC m=+0.018071855 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:28:12.627 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 00:28:12 smithi153 podman[70002]: 2024-04-25 00:28:12.389956579 +0000 UTC m=+0.134818743 container create 8146b2943aff755c7a5d0a04cb5e3d3ea1f4127437fece5571f92b0dead4c658 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-25T00:28:12.627 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 00:28:12 smithi153 podman[70002]: 2024-04-25 00:28:12.627043533 +0000 UTC m=+0.371905702 container init 8146b2943aff755c7a5d0a04cb5e3d3ea1f4127437fece5571f92b0dead4c658 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_CLEAN=True) 2024-04-25T00:28:12.925 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 00:28:12 smithi153 podman[70002]: 2024-04-25 00:28:12.630843932 +0000 UTC m=+0.375706094 container start 8146b2943aff755c7a5d0a04cb5e3d3ea1f4127437fece5571f92b0dead4c658 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-25T00:28:12.925 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 00:28:12 smithi153 podman[70002]: 2024-04-25 00:28:12.735015253 +0000 UTC m=+0.479877425 container attach 8146b2943aff755c7a5d0a04cb5e3d3ea1f4127437fece5571f92b0dead4c658 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T00:28:13.277 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 00:28:12 smithi153 podman[70002]: 2024-04-25 00:28:12.925242189 +0000 UTC m=+0.670104361 container died 8146b2943aff755c7a5d0a04cb5e3d3ea1f4127437fece5571f92b0dead4c658 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:28:13.641 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.5.service' 2024-04-25T00:28:13.679 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:28:13.680 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-25T00:28:13.680 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-25T00:28:13.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.6 2024-04-25T00:28:14.008 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 00:28:13 smithi027 systemd[1]: Stopping Ceph osd.6 for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:28:14.442 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 00:28:14 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-6[54396]: 2024-04-25T00:28:14.007+0000 7fa1063af700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:28:14.443 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 00:28:14 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-6[54396]: 2024-04-25T00:28:14.007+0000 7fa1063af700 -1 osd.6 100 *** Got signal Terminated *** 2024-04-25T00:28:14.443 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 00:28:14 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-6[54396]: 2024-04-25T00:28:14.007+0000 7fa1063af700 -1 osd.6 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T00:28:19.943 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 00:28:19 smithi027 podman[96889]: 2024-04-25 00:28:19.691336084 +0000 UTC m=+5.810252964 container died 3f295298506bf08b85ed5252416f765a6c1b092b149ff7312b6e3b8e655b057e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-6, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T00:28:20.350 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 00:28:20 smithi027 podman[96889]: 2024-04-25 00:28:20.040882407 +0000 UTC m=+6.159799285 container cleanup 3f295298506bf08b85ed5252416f765a6c1b092b149ff7312b6e3b8e655b057e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-6, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T00:28:20.350 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 00:28:20 smithi027 bash[96889]: ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-6 2024-04-25T00:28:20.654 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 00:28:20 smithi027 podman[96918]: 2024-04-25 00:28:20.350586819 +0000 UTC m=+0.655877994 container remove 3f295298506bf08b85ed5252416f765a6c1b092b149ff7312b6e3b8e655b057e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-6, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, ceph=True) 2024-04-25T00:28:20.942 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 00:28:20 smithi027 podman[97013]: 2024-04-25 00:28:20.557843936 +0000 UTC m=+0.018621355 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:28:20.943 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 00:28:20 smithi027 podman[97013]: 2024-04-25 00:28:20.684677554 +0000 UTC m=+0.145454956 container create 84315cbc1dc11fc4fc22b5c838d54b116e7b4ee1d1a78a6996ddd6c872e38a87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T00:28:20.943 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 00:28:20 smithi027 podman[97013]: 2024-04-25 00:28:20.895463198 +0000 UTC m=+0.356240597 container init 84315cbc1dc11fc4fc22b5c838d54b116e7b4ee1d1a78a6996ddd6c872e38a87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:28:20.943 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 00:28:20 smithi027 podman[97013]: 2024-04-25 00:28:20.898673196 +0000 UTC m=+0.359450605 container start 84315cbc1dc11fc4fc22b5c838d54b116e7b4ee1d1a78a6996ddd6c872e38a87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424) 2024-04-25T00:28:21.206 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 00:28:21 smithi027 podman[97013]: 2024-04-25 00:28:21.004044509 +0000 UTC m=+0.464821918 container attach 84315cbc1dc11fc4fc22b5c838d54b116e7b4ee1d1a78a6996ddd6c872e38a87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T00:28:21.549 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 00:28:21 smithi027 podman[97013]: 2024-04-25 00:28:21.206238081 +0000 UTC m=+0.667015497 container died 84315cbc1dc11fc4fc22b5c838d54b116e7b4ee1d1a78a6996ddd6c872e38a87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-25T00:28:21.868 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.6.service' 2024-04-25T00:28:21.909 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:28:21.910 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-25T00:28:21.910 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-25T00:28:21.910 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.7 2024-04-25T00:28:22.253 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 00:28:21 smithi139 systemd[1]: Stopping Ceph osd.7 for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:28:22.753 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 00:28:22 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-7[51188]: 2024-04-25T00:28:22.287+0000 7fd1f0807700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:28:22.753 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 00:28:22 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-7[51188]: 2024-04-25T00:28:22.287+0000 7fd1f0807700 -1 osd.7 100 *** Got signal Terminated *** 2024-04-25T00:28:22.753 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 00:28:22 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-7[51188]: 2024-04-25T00:28:22.287+0000 7fd1f0807700 -1 osd.7 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T00:28:27.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:26 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:26.900+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:27.985 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:27 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:27.701+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:28.253 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 00:28:27 smithi139 podman[69214]: 2024-04-25 00:28:27.985068405 +0000 UTC m=+5.876868646 container died 4d6a184c52f3fee84e14ff94a328899916084eb0bf953ba6a96505b04f689c5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-25T00:28:28.276 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:27 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:27.901+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:28.752 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 00:28:28 smithi139 podman[69214]: 2024-04-25 00:28:28.31328083 +0000 UTC m=+6.205081071 container cleanup 4d6a184c52f3fee84e14ff94a328899916084eb0bf953ba6a96505b04f689c5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T00:28:28.752 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 00:28:28 smithi139 bash[69214]: ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-7 2024-04-25T00:28:28.752 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 00:28:28 smithi139 podman[69231]: 2024-04-25 00:28:28.497918996 +0000 UTC m=+0.509626341 container remove 4d6a184c52f3fee84e14ff94a328899916084eb0bf953ba6a96505b04f689c5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-25T00:28:29.006 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 00:28:28 smithi139 podman[69325]: 2024-04-25 00:28:28.831737224 +0000 UTC m=+0.018084979 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:28:29.006 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 00:28:28 smithi139 podman[69325]: 2024-04-25 00:28:28.965821325 +0000 UTC m=+0.152169071 container create 8153360994a647cb6e8a2f0da027bab2dc1ae5ced0da4d0b4ab267e0d80253ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-25T00:28:29.006 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:28 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:28.750+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:29.276 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:28 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:28.931+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:29.503 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 00:28:29 smithi139 podman[69325]: 2024-04-25 00:28:29.218690199 +0000 UTC m=+0.405037945 container init 8153360994a647cb6e8a2f0da027bab2dc1ae5ced0da4d0b4ab267e0d80253ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-25T00:28:29.504 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 00:28:29 smithi139 podman[69325]: 2024-04-25 00:28:29.221776689 +0000 UTC m=+0.408124436 container start 8153360994a647cb6e8a2f0da027bab2dc1ae5ced0da4d0b4ab267e0d80253ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9) 2024-04-25T00:28:29.504 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 00:28:29 smithi139 podman[69325]: 2024-04-25 00:28:29.301889741 +0000 UTC m=+0.488237493 container attach 8153360994a647cb6e8a2f0da027bab2dc1ae5ced0da4d0b4ab267e0d80253ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-25T00:28:29.692 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:29 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:29.423+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:21.058654+0000 front 2024-04-25T00:27:21.058703+0000 (oldest deadline 2024-04-25T00:28:28.659326+0000) 2024-04-25T00:28:29.788 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 00:28:29 smithi139 podman[69325]: 2024-04-25 00:28:29.50549964 +0000 UTC m=+0.691847382 container died 8153360994a647cb6e8a2f0da027bab2dc1ae5ced0da4d0b4ab267e0d80253ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T00:28:29.788 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:29 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:29.712+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:30.062 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.7.service' 2024-04-25T00:28:30.100 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:28:30.100 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-25T00:28:30.101 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-25T00:28:30.101 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.8 2024-04-25T00:28:30.276 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 00:28:30 smithi153 systemd[1]: Stopping Ceph osd.8 for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:28:30.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:29 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:29.971+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:30.692 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:30 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:30.436+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:21.058654+0000 front 2024-04-25T00:27:21.058703+0000 (oldest deadline 2024-04-25T00:28:28.659326+0000) 2024-04-25T00:28:30.776 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 00:28:30 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-8[51595]: 2024-04-25T00:28:30.435+0000 7fda35192700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:28:30.777 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 00:28:30 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-8[51595]: 2024-04-25T00:28:30.435+0000 7fda35192700 -1 osd.8 100 *** Got signal Terminated *** 2024-04-25T00:28:30.777 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 00:28:30 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-8[51595]: 2024-04-25T00:28:30.435+0000 7fda35192700 -1 osd.8 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T00:28:31.003 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:30 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:30.668+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:31.276 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:31 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:31.015+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:31.692 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:31 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:31.464+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:21.058654+0000 front 2024-04-25T00:27:21.058703+0000 (oldest deadline 2024-04-25T00:28:28.659326+0000) 2024-04-25T00:28:32.003 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:31 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:31.706+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:32.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:31 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:31.968+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:32.692 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:32 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:32.423+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:21.058654+0000 front 2024-04-25T00:27:21.058703+0000 (oldest deadline 2024-04-25T00:28:28.659326+0000) 2024-04-25T00:28:33.003 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:32 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:32.720+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:33.276 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:32 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:32.923+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:33.692 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:33 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:33.384+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:21.058654+0000 front 2024-04-25T00:27:21.058703+0000 (oldest deadline 2024-04-25T00:28:28.659326+0000) 2024-04-25T00:28:34.003 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:33 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:33.676+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:34.276 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:33 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:33.918+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:34.692 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:34 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:34.424+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:21.058654+0000 front 2024-04-25T00:27:21.058703+0000 (oldest deadline 2024-04-25T00:28:28.659326+0000) 2024-04-25T00:28:35.003 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:34 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:34.651+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:35.276 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:34 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:34.904+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:35.692 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:35 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:35.412+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:21.058654+0000 front 2024-04-25T00:27:21.058703+0000 (oldest deadline 2024-04-25T00:28:28.659326+0000) 2024-04-25T00:28:36.003 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:35 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:35.694+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:36.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 00:28:36 smithi153 podman[70121]: 2024-04-25 00:28:36.142128659 +0000 UTC m=+5.841136255 container died ab31b3015dc98c77ade3d04b7c63c18114c7aedd01760f2590114b6e612dd4f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-8, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True) 2024-04-25T00:28:36.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:35 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:35.919+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:36.692 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:36 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:36.433+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:21.058654+0000 front 2024-04-25T00:27:21.058703+0000 (oldest deadline 2024-04-25T00:28:28.659326+0000) 2024-04-25T00:28:36.776 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 00:28:36 smithi153 podman[70121]: 2024-04-25 00:28:36.527030589 +0000 UTC m=+6.226038192 container cleanup ab31b3015dc98c77ade3d04b7c63c18114c7aedd01760f2590114b6e612dd4f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-8, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T00:28:36.777 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 00:28:36 smithi153 bash[70121]: ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-8 2024-04-25T00:28:37.003 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:36 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:36.655+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:37.129 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:36 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:36.953+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:37.129 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 00:28:36 smithi153 podman[70138]: 2024-04-25 00:28:36.786330274 +0000 UTC m=+0.641615621 container remove ab31b3015dc98c77ade3d04b7c63c18114c7aedd01760f2590114b6e612dd4f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-8, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-25T00:28:37.390 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 00:28:37 smithi153 podman[70226]: 2024-04-25 00:28:37.129582519 +0000 UTC m=+0.110017984 container create 16ce61c85ee335dd74953b6665987b6baf38e4c434ac08c1cdc78446f08fb9ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-04-25T00:28:37.390 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 00:28:37 smithi153 podman[70226]: 2024-04-25 00:28:37.037704586 +0000 UTC m=+0.018140015 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:28:37.686 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 00:28:37 smithi153 podman[70226]: 2024-04-25 00:28:37.390295411 +0000 UTC m=+0.370730831 container init 16ce61c85ee335dd74953b6665987b6baf38e4c434ac08c1cdc78446f08fb9ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T00:28:37.686 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 00:28:37 smithi153 podman[70226]: 2024-04-25 00:28:37.393790564 +0000 UTC m=+0.374225992 container start 16ce61c85ee335dd74953b6665987b6baf38e4c434ac08c1cdc78446f08fb9ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T00:28:37.686 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 00:28:37 smithi153 podman[70226]: 2024-04-25 00:28:37.481977714 +0000 UTC m=+0.462413135 container attach 16ce61c85ee335dd74953b6665987b6baf38e4c434ac08c1cdc78446f08fb9ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T00:28:37.942 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:37 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:37.466+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:21.058654+0000 front 2024-04-25T00:27:21.058703+0000 (oldest deadline 2024-04-25T00:28:28.659326+0000) 2024-04-25T00:28:37.988 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 00:28:37 smithi153 podman[70226]: 2024-04-25 00:28:37.686517153 +0000 UTC m=+0.666952580 container died 16ce61c85ee335dd74953b6665987b6baf38e4c434ac08c1cdc78446f08fb9ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:28:38.003 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:37 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:37.643+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:38.276 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:37 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:37.988+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:38.296 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.8.service' 2024-04-25T00:28:38.335 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:28:38.335 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-25T00:28:38.336 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-25T00:28:38.336 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.9 2024-04-25T00:28:38.653 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:38 smithi027 systemd[1]: Stopping Ceph osd.9 for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:28:38.653 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:38 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:38.461+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:21.058654+0000 front 2024-04-25T00:27:21.058703+0000 (oldest deadline 2024-04-25T00:28:28.659326+0000) 2024-04-25T00:28:38.942 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:38 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:38.651+0000 7f10e38f2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:28:38.942 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:38 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:38.651+0000 7f10e38f2700 -1 osd.9 100 *** Got signal Terminated *** 2024-04-25T00:28:38.942 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:38 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:38.651+0000 7f10e38f2700 -1 osd.9 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T00:28:39.003 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:38 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:38.672+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:39.276 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:39 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:39.013+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:39.942 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:39 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:39.464+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:21.058654+0000 front 2024-04-25T00:27:21.058703+0000 (oldest deadline 2024-04-25T00:28:28.659326+0000) 2024-04-25T00:28:39.943 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:39 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:39.464+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:35.661470+0000 front 2024-04-25T00:27:35.661422+0000 (oldest deadline 2024-04-25T00:28:39.161459+0000) 2024-04-25T00:28:40.003 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:39 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:39.717+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:40.003 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:39 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:39.717+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:34.645033+0000 front 2024-04-25T00:27:34.645014+0000 (oldest deadline 2024-04-25T00:28:38.745130+0000) 2024-04-25T00:28:40.276 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:40 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:39.999+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:40.942 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:40 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:40.510+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:21.058654+0000 front 2024-04-25T00:27:21.058703+0000 (oldest deadline 2024-04-25T00:28:28.659326+0000) 2024-04-25T00:28:40.942 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:40 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:40.510+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:35.661470+0000 front 2024-04-25T00:27:35.661422+0000 (oldest deadline 2024-04-25T00:28:39.161459+0000) 2024-04-25T00:28:41.003 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:40 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:40.752+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:41.003 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:40 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:40.752+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:34.645033+0000 front 2024-04-25T00:27:34.645014+0000 (oldest deadline 2024-04-25T00:28:38.745130+0000) 2024-04-25T00:28:41.276 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:41 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:41.022+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:41.942 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:41 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:41.462+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:21.058654+0000 front 2024-04-25T00:27:21.058703+0000 (oldest deadline 2024-04-25T00:28:28.659326+0000) 2024-04-25T00:28:41.942 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:41 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:41.462+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:35.661470+0000 front 2024-04-25T00:27:35.661422+0000 (oldest deadline 2024-04-25T00:28:39.161459+0000) 2024-04-25T00:28:42.003 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:41 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:41.746+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:42.003 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:41 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:41.746+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:34.645033+0000 front 2024-04-25T00:27:34.645014+0000 (oldest deadline 2024-04-25T00:28:38.745130+0000) 2024-04-25T00:28:42.276 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:41 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:41.994+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:42.942 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:42 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:42.443+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:21.058654+0000 front 2024-04-25T00:27:21.058703+0000 (oldest deadline 2024-04-25T00:28:28.659326+0000) 2024-04-25T00:28:42.943 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:42 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:42.443+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:35.661470+0000 front 2024-04-25T00:27:35.661422+0000 (oldest deadline 2024-04-25T00:28:39.161459+0000) 2024-04-25T00:28:43.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:42 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:42.773+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:43.254 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:42 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:42.773+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:34.645033+0000 front 2024-04-25T00:27:34.645014+0000 (oldest deadline 2024-04-25T00:28:38.745130+0000) 2024-04-25T00:28:43.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:43 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:43.036+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:43.692 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:43 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:43.411+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:21.058654+0000 front 2024-04-25T00:27:21.058703+0000 (oldest deadline 2024-04-25T00:28:28.659326+0000) 2024-04-25T00:28:43.692 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:43 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9[61291]: 2024-04-25T00:28:43.411+0000 7f10dfee6700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:35.661470+0000 front 2024-04-25T00:27:35.661422+0000 (oldest deadline 2024-04-25T00:28:39.161459+0000) 2024-04-25T00:28:44.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:43 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:43.766+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:44.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:43 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:43.766+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:34.645033+0000 front 2024-04-25T00:27:34.645014+0000 (oldest deadline 2024-04-25T00:28:38.745130+0000) 2024-04-25T00:28:44.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:44 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:44.057+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:44.728 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:44 smithi027 podman[97119]: 2024-04-25 00:28:44.455410366 +0000 UTC m=+5.925562245 container died 9fde1ca8e12cd06c2b4b3edf242f4dc0875b5baf424d567bd1ae5c9f45bba0c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-25T00:28:45.046 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:44 smithi027 podman[97119]: 2024-04-25 00:28:44.843989195 +0000 UTC m=+6.314141057 container cleanup 9fde1ca8e12cd06c2b4b3edf242f4dc0875b5baf424d567bd1ae5c9f45bba0c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-25T00:28:45.046 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:44 smithi027 bash[97119]: ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9 2024-04-25T00:28:45.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:44 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:44.797+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:45.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:44 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:44.797+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:34.645033+0000 front 2024-04-25T00:27:34.645014+0000 (oldest deadline 2024-04-25T00:28:38.745130+0000) 2024-04-25T00:28:45.442 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:45 smithi027 podman[97136]: 2024-04-25 00:28:45.136956189 +0000 UTC m=+0.683093911 container remove 9fde1ca8e12cd06c2b4b3edf242f4dc0875b5baf424d567bd1ae5c9f45bba0c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-d2c45d9) 2024-04-25T00:28:45.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:45 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:45.057+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:45.773 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:45 smithi027 podman[97236]: 2024-04-25 00:28:45.375884176 +0000 UTC m=+0.018716730 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:28:45.773 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:45 smithi027 podman[97236]: 2024-04-25 00:28:45.48770531 +0000 UTC m=+0.130537860 container create 1faf003c07b44aa43bbe094c1e6ae6a492166637e936a23bb42f421421b0f53a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T00:28:45.773 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:45 smithi027 podman[97236]: 2024-04-25 00:28:45.698352104 +0000 UTC m=+0.341184649 container init 1faf003c07b44aa43bbe094c1e6ae6a492166637e936a23bb42f421421b0f53a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9) 2024-04-25T00:28:45.774 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:45 smithi027 podman[97236]: 2024-04-25 00:28:45.701666266 +0000 UTC m=+0.344498819 container start 1faf003c07b44aa43bbe094c1e6ae6a492166637e936a23bb42f421421b0f53a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9) 2024-04-25T00:28:46.192 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:45 smithi027 podman[97236]: 2024-04-25 00:28:45.773335073 +0000 UTC m=+0.416167639 container attach 1faf003c07b44aa43bbe094c1e6ae6a492166637e936a23bb42f421421b0f53a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-25T00:28:46.192 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 00:28:46 smithi027 podman[97236]: 2024-04-25 00:28:46.015965535 +0000 UTC m=+0.658798079 container died 1faf003c07b44aa43bbe094c1e6ae6a492166637e936a23bb42f421421b0f53a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T00:28:46.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:45 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:45.837+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:46.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:45 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:45.837+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:34.645033+0000 front 2024-04-25T00:27:34.645014+0000 (oldest deadline 2024-04-25T00:28:38.745130+0000) 2024-04-25T00:28:46.525 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.9.service' 2024-04-25T00:28:46.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:46 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:46.037+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:46.565 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:28:46.565 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-25T00:28:46.566 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-25T00:28:46.566 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.10 2024-04-25T00:28:46.858 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:46 smithi139 systemd[1]: Stopping Ceph osd.10 for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:28:47.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:46 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:46.857+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:47.254 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:46 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:46.857+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:34.645033+0000 front 2024-04-25T00:27:34.645014+0000 (oldest deadline 2024-04-25T00:28:38.745130+0000) 2024-04-25T00:28:47.254 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:46 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:46.865+0000 7f8583571700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:28:47.254 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:46 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:46.865+0000 7f8583571700 -1 osd.10 100 *** Got signal Terminated *** 2024-04-25T00:28:47.254 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:46 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:46.865+0000 7f8583571700 -1 osd.10 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T00:28:47.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:47 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:47.085+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:48.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:47 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:47.812+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:48.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:47 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:47.812+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:34.645033+0000 front 2024-04-25T00:27:34.645014+0000 (oldest deadline 2024-04-25T00:28:38.745130+0000) 2024-04-25T00:28:48.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:47 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:47.812+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T00:27:44.646357+0000 front 2024-04-25T00:27:44.646225+0000 (oldest deadline 2024-04-25T00:28:47.546417+0000) 2024-04-25T00:28:48.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:48 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:48.098+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:48.527 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:48 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:48.098+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T00:27:43.359366+0000 front 2024-04-25T00:27:43.359245+0000 (oldest deadline 2024-04-25T00:28:48.059416+0000) 2024-04-25T00:28:49.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:48 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:48.844+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:49.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:48 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:48.844+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:34.645033+0000 front 2024-04-25T00:27:34.645014+0000 (oldest deadline 2024-04-25T00:28:38.745130+0000) 2024-04-25T00:28:49.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:48 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:48.844+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T00:27:44.646357+0000 front 2024-04-25T00:27:44.646225+0000 (oldest deadline 2024-04-25T00:28:47.546417+0000) 2024-04-25T00:28:49.276 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:49 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:49.130+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:49.276 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:49 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:49.130+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T00:27:43.359366+0000 front 2024-04-25T00:27:43.359245+0000 (oldest deadline 2024-04-25T00:28:48.059416+0000) 2024-04-25T00:28:50.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:49 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:49.861+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:50.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:49 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:49.861+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:34.645033+0000 front 2024-04-25T00:27:34.645014+0000 (oldest deadline 2024-04-25T00:28:38.745130+0000) 2024-04-25T00:28:50.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:49 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:49.861+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T00:27:44.646357+0000 front 2024-04-25T00:27:44.646225+0000 (oldest deadline 2024-04-25T00:28:47.546417+0000) 2024-04-25T00:28:50.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:50 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:50.128+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:50.527 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:50 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:50.128+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T00:27:43.359366+0000 front 2024-04-25T00:27:43.359245+0000 (oldest deadline 2024-04-25T00:28:48.059416+0000) 2024-04-25T00:28:51.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:50 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:50.870+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:51.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:50 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:50.870+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:34.645033+0000 front 2024-04-25T00:27:34.645014+0000 (oldest deadline 2024-04-25T00:28:38.745130+0000) 2024-04-25T00:28:51.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:50 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:50.870+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T00:27:44.646357+0000 front 2024-04-25T00:27:44.646225+0000 (oldest deadline 2024-04-25T00:28:47.546417+0000) 2024-04-25T00:28:51.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:51 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:51.128+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:51.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:51 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:51.128+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T00:27:43.359366+0000 front 2024-04-25T00:27:43.359245+0000 (oldest deadline 2024-04-25T00:28:48.059416+0000) 2024-04-25T00:28:52.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:51 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:51.833+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.543199+0000 front 2024-04-25T00:27:23.543140+0000 (oldest deadline 2024-04-25T00:28:27.043164+0000) 2024-04-25T00:28:52.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:51 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:51.833+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T00:27:34.645033+0000 front 2024-04-25T00:27:34.645014+0000 (oldest deadline 2024-04-25T00:28:38.745130+0000) 2024-04-25T00:28:52.253 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:51 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10[57759]: 2024-04-25T00:28:51.833+0000 7f857fb65700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T00:27:44.646357+0000 front 2024-04-25T00:27:44.646225+0000 (oldest deadline 2024-04-25T00:28:47.546417+0000) 2024-04-25T00:28:52.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:52 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:52.144+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:52.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:52 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:52.144+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T00:27:43.359366+0000 front 2024-04-25T00:27:43.359245+0000 (oldest deadline 2024-04-25T00:28:48.059416+0000) 2024-04-25T00:28:52.708 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:52 smithi139 podman[69442]: 2024-04-25 00:28:52.427778439 +0000 UTC m=+5.666839698 container died 263ff50ab294c00c154407f2c7a7990883f3b3ae2f10d5ef55dad7cf4efe3c7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T00:28:53.003 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:52 smithi139 podman[69442]: 2024-04-25 00:28:52.798190968 +0000 UTC m=+6.037252216 container cleanup 263ff50ab294c00c154407f2c7a7990883f3b3ae2f10d5ef55dad7cf4efe3c7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T00:28:53.003 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:52 smithi139 bash[69442]: ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10 2024-04-25T00:28:53.400 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:53 smithi139 podman[69460]: 2024-04-25 00:28:53.049692822 +0000 UTC m=+0.622204162 container remove 263ff50ab294c00c154407f2c7a7990883f3b3ae2f10d5ef55dad7cf4efe3c7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-25T00:28:53.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:53 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:53.131+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:53.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:53 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:53.131+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T00:27:43.359366+0000 front 2024-04-25T00:27:43.359245+0000 (oldest deadline 2024-04-25T00:28:48.059416+0000) 2024-04-25T00:28:53.660 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:53 smithi139 podman[69551]: 2024-04-25 00:28:53.400307802 +0000 UTC m=+0.106046532 container create a9d7b8181b835907dec3af3e1f2d9801224d66741937f1c11155e85a1bc384e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9) 2024-04-25T00:28:53.661 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:53 smithi139 podman[69551]: 2024-04-25 00:28:53.312097598 +0000 UTC m=+0.017836354 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:28:53.661 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:53 smithi139 podman[69551]: 2024-04-25 00:28:53.660864672 +0000 UTC m=+0.366603403 container init a9d7b8181b835907dec3af3e1f2d9801224d66741937f1c11155e85a1bc384e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T00:28:53.957 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:53 smithi139 podman[69551]: 2024-04-25 00:28:53.663966913 +0000 UTC m=+0.369705652 container start a9d7b8181b835907dec3af3e1f2d9801224d66741937f1c11155e85a1bc384e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T00:28:53.957 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:53 smithi139 podman[69551]: 2024-04-25 00:28:53.744710063 +0000 UTC m=+0.450448807 container attach a9d7b8181b835907dec3af3e1f2d9801224d66741937f1c11155e85a1bc384e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T00:28:54.230 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:53 smithi139 podman[69551]: 2024-04-25 00:28:53.958045236 +0000 UTC m=+0.663783984 container died a9d7b8181b835907dec3af3e1f2d9801224d66741937f1c11155e85a1bc384e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:28:54.485 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 00:28:54 smithi139 podman[69578]: 2024-04-25 00:28:54.464884606 +0000 UTC m=+0.501396015 container remove a9d7b8181b835907dec3af3e1f2d9801224d66741937f1c11155e85a1bc384e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-d2c45d9, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:28:54.490 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.10.service' 2024-04-25T00:28:54.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:54 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:54.123+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:54.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:54 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:54.123+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T00:27:43.359366+0000 front 2024-04-25T00:27:43.359245+0000 (oldest deadline 2024-04-25T00:28:48.059416+0000) 2024-04-25T00:28:54.528 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:28:54.528 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-25T00:28:54.528 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-25T00:28:54.528 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.11 2024-04-25T00:28:55.026 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:54 smithi153 systemd[1]: Stopping Ceph osd.11 for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:28:55.027 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:54 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:54.795+0000 7fc095898700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:28:55.027 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:54 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:54.795+0000 7fc095898700 -1 osd.11 100 *** Got signal Terminated *** 2024-04-25T00:28:55.027 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:54 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:54.795+0000 7fc095898700 -1 osd.11 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T00:28:55.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:55 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:55.167+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:55.527 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:55 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:55.167+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T00:27:43.359366+0000 front 2024-04-25T00:27:43.359245+0000 (oldest deadline 2024-04-25T00:28:48.059416+0000) 2024-04-25T00:28:56.527 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:56 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:56.133+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:56.527 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:56 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:56.133+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T00:27:43.359366+0000 front 2024-04-25T00:27:43.359245+0000 (oldest deadline 2024-04-25T00:28:48.059416+0000) 2024-04-25T00:28:57.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:57 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:57.151+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:57.527 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:57 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:57.151+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T00:27:43.359366+0000 front 2024-04-25T00:27:43.359245+0000 (oldest deadline 2024-04-25T00:28:48.059416+0000) 2024-04-25T00:28:57.527 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:57 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:57.151+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T00:27:51.560449+0000 front 2024-04-25T00:27:51.560368+0000 (oldest deadline 2024-04-25T00:28:56.260369+0000) 2024-04-25T00:28:58.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:58 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:58.190+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:58.527 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:58 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:58.190+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T00:27:43.359366+0000 front 2024-04-25T00:27:43.359245+0000 (oldest deadline 2024-04-25T00:28:48.059416+0000) 2024-04-25T00:28:58.527 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:58 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:58.190+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T00:27:51.560449+0000 front 2024-04-25T00:27:51.560368+0000 (oldest deadline 2024-04-25T00:28:56.260369+0000) 2024-04-25T00:28:59.526 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:59 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:59.224+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T00:27:23.555166+0000 front 2024-04-25T00:27:23.555278+0000 (oldest deadline 2024-04-25T00:28:26.455304+0000) 2024-04-25T00:28:59.527 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:59 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:59.224+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T00:27:43.359366+0000 front 2024-04-25T00:27:43.359245+0000 (oldest deadline 2024-04-25T00:28:48.059416+0000) 2024-04-25T00:28:59.527 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:28:59 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11[58362]: 2024-04-25T00:28:59.224+0000 7fc08de84700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T00:27:51.560449+0000 front 2024-04-25T00:27:51.560368+0000 (oldest deadline 2024-04-25T00:28:56.260369+0000) 2024-04-25T00:29:00.863 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:29:00 smithi153 podman[70343]: 2024-04-25 00:29:00.579825234 +0000 UTC m=+5.859332328 container died dd4490cb99aabf189492633aeafdded4247c9abee38aee4d553698d5b90fc92e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-25T00:29:01.204 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:29:00 smithi153 podman[70343]: 2024-04-25 00:29:00.961559222 +0000 UTC m=+6.241066309 container cleanup dd4490cb99aabf189492633aeafdded4247c9abee38aee4d553698d5b90fc92e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T00:29:01.204 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:29:00 smithi153 bash[70343]: ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11 2024-04-25T00:29:01.514 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:29:01 smithi153 podman[70360]: 2024-04-25 00:29:01.204301233 +0000 UTC m=+0.618946624 container remove dd4490cb99aabf189492633aeafdded4247c9abee38aee4d553698d5b90fc92e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-25T00:29:01.777 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:29:01 smithi153 podman[70452]: 2024-04-25 00:29:01.513768829 +0000 UTC m=+0.103757565 container create b6cef82322aeb30280af8423a8c1be96ef2807e196978573d70e1de8286bc8a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T00:29:01.777 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:29:01 smithi153 podman[70452]: 2024-04-25 00:29:01.427855622 +0000 UTC m=+0.017844316 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T00:29:01.777 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:29:01 smithi153 podman[70452]: 2024-04-25 00:29:01.74958635 +0000 UTC m=+0.339575048 container init b6cef82322aeb30280af8423a8c1be96ef2807e196978573d70e1de8286bc8a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T00:29:01.777 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:29:01 smithi153 podman[70452]: 2024-04-25 00:29:01.753076335 +0000 UTC m=+0.343065029 container start b6cef82322aeb30280af8423a8c1be96ef2807e196978573d70e1de8286bc8a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11-deactivate, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T00:29:02.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:29:01 smithi153 podman[70452]: 2024-04-25 00:29:01.841079491 +0000 UTC m=+0.431068201 container attach b6cef82322aeb30280af8423a8c1be96ef2807e196978573d70e1de8286bc8a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T00:29:02.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 00:29:02 smithi153 podman[70452]: 2024-04-25 00:29:02.046314394 +0000 UTC m=+0.636303097 container died b6cef82322aeb30280af8423a8c1be96ef2807e196978573d70e1de8286bc8a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T00:29:02.633 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@osd.11.service' 2024-04-25T00:29:02.672 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:29:02.673 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-25T00:29:02.673 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-25T00:29:02.673 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.a 2024-04-25T00:29:02.964 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:29:02 smithi027 systemd[1]: Stopping Ceph mds.a for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:29:03.442 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:29:02 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-a[67853]: 2024-04-25T00:29:02.964+0000 7f7e18e2a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:29:03.442 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:29:02 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-a[67853]: 2024-04-25T00:29:02.964+0000 7f7e18e2a700 -1 mds.a *** got signal Terminated *** 2024-04-25T00:29:07.243 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:29:06 smithi027 podman[97342]: 2024-04-25 00:29:06.97628702 +0000 UTC m=+4.102685288 container died 08bfbb52e555de6afb24f7f55187f64dffc302c3b383dae47d0af62d8f1f0fbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T00:29:07.548 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:29:07 smithi027 podman[97342]: 2024-04-25 00:29:07.314048616 +0000 UTC m=+4.440446923 container cleanup 08bfbb52e555de6afb24f7f55187f64dffc302c3b383dae47d0af62d8f1f0fbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T00:29:07.548 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:29:07 smithi027 bash[97342]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-a 2024-04-25T00:29:07.799 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 00:29:07 smithi027 podman[97356]: 2024-04-25 00:29:07.548514013 +0000 UTC m=+0.560603289 container remove 08bfbb52e555de6afb24f7f55187f64dffc302c3b383dae47d0af62d8f1f0fbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:29:07.802 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.a.service' 2024-04-25T00:29:07.838 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:29:07.838 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-25T00:29:07.838 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-25T00:29:07.838 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.d 2024-04-25T00:29:08.177 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 00:29:07 smithi027 systemd[1]: Stopping Ceph mds.d for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:29:08.442 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 00:29:08 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-d[66898]: 2024-04-25T00:29:08.176+0000 7fc4b0137700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:29:08.442 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 00:29:08 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-d[66898]: 2024-04-25T00:29:08.176+0000 7fc4b0137700 -1 mds.d *** got signal Terminated *** 2024-04-25T00:29:10.728 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 00:29:10 smithi027 podman[97462]: 2024-04-25 00:29:10.464948419 +0000 UTC m=+2.403551086 container died def632097920c29825e5d2b372618bc00719ec2d7f5df6310bdb1b4e60816061 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9) 2024-04-25T00:29:10.986 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 00:29:10 smithi027 podman[97462]: 2024-04-25 00:29:10.818041415 +0000 UTC m=+2.756644091 container cleanup def632097920c29825e5d2b372618bc00719ec2d7f5df6310bdb1b4e60816061 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-25T00:29:10.986 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 00:29:10 smithi027 bash[97462]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-d 2024-04-25T00:29:11.195 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.d.service' 2024-04-25T00:29:11.228 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:29:11.229 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-25T00:29:11.229 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-25T00:29:11.229 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.g 2024-04-25T00:29:11.572 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:29:11 smithi027 systemd[1]: Stopping Ceph mds.g for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:29:11.942 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:29:11 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-g[67504]: 2024-04-25T00:29:11.572+0000 7f26290e7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:29:11.942 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:29:11 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-g[67504]: 2024-04-25T00:29:11.572+0000 7f26290e7700 -1 mds.g *** got signal Terminated *** 2024-04-25T00:29:15.839 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:29:15 smithi027 podman[97582]: 2024-04-25 00:29:15.587938897 +0000 UTC m=+4.142985195 container died 8ec929729bb67d74985e98123c026f82740492c4083966af08110725f203f701 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9) 2024-04-25T00:29:16.123 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:29:15 smithi027 podman[97582]: 2024-04-25 00:29:15.930143286 +0000 UTC m=+4.485189577 container cleanup 8ec929729bb67d74985e98123c026f82740492c4083966af08110725f203f701 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-g, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-25T00:29:16.123 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:29:15 smithi027 bash[97582]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-g 2024-04-25T00:29:16.356 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.g.service' 2024-04-25T00:29:16.377 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:29:16 smithi027 podman[97594]: 2024-04-25 00:29:16.122918411 +0000 UTC m=+0.532947714 container remove 8ec929729bb67d74985e98123c026f82740492c4083966af08110725f203f701 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-25T00:29:16.377 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:29:16 smithi027 systemd[1]: ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.g.service: Deactivated successfully. 2024-04-25T00:29:16.377 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:29:16 smithi027 systemd[1]: Stopped Ceph mds.g for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:29:16.377 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 00:29:16 smithi027 systemd[1]: ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.g.service: Consumed 1.970s CPU time. 2024-04-25T00:29:16.388 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:29:16.388 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-25T00:29:16.388 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-25T00:29:16.389 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.j 2024-04-25T00:29:16.692 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 00:29:16 smithi027 systemd[1]: Stopping Ceph mds.j for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:29:17.192 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 00:29:16 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-j[66549]: 2024-04-25T00:29:16.712+0000 7f7cd5eb6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:29:17.192 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 00:29:16 smithi027 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-j[66549]: 2024-04-25T00:29:16.712+0000 7f7cd5eb6700 -1 mds.j *** got signal Terminated *** 2024-04-25T00:29:19.752 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 00:29:19 smithi027 podman[97701]: 2024-04-25 00:29:19.451643785 +0000 UTC m=+2.844996992 container died 9926742db09d993558e2a1b8f4d83c9b6c9b0f256d71f5d9875311fb5201cacd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-j, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T00:29:20.068 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 00:29:19 smithi027 podman[97701]: 2024-04-25 00:29:19.850815452 +0000 UTC m=+3.244168660 container cleanup 9926742db09d993558e2a1b8f4d83c9b6c9b0f256d71f5d9875311fb5201cacd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T00:29:20.068 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 00:29:19 smithi027 bash[97701]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-j 2024-04-25T00:29:20.226 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.j.service' 2024-04-25T00:29:20.260 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:29:20.260 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-25T00:29:20.260 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-25T00:29:20.261 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.b 2024-04-25T00:29:20.547 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:29:20 smithi139 systemd[1]: Stopping Ceph mds.b for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:29:20.805 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:29:20 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-b[62989]: 2024-04-25T00:29:20.546+0000 7f187f458700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:29:20.805 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:29:20 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-b[62989]: 2024-04-25T00:29:20.546+0000 7f187f458700 -1 mds.b *** got signal Terminated *** 2024-04-25T00:29:24.091 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:29:23 smithi139 podman[69658]: 2024-04-25 00:29:23.810607431 +0000 UTC m=+3.361823333 container died 3e317c4309d5710c6b99a6a1534a388826ca6d458be6e262f9ee2f0ca02d710a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T00:29:24.473 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:29:24 smithi139 podman[69658]: 2024-04-25 00:29:24.189460142 +0000 UTC m=+3.740676057 container cleanup 3e317c4309d5710c6b99a6a1534a388826ca6d458be6e262f9ee2f0ca02d710a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True) 2024-04-25T00:29:24.473 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 00:29:24 smithi139 bash[69658]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-b 2024-04-25T00:29:24.589 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.b.service' 2024-04-25T00:29:24.623 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:29:24.624 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-25T00:29:24.624 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-25T00:29:24.624 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.e 2024-04-25T00:29:24.753 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:29:24 smithi139 systemd[1]: Stopping Ceph mds.e for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:29:25.253 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:29:24 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-e[62368]: 2024-04-25T00:29:24.934+0000 7f43c3349700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:29:25.253 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:29:24 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-e[62368]: 2024-04-25T00:29:24.934+0000 7f43c3349700 -1 mds.e *** got signal Terminated *** 2024-04-25T00:29:28.077 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:29:27 smithi139 podman[69790]: 2024-04-25 00:29:27.811227584 +0000 UTC m=+2.972147028 container died 8faab5e9836d4a198e7a50570f709197e79195cd9b5fecc8d843c623251c5388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-25T00:29:28.343 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:29:28 smithi139 podman[69790]: 2024-04-25 00:29:28.151067823 +0000 UTC m=+3.311987263 container cleanup 8faab5e9836d4a198e7a50570f709197e79195cd9b5fecc8d843c623251c5388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0) 2024-04-25T00:29:28.344 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:29:28 smithi139 bash[69790]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-e 2024-04-25T00:29:28.578 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.e.service' 2024-04-25T00:29:28.599 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:29:28 smithi139 podman[69804]: 2024-04-25 00:29:28.343711406 +0000 UTC m=+0.529533807 container remove 8faab5e9836d4a198e7a50570f709197e79195cd9b5fecc8d843c623251c5388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS) 2024-04-25T00:29:28.599 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:29:28 smithi139 systemd[1]: ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.e.service: Deactivated successfully. 2024-04-25T00:29:28.599 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:29:28 smithi139 systemd[1]: Stopped Ceph mds.e for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:29:28.599 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 00:29:28 smithi139 systemd[1]: ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.e.service: Consumed 2.142s CPU time. 2024-04-25T00:29:28.611 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:29:28.612 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-25T00:29:28.612 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-25T00:29:28.612 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.h 2024-04-25T00:29:28.929 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:29:28 smithi139 systemd[1]: Stopping Ceph mds.h for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:29:29.253 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:29:28 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-h[62679]: 2024-04-25T00:29:28.929+0000 7f2d1a74b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:29:29.253 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:29:28 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-h[62679]: 2024-04-25T00:29:28.929+0000 7f2d1a74b700 -1 mds.h *** got signal Terminated *** 2024-04-25T00:29:32.071 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:29:31 smithi139 podman[69910]: 2024-04-25 00:29:31.815177632 +0000 UTC m=+2.988960588 container died 82223188b6cc765aff24143766a63beef588e22b935024d3b8883cf276923378 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T00:29:32.472 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:29:32 smithi139 podman[69910]: 2024-04-25 00:29:32.170908874 +0000 UTC m=+3.344691838 container cleanup 82223188b6cc765aff24143766a63beef588e22b935024d3b8883cf276923378 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T00:29:32.472 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 00:29:32 smithi139 bash[69910]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-h 2024-04-25T00:29:32.608 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.h.service' 2024-04-25T00:29:32.640 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:29:32.640 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-25T00:29:32.640 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-25T00:29:32.640 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.k 2024-04-25T00:29:32.753 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:29:32 smithi139 systemd[1]: Stopping Ceph mds.k for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:29:33.253 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:29:32 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-k[62044]: 2024-04-25T00:29:32.974+0000 7f1fab6fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:29:33.253 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:29:32 smithi139 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-k[62044]: 2024-04-25T00:29:32.974+0000 7f1fab6fb700 -1 mds.k *** got signal Terminated *** 2024-04-25T00:29:36.083 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:29:35 smithi139 podman[70031]: 2024-04-25 00:29:35.812974977 +0000 UTC m=+2.955252211 container died 7024280520b2158248c5b1fe3b149d309f87206bb66e7aef0585dbb72718fd98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T00:29:36.334 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:29:36 smithi139 podman[70031]: 2024-04-25 00:29:36.149635081 +0000 UTC m=+3.291912317 container cleanup 7024280520b2158248c5b1fe3b149d309f87206bb66e7aef0585dbb72718fd98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-25T00:29:36.334 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:29:36 smithi139 bash[70031]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-k 2024-04-25T00:29:36.583 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.k.service' 2024-04-25T00:29:36.602 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:29:36 smithi139 podman[70045]: 2024-04-25 00:29:36.333772861 +0000 UTC m=+0.513706211 container remove 7024280520b2158248c5b1fe3b149d309f87206bb66e7aef0585dbb72718fd98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T00:29:36.603 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:29:36 smithi139 systemd[1]: ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.k.service: Deactivated successfully. 2024-04-25T00:29:36.603 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:29:36 smithi139 systemd[1]: Stopped Ceph mds.k for 518857f8-0297-11ef-bc93-c7b262605968. 2024-04-25T00:29:36.603 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 00:29:36 smithi139 systemd[1]: ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.k.service: Consumed 2.059s CPU time. 2024-04-25T00:29:36.613 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:29:36.613 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-25T00:29:36.613 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-25T00:29:36.614 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.c 2024-04-25T00:29:36.917 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:29:36 smithi153 systemd[1]: Stopping Ceph mds.c for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:29:37.179 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:29:36 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-c[63662]: 2024-04-25T00:29:36.916+0000 7f4a5ff9c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:29:37.179 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:29:36 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-c[63662]: 2024-04-25T00:29:36.916+0000 7f4a5ff9c700 -1 mds.c *** got signal Terminated *** 2024-04-25T00:29:40.060 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:29:39 smithi153 podman[70559]: 2024-04-25 00:29:39.777296143 +0000 UTC m=+2.966206726 container died 1da4ca3353f4fa644d85e23fff5e12eeca3a8c3b821020bfb1020d7364099f41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-c, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T00:29:40.410 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:29:40 smithi153 podman[70559]: 2024-04-25 00:29:40.15940239 +0000 UTC m=+3.348312974 container cleanup 1da4ca3353f4fa644d85e23fff5e12eeca3a8c3b821020bfb1020d7364099f41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T00:29:40.410 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 00:29:40 smithi153 bash[70559]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-c 2024-04-25T00:29:40.540 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.c.service' 2024-04-25T00:29:40.575 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:29:40.576 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-25T00:29:40.576 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-25T00:29:40.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.f 2024-04-25T00:29:40.776 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:29:40 smithi153 systemd[1]: Stopping Ceph mds.f for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:29:41.276 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:29:40 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-f[63355]: 2024-04-25T00:29:40.896+0000 7f54f3501700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:29:41.277 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:29:40 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-f[63355]: 2024-04-25T00:29:40.896+0000 7f54f3501700 -1 mds.f *** got signal Terminated *** 2024-04-25T00:29:44.026 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:29:43 smithi153 podman[70692]: 2024-04-25 00:29:43.776659712 +0000 UTC m=+2.986411873 container died fe42e8756c601342a29b81dd6154e1e625be3453b81afabbd5741faa749009a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T00:29:44.505 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:29:44 smithi153 podman[70692]: 2024-04-25 00:29:44.279193094 +0000 UTC m=+3.488945237 container cleanup fe42e8756c601342a29b81dd6154e1e625be3453b81afabbd5741faa749009a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-f, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T00:29:44.505 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 00:29:44 smithi153 bash[70692]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-f 2024-04-25T00:29:44.646 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.f.service' 2024-04-25T00:29:44.679 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:29:44.680 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-25T00:29:44.680 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-25T00:29:44.680 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.i 2024-04-25T00:29:45.016 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:29:44 smithi153 systemd[1]: Stopping Ceph mds.i for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:29:45.276 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:29:45 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-i[63969]: 2024-04-25T00:29:45.015+0000 7fd5b7289700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:29:45.276 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:29:45 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-i[63969]: 2024-04-25T00:29:45.015+0000 7fd5b7289700 -1 mds.i *** got signal Terminated *** 2024-04-25T00:29:47.991 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:29:47 smithi153 podman[70815]: 2024-04-25 00:29:47.78193207 +0000 UTC m=+2.881903754 container died 4a115f16103e9e64e5a69a59c161aa21f323a121d19946a3a42de572e80940c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-25T00:29:48.276 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:29:48 smithi153 podman[70815]: 2024-04-25 00:29:48.074045022 +0000 UTC m=+3.174016716 container cleanup 4a115f16103e9e64e5a69a59c161aa21f323a121d19946a3a42de572e80940c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-25T00:29:48.277 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 00:29:48 smithi153 bash[70815]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-i 2024-04-25T00:29:48.455 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.i.service' 2024-04-25T00:29:48.486 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:29:48.486 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-25T00:29:48.487 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-25T00:29:48.487 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.l 2024-04-25T00:29:48.776 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:29:48 smithi153 systemd[1]: Stopping Ceph mds.l for 518857f8-0297-11ef-bc93-c7b262605968... 2024-04-25T00:29:49.276 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:29:48 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-l[63030]: 2024-04-25T00:29:48.798+0000 7fbdf640b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T00:29:49.276 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:29:48 smithi153 ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-l[63030]: 2024-04-25T00:29:48.798+0000 7fbdf640b700 -1 mds.l *** got signal Terminated *** 2024-04-25T00:29:52.069 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:29:51 smithi153 podman[70935]: 2024-04-25 00:29:51.780425906 +0000 UTC m=+3.079008723 container died c058c58a68b601d0ec629950f6e70b581c1e485d43b0a47cdae667600b23ffd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-25T00:29:52.436 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:29:52 smithi153 podman[70935]: 2024-04-25 00:29:52.152339237 +0000 UTC m=+3.450922049 container cleanup c058c58a68b601d0ec629950f6e70b581c1e485d43b0a47cdae667600b23ffd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-25T00:29:52.436 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 00:29:52 smithi153 bash[70935]: ceph-518857f8-0297-11ef-bc93-c7b262605968-mds-l 2024-04-25T00:29:52.570 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-518857f8-0297-11ef-bc93-c7b262605968@mds.l.service' 2024-04-25T00:29:52.601 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T00:29:52.601 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-25T00:29:52.602 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 518857f8-0297-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T00:29:52.800 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:29:55.567 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 518857f8-0297-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T00:29:55.757 INFO:teuthology.orchestra.run.smithi139.stdout:Deleting cluster with fsid: 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:29:58.258 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 518857f8-0297-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T00:29:58.447 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:30:00.860 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T00:30:00.893 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T00:30:00.944 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T00:30:00.969 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-25T00:30:00.971 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672232/remote/smithi027/crash 2024-04-25T00:30:00.972 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/crash -- . 2024-04-25T00:30:01.009 INFO:teuthology.orchestra.run.smithi027.stderr:tar: /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T00:30:01.009 INFO:teuthology.orchestra.run.smithi027.stderr:tar: Error is not recoverable: exiting now 2024-04-25T00:30:01.010 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672232/remote/smithi139/crash 2024-04-25T00:30:01.011 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/crash -- . 2024-04-25T00:30:01.051 INFO:teuthology.orchestra.run.smithi139.stderr:tar: /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T00:30:01.051 INFO:teuthology.orchestra.run.smithi139.stderr:tar: Error is not recoverable: exiting now 2024-04-25T00:30:01.054 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672232/remote/smithi153/crash 2024-04-25T00:30:01.055 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/crash -- . 2024-04-25T00:30:01.092 INFO:teuthology.orchestra.run.smithi153.stderr:tar: /var/lib/ceph/518857f8-0297-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T00:30:01.092 INFO:teuthology.orchestra.run.smithi153.stderr:tar: Error is not recoverable: exiting now 2024-04-25T00:30:01.093 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-25T00:30:01.093 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-25T00:30:01.193 INFO:tasks.cephadm:Compressing logs... 2024-04-25T00:30:01.194 DEBUG:teuthology.orchestra.run.smithi027:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T00:30:01.196 DEBUG:teuthology.orchestra.run.smithi139:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T00:30:01.199 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T00:30:01.224 INFO:teuthology.orchestra.run.smithi139.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.log 2024-04-25T00:30:01.224 INFO:teuthology.orchestra.run.smithi139.stderr:: No such file or directory 2024-04-25T00:30:01.227 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T00:30:01.227 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-25T00:30:01.227 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-25T00:30:01.227 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-25T00:30:01.227 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-25T00:30:01.227 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-25T00:30:01.227 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-25T00:30:01.227 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-25T00:30:01.228 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T00:30:01.228 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-25T00:30:01.229 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T00:30:01.229 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-25T00:30:01.229 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-25T00:30:01.229 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T00:30:01.232 INFO:teuthology.orchestra.run.smithi027.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-25T00:30:01.234 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.log 2024-04-25T00:30:01.235 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T00:30:01.235 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-25T00:30:01.235 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-25T00:30:01.235 INFO:teuthology.orchestra.run.smithi027.stderr:gzip/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.g.log: -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-25T00:30:01.236 INFO:teuthology.orchestra.run.smithi153.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-25T00:30:01.237 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-25T00:30:01.237 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-25T00:30:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-25T00:30:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.0.log: /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-25T00:30:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T00:30:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mon.a.log: /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-25T00:30:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T00:30:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.cephadm.log: 87.3% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T00:30:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-25T00:30:01.240 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-25T00:30:01.240 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T00:30:01.241 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-25T00:30:01.241 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/cephadm.log: 94.6% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-25T00:30:01.243 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.log 2024-04-25T00:30:01.243 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-25T00:30:01.243 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-25T00:30:01.243 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T00:30:01.243 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.f.log: --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-25T00:30:01.243 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-25T00:30:01.243 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-25T00:30:01.243 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.5.log: /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-25T00:30:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-25T00:30:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T00:30:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mgr.z.log: /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-25T00:30:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-25T00:30:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T00:30:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-25T00:30:01.244 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T00:30:01.280 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.log: 93.8% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-25T00:30:01.334 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 92.9% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-25T00:30:01.335 INFO:teuthology.orchestra.run.smithi027.stderr: 94.0% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-25T00:30:01.335 INFO:teuthology.orchestra.run.smithi027.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-25T00:30:01.349 INFO:teuthology.orchestra.run.smithi027.stderr: 91.1% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T00:30:01.366 INFO:teuthology.orchestra.run.smithi153.stderr: 92.1% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-25T00:30:01.377 INFO:teuthology.orchestra.run.smithi153.stderr: 84.0% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T00:30:01.384 INFO:teuthology.orchestra.run.smithi153.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T00:30:01.386 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 87.3% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T00:30:01.418 INFO:teuthology.orchestra.run.smithi153.stderr: 90.2% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T00:30:01.424 INFO:teuthology.orchestra.run.smithi027.stderr: 84.2% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T00:30:01.429 INFO:teuthology.orchestra.run.smithi027.stderr: 90.0% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T00:30:01.435 INFO:teuthology.orchestra.run.smithi139.stderr: 91.0% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T00:30:01.474 INFO:teuthology.orchestra.run.smithi139.stderr: 84.1% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T00:30:01.479 INFO:teuthology.orchestra.run.smithi139.stderr: 90.1% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T00:30:01.493 INFO:teuthology.orchestra.run.smithi139.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T00:30:01.546 INFO:teuthology.orchestra.run.smithi027.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T00:30:01.559 INFO:teuthology.orchestra.run.smithi153.stderr: 92.8% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-25T00:30:01.565 INFO:teuthology.orchestra.run.smithi027.stderr: 94.6% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-25T00:30:01.593 INFO:teuthology.orchestra.run.smithi153.stderr: 87.3% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T00:30:01.637 INFO:teuthology.orchestra.run.smithi139.stderr: 93.0% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-25T00:30:01.638 INFO:teuthology.orchestra.run.smithi153.stderr: 91.0% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T00:30:01.687 INFO:teuthology.orchestra.run.smithi139.stderr: 92.0% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-25T00:30:02.137 INFO:teuthology.orchestra.run.smithi139.stderr: 94.0% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-25T00:30:02.733 INFO:teuthology.orchestra.run.smithi153.stderr: 94.1% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-25T00:30:02.955 INFO:teuthology.orchestra.run.smithi027.stderr: 90.3% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-25T00:30:03.264 INFO:teuthology.orchestra.run.smithi153.stderr: 93.1% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-25T00:30:05.207 INFO:teuthology.orchestra.run.smithi153.stderr: 91.8% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-25T00:30:06.899 INFO:teuthology.orchestra.run.smithi139.stderr: 91.9% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-25T00:30:11.919 INFO:teuthology.orchestra.run.smithi139.stderr: 95.2% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-25T00:30:13.632 INFO:teuthology.orchestra.run.smithi027.stderr: 90.4% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-25T00:30:37.671 INFO:teuthology.orchestra.run.smithi139.stderr: 96.5% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-25T00:34:19.036 INFO:teuthology.orchestra.run.smithi153.stderr: 91.7% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-25T00:41:47.095 INFO:teuthology.orchestra.run.smithi153.stderr: 91.9% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-25T00:43:13.523 INFO:teuthology.orchestra.run.smithi139.stderr: 92.1% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-25T00:45:07.367 INFO:teuthology.orchestra.run.smithi153.stderr: 91.6% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-25T00:45:44.398 INFO:teuthology.orchestra.run.smithi153.stderr: 91.5% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-25T00:45:44.400 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-04-25T00:45:44.400 INFO:teuthology.orchestra.run.smithi153.stderr:real 15m43.186s 2024-04-25T00:45:44.400 INFO:teuthology.orchestra.run.smithi153.stderr:user 7m39.869s 2024-04-25T00:45:44.400 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m19.276s 2024-04-25T00:45:46.152 INFO:teuthology.orchestra.run.smithi139.stderr: 91.9% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-25T00:48:35.833 INFO:teuthology.orchestra.run.smithi027.stderr: 91.6% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-25T00:49:34.692 INFO:teuthology.orchestra.run.smithi139.stderr: 91.7% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-25T00:49:53.380 INFO:teuthology.orchestra.run.smithi139.stderr: 91.4% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-25T00:49:53.382 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-04-25T00:49:53.382 INFO:teuthology.orchestra.run.smithi139.stderr:real 19m52.168s 2024-04-25T00:49:53.382 INFO:teuthology.orchestra.run.smithi139.stderr:user 7m15.100s 2024-04-25T00:49:53.382 INFO:teuthology.orchestra.run.smithi139.stderr:sys 0m19.396s 2024-04-25T00:50:34.192 INFO:teuthology.orchestra.run.smithi027.stderr: 91.5% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-25T00:54:50.852 INFO:teuthology.orchestra.run.smithi027.stderr: 91.7% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-25T00:55:10.776 INFO:teuthology.orchestra.run.smithi027.stderr: 90.6% -- replaced with /var/log/ceph/518857f8-0297-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-25T00:55:10.824 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-04-25T00:55:10.825 INFO:teuthology.orchestra.run.smithi027.stderr:real 25m9.614s 2024-04-25T00:55:10.825 INFO:teuthology.orchestra.run.smithi027.stderr:user 10m10.966s 2024-04-25T00:55:10.825 INFO:teuthology.orchestra.run.smithi027.stderr:sys 0m25.144s 2024-04-25T00:55:10.825 INFO:tasks.cephadm:Archiving logs... 2024-04-25T00:55:10.826 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672232/remote/smithi027/log 2024-04-25T00:55:10.827 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T00:56:52.444 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672232/remote/smithi139/log 2024-04-25T00:56:52.448 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T00:58:05.965 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672232/remote/smithi153/log 2024-04-25T00:58:05.966 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T00:59:23.025 INFO:tasks.cephadm:Removing cluster... 2024-04-25T00:59:23.026 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 518857f8-0297-11ef-bc93-c7b262605968 --force 2024-04-25T00:59:23.678 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:59:24.626 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 518857f8-0297-11ef-bc93-c7b262605968 --force 2024-04-25T00:59:25.214 INFO:teuthology.orchestra.run.smithi139.stdout:Deleting cluster with fsid: 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:59:25.954 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 518857f8-0297-11ef-bc93-c7b262605968 --force 2024-04-25T00:59:26.430 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: 518857f8-0297-11ef-bc93-c7b262605968 2024-04-25T00:59:27.237 INFO:tasks.cephadm:Removing cephadm ... 2024-04-25T00:59:27.237 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T00:59:27.257 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T00:59:27.276 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T00:59:27.292 INFO:tasks.cephadm:Teardown complete 2024-04-25T00:59:27.292 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T00:59:27.305 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T00:59:27.305 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T00:59:27.308 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T00:59:27.319 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T00:59:27.460 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T00:59:27.471 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T00:59:27.471 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T00:59:27.474 DEBUG:teuthology.orchestra.run.smithi139:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T00:59:27.476 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T00:59:27.488 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-04-25T00:59:27.490 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-04-25T00:59:27.491 INFO:teuthology.orchestra.run.smithi139.stderr:bash: line 1: ntpq: command not found 2024-04-25T00:59:27.506 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T00:59:27.507 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-04-25T00:59:27.507 INFO:teuthology.orchestra.run.smithi153.stdout:^- hv01.front.sepia.ceph.com 2 6 377 60 +1023us[+1031us] +/- 569ms 2024-04-25T00:59:27.507 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -100us[ -93us] +/- 71ms 2024-04-25T00:59:27.507 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 +136us[ +136us] +/- 80ms 2024-04-25T00:59:27.507 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T00:59:27.527 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T00:59:27.527 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2024-04-25T00:59:27.527 INFO:teuthology.orchestra.run.smithi139.stdout:^- hv01.front.sepia.ceph.com 2 6 377 58 +1053us[+1053us] +/- 569ms 2024-04-25T00:59:27.527 INFO:teuthology.orchestra.run.smithi139.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -116us[ -106us] +/- 71ms 2024-04-25T00:59:27.527 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +144us[ +144us] +/- 80ms 2024-04-25T00:59:27.527 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T00:59:27.576 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T00:59:27.576 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-04-25T00:59:27.576 INFO:teuthology.orchestra.run.smithi027.stdout:^- hv01.front.sepia.ceph.com 2 6 377 63 +1010us[+1006us] +/- 569ms 2024-04-25T00:59:27.576 INFO:teuthology.orchestra.run.smithi027.stdout:^* hv02.front.sepia.ceph.com 3 6 377 54 -118us[ -121us] +/- 71ms 2024-04-25T00:59:27.576 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +135us[ +131us] +/- 80ms 2024-04-25T00:59:27.576 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T00:59:27.594 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T00:59:27.605 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T00:59:27.606 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T00:59:27.616 DEBUG:teuthology.orchestra.run.smithi027:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T00:59:27.802 DEBUG:teuthology.orchestra.run.smithi139:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T00:59:27.886 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T00:59:28.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T00:59:28.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:59:28.056 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T00:59:28.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:59:28.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T00:59:28.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:59:28.117 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T00:59:28.128 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T00:59:28.139 INFO:teuthology.task.internal:Duration was 3731.570841 seconds 2024-04-25T00:59:28.139 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T00:59:28.150 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T00:59:28.150 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T00:59:28.153 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T00:59:28.156 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T00:59:28.206 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T00:59:28.208 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T00:59:28.224 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T00:59:28.576 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T00:59:28.576 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T00:59:28.576 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T00:59:28.609 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi139.front.sepia.ceph.com 2024-04-25T00:59:28.610 DEBUG:teuthology.orchestra.run.smithi139:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T00:59:28.641 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-04-25T00:59:28.641 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T00:59:28.674 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T00:59:28.675 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T00:59:28.677 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T00:59:28.684 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T00:59:28.744 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T00:59:28.744 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T00:59:28.747 DEBUG:teuthology.orchestra.run.smithi139:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T00:59:28.757 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T00:59:29.249 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T00:59:29.262 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T00:59:29.262 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T00:59:29.316 DEBUG:teuthology.orchestra.run.smithi139:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T00:59:29.345 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T00:59:29.393 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T00:59:29.405 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T00:59:29.407 DEBUG:teuthology.orchestra.run.smithi139:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T00:59:29.410 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T00:59:29.438 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = core 2024-04-25T00:59:29.439 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2024-04-25T00:59:29.448 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-04-25T00:59:29.475 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T00:59:29.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:59:29.518 DEBUG:teuthology.orchestra.run.smithi139:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T00:59:29.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:59:29.532 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T00:59:29.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:59:29.546 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T00:59:29.558 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T00:59:29.560 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672232/remote/smithi027 2024-04-25T00:59:29.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T00:59:29.602 DEBUG:teuthology.misc:Transferring archived files from smithi139:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672232/remote/smithi139 2024-04-25T00:59:29.602 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T00:59:29.637 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672232/remote/smithi153 2024-04-25T00:59:29.638 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T00:59:29.675 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T00:59:29.676 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T00:59:29.678 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T00:59:29.681 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T00:59:29.724 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T00:59:29.736 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T00:59:29.736 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T00:59:29.748 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T00:59:29.748 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T00:59:29.750 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T00:59:29.753 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T00:59:29.762 INFO:teuthology.orchestra.run.smithi027.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 00:59 /home/ubuntu/cephtest 2024-04-25T00:59:29.764 INFO:teuthology.orchestra.run.smithi139.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 00:59 /home/ubuntu/cephtest 2024-04-25T00:59:29.779 INFO:teuthology.orchestra.run.smithi153.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 00:59 /home/ubuntu/cephtest 2024-04-25T00:59:29.786 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T00:59:29.799 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T00:59:29.988 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} duration: 3731.5708413124084 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-25T00:59:29.988 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T00:59:30.087 INFO:teuthology.run:pass